Distributed Observability In Rust – Manjunath Gangappa

Configuring Services to Export Each service needs two environment variables to know where to send telemetry: # In docker-compose.yml, for each service: environment: – OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://jaeger:4317 – OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://prometheus:9090/api/v1/otlp/v1/metrics – OTEL_SERVICE_NAME=orders # Becomes the ‘job’ label in Prometheus Notice the different protocols: traces use gRPC (port 4317), metrics use HTTP with a specific path ( /api/v1/otlp/v1/metrics ). Our init_telemetry function from Section 6.5 reads these environment variables ( OTEL_EXPORTER_OTLP_TRACES_ENDPOINT and OTEL_EXPORTER_OTLP_METRICS_ENDPOINT , falling back to OTEL_EXPORTER_OTLP_ENDPOINT ) and configures the exporters accordingly.

Anything passed explicitly to .with_endpoint(…) in code overrides whatever the environment provides. The OTEL_SERVICE_NAME is particularly important for metrics. Prometheus converts this to a job label, which we use in queries to filter by service: http_server_request_duration_seconds_count{job=”orders”} . Verifying the Pipeline Once the stack starts, you can verify the pipeline is working by querying Prometheus directly: curl ‘http://localhost:9090/api/v1/query?query=http_server_request_duration_seconds_count’ \ | jq ‘.data.result[] | {job: .metric.job}’ This queries for HTTP request metrics and groups them by service.

A working pipeline shows: {“job”: “products”} {“job”: “inventory”} {“job”: “orders”} {“job”: “otelmart”} If metrics don’t appear, the problem is usually one of three things: 1. Services can’t reach Prometheus : Check Docker networking and that services can resolve prometheus:9090 2. Prometheus didn’t start with OTLP enabled : Verify ‑‑web.enable‑otlp‑receiver in the command 3. Exporters are configured incorrectly : Check service logs for OTLP export errors The beauty of this architecture is its simplicity.

No agents, no sidecars, no collectors. Services push, Prometheus stores, Grafana visualizes. This directness makes debugging straightforward that follow the path from instrument to storage. Dashboards: Visualizing the Request Journey Metrics in Prometheus are useful for querying, but their power emerges when visualized. Grafana transforms time series data into dashboards that reveal patterns invisible in raw numbers.

This section teaches you to build dashboards that show system health at a glance. Understanding Metric Name Transformations Before querying metrics in Grafana, you need to understand how Prometheus stores OTLP metrics.

Mastering Distributed Observability in Rust Copyright © 2026 Packt Publishing All rights reserved . No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented.

However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals.

However, Packt Publishing cannot guarantee the accuracy of this information. Portfolio Director: Kunal Chaudhary Relationship Lead: Srishti Seth Pro gram Manager : K. Loganathan Project Manager : Ankit Maroli Content Engineer : Sushma Reddy Technical Editor: Irfa Ansari Indexer: Rekha Nair Production Designer: Prashant Ghare Growth Lead : Vinishka Kalra First published: June 2026 Production reference: 1230626 Published by Packt Publishing Ltd.

Grosvenor House 11 St Paul’s Square Birmingham B3 1RB, UK. ISBN 978-1-80667-179-3 www.packtpub.com To my wife Shreedevi, for her unwavering support; Kushan and Gagan, for their constant inspiration; and my parents, my foundation. — Manjunath Gangappa To my wife, Amsaveni, for her patience, love, and encouragement through this journey. To my sons, Nikil Pranav and Reyan, whose curiosity, energy, and smiles make every day brighter. To my parents, Rangaraj and Chandravadivu, whose sacrifices and values shaped the person I am today.

And to my sister, Latha, for her constant love, encouragement, and support. — Rajkumar Rangaraj Foreword I serve on the OpenTelemetry Governance Committee and maintain the OpenTelemetry semantic conventions, including HTTP, database, and Generative AI (GenAI) conventions, the parts of the standard used throughout this book. Which is why I was eager to read this book.

This is a short excerpt from the opening of “” by Unknown, quoted for review and introduction purposes. All rights belong to the copyright holders.

Book Information

• Unique ID: 180122817997d992
• File Extension: .pdf
• File Size: 25,192,972 bytes (24.026 MB)
• Title: –
• Author: Unknown
• ISBN: 9781806671793
• Pages: 529
• Language: English (en)

Reading & Word Statistics

• Estimated Reading Time: 547.83 minutes
• Total Words: 109,566
• Total Characters: 783,422
• Average Words per Page: 207.12
• Average Characters per Page: 1480.95

Most Frequent Words

service (607), span (594), metrics (566), checkout (496), http (493), trace (475), request (454), chapter (431), error (422), orders (421), pool (367), tracing (364), database (359), inventory (335), traces (320), opentelemetry (317), name (316), rate (304), time (303), product (292), let (285), use (281), order (272), duration (266), observability (263), query (261), logs (260), business (259), code (253), async (247), new (247), system (245), log (245), spans (243), latency (243), data (235), telemetry (228), transaction (226), products (225), stock (225), services (222), client (222), context (207), requests (202), otelmart (201), rust (194), every (194), state (193), wait (192), response (184), incident (175), total (175), model (172), jaeger (171), across (165), without (163), instrumentation (159), connection (156), seconds (155), slow (154), server (152), one (151), dashboard (151), memory (150), instrument (150), single (148), otlp (146), see (144), note (143), like (142), record (141), lock (139), uuid (138), operation (137), layer (135), three (133), attributes (133), await (133), user (132), distributed (131), tokio (130), prometheus (130), task (129), count (129), queries (127), blocking (125), budget (123), build (122), call (119), metric (118), errors (117), per (117), api (117), select (117), gen (117), histogram (116), provider (114), reserve (114), now (113), loki (113).

PDF Download