Observability Basics: Traces, Metrics, Logs — Practical Guide (Sep 21, 2025)

Observability Basics: Traces, Metrics, Logs

Modern software systems are complex, often distributed, and dynamic. To maintain reliability, debug effectively, and optimize performance, engineers rely on
observability. This article demystifies observability’s three core pillars—traces, metrics, and logs—offering foundational concepts, actionable code examples, real-world usage scenarios, common pitfalls, and a handy checklist.

What Is Observability?

Observability refers to how well you can infer a system’s internal state based on the external outputs it generates. Unlike simple monitoring, which alerts you when something goes wrong, observability helps you understand why and how problems occur—even in complex, distributed environments.

The three pillars of observability—traces, metrics, and logs—each provide distinct types of information that complement one another:

Traces: Record the execution path of requests through a system, showing timing and causal relationships.
Metrics: Numeric measurements aggregated over time, like latency or error rates.
Logs: Timestamped, unstructured or structured text entries capturing detailed events and context.

1. Distributed Tracing: Understanding System Flow

Distributed tracing lets you track a single request as it travels across microservices, APIs, databases, and queues. This is crucial for debugging latency issues and understanding failure points in modern backend or cloud-native architectures.

Key Concepts

Span: A single unit of work with start and end timestamps (e.g., an HTTP request or DB query).
Trace: A tree or graph of spans forming the full journey of a request.
Context Propagation: Passing trace IDs through services, often using HTTP headers like traceparent.

Example: Adding OpenTelemetry tracing in a Node.js Express app

const express = require('express');
const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { registerInstrumentations } = require('@opentelemetry/instrumentation');
const { ExpressInstrumentation } = require('@opentelemetry/instrumentation-express');
const { SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const { ConsoleSpanExporter } = require('@opentelemetry/sdk-trace-base');

const provider = new NodeTracerProvider();
provider.addSpanProcessor(new SimpleSpanProcessor(new ConsoleSpanExporter()));
provider.register();

registerInstrumentations({
  instrumentations: [
    new ExpressInstrumentation(),
  ],
});

const app = express();

app.get('/checkout', (req, res) => {
  // Your business logic here
  res.send('Checkout processed');
});

app.listen(3000, () => console.log('Server running on http://localhost:3000'));

Tip: Use a distributed tracing backend like Jaeger or Honeycomb to visualize traces, especially in microservices or CI/CD pipelines where latency has many contributors.

2. Metrics: Quantitative Health Indicators

Metrics offer aggregated snapshots of system health and performance. Common metrics include request latency, CPU usage, error rates, or throughput. These enable dashboards and alerting, critical for both frontend (mobile/web) performance and backend/cloud infrastructure.

Metric Types

Counter: A value that only increases (e.g., number of requests).
Gauge: A value that can go up or down (e.g., CPU load).
Histogram: Distribution of latency or size measurements.

Example: Integrating Prometheus metrics in a Python Flask app

from flask import Flask
from prometheus_client import Counter, generate_latest, CONTENT_TYPE_LATEST

app = Flask(__name__)
REQUEST_COUNT = Counter('app_request_count', 'Total app requests', ['endpoint'])

@app.route('/metrics')
def metrics():
    return generate_latest(), 200, {'Content-Type': CONTENT_TYPE_LATEST}

@app.route('/login')
def login():
    REQUEST_COUNT.labels(endpoint='/login').inc()
    return "User logged in"

if __name__ == "__main__":
    app.run(port=5000)

In a cloud or CI/CD context, collecting metrics from your infrastructure (CPU, memory, disk), Kubernetes pods, and security tools helps detect anomalies early and supports capacity planning.

3. Logs: Rich Context for Troubleshooting

Logs capture detailed events and errors with rich context, essential for postmortem analysis and incident investigations. Logs are typically timestamped lines of text but should be structured (e.g., JSON) for easier querying and filtering.

Logging Best Practices

Structure Logs: Use structured logging (JSON) so you can filter by fields like request IDs, error codes, user IDs.
Include Context: Add trace IDs or user session IDs to correlate with traces and metrics.
Log at Appropriate Levels: Use levels such as DEBUG, INFO, WARN, ERROR appropriately to filter noise.

Example: Structured logging in a Go backend service

package main

import (
  "log"
  "os"
  "github.com/sirupsen/logrus"
)

func main() {
  log := logrus.New()
  log.Out = os.Stdout
  log.Formatter = &logrus.JSONFormatter{}

  traceID := "1234567890abcdef"
  userID := "user42"
  log.WithFields(logrus.Fields{
    "trace_id": traceID,
    "user_id":  userID,
  }).Info("User login succeeded")

  log.Warn("Disk space low")
  log.Error("Failed to load config file")
}

In a security or testing pipeline, logs are vital for audit trails and forensic investigation. Centralize them with tools like Elasticsearch or Splunk for effective searching and alerting.

Common Pitfalls and How to Avoid Them

Observability blind spots: Missing trace propagation or inconsistent logging headers can lead to fragmented data.
High cardinality in metrics: Avoid overly granular metric labels that explode time series and degrade performance.
Logs without context: Failing to correlate logs, traces, and metrics makes root cause analysis slow and painful.
Ignoring alert fatigue: Fine-tune alerts to actionable thresholds and combined signals rather than simple thresholds.
Not instrumenting early: Add observability from day one; retrofitting is harder and error-prone.

Final Checklist for Effective Observability

✅ Implement distributed tracing with context propagation across microservices and key components.
✅ Collect metrics for performance, errors, resource usage with proper aggregation and label hygiene.
✅ Use structured logging including trace and user/session identifiers to correlate events.
✅ Integrate observability data into centralized platforms (e.g., Prometheus + Grafana, Jaeger, ELK stack).
✅ Define meaningful alerts combining traces, metrics, and logs to reduce noise and escalate critical issues.
✅ Continuously review and evolve instrumentation as your architecture and business needs grow.

Conclusion

Observability—across