How do metrics, logs and traces differ, and what do you instrument first?

Metrics are pre-aggregated numeric time series - cheap to store and query, ideal for alerting, but they cannot explain an individual request. Logs are discrete structured events with full context - ideal for investigation, expensive at scale. Traces stitch one request's spans across every service, showing where the latency and the failure actually occurred. Instrument metrics first because they drive alerting: for every service, the RED metrics - request rate, error rate and duration percentiles - get you most of the way. | Metrics | Logs | Traces. Shape | Numeric time series | Discrete events | Spans in a causal tree. Cost at scale | Low | High | Medium (with sampling). Answers | Is it broken? Since when? | What exactly happened to this request? | Where did the time go?. Cardinality limits | Severe | None | Moderate. Retention | Months to years | Days to weeks | Days. Use for alerting | Yes | Rarely | No RED and USE: what to measure RED - for services: Rate (requests/sec), Errors (failures/sec or error ratio), Duration (p50/p95/p99). Three metrics per endpoint covers most of what you need.. USE - for resources: Utilisation (% busy), Saturation (queue depth or wait time), Errors. Applies to CPU, memory, disk, connection pools and thread pools.. The four golden signals are RED plus saturation - the same idea from a different source. Cardinality is what makes metrics expensive: A metric's cost is the product of its label cardinality. http_requestsmethod, status, endpoint with 5 methods × 10 statuses × 50 endpoints = 2,500 series: fine. Add user_id with a million users and you have 2.5 billion series, which will take down your metrics backend and produce a memorable bill. Never put unbounded identifiers in metric labels - user IDs, request IDs, URLs with path parameter

Observability: Metrics, Logs and Distributed Tracing Explained

IntermediateobservabilitymonitoringtracingAsked at Datadog, Google

30-second answer

Metrics are pre-aggregated numeric time series - cheap to store and query, ideal for alerting, but they cannot explain an individual request. Logs are discrete structured events with full context - ideal for investigation, expensive at scale. Traces stitch one request's spans across every service, showing where the latency and the failure actually occurred. Instrument metrics first because they drive alerting: for every service, the RED metrics - request rate, error rate and duration percentiles - get you most of the way.

	Metrics	Logs	Traces
Shape	Numeric time series	Discrete events	Spans in a causal tree
Cost at scale	Low	High	Medium (with sampling)
Answers	Is it broken? Since when?	What exactly happened to this request?	Where did the time go?
Cardinality limits	Severe	None	Moderate
Retention	Months to years	Days to weeks	Days
Use for alerting	Yes	Rarely	No

RED and USE: what to measure

RED - for services: Rate (requests/sec), Errors (failures/sec or error ratio), Duration (p50/p95/p99). Three metrics per endpoint covers most of what you need.
USE - for resources: Utilisation (% busy), Saturation (queue depth or wait time), Errors. Applies to CPU, memory, disk, connection pools and thread pools.
The four golden signals are RED plus saturation - the same idea from a different source.

jsonStructured logs: fields, not sentences

// ❌ Unparseable at scale - you cannot aggregate or filter on prose.
"ERROR: failed to charge user 4711 for order 991 after 3 retries"

// ✅ Structured: queryable, aggregatable, and joins to traces.
{
  "ts":        "2026-07-26T09:14:22.113Z",
  "level":     "error",
  "msg":       "payment_charge_failed",
  "service":   "payments",
  "trace_id":  "4bf92f3577b34da6a3ce929d0e0e4736",   // joins to the trace
  "span_id":   "00f067aa0ba902b7",
  "user_id":   "4711",
  "order_id":  "991",
  "attempts":  3,
  "provider":  "stripe",
  "error_code":"card_declined",
  "duration_ms": 1840
}

Alerting on symptoms, not causes

Bad alert (cause)	Good alert (symptom)
CPU above 80%	p99 checkout latency above 2 s for 5 minutes
Disk 75% full	Error rate above 1% for 3 minutes
A pod restarted	Successful order rate down 20% versus the same time last week
Queue depth above 10,000	Oldest unprocessed message older than 5 minutes

Cause alerts fire constantly and mean nothing on their own. Symptom alerts fire when users are affected - which is the only reason to wake someone.

Sampling traces without losing the interesting ones

Head sampling - decide at the start, e.g. keep 1%. Cheap and simple, but you will miss most errors because errors are rare.
Tail sampling - buffer the full trace and decide after it completes. Keep 100% of errors and slow requests, 1% of the rest. Much more useful, and needs a collector that can hold traces in flight.
Always keep anything with an error, anything above the p99 latency threshold, and a small random baseline so you can still compute distributions.

Say these points

Metrics alert, logs explain, traces localise - you need all three.
RED for services, USE for resources; instrument these first.
Never put unbounded identifiers in metric labels - cardinality is multiplicative.
Propagate a trace ID into every log line and return it to the client.
Alert on user-visible symptoms, and make every page actionable.

Avoid these mistakes

High-cardinality metric labels taking down the metrics backend.
Unstructured log messages that cannot be aggregated.
Alerting on CPU and disk instead of user-visible impact.
Head sampling only, so nearly every error trace is discarded.
Logging enough personally identifiable information to create a compliance problem.

Expect these follow-ups

A user reports a failure at 14:03. Walk me through your investigation.
How do you keep logging costs down without losing debuggability?
What would you instrument to detect a slow memory leak?

Observability: Logs, Metrics and Distributed Tracing