Observability and Instrumentation

Code you can't observe is code you can't operate. Instrumentation is written alongside the feature, not as a post-launch add-on.

When to Use

• Building any feature that will run in production
• Adding a new service, endpoint, background job, or external integration
• A production incident took too long to diagnose
• Setting up or reviewing alerting rules

Process

1. Define "working" before instrumenting

Before adding any instrumentation, write down 2–4 questions an on-call engineer will ask:

FEATURE: checkout payment retry
QUESTIONS ON-CALL WILL ASK:
1. What fraction of payments succeed on first attempt vs after retry?
2. When a payment fails permanently, why?
3. Is the payment provider slower than usual?
→ Every signal below must help answer one of these.

2. Pick the right signal for each question

Signal	Answers	Cost profile
Structured log	"What happened in this specific case?"	Per-event
Metric	"How often / how fast, in aggregate?"	Fixed per series
Trace	"Where did time go across services?"	Per-request

Metrics tell you that something is wrong, traces tell you where, logs tell you why.

3. Structured Logging

Log events, not prose:

// BAD: string interpolation — unqueryable
logger.info(`Payment ${id} failed for user ${userId} after ${n} retries`);

// GOOD: stable event name + structured fields
logger.warn({
  event: 'payment_failed',
  paymentId: id,
  provider: 'stripe',
  errorCode: err.code,
  attempt: n,
}, 'payment failed');

Log levels:

Level	Meaning	On-call action
error	Invariant broken; someone may need to act	Investigate
warn	Degraded but handled	Watch for trends
info	Significant business event	None
debug	Diagnostic detail	Off in production

Correlation IDs are mandatory:

app.use((req, res, next) => {
  req.id = req.headers['x-request-id'] ?? crypto.randomUUID();
  req.log = logger.child({ requestId: req.id });
  res.setHeader('x-request-id', req.id);
  next();
});

4. Metrics

For request-driven services, instrument RED on every endpoint: Rate, Errors, Duration.

import { Histogram } from 'prom-client';

const httpDuration = new Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration',
  labelNames: ['method', 'route', 'status_class'],
  buckets: [0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
});

Cardinality is the failure mode. Never use user IDs, raw URLs, or error messages as labels.

Track averages never, percentiles always.

5. Distributed Tracing

Use OpenTelemetry — it's the vendor-neutral standard:

import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';

const sdk = new NodeSDK({
  serviceName: 'checkout-service',
  instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();

6. Alerting

Alert on symptoms users feel, not on causes:

SYMPTOM (page-worthy):           CAUSE (dashboard, not a page):
error rate > 1% for 5 min        CPU at 85%
p99 latency > 2s                 one pod restarted
queue age > 10 min               disk at 70%

Rules for every alert:

1. It must be actionable. If the response is "ignore it", delete the alert.
2. It links to a runbook.
3. It has a threshold and duration justified by the SLO.

Verification

After instrumenting a feature:

• The on-call questions are written down, and each signal maps to one
• All log output is structured (JSON) with a correlation ID
• No secrets or unredacted PII in any log line
• RED metrics exist for every new endpoint
• Latency is a histogram; p95/p99 are queryable
• A single request can be followed end-to-end in the tracing UI
• Every new alert is symptom-based with a runbook link
• An induced failure in staging was located via telemetry alone