mcpskills.net — Curated AI Skills, MCPs & Agents

mcpskills.net

Code you can't observe is code you can't operate. Instrumentation is written alongside the feature, not as a post-launch add-on.

When to Use

Building any feature that will run in production
Adding a new service, endpoint, background job, or external integration
A production incident took too long to diagnose
Setting up or reviewing alerting rules

Process

1. Define "working" before instrumenting

Before adding any instrumentation, write down 2–4 questions an on-call engineer will ask:

FEATURE: checkout payment retry
QUESTIONS ON-CALL WILL ASK:
1. What fraction of payments succeed on first attempt vs after retry?
2. When a payment fails permanently, why?
3. Is the payment provider slower than usual?
→ Every signal below must help answer one of these.

2. Pick the right signal for each question

| Signal | Answers | Cost profile | |---|---|---| | Structured log | "What happened in this specific case?" | Per-event | | Metric | "How often / how fast, in aggregate?" | Fixed per series | | Trace | "Where did time go across services?" | Per-request |

Metrics tell you that something is wrong, traces tell you where, logs tell you why.

3. Structured Logging

Log events, not prose:

// BAD: string interpolation — unqueryable
logger.info(`Payment ${id} failed for user ${userId} after ${n} retries`);

// GOOD: stable event name + structured fields
logger.warn({
  event: 'payment_failed',
  paymentId: id,
  provider: 'stripe',
  errorCode: err.code,
  attempt: n,
}, 'payment failed');

Log levels:

| Level | Meaning | On-call action | |---|---|---| | error | Invariant broken; someone may need to act | Investigate | | warn | Degraded but handled | Watch for trends | | info | Significant business event | None | | debug | Diagnostic detail | Off in production |

Correlation IDs are mandatory:

app.use((req, res, next) => {
  req.id = req.headers['x-request-id'] ?? crypto.randomUUID();
  req.log = logger.child({ requestId: req.id });
  res.setHeader('x-request-id', req.id);
  next();
});

4. Metrics

For request-driven services, instrument RED on every endpoint: Rate, Errors, Duration.

import { Histogram } from 'prom-client';

const httpDuration = new Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration',
  labelNames: ['method', 'route', 'status_class'],
  buckets: [0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
});

Cardinality is the failure mode. Never use user IDs, raw URLs, or error messages as labels.

Track averages never, percentiles always.

5. Distributed Tracing

Use OpenTelemetry — it's the vendor-neutral standard:

import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';

const sdk = new NodeSDK({
  serviceName: 'checkout-service',
  instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();

6. Alerting

Alert on symptoms users feel, not on causes:

SYMPTOM (page-worthy):           CAUSE (dashboard, not a page):
error rate > 1% for 5 min        CPU at 85%
p99 latency > 2s                 one pod restarted
queue age > 10 min               disk at 70%

Rules for every alert:

It must be actionable. If the response is "ignore it", delete the alert.
It links to a runbook.
It has a threshold and duration justified by the SLO.

Verification

After instrumenting a feature:

[ ] The on-call questions are written down, and each signal maps to one
[ ] All log output is structured (JSON) with a correlation ID
[ ] No secrets or unredacted PII in any log line
[ ] RED metrics exist for every new endpoint
[ ] Latency is a histogram; p95/p99 are queryable
[ ] A single request can be followed end-to-end in the tracing UI
[ ] Every new alert is symptom-based with a runbook link
[ ] An induced failure in staging was located via telemetry alone