Structured Logging

Structured logging emits log records as machine-parseable JSON rather than freeform strings, enabling reliable indexing, filtering, and alerting without regex fragility. Every log record should carry a minimum set of contextual fields: timestamp, level, service, trace_id, span_id, and a human-readable message. PII redaction is non-negotiable in regulated environments — user identifiers must be masked or tokenized before the log is written, not filtered at the aggregation layer. Log levels (DEBUG, INFO, WARN, ERROR, FATAL) must be dynamically adjustable at runtime without restart to enable surgical debugging in production.

Key Points

• Always use an established logging library (Winston, Logback, structlog, Zap) — never format JSON manually; it breaks on nested objects and special characters
• Mandatory fields: timestamp (ISO-8601 with milliseconds), level, service name, version/commit SHA, trace_id, span_id, environment
• PII masking patterns: mask all but last 4 digits of user IDs, hash emails, redact credit card numbers using regex before serialization
• Dynamic log level changes (via config reload or feature flag) prevent log storms in production while retaining DEBUG capability when needed
• Avoid logging inside tight loops or per-row DB operations; use sampling (e.g., log 1-in-1000) or batch-level summaries instead
• Correlation with traces: always include trace_id and span_id from the active OpenTelemetry span context so logs can be linked to traces
• Log verbosity by severity: ERROR/FATAL always include full stack traces; WARN includes context fields; DEBUG includes request/response bodies
• Test log output as part of CI: assert that sensitive fields are absent and that required fields are present using log capture fixtures

Structured JSON log: PII-masked user_id, W3C-compatible trace/span IDs, and machine-parseable fields for alerting

{
  "timestamp": "2024-01-15T10:23:45.123Z",
  "level": "ERROR",
  "service": "payment-service",
  "trace_id": "abc123def456",
  "span_id": "789xyz",
  "user_id": "usr_****1234",
  "order_id": "ord_987654",
  "message": "Payment processing failed",
  "error": "InsufficientFunds",
  "duration_ms": 342,
  "env": "production"
}