Skip to content

observability.md

Latest commit

 

History

History
152 lines (106 loc) · 4.1 KB

observability.md

File metadata and controls

152 lines (106 loc) · 4.1 KB
Raw

Observability

SearchOps Phase 6 adds small, local-friendly backend observability. It does not add a production monitoring stack.

Phase 7 keeps this scope unchanged and documents the implemented /metrics endpoint in OpenAPI and README. CI verifies the endpoint through backend tests; it does not provision Prometheus, Grafana, tracing, alerting, or production log aggregation.

Implemented In Phase 6

Structured Backend Logs

The backend uses Pino for JSON structured logs.

Startup, shutdown, request completion, unhandled request errors, unhandled promise rejections, and uncaught exceptions are logged through the shared application logger.

Request completion logs include:

  • requestId
  • method
  • path
  • route
  • statusCode
  • durationMs

Log levels:

  • info for normal completed requests
  • warn for slow completed requests
  • error for 5xx responses and unhandled errors

The default slow request threshold is 500ms. It can be changed locally with SLOW_REQUEST_THRESHOLD_MS.

Request IDs

The existing x-request-id behavior is preserved.

If a request includes x-request-id, the backend reuses it. Otherwise the backend generates one and returns it in the x-request-id response header and API response meta.requestId.

Metrics Endpoint

The backend exposes:

GET /metrics

This is an infrastructure endpoint and is not mounted under /api.

The response is Prometheus-style text and is not wrapped in the normal JSON API envelope.

Implemented metrics:

  • http_requests_total
  • http_errors_total
  • http_request_duration_ms histogram
  • search_latency_simulation_total

HTTP metrics use these labels:

  • method
  • route
  • statusCode

Example:

http_requests_total{method="GET",route="/api/jobs",statusCode="200"} 3
http_request_duration_ms_count{method="GET",route="/api/jobs",statusCode="200"} 3
http_errors_total{method="GET",route="/missing",statusCode="404"} 1
search_latency_simulation_total{route="/api/jobs"} 1

Local Search Latency Simulation

Phase 6 includes a controlled local-only incident simulation for GET /api/jobs.

The simulation only runs when both are true:

  • ENABLE_INCIDENT_SIMULATION=true
  • NODE_ENV is not production

The delay is triggered per request, not globally for every request.

Supported triggers:

GET /api/jobs?simulateSlowSearch=true

or:

x-searchops-simulate-latency-ms: 1200

The maximum simulated delay is capped at 3000ms.

When triggered, the backend:

  • delays only that GET /api/jobs request
  • increments search_latency_simulation_total
  • logs search latency simulation triggered with requestId, route, and delayMs
  • records the slower request in http_request_duration_ms

Local Inspection Commands

curl http://localhost:5000/health
curl http://localhost:5000/metrics
curl "http://localhost:5000/api/jobs?limit=3"
curl "http://localhost:5000/api/jobs?limit=3&simulateSlowSearch=true"
curl -H "x-searchops-simulate-latency-ms: 1200" "http://localhost:5000/api/jobs?limit=3"

Future Work

These are intentionally not implemented in Phase 6:

  • Prometheus server
  • Grafana dashboards
  • OpenTelemetry tracing
  • distributed tracing
  • alert rules or paging
  • log aggregation
  • log rotation
  • CloudWatch or other managed monitoring
  • production dashboards
  • authentication
  • rate limiting
  • deployment monitoring

Future production work could add those pieces around the Phase 6 logs and metrics. Until then, /metrics and container logs are local operational signals only.

Phase 6 Checklist

  • Pino JSON structured application logs
  • request completion logs with request IDs and durations
  • warning logs for slow requests
  • error logs for 5xx responses and unhandled request errors
  • GET /health preserved
  • GET /metrics Prometheus-style text endpoint
  • HTTP request, error, and duration metrics
  • local-only controlled search latency simulation
  • simulation metric and log signal
  • tracing
  • alerting
  • dashboards
  • production log aggregation

See incident-response.md for the local incident simulation runbook.