Every production SaaS should expose Prometheus-compatible metrics on a /metrics endpoint so on-call engineers can alert on user-visible failure—not CPU graphs alone. The practical baseline is RED (Rate, Errors, Duration): request throughput, error ratio, and latency distribution per route or service boundary. DarDev instruments internal APIs—sync workers, mail handlers, admin health checks—with this pattern before adding custom business counters.
If you already run Grafana, the gap is usually missing HTTP-level counters—not another node exporter. Start with RED on your three busiest endpoints.
RED metrics: rate, errors, duration
RED gives three PromQL-friendly signals without inventing a custom taxonomy. Rate comes from a counter such as http_requests_total, scraped every 15–30 seconds; derive requests per second with rate(http_requests_total[5m]). Errors filter the same counter by status=~"5.." or use a dedicated http_errors_total incremented only on server failures. Duration requires a histogram—never a gauge that averages latency inside the app, because averages hide tail latency that users feel.
- Rate: counter + rate() or increase() over a sliding window
- Errors: 5xx ratio or explicit error counter by error_type
- Duration: histogram with _bucket, _sum, and _count suffixes
Label with method, route template, and status. Background workers map RED to jobs_processed_total, jobs_failed_total, and job_duration_seconds.
/metrics endpoint patterns
Expose GET /metrics on a dedicated port or path, reachable only inside the cluster network—never on the public internet without mTLS or strict IP allowlists. In Kubernetes, annotate pods with prometheus.io/scrape, prometheus.io/path, and prometheus.io/port, or define a ServiceMonitor when using the Prometheus Operator.
- One process, one registry—avoid duplicate metric names across sidecars
- OpenMetrics text format; official client libraries emit HELP and TYPE lines
- Keep liveness on /health or /ready; do not fold probe traffic into RED counters
- Optional admin listener on :9090/metrics keeps scrape load off the customer-facing port
Fifteen-second scrape intervals suit most APIs. Fix cardinality before raising frequency. For how metrics fit with logs and traces, see logs vs metrics vs traces.
Histograms without guessing buckets
Histograms pre-aggregate latency into buckets chosen at instrumentation time. The Go client defaults favor short requests; for APIs with 200ms–5s tails, define buckets explicitly—for example 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, and 10 seconds.
Compute p95 with histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, route)). Do not treat _sum divided by _count as a percentile—that is a mean, not p95. Summaries with client-side quantiles are a poor fit for horizontally scaled replicas; prefer histograms so Prometheus aggregates across pods correctly.
Cardinality traps that take down Prometheus
Each unique label combination becomes a new time series. High-cardinality labels—user IDs, email addresses, raw URLs with UUIDs, unbounded exception messages—can explode scrape payload size and TSDB memory within hours.
- Normalize routes: /api/v1/invoices/{id}, not one series per invoice UUID
- Use a low-cardinality error_type enum instead of full stack traces in labels
- Never put tenant_id on every request counter unless you run a federation tier with budget
- Watch scrape_duration_seconds and series count after each deploy
Per-tenant SLOs belong in recording rules or traces—not unbounded labels on hot counters.
Instrumentation in Go and Node (concept level)
Go: register promhttp.Handler() on /metrics and wrap handlers with promhttp.InstrumentHandlerDuration plus a CounterVec labeled by method and route template. Node: use prom-client—a Registry, Counter, Histogram, and a /metrics route returning registry.metrics(). Record after routing resolves the handler so 401 and 500 hit the correct labels. Instrument the HTTP boundary first; add DB and cache histograms only where RED shows pain.
Alerts and SLOs that follow metrics
Page on error rate above threshold for five consecutive minutes and on p95 latency regression versus a one-week baseline on your top three routes. Tie deploy events to Prometheus annotations so on-call correlates spikes with pipeline IDs—concrete PromQL examples live in PromQL queries for deploy health.
After RED is stable, add error budgets per slos-internal-customer-apps. Until then, resist paging on every new counter.
What DarDev exports internally
On the unified mailer stack we scrape queue depth, SMTP counters, and HTTP histograms internally—not per-subscriber Listmonk metrics, which would overwhelm single-node Prometheus. See how we observe the Unified Mailer stack. Platform overview: dardev.net/products.
What is the minimum metric set for a new SaaS API?
http_requests_total (or equivalent) with method, route template, and status labels; http_request_duration_seconds histogram on the same labels; process and Go/Node runtime metrics from the official client defaults. Add business counters only after RED alerts work.
Should /metrics be public?
No. Expose it on an internal network, admin port, or ServiceMonitor target inside the cluster. If external monitoring must reach it, use mTLS, authentication, or a reverse proxy with IP allowlists—metric endpoints leak route names and label values useful to attackers.
Counter vs gauge for queue depth?
Queue depth is a gauge—it goes up and down. Counters only increase (reset on process restart). Use a gauge labeled by queue name; never increment a counter when messages are dequeued without a paired decrement pattern.
How many histogram buckets should we use?
Roughly ten buckets spanning your SLO range plus one +Inf bucket. Too few buckets flatten p99 estimates; too many add series overhead without better accuracy. Align bucket upper bounds with latency targets—e.g. 500ms and 2s if those are contractual SLO knees.
When do we need traces if we have RED metrics?
Metrics tell you that checkout p95 doubled; traces show which downstream call regressed. Keep RED for paging and SLO dashboards; add tracing when error counters spike but logs lack request correlation. The three signals complement each other—see logs vs metrics vs traces for decision criteria.
