Observability lab: metrics from zero

In this DarDevLab observability lab you install Prometheus, deploy node_exporter on a Linux host, confirm scrape targets are healthy, and build one Grafana panel that shows CPU utilization from real time-series data—not a screenshot from a tutorial. By the end you have a minimal metrics stack that mirrors what platform teams run before they add logs and traces.

DarDevLab teaches on production-grade stacks—GitLab CI, Kubernetes, Prometheus, Grafana. Finish this lab before our logs vs metrics vs traces guide or deploy-health PromQL modules.

What you will build

1. A single-node Prometheus server listening on port 9090 with persistent TSDB storage
2. node_exporter on the same VM (or a second lab VM) exposing host metrics on port 9100
3. A prometheus.yml scrape job named node that shows UP in Status → Targets
4. Grafana connected to Prometheus as a data source with one dashboard panel for CPU usage
5. A short homework submission: screenshot, scrape config snippet, and one-sentence explanation of the query

Lab prerequisites

Use Ubuntu 22.04+ with 2 vCPU, 4 GB RAM, and outbound HTTPS. Open ports 9090, 9100, and 3000 only on the lab VLAN or via SSH tunnel.

• SSH access with sudo on the lab host
• curl and tar installed (or Docker if your cohort uses the container track)
• A student GitLab repo or DarDevLab assignment folder for config commits
• Optional second VM to practice multi-target scraping—recommended for advanced cohorts

Step 1 — Install Prometheus

Download Prometheus for linux-amd64, create a prometheus user, install binaries to /usr/local/bin, and store TSDB data under /var/lib/prometheus. Prometheus pulls metrics from HTTP endpoints on the scrape schedule in prometheus.yml.

1. Create service user and directories

   sudo useradd --no-create-home --shell /bin/false prometheus. Create /etc/prometheus and /var/lib/prometheus owned by that user.

2. Install binaries

   Extract prometheus and promtool into /usr/local/bin. Place the default prometheus.yml into /etc/prometheus/prometheus.yml.

3. Systemd unit

   Write a unit file that sets --config.file=/etc/prometheus/prometheus.yml and --storage.tsdb.path=/var/lib/prometheus. Run sudo systemctl enable --now prometheus.

4. Verify

   curl -s localhost:9090/-/healthy should return Prometheus is Healthy. Open the UI and confirm Graph tab loads.

Step 2 — Install node_exporter

node_exporter exposes host CPU, memory, disk, and network metrics in Prometheus text format. One instance on the lab VM is enough; a second host is optional practice for multi-target scraping.

1. Download and install

   Fetch the node_exporter tarball matching your architecture. Install the binary to /usr/local/bin/node_exporter.

2. Systemd service

   Run as user node_exporter. Default listen address is :9100. Enable and start the service.

3. Smoke test

   curl -s localhost:9100/metrics | head should show lines like node_cpu_seconds_total and node_memory_MemAvailable_bytes.

Step 3 — Configure scraping

Edit /etc/prometheus/prometheus.yml: keep the self-scrape job on localhost:9090 and add a node job with static_configs targeting node_exporter (usually localhost:9100). Set scrape_interval to 15s. Restart Prometheus, then confirm Status → Targets shows both jobs UP. If node is DOWN, check the exporter service, firewall rules, and target addresses—document the fix in homework.

Step 4 — Grafana and your first panel

Install Grafana on port 3000, set the admin password, and add Prometheus as a data source (http://localhost:9090 or http://prometheus:9090 in Compose). Create one Time series panel with query: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100). Title it Lab CPU usage, set percent units, refresh 30s, and generate load with stress-ng so the line moves. See Grafana dashboards for small platform teams for layout patterns beyond this first panel.

Student homework rubric

Submit a PDF or Markdown file in the DarDevLab portal. Total: 100 points.

• Targets screenshot (20 pts): Prometheus Status → Targets with prometheus and node UP, including scrape URL and last scrape timestamp visible
• Config snippet (20 pts): prometheus.yml scrape job for node with correct targets and scrape_interval; comments explain why static_configs is used in this lab
• Grafana panel (25 pts): exported dashboard JSON or screenshot showing panel title, PromQL query, unit, and a visible data line after load test
• Query explanation (15 pts): two to four sentences describing what rate() does to node_cpu_seconds_total and why idle mode is subtracted from 100
• Failure recovery (20 pts): brief note documenting one issue you hit (firewall, typo, service not started) and the command or config change that fixed it

Common lab mistakes

• Leaving default prometheus.yml without a node job—Graph works but only self-metrics exist
• Querying node_cpu_seconds_total without rate()—Prometheus returns counters that look like a ramp, not a percentage
• Pointing Grafana at the wrong URL inside Docker networks (localhost inside the container is not the host)
• Opening 9090 and 9100 to 0.0.0.0/0 on a cloud VM—lab policy violation and a security deduction
• Skipping homework config snippets and submitting only screenshots—instructors cannot verify reproducibility

What comes next in DarDevLab

Next modules cover Alertmanager, application /metrics endpoints (Prometheus metrics every SaaS should expose), and deploy correlation. Enroll at dardevlab.com for graded cohorts and certification prep.