Monitoring & Alerting Design: From Metrics Collection to Alert Convergence

The Bottom Line: SLO First, Monitoring Second, Alerting Last

The most common mistake is doing it in reverse — building metric collection first, adding alert rules second, and then struggling with hundreds of daily alerts until you cannot tell what matters.

The correct order:

1. Define SLOs — user-visible metrics and thresholds
2. Set up collection & storage — only the metrics you need
3. Build dashboards — make SLOs visible at a glance
4. Configure alert convergence — alert only on SLO violations, with noise suppression

---

1. SLO First: What Deserves an Alert

Not every anomaly is worth an alert. Only alert on things the user can feel.

Recommended Alert Trio

Availability SLO: API success rate ≥ 99.9%
Latency SLO:    API P99 response time ≤ 500ms
Freshness SLO:  Data sync delay ≤ 5 minutes

Do NOT Alert On (Put These on a Dashboard Instead)

Metric	Why Not
CPU > 80%	Not user-visible; monitor capacity instead
Disk > 85%	Operational item; schedule cleanup or auto-scale
Single 5xx error	Rate, not absolute count — 1/10000 is normal
Single Pod restart	Frequency matters; occasional restarts are fine

---

2. Collection Layer: Prometheus + Exporters

Architecture

App service (/metrics) → Prometheus Server → Grafana
Node Exporter         → Prometheus Server → Alertmanager
PostgreSQL Exporter   → Prometheus Server     ↓
                                          Notifications

Key Configuration

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - 'alerts/*.yml'

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']

  - job_name: 'app'
    static_configs:
      - targets: ['localhost:3000']

Rule of thumb: never scrape more frequently than every 15 seconds. Shorter intervals increase storage cost and network pressure without improving alert accuracy.

---

3. Alert Convergence: Three-Layer Suppression

Layer 1: Dependency Inhibition

inhibit_rules:
  - source_match:
      severity: 'critical'
      alertname: 'InstanceDown'
    target_match:
      severity: 'warning'
    equal: ['instance']

  - source_match:
      alertname: 'PostgresDown'
    target_match:
      alertname: 'APIHighLatency'
    equal: ['cluster']

Layer 2: Group Aggregation

route:
  group_by: ['alertname', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

Layer 3: Silent Windows

time_intervals:
  - name: 'business_hours'
    time_intervals:
      - weekdays: ['monday:friday']
        times:
          - start_time: '09:00'
            end_time: '19:00'

---

4. Grafana Dashboard Design: The 3-Second Rule

An operator should be able to tell whether the system is healthy within 3 seconds of opening the dashboard.

Layout

┌──────────────────────────────────────┐
│ Row 1: SLO Overview (Red/Yellow/Green)│
│  API Avail │ P99 Latency │ Error Rate │
├──────────────────────────────────────┤
│ Row 2: Service Status (per-service)  │
│  auth │ orders │ payment │ notify    │
├──────────────────────────────────────┤
│ Rows 3-4: Detailed Metrics           │
│  CPU/Mem/Disk/Net │ Slow SQL        │
└──────────────────────────────────────┘

Watch only red: if the SLO overview row is all green, the system is healthy. Do not scroll down.

---

5. Recommended Stack by Team Size

Size	Stack	Monthly Ops Cost
Solo / Small	Prometheus + Grafana + Alertmanager	1 × 2C4G server
Medium	+ Loki (logs) + Tempo (traces)	2-3 servers
Large	VictoriaMetrics + Grafana OnCall + PagerDuty	Depends on volume

---

Checklist

Before launch:

• SLOs defined and agreed (≤ 5 metrics)
• Key services expose /metrics endpoints
• Prometheus targets reachable, scrape succeeding
• Alert rules have dependency inhibition configured
• Runbook written — what to do first when an alert fires
• Grafana dashboards pass the 3-second test

Ongoing:

• Review Grafana dashboards weekly (5 min)
• Review alert rules monthly — remove unnecessary ones
• Run a P0 alert drill quarterly

---

Related Reading

• Observability for AI Applications — Monitoring for LLM-based systems
• CI/CD Tooling: GitHub Actions vs GitLab CI — Automated ops via alert triggers
• Ops Automation Patterns — Self-healing systems driven by alerts

Need help building a monitoring stack? Contact us for a free architecture consultation and fixed-price quote.