LFCE 131 🐧 Alertmanager — Rules and Receivers
Prometheus alerting is a two-part system. The first part is the alerting rule, which lives in Prometheus and defines the condition that should trigger an alert. The second part is the receiver, which lives in Alertmanager and defines where the notification goes. The rule and the receiver are connected by labels: the rule attaches labels like severity: critical or team: database, and Alertmanager’s routing tree matches those labels to the appropriate receiver.
This chapter covers both parts. It explains the alerting rule file structure, the for clause that debounces transient spikes, and the labels and annotations that carry routing metadata and human-readable context. It also covers the receiver configuration for the three most common channels—Slack, email, and PagerDuty—and the amtool commands that verify a routing tree before an alert ever fires.
Key point: A rule without a matching receiver is an alert that fires into the void. The rule’s labels are the contract; the receiver’s configuration is the fulfillment. Test the routing tree with amtool config routes test before trusting it.
Why rules and receivers are separate
The evaluation problem. Prometheus evaluates rules against the time series database. It has the metrics, the PromQL engine, and the evaluation loop. Alertmanager does not. It receives alerts that are already firing, and its job is to decide who should be notified and how often. Separating the two means Prometheus can focus on evaluation and Alertmanager can focus on notification without duplicating either concern .
The routing problem. A single Prometheus instance can send alerts to multiple Alertmanager instances, and a single Alertmanager instance can receive alerts from multiple Prometheus servers. The routing tree in Alertmanager matches on the labels that Prometheus attached. The rule defines the labels; the routing tree decides what those labels mean in terms of notification channels .
The debounce problem. A rule without a for clause fires on the first evaluation where the condition is true. A transient spike—a brief CPU surge, a momentary error rate increase—produces a notification that requires no action. The for clause tells Prometheus to wait until the condition has held for a defined duration before moving the alert from pending to firing. The pending state is the debounce window .
a. The alerting rule file
An alerting rule lives in a YAML file referenced by the rule_files section of prometheus.yml. The file contains one or more groups, and each group contains one or more rules.
# /etc/prometheus/rules/platform.yml
groups:
- name: platform-alerts
rules:
- alert: HighErrorRate
expr: |
rate(http_requests_total{code=~"5.."}[5m]) > 1
for: 5m
labels:
severity: critical
team: platform
annotations:
summary: "High 5xx rate on {{ $labels.instance }}"
description: "More than 1 5xx error per second over the last 5m."
runbook: "https://runbook.example.com/high-5xx-rate"The alert field is the alert name. It appears in Alertmanager and in the notification. The expr field is a PromQL expression that returns a vector when the condition is true. The for field is the debounce duration. The labels field adds metadata that the routing tree matches on. The annotations field carries human-readable content: a summary, a description, and a runbook link .
The expr must return a vector, not a scalar. rate(http_requests_total[5m]) > 1 returns one or more time series, each representing an instance or label combination that exceeds the threshold. The alert fires for each element in the vector, with the labels of that element attached .
b. The for clause and alert states
The for clause creates the pending state. When the expression first returns a vector element, the alert enters pending. If the condition remains true for the full for duration, the alert transitions to firing. If the condition becomes false before the duration elapses, the alert returns to inactive without ever firing .
# Fires after 10 minutes of sustained latency
- alert: APIHighRequestLatency
expr: api_http_request_latencies_second{quantile="0.5"} > 1
for: 10m
labels:
severity: warning
annotations:
summary: "High request latency on {{ $labels.instance }}"
description: "{{ $labels.instance }} has a median request latency above 1s (current value: {{ $value }}s)"The templating variables are available in labels and annotations. $labels holds the label key-value pairs of the firing element. $value holds the evaluated numeric value. $externalLabels holds the external labels configured in prometheus.yml .
The duration choice is a trade-off. Too short, and the alert fires on transient noise. Too long, and the notification arrives after the incident has already caused damage. The KodeKloud documentation suggests durations between a couple of minutes and ten minutes, depending on the service’s tolerance for brief errors .
c. Receiver configuration
A receiver is a named notification integration. The receivers section of alertmanager.yml defines each receiver with the connection details for its channel.
# /etc/alertmanager/alertmanager.yml
receivers:
- name: 'slack-critical'
slack_configs:
- api_url: 'https://hooks.slack.com/services/...'
channel: '#alerts-critical'
send_resolved: true
- name: 'email-ops'
email_configs:
- to: 'ops@example.com'
from: 'alerts@example.com'
smarthost: 'smtp.example.com:587'
auth_username: 'alerts@example.com'
auth_password: '<password>'
- name: 'pagerduty-critical'
pagerduty_configs:
- routing_key: '<integration-key>'
severity: 'critical'
send_resolved: trueThe slack_configs block requires a webhook URL and a channel. The email_configs block requires an SMTP server, a sender address, and a recipient. The pagerduty_configs block requires an integration key. Each block accepts send_resolved: true to send a follow-up notification when the alert resolves .
The routing tree connects the rule’s labels to the receiver. A route with matchers: [severity = "critical"] sends every alert with severity: critical to the receiver specified in that route.
route:
receiver: 'slack-default'
group_by: ['alertname', 'instance']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- matchers: [severity = "critical"]
receiver: 'pagerduty-critical'
repeat_interval: 1h
- matchers: [team = "database"]
receiver: 'email-dba'The routing tree is first-match, top-down. The most specific routes must come first. A route without a continue: true stops evaluation, so any routes below it are unreachable for matching alerts .
Complete Example Session
# ============================================
# PART 1: THE RULE FILE
# ============================================
# /etc/prometheus/rules/platform.yml
groups:
- name: platform-alerts
rules:
- alert: InstanceDown
expr: up == 0
for: 5m
labels:
severity: critical
team: platform
annotations:
summary: "Instance {{ $labels.instance }} down"
description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes."# ============================================
# PART 2: A SECOND RULE WITH WARNING SEVERITY
# ============================================
- alert: HighMemoryUsage
expr: |
(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes)
/ node_memory_MemTotal_bytes > 0.85
for: 10m
labels:
severity: warning
team: platform
annotations:
summary: "Memory usage above 85% on {{ $labels.instance }}"# ============================================
# PART 3: REFERENCE THE RULES IN PROMETHEUS
# ============================================
# /etc/prometheus/prometheus.yml
rule_files:
- "rules/*.yml"
alerting:
alertmanagers:
- static_configs:
- targets: ['localhost:9093']# ============================================
# PART 4: THE SLACK RECEIVER
# ============================================
# /etc/alertmanager/alertmanager.yml
receivers:
- name: 'slack-platform'
slack_configs:
- api_url: 'https://hooks.slack.com/services/...'
channel: '#platform-alerts'
send_resolved: true# ============================================
# PART 5: THE PAGERDUTY RECEIVER
# ============================================
- name: 'pagerduty-critical'
pagerduty_configs:
- routing_key: '<integration-key>'
severity: 'critical'
send_resolved: true# ============================================
# PART 6: THE EMAIL RECEIVER
# ============================================
- name: 'email-ops'
email_configs:
- to: 'ops@example.com'
from: 'alerts@example.com'
smarthost: 'smtp.example.com:587'
auth_username: 'alerts@example.com'
auth_password: '<password>'# ============================================
# PART 7: THE ROUTING TREE
# ============================================
route:
receiver: 'slack-platform'
group_by: ['alertname', 'instance']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- matchers: [severity = "critical"]
receiver: 'pagerduty-critical'
repeat_interval: 1h
- matchers: [severity = "warning"]
receiver: 'slack-platform'
repeat_interval: 4h# ============================================
# PART 8: VALIDATE THE RULE FILE
# ============================================
# promtool check rules /etc/prometheus/rules/platform.yml
# Checking /etc/prometheus/rules/platform.yml
# SUCCESS: 2 rules found# ============================================
# PART 9: VALIDATE THE ROUTING TREE
# ============================================
# amtool check-config /etc/alertmanager/alertmanager.yml
# amtool config routes test --config.file=/etc/alertmanager/alertmanager.yml severity=critical team=platform
# pagerduty-critical# ============================================
# PART 10: RELOAD BOTH SERVICES
# ============================================
# systemctl reload prometheus
# systemctl reload alertmanagerThe ten parts covered the rule file, a warning-severity rule, the Prometheus reference, three receiver types, the routing tree, rule validation, routing validation, and service reload.
Quick Reference
Rule fields
| Field | Required | Purpose |
|---|---|---|
alert | Yes | Alert name |
expr | Yes | PromQL condition |
for | No | Debounce duration |
labels | No | Routing metadata |
annotations | No | Human-readable context |
keep_firing_for | No | Keep firing after condition clears |
Alert states
| State | Meaning |
|---|---|
inactive | Condition is false |
pending | Condition is true, for duration not elapsed |
firing | Condition has been true for for duration |
Receiver types
| Type | Required fields |
|---|---|
| Slack | api_url, channel |
to, from, smarthost, auth | |
| PagerDuty | routing_key |
| Webhook | url |
Validation commands
| Command | Purpose |
|---|---|
promtool check rules <file> | Validate rule syntax |
amtool check-config <file> | Validate Alertmanager config |
amtool config routes test <labels> | Test routing for a label set |
amtool config routes show | Print the routing tree |
Best Practices
✅ Do This:
# Use a for clause to debounce transient spikes
for: 5m # ✅# Attach routing labels to the rule
labels:
severity: critical
team: platform # ✅# Validate rules before reloading
promtool check rules /etc/prometheus/rules/*.yml # ✅# Test the routing tree with amtool
amtool config routes test --config.file=... severity=critical # ✅❌ Don’t Do This:
# Fire on the first evaluation without debounce
# No for clause # ❌# Route without a matching receiver
# receiver: 'typo-receiver' # ❌# Put a broad route before a specific one
# The specific route becomes unreachable # ❌# Reload without validating
systemctl reload alertmanager # ❌Common Pitfalls
| Pitfall | Why It Happens | Fix |
|---|---|---|
| Alert fires but no notification | No route matches the rule’s labels | Add a route with the matching matcher |
| Alert never fires | for duration too long or expr returns no vector | Check the Prometheus Alerts tab for pending state |
| Notification storm | group_by too granular or no for | Group by fewer labels; add for |
| Routing tree unreachable route | Broad route before specific route without continue | Reorder routes; add continue: true |
| Validation fails | YAML syntax error or invalid PromQL | Use promtool and amtool to identify the line |
Real-World Examples
1. Instance down alert
- alert: InstanceDown
expr: up == 0
for: 5m
labels:
severity: critical2. High error rate alert
- alert: HighErrorRate
expr: rate(http_requests_total{code=~"5.."}[5m]) > 1
for: 5m3. Slack receiver
slack_configs:
- api_url: 'https://hooks.slack.com/...'
channel: '#alerts'4. PagerDuty receiver
pagerduty_configs:
- routing_key: '<key>'5. Email receiver
email_configs:
- to: 'ops@example.com'
smarthost: 'smtp:587'6. Critical route
routes:
- matchers: [severity = "critical"]
receiver: 'pagerduty'7. Warning route
- matchers: [severity = "warning"]
receiver: 'slack'
8. Validate rules
promtool check rules rules.yml9. Validate routing
amtool config routes test severity=critical10. Reload services
systemctl reload prometheus alertmanagerVisual
The rule-to-receiver flow
┌─────────────────────────────────────────────────────────────┐
│ RULE → RECEIVER │
│ │
│ Prometheus rule │
│ expr: rate(errors[5m]) > 1 │
│ for: 5m │
│ labels: {severity: critical, team: platform} │
│ │ │
│ ▼ │
│ Alertmanager receives alert with labels │
│ │ │
│ ▼ │
│ Routing tree matches: severity = critical │
│ │ │
│ ▼ │
│ Receiver: pagerduty-critical │
└─────────────────────────────────────────────────────────────┘The pending state
┌─────────────────────────────────────────────────────────────┐
│ ALERT STATES │
│ │
│ expr returns true │
│ │ │
│ ▼ │
│ pending (for duration starts) │
│ │ │
│ ├── condition false ──► inactive (no notification) │
│ │ │
│ └── for elapsed ──► firing (notification sent) │
└─────────────────────────────────────────────────────────────┘Routing tree order
┌─────────────────────────────────────────────────────────────┐
│ ROUTING TREE (first match wins) │
│ │
│ route: │
│ receiver: 'default' │
│ routes: │
│ - matchers: [severity = "critical"] │
│ receiver: 'pagerduty' ← specific, first │
│ - matchers: [team = "db"] │
│ receiver: 'email-dba' │
│ - matchers: [severity = "warning"] │
│ receiver: 'slack' ← broad, last │
│ │
│ A critical alert matches the first route and stops. │
└─────────────────────────────────────────────────────────────┘Validation pipeline
┌─────────────────────────────────────────────────────────────┐
│ VALIDATION │
│ │
│ Rule file ──► promtool check rules ──► syntax ok │
│ │ │
│ ▼ │
│ Prometheus reload │
│ │ │
│ ▼ │
│ Alertmanager config ──► amtool check-config ──► ok │
│ │ │
│ ▼ │
│ amtool config routes test ──► receiver name │
│ │ │
│ ▼ │
│ Alertmanager reload │
└─────────────────────────────────────────────────────────────┘Summary
| Item | Value |
|---|---|
| Rule location | rule_files in prometheus.yml |
| Rule fields | alert, expr, for, labels, annotations |
| Debounce | for clause |
| Alert states | inactive, pending, firing |
| Receiver location | receivers in alertmanager.yml |
| Routing | route.routes with matchers |
| Slack receiver | slack_configs with api_url, channel |
| Email receiver | email_configs with to, smarthost |
| PagerDuty receiver | pagerduty_configs with routing_key |
| Rule validation | promtool check rules |
| Routing validation | amtool config routes test |
Key takeaways:
- The rule defines the condition and the labels; the receiver defines the destination. The rule’s
labelsfield is the contract that connects the two. A rule without a matching route in Alertmanager fires into the void . - The
forclause is the debounce. Without it, the alert fires on the first evaluation where the condition is true, including transient spikes. With it, the alert stayspendinguntil the condition has held for the configured duration . - The routing tree is first-match, top-down. Specific routes must come before broad routes. A route without
continue: truestops evaluation, so any routes below it are unreachable for matching alerts . - Each receiver type has its own configuration block. Slack needs
api_urlandchannel. Email needsto,smarthost, and authentication. PagerDuty needs arouting_key. Thesend_resolved: trueoption sends a follow-up when the alert resolves . - Validate before reloading.
promtool check rulescatches rule syntax errors.amtool check-configcatches Alertmanager configuration errors.amtool config routes testverifies that a specific label set reaches the intended receiver without sending any notification . - A rule without a
forclause is a rule that fires on noise. The pending state exists to separate a sustained condition from a momentary blip. Use it.
Remember: An alerting rule is a PromQL expression plus a duration plus a set of labels. The expression determines when the condition is true. The duration determines how long it must stay true. The labels determine where the notification goes. The receiver is the destination: a Slack channel, an email address, a PagerDuty integration. The routing tree connects the two. Validate the rule with promtool, validate the routing with amtool, and reload both services only after the validation passes. A rule that fires without a receiver is not an alert; it is a log entry that nobody reads.
Stop using slow, ad-bloated tool sites! 🤮
🔎 Search “KandZ Tools” on Google to use many professional utilities for free.
KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)
⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free
🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!