LFCE 130 🐧 Alertmanager — Concepts
Alertmanager is the notification router for Prometheus alerts. Prometheus evaluates alerting rules and sends firing alerts to Alertmanager. Alertmanager then decides whether to send a notification, to whom, and how often. It does not evaluate rules, store metrics, or query data. Its entire responsibility is the space between “an alert fired” and “a human received a message” .
This chapter covers the four mechanisms that define Alertmanager’s behavior: grouping, which batches related alerts into a single notification; inhibition, which suppresses alerts when a more important alert is already firing; silences, which mute alerts for a scheduled window; and the routing tree, which determines which receiver gets which alert. It also covers the high-availability model, which is not a load-balanced cluster but a gossip mesh where every instance receives every alert.
Key point: Grouping, inhibition, and silences suppress notifications, not alerts. The alert is still firing, still visible in the Alertmanager UI, and still queryable. Only the notification is withheld.
Why Alertmanager exists
The notification storm problem. A network partition or a host failure does not produce one alert. It produces dozens: the host is down, every service on the host is down, every check that depends on those services is failing, and every dependency of those services is reporting errors. Sending each of those as a separate notification is noise. Alertmanager’s grouping batches alerts that share labels into a single notification, so a hundred instances of the same problem arrive as one message listing the affected instances .
The causal chain problem. When a host is down, the disk-space alert for that host is not useful information. The host is down; the disk is irrelevant. Inhibition expresses this causality: if an alert matching InstanceDown is firing for an instance, suppress the warning alerts that share the same instance label . The operator sees the cause, not the consequences.
The planned maintenance problem. A database upgrade will take the service offline for thirty minutes. The alerts that fire during that window are expected. A silence mutes notifications for alerts matching a set of label matchers for a defined time period. Unlike inhibition, a silence is not driven by another alert; it is an explicit operator action .
The routing problem. A critical database alert should page the on-call engineer. An informational alert about a certificate expiring in thirty days should go to a Slack channel. A warning about a staging environment should go to a different channel than the same warning in production. The routing tree matches alerts on their labels and sends them to the appropriate receiver, with per-route settings for grouping and repeat intervals .
a. Grouping
Grouping categorizes alerts of similar nature into a single notification. The grouping is controlled by the group_by list in the route configuration. Alerts that share the same values for every label in group_by are grouped together.
route:
group_by: ['alertname', 'cluster']
group_wait: 30s
group_interval: 5m
repeat_interval: 4hThe three timing parameters control the notification lifecycle. group_wait is how long Alertmanager waits after the first alert in a group arrives before sending the first notification, to allow more alerts to join the group. group_interval is how long it waits before sending an update for a group that has new alerts. repeat_interval is how long it waits before re-sending a notification for a group that is still firing .
The documentation’s canonical example is a network partition affecting half the service instances. Hundreds of alerts fire, one per instance. With grouping by cluster and alertname, Alertmanager sends a single notification that lists every affected instance, instead of hundreds of separate messages .
b. Inhibition
Inhibition suppresses notifications for target alerts when a source alert is already firing. The rule has three parts: source_matchers selects the alert that causes the suppression, target_matchers selects the alerts that are suppressed, and equal lists the labels that must have identical values on both sides for the rule to apply .
inhibit_rules:
- source_matchers: [alertname = "InstanceDown"]
target_matchers: [severity =~ "warning|critical"]
equal: [instance]This rule reads as: while an InstanceDown alert is firing, mute any warning or critical alert that has the same instance value. The equal clause is what scopes the suppression to the dead host. A disk alert on a different, healthy host has a different instance value and is not suppressed .
Inhibition is live state, not history. The moment the source alert resolves, the target alerts start notifying again on their own. The suppression is not a permanent mute; it is a conditional one that lasts exactly as long as the cause is present.
c. Silences
A silence mutes notifications for alerts that match its label matchers, for a defined start and end time. Unlike inhibition, a silence is not triggered by another alert. It is created by an operator, either through the Alertmanager web interface or through the API .
A silence is defined by a set of matchers, using the same operators as routing: =, !=, =~, and !~. An incoming alert is checked against every active silence. If it matches all the matchers of a silence, no notification is sent for that alert .
Silences are the correct tool for planned maintenance. A database upgrade window, a network change, or a certificate rotation can be scheduled, and a silence covering the affected labels prevents the expected alerts from paging anyone. The silence expires automatically at the configured end time, which prevents the common failure mode of a temporary mute that is never removed.
d. The routing tree and receivers
The routing tree is a hierarchical structure that matches alerts on their labels and sends them to receivers. The root route is the default; child routes are evaluated in order, and the first match wins unless continue: true is set .
route:
receiver: 'default-slack'
group_by: ['alertname']
routes:
- matchers: [severity = "critical"]
receiver: 'pagerduty-critical'
repeat_interval: 1h
- matchers: [team =~ "database|dba"]
receiver: 'db-team-slack'A receiver is a named notification integration: email, Slack, PagerDuty, OpsGenie, a generic webhook, or any other supported channel. The receiver configuration contains the connection details and the notification template. The routing tree decides which receiver is invoked for a given alert .
The routing tree is first-match, top-down. Specific routes must come before general ones. An alert that matches no child route falls through to the root route’s receiver. A route that points to a nonexistent receiver name is a configuration error that amtool check-config catches before the service fails at 3 a.m. .
Complete Example Session
# ============================================
# PART 1: THE MINIMAL CONFIGURATION
# ============================================
route:
receiver: 'default'
receivers:
- name: 'default'
# notification integration details here# ============================================
# PART 2: GROUPING
# ============================================
route:
receiver: 'default'
group_by: ['alertname', 'cluster']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h# ============================================
# PART 3: ROUTING BY SEVERITY
# ============================================
route:
receiver: 'default-slack'
routes:
- matchers: [severity = "critical"]
receiver: 'pagerduty'
- matchers: [severity = "warning"]
receiver: 'slack-warnings'# ============================================
# PART 4: INHIBITION RULE
# ============================================
inhibit_rules:
- source_matchers: [alertname = "InstanceDown"]
target_matchers: [severity =~ "warning|critical"]
equal: [instance]# ============================================
# PART 5: INHIBITION FOR CLUSTER OUTAGE
# ============================================
inhibit_rules:
- source_matchers: [alertname = "ClusterUnreachable"]
target_matchers: [alertname =~ "InstanceDown|ServiceDown"]
equal: [cluster]# ============================================
# PART 6: SILENCE (CREATED VIA UI OR API)
# ============================================
# Matchers: alertname="DiskSpaceWarning", instance="db01"
# Start: 2026-11-05T22:00:00Z
# End: 2026-11-05T23:00:00Z
# Comment: "Planned disk expansion"# ============================================
# PART 7: RECEIVER DEFINITION
# ============================================
receivers:
- name: 'pagerduty'
pagerduty_configs:
- routing_key: '<key>'
send_resolved: true# ============================================
# PART 8: VALIDATION
# ============================================
# amtool check-config /etc/alertmanager/alertmanager.yml# ============================================
# PART 9: HIGH AVAILABILITY FLAGS
# ============================================
# alertmanager \
# --cluster.listen-address=0.0.0.0:9094 \
# --cluster.peer=am-1:9094 \
# --cluster.peer=am-2:9094# ============================================
# PART 10: THE COMPLETE CONFIGURATION
# ============================================
route:
receiver: 'default-slack'
group_by: ['alertname', 'instance']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- matchers: [severity = "critical"]
receiver: 'pagerduty'
repeat_interval: 1h
inhibit_rules:
- source_matchers: [alertname = "InstanceDown"]
target_matchers: [severity =~ "warning|critical"]
equal: [instance]
receivers:
- name: 'default-slack'
slack_configs: [...]
- name: 'pagerduty'
pagerduty_configs: [...]The ten parts covered the minimal configuration, grouping, routing, inhibition, cluster-level inhibition, silences, receivers, validation, high availability, and the complete configuration.
Quick Reference
The four mechanisms
| Mechanism | Trigger | Scope | Configured in |
|---|---|---|---|
| Grouping | Automatic | Batches alerts sharing labels | route.group_by |
| Inhibition | Source alert firing | Suppresses target alerts matching equal | inhibit_rules |
| Silences | Operator action | Mutes matching alerts until end time | Web UI or API |
| Routing | Every alert | Sends to receiver | route.routes |
Grouping parameters
| Parameter | Default | Purpose |
|---|---|---|
group_by | ['alertname'] | Labels that define a group |
group_wait | 30s | Wait before first notification |
group_interval | 5m | Wait before updates |
repeat_interval | 4h | Re-notify interval |
Inhibition fields
| Field | Purpose |
|---|---|
source_matchers | The alert that causes suppression |
target_matchers | The alerts that are suppressed |
equal | Labels that must match on both sides |
High availability
| Flag | Purpose |
|---|---|
--cluster.listen-address | Bind address for gossip |
--cluster.peer | Peer addresses (repeatable) |
--cluster.advertise-address | Address advertised to peers |
Best Practices
✅ Do This:
# Group by alertname and cluster to batch storms
group_by: ['alertname', 'cluster'] # ✅# Use inhibition to suppress consequences of a known cause
source_matchers: [alertname = "InstanceDown"]
equal: [instance] # ✅# Validate configuration before restarting
amtool check-config /etc/alertmanager/alertmanager.yml # ✅# Point Prometheus to all Alertmanager instances, not a load balancer
alertmanagers:
- static_configs:
- targets: ['am-1:9093', 'am-2:9093'] # ✅❌ Don’t Do This:
# Route to a receiver that does not exist
routes:
- receiver: 'typo-receiver' # not defined # ❌# Use inhibition without the equal clause
inhibit_rules:
- source_matchers: [alertname = "InstanceDown"]
target_matchers: [severity = "warning"]
# no equal → suppresses every warning everywhere # ❌# Load balance Prometheus traffic across Alertmanagers
# Alertmanager expects every alert on every instance # ❌# Create a silence without an end time
# A silence that never expires is a forgotten mute # ❌Common Pitfalls
| Pitfall | Why It Happens | Fix |
|---|---|---|
| Alerts not delivered | Receiver name typo or missing route | amtool check-config; verify receiver names |
| Notification storm | group_by too granular | Group by fewer labels |
| Inhibition suppresses too much | equal clause missing or too broad | Add specific labels to equal |
| Silence does not mute | Matchers do not match the alert labels | Verify the alert’s labels and the silence matchers |
| Duplicate notifications in HA | Prometheus load-balanced across Alertmanagers | Point Prometheus to all instances |
amtool command not found | Package not installed | Install amtool or use the full path |
Real-World Examples
1. Basic routing
route:
receiver: 'slack'
routes:
- matchers: [severity = "critical"]
receiver: 'pagerduty'2. Grouping with timers
route:
group_by: ['alertname', 'instance']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h3. Host-down inhibition
inhibit_rules:
- source_matchers: [alertname = "InstanceDown"]
target_matchers: [severity =~ "warning|critical"]
equal: [instance]4. Cluster-down inhibition
inhibit_rules:
- source_matchers: [alertname = "ClusterUnreachable"]
target_matchers: [alertname =~ "InstanceDown|ServiceDown"]
equal: [cluster]5. Silence via API
curl -X POST http://alertmanager:9093/api/v2/silences -d '{
"matchers": [{"name": "alertname", "value": "DiskSpaceWarning", "isRegex": false}],
"startsAt": "2026-11-05T22:00:00Z",
"endsAt": "2026-11-05T23:00:00Z",
"createdBy": "admin",
"comment": "Planned disk expansion"
}'6. Receiver definition
receivers:
- name: 'pagerduty'
pagerduty_configs:
- routing_key: '<key>'7. Validate config
amtool check-config /etc/alertmanager/alertmanager.yml8. Test routing
amtool config routes test --config.file=alertmanager.yml severity=critical9. HA cluster
alertmanager --cluster.listen-address=0.0.0.0:9094 --cluster.peer=am-2:909410. Query active alerts
amtool alert queryVisual
The Alertmanager pipeline
┌─────────────────────────────────────────────────────────────┐
│ ALERTMANAGER PIPELINE │
│ │
│ Prometheus │
│ │ │
│ ▼ │
│ Alertmanager │
│ │ │
│ ├── Deduplication │
│ │ │
│ ├── Grouping (batch related alerts) │
│ │ │
│ ├── Inhibition (suppress consequences) │
│ │ │
│ ├── Silences (mute for a window) │
│ │ │
│ ▼ │
│ Routing tree (match labels → receiver) │
│ │ │
│ ▼ │
│ Receiver (email, Slack, PagerDuty, webhook) │
└─────────────────────────────────────────────────────────────┘Grouping timers
┌─────────────────────────────────────────────────────────────┐
│ GROUPING TIMERS │
│ │
│ Alert 1 arrives │
│ │ │
│ ▼ │
│ group_wait (30s) ──► more alerts join the group │
│ │ │
│ ▼ │
│ First notification sent (group of N alerts) │
│ │ │
│ ▼ │
│ New alert joins group │
│ │ │
│ ▼ │
│ group_interval (5m) ──► update notification sent │
│ │ │
│ ▼ │
│ Group still firing │
│ │ │
│ ▼ │
│ repeat_interval (4h) ──► reminder notification │
└─────────────────────────────────────────────────────────────┘Inhibition scoping
┌─────────────────────────────────────────────────────────────┐
│ INHIBITION: equal DEFINES THE BLAST RADIUS │
│ │
│ source: InstanceDown{instance="web01"} │
│ target: DiskSpaceWarning{instance="web01"} │
│ equal: [instance] │
│ │
│ Result: DiskSpaceWarning on web01 is suppressed. │
│ │
│ DiskSpaceWarning on web02 (different instance) is NOT │
│ suppressed. The equal clause scopes the rule to web01. │
│ │
│ Without equal: every warning is suppressed, everywhere. │
└─────────────────────────────────────────────────────────────┘HA gossip mesh
┌─────────────────────────────────────────────────────────────┐
│ HIGH AVAILABILITY │
│ │
│ Prometheus sends every alert to every Alertmanager. │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ AM-1 │◄──►│ AM-2 │◄──►│ AM-3 │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ ▲ ▲ ▲ │
│ └───────────────┴───────────────┘ │
│ gossip │
│ │
│ Each instance deduplicates against the shared log. │
│ During a partition: duplicate notifications (fail open). │
│ After heal: logs merge, deduplication resumes. │
└─────────────────────────────────────────────────────────────┘Summary
| Item | Value |
|---|---|
| Purpose | Route, group, deduplicate, inhibit, silence alerts |
| Input | Alerts from Prometheus |
| Output | Notifications to receivers |
| Grouping | Batches alerts sharing group_by labels |
| Inhibition | Suppresses target alerts when source fires |
| Silences | Operator-defined mute for a time window |
| Routing tree | Matches labels to receivers |
| HA model | Gossip mesh, every alert to every instance |
| Default port | 9093 |
| Validation tool | amtool check-config |
Key takeaways:
- Alertmanager does not evaluate rules. Prometheus evaluates the rules and sends firing alerts. Alertmanager decides who hears about them and how often. The separation means the alerting logic is in Prometheus and the notification logic is in Alertmanager .
- Grouping is the primary defense against notification storms. A hundred alerts from a hundred instances of the same problem become one notification listing the affected instances. The
group_bylabels define what “the same problem” means . - Inhibition encodes causality. When a host is down, its disk alert is noise. The
equalclause scopes the suppression to the affected host; without it, the rule suppresses every alert of the target type everywhere . - Silences are for planned maintenance. They mute matching alerts for a defined window and expire automatically. They are not a substitute for fixing a broken alert rule; they are a temporary mute for an expected condition .
- The routing tree is first-match, top-down. Specific routes must come before general ones. A route that points to a nonexistent receiver is a configuration error that
amtool check-configcatches before the service fails . - High availability is a gossip mesh, not a load-balanced cluster. Prometheus must send every alert to every Alertmanager instance. Load balancing between Prometheus and Alertmanager breaks the deduplication model .
Remember: Alertmanager is the filter between the alert and the human. Grouping reduces volume. Inhibition removes consequences of a known cause. Silences mute expected conditions. Routing sends the right alert to the right place. Each mechanism addresses a different failure mode of a naive notification system. The configuration file is where all four are defined, and amtool check-config is what prevents a typo from becoming a silent outage. Get grouping and inhibition right first: they are the two mechanisms that turn an alert storm into a signal.
Stop using slow, ad-bloated tool sites! 🤮
🔎 Search “KandZ Tools” on Google to use many professional utilities for free.
KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)
⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free
🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!