LFCE 127 🐧 Prometheus — Scrape Configs
The scrape_configs section of prometheus.yml is where Prometheus learns what to monitor. Each entry defines a job: a named group of targets that share the same scrape behavior. A job can list targets statically, discover them from a file, or query a service discovery mechanism. It can override the global scrape interval, change the metrics path, attach labels, and apply relabeling rules before a sample is ever ingested.
This chapter covers the structure of a scrape job, the four target discovery mechanisms that matter in production (static_configs, file_sd_configs, dns_sd_configs, and kubernetes_sd_configs), the per-job overrides that change how a target is scraped, and the relabel_configs rules that filter and transform targets before scraping. By the end, you will be able to write a scrape configuration that discovers targets dynamically, attaches meaningful labels, and avoids scraping things it should not.
Key point: relabel_configs run before the scrape. They can drop a target entirely, rewrite its address, or add labels that appear on every metric from that target. metric_relabel_configs run after the scrape and can only drop or rewrite individual samples.
Why scrape configs have this structure
The target discovery problem. A static list of targets works for a handful of servers. It does not work for a Kubernetes cluster where pods are created and destroyed continuously, or a cloud environment where instances are ephemeral. The scrape config supports multiple discovery mechanisms because no single mechanism fits every environment. static_configs is the simplest, file_sd_configs delegates discovery to an external script, and kubernetes_sd_configs queries the Kubernetes API directly .
The label attachment problem. A metric’s identity is its name plus its labels. The scrape config controls two sets of labels: the labels attached to the target (labels in static_configs, or metadata labels from service discovery), and the labels attached to every sample from that target (the job label, which is the job_name, and any instance label derived from __address__). The relabel_configs section can add, modify, or remove these labels before the scrape .
The filtering problem. Service discovery returns everything it knows about. A Kubernetes discovery role returns every pod, including the ones that do not expose metrics. A file-based discovery might return targets that are temporarily down. The relabel_configs section is where filtering happens: a keep action drops targets that do not match a regex, and a drop action removes targets that do. Without relabeling, Prometheus scrapes every target the discovery mechanism returns, including the ones that should be ignored .
a. The basic job structure
A scrape job is a YAML mapping with job_name as the only required field. Everything else has a default. The job_name becomes the value of the job label on every metric scraped by that job.
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]This is the minimal job. It scrapes http://localhost:9090/metrics every 15 seconds (the global default), and every metric it collects carries job="prometheus" and instance="localhost:9090" .
The static_configs block accepts a list of target groups. Each group has a targets list and an optional labels mapping. The labels in a static config group are attached to every metric from every target in that group.
scrape_configs:
- job_name: "node"
static_configs:
- targets: ["10.0.1.10:9100", "10.0.1.11:9100"]
labels:
env: "production"
role: "web"The two targets share the env and role labels, but each gets its own instance label derived from its address.
b. Per-job overrides
Every field in a scrape job can override the global setting. The most commonly overridden fields are scrape_interval, scrape_timeout, metrics_path, and scheme.
scrape_configs:
- job_name: "slow-service"
scrape_interval: 60s
scrape_timeout: 30s
metrics_path: "/custom/metrics"
scheme: "https"
static_configs:
- targets: ["10.0.2.20:8443"]A job that scrapes a slow endpoint can use a longer interval and timeout without affecting the other jobs. A job that exposes metrics on a non-standard path can set metrics_path. A job that requires HTTPS can set scheme and provide tls_config for certificate validation .
The params field passes query parameters to the scrape request. It is a mapping of parameter names to lists of values.
- job_name: "api-metrics"
params:
module: ["http_2xx"]
static_configs:
- targets: ["10.0.3.30:9115"]
The params field is useful for exporters that require a module parameter, such as the blackbox exporter.
c. File-based service discovery
file_sd_configs reads target lists from JSON or YAML files and re-reads them at a configured interval. It is the bridge between Prometheus and any external system that can write a file: configuration management tools, custom scripts, or a controller that watches a cloud API .
scrape_configs:
- job_name: "dynamic-targets"
file_sd_configs:
- files:
- "/etc/prometheus/targets/*.json"
- "/etc/prometheus/targets/*.yml"
refresh_interval: "5m"The JSON file format is a list of target groups, each with targets and labels:
[
{
"targets": ["10.0.4.40:9100", "10.0.4.41:9100"],
"labels": {"env": "staging", "team": "platform"}
}
]The YAML format is equivalent. Prometheus watches the files and reloads them when they change, without a restart. The refresh_interval controls how often it checks for changes. File-based discovery is the recommended approach for targets managed by Ansible, Puppet, or a custom inventory script, because the script writes the file and Prometheus picks up the change .
d. Relabeling: filtering and transforming targets
relabel_configs is the most powerful and most misunderstood part of the scrape config. It processes every target that a discovery mechanism returns, before the target is scraped. Each rule has a source_labels list, a separator, a regex, and an action. The action determines what happens when the concatenated source labels match the regex.
The most common actions are keep, drop, replace, and labelmap.
scrape_configs:
- job_name: "kubernetes-pods"
kubernetes_sd_configs:
- role: pod
relabel_configs:
# Only scrape pods with prometheus.io/scrape=true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: "true"
# Use the port from the prometheus.io/port annotation
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
target_label: __address__
regex: "(.+):(\\d+)"
replacement: "$1:$2"
# Add the pod name as a label
- source_labels: [__meta_kubernetes_pod_name]
target_label: podThe __meta_* labels are internal labels provided by the discovery mechanism. They are available during relabeling but are removed before the sample is ingested. The __address__ label is the target’s host:port, and rewriting it changes where Prometheus sends the scrape. The __metrics_path__ label can be rewritten to change the path per target .
The keep action drops every target that does not match the regex. The drop action drops every target that does match. These two actions are how a job filters the output of a discovery mechanism that returns more targets than it should scrape.
Complete Example Session
# ============================================
# PART 1: THE MINIMAL JOB
# ============================================
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
# Scrapes http://localhost:9090/metrics every 15s.
# Adds job="prometheus" and instance="localhost:9090".# ============================================
# PART 2: STATIC TARGETS WITH LABELS
# ============================================
scrape_configs:
- job_name: "node"
static_configs:
- targets: ["10.0.1.10:9100", "10.0.1.11:9100"]
labels:
env: "production"
role: "web"
# Both targets carry env="production" and role="web".# ============================================
# PART 3: PER-JOB OVERRIDES
# ============================================
- job_name: "slow-api"
scrape_interval: 60s
scrape_timeout: 30s
metrics_path: "/api/metrics"
scheme: "https"
static_configs:
- targets: ["10.0.2.20:8443"]
# This job scrapes a different path, over HTTPS,
# with a longer interval than the global default.# ============================================
# PART 4: FILE-BASED DISCOVERY
# ============================================
- job_name: "dynamic"
file_sd_configs:
- files:
- "/etc/prometheus/targets/*.json"
refresh_interval: "5m"
# Prometheus re-reads the JSON files every 5 minutes.
# A script can write new targets without a reload.# ============================================
# PART 5: THE JSON TARGET FILE
# ============================================
[
{
"targets": ["10.0.4.40:9100"],
"labels": {"env": "staging"}
}
]
# Placed in /etc/prometheus/targets/staging.json.# ============================================
# PART 6: KUBERNETES POD DISCOVERY
# ============================================
- job_name: "kubernetes-pods"
kubernetes_sd_configs:
- role: pod
# Discovers every pod in the cluster.
# Returns far more targets than should be scraped.# ============================================
# PART 7: FILTER WITH KEEP
# ============================================
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: "true"
# Only pods with the annotation prometheus.io/scrape=true.
# Everything else is dropped before scraping.# ============================================
# PART 8: REWRITE THE ADDRESS
# ============================================
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: "([^:]+)(?::\\d+)?;(\\d+)"
replacement: "$1:$2"
target_label: __address__
# Replaces the pod IP with IP:annotation_port.
# The port comes from the prometheus.io/port annotation.# ============================================
# PART 9: ADD A LABEL
# ============================================
- source_labels: [__meta_kubernetes_pod_name]
target_label: pod
# Every metric from this target gets pod="<name>".# ============================================
# PART 10: VALIDATE AND RELOAD
# ============================================
promtool check config /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus
# promtool catches syntax errors before reload.
# A successful reload applies the new scrape configs.The ten parts covered the minimal job, static labels, per-job overrides, file-based discovery, the JSON target file, Kubernetes discovery, keep filtering, address rewriting, label addition, and validation.
Quick Reference
Job fields
| Field | Default | Purpose |
|---|---|---|
job_name | Required | Job identifier, becomes job label |
scrape_interval | Global | Per-job scrape frequency |
scrape_timeout | Global | Per-job timeout |
metrics_path | /metrics | HTTP path to scrape |
scheme | http | Protocol scheme |
params | — | Query parameters |
static_configs | — | Static target list |
file_sd_configs | — | Targets from files |
kubernetes_sd_configs | — | Targets from Kubernetes API |
relabel_configs | — | Pre-scrape target transformation |
metric_relabel_configs | — | Post-scrape sample transformation |
Discovery mechanisms
| Mechanism | Use case | Dynamic |
|---|---|---|
static_configs | Fixed targets | No |
file_sd_configs | External inventory | Yes (file watch) |
dns_sd_configs | DNS SRV records | Yes |
kubernetes_sd_configs | Kubernetes resources | Yes |
Relabel actions
| Action | Effect |
|---|---|
keep | Keep targets matching regex |
drop | Drop targets matching regex |
replace | Replace target label with regex capture |
labelmap | Map discovered labels to new label names |
labeldrop | Drop matching labels |
labelkeep | Keep matching labels |
Common relabel patterns
| Pattern | Purpose |
|---|---|
keep on __meta_*_annotation_prometheus_io_scrape | Only scrape annotated pods |
replace on __address__ | Change scrape port |
replace on __metrics_path__ | Change metrics path per target |
replace on __meta_*_name → pod/node | Add human-readable label |
Best Practices
✅ Do This:
# Use file_sd_configs for targets managed by external tools
file_sd_configs:
- files: ["/etc/prometheus/targets/*.json"] # ✅# Filter Kubernetes discovery with keep rules
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: "true" # ✅# Override scrape_interval per job when needed
scrape_interval: 60s # ✅# Validate with promtool before reloading
promtool check config /etc/prometheus/prometheus.yml # ✅❌ Don’t Do This:
# Scrape every Kubernetes pod without filtering
kubernetes_sd_configs:
- role: pod
# no relabel_configs # ❌# Use static_configs for a dynamic environment
static_configs:
- targets: ["10.0.1.10:9100"] # pods change constantly # ❌# Rewrite __address__ without preserving the port
target_label: __address__
replacement: "$1" # drops the port # ❌# Set scrape_timeout greater than scrape_interval
scrape_interval: 15s
scrape_timeout: 30s # ❌ (timeout > interval)Common Pitfalls
| Pitfall | Why It Happens | Fix |
|---|---|---|
Target shows up=0 | Wrong metrics_path, scheme, or port | Check the target’s actual endpoint |
| Kubernetes job scrapes too many targets | No keep relabel rule | Add keep on the scrape annotation |
| File discovery not updating | refresh_interval not set or file path wrong | Check the glob and the interval |
relabel_configs has no effect | Source labels use wrong __meta_* name | Verify the discovery mechanism’s label names |
| Scrape timeout errors | scrape_timeout too short for slow endpoint | Increase per-job timeout |
| Port missing after address rewrite | Replacement drops the port | Include $2 in the replacement |
Real-World Examples
1. Minimal Prometheus self-scrape
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]2. Node exporter static targets
- job_name: "node"
static_configs:
- targets: ["10.0.1.10:9100", "10.0.1.11:9100"]3. Per-job interval override
- job_name: "slow"
scrape_interval: 60s
static_configs:
- targets: ["10.0.2.20:9100"]4. File-based discovery
- job_name: "dynamic"
file_sd_configs:
- files: ["/etc/prometheus/targets/*.json"]5. JSON target file
[{"targets": ["10.0.4.40:9100"], "labels": {"env": "prod"}}]6. Kubernetes pod discovery
- job_name: "pods"
kubernetes_sd_configs:
- role: pod7. Keep annotated pods
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: "true"8. Rewrite address with annotation port
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: "([^:]+)(?::\\d+)?;(\\d+)"
replacement: "$1:$2"
target_label: __address__9. Add pod name label
- source_labels: [__meta_kubernetes_pod_name]
target_label: pod10. Validate config
promtool check config /etc/prometheus/prometheus.ymlVisual
The scrape config lifecycle
┌─────────────────────────────────────────────────────────────┐
│ SCRAPE CONFIG LIFECYCLE │
│ │
│ Discovery mechanism │
│ (static, file, k8s, dns) │
│ │ │
│ ▼ │
│ Raw targets with __meta_* labels │
│ │ │
│ ▼ │
│ relabel_configs │
│ ├── keep/drop: filter targets │
│ ├── replace: rewrite __address__, labels │
│ └── labelmap: map discovered labels │
│ │ │
│ ▼ │
│ Final targets │
│ │ │
│ ▼ │
│ HTTP GET /metrics │
│ │ │
│ ▼ │
│ metric_relabel_configs │
│ ├── drop: remove samples │
│ └── replace: rewrite sample labels │
│ │ │
│ ▼ │
│ TSDB ingestion │
└─────────────────────────────────────────────────────────────┘Static vs file discovery
┌─────────────────────────────────────────────────────────────┐
│ STATIC_CONFIGS FILE_SD_CONFIGS │
│ │
│ Targets in prometheus.yml Targets in external files │
│ Reload required Auto-refreshed │
│ Good for fixed hosts Good for dynamic inventory │
│ No external dependency Requires file writer │
└─────────────────────────────────────────────────────────────┘Kubernetes relabel flow
┌─────────────────────────────────────────────────────────────┐
│ KUBERNETES RELABELING │
│ │
│ kubernetes_sd_configs: role: pod │
│ │ │
│ ▼ │
│ Every pod with __meta_kubernetes_pod_* labels │
│ │ │
│ ▼ │
│ keep: __meta_kubernetes_pod_annotation_prometheus_io_scrape │
│ = "true" │
│ │ │
│ ▼ │
│ replace: __address__ = pod_ip:annotation_port │
│ │ │
│ ▼ │
│ replace: pod = __meta_kubernetes_pod_name │
│ │ │
│ ▼ │
│ Scrape the pod │
└─────────────────────────────────────────────────────────────┘Per-job override decision
┌─────────────────────────────────────────────────────────────┐
│ WHEN TO OVERRIDE │
│ │
│ Slow endpoint? │
│ └── scrape_interval: 60s, scrape_timeout: 30s │
│ │
│ Non-standard path? │
│ └── metrics_path: /custom/metrics │
│ │
│ HTTPS with client cert? │
│ └── scheme: https, tls_config: {...} │
│ │
│ Requires query parameter? │
│ └── params: {module: ["http_2xx"]} │
└─────────────────────────────────────────────────────────────┘Summary
| Item | Value |
|---|---|
| Required job field | job_name |
| Default metrics path | /metrics |
| Default scheme | http |
| Static targets | static_configs |
| File-based discovery | file_sd_configs with files glob |
| Kubernetes discovery | kubernetes_sd_configs with role |
| Pre-scrape filtering | relabel_configs |
| Post-scrape filtering | metric_relabel_configs |
| Target address label | __address__ |
| Metrics path label | __metrics_path__ |
Key takeaways:
- Every scrape job needs a
job_name, and that name becomes thejoblabel on every metric it collects. Theinstancelabel is derived from__address__. Together, these two labels identify the source of every sample . static_configsis for fixed targets;file_sd_configsis for everything else. A file-based discovery lets an external script or configuration management tool write the target list, and Prometheus picks up changes without a restart .relabel_configsruns before the scrape. It can drop targets (keep,drop), rewrite the address (__address__), change the metrics path (__metrics_path__), and add labels. The__meta_*labels from service discovery are available only during this phase.- Kubernetes discovery returns every resource for the chosen role. Without a
keeprule that filters on an annotation or label, Prometheus scrapes pods that do not expose metrics, producingup=0noise and wasted scrape cycles . - Per-job overrides isolate slow or special endpoints. A job that scrapes a slow API can use a longer interval and timeout without changing the behavior of the jobs that scrape fast exporters .
- Validate with
promtoolbefore reloading. A YAML syntax error or an invalid relabel rule prevents Prometheus from starting.promtool check configcatches these before the service is affected.
Remember: A scrape config is a contract between Prometheus and a set of targets. The discovery mechanism determines which targets exist. The relabel rules determine which ones are scraped and what labels they carry. The per-job overrides determine how the scrape is performed. Get the discovery right first: static_configs for a handful of fixed hosts, file_sd_configs for anything managed by an external system, kubernetes_sd_configs for pods and services. Then add a keep rule to filter out the targets that should not be scraped. Without it, the job scrapes everything the discovery returns, and the up metric becomes noise instead of signal.
Stop using slow, ad-bloated tool sites! 🤮
🔎 Search “KandZ Tools” on Google to use many professional utilities for free.
KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)
⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free
🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!