LFCE 125 🐧 Prometheus — Concepts
Prometheus is a monitoring platform that collects metrics from targets by scraping HTTP endpoints, stores them as time series in a local database, and provides a query language to analyze them. It was originally built at SoundCloud in 2012, joined the Cloud Native Computing Foundation in 2016, and has become the de facto standard for cloud-native monitoring . The design is deliberately different from traditional monitoring systems: Prometheus pulls metrics rather than receiving them, stores data locally rather than in a distributed cluster, and identifies every time series by a metric name plus a set of key-value labels rather than by a hierarchical path.
This chapter covers the conceptual foundation of Prometheus: the pull-based collection model and its trade-offs, the multi-dimensional data model that distinguishes it from earlier monitoring systems, the four metric types that client libraries expose, the component architecture that separates scraping, storage, querying, and alerting, and the PromQL query language that operates on the label dimensions. By the end, you will understand why Prometheus is described as a time series database with a query language rather than a general-purpose monitoring solution, and what it is not designed to do.
Key point: Prometheus stores numeric time series, not logs or events. Every sample is a timestamped value identified by a metric name and a set of labels. The label schema determines what you can query, and high-cardinality labels are the primary cause of memory and storage growth.
Why Prometheus exists
The push-based monitoring problem. Traditional monitoring systems like Nagios and Zabbix use a push model: agents or scripts send status updates to a central server, or the server runs check scripts against targets. This works for simple up/down checks but scales poorly for dynamic environments where targets appear and disappear, and it makes it difficult to tell whether a target is down or the monitoring system itself is broken. Prometheus inverts the model: the server pulls metrics from targets at a configured interval, which means a failed scrape is an immediate signal that the target is unreachable .
The hierarchical metric problem. Earlier systems like Graphite organize metrics as hierarchical paths (servers.web01.cpu.user). This is readable but inflexible: you cannot ask “what is the average CPU across all web servers in us-east-1” without knowing the path structure in advance. Prometheus’s data model uses labels instead of hierarchy. A CPU metric has a name (node_cpu_seconds_total) and labels like instance, job, mode, and cpu. Any label can be used to filter or aggregate, which makes ad-hoc queries possible without pre-planning the metric namespace .
The distributed storage problem. Prometheus stores its data in a local time series database on the server’s own disk. There is no dependency on network storage, a separate database cluster, or a distributed consensus layer. The documentation is explicit about the reason: “Prometheus is designed for reliability, to be the system you go to during an outage to allow you to quickly diagnose problems. Each Prometheus server is standalone, not depending on network storage or other remote services” . The trade-off is that a single Prometheus instance is a single point of failure, and long-term storage requires external systems via remote_write.
a. The pull-based collection model
Prometheus collects metrics by sending HTTP GET requests to a /metrics endpoint on each target. The target responds with a plain-text or protobuf payload containing the current values of all its metrics. The scrape interval is configured globally and can be overridden per job .
# prometheus.yml
global:
scrape_interval: 15s
scrape_configs:
- job_name: prometheus
static_configs:
- targets: ['localhost:9090']The pull model has several practical consequences. First, target health is visible directly: if a scrape fails, the up metric for that target is set to 0. Second, the Prometheus server controls the scrape rate, which prevents a target from overwhelming the monitoring system with a burst of pushed data. Third, the target list is centralized: the Prometheus configuration or a service discovery mechanism determines what is monitored, rather than each target deciding whether to report .
The exception to the pull model is short-lived jobs that cannot be scraped reliably. For these, Prometheus provides the Pushgateway: the job pushes its metrics to the gateway, and Prometheus scrapes the gateway . The Pushgateway is not a general-purpose push endpoint; it is a workaround for batch jobs that exist for less than one scrape interval.
b. The data model and metric types
Every time series in Prometheus is identified by a metric name and a set of labels. The notation is metric_name{label="value", ...}. A metric with the name node_cpu_seconds_total and labels instance="web01", mode="user", and cpu="0" is one time series. Changing any label value creates a new time series .
node_cpu_seconds_total{instance="web01", job="node", mode="user", cpu="0"} 12345.67Prometheus client libraries expose four metric types, each suited to a different kind of measurement :
Counter is a monotonically increasing value that resets to zero on restart. It is used for counts: requests served, errors occurred, bytes sent. The value only goes up, which makes rate() queries meaningful.
Gauge is a value that can go up or down arbitrarily. It is used for current states: memory usage, temperature, queue length, number of active connections.
Histogram samples observations into configurable buckets and also provides a sum and count of all observations. It is used for request durations and response sizes, where you want to know the distribution, not just the average.
Summary is similar to a histogram but calculates configurable quantiles over a sliding time window. It is used when you need precise quantiles without knowing the bucket boundaries in advance, at the cost of higher client-side computation and the inability to aggregate quantiles across instances.
The choice of metric type is not cosmetic. A counter queried with rate() produces a per-second rate. A gauge queried with rate() produces meaningless noise. A histogram produces bucket time series that can be aggregated with histogram_quantile(). A summary’s quantiles cannot be aggregated across instances.
c. The component architecture
A Prometheus deployment consists of several components that can run as a single binary or as separate services. The core server has three responsibilities: the retrieval worker that scrapes targets, the time series database that stores samples locally, and the HTTP server that provides the query interface .
The retrieval worker reads the scrape configuration, sends HTTP requests to each target’s /metrics endpoint at the configured interval, parses the response, and writes the samples to the TSDB. If a scrape fails, it records the failure as the up metric.
The TSDB stores samples in time-bounded blocks, typically two hours each, on the local filesystem. The blocks are compressed and indexed by metric name and labels. The default retention is 15 days, configurable with --storage.tsdb.retention.time . The TSDB is not designed for network-attached storage; it expects fast, local POSIX filesystem access.
The HTTP server exposes the PromQL expression browser, the /metrics endpoint for Prometheus’s own metrics, and the HTTP API that Grafana and other tools use for querying.
External to the server, exporters expose metrics for systems that do not natively support Prometheus. The Node Exporter exposes host-level metrics (CPU, memory, disk, network) on Linux. The JMX Exporter exposes JVM metrics. Database, web server, and load balancer exporters exist for most common services .
The Alertmanager is a separate service that receives alerts from Prometheus, deduplicates them, groups them, and routes them to notification channels like email, Slack, or PagerDuty. Prometheus evaluates alerting rules and sends firing alerts to Alertmanager; it does not send notifications itself .
Service discovery replaces static target lists in dynamic environments. Prometheus supports Kubernetes, EC2, GCE, Consul, DNS, and file-based discovery. Each discovery mechanism populates the target list, and relabeling rules can filter and transform targets before scraping .
d. PromQL and the query model
PromQL is a functional expression language for selecting and aggregating time series. It operates on the label dimensions: you can filter by label, aggregate by label, and compute rates and derivatives over time windows .
# Select all time series for a metric
http_requests_total
# Filter by labels
http_requests_total{job="apiserver", handler="/api/comments"}
# Per-second rate over 5 minutes
rate(http_requests_total[5m])
# Sum the rate by job, preserving the job label
sum by (job) (rate(http_requests_total[5m]))
# Top 3 CPU users by app and process
topk(3, sum by (app, proc) (rate(instance_cpu_time_ns[5m])))The query model distinguishes between instant vectors (one sample per time series at a single timestamp) and range vectors (a range of samples over a time window). Functions like rate(), irate(), and increase() take range vectors. Aggregation operators like sum(), avg(), and topk() take instant vectors. The result of a query can be graphed, viewed in the expression browser, or consumed via the HTTP API .
PromQL is not SQL. It does not have joins, subqueries (in the traditional sense), or arbitrary expressions. It is designed for the specific operations that are useful on time series: filtering by label, aggregating across labels, computing rates over time windows, and comparing one metric to another. This narrowness is a feature: it makes queries predictable and the query engine optimizable.
Complete Example Session
# ============================================
# PART 1: THE PULL MODEL
# ============================================
# Prometheus scrapes targets at a configured interval.
# Target: http://web01:9100/metrics
# Interval: 15 seconds
#
# If the scrape fails, up{instance="web01"} = 0.
# The target does not push data to Prometheus.# ============================================
# PART 2: THE DATA MODEL
# ============================================
# A time series is identified by metric name + labels.
# node_cpu_seconds_total{instance="web01", mode="user", cpu="0"}
#
# Changing any label value creates a new time series.
# Labels are the query dimensions.# ============================================
# PART 3: COUNTER
# ============================================
# http_requests_total{job="api", status="200"} 12345
#
# Monotonically increasing.
# Resets to 0 on restart.
# Query with rate() to get per-second rate.# ============================================
# PART 4: GAUGE
# ============================================
# node_memory_MemAvailable_bytes{instance="web01"} 2147483648
#
# Can go up or down.
# Represents a current state.
# Query directly or with avg_over_time().# ============================================
# PART 5: HISTOGRAM
# ============================================
# http_request_duration_seconds_bucket{le="0.1"} 500
# http_request_duration_seconds_bucket{le="0.5"} 900
# http_request_duration_seconds_bucket{le="+Inf"} 1000
# http_request_duration_seconds_sum 125.3
# http_request_duration_seconds_count 1000
#
# Buckets + sum + count.
# Query with histogram_quantile() for percentiles.# ============================================
# PART 6: SUMMARY
# ============================================
# rpc_duration_seconds{quantile="0.5"} 0.05
# rpc_duration_seconds{quantile="0.9"} 0.12
# rpc_duration_seconds_sum 25.3
# rpc_duration_seconds_count 500
#
# Client-side quantiles.
# Cannot be aggregated across instances.# ============================================
# PART 7: THE ARCHITECTURE
# ============================================
# Retrieval Worker ──► TSDB ──► HTTP Server
# │ │
# ▼ ▼
# Exporters PromQL / API
# │
# ▼
# Targets (/metrics)# ============================================
# PART 8: ALERTING FLOW
# ============================================
# Prometheus evaluates alerting rules.
# Firing alerts are sent to Alertmanager.
# Alertmanager deduplicates, groups, routes.
# Notifications go to email, Slack, PagerDuty.# ============================================
# PART 9: STORAGE RETENTION
# ============================================
# Default retention: 15 days.
# --storage.tsdb.retention.time=15d
# --storage.tsdb.retention.size=10G
#
# Local TSDB is not for long-term storage.
# Use remote_write to Thanos or Mimir for that.# ============================================
# PART 10: WHAT PROMETHEUS IS NOT
# ============================================
# Not a log system. Use Loki for logs.
# Not 100% accurate. Not for billing.
# Not a distributed database. Single-node by design.
# Not a push-based system. Pull is the default.The ten parts covered the pull model, the data model, the four metric types, the component architecture, the alerting flow, storage retention, and the boundaries of Prometheus’s design.
Quick Reference
Core components
| Component | Responsibility |
|---|---|
| Retrieval worker | Scrapes /metrics endpoints |
| TSDB | Stores samples locally in time-bounded blocks |
| HTTP server | Serves PromQL queries and API |
| Exporters | Expose metrics for non-native systems |
| Alertmanager | Deduplicates, groups, routes alerts |
| Pushgateway | Receives pushed metrics from short-lived jobs |
Metric types
| Type | Behavior | Use case |
|---|---|---|
| Counter | Monotonically increasing | Request counts, errors |
| Gauge | Arbitrary up/down | Memory, temperature, queue length |
| Histogram | Buckets + sum + count | Request durations, response sizes |
| Summary | Client-side quantiles | Precise percentiles per instance |
Data model notation
| Element | Example |
|---|---|
| Metric name | http_requests_total |
| Labels | {job="api", status="200"} |
| Full series | http_requests_total{job="api", status="200"} |
Common PromQL operations
| Query | Purpose |
|---|---|
metric_name | Select all series for a metric |
metric{label="value"} | Filter by label |
rate(metric[5m]) | Per-second rate over 5 minutes |
sum by (job) (rate(...)) | Aggregate by label |
topk(3, ...) | Top 3 series by value |
Best Practices
✅ Do This:
# Use labels for dimensions you filter by
node_cpu_seconds_total{instance="web01", mode="user"} # ✅# Use rate() for counters
rate(http_requests_total[5m]) # ✅# Use service discovery in dynamic environments
kubernetes_sd_configs: # ✅# Set retention limits
--storage.tsdb.retention.time=15d # ✅❌ Don’t Do This:
# Use rate() on a gauge
rate(node_memory_MemAvailable_bytes[5m]) # ❌ (meaningless)# Use high-cardinality labels
http_requests_total{user_id="12345"} # ❌ (cardinality explosion)# Expect Prometheus to store logs
# Use Loki for logs # ❌# Rely on local TSDB for long-term retention
# Use remote_write for long-term storage # ❌Common Pitfalls
| Pitfall | Why It Happens | Fix |
|---|---|---|
| Memory grows unbounded | High-cardinality labels | Reduce label cardinality; drop unused series |
rate() returns nothing | Query uses instant vector instead of range vector | Use rate(metric[5m]), not rate(metric) |
| Histogram quantiles are wrong | Bucket boundaries too coarse | Adjust bucket configuration |
Target shows up=0 | Scrape failed (network, auth, wrong path) | Check target reachability and /metrics endpoint |
| Retention too short | Default 15 days | Increase --storage.tsdb.retention.time |
| Single instance is a SPOF | Local storage, no HA | Run duplicate Prometheus servers with Alertmanager deduplication |
Real-World Examples
1. Scrape a Node Exporter
scrape_configs:
- job_name: node
static_configs:
- targets: ['web01:9100']2. Scrape a counter metric
http_requests_total{job="api"}3. Compute a rate
rate(http_requests_total[5m])4. Aggregate by label
sum by (status) (rate(http_requests_total[5m]))5. Query a gauge
node_memory_MemAvailable_bytes{instance="web01"}6. Compute a histogram quantile
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))7. Top K series
topk(3, sum by (app, proc) (rate(instance_cpu_time_ns[5m])))8. Alerting rule
- alert: HighCPU
expr: (sum(rate(container_cpu_usage_seconds_total[5m])) by (instance) * 100) > 80
for: 5m9. Remote write
remote_write:
- url: https://thanos.example.com/api/v1/write10. Pushgateway for batch job
echo "batch_job_duration_seconds 42.5" | curl --data-binary @- http://pushgateway:9091/metrics/job/batchVisual
The Prometheus architecture
┌─────────────────────────────────────────────────────────────┐
│ PROMETHEUS SERVER │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Retrieval │───►│ TSDB │───►│ HTTP Server │ │
│ │ Worker │ │ (local disk)│ │ (PromQL) │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │ │ │
│ │ ▼ │
│ │ Grafana / API │
│ ▼ │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Exporters │ │ Alertmanager│ │
│ │ (/metrics) │ │ (notifies) │ │
│ └──────────────┘ └──────────────┘ │
│ │ │
│ ▼ │
│ Targets (nodes, apps, databases) │
└─────────────────────────────────────────────────────────────┘The data model
┌─────────────────────────────────────────────────────────────┐
│ TIME SERIES │
│ │
│ Metric name: http_requests_total │
│ Labels: {job="api", instance="web01", status="200"} │
│ │
│ Timestamp Value │
│ 2026-11-05T10:00 12345 │
│ 2026-11-05T10:01 12389 │
│ 2026-11-05T10:02 12401 │
│ │
│ Each unique label combination is a separate time series. │
└─────────────────────────────────────────────────────────────┘Metric type behavior
┌─────────────────────────────────────────────────────────────┐
│ COUNTER GAUGE │
│ │
│ 12345 ──► 12389 ──► 12401 2147483648 ──► 1073741824 │
│ (always up) (up and down) │
│ rate() = meaningful rate() = noise │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ HISTOGRAM SUMMARY │
│ │
│ Buckets + sum + count Client-side quantiles │
│ Aggregatable across NOT aggregatable across │
│ instances instances │
│ histogram_quantile() Direct quantile query │
└─────────────────────────────────────────────────────────────┘What Prometheus is not
┌─────────────────────────────────────────────────────────────┐
│ BOUNDARIES │
│ │
│ IS: IS NOT: │
│ Metrics Logs (use Loki) │
│ Pull-based Push-based (except Pushgateway) │
│ Single-node Distributed database │
│ Numeric time series Event tracking │
│ Reliable under failure 100% accurate (not for billing) │
└─────────────────────────────────────────────────────────────┘Summary
| Item | Value |
|---|---|
| Collection model | Pull (HTTP scrape of /metrics) |
| Default scrape interval | 15 seconds |
| Data model | Metric name + labels → time series |
| Metric types | Counter, Gauge, Histogram, Summary |
| Storage | Local TSDB, ~2-hour blocks |
| Default retention | 15 days |
| Query language | PromQL |
| Alerting | Prometheus evaluates, Alertmanager notifies |
| Long-term storage | remote_write to external systems |
| Log support | None (use Loki) |
Key takeaways:
- Prometheus pulls metrics, it does not receive them. The server scrapes targets at a configured interval, which makes target health directly observable through the
upmetric and gives the server control over collection load. Pushgateway exists only for short-lived batch jobs . - The data model is metric name plus labels. Every unique label combination is a separate time series. Labels are the query dimensions, and high-cardinality labels are the primary cause of memory and storage growth .
- The four metric types have different query semantics. Counters are queried with
rate(). Gauges are queried directly. Histograms produce aggregatable bucket series. Summaries produce client-side quantiles that cannot be aggregated across instances . - The architecture separates scraping, storage, querying, and alerting. The retrieval worker scrapes, the TSDB stores, the HTTP server queries, and Alertmanager routes alerts. Exporters bridge systems that do not natively expose Prometheus metrics .
- PromQL is narrow by design. It filters by label, aggregates across labels, and computes rates over time windows. It does not have joins or arbitrary expressions, which makes queries predictable and the engine optimizable .
- Prometheus is not a log system, not a distributed database, and not 100% accurate. It is designed for reliability during outages, not for billing-grade precision or event tracking. Logs belong in Loki; long-term storage belongs in Thanos or Mimir via
remote_write.
Remember: Prometheus is a time series database with a pull-based collection model and a query language. The pull model makes target health visible and collection load controllable. The label-based data model makes ad-hoc queries possible without pre-planning a metric hierarchy. The four metric types encode different kinds of measurements, and using the wrong type produces meaningless queries. The architecture separates concerns: scraping, storage, querying, and alerting each have their own component. And the boundaries are deliberate: Prometheus stores metrics, not logs; it runs as a standalone server, not a distributed cluster; it prioritizes reliability over perfect accuracy. Understanding those boundaries is what prevents the mistake of trying to make Prometheus do something it was not designed for.
Stop using slow, ad-bloated tool sites! 🤮
🔎 Search “KandZ Tools” on Google to use many professional utilities for free.
KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)
⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free
🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!