LFCE 123 🐧 Loki — Concepts
Loki is a log aggregation system built by Grafana Labs with a design philosophy that inverts the assumptions of traditional log platforms. Where Elasticsearch and Splunk index the full text of every log line, Loki indexes only a small set of labels attached to each log stream. The log content itself is compressed into chunks and stored in object storage. This trade-off dramatically reduces storage cost and operational complexity, but it shifts the burden of query performance onto the label schema: a well-designed set of low-cardinality labels makes queries fast, while a high-cardinality schema makes them slow and expensive .
This chapter covers the conceptual foundation of Loki: the label-based indexing model that distinguishes it from full-text search systems, the components that make up a Loki deployment and how data flows through them, the LogQL query language that provides “schema at query time,” and the label cardinality problem that determines whether a Loki deployment performs well or collapses under its own index. By the end, you will understand why Loki is not a drop-in replacement for Elasticsearch, and what kinds of logging workloads it is designed to serve.
Key point: Loki indexes labels, not log content. Every unique combination of label values creates a new log stream with its own index entry. High-cardinality labels—user IDs, request IDs, trace IDs—cause a cardinality explosion that degrades ingestion and query performance.
Why Loki exists
The full-text indexing cost problem. Elasticsearch and similar systems build an inverted index over every field of every log record. This makes arbitrary full-text queries fast, but the index is typically 3–10 times the size of the raw log data, and the CPU cost of tokenization dominates the ingest path . For organizations that generate terabytes of logs per day but query only a small fraction of them, this cost is difficult to justify. Loki’s creators observed that most operational queries filter by a small set of dimensions—service name, environment, cluster—and then scan the matching logs for a string. Indexing the full text is unnecessary for that pattern .
The object storage problem. Traditional log systems store their indexes and data on local disks or block storage attached to the cluster nodes. Scaling requires adding nodes and rebalancing shards. Loki stores compressed chunks in object storage such as S3, GCS, or Azure Blob Storage, and keeps only a small index that maps labels to chunk offsets . This means storage scales independently of compute, and the cost per gigabyte is the cost of object storage rather than the cost of provisioned SSD or HDD. The trade-off is that queries must fetch chunks from object storage and decompress them, which is slower than querying an in-memory inverted index .
The operational simplicity problem. A production Elasticsearch cluster requires careful sizing of shards, management of heap and garbage collection, and monitoring of cluster state. Loki’s architecture is simpler: a distributor routes incoming streams to ingesters, ingesters build chunks and flush them to object storage, and queriers fetch and scan those chunks . The system has fewer moving parts, and the parts that exist are stateless or nearly so. For teams that already run Prometheus and Grafana, Loki integrates with the same label model and the same visualization layer .
a. The label-based indexing model
Loki’s core architectural decision is that it indexes metadata, not content. A log stream is defined by a set of labels, and every unique combination of label values creates a separate stream. When a log line arrives, Loki assigns it to a stream based on its labels, appends it to that stream’s in-memory chunk, and eventually flushes the chunk to object storage. The index records which chunks belong to which label combination, but it does not record the words inside the log lines .
# A stream is identified by labels
{service_name="nginx", environment="prod", cluster="us-east-1"}
# This combination of labels creates one stream.
# A different environment value creates a different stream.
{service_name="nginx", environment="staging", cluster="us-east-1"}The consequence is that a query must specify at least one label selector. Loki uses the selector to find the relevant chunks, fetches them from storage, decompresses them, and then applies any filter expressions to the log content. This is why Loki queries are fast when the label selector narrows the data to a small number of chunks, and slow when it selects a large fraction of the stored logs .
The contrast with full-text indexing is stark. Elasticsearch can answer “find all logs containing ‘connection refused'” by looking up the term in its inverted index. Loki must scan every log line in the streams that match the selector. For a small number of streams, this is fast. For a broad selector that matches terabytes of logs, it is slow and resource-intensive .
b. The architecture and data flow
A Loki deployment consists of several components that can be run as a single binary for development, as a monolithic service for moderate scale, or as separate microservices for large-scale production. The core components and their responsibilities are consistent across deployment modes.
The distributor is the entry point for log ingestion. It receives streams from agents like Grafana Alloy or Promtail, validates the labels, applies rate limits, and uses consistent hashing to route each stream to the appropriate ingesters. The same stream always goes to the same set of ingesters, which is what allows the ingesters to build chunks for that stream .
The ingester buffers log lines in memory, building chunks per stream. When a chunk reaches a target size or age, the ingester compresses it and flushes it to object storage. Ingesters also replicate each stream across multiple instances, typically three, so that a single ingester failure does not lose data. Recent logs remain queryable from the ingester’s in-memory buffer, which is why recent queries are faster than historical ones .
The querier handles LogQL queries. It determines which chunks to fetch based on the label selector, retrieves recent data from ingesters and historical data from object storage, applies the filter and parser expressions, and returns the matching log lines. The query frontend sits in front of the queriers, splitting large queries into smaller parallel queries, queuing them, and caching results to reduce load .
# Simplified data flow
# Agent ──► Distributor ──► Ingester ──► Object Storage
# ▲
# Client ──► Query Frontend ──► Querier ┘The compactor runs in the background, merging small chunks into larger ones, deduplicating index entries, and enforcing retention policies. Without compaction, the number of chunks grows unbounded and query performance degrades .
c. LogQL and schema at query time
LogQL is Loki’s query language. It is syntactically similar to PromQL, which makes it familiar to teams already using Prometheus, but it operates on logs rather than metrics. A LogQL query has two parts: a mandatory log stream selector enclosed in braces, and an optional log pipeline that filters, parses, and formats the log lines .
# Stream selector only
{service_name="api", environment="prod"}
# Stream selector with a line filter
{service_name="api"} |= "error"
# Stream selector with JSON parsing and field filter
{service_name="api"} | json | status_code >= 500The stream selector uses label matching operators: = for exact match, != for inequality, =~ for regex match, and !~ for regex non-match. The pipeline uses line filters (|= for contains, |~ for regex), parsers (| json, | logfmt, | regexp), and formatters (| line_format, | label_format) .
The key concept is “schema at query time.” Because Loki does not require a schema at ingest, a log line can be parsed as JSON, logfmt, or a custom pattern when the query runs. Fields extracted by a parser exist only for the duration of the query and add no indexing overhead. This means a team can change how they parse their logs without changing how the logs are stored .
LogQL also supports metric queries that generate time series from log data. sum(rate({service_name="api"} |= "error" [5m])) returns a rate of error lines over time, which can be graphed in Grafana alongside Prometheus metrics. This blurs the line between logs and metrics and is one of Loki’s integration advantages .
d. The cardinality problem
The single most important operational consideration in Loki is label cardinality. Every unique combination of label values creates a new log stream, and every stream has its own index entry and its own set of chunks. A small number of low-cardinality labels produces a manageable number of streams. A high-cardinality label produces an explosion of streams that overwhelms the index and degrades performance .
# Good labels: low cardinality, stable values
{service_name="payment-api", environment="prod", cluster="us-east-1"}
# Bad labels: high cardinality, unique per request
{service_name="payment-api", user_id="12345", request_id="abc-def-ghi"}The distinction is not about the number of distinct values in the entire dataset. It is about the number of distinct values per unit of time and the number of streams that result. A label like http_status has a small set of possible values (200, 404, 500), but if a service generates thousands of requests per second across hundreds of status code combinations, the stream count can still be large. A label like request_id is catastrophically bad because every request creates a new stream .
The correct pattern is to use labels for dimensions that are filtered in every query, and to keep those dimensions low-cardinality. Values that are high-cardinality but occasionally needed for filtering should be placed in the log line itself and extracted at query time with a parser, or attached as structured metadata in Loki 3.0 and later, which does not create new streams .
Complete Example Session
# ============================================
# PART 1: THE LABEL-BASED INDEX
# ============================================
# A log stream is defined by its labels.
# {service_name="nginx", env="prod"}
#
# Every unique combination creates a new stream.
# The index maps labels to chunks, not words to lines.# ============================================
# PART 2: THE DATA FLOW
# ============================================
# Agent ──► Distributor ──► Ingester ──► Object Storage
# ▲
# Client ──► Query Frontend ──► Querier ┘
#
# Distributor: validates, hashes, routes
# Ingester: buffers, builds chunks, flushes
# Querier: fetches chunks, applies pipeline# ============================================
# PART 3: THE STREAM SELECTOR
# ============================================
# Mandatory part of every LogQL query
{service_name="api", environment="prod"}
#
# Operators:
# = exact match
# != not equal
# =~ regex match
# !~ regex not match# ============================================
# PART 4: THE LOG PIPELINE
# ============================================
# Optional filters and parsers applied after selection
{service_name="api"} |= "error" | json | status >= 500
#
# |= line contains string
# |~ line matches regex
# | json parse JSON fields
# | logfmt parse logfmt fields# ============================================
# PART 5: SCHEMA AT QUERY TIME
# ============================================
# No schema required at ingest.
# Parse as JSON, logfmt, or regex at query time.
#
# Fields extracted by a parser exist only for the query.
# They add no indexing overhead.# ============================================
# PART 6: METRIC QUERIES
# ============================================
# Generate time series from logs
sum(rate({service_name="api"} |= "error" [5m]))
#
# Returns a rate of error lines over time.
# Can be graphed alongside Prometheus metrics.# ============================================
# PART 7: GOOD LABELS
# ============================================
# Low cardinality, stable values
{env="prod", cluster="us-east", app="payment"}
#
# These dimensions are filtered in every query.
# They have a small, predictable set of values.# ============================================
# PART 8: BAD LABELS
# ============================================
# High cardinality, unique values
{user_id="12345", request_id="abc", trace_id="xyz"}
#
# Every unique value creates a new stream.
# This causes cardinality explosion.# ============================================
# PART 9: THE COMPACTOR
# ============================================
# Background process
# - Merges small chunks into larger ones
# - Deduplicates index entries
# - Enforces retention policies
#
# Without compaction, chunk count grows unbounded.# ============================================
# PART 10: LOKI VS ELASTICSEARCH
# ============================================
# Elasticsearch: full-text index, fast arbitrary queries,
# 3-10x storage overhead, high CPU ingest
#
# Loki: label index, fast selector queries, cheap storage,
# slow full-text scan, low operational overheadThe ten parts covered the label-based index, the data flow, the stream selector, the log pipeline, schema at query time, metric queries, good labels, bad labels, the compactor, and the comparison with Elasticsearch.
Quick Reference
Core components
| Component | Responsibility |
|---|---|
| Distributor | Validates, hashes, routes incoming streams |
| Ingester | Buffers, builds chunks, flushes to storage |
| Querier | Fetches chunks, applies LogQL pipeline |
| Query Frontend | Splits, queues, caches queries |
| Compactor | Merges chunks, enforces retention |
LogQL query structure
| Part | Required | Example |
|---|---|---|
| Stream selector | Yes | {service_name="api"} |
| Line filter | No | |= "error" |
| Parser | No | | json |
| Field filter | No | | status >= 500 |
| Formatter | No | | line_format "{{.msg}}" |
Label cardinality
| Label type | Example | Recommendation |
|---|---|---|
| Low cardinality | env, cluster, app | Use as labels |
| High cardinality | user_id, request_id, trace_id | Use structured metadata or parse at query time |
Storage model
| Data | Where it lives | Indexed |
|---|---|---|
| Labels | Index (TSDB) | Yes |
| Log content | Compressed chunks in object storage | No |
| Chunk metadata | Index | Yes (offsets) |
Best Practices
✅ Do This:
# Use low-cardinality labels for dimensions filtered in every query
{env="prod", app="api", cluster="us-east"} # ✅# Parse high-cardinality fields at query time
{app="api"} | json | user_id="12345" # ✅# Use structured metadata for high-cardinality values (Loki 3.0+)
# Attach trace_id as metadata, not a label # ✅# Use the stream selector to narrow the dataset before filtering
{app="api"} |= "error" # ✅❌ Don’t Do This:
# Use high-cardinality values as labels
{app="api", request_id="abc-123"} # ❌ (cardinality explosion)# Query without a label selector
{} |= "error" # ❌ (scans everything)# Treat Loki as a full-text search engine
# Arbitrary content search is slow and expensive # ❌# Skip compaction in production
# Chunk count grows unbounded # ❌Common Pitfalls
| Pitfall | Why It Happens | Fix |
|---|---|---|
| Cardinality explosion | High-cardinality values used as labels | Use low-cardinality labels; parse high-cardinality fields at query time |
| Slow queries | Stream selector matches too many chunks | Narrow the selector; add a label filter |
service_name shows unknown_service | Agent not forwarding service metadata | Configure the agent to include service_name or a fallback label |
| Ingestion rate limit (HTTP 429) | Too many streams or too much volume | Reduce cardinality; increase limits; check agent configuration |
| Storage grows unexpectedly | Retention not configured | Enable compactor retention policies |
| Out-of-order writes rejected | Log timestamps not ascending within a stream | Ensure agents send in order; configure out-of-order acceptance if needed |
Real-World Examples
1. Basic stream selector
{service_name="nginx", environment="prod"}2. Line filter
{service_name="api"} |= "connection refused"3. JSON parser and field filter
{service_name="api"} | json | status_code >= 5004. Logfmt parser
{service_name="api"} | logfmt | level="error"5. Regex parser
{service_name="api"} | regexp `(?P<method>\w+) (?P<path>\S+)`6. Metric query
sum(rate({service_name="api"} |= "error" [5m]))7. Good label set
{env="prod", cluster="us-east", app="payment"}8. Bad label set
{app="payment", user_id="12345", request_id="abc"}9. Structured metadata (Loki 3.0+)
# Attach trace_id as metadata, not a stream label10. Narrow selector for performance
{app="api", env="prod", cluster="us-east"} |= "timeout"Visual
Label-based vs full-text indexing
┌─────────────────────────────────────────────────────────────┐
│ ELASTICSEARCH │
│ │
│ Log line ──► Tokenize every field ──► Inverted index │
│ │
│ Index contains: word → document ID │
│ Storage: 3-10x raw size │
│ Query: fast for any field │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ LOKI │
│ │
│ Log line ──► Extract labels ──► Compress chunk ──► S3 │
│ │ │
│ ▼ │
│ Index: labels → chunk offsets │
│ │
│ Index contains: label combination → chunks │
│ Storage: near raw size (compressed) │
│ Query: fast for selector, slow for content scan │
└─────────────────────────────────────────────────────────────┘Data flow
┌─────────────────────────────────────────────────────────────┐
│ LOKI DATA FLOW │
│ │
│ Agent (Alloy/Promtail) │
│ │ │
│ ▼ │
│ Distributor ──► hash ring ──► Ingester │
│ │ │
│ ▼ │
│ Chunk buffer │
│ │ │
│ ▼ │
│ Object Storage (S3/GCS) │
│ ▲ │
│ │ │
│ Client (Grafana) ──► Query Frontend ──► Querier │
│ │
│ Querier fetches chunks from storage and ingesters. │
└─────────────────────────────────────────────────────────────┘Cardinality explosion
┌─────────────────────────────────────────────────────────────┐
│ LABEL CARDINALITY │
│ │
│ {app="api", env="prod"} │
│ → 1 stream per app/env combination │
│ → 10 apps × 3 envs = 30 streams │
│ │
│ {app="api", env="prod", user_id="12345"} │
│ → 1 stream per user │
│ → 10 apps × 3 envs × 100,000 users = 3,000,000 streams │
│ │
│ Each stream has its own index entry and chunk set. │
│ The index and memory usage grow with stream count. │
└─────────────────────────────────────────────────────────────┘LogQL query anatomy
┌─────────────────────────────────────────────────────────────┐
│ LOGQL QUERY │
│ │
│ {app="api", env="prod"} |= "error" | json | status >= 500 │
│ └─────────────────────┘ └────────┘ └────┘ └────────────┘ │
│ selector line filter parser field filter │
│ │
│ Selector: mandatory. Narrows chunks from index. │
│ Pipeline: optional. Filters and parses log content. │
│ │
│ The selector does the expensive work (chunk retrieval). │
│ The pipeline does the cheap work (in-memory scan). │
└─────────────────────────────────────────────────────────────┘Summary
| Item | Value |
|---|---|
| Index model | Labels only, not log content |
| Storage | Compressed chunks in object storage (S3/GCS) |
| Query language | LogQL |
| Mandatory query part | Stream selector {label="value"} |
| Optional query part | Log pipeline |=, | json, etc. |
| Cardinality risk | High-cardinality labels create stream explosion |
| Architecture | Distributor, Ingester, Querier, Query Frontend, Compactor |
| Main trade-off | Cheap storage and simple ops vs slow full-text search |
Key takeaways:
- Loki indexes labels, not content. The index maps label combinations to chunk offsets. A query must specify a stream selector to narrow the dataset before scanning log lines. Arbitrary full-text search is possible but slow and expensive compared to Elasticsearch .
- Cardinality is the single most important operational concern. Every unique label combination creates a new stream. Low-cardinality labels like
env,cluster, andappare safe. High-cardinality labels likeuser_id,request_id, andtrace_idcause a cardinality explosion that degrades ingestion, query performance, and memory usage . - LogQL provides schema at query time. Logs do not need a schema at ingest. The
| json,| logfmt, and| regexpparsers extract fields during the query, and those fields add no indexing overhead. This makes the label schema the only schema decision that matters at ingest time . - The architecture separates ingestion from querying. The distributor routes streams, the ingester buffers and flushes chunks, and the querier fetches and scans chunks. The query frontend splits and caches queries, and the compactor manages chunk lifecycle and retention .
- Loki is not Elasticsearch. It does not replace full-text search for complex analytics. It replaces it for operational logging where the query pattern is “filter by service and environment, then grep for a string.” For that pattern, Loki is dramatically cheaper and simpler. For arbitrary content search across terabytes of logs, it is the wrong tool .
Remember: Loki’s design is a deliberate trade-off. It gives up the ability to answer arbitrary content queries quickly in exchange for storage that scales with object storage and an operational model that does not require managing a search cluster. The label schema is the contract: get it right, and Loki is fast and cheap. Get it wrong, and the index becomes the bottleneck. Use labels for the dimensions you filter by in every query. Keep them low-cardinality. Parse everything else at query time. And do not expect Loki to replace a full-text search engine for workloads that need it.
Stop using slow, ad-bloated tool sites! 🤮
🔎 Search “KandZ Tools” on Google to use many professional utilities for free.
KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)
⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free
🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!