LFCE 124 🐧 Promtail — Setup - KandZ

⏱️ Estimated read: 13 min | |

Promtail is the log collection agent designed for Grafana Loki. Its job is narrow: discover log files on a host, attach labels to the log streams, track how far it has read into each file, and push the lines to a Loki server over HTTP. It does not index, store, or query logs. It reads from the filesystem and writes to Loki’s push API, and it maintains a positions file so that a restart does not re-send or skip lines.

Quick How-To Guides & Fast FixesNeed a quick fix or step-by-step solution? Browse our fast answers on HowTo.KandZ.me:

Operating Systems CLI →Dev Environment Languages →Web Network Tools →Security Accounts →Security Access Tech →Device Phone Basics →Social Messaging →Digital Files Media →

This chapter covers the setup of Promtail on a Linux host: downloading the binary, writing a configuration that defines the Loki endpoint and the log files to scrape, configuring the positions file, and running Promtail as a systemd service. It also covers the pipeline stages that attach labels and parse structured content from log lines, because a Promtail configuration without labels produces streams that are difficult to query in Loki.

Key point: Promtail does not send logs anywhere without a clients block, and it does not read any files without a scrape_configs block. A configuration missing either one starts successfully and does nothing useful.

Why Promtail setup has this structure

The positions file problem. Promtail tracks the last read offset for each log file in a positions file. If this file is deleted or unwritable, Promtail cannot resume from where it stopped after a restart. It either re-reads files from the beginning or fails to track progress. The default location is /tmp/positions.yaml, which is cleared on reboot on many systems. A production setup must place the positions file on persistent storage, typically /var/lib/promtail/positions.yaml .

The label problem. Loki indexes labels, not log content. A Promtail configuration that scrapes /var/log/*.log without attaching labels produces streams identified only by the file path, which is not a queryable dimension. The labels block in each static_configs entry attaches static labels like job, host, and environment to every line from that file. Without these labels, the logs are stored but difficult to find .

The pipeline problem. Raw log lines often contain structured data inside the text: JSON objects, logfmt key-value pairs, or application-specific formats. Promtail’s pipeline stages parse this data and promote extracted fields to labels or structured metadata, which makes them queryable in Loki. A setup without pipeline stages treats every line as opaque text .

a. Downloading and installing the binary

Promtail is distributed as a single binary. The download URL follows the pattern https://github.com/grafana/loki/releases/download/v<version>/promtail-linux-amd64.zip. The binary can be placed in /usr/local/bin or run from any directory.

# Download the latest release for Linux x86_64
curl -O -L "https://github.com/grafana/loki/releases/latest/download/promtail-linux-amd64.zip"

# Extract and install
unzip promtail-linux-amd64.zip
chmod a+x promtail-linux-amd64
sudo mv promtail-linux-amd64 /usr/local/bin/promtail

# Verify
promtail --version

For Debian and Ubuntu, the promtail package is available in some repositories. The binary method is version-independent and works on any distribution .

b. The configuration file

Promtail reads a YAML configuration file passed with --config.file. The file has four top-level sections that matter for a basic setup: server, positions, clients, and scrape_configs .

# /etc/promtail/config.yaml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /var/lib/promtail/positions.yaml

clients:
  - url: http://loki.example.com:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          host: ${HOSTNAME}
          __path__: /var/log/*.log

The server block sets the HTTP port for Promtail’s own metrics and health endpoints. The positions block sets the path to the offset file. The clients block sets the Loki push URL. The scrape_configs block defines which files to read and what labels to attach .

The __path__ label is special: it tells Promtail which files to read. Every other label in the labels block is attached to the log stream and becomes a queryable dimension in Loki. A configuration can have multiple scrape_configs entries, each with its own job name, path, and labels.

c. Pipeline stages

Pipeline stages transform log lines before they are sent to Loki. They are defined inside a scrape_configs entry under pipeline_stages. The stages run in order, and each stage can parse, transform, or act on the extracted data .

scrape_configs:
  - job_name: application
    static_configs:
      - targets: [localhost]
        labels:
          job: app
          __path__: /var/log/app/*.log
    pipeline_stages:
      - json:
          expressions:
            level: level
            message: message
            user_id: user.id
      - labels:
          level:
      - timestamp:
          source: time
          format: RFC3339

The json stage parses the log line as JSON and extracts the specified expressions. The labels stage promotes the level field to a label, which means every unique value of level creates a stream. The timestamp stage replaces the log line’s timestamp with the value from the time field, parsed as RFC3339 .

A label promotion is not free. Promoting a high-cardinality field like user_id to a label creates a new stream for every unique value, which causes the cardinality explosion described in the Loki chapter. The json stage can extract user_id for use in later stages without promoting it to a label. Only fields that are filtered on in queries should become labels.

d. Running as a systemd service

A systemd unit makes Promtail start at boot, restart on failure, and log its own output. The unit should run Promtail as a dedicated user, or as root if the log files are not readable by other users.

# /etc/systemd/system/promtail.service
[Unit]
Description=Promtail log collector
After=network.target

[Service]
Type=simple
User=promtail
Group=promtail
ExecStart=/usr/local/bin/promtail --config.file=/etc/promtail/config.yaml
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

If Promtail runs as a non-root user, that user must have read access to the log files and write access to the positions directory. The systemd-journal group is required if Promtail reads from the journal. The Ansible role documentation notes that Promtail defaults to a rootless mode using ACLs to read logs without root permissions .

After creating the unit, enable and start it:

sudo systemctl daemon-reload
sudo systemctl enable --now promtail
systemctl status promtail --no-pager

The Promtail HTTP endpoint on port 9080 exposes metrics and a /ready endpoint for health checks. A quick verification is curl http://localhost:9080/ready.

Complete Example Session

# ============================================
# PART 1: DOWNLOAD AND INSTALL
# ============================================
curl -O -L "https://github.com/grafana/loki/releases/latest/download/promtail-linux-amd64.zip"
unzip promtail-linux-amd64.zip
chmod a+x promtail-linux-amd64
sudo mv promtail-linux-amd64 /usr/local/bin/promtail
promtail --version

# ============================================
# PART 2: CREATE THE CONFIG DIRECTORY
# ============================================
sudo mkdir -p /etc/promtail /var/lib/promtail

# ============================================
# PART 3: WRITE THE CONFIGURATION
# ============================================
sudo tee /etc/promtail/config.yaml <<'EOF'
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /var/lib/promtail/positions.yaml

clients:
  - url: http://loki.example.com:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets: [localhost]
        labels:
          job: varlogs
          host: ${HOSTNAME}
          __path__: /var/log/*.log
EOF

# ============================================
# PART 4: VALIDATE THE CONFIGURATION
# ============================================
promtail --config.file=/etc/promtail/config.yaml --dry-run

# Promtail parses the config and exits without starting.
# A syntax error is reported at this stage.

# ============================================
# PART 5: RUN PROMTAIL IN THE FOREGROUND (TEST)
# ============================================
promtail --config.file=/etc/promtail/config.yaml

# The log output shows:
# - the Loki client URL
# - the files being tailed
# - any errors reaching Loki

# ============================================
# PART 6: CREATE THE SYSTEMD UNIT
# ============================================
sudo tee /etc/systemd/system/promtail.service <<'EOF'
[Unit]
Description=Promtail log collector
After=network.target

[Service]
Type=simple
User=promtail
Group=promtail
ExecStart=/usr/local/bin/promtail --config.file=/etc/promtail/config.yaml
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

# ============================================
# PART 7: CREATE THE PROMTAIL USER
# ============================================
sudo useradd --system --no-create-home --shell /usr/sbin/nologin promtail
sudo chown promtail:promtail /var/lib/promtail

# Grant read access to log files
# Option A: add to systemd-journal group for journal access
sudo usermod -aG systemd-journal promtail

# Option B: use ACLs for specific log directories
# sudo setfacl -R -m u:promtail:rX /var/log/app

# ============================================
# PART 8: START THE SERVICE
# ============================================
sudo systemctl daemon-reload
sudo systemctl enable --now promtail
systemctl status promtail --no-pager

# ============================================
# PART 9: VERIFY THE HEALTH ENDPOINT
# ============================================
curl http://localhost:9080/ready
# Expected: "ready"

curl http://localhost:9080/metrics | grep promtail_sent_bytes_total
# Shows bytes sent to Loki.

# ============================================
# PART 10: ADD A PIPELINE STAGE
# ============================================
sudo tee /etc/promtail/config.yaml <<'EOF'
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /var/lib/promtail/positions.yaml

clients:
  - url: http://loki.example.com:3100/loki/api/v1/push

scrape_configs:
  - job_name: application
    static_configs:
      - targets: [localhost]
        labels:
          job: app
          host: ${HOSTNAME}
          __path__: /var/log/app/*.log
    pipeline_stages:
      - json:
          expressions:
            level: level
            message: message
      - labels:
          level:
EOF

sudo systemctl restart promtail

The ten parts covered installation, directory creation, configuration, validation, foreground testing, the systemd unit, user creation, service start, health verification, and a pipeline stage.

Quick Reference

Configuration sections

Section	Purpose	Required
`server`	HTTP/gRPC listen ports	No (defaults apply)
`positions`	Offset file path	Recommended
`clients`	Loki push URL	Yes
`scrape_configs`	Files to read and labels	Yes

Key parameters

Parameter	Example	Purpose
`http_listen_port`	`9080`	Promtail metrics/health port
`positions.filename`	`/var/lib/promtail/positions.yaml`	Persistent offset file
`clients[].url`	`http://loki:3100/loki/api/v1/push`	Loki endpoint
`scrape_configs[].job_name`	`system`	Job identifier
`__path__`	`/var/log/*.log`	Files to scrape
`labels`	`job`, `host`	Stream labels

Common pipeline stages

Stage	Purpose	Example
`json`	Parse JSON log lines	`expressions: {level: level}`
`regex`	Parse with regex	`expression: '(?P<level>\w+)'`
`labels`	Promote extracted field to label	`labels: [level]`
`timestamp`	Replace log timestamp	`source: time`
`match`	Conditional stage execution	`selector: '{job="app"}'`

Verification

Check	Command
Config syntax	`promtail --dry-run`
Service status	`systemctl status promtail`
Health endpoint	`curl localhost:9080/ready`
Sent bytes	`curl localhost:9080/metrics \| grep sent_bytes`

Best Practices

✅ Do This:

# Place the positions file on persistent storage
positions:
  filename: /var/lib/promtail/positions.yaml               # ✅

# Attach labels that are queried in Loki
labels:
  job: varlogs
  host: ${HOSTNAME}                                        # ✅

# Validate before starting the service
promtail --config.file=/etc/promtail/config.yaml --dry-run # ✅

# Run as a non-root user with read access to logs
User=promtail                                              # ✅

❌ Don’t Do This:

# Leave the positions file in /tmp
positions:
  filename: /tmp/positions.yaml                            # ❌ (lost on reboot)

# Scrape without labels
static_configs:
  - targets: [localhost]
    labels:
      __path__: /var/log/*.log                             # ❌ (no queryable dimensions)

# Promote high-cardinality fields to labels
- labels:
    user_id:                                              # ❌ (cardinality explosion)

# Start the service without testing the config
systemctl start promtail                                   # ❌ (syntax errors fail silently)

Common Pitfalls

Pitfall	Why It Happens	Fix
Promtail starts but sends nothing	`clients` block missing or wrong URL	Verify `clients[].url` points to Loki
No logs appear in Loki	`__path__` does not match any files	Check the path glob and file permissions
Duplicate logs after restart	Positions file lost or unwritable	Set `positions.filename` to a persistent path
Promtail cannot read logs	Running as a user without read permission	Add user to appropriate group or use ACLs
Cardinality explosion	High-cardinality field promoted to label	Remove from `labels` stage; use `structured_metadata`
Service fails to start	Invalid YAML syntax	Run `promtail --dry-run` to find the error

Real-World Examples

1. Minimal configuration

server:
  http_listen_port: 9080
positions:
  filename: /var/lib/promtail/positions.yaml
clients:
  - url: http://loki:3100/loki/api/v1/push
scrape_configs:
  - job_name: system
    static_configs:
      - targets: [localhost]
        labels:
          job: varlogs
          __path__: /var/log/*.log

2. JSON pipeline

pipeline_stages:
  - json:
      expressions:
        level: level
  - labels:
      level:

3. Regex pipeline

pipeline_stages:
  - regex:
      expression: '^(?P<time>\S+) (?P<level>\w+) (?P<msg>.*)$'
  - labels:
      level:

4. Static labels

labels:
  environment: production
  cluster: us-east-1

5. Multiple scrape jobs

scrape_configs:
  - job_name: system
    static_configs:
      - targets: [localhost]
        labels:
          job: varlogs
          __path__: /var/log/*.log
  - job_name: app
    static_configs:
      - targets: [localhost]
        labels:
          job: applogs
          __path__: /var/log/app/*.log

6. Systemd journal scraping

scrape_configs:
  - job_name: journal
    journal:
      labels:
        job: systemd-journal

7. Timestamp parsing

pipeline_stages:
  - json:
      expressions:
        time: time
  - timestamp:
      source: time
      format: RFC3339

8. Drop stage

pipeline_stages:
  - drop:
      expression: '.*healthcheck.*'

9. Multi-line logs

pipeline_stages:
  - multiline:
      firstline: '^\d{4}-\d{2}-\d{2}'

10. Dry run

promtail --config.file=/etc/promtail/config.yaml --dry-run

Visual

Promtail data flow

┌─────────────────────────────────────────────────────────────┐
│  PROMTAIL DATA FLOW                                          │
│                                                              │
│  /var/log/*.log                                              │
│       │                                                      │
│       ▼                                                      │
│  Discovery (__path__ glob)                                   │
│       │                                                      │
│       ▼                                                      │
│  Read new lines (positions file tracks offset)               │
│       │                                                      │
│       ▼                                                      │
│  Pipeline stages (parse, label, transform)                   │
│       │                                                      │
│       ▼                                                      │
│  Loki push API (HTTP POST)                                   │
│       │                                                      │
│       ▼                                                      │
│  Loki                                                        │
│                                                              │
│  Positions file prevents re-reading on restart.              │
└─────────────────────────────────────────────────────────────┘

Configuration structure

┌─────────────────────────────────────────────────────────────┐
│  PROMTAIL CONFIG                                             │
│                                                              │
│  server:                                                     │
│    http_listen_port: 9080                                    │
│                                                              │
│  positions:                                                  │
│    filename: /var/lib/promtail/positions.yaml                │
│                                                              │
│  clients:                                                    │
│    - url: http://loki:3100/loki/api/v1/push                 │
│                                                              │
│  scrape_configs:                                             │
│    - job_name: system                                        │
│      static_configs:                                         │
│        - targets: [localhost]                                │
│          labels:                                             │
│            job: varlogs                                      │
│            __path__: /var/log/*.log                          │
│      pipeline_stages:                                        │
│        - json: { ... }                                       │
│        - labels: [ ... ]                                     │
└─────────────────────────────────────────────────────────────┘

Label vs pipeline decision

┌─────────────────────────────────────────────────────────────┐
│  LABEL OR NOT?                                               │
│                                                              │
│  Does the field have a small set of values?                  │
│    (level, status, environment)                              │
│       │                                                      │
│       ▼                                                      │
│  Promote to label via labels stage                           │
│  → Queryable dimension, creates streams per value            │
│                                                              │
│  Does the field have a large set of values?                  │
│    (user_id, request_id, trace_id)                           │
│       │                                                      │
│       ▼                                                      │
│  Extract with parser, keep in log line                       │
│  → Not a label, no stream explosion                          │
│  → Query with `| json | user_id = "..."`                    │
└─────────────────────────────────────────────────────────────┘

Systemd service lifecycle

┌─────────────────────────────────────────────────────────────┐
│  SYSTEMD SERVICE                                             │
│                                                              │
│  systemctl enable --now promtail                             │
│       │                                                      │
│       ▼                                                      │
│  ExecStart=/usr/local/bin/promtail --config.file=...        │
│       │                                                      │
│       ▼                                                      │
│  Promtail starts                                             │
│       │                                                      │
│       ├── Reads config                                       │
│       ├── Opens positions file                               │
│       ├── Starts HTTP server on 9080                         │
│       └── Begins tailing __path__ files                      │
│                                                              │
│  Restart=on-failure: restarts if Promtail crashes            │
└─────────────────────────────────────────────────────────────┘

Summary

Item	Value
Binary	`promtail`
Config flag	`--config.file`
Default HTTP port	9080
Positions file	Tracks read offset, must be persistent
Client endpoint	Loki push API (`/loki/api/v1/push`)
Scrape path	`__path__` label in `static_configs`
Pipeline stages	Parse, transform, label, timestamp
Systemd unit	`promtail.service`
Health check	`curl localhost:9080/ready`

Key takeaways:

Promtail is a courier, not a database. It reads log files, attaches labels, and pushes to Loki. It does not store or index anything. Its configuration is entirely about where to read from and where to send to .
The positions file must be persistent. The default /tmp/positions.yaml is lost on reboot, causing Promtail to re-read files from the beginning or lose track of its offset. Set positions.filename to a path under /var/lib/ .
Labels are the queryable dimensions. A scrape config without labels produces streams that are difficult to query in Loki. Attach job, host, and environment labels that match how you will filter logs. Do not promote high-cardinality fields like user_id to labels .
Pipeline stages parse structured content. The json, regex, logfmt, and labels stages extract fields from log lines and promote the useful ones to labels. This is how a Promtail setup turns opaque log text into queryable data .
Run Promtail as a systemd service. A foreground run is for testing. The service ensures Promtail starts at boot, restarts on failure, and logs its own output. Run it as a non-root user with read access to the log files .
Validate with --dry-run before starting. Promtail parses the configuration and exits without sending anything. A YAML syntax error or an invalid stage configuration is reported at this stage, not after the service fails.

Remember: Promtail’s setup is two decisions: where to send logs (clients) and which files to read (scrape_configs). The positions file is the third requirement that makes the setup reliable across restarts. The labels in the scrape config determine what you can query in Loki. The pipeline stages determine what structured data is available for labeling. Get the positions file right, attach low-cardinality labels, and test with --dry-run before enabling the service. A Promtail setup that starts without errors but sends nothing is almost always missing a clients block, an incorrect __path__, or a permissions problem on the log files.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!