LFCS 134 🐧 Monitoring — vmstat and iostat - KandZ

⏱️ Estimated read: 11 min | |

The previous chapter covered process monitoring with top and htop, which show the CPU and memory usage of individual processes. This chapter covers the system-wide monitoring tools that reveal the aggregate state of memory, swap, I/O, and CPU: vmstat and iostat. These tools answer a different question. top answers “which process is using the CPU?” vmstat answers “is the system swapping, and is the CPU idle because it is waiting for I/O?” iostat answers “which disk is saturated, and what is the average latency?” The LFCS exam tests the field interpretation: the r and b process counts, the si and so swap rates, the bi and bo block I/O rates, and the %util and await disk metrics. The critical trap is the first line of vmstat and iostat: it reports the average since boot, not the current state. The first line must be discarded, and the subsequent lines are the real measurements.

Key point: vmstat reports virtual memory statistics: processes, memory, swap, block I/O, system, and CPU. The first report is the average since boot. The delay argument sets the interval, and the count argument sets the number of reports. iostat reports CPU and per-device I/O statistics. The -x flag enables the extended device report with %util and await. The -d flag shows only the device report, and the -z flag omits devices with no activity. Both tools are part of the procps and sysstat packages respectively.

Why vmstat and iostat require a specific mental model

The first-line problem. The first line of vmstat output is the average since the last reboot. It is almost never useful. The second line and every subsequent line are the measurements for the interval specified by the delay argument. The vmstat 1 command produces the boot average first, then one report per second. The vmstat 1 5 command produces the boot average first, then five reports at one-second intervals. The -y flag (on newer versions) skips the first line entirely . The same applies to iostat: the first report is the average since boot, and the subsequent reports are the interval measurements .

The r and b problem. The r field is the number of processes in the run queue: running or waiting for CPU. The b field is the number of processes in uninterruptible sleep, typically waiting for I/O. A high r means the CPU is oversubscribed. A high b means the I/O subsystem is the bottleneck. The b field is the direct connection to the %wa (I/O wait) in top: both indicate that processes are stuck waiting for disk .

The swap problem. The si and so fields are the swap-in and swap-out rates in kilobytes per second. A non-zero si or so on a sustained basis means the system is under memory pressure. The operating system is moving pages to disk to free RAM, and the processes that need those pages must wait for the disk. The swpd field is the total swap used, but it is the rate (si/so) that indicates active swapping. A system with swpd non-zero but si/so zero is not currently swapping; it has swapped in the past .

The iostat %util and await problem. The %util field is the percentage of time the device had I/O requests in flight. A %util of 100% means the device is saturated. The await field is the average time in milliseconds for an I/O request to be served, including the time in queue and the time in service. A high await with a high %util means the device is the bottleneck. A high await with a low %util may indicate a slow device or a queue that is not the limiting factor .

The trade-off. vmstat provides a complete system overview in a single line, which is ideal for a quick check and for logging. iostat provides per-device detail, which is essential for identifying which disk is slow. The two tools are complementary. The vmstat shows the symptom (I/O wait, swapping), and the iostat shows the cause (the specific device and its latency).

a. The vmstat output and field interpretation

The vmstat command reports six categories: procs, memory, swap, io, system, and cpu. The default output has one line per report.

# One report per second, five reports
vmstat 1 5

The output has these columns:

Column	Category	Meaning
`r`	procs	Runnable processes (running or waiting for CPU)
`b`	procs	Uninterruptible sleep (waiting for I/O)
`swpd`	memory	Swap used (KB)
`free`	memory	Idle memory (KB)
`buff`	memory	Buffer cache (KB)
`cache`	memory	Page cache (KB)
`si`	swap	Swap-in rate (KB/s)
`so`	swap	Swap-out rate (KB/s)
`bi`	io	Blocks received from block device (blocks/s)
`bo`	io	Blocks sent to block device (blocks/s)
`in`	system	Interrupts per second
`cs`	system	Context switches per second
`us`	cpu	User time (%)
`sy`	cpu	System time (%)
`id`	cpu	Idle time (%)
`wa`	cpu	I/O wait time (%)
`st`	cpu	Steal time (%)

The r field is the number of processes waiting for CPU. If r is consistently higher than the number of CPU cores, the CPU is the bottleneck. The b field is the number of processes waiting for I/O. A high b with a high wa confirms the disk is the bottleneck .

The si and so fields are the swap activity. A non-zero value on a sustained basis means the system is short on memory. The free field is the idle memory, but the buff and cache are reclaimable, so the total available memory is free + buff + cache .

The us, sy, id, and wa fields are the CPU breakdown. The wa is the time the CPU spent idle while waiting for I/O. A high wa is not CPU usage; it is the disk being slow .

b. The iostat output and per-device metrics

The iostat command reports CPU and device statistics. The -x flag enables the extended device report.

# Extended device report, one-second intervals, five reports
iostat -x 1 5

The device report has these columns:

Column	Meaning
`Device`	Device name
`r/s`	Read requests per second
`w/s`	Write requests per second
`rkB/s`	Kilobytes read per second
`wkB/s`	Kilobytes written per second
`await`	Average I/O latency (ms)
`%util`	Device utilization (%)

The r/s and w/s are the request rates. The rkB/s and wkB/s are the throughput. The await is the average latency for all requests. The %util is the percentage of time the device was busy .

The -d flag shows only the device report, omitting the CPU report. The -z flag omits devices with no activity, which is useful on systems with many idle disks. The -m flag switches the units to megabytes per second.

# Device report only
iostat -d 1 5

# Omit idle devices
iostat -xz 1 5

# Megabyte units
iostat -xm 1 5

The iostat output also includes a CPU report by default. The %iowait field is the percentage of time the CPU was idle with outstanding I/O requests. The %idle is the percentage of time the CPU was idle with no outstanding I/O .

c. Combining vmstat and iostat for diagnosis

The two tools are used together. The vmstat shows the symptom, and the iostat shows the device.

A high wa in vmstat and a high %util on a specific device in iostat means the disk is the bottleneck. The await value indicates how slow the device is. A high b in vmstat (uninterruptible processes) confirms that processes are stuck waiting for the disk.

A high r in vmstat with a low wa and a low %util means the CPU is the bottleneck. The processes are runnable but waiting for CPU time.

A non-zero si and so in vmstat means the system is swapping. The iostat may show high I/O on the swap device, which is often the same disk as the root filesystem.

The diagnostic sequence is: run vmstat 1 to see the overall picture, then run iostat -xz 1 to identify the specific device if the I/O wait is high.

# Step 1: overall picture
vmstat 1 5

# Step 2: if wa is high, identify the device
iostat -xz 1 5

Complete Example Session

# ============================================
# PART 1: BASIC VMSTAT
# ============================================

vmstat 1 3
# First line: average since boot.
# Second and third lines: one-second intervals.

# ============================================
# PART 2: INTERPRET VMSTAT
# ============================================

# procs r=2, b=0 → CPU not oversubscribed
# swap si=0, so=0 → no swapping
# cpu us=5, sy=2, id=92, wa=1 → mostly idle

# ============================================
# PART 3: HIGH I/O WAIT
# ============================================

vmstat 1 3
# cpu wa=40 → high I/O wait
# procs b=5 → processes blocked on I/O

# ============================================
# PART 4: IDENTIFY THE DEVICE
# ============================================

iostat -xz 1 3
# Look for the device with high %util and high await.

# ============================================
# PART 5: INTERPRET IOSTAT
# ============================================

# Device sda: %util=98.5, await=45.2
# → sda is saturated, average latency 45 ms

# ============================================
# PART 6: SWAPPING
# ============================================

vmstat 1 3
# swap si=1024, so=2048
# → system is actively swapping, memory pressure

# ============================================
# PART 7: DEVICE REPORT ONLY
# ============================================

iostat -d 1 3
# Shows only the device report, no CPU.

# ============================================
# PART 8: OMIT IDLE DEVICES
# ============================================

iostat -xz 1 3
# Only devices with activity are shown.

# ============================================
# PART 9: MEGABYTE UNITS
# ============================================

iostat -xm 1 3
# Throughput in MB/s instead of KB/s.

The nine parts trace the monitoring: basic vmstat, field interpretation, high I/O wait, device identification, iostat interpretation, swapping detection, device-only report, idle device omission, and megabyte units.

Quick Reference

vmstat Fields

Field	Meaning
`r`	Runnable processes
`b`	Uninterruptible (I/O wait)
`swpd`	Swap used (KB)
`free`	Idle memory (KB)
`si`	Swap-in rate (KB/s)
`so`	Swap-out rate (KB/s)
`bi`	Blocks in (blocks/s)
`bo`	Blocks out (blocks/s)
`us`	User CPU (%)
`sy`	System CPU (%)
`id`	Idle CPU (%)
`wa`	I/O wait (%)

iostat Flags

Flag	Purpose
`-x`	Extended device report
`-d`	Device report only
`-z`	Omit idle devices
`-m`	Megabyte units
`-c`	CPU report only
`interval count`	Frequency and number of reports

iostat Key Fields

Field	Meaning
`r/s`	Read requests per second
`w/s`	Write requests per second
`await`	Average latency (ms)
`%util`	Device utilization (%)

Diagnostic Patterns

Pattern	Meaning
High `r`, low `wa`	CPU bottleneck
High `b`, high `wa`, high `%util`	Disk bottleneck
Non-zero `si`/`so`	Memory pressure / swapping
High `await`, high `%util`	Saturated device

Best Practices

✅ Do This:

vmstat 1 5
<!-- Discard the first line (boot average) -->        # ✅

iostat -xz 1 5
<!-- Use -x for extended, -z to omit idle -->        # ✅

vmstat 1
<!-- Run vmstat first, then iostat -->              # ✅

❌ Don’t Do This:

# Interpret the first line as current state
<!-- It is the average since boot -->                # ❌

# Use iostat without -x for bottleneck analysis
<!-- The basic report lacks await and %util -->      # ❌

# Ignore si/so when diagnosing performance
<!-- Swapping is the cause of I/O wait -->           # ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
First line misleading	Average since boot	Use `-y` or discard first line
High `wa` but disk looks fine	Swap on same disk	Check `si`/`so`
`iostat` shows no device	Device not in `/proc/diskstats`	Check kernel config
`vmstat` not found	`procps` not installed	Install `procps`
`iostat` not found	`sysstat` not installed	Install `sysstat`

Real-World Examples

1. Basic vmstat

vmstat 1 5

2. vmstat Swap Check

vmstat 1 5
# Look at si and so columns.

3. iostat Extended

iostat -x 1 5

4. iostat Omit Idle

iostat -xz 1 5

5. iostat Device Only

iostat -d 1 5

6. iostat Megabytes

iostat -xm 1 5

7. High I/O Wait

vmstat 1 3
# wa column high → run iostat.

8. Identify Saturated Disk

iostat -xz 1 3
# Look for %util near 100.

9. Check Swapping

vmstat 1 3
# si and so non-zero → memory pressure.

10. CPU Bottleneck

vmstat 1 3
# r high, wa low → CPU is the bottleneck.

Visual

vmstat Categories

┌──────────────────────────────────────────────┐
│  procs: r (runnable), b (blocked I/O)        │
├──────────────────────────────────────────────┤
│  memory: swpd, free, buff, cache             │
├──────────────────────────────────────────────┤
│  swap: si (in), so (out)                     │
├──────────────────────────────────────────────┤
│  io: bi (in), bo (out)                       │
├──────────────────────────────────────────────┤
│  system: in (interrupts), cs (context)       │
├──────────────────────────────────────────────┤
│  cpu: us, sy, id, wa                         │
└──────────────────────────────────────────────┘

Diagnostic Flow

┌──────────────────────────────────────────────┐
│  vmstat 1                                    │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────┐
│  wa high? → iostat -xz 1                     │
│  si/so non-zero? → memory pressure           │
│  r high, wa low? → CPU bottleneck            │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────┐
│  iostat: identify device with %util > 90     │
│  and await > 20ms                            │
└──────────────────────────────────────────────┘

Summary

Item	Value
Virtual memory stats	`vmstat`
Device I/O stats	`iostat`
First line	Average since boot
`vmstat` interval	`vmstat <delay> <count>`
`iostat` extended	`-x`
`iostat` omit idle	`-z`
Swap activity	`si`, `so`
I/O wait	`wa`
Device utilization	`%util`
Device latency	`await`

Key takeaways:

The first line of vmstat and iostat is the boot average. It is not the current state. The subsequent lines are the interval measurements. The -y flag skips the first line on newer versions.
The r and b fields distinguish CPU and I/O bottlenecks. A high r means the CPU is oversubscribed. A high b means processes are waiting for I/O. The wa field confirms the I/O wait.
The si and so fields indicate swapping. A non-zero rate on a sustained basis means the system is short on memory. The swpd field is the total swap used, not the current activity.
The iostat -x report is the per-device detail. The %util is the device saturation. The await is the average latency. A high %util with a high await identifies the bottleneck disk.
The two tools are complementary. vmstat shows the symptom. iostat shows the cause. Run vmstat first, then iostat if the I/O wait is high.

Remember: top shows which process is using the CPU. vmstat shows whether the CPU is idle because of I/O. iostat shows which disk is slow. The exam tests the field interpretation: r, b, si, so, wa, %util, await. The first line is the average; the rest are the real-time measurements.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!