Docker 92 🐳 Storage Performance Tuning: I/O Bottlenecks, Direct I/O, and Benchmark Tooling inside Containers
Container storage performance is not the same as host storage performance. The storage driver, the writable layer, and the volume configuration all sit between the application and the disk, and each can become the bottleneck that limits throughput or inflates latency. The diagnostic challenge is that the bottleneck is often not the disk itself: a container writing to its writable layer triggers copy_up operations, a volume on a slow backing store delivers slow I/O regardless of the driver, and a misconfigured backing array can queue requests while the individual NVMe drives sit idle . This chapter covers the storage driver’s role in I/O performance, the direct I/O measurement technique that bypasses page cache to reveal true device behavior, and the benchmark tooling—fio, dd, ioping—that produces numbers you can act on.
Key point: The storage driver is not the place for write-heavy data. Docker’s own documentation recommends volumes for write-intensive workloads because writes to the container’s writable layer consume space in the thin top layer and can incur copy_up overhead . A benchmark that measures the writable layer is measuring the wrong thing if the production workload should be using a volume.
Why container storage performance requires its own tuning
The writable layer problem. Every write to a container’s filesystem goes to a thin writable layer on top of the read-only image layers. The first write to a file that exists in a lower layer triggers a copy_up: the entire file is copied into the writable layer before the modification is applied . For a database or a log-heavy application, this is the wrong place to write. The copy_up overhead is paid once per file, but the space consumption and the storage driver’s copy-on-write mechanics continue to affect every subsequent write.
The volume indirection problem. A Docker volume bypasses the storage driver’s copy-on-write path. I/O to a volume goes directly to the backing filesystem, which means the performance is closer to the raw device . The trade-off is that volume performance depends entirely on what the volume is backed by. A named volume on a slow NFS mount is slow; a bind mount to a local NVMe drive is fast. The storage driver is no longer the variable, but the backing store is.
The page cache problem. A benchmark without direct I/O measures the page cache, not the disk. The first read of a file is served from disk; subsequent reads are served from memory. A benchmark that does not drop caches between runs, or does not use O_DIRECT/--direct=1, reports throughput that the application will only see on the first pass . This is the difference between a number that reflects the device and a number that reflects the kernel’s caching behavior.
The backing array problem. A container’s I/O can be queued at a layer below the filesystem. A software RAID5 array, for example, has a single-threaded stripe dispatch path that caps sequential write throughput around 2.5 GB/s regardless of how fast the underlying NVMe drives are . The container sees high latency and deep queues while the individual drives report low utilization. The bottleneck is the RAID layer, not the disk, the filesystem, or Docker .
a. Storage Driver Choice and Volume Placement
Docker’s default storage driver on modern Linux distributions is overlay2, which is recommended for most use cases . It provides good overall performance and broad compatibility. The vfs driver is not a union filesystem and performs a deep copy of every layer, which makes it slow and space-inefficient; it is intended for testing and debugging . The btrfs and zfs drivers offer snapshots and other advanced features but require the backing filesystem to be configured correctly, and their copy-on-write behavior can degrade random write performance for certain workloads .
The more consequential decision is where the data lives. For write-heavy workloads, the correct pattern is a volume or bind mount, not the container’s writable layer. A bind mount to a local directory on a fast disk gives the container direct access to that disk’s performance. A named volume gives Docker control over placement, but the performance is only as good as the volume’s backing store.
# Check the current storage driver
docker info | grep "Storage Driver"
# Run a write-heavy container with a bind mount to local storage
docker run -v /mnt/nvme/app-data:/data my-app
# Run with a named volume (backing store determined by Docker)
docker run -v app-data:/data my-appThe benchmark should match the deployment. If production uses a bind mount, the benchmark should use a bind mount. If production uses a named volume, the benchmark should use a named volume on the same backing store. Measuring the writable layer when production uses a volume produces a number that does not correspond to any real workload .
b. Direct I/O and Benchmark Tooling
fio is the standard tool for storage benchmarking because it separates the measurement from the cache. The --direct=1 flag enables direct I/O, which bypasses the page cache and forces the I/O to reach the device. Without it, the numbers reflect memory bandwidth as much as disk performance . The --ioengine flag selects the I/O mechanism: libaio for asynchronous I/O on Linux, posixaio for portability, and sync for a simple synchronous path.
# Random 4K read, direct I/O, 32 outstanding requests
fio --name=randread --ioengine=libaio --direct=1 --bs=4k \
--rw=randread --iodepth=32 --size=2G --runtime=60 \
--time_based --group_reporting --filename=/data/fio-test
# Random 4K mixed read/write (70% read)
fio --name=mixed --ioengine=libaio --direct=1 --bs=4k \
--rw=randrw --rwmixread=70 --iodepth=32 --size=2G \
--runtime=60 --time_based --filename=/data/fio-testdd is the simpler alternative for a quick throughput check. The oflag=dsync flag forces synchronous writes, which provides a rough measure of write latency without the page cache absorbing the writes. It is not a substitute for fio, but it is available in almost every container image and requires no installation .
# Basic write throughput with synchronous writes
dd if=/dev/zero of=/data/test.img bs=1G count=1 oflag=dsync
# Read throughput (drop caches first on the host)
dd if=/data/test.img of=/dev/null bs=1G count=1ioping measures latency rather than throughput. It sends individual I/O requests and reports the response time, which is useful for diagnosing whether a volume feels slow because of low bandwidth or high per-request latency.
# Measure I/O latency (20 requests, 4K each)
ioping -c 20 -s 4K /dataThe benchmark must run inside the container, not just on the host. The host benchmark establishes the baseline; the container benchmark reveals the overhead added by the storage driver, the volume mount, and any cgroup I/O limits. Docker volumes on overlay2 should match the host baseline within 5–10 percent for direct I/O; a larger gap indicates a configuration problem or a slow backing store .
c. Diagnosing Bottlenecks Below the Container
When the container benchmark is slow, the next step is determining which layer is responsible. The container’s I/O passes through the storage driver, the filesystem, possibly a volume plugin, and the backing block device. iostat on the host shows whether the disk is actually busy.
# Host-level I/O statistics: utilization, latency, queue depth
iostat -xm 1The interpretation depends on where the latency appears. If the container sees high latency but the host’s disk shows low %util and low await, the bottleneck is above the disk: the storage driver, the filesystem, or a volume plugin. If the disk shows high %util and high await, the disk itself is saturated. If the disk shows low utilization but the queue is deep, the bottleneck is below the filesystem—a RAID layer, a network volume, or a hypervisor .
For a RAID array, the stripe_cache_size parameter can be the limiting factor for write throughput. The default value of 256 is often too small for write-heavy container workloads, and increasing it reduces queue depth and latency . This is a host-level tuning step, not a container-level one, but it affects every container using that array.
# Check current stripe cache size (RAID5 example)
cat /sys/block/md0/md/stripe_cache_size
# Increase for write-heavy workloads (host-level)
echo 8192 > /sys/block/md0/md/stripe_cache_sizeThe container’s own I/O limits can also be the bottleneck. Docker’s --device-read-bps and --device-write-bps flags impose throttling. If the benchmark is slow and the host disk is idle, check whether the container was started with limits that are lower than expected.
Complete Example Session
# ============================================
# PART 1: CHECK THE STORAGE DRIVER
# ============================================
docker info | grep "Storage Driver"
# Output: overlay2 (typical)
# ============================================
# PART 2: CREATE A BENCHMARK VOLUME
# ============================================
docker volume create bench-vol
# Or use a bind mount to local NVMe storage:
# docker run -v /mnt/nvme:/data ...
# ============================================
# PART 3: RUN A DIRECT I/O BENCHMARK INSIDE THE CONTAINER
# ============================================
docker run --rm -v bench-vol:/data ubuntu:24.04 bash -c \
'apt update && apt install -y -qq fio && \
fio --name=vol-4k --ioengine=libaio --direct=1 --bs=4k \
--rw=randread --iodepth=32 --size=2G --runtime=30 \
--time_based --group_reporting --filename=/data/fio-test'
# ============================================
# PART 4: ESTABLISH THE HOST BASELINE
# ============================================
fio --name=host-4k --ioengine=libaio --direct=1 --bs=4k \
--rw=randread --iodepth=32 --size=2G --runtime=30 \
--time_based --group_reporting --filename=/tmp/host-fio-test
# ============================================
# PART 5: COMPARE CONTAINER AND HOST NUMBERS
# ============================================
# Docker volume on overlay2 should be within 5-10% of host
# A larger gap indicates a configuration issue
# ============================================
# PART 6: MEASURE LATENCY WITH IOPING
# ============================================
docker run --rm -v bench-vol:/data ubuntu:24.04 bash -c \
'apt install -y -qq ioping && ioping -c 20 -s 4K /data'
# ============================================
# PART 7: QUICK CHECK WITH DD (NO INSTALL NEEDED)
# ============================================
docker run --rm -v bench-vol:/data ubuntu:24.04 \
dd if=/dev/zero of=/data/test.img bs=1G count=1 oflag=dsync
# ============================================
# PART 8: HOST-LEVEL DIAGNOSTICS
# ============================================
# While the container benchmark runs, check the host
iostat -xm 1
# Look for: high %util and high await (disk saturated)
# Or: low %util and deep queue (bottleneck below filesystem)
# ============================================
# PART 9: CHECK FOR RAID BOTTLENECKS (IF APPLICABLE)
# ============================================
# If the host uses md RAID5 and write latency is high:
cat /sys/block/md0/md/stripe_cache_size
# Increase if write-heavy: echo 8192 > /sys/block/md0/md/stripe_cache_size
# ============================================
# PART 10: CLEAN UP
# ============================================
docker volume rm bench-volThe ten parts trace the benchmarking workflow: identify the driver, create the volume, run direct I/O, compare with the host baseline, measure latency, diagnose at the host level, and check for bottlenecks below the filesystem. The critical comparison is Part 5: if the container’s volume is significantly slower than the host baseline, the problem is in the Docker storage configuration, not the disk.
Quick Reference
Storage Drivers
| Driver | Characteristics | Recommendation |
|---|---|---|
overlay2 | Union filesystem, good performance | Default for most workloads |
btrfs | Snapshots, CoW overhead | Only if backing FS is btrfs |
zfs | Snapshots, requires setup | Only if backing FS is zfs |
vfs | Deep copy, no CoW, slow | Testing only |
Benchmark Tool Flags
| Tool | Flag | Purpose |
|---|---|---|
fio | --direct=1 | Bypass page cache |
fio | --ioengine=libaio | Async I/O on Linux |
fio | --iodepth=32 | Outstanding requests |
dd | oflag=dsync | Synchronous writes |
ioping | -c 20 | Request count for latency |
Diagnostic Signals
| Observation | Likely Bottleneck |
|---|---|
| Container slow, host disk idle | Storage driver, volume plugin, or cgroup limit |
Host disk high %util, high await | Disk saturated |
Host disk low %util, deep queue | RAID layer, network volume, or hypervisor |
| Writes slow, reads fast | Writable layer or RAID5 stripe dispatch |
Best Practices
✅ Do This:
# Use volumes for write-heavy workloads
docker run -v app-data:/data my-db # ✅# Use direct I/O for benchmarks
fio --direct=1 ... # ✅# Compare container and host benchmarks
# A >10% gap indicates Docker-level overhead # ✅# Check the host disk while the container benchmark runs
iostat -xm 1 # ✅❌ Don’t Do This:
# Benchmark the writable layer when production uses a volume
docker run my-app # no volume, writes to thin layer # ❌# Benchmark without direct I/O and trust the numbers
fio --direct=0 ... # measures page cache, not disk # ❌# Assume the disk is the bottleneck without checking iostat
# The bottleneck may be above or below the disk # ❌# Use the vfs driver for production
# Deep copy on every layer, poor performance # ❌Common Pitfalls
| Pitfall | Why It Happens | Fix |
|---|---|---|
| Benchmark shows high throughput but app is slow | Page cache absorbed writes | Use --direct=1 or oflag=dsync |
| Volume slower than expected | Named volume on slow backing store | Use bind mount to local disk |
| Container slow, host disk idle | Storage driver or cgroup limit | Check driver, check --device-write-bps |
| Write latency high on RAID5 | Stripe cache too small | Increase stripe_cache_size |
| First write to file is slow | copy_up from lower image layer | Use volume for data that changes |
Real-World Examples
1. Check Storage Driver
docker info | grep "Storage Driver"2. Direct I/O Random Read
fio --name=read --ioengine=libaio --direct=1 --bs=4k --rw=randread --iodepth=32 --size=2G --runtime=30 --filename=/data/fio-test3. Direct I/O Random Write
fio --name=write --ioengine=libaio --direct=1 --bs=4k --rw=randwrite --iodepth=32 --size=2G --runtime=30 --filename=/data/fio-test4. Mixed Workload
fio --name=mixed --ioengine=libaio --direct=1 --bs=4k --rw=randrw --rwmixread=70 --iodepth=32 --size=2G --runtime=60 --filename=/data/fio-test5. Quick Write Check
dd if=/dev/zero of=/data/test.img bs=1G count=1 oflag=dsync6. Latency Measurement
ioping -c 20 -s 4K /data7. Host Disk Utilization
iostat -xm 18. RAID Stripe Cache
cat /sys/block/md0/md/stripe_cache_size9. Bind Mount for Performance
docker run -v /mnt/nvme/app:/data my-app10. Volume Baseline Comparison
# Run fio on host, then on container volume, compareVisual
Container I/O Path
┌──────────────────────────────────────────────────────────────┐
│ APPLICATION │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ Container FS │ Writable layer (thin, copy_up) │
│ │ (overlay2) │ or volume (direct) │
│ └────────┬────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ Storage Driver │ overlay2 union mount │
│ └────────┬────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ Backing FS │ ext4, xfs, or volume plugin │
│ └────────┬────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ Block Device │ NVMe, RAID, or network volume │
│ └────────┬────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ Physical Disk │ The actual hardware │
│ └─────────────────┘ │
│ │
│ Writable layer adds copy_up overhead. │
│ Volumes bypass the storage driver. │
│ Direct I/O bypasses the page cache. │
│ │
└──────────────────────────────────────────────────────────────┘Direct I/O vs. Buffered I/O
┌──────────────────────────────────────────────────────────────┐
│ BUFFERED I/O DIRECT I/O │
│ │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Application │ │ Application │ │
│ └────────┬────────┘ └────────┬────────┘ │
│ ▼ │ │
│ ┌─────────────────┐ │ │
│ │ Page Cache │ Absorbs writes │ │
│ │ (kernel memory) │ Serves reads │ │
│ └────────┬────────┘ │ │
│ ▼ ▼ │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Block Device │ │ Block Device │ │
│ └─────────────────┘ └─────────────────┘ │
│ │
│ Benchmark measures cache. Benchmark measures disk. │
│ Fast but misleading. Slower but accurate. │
│ │
│ Use --direct=1 in fio to bypass the cache. │
│ │
└──────────────────────────────────────────────────────────────┘Bottleneck Diagnosis
┌──────────────────────────────────────────────────────────────┐
│ CONTAINER BENCHMARK IS SLOW │
│ │
│ ┌─────────────────┐ │
│ │ Check host │────► Disk %util high? │
│ │ iostat │ YES → Disk is saturated │
│ └────────┬────────┘ │
│ │ NO │
│ ▼ │
│ ┌─────────────────┐ │
│ │ Disk %util low? │────► YES → Bottleneck above disk │
│ │ │ Storage driver, volume plugin, │
│ │ │ cgroup limit │
│ └────────┬────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ Queue deep, │────► YES → Bottleneck below filesystem │
│ │ disk idle? │ RAID stripe cache, network volume │
│ └─────────────────┘ hypervisor │
│ │
│ Each layer has a distinct signature in iostat. │
│ │
└──────────────────────────────────────────────────────────────┘Volume vs. Writable Layer
┌──────────────────────────────────────────────────────────────┐
│ WRITABLE LAYER VOLUME │
│ │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Thin top layer │ │ Direct access │ │
│ │ copy_up on │ │ to backing FS │ │
│ │ first write │ │ │ │
│ └────────┬────────┘ └────────┬────────┘ │
│ ▼ ▼ │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Storage driver │ │ No driver │ │
│ │ overhead │ │ overhead │ │
│ └────────┬────────┘ └────────┬────────┘ │
│ ▼ ▼ │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Disk │ │ Disk │ │
│ └─────────────────┘ └─────────────────┘ │
│ │
│ Write-heavy workloads should use volumes. │
│ Docker docs recommend this explicitly . │
│ │
└──────────────────────────────────────────────────────────────┘Summary
| Item | Value |
|---|---|
| Default driver | overlay2 |
| Write-heavy workloads | Use volumes, not writable layer |
| Direct I/O flag | --direct=1 in fio |
| Quick write test | dd oflag=dsync |
| Latency tool | ioping |
| Host diagnostics | iostat -xm 1 |
| Volume vs host gap | Should be within 5–10% |
| RAID5 write limit | Stripe cache and single-thread dispatch |
Key takeaways:
- The storage driver is not the right place for write-heavy data. Docker’s own documentation recommends volumes for write-intensive workloads because writes to the writable layer consume space in the thin top layer and can trigger
copy_upoperations . Benchmark the volume, not the writable layer. - Direct I/O separates the device from the cache. A benchmark without
--direct=1measures page cache, not disk. The numbers may look good but do not reflect what the application will experience when the cache is cold . - Compare container and host. A Docker volume on
overlay2should be within 5–10 percent of the host baseline for direct I/O. A larger gap indicates a configuration issue—the storage driver, a volume plugin, or a cgroup limit—not a slow disk . - The bottleneck is often below the filesystem. A RAID5 array with a small stripe cache queues requests while the NVMe drives sit idle.
iostaton the host reveals the signature: low disk utilization with deep queues means the bottleneck is the RAID layer, not the disk . - Use the tool that matches the question.
fiofor throughput and IOPS,ddfor a quick write check,iopingfor latency,iostatfor host-level diagnosis. Each answers a different question about where the time is going.
Remember: Container storage performance is a layered problem. The application writes to a filesystem, the filesystem sits on a storage driver or a volume, the volume sits on a backing store, and the backing store sits on hardware that may itself be a RAID array or a network volume. A benchmark that measures only one layer produces a number that does not explain the others. The correct approach is to measure inside the container, compare with the host, and use host-level tools to identify which layer is responsible when the numbers diverge. The storage driver is the variable that Docker controls; the volume backing store and the hardware are the variables that the infrastructure controls. Tuning starts with knowing which one is limiting the workload.
Stop using slow, ad-bloated tool sites! 🤮
🔎 Search “KandZ Tools” on Google to use many professional utilities for free.
KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)
⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free
🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!