Docker 30 🐳 Storage Drivers Deep Dive: Overlay2 Architecture, Inodes, and Performance Benchmarks - KandZ

The storage driver is the layer between a container’s filesystem and the host’s disk. It determines how image layers are stacked, how writes are handled, and how much space and inode overhead the container incurs. Docker supports several drivers, but overlay2 is the default on most Linux distributions and the one most LFCS and Docker exam candidates will encounter. Understanding its architecture, its inode behavior, and its performance characteristics is necessary for diagnosing storage problems and making informed driver choices.

Key point: overlay2 is a union filesystem that stacks read-only image layers (lowerdir) beneath a writable container layer (upperdir), presenting a unified view through a merged directory. Writes trigger a copy-up of the entire file from the lower layer to the upper layer. The driver is file-level copy-on-write, which means large-file modifications are slow but container startup is fast. Inode exhaustion and layer depth are the two most common failure modes.

Why overlay2 is the default

The union mount problem. A Docker image is a stack of layers. Each layer is a diff—a set of files that were added or changed in that step. When a container runs, it needs to see a single, coherent filesystem that merges all the image layers plus its own writable layer. OverlayFS provides this union mount. The overlay2 driver is Docker’s implementation of that union mount, and it is more efficient than the legacy overlay driver in inode utilization and layer count support .

The inode problem. The legacy overlay driver consumed excessive inodes, especially as the number of images grew. overlay2 takes advantage of kernel features added in Linux 4.0 to avoid this excessive consumption . It supports up to 128 lower layers natively, which allows multi-layer images to be stacked directly rather than through hard links .

The compatibility problem. overlay2 requires a backing filesystem that supports d_type=true. On XFS, this means the filesystem must be formatted with ftype=1. On ext4, d_type is enabled by default in modern versions. Without d_type, overlay2 degrades to a slower mode and may fail .

The performance problem. overlay2 provides the fastest container startup among the storage drivers, averaging around 50ms, compared to 80–90ms for btrfs and zfs . It also has good sequential write and random I/O performance. Its main weakness is copy-on-write of large files, which copies the entire file rather than just the changed blocks .

a. Overlay2 architecture: lowerdir, upperdir, merged

The overlay2 driver constructs a container’s filesystem from three directories and one file, all managed by Docker under /var/lib/docker/overlay2/. You should never manipulate these directly .

Component	Role
`lowerdir`	Read-only image layers, stacked in order
`upperdir`	Writable container layer
`merged`	Unified view, the container’s mount point
`workdir`	Internal to OverlayFS, for atomic operations
`lower-id`	File containing the ID of the top image layer

The lowerdir is the image. It is read-only. When a container is created, Docker combines the image’s top layer with a new writable directory for the container. The image layers become the lowerdir; the new directory becomes the upperdir .

The merged directory is the union of the two. From inside the container, / is the merged directory. The container sees the files from the image layers and its own writable layer as a single filesystem.

The workdir is required by OverlayFS for internal operations. It must be on the same filesystem as the upperdir .

When a file exists in both the image layer and the container layer, the container layer’s version wins. The image layer’s file is obscured. This is how modifications are represented: the modified file is copied to the upper layer, and the lower layer’s version is hidden .

b. Copy-up, whiteouts, and inode behavior

Copy-up. The first time a container writes to a file that exists in the image layer, overlay2 performs a copy-up. The entire file is copied from the lowerdir to the upperdir. The container’s write then modifies the copy in the upper layer. The lower layer’s original file is unchanged .

This is file-level copy-on-write. If the file is 1 GB and the container modifies one byte, the entire 1 GB is copied. This is the primary performance weakness of overlay2 compared to btrfs or zfs, which copy only the changed blocks .

Subsequent writes to the same file operate on the copy in the upperdir. The copy-up happens only once per file .

Whiteouts. When a container deletes a file that exists in the image layer, the file cannot be removed from the read-only lowerdir. Instead, overlay2 creates a whiteout file in the upperdir that obscures the lower layer’s file. The file is effectively deleted from the container’s view, but the image layer’s copy still exists .

Inode consumption. Inodes are consumed by the files and directories in the upperdir. A container that creates many small files—a build process, a cache, a log directory—consumes inodes in proportion to the number of files. The overlay2 driver itself has a fixed inode footprint per layer, and the legacy overlay driver’s excessive inode consumption is what overlay2 was designed to fix .

Inode exhaustion. The most common storage failure in Docker is inode exhaustion, not disk space exhaustion. The error message is No space left on device, but df -h shows free space. The fix is df -i, which shows inode usage. If the inode count is near 100%, the filesystem cannot create new files, even though there is space .

The mitigation is to monitor df -i /var/lib/docker and clean up unused images, containers, and volumes with docker system prune -a --volumes . For long-term prevention, the backing filesystem can be formatted with a higher inode density: mkfs.ext4 -i 131072 allocates one inode per 128 KB, which is more dense than the default .

c. Performance benchmarks and comparison

Container start time. overlay2 is the fastest. A typical container starts in around 50ms, compared to 80ms for btrfs and 90ms for zfs .

Sequential write. All three drivers perform well for sequential writes. overlay2 is good, btrfs is good, and zfs is good with compression enabled .

Random I/O. overlay2 and zfs perform well. btrfs is moderate. This matters for databases and applications that do many small reads and writes .

Copy-on-write of large files. This is where overlay2 is weakest. Because it copies the entire file, modifying a large file is slow. btrfs and zfs copy only changed blocks, making them significantly faster for this operation .

Storage efficiency. overlay2 is good. btrfs and zfs are excellent when compression is enabled, but compression is optional and must be configured .

The performance trade-off. overlay2 is the default for a reason: it provides the best balance of startup speed, I/O performance, and compatibility for most workloads. It is not the best for write-heavy workloads that modify large files, but those workloads are usually better served by volumes than by the container’s writable layer anyway.

Complete Example Session

# ============================================
# PART 1: CHECK THE CURRENT DRIVER
# ============================================
docker info | grep -E "(Storage Driver|Backing Filesystem|Supports d_type)"

# ============================================
# PART 2: VIEW OVERLAY MOUNTS
# ============================================
mount | grep overlay
# Shows the lowerdir, upperdir, and workdir for each running container

# ============================================
# PART 3: INSPECT A CONTAINER'S LAYER DIRECTORIES
# ============================================
ls -l /var/lib/docker/overlay2/
# Shows the layer directories
ls -l /var/lib/docker/overlay2/<container-id>/
# Shows lower, upper, merged, work

# ============================================
# PART 4: VERIFY THE MERGED VIEW
# ============================================
ls -l /var/lib/docker/overlay2/<container-id>/merged/
# Shows the container's filesystem

# ============================================
# PART 5: CHECK INODE USAGE
# ============================================
df -i /var/lib/docker
# If near 100%, inode exhaustion

# ============================================
# PART 6: CHECK DISK SPACE
# ============================================
df -h /var/lib/docker
# Space used by overlay2

# ============================================
# PART 7: CLEAN UP UNUSED RESOURCES
# ============================================
docker system prune -a --volumes
# Removes unused images, containers, volumes, networks

# ============================================
# PART 8: CHECK FILESYSTEM TYPE
# ============================================
xfs_info /var/lib/docker | grep ftype
# or
tune2fs -l /dev/sdX | grep "Filesystem features"

# ============================================
# PART 9: CONFIGURE OVERLAY2 IN DAEMON.JSON
# ============================================
cat > /etc/docker/daemon.json << EOF
{
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.mountopt=nodev,metacopy=on"
  ]
}
EOF
systemctl restart docker

# ============================================
# PART 10: BENCHMARK I/O WITH DD
# ============================================
docker run -it --rm ubuntu bash -c "dd if=/dev/zero of=test.img bs=1G count=1 oflag=dsync"

The ten parts covered checking the driver, viewing overlay mounts, inspecting layer directories, verifying the merged view, checking inode usage, checking disk space, cleaning up, checking the filesystem type, configuring overlay2, and benchmarking I/O.

Quick Reference

Overlay2 Components

Component	Path	Role
`lowerdir`	`/var/lib/docker/overlay2/<id>/diff`	Read-only image layers
`upperdir`	`/var/lib/docker/overlay2/<id>/diff`	Writable container layer
`merged`	`/var/lib/docker/overlay2/<id>/merged`	Unified view
`workdir`	`/var/lib/docker/overlay2/<id>/work`	Internal
`lower-id`	`/var/lib/docker/overlay2/<id>/lower-id`	Top image layer ID

Overlay2 vs Other Drivers

Metric	overlay2	btrfs	zfs
Container start	~50ms	~80ms	~90ms
Sequential write	Good	Good	Good
Random I/O	Good	Moderate	Good
CoW (large files)	Slow (file-level)	Fast (block-level)	Fast (block-level)
Storage efficiency	Good	Excellent (compression)	Excellent (compression)
Best for	General purpose	Snapshot-heavy	Data integrity

Requirements

Requirement	Detail
Kernel	4.0+ (or RHEL 3.10.0-514+)
ext4	`d_type` enabled (default in modern)
XFS	`ftype=1` (format with `-n ftype=1`)
Backing FS	ext4 or XFS (not btrfs/zfs)

Inode Management

Task	Command
Check inode usage	`df -i /var/lib/docker`
Clean unused	`docker system prune -a --volumes`
High-density format	`mkfs.ext4 -i 131072 /dev/sdX`
Monitor	`df -i` above 90% needs action

Best Practices

✅ Do This:

# Verify the backing filesystem supports d_type
docker info | grep "Supports d_type"                         # ✅

# Monitor inode usage, not just disk space
df -i /var/lib/docker                                        # ✅

# Clean up unused resources regularly
docker system prune -a --volumes                             # ✅

# Use volumes for write-heavy data
docker run -v mydata:/var/lib/postgresql/data postgres        # ✅

# Reduce image layers in the Dockerfile
# Use multi-stage builds, combine RUN commands                 # ✅

# Enable metacopy for faster copy-up (kernel 4.19+)
"storage-opts": ["overlay2.mountopt=metacopy=on"]             # ✅

# Use XFS with ftype=1 for the backing filesystem
mkfs.xfs -n ftype=1 /dev/sdX                                 # ✅

❌ Don’t Do This:

# Don't manually manipulate overlay2 directories
rm -rf /var/lib/docker/overlay2/*                            # ❌

# Don't assume "No space left on device" means disk full
# Check inodes first                                            # ❌

# Don't use overlay2 on btrfs or zfs
# Unsupported and will fail                                     # ❌

# Don't ignore d_type requirements
# overlay2 degrades without it                                  # ❌

# Don't store databases in the container writable layer
# Use volumes for persistence                                   # ❌

# Don't let layer count grow without reason
# Deep stacks slow down file lookups                            # ❌

# Don't prune without checking what will be removed
docker system prune -a  # removes all unused images            # ⚠️

Common Pitfalls

Pitfall	Why It Happens	Fix
`No space left on device`	Inode exhaustion	`df -i`, `docker system prune`
Slow writes to large files	File-level copy-up	Use a volume for write-heavy data
overlay2 not supported	Backing FS is btrfs/zfs	Use ext4 or XFS
`d_type` not supported	XFS without `ftype=1`	Reformat with `-n ftype=1`
Container start slow	Deep layer stack	Reduce image layers
Disk space grows	upperdir accumulates	Monitor, clean, use volumes
Mount fails	workdir on different FS	Ensure same filesystem

Real-World Examples

1. Check Driver

docker info | grep "Storage Driver"
# Storage Driver: overlay2

2. View Overlay Mounts

mount | grep overlay

3. Check Inodes

df -i /var/lib/docker
# Filesystem      Inodes  IUsed  IFree IUse% Mounted on
# /dev/sda1      6553600  98120 6455480    2% /

4. Clean Up

docker system prune -a --volumes

5. Configure overlay2

{
  "storage-driver": "overlay2",
  "storage-opts": ["overlay2.mountopt=nodev,metacopy=on"]
}

6. Check XFS ftype

xfs_info /var/lib/docker | grep ftype
# ftype=1

7. Format XFS with ftype

mkfs.xfs -n ftype=1 /dev/sdb

8. High-Density Inodes

mkfs.ext4 -i 131072 /dev/sdb

9. Benchmark Write

docker run -it --rm ubuntu bash -c "dd if=/dev/zero of=test bs=1G count=1 oflag=dsync"

10. Check Upperdir Size

du -sh /var/lib/docker/overlay2/*/diff

Visual

Overlay2 Layer Stack

┌─────────────────────────────────────────────────────────────┐
│  CONTAINER VIEW (merged)                                    │
│  ┌─────────────────────────────────────────────────────┐    │
│  │  /usr/bin/app      (from image)                     │    │
│  │  /etc/config.yml   (from image)                     │    │
│  │  /data/cache.db    (modified — in upperdir)         │    │
│  │  /tmp/scratch      (new file — in upperdir)         │    │
│  └─────────────────────────────────────────────────────┘    │
│                                                             │
│  ┌─────────────────────────────────────────────────────┐    │
│  │  UPPERDIR (writable container layer)                │    │
│  │  /data/cache.db    (copy-up from lower)             │    │
│  │  /tmp/scratch      (new)                            │    │
│  └─────────────────────────────────────────────────────┘    │
│  ┌─────────────────────────────────────────────────────┐    │
│  │  LOWERDIR (read-only image layers)                  │    │
│  │  Layer 3: /etc/config.yml                           │    │
│  │  Layer 2: /usr/bin/app                              │    │
│  │  Layer 1: /bin, /lib, etc.                          │    │
│  │  /data/cache.db    (obscured by upperdir)           │    │
│  └─────────────────────────────────────────────────────┘    │
│                                                             │
│  Writes go to upperdir. Lowerdir is never modified.         │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Copy-Up Operation

┌─────────────────────────────────────────────────────────────┐
│  BEFORE WRITE                                               │
│                                                             │
│  lowerdir: /data/large-file (1 GB)                          │
│  upperdir: (empty)                                          │
│                                                             │
│  Container writes 1 byte to /data/large-file                │
│    │                                                        │
│    ▼                                                        │
│  COPY-UP: entire 1 GB file copied to upperdir               │
│                                                             │
│  ┌─────────────────────────────────────────────────────┐    │
│  │  upperdir: /data/large-file (1 GB copy)             │    │
│  │  lowerdir: /data/large-file (original, unchanged)   │    │
│  └─────────────────────────────────────────────────────┘    │
│                                                             │
│  Write modifies the copy. Lowerdir is untouched.            │
│  Slow for large files. Fast for small files.                │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Inode vs Disk Space Exhaustion

┌─────────────────────────────────────────────────────────────┐
│  DISK SPACE EXHAUSTION                                      │
│                                                             │
│  df -h /var/lib/docker                                      │
│  Filesystem  Size  Used Avail Use%                          │
│  /dev/sda1   100G   99G  1.0G  99%                          │
│                                                             │
│  Fix: delete files, prune images                            │
│                                                             │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  INODE EXHAUSTION                                           │
│                                                             │
│  df -i /var/lib/docker                                      │
│  Filesystem  Inodes IUsed IFree IUse%                       │
│  /dev/sda1   6.5M   6.5M  0     100%                        │
│                                                             │
│  df -h shows space available, but files cannot be created.  │
│                                                             │
│  Fix: docker system prune, reformat with higher inode density│
│                                                             │
└─────────────────────────────────────────────────────────────┘

Driver Performance Comparison

┌─────────────────────────────────────────────────────────────┐
│  CONTAINER START TIME                                       │
│                                                             │
│  overlay2  ████████ 50ms                                     │
│  btrfs     ████████████████ 80ms                             │
│  zfs       ██████████████████████ 90ms                       │
│                                                             │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  LARGE FILE MODIFICATION (CoW)                              │
│                                                             │
│  overlay2  ████████████████████████████████████ Slow         │
│  btrfs     ████████ Fast                                     │
│  zfs       ████████ Fast                                     │
│                                                             │
│  overlay2 copies the entire file.                           │
│  btrfs and zfs copy only changed blocks.                    │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Summary

Item	Value
Default driver	`overlay2`
Backing filesystems	ext4, XFS with `ftype=1`
Kernel requirement	4.0+
Layers supported	128
Copy-on-write	File-level
Container start	~50ms
Weakness	Large-file modifications
Strength	Startup, general I/O
Inode fix	`df -i`, `docker system prune`
Config file	`/etc/docker/daemon.json`

Key takeaways:

overlay2 is the default storage driver on most Linux distributions. It uses OverlayFS to stack read-only image layers beneath a writable container layer, presenting a unified view through the merged directory. The image layers are the lowerdir; the container layer is the upperdir .
Writes trigger a file-level copy-up. The first time a container writes to a file from the image, the entire file is copied to the container layer. This is fast for small files and slow for large files. Subsequent writes operate on the copy .
Inode exhaustion is the most common storage failure. The error is No space left on device, but df -h shows free space. The fix is df -i and docker system prune .
overlay2 requires d_type support. On XFS, the filesystem must be formatted with ftype=1. Without it, overlay2 degrades to a slower mode .
Container startup is fastest with overlay2. At around 50ms, it beats btrfs and zfs. Its weakness is large-file copy-up, which is significantly slower than block-level drivers like btrfs and zfs .
The upperdir is the container’s writable layer. It accumulates files as the container runs. For write-heavy workloads, use volumes instead of the container’s writable layer. Volumes bypass the storage driver and are faster and persistent .
Reduce image layers to reduce overhead. Deep layer stacks slow down file lookups. Use multi-stage builds and combine RUN commands in the Dockerfile .

Remember: overlay2 is the workhorse of Docker storage. It provides the best balance of performance and compatibility for most workloads, and it is the driver you will encounter in the vast majority of installations. Its architecture is a union mount of read-only image layers and a writable container layer. Its copy-up behavior is file-level, which is fast for startup and small files but slow for large-file modifications. Its failure mode is inode exhaustion, which masquerades as disk space exhaustion. Monitor df -i, clean up unused resources, use volumes for write-heavy data, and reduce image layers. The driver is not something you configure once and forget—it is something you monitor and manage.

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!