LFCS 95 🐧 RAID with mdadm — Managing and Recovering - KandZ

⏱️ Estimated read: 9 min | |

An array that is created and forgotten is a liability. Drives fail, sectors go bad, and arrays silently degrade into a state where a second failure means total data loss. Managing an mdadm array is not about reacting to catastrophe—it is about routine monitoring, deliberate failure testing, and a recovery procedure that you have practiced before the disk actually dies. The LFCS exam tests the sequence: identify the failed disk, mark it as failed, remove it, add a replacement, and monitor the rebuild.

Key point: The recovery procedure is --fail, --remove, --add. The --fail step is not optional when a disk is still responding but corrupted—it forces the array to stop trusting the device before you remove it.

Why management and recovery exist

The silent degradation problem. When a disk fails in a RAID 1 or RAID 5 array, the array continues to operate. This is the point of RAID. But the array is now degraded: performance is reduced, and a second failure is unrecoverable. If no one is monitoring /proc/mdstat, the degraded state can persist for months until the second disk fails and the data is gone. Monitoring is the first line of defense .

The fail-first rule. A disk that is returning I/O errors but not completely dead will continue to be used by the array unless explicitly marked as failed. The kernel’s md driver may not immediately recognize that the disk is unreliable. mdadm --fail forces the array to stop using the device, allowing a replacement to be added without waiting for the disk to fail completely .

The rebuild risk. When a replacement disk is added, the array begins rebuilding. For RAID 1, this is a simple copy from the surviving mirror. For RAID 5, the array must read all surviving data and parity blocks to reconstruct the lost information. This places heavy I/O load on the remaining disks, increasing the chance of a second failure during the rebuild window. Monitoring the rebuild and adjusting speed limits are part of recovery .

The scrub problem. Even without a disk failure, RAID arrays can develop silent data corruption—mismatches between mirror copies or parity inconsistencies. A periodic scrub reads all data and verifies consistency. The kernel exposes this through /sys/block/mdX/md/sync_action, where writing check initiates a read-only verification and repair attempts to correct mismatches .

The configuration persistence problem. An array that is not listed in /etc/mdadm.conf may not assemble automatically on boot. After any change to the array—adding a disk, replacing a member—the configuration should be regenerated with mdadm --detail --scan .

a. Monitoring array health

The first command in any management workflow is cat /proc/mdstat. It shows the array device, the RAID level, the member disks with their slot numbers, and the state of each disk. The notation [UU] means both disks are up; [U_] means the second disk has failed. During a rebuild, the progress is shown with a percentage and estimated time .

# Quick status
cat /proc/mdstat

# Detailed view
sudo mdadm --detail /dev/md0

mdadm --detail provides the array UUID, chunk size, RAID level, and the state of each member disk. It is the command to run when /proc/mdstat shows a problem, because it identifies which specific device has failed .

For automated monitoring, mdadm --monitor can run in daemon mode and send alerts when events occur. The --oneshot flag checks the array once and exits, which is suitable for cron jobs .

# Check once and report degraded arrays
sudo mdadm --monitor --scan --oneshot

b. Replacing a failed disk

The recovery sequence has four steps. First, mark the disk as failed if the kernel has not already done so. Second, remove the failed disk from the array. Third, physically replace the disk (or prepare the new device). Fourth, add the new disk to the array and monitor the rebuild.

# Mark as failed (required if the disk still responds)
sudo mdadm /dev/md0 --fail /dev/sdc

# Remove from array
sudo mdadm /dev/md0 --remove /dev/sdc

# Add replacement (after physical replacement or if using a spare)
sudo mdadm /dev/md0 --add /dev/sdd

The --add command triggers the rebuild automatically. The array continues to serve data during the rebuild, but performance is reduced. The progress can be monitored with cat /proc/mdstat or watch -n 5 cat /proc/mdstat .

The rebuild speed can be adjusted through the kernel parameters /proc/sys/dev/raid/speed_limit_min and /proc/sys/dev/raid/speed_limit_max. Increasing the minimum speed accelerates the rebuild at the cost of foreground I/O performance .

c. Scrubbing and verifying integrity

A scrub reads every block in the array and verifies that mirrors match or that parity is consistent. It does not require unmounting the array. The kernel exposes this through sysfs.

# Start a read-only check
echo check | sudo tee /sys/block/md0/md/sync_action

# Monitor progress
cat /proc/mdstat

# After completion, check for mismatches
cat /sys/block/md0/md/mismatch_cnt

# If mismatches are found, repair them
echo repair | sudo tee /sys/block/md0/md/sync_action

A mismatch_cnt of zero means the array is consistent. A non-zero value indicates that mirror copies diverged or parity did not match the data. The repair action attempts to correct these inconsistencies . Scrubbing is not a substitute for backups—it detects and fixes consistency errors, not user mistakes or malware.

Complete Example Session

# ============================================
# PART 1: CHECK ARRAY STATUS
# ============================================

cat /proc/mdstat
# md0 : active raid1 sdc[1] sdb[0]
#       976630464 blocks super 1.2 [2/2] [UU]

# ============================================
# PART 2: IDENTIFY A FAILED DISK
# ============================================

# Simulate a failure (for testing)
sudo mdadm /dev/md0 --fail /dev/sdc

# Verify the array is degraded
cat /proc/mdstat
# [2/1] [U_]

# ============================================
# PART 3: REMOVE THE FAILED DISK
# ============================================

sudo mdadm /dev/md0 --remove /dev/sdc
# The slot is now empty.

# ============================================
# PART 4: ADD A REPLACEMENT
# ============================================

# Physically replace the disk, then:
sudo mdadm /dev/md0 --add /dev/sdd
# Rebuild starts automatically.

# ============================================
# PART 5: MONITOR REBUILD
# ============================================

watch -n 5 cat /proc/mdstat
# [=>...] recovery = 15% ...

# ============================================
# PART 6: ADJUST REBUILD SPEED
# ============================================

# Increase minimum speed for faster rebuild
echo 50000 | sudo tee /proc/sys/dev/raid/speed_limit_min

# ============================================
# PART 7: SCRUB THE ARRAY
# ============================================

echo check | sudo tee /sys/block/md0/md/sync_action
# Wait for completion, then:
cat /sys/block/md0/md/mismatch_cnt

# ============================================
# PART 8: REPAIR MISMATCHES
# ============================================

# If mismatch_cnt > 0:
echo repair | sudo tee /sys/block/md0/md/sync_action

# ============================================
# PART 9: PERSIST CONFIGURATION
# ============================================

sudo mdadm --detail --scan | sudo tee -a /etc/mdadm.conf

The nine parts trace the management lifecycle: status check, simulated failure, removal, replacement, rebuild monitoring, speed tuning, scrubbing, repair, and configuration persistence.

Quick Reference

Management Commands

Command	Purpose
`mdadm --detail /dev/md0`	Full array details
`mdadm /dev/md0 --fail /dev/sdX`	Mark disk as failed
`mdadm /dev/md0 --remove /dev/sdX`	Remove failed disk
`mdadm /dev/md0 --add /dev/sdX`	Add replacement, start rebuild
`cat /proc/mdstat`	Quick status
`mdadm --monitor --scan --oneshot`	Single check for cron

Scrubbing Commands

Command	Purpose
`echo check > /sys/block/md0/md/sync_action`	Read-only verification
`echo repair > /sys/block/md0/md/sync_action`	Correct mismatches
`cat /sys/block/md0/md/mismatch_cnt`	Count inconsistencies

Rebuild Speed

Parameter	Default	Purpose
`speed_limit_min`	1000 KB/s	Minimum rebuild rate
`speed_limit_max`	200000 KB/s	Maximum rebuild rate

Best Practices

✅ Do This:

sudo mdadm /dev/md0 --fail /dev/sdc
sudo mdadm /dev/md0 --remove /dev/sdc

✅ Fail first, then remove

watch -n 5 cat /proc/mdstat

✅ Monitor rebuild progress

echo check | sudo tee /sys/block/md0/md/sync_action

✅ Periodic scrub for silent corruption

❌ Don’t Do This:

sudo mdadm /dev/md0 --remove /dev/sdc

❌ Remove without failing if the disk still responds

# Ignore degraded array

❌ A degraded array is one failure from total loss

sudo mdadm --detail --scan > /etc/mdadm.conf

❌ Overwrites existing entries; use >>

Common Pitfalls

Pitfall	Why It Happens	Fix
Remove fails	Disk still active in array	`--fail` first, then `--remove`
Array not assembled on boot	`mdadm.conf` stale	`mdadm --detail --scan >> /etc/mdadm.conf`
Rebuild too slow	Default speed limit	Increase `speed_limit_min`
Silent corruption undetected	No scrub scheduled	Periodic `echo check > sync_action`
Second disk fails during rebuild	Rebuild stress on remaining disks	Monitor; consider RAID 6 or 10

Real-World Examples

1. Check Status

cat /proc/mdstat

2. Detailed View

sudo mdadm --detail /dev/md0

3. Mark Failed

sudo mdadm /dev/md0 --fail /dev/sdc

4. Remove Disk

sudo mdadm /dev/md0 --remove /dev/sdc

5. Add Replacement

sudo mdadm /dev/md0 --add /dev/sdd

6. Monitor Rebuild

watch -n 5 cat /proc/mdstat

7. Start Scrub

echo check | sudo tee /sys/block/md0/md/sync_action

8. Repair Mismatches

echo repair | sudo tee /sys/block/md0/md/sync_action

9. Check Mismatches

cat /sys/block/md0/md/mismatch_cnt

10. Persist Config

sudo mdadm --detail --scan >> /etc/mdadm.conf

Visual

Recovery Sequence

┌──────────────────────────────────────────────┐
│  Disk failure detected                       │
│  /proc/mdstat shows [U_]                     │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────┐
│  mdadm /dev/md0 --fail /dev/sdc              │
│  (if disk still responds)                    │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────┐
│  mdadm /dev/md0 --remove /dev/sdc            │
│  (slot becomes empty)                        │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────┐
│  Physically replace disk                     │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────┐
│  mdadm /dev/md0 --add /dev/sdd               │
│  (rebuild starts automatically)              │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────┐
│  Monitor rebuild                             │
│  watch -n 5 cat /proc/mdstat                 │
└──────────────────────────────────────────────┘

The sequence is deliberate. Failing first prevents the array from using a device that may return corrupted data.

Scrub Decision

┌──────────────────────────────────────────────┐
│  Periodic check (cron or manual)             │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────┐
│  echo check > /sys/block/md0/md/sync_action  │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────┐
│  Wait for completion                         │
│  cat /proc/mdstat shows progress             │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────┐
│  mismatch_cnt > 0?                           │
└──────────┬───────────────────────┬───────────┘
           │ YES                   │ NO
           ▼                       ▼
┌──────────────────────┐  ┌──────────────────────┐
│  echo repair         │  │  Array is consistent │
│  Correct mismatches  │  │  No action needed    │
└──────────────────────┘  └──────────────────────┘

A scrub does not require unmounting. It runs in the background and reports mismatches when complete.

Summary

Item	Value
Status check	`cat /proc/mdstat`
Detailed view	`mdadm --detail /dev/md0`
Mark failed	`mdadm /dev/md0 --fail /dev/sdX`
Remove disk	`mdadm /dev/md0 --remove /dev/sdX`
Add replacement	`mdadm /dev/md0 --add /dev/sdX`
Start scrub	`echo check > /sys/block/md0/md/sync_action`
Repair mismatches	`echo repair > /sys/block/md0/md/sync_action`
Persist config	`mdadm --detail --scan >> /etc/mdadm.conf`

Key takeaways:

Monitoring is the first responsibility. A degraded array is one failure from data loss. Check /proc/mdstat regularly and configure mdadm --monitor for alerts .
Fail before removing. A disk that is responding but corrupted will continue to be used unless explicitly failed. mdadm --fail forces the array to stop trusting it .
The rebuild starts automatically. After --add, the kernel begins reconstruction. The array remains usable, but performance is reduced until the rebuild completes .
Scrubbing detects silent corruption. Periodic checks via sync_action find mismatches that would otherwise go unnoticed until a disk fails and the parity cannot reconstruct the data .
Configuration must be persisted. After any array change, regenerate /etc/mdadm.conf with mdadm --detail --scan to ensure the array assembles on boot .

Remember: Managing an mdadm array is a discipline of verification. Check the status, fail the disk before removing it, monitor the rebuild, and scrub periodically. The commands are simple, but the sequence matters. An array that is not monitored is an array that will fail silently.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!