LFCE 119 🐧 mdadm — Managing and Recovering - KandZ

⏱️ Estimated read: 13 min | |

Creating an array is the beginning. The work that determines whether the array survives a disk failure—or a second failure during a rebuild—is the management that follows. mdadm provides the commands to mark a disk as failed, remove it, add a replacement, monitor the rebuild, and assemble a degraded array when the system boots with fewer disks than expected. The difference between an array that recovers cleanly and one that loses data is usually not the RAID level; it is whether the administrator knew which commands to run and in what order.

Quick How-To Guides & Fast FixesNeed a quick fix or step-by-step solution? Browse our fast answers on HowTo.KandZ.me:

Operating Systems CLI →Dev Environment Languages →Web Network Tools →Security Accounts →Security Access Tech →Device Phone Basics →Social Messaging →Digital Files Media →

This chapter covers the management and recovery lifecycle: checking array status with --detail and /proc/mdstat, the fail-remove-add sequence for replacing a disk, the rebuild process and how to monitor it, the assembly of degraded arrays after a reboot, and the --force and --re-add options that exist for edge cases. It also covers the persistence of array configuration and the monitoring setup that makes failures visible before they become data loss.

Key point: A failed disk must be marked failed with --fail before it can be removed with --remove. Skipping the fail step leaves the array in a state where the disk is still considered a member, and the remove command will be refused.

Why the recovery sequence has this order

The fail-then-remove distinction. A disk can fail in two ways: the kernel notices an I/O error and marks the disk faulty automatically, or the administrator notices a degraded array and must mark the disk manually. In both cases, mdadm --fail is what changes the disk’s state in the array metadata from “active” to “faulty.” Only after that state change does --remove work, because --remove detaches a disk that is already marked as not participating . Removing a disk without failing it first is refused, because it would be indistinguishable from pulling a healthy disk.

The add-and-rebuild process. Adding a replacement disk with --add does not instantly restore redundancy. The kernel begins a rebuild: it reads the surviving data and parity, reconstructs the missing blocks, and writes them to the new disk. The rebuild runs in the background, and the array remains usable during it, but performance is reduced. The rebuild completes when every block has been reconstructed, at which point the new disk transitions from “spare rebuilding” to “active sync” . The array is only fully redundant again at that point.

The assembly problem after reboot. When a system boots with a degraded array, mdadm must decide whether to assemble and start the array with fewer disks than expected. By default, if the expected number of devices is not found and --scan is used, the array is assembled but not started. The --run option forces it to start anyway, which is what allows a degraded array to come online . Without --run, the array sits assembled but inactive, and the filesystem on it cannot be mounted.

The force option and its risk. The --force option allows assembly even when the metadata on some devices appears out of date, and it can mark devices recorded as failed as working again so the array can start . The manual is explicit about the risk: “An array which requires --force to be started may contain data corruption” . It exists for recovery scenarios where the alternative is data loss, not as a routine option.

a. Checking array status

The two primary status commands are mdadm --detail and cat /proc/mdstat. They show different views of the same array.

# Detailed view: state, devices, UUID, rebuild progress
sudo mdadm --detail /dev/md0

# Brief view: all arrays, active/inactive state, resync percentage
cat /proc/mdstat

The --detail output includes a “State” line (clean, active, degraded, recovering, resyncing) and a device table at the bottom showing each member’s state: active sync, spare, faulty, removed, or spare rebuilding . The /proc/mdstat output shows the array’s personality, the member devices with their slot numbers, and a progress bar if a rebuild or resync is running.

b. Replacing a failed disk

The sequence is fail, remove, add. The fail step marks the disk as faulty in the metadata. The remove step detaches it from the array. The add step attaches the replacement and triggers the rebuild.

# Step 1: Mark the failed disk as faulty
sudo mdadm --manage /dev/md0 --fail /dev/sdd

# Step 2: Remove the faulty disk from the array
sudo mdadm --manage /dev/md0 --remove /dev/sdd

# Step 3: Add the replacement disk
sudo mdadm --manage /dev/md0 --add /dev/sdf

If the kernel has already marked the disk as faulty due to I/O errors, the --fail step may be redundant, but running it is harmless and ensures the state is explicit. After --add, the new disk appears as spare rebuilding in --detail and the rebuild percentage appears in /proc/mdstat . The rebuild can take hours for large disks, and performance is reduced until it completes.

c. Assembling a degraded array after reboot

When a system boots with a missing disk, mdadm may assemble the array but not start it, because the expected number of devices is not present. The --run option forces the array to start in degraded mode.

# Assemble and start even with fewer devices than expected
sudo mdadm --assemble --run /dev/md0

# Or scan all arrays and start those found
sudo mdadm --assemble --scan --run

The --run option is documented as attempting to start the array “even if fewer drives were given than were present last time the array was active” . Without it, the array is assembled but remains inactive. Once started, the filesystem can be mounted and the disk replacement can proceed.

d. The force and re-add options

The --force option is for recovery from inconsistent metadata. It assembles the array even if some devices have out-of-date metadata, and it can mark failed devices as working if that is the only way to start the array . The manual warns that this may produce data corruption, so it is a last resort.

The --re-add option is for a different scenario: a disk that was removed due to a transient failure (a loose cable, a controller glitch) and is now working again. Re-adding it allows the array to catch up only the blocks that changed while it was absent, using the write-intent bitmap, rather than rebuilding the entire disk.

# Re-add a disk that was transiently removed
sudo mdadm /dev/md0 --re-add /dev/sdd

SUSE documentation notes that mdadm cannot reliably determine why a disk failed, so it treats any failed device as faulty until explicitly told otherwise . The --re-add option is that explicit instruction. For systems with unreliable connections, the POLICY action=re-add line in /etc/mdadm.conf automates the process when a device reappears .

Complete Example Session

# ============================================
# PART 1: CHECK ARRAY STATUS
# ============================================
sudo mdadm --detail /dev/md0 | grep -E "State|RaidDevice|faulty|active"

# Look for "degraded" in the State line.
# Look for a device marked "faulty" or "removed" at the bottom.

# ============================================
# PART 2: IDENTIFY THE FAILED DISK
# ============================================
# Check kernel logs for the failure message
sudo journalctl -k | grep -i "md0.*fail"

# Or examine each member
sudo mdadm --examine /dev/sdd | grep -E "State|Faulty"

# ============================================
# PART 3: MARK THE DISK AS FAILED
# ============================================
sudo mdadm --manage /dev/md0 --fail /dev/sdd

# Verify the state changed to "faulty"
sudo mdadm --detail /dev/md0 | grep sdd

# ============================================
# PART 4: REMOVE THE FAILED DISK
# ============================================
sudo mdadm --manage /dev/md0 --remove /dev/sdd

# The device slot is now marked "removed"
# /proc/mdstat shows the array as degraded
cat /proc/mdstat

# ============================================
# PART 5: ADD THE REPLACEMENT DISK
# ============================================
# If the replacement disk was previously part of another array,
# zero its superblock first:
# sudo mdadm --zero-superblock /dev/sdf

sudo mdadm --manage /dev/md0 --add /dev/sdf

# The rebuild begins automatically.

# ============================================
# PART 6: MONITOR THE REBUILD
# ============================================
watch cat /proc/mdstat

# Example output:
# md0 : active raid5 sdf[3] sdc[1] sdb[0]
#       1953262592 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/2] [UU_]
#       [==>....................]  recovery = 11.2% (219.../195...)

# ============================================
# PART 7: VERIFY AFTER REBUILD
# ============================================
sudo mdadm --detail /dev/md0 | grep -E "State|sdf"

# Expected: State: clean
# sdf state: active sync

# ============================================
# PART 8: ASSEMBLE A DEGRADED ARRAY AFTER REBOOT
# ============================================
# If the array is assembled but not started:
sudo mdadm --assemble --run /dev/md0

# Or scan for all arrays:
sudo mdadm --assemble --scan --run

# Verify it started
cat /proc/mdstat

# ============================================
# PART 9: RE-ADD A TRANSIENTLY REMOVED DISK
# ============================================
# If a disk was removed due to a cable issue and is now back:
sudo mdadm /dev/md0 --re-add /dev/sdd

# The bitmap allows it to catch up quickly.
# If re-add fails, fall back to --add and full rebuild.

# ============================================
# PART 10: CONFIGURE MONITORING
# ============================================
# Save configuration (should already be present)
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm.conf

# Add email alerts for failures
echo "MAILADDR admin@example.com" | sudo tee -a /etc/mdadm.conf

# Test the alert
sudo mdadm --monitor --scan --test --oneshot

The ten parts covered status checking, failure identification, the fail step, the remove step, the add step, rebuild monitoring, post-rebuild verification, degraded assembly with --run, re-adding a transient disk, and monitoring configuration.

Quick Reference

Management commands

Command	Effect
`mdadm --manage /dev/md0 --fail /dev/sdX`	Mark a disk as faulty
`mdadm --manage /dev/md0 --remove /dev/sdX`	Remove a faulty disk
`mdadm --manage /dev/md0 --add /dev/sdX`	Add a replacement disk
`mdadm /dev/md0 --re-add /dev/sdX`	Re-add a transiently removed disk
`mdadm --assemble --run /dev/md0`	Start a degraded array
`mdadm --assemble --scan --run`	Scan and start all degraded arrays

Status commands

Command	Purpose
`mdadm --detail /dev/md0`	Detailed array status
`cat /proc/mdstat`	Brief status, rebuild progress
`mdadm --examine /dev/sdX`	Superblock details on a member

Device states

State	Meaning
`active sync`	Healthy member, in sync
`spare`	Available but not in use
`faulty`	Marked failed, still attached
`removed`	Detached from the array
`spare rebuilding`	Replacement disk being rebuilt

Recovery options

Option	Purpose	Risk
`--run`	Start degraded array	None, expected behavior
`--force`	Assemble with out-of-date metadata	May cause data corruption
`--re-add`	Re-add transiently removed disk	Requires bitmap for fast catch-up

Best Practices

✅ Do This:

# Always fail before remove
mdadm --manage /dev/md0 --fail /dev/sdd                   # ✅

# Monitor rebuild with /proc/mdstat
watch cat /proc/mdstat                                     # ✅

# Use --run to start degraded arrays after reboot
mdadm --assemble --scan --run                              # ✅

# Configure email alerts for failures
echo "MAILADDR admin@example.com" >> /etc/mdadm.conf       # ✅

# Try --re-add before --add for transient failures
mdadm /dev/md0 --re-add /dev/sdd                           # ✅

❌ Don’t Do This:

# Remove without failing first
mdadm --manage /dev/md0 --remove /dev/sdd                  # ❌ (refused)

# Use --force routinely
mdadm --assemble --force /dev/md0                          # ❌ (corruption risk)

# Ignore a degraded array
# A second failure destroys it                              # ❌

# Assume the rebuild is instantaneous
# Large disks take hours                                     # ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
Remove command refused	Disk not marked faulty first	Run `--fail` before `--remove`
Array not started after reboot	Missing `--run` flag	Use `--assemble --scan --run`
Rebuild takes too long	Large disk, heavy I/O load	Monitor with `/proc/mdstat`; avoid heavy load
Re-add fails	Bitmap not present or disk too far out of sync	Fall back to `--add` (full rebuild)
Second disk fails during rebuild	No redundancy while degraded	Monitor rebuild; replace disks promptly
`--force` causes corruption	Metadata inconsistent across devices	Only use as last resort; back up first

Real-World Examples

1. Mark a disk as failed

mdadm --manage /dev/md0 --fail /dev/sdd

2. Remove a failed disk

mdadm --manage /dev/md0 --remove /dev/sdd

3. Add a replacement disk

mdadm --manage /dev/md0 --add /dev/sdf

4. Monitor rebuild progress

watch cat /proc/mdstat

5. Start a degraded array after reboot

mdadm --assemble --scan --run

6. Re-add a transiently removed disk

mdadm /dev/md0 --re-add /dev/sdd

7. Check array state

mdadm --detail /dev/md0 | grep State

8. Examine a member’s superblock

mdadm --examine /dev/sdd | grep -E "State|Slot"

9. Force assemble (last resort)

mdadm --assemble --force /dev/md0 /dev/sdb1 /dev/sdc1

10. Test monitoring alerts

mdadm --monitor --scan --test --oneshot

Visual

The fail-remove-add sequence

┌─────────────────────────────────────────────────────────────┐
│  REPLACING A FAILED DISK                                     │
│                                                              │
│  Healthy array:                                              │
│    [sdb active sync] [sdc active sync] [sdd active sync]    │
│                                                              │
│  Step 1: --fail /dev/sdd                                     │
│    [sdb active sync] [sdc active sync] [sdd faulty]         │
│                                                              │
│  Step 2: --remove /dev/sdd                                   │
│    [sdb active sync] [sdc active sync] [removed]            │
│                                                              │
│  Step 3: --add /dev/sdf                                      │
│    [sdb active sync] [sdc active sync] [sdf rebuilding]     │
│                                                              │
│  Rebuild completes:                                          │
│    [sdb active sync] [sdc active sync] [sdf active sync]    │
└─────────────────────────────────────────────────────────────┘

Degraded array assembly

┌─────────────────────────────────────────────────────────────┐
│  BOOT WITH A MISSING DISK                                    │
│                                                              │
│  mdadm --assemble --scan (default):                          │
│    Array assembled but NOT started.                          │
│    Expected 3 devices, found 2.                              │
│    State: inactive                                           │
│                                                              │
│  mdadm --assemble --scan --run:                              │
│    Array assembled AND started.                              │
│    State: degraded, active                                   │
│    Filesystem can be mounted.                                │
│                                                              │
│  --run is what allows degraded operation.                    │
└─────────────────────────────────────────────────────────────┘

Device states during recovery

┌─────────────────────────────────────────────────────────────┐
│  DEVICE STATE TRANSITIONS                                    │
│                                                              │
│  Healthy:  active sync                                       │
│       │                                                      │
│       ▼ I/O error                                           │
│  auto or --fail: faulty                                      │
│       │                                                      │
│       ▼ --remove                                             │
│  removed                                                     │
│       │                                                      │
│       ▼ --add                                                │
│  spare rebuilding                                            │
│       │                                                      │
│       ▼ rebuild complete                                     │
│  active sync                                                 │
└─────────────────────────────────────────────────────────────┘

–force risk

┌─────────────────────────────────────────────────────────────┐
│  --force ASSEMBLY                                            │
│                                                              │
│  Normal assembly:                                            │
│    All devices present, metadata consistent.                 │
│    Array starts, data is correct.                            │
│                                                              │
│  Forced assembly:                                            │
│    Some devices have out-of-date metadata.                   │
│    mdadm starts the array anyway.                            │
│    The most recent writes may be missing.                    │
│    Data corruption is possible .                 │
│                                                              │
│  Use only when the alternative is certain data loss.        │
└─────────────────────────────────────────────────────────────┘

Summary

Item	Value
Fail command	`mdadm --manage /dev/md0 --fail /dev/sdX`
Remove command	`mdadm --manage /dev/md0 --remove /dev/sdX`
Add command	`mdadm --manage /dev/md0 --add /dev/sdX`
Re-add command	`mdadm /dev/md0 --re-add /dev/sdX`
Degraded assembly	`mdadm --assemble --scan --run`
Status detail	`mdadm --detail /dev/md0`
Rebuild progress	`cat /proc/mdstat`
Force assembly risk	Possible data corruption
Monitoring config	`MAILADDR` in `/etc/mdadm.conf`

Key takeaways:

Fail before remove. The --fail command changes the disk’s metadata state from active to faulty. Only a faulty disk can be removed from the array. Skipping this step causes the remove command to be refused.
The rebuild is automatic but not instant. Adding a replacement disk triggers a background rebuild. The array is usable during the rebuild but not fully redundant until it completes.
Degraded arrays need --run to start after a reboot. Without it, mdadm assembles the array but leaves it inactive because the expected number of devices is missing.
--force is a recovery tool with a data corruption warning. The manual states that an array requiring --force may contain data corruption . It is not a routine option.
--re-add is for transient failures. A disk that disappeared due to a cable or controller glitch can rejoin without a full rebuild if the bitmap is present. If it fails, fall back to --add.
Monitoring is part of management. Configuring MAILADDR in /etc/mdadm.conf ensures that a degraded array sends an alert before a second failure destroys the data.

Remember: The recovery sequence is fail, remove, add, rebuild. Each step has a purpose, and skipping one produces a command that is refused or an array that is not fully recovered. The most common mistake is trying to remove a disk without failing it first. The second most common is rebooting a degraded system and assuming the array came up automatically, when --run was required. The third is ignoring a degraded array until a second disk fails during the rebuild, at which point recovery may not be possible. Configure monitoring, replace disks promptly, and verify that the rebuild reached active sync before considering the array recovered.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!