LFCS 151 🐧 Testing Backups - KandZ

⏱️ Estimated read: 11 min | |

The previous chapters covered the backup strategies and the tools that create the backups: tar and rsync. This chapter covers the step that determines whether the backups are worth anything: the testing. A backup that has never been restored is a backup that might not work. The LFCS exam tests the conceptual model: the distinction between the archive verification and the full restore test, the checksum verification, the restore to an isolated environment, and the documentation of the restore procedure. The critical trap is assuming that the backup is valid because the backup job reported success. A backup that completed without errors is a backup that was written to disk. It is not necessarily a backup that can be read back. The corruption can happen in the storage, in the compression, or in the network transfer. The only way to know is to test.

Key point: The backup testing has two levels. The archive verification confirms that the backup file is readable and that the contents are intact. The full restore test confirms that the restore procedure works and that the restored data matches the original. The verification is fast and can run on every backup. The full restore test is slow and runs on a schedule. The tar -t lists the archive. The sha256sum compares the checksums. The diff -r compares the directories. The restore to a test directory is the safe approach.

Why backup testing requires a specific mental model

The success is not the proof problem. The backup job can exit with the status 0 while the archive is unusable. The tar can succeed in creating a file that is truncated because the disk is full. The rsync can succeed in transferring the files while the checksum mismatch goes unnoticed. The success status confirms the command ran, not the data is recoverable. The test is the proof.

The verification versus restore problem. The archive verification reads the backup file and checks the internal structure. The tar -t reads the index and lists the files. The gzip -t reads the compressed stream and checks the integrity. The verification is fast and does not need the target disk. The full restore test extracts the backup to a disk and compares it to the source. The full test catches the errors that the verification misses: the missing files, the wrong permissions, the restored applications that do not start. The two levels are complementary.

The isolated environment problem. The full restore test should not overwrite the production data. The restore goes to an isolated directory, a separate disk, or a test VM. The restore to a test directory is the simplest: the tar -x to /tmp/restore-test/ and the diff -r against the source. The restore to a test VM is the more thorough: the entire system, the bootloader, the services. The isolated environment is the safety net.

The checksum problem. The sha256sum computes the checksum of the backup file. The checksum is stored with the backup. The restore test computes the checksum again and compares. The mismatch means the backup file was modified or corrupted. The checksum is the integrity check that does not require the extraction. The sha256sum -c verifies the checksum from the file.

The documentation problem. The restore procedure must be documented and tested by someone who is not the backup author. The author knows the steps that are not in the documentation. The other person discovers the missing steps during the test. The documented procedure is the recovery plan. The undocumented procedure is the recovery improvisation.

The trade-off. The verification is fast and cheap. The full restore is slow and resource-intensive. The correct strategy is the verification on every backup and the full restore test on a schedule (monthly or quarterly). The full restore test is the proof that the procedure works, and it is the only way to catch the missing files and the wrong permissions.

a. The archive verification

The archive verification confirms that the backup file is readable. The tar -t lists the contents without extracting. The tar -tv adds the permissions, size, and timestamp. The tar -tz decompresses the gzip stream to read the index.

# List the archive contents
tar -tzvf /backup/full-2026-10-30.tar.gz | head

# Count the files
tar -tzf /backup/full-2026-10-30.tar.gz | wc -l

# Check the gzip integrity
gzip -t /backup/full-2026-10-30.tar.gz
echo $?
# 0 means the gzip stream is valid

The gzip -t command reads the compressed stream and verifies the internal checksums. The 0 exit code means the stream is intact. The non-zero exit code means the file is corrupted.

The tar -t reads the entire archive to list the contents. The command fails if the archive is truncated or if the gzip stream is corrupted. The output is the file list, which is the confirmation that the archive is readable.

The checksum is the second verification.

# Compute the checksum
sha256sum /backup/full-2026-10-30.tar.gz

# Store the checksum
sha256sum /backup/full-2026-10-30.tar.gz > /backup/full-2026-10-30.tar.gz.sha256

# Verify the checksum later
sha256sum -c /backup/full-2026-10-30.tar.gz.sha256
# Output: full-2026-10-30.tar.gz: OK

The sha256sum -c reads the checksum file and compares the computed checksum to the recorded one. The OK means the file matches. The FAILED means the file was modified.

b. The full restore test to an isolated directory

The full restore test extracts the backup to a test directory and compares it to the source.

# Create the test directory
mkdir -p /tmp/restore-test

# Extract the full backup
sudo tar -xpzvf /backup/full-2026-10-30.tar.gz -C /tmp/restore-test/

# Extract the incrementals in order
sudo tar -xpzvf /backup/inc-2026-10-31.tar.gz -C /tmp/restore-test/

# Compare the restored directory to the source
sudo diff -r /home/user/ /tmp/restore-test/home/user/

The diff -r compares the files recursively. The no output means the directories are identical. The differences are the potential problems: the missing files, the changed content, the wrong permissions.

The permissions and ownership check is the second step.

# List the restored files with the permissions
ls -la /tmp/restore-test/home/user/

# Compare the ownership
stat /home/user/file.txt /tmp/restore-test/home/user/file.txt

The ls -la and the stat show the owner, the group, and the mode. The mismatch is the permission problem. The tar -p restores the permissions, but only if the extracting user has the permission to set them. The root is required for the ownership.

The rsync restore test is the alternative. The rsync -a --dry-run shows the differences between the backup and the source without transferring. The --dry-run output is the verification.

# Compare the backup to the source
rsync -avn /backup/home/ /home/user/

The output shows the files that differ. The empty output means the backup matches the source. The differences are the changes since the last backup, not the errors.

c. The documentation and the scheduled test

The restore procedure must be documented in a runbook. The runbook has the steps, the commands, and the expected outputs. The runbook is tested by the backup author and by the other administrator.

# Restore Runbook

## Prerequisites
- The backup file: /backup/full-YYYY-MM-DD.tar.gz
- The target disk: /dev/sdb1
- The root access

## Steps
1. Mount the target disk: `mount /dev/sdb1 /mnt/new-root`
2. Extract the full backup: `tar -xpzvf /backup/full.tar.gz -C /mnt/new-root/`
3. Extract the incrementals in order
4. Restore the bootloader: `grub-install /dev/sdb`
5. Relabel SELinux: `restorecon -Rv /mnt/new-root/`
6. Reboot and verify

## Expected Output
- The system boots
- The services start
- The files match the backup

The scheduled test is the quarterly full restore in the test environment. The test is documented in the audit log. The failure of the test is the incident that triggers the investigation.

The cron job for the verification.

# /etc/cron.d/backup-verify
0 3 * * * root tar -tzf /backup/latest.tar.gz > /dev/null || echo "Backup verification failed" | mail -s "Backup" admin@example.com

The job runs the tar -t on the latest backup and sends an email if the verification fails. The verification is the automated check. The full restore is the manual periodic test.

Complete Example Session

# ============================================
# PART 1: LIST THE ARCHIVE
# ============================================

tar -tzvf /backup/full-2026-10-30.tar.gz | head

# ============================================
# PART 2: CHECK THE GZIP INTEGRITY
# ============================================

gzip -t /backup/full-2026-10-30.tar.gz
echo $?
# 0

# ============================================
# PART 3: COMPUTE THE CHECKSUM
# ============================================

sha256sum /backup/full-2026-10-30.tar.gz > /backup/full-2026-10-30.tar.gz.sha256

# ============================================
# PART 4: VERIFY THE CHECKSUM
# ============================================

sha256sum -c /backup/full-2026-10-30.tar.gz.sha256
# full-2026-10-30.tar.gz: OK

# ============================================
# PART 5: RESTORE TO TEST DIRECTORY
# ============================================

mkdir -p /tmp/restore-test
sudo tar -xpzvf /backup/full-2026-10-30.tar.gz -C /tmp/restore-test/

# ============================================
# PART 6: COMPARE THE RESTORE
# ============================================

sudo diff -r /home/user/ /tmp/restore-test/home/user/
# No output: identical

# ============================================
# PART 7: CHECK PERMISSIONS
# ============================================

stat /home/user/file.txt /tmp/restore-test/home/user/file.txt

# ============================================
# PART 8: DOCUMENT THE PROCEDURE
# ============================================

# Write the restore runbook

# ============================================
# PART 9: SCHEDULE THE VERIFICATION
# ============================================

0 3 * * * root tar -tzf /backup/latest.tar.gz > /dev/null || echo "Backup verification failed" | mail -s "Backup" admin@example.com

The nine parts trace the backup testing: list, gzip check, checksum, checksum verify, restore to test, compare, permissions, document, schedule.

Quick Reference

Archive Verification

Command	Purpose
`tar -tzf <file>`	List contents
`gzip -t <file>`	Check gzip stream
`sha256sum <file>`	Compute checksum
`sha256sum -c <file>.sha256`	Verify checksum

Restore Test

Command	Purpose
`tar -xpzvf <file> -C <dir>`	Extract to test dir
`diff -r <src> <restored>`	Compare directories
`rsync -avn <src> <restored>`	Compare (dry run)
`stat <file>`	Check permissions

Documentation

Step	Purpose
Runbook	Document the procedure
Schedule	Quarterly full restore
Automated verification	Cron job with `tar -t`

Best Practices

✅ Do This:

tar -tzf /backup/latest.tar.gz
<!-- Verify the archive on every backup -->           # ✅

sha256sum -c /backup/full.tar.gz.sha256
<!-- Verify the checksum -->                          # ✅

diff -r /home/user/ /tmp/restore-test/home/user/
<!-- Compare the restore to the source -->            # ✅

❌ Don’t Do This:

# Assume the backup is valid
<!-- The success status is not the proof -->          # ❌

# Restore over the production data
<!-- Use the isolated test directory -->             # ❌

# Skip the documentation
<!-- The procedure is lost with the author -->       # ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
Backup corrupted	Storage or network error	Checksum on every backup
Restore fails	Wrong order or missing incrementals	Restore full + incrementals in order
Permissions wrong	No `-p` flag	Use `-p` or `-a`
Procedure unknown	No documentation	Write the runbook
Restore test not run	No schedule	Quarterly test

Real-World Examples

1. List Archive

tar -tzf /backup/full.tar.gz

2. Gzip Check

gzip -t /backup/full.tar.gz

3. Checksum

sha256sum /backup/full.tar.gz

4. Verify Checksum

sha256sum -c /backup/full.tar.gz.sha256

5. Restore to Test

tar -xpzvf /backup/full.tar.gz -C /tmp/restore-test/

6. Compare

diff -r /home/user/ /tmp/restore-test/home/user/

7. Permissions

stat /home/user/file.txt

8. Document

Write the restore runbook

9. Schedule Verification

0 3 * * * root tar -tzf /backup/latest.tar.gz > /dev/null || echo "Fail" | mail

10. Full Restore Test

Quarterly full restore in the test VM

Visual

Testing Levels

┌──────────────────────────────────────────────┐
│  Archive verification                        │
│  ├── tar -t                                  │
│  ├── gzip -t                                 │
│  └── sha256sum -c                            │
│  Fast, automated, runs on every backup       │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────┐
│  Full restore test                           │
│  ├── Extract to test directory               │
│  ├── Compare with diff -r                    │
│  └── Check permissions                       │
│  Slow, scheduled (monthly/quarterly)         │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────┐
│  Documentation                               │
│  ├── Restore runbook                         │
│  └── Tested by another admin                 │
└──────────────────────────────────────────────┘

Restore to Test

┌──────────────────────────────────────────────┐
│  Backup: /backup/full.tar.gz                 │
└──────────────────┬───────────────────────────┘
                   │
                   │ tar -xpzvf
                   ▼
┌──────────────────────────────────────────────┐
│  Test dir: /tmp/restore-test/                │
└──────────────────┬───────────────────────────┘
                   │
                   │ diff -r
                   ▼
┌──────────────────────────────────────────────┐
│  Source: /home/user/                         │
│  No output: identical                        │
└──────────────────────────────────────────────┘

Summary

Item	Value
Archive list	`tar -tzf`
Gzip check	`gzip -t`
Checksum	`sha256sum`
Checksum verify	`sha256sum -c`
Restore test	`tar -x` to test dir
Compare	`diff -r`
Permissions	`stat`
Documentation	Restore runbook
Scheduled test	Quarterly full restore

Key takeaways:

The success status is not the proof. The backup job can exit with 0 while the archive is corrupted or truncated. The verification is the proof. The tar -t and the sha256sum -c are the mandatory checks.
The archive verification and the full restore test are complementary. The verification is fast and checks the backup file. The full restore is slow and checks the procedure. Both are necessary.
The restore test goes to an isolated environment. The test directory or the test VM is the safety net. The restore over the production data is the disaster, not the test.
The checksum is the integrity check. The sha256sum is computed at backup time and verified at restore time. The mismatch means the file was modified or corrupted.
The documentation is the recovery plan. The runbook documents the steps, the commands, and the expected outputs. The runbook must be tested by someone who is not the author.

Remember: The backup is the insurance. The test is the claim. The tar -t is the fast check. The sha256sum -c is the integrity check. The full restore is the procedure test. The runbook is the plan. The exam tests the verification commands and the restore procedure.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!