LFCS 119 🐧 Remote Sync — rsync Advanced - KandZ

⏱️ Estimated read: 9 min | |

The basic rsync command handles the common case: copy a directory, mirror a destination, transfer over SSH. The advanced features handle the cases that the basic command cannot express: excluding files by pattern, limiting bandwidth, synchronizing over a non-standard port, preserving hard links and extended attributes, and resuming interrupted transfers without retransmitting the entire file. The LFCS exam tests the flags that control these behaviors, and it tests the specific sequence for verifying a transfer with the --checksum flag before making it permanent. The critical distinction is between --exclude and --include: the first skips matching files, and the second overrides a previous exclusion. The order of the rules matters.

Key point: rsync filters are evaluated in the order they appear on the command line. The first matching rule wins. A --include after an --exclude can re-include a file that the exclude would have skipped. The --delete-excluded flag removes excluded files from the destination, which is different from --delete (which removes files that are missing from the source). The --checksum flag is slower than the default size-and-time comparison, but it is the correct verification when timestamps are unreliable.

Why advanced rsync requires explicit flags

The filter-order problem. The --exclude flag is simple when there is only one pattern. With multiple patterns, the order matters. A broad exclusion like --exclude='*.log' followed by a narrow inclusion like --include='important.log' does not work as expected: the exclusion matches first, and the inclusion is never evaluated. The correct order is to place the inclusion before the exclusion, or to use the --filter rule syntax with explicit + and - prefixes.

The bandwidth problem. A full backup over a metered or shared network connection can saturate the link. The --bwlimit flag caps the transfer rate in kilobytes per second. The value 1000 means 1 MB/s. Without it, rsync uses as much bandwidth as the network provides.

The hard-link problem. A backup of a filesystem with hard links (e.g., a mail spool, a deduplicated archive) must preserve the link relationships. Without -H, rsync copies each link as a separate file, breaking the deduplication. The -H flag tells rsync to track hard links and recreate them on the destination.

The extended attribute problem. SELinux contexts, ACLs, and user-defined extended attributes are not preserved by the -a flag alone. The -A flag preserves ACLs. The -X flag preserves extended attributes. The -aAX combination is the standard for a complete backup on a system that uses these features.

The resume problem. An interrupted transfer of a 10 GB file with --partial keeps the partial file, but rsync restarts the transfer from the beginning unless --append or --append-verify is used. The --append flag appends data to the partial file without verification, which is faster but unsafe. The --append-verify flag verifies the existing data before appending, which is safe but slower. The -P flag combines --partial and --progress, and it is the standard for large transfers.

a. Filters, includes, and excludes

The --exclude flag skips files that match the pattern. The --include flag re-includes files that would otherwise be excluded. The order of the rules on the command line determines the precedence.

# Exclude all logs
rsync -av --exclude='*.log' /src/ /dest/

# Include important.log before excluding all logs
rsync -av --include='important.log' --exclude='*.log' /src/ /dest/

The --exclude-from flag reads patterns from a file. Each line is a pattern. Blank lines and lines starting with # are comments.

# Exclusions from a file
rsync -av --exclude-from=exclude.txt /src/ /dest/

The --filter flag provides a more expressive syntax. The rules are prefixed with + (include) or - (exclude). The --filter='- *.log' is equivalent to --exclude='*.log'.

# Filter with explicit rules
rsync -av --filter='+ important.log' --filter='- *.log' /src/ /dest/

The --delete-excluded flag removes excluded files from the destination. This is different from --delete, which removes files that are missing from the source. If a file is excluded from the transfer, it is still present on the destination. The --delete-excluded flag deletes it.

# Delete excluded files from the destination
rsync -av --delete --delete-excluded --exclude='*.log' /src/ /dest/

b. Bandwidth, ports, and remote shell

The --bwlimit flag caps the transfer rate. The --port flag specifies the SSH port. The -e flag specifies the remote shell command.

# Limit to 1 MB/s
rsync -av --bwlimit=1000 /src/ user@host:/dest/

# Use a non-standard SSH port
rsync -av -e 'ssh -p 2222' /src/ user@host:/dest/

# Use a specific SSH key
rsync -av -e 'ssh -i ~/.ssh/work_key' /src/ user@host:/dest/

The --timeout flag sets the I/O timeout in seconds. If no data is transferred within the timeout, rsync exits. This is useful for scripts that should not hang indefinitely on a dead connection.

# Timeout after 60 seconds of no data
rsync -av --timeout=60 /src/ user@host:/dest/

The -c or --checksum flag changes the comparison algorithm. By default, rsync compares file size and modification time. With --checksum, it computes a checksum of each file and compares the checksums. This is slower but more accurate when timestamps are unreliable (e.g., after a restore, or when files are touched without content changes).

# Checksum-based comparison
rsync -avc /src/ /dest/

c. Hard links, ACLs, and extended attributes

The -H flag preserves hard links. The -A flag preserves ACLs. The -X flag preserves extended attributes. The -aAX combination is the standard for a complete backup.

# Preserve hard links, ACLs, and extended attributes
rsync -aHAX /src/ /dest/

The --numeric-ids flag prevents the mapping of user and group names to IDs. This is important when the source and destination systems have different user databases. Without it, rsync maps names to local IDs, which can produce incorrect ownership on the destination.

# Preserve numeric IDs
rsync -aHAX --numeric-ids /src/ user@host:/dest/

The --sparse flag handles sparse files efficiently. A sparse file is a file with large blocks of zero bytes. Without --sparse, rsync writes the zeros to the destination, which can be much larger than the source. With --sparse, rsync writes the zero blocks as sparse blocks, preserving the file’s apparent size but not the disk usage.

# Preserve sparse files
rsync -av --sparse /src/ /dest/

Complete Example Session

# ============================================
# PART 1: BASIC EXCLUDE
# ============================================

rsync -av --exclude='*.log' /src/ /dest/
# Skips all files ending in .log.

# ============================================
# PART 2: INCLUDE BEFORE EXCLUDE
# ============================================

rsync -av --include='important.log' --exclude='*.log' /src/ /dest/
# Includes important.log, excludes all other .log files.

# ============================================
# PART 3: EXCLUDE FILE
# ============================================

cat > exclude.txt <<'EOF'
*.log
.cache/
node_modules/
EOF

rsync -av --exclude-from=exclude.txt /src/ /dest/

# ============================================
# PART 4: DELETE EXCLUDED
# ============================================

rsync -av --delete --delete-excluded --exclude='*.log' /src/ /dest/
# Removes .log files from the destination.

# ============================================
# PART 5: BANDWIDTH LIMIT
# ============================================

rsync -av --bwlimit=1000 /src/ user@host:/dest/
# Caps the transfer at 1 MB/s.

# ============================================
# PART 6: NON-STANDARD PORT
# ============================================

rsync -av -e 'ssh -p 2222' /src/ user@host:/dest/

# ============================================
# PART 7: CHECKSUM COMPARISON
# ============================================

rsync -avc /src/ /dest/
# Uses checksums instead of size/time.

# ============================================
# PART 8: HARD LINKS, ACLS, EXTENDED ATTRIBUTES
# ============================================

rsync -aHAX --numeric-ids /src/ user@host:/dest/
# Preserves all filesystem metadata.

# ============================================
# PART 9: VERIFY THE TRANSFER
# ============================================

rsync -avcn --delete /src/ /dest/
# Dry run with checksums: confirms the destination is identical.

The nine parts trace the advanced workflow: basic exclude, include-before-exclude, exclude file, delete-excluded, bandwidth limit, non-standard port, checksum, hard links, and verification.

Quick Reference

Filter Flags

Flag	Purpose
`--exclude`	Skip matching files
`--include`	Re-include files
`--exclude-from`	Read patterns from file
`--filter`	Explicit `+`/`-` rules
`--delete-excluded`	Remove excluded files from destination

Transfer Flags

Flag	Purpose
`--bwlimit`	Limit bandwidth (KB/s)
`--timeout`	I/O timeout (seconds)
`-e 'ssh -p <port>'`	Non-standard SSH port
`-e 'ssh -i <key>'`	Specific SSH key
`-c`	Checksum comparison
`-H`	Preserve hard links
`-A`	Preserve ACLs
`-X`	Preserve extended attributes
`--numeric-ids`	No name-to-ID mapping
`--sparse`	Handle sparse files
`--append-verify`	Resume with verification

Archive Components

Flag	Meaning
`-a`	Archive mode
`-H`	Hard links
`-A`	ACLs
`-X`	Extended attributes
`-aHAX`	Complete backup

Best Practices

✅ Do This:

rsync -av --include='important.log' --exclude='*.log' /src/ /dest/
<!-- Include before exclude -->                       # ✅

rsync -aHAX --numeric-ids /src/ user@host:/dest/
<!-- Preserve all metadata for backup -->             # ✅

rsync -avcn --delete /src/ /dest/
<!-- Dry run with checksums before real transfer -->  # ✅

❌ Don’t Do This:

# Exclude before include
<!-- The include is never evaluated -->              # ❌

# Use --append without verification
<!-- Resumes at the wrong position if the file changed --> # ❌

# Use --delete-excluded without understanding it
<!-- Removes excluded files from destination -->      # ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
Include not working	Exclude matched first	Place include before exclude
Bandwidth limit ignored	Wrong unit	`--bwlimit` is KB/s
Port ignored	`-e` missing	Use `-e 'ssh -p <port>'`
Ownership wrong	Name-to-ID mapping	`--numeric-ids`
Sparse file grows	No `--sparse`	Add `--sparse`
Resume starts from zero	No `--append-verify`	Add `--append-verify`

Real-World Examples

1. Exclude Pattern

rsync -av --exclude='*.log' /src/ /dest/

2. Include Before Exclude

rsync -av --include='important.log' --exclude='*.log' /src/ /dest/

3. Exclude File

rsync -av --exclude-from=exclude.txt /src/ /dest/

4. Delete Excluded

rsync -av --delete --delete-excluded --exclude='*.log' /src/ /dest/

5. Bandwidth Limit

rsync -av --bwlimit=1000 /src/ user@host:/dest/

6. Non-Standard Port

rsync -av -e 'ssh -p 2222' /src/ user@host:/dest/

7. Checksum Comparison

rsync -avc /src/ /dest/

8. Preserve Hard Links

rsync -aH /src/ /dest/

9. Preserve ACLs and Extended Attributes

rsync -aAX /src/ /dest/

10. Verify with Dry Run

rsync -avcn --delete /src/ /dest/

Visual

Filter Order

┌──────────────────────────────────────────────┐
│  Rule 1: --include='important.log'           │
│  Matches → include the file                  │
└──────────────────┬───────────────────────────┘
                   │ no match
                   ▼
┌──────────────────────────────────────────────┐
│  Rule 2: --exclude='*.log'                   │
│  Matches → exclude the file                  │
└──────────────────┬───────────────────────────┘
                   │ no match
                   ▼
┌──────────────────────────────────────────────┐
│  No match → include by default               │
└──────────────────────────────────────────────┘

Backup Metadata

┌──────────────────────────────────────────────┐
│  -a    → permissions, times, symlinks        │
│  -H    → hard links                          │
│  -A    → ACLs                                │
│  -X    → extended attributes                 │
│  --numeric-ids → preserve UIDs/GIDs          │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────┐
│  rsync -aHAX --numeric-ids /src/ /dest/      │
│  Complete filesystem backup                  │
└──────────────────────────────────────────────┘

Summary

Item	Value
Exclude	`--exclude`
Include	`--include` (before exclude)
Exclude file	`--exclude-from`
Filter syntax	`--filter='+ pattern'`
Delete excluded	`--delete-excluded`
Bandwidth limit	`--bwlimit` (KB/s)
Non-standard port	`-e 'ssh -p <port>'`
Checksum	`-c`
Hard links	`-H`
ACLs	`-A`
Extended attributes	`-X`
Numeric IDs	`--numeric-ids`
Sparse files	`--sparse`
Resume with verify	`--append-verify`

Key takeaways:

Filter order matters. --include rules must appear before --exclude rules to take effect. The first matching rule wins, and the --filter syntax with explicit + and - prefixes makes the order visible .
--delete-excluded is not --delete. The --delete flag removes files missing from the source. The --delete-excluded flag removes files that are excluded from the transfer. Both are destructive and require a dry run.
-aHAX is the complete backup. The archive flag preserves permissions, times, and symlinks. The -H flag preserves hard links. The -A and -X flags preserve ACLs and extended attributes .
--numeric-ids prevents mapping errors. On systems with different user databases, name-based mapping produces wrong ownership. The --numeric-ids flag preserves the original UIDs and GIDs.
--append-verify is the safe resume. The --append flag is faster but unsafe. The --append-verify flag verifies the existing data before appending, which is the correct behavior for large files.

Remember: Advanced rsync is about control. The filters control what is transferred. The bandwidth limit controls how fast. The preservation flags control what metadata is kept. The resume flags control how interrupted transfers are handled. The exam tests the flags and the order. The dry run is the verification step. The real run is the commitment.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!