Bash Scripting 29 🐧 Remote Automation — Executing commands securely via SSH loops
A script that runs commands on one remote host is straightforward. A script that runs the same command on dozens of hosts, collects the output, and reports which ones succeeded and which failed is a different problem. The loop itself is trivial. The security and reliability concerns are not: how the script authenticates without prompting for a password, how it handles a host that is unreachable, how it distinguishes a failed remote command from a failed SSH connection, and how it avoids re-negotiating a new TCP connection for every command on the same host.
This chapter covers the SSH loop pattern: key-based authentication that works without a terminal, the BatchMode and ConnectTimeout options that make SSH fail instead of prompt, the for loop and while read constructs that iterate over a host list, the exit-status convention where 255 means connection failure and any other non-zero value means command failure, and the ControlMaster multiplexing that reuses a single connection across multiple commands to the same host. It also covers the safe handling of secrets, which should never be embedded in a script that is checked into a repository.
Key point: ssh exits with the remote command’s status, or with 255 if the connection itself failed. A loop that treats every non-zero status the same cannot distinguish “the command failed on the host” from “the host is unreachable.” Check for 255 separately.
Why SSH loops need these safeguards
The interactive prompt problem. By default, ssh prompts for a password if key authentication fails, and prompts for confirmation if the host key is unknown. In a script running from cron or a CI pipeline, there is no terminal to answer these prompts. The script hangs until it times out. The BatchMode=yes option disables all interactive prompting: if authentication requires input, ssh fails immediately instead of waiting . The ConnectTimeout option bounds how long connection establishment can take, so an unreachable host does not stall the loop .
The exit-status ambiguity problem. ssh has a single special exit code: 255 means the connection failed for any reason (DNS resolution, TCP connection, authentication, host key mismatch). Any other exit code is the exit status of the remote command . A loop that reports “host failed” for every non-zero status will misreport a command that returned 1 as an unreachable host. The script must check for 255 first, then interpret the remaining codes as command results.
The reconnection overhead problem. A loop that runs three commands on the same host via three separate ssh invocations performs three TCP handshakes, three authentication exchanges, and three host-key verifications. On a high-latency link, this overhead dominates the actual work. The ControlMaster feature of OpenSSH allows the first connection to become a master, and subsequent connections to reuse it through a Unix socket . The handshake happens once; the remaining commands are sent over the existing channel.
The secret handling problem. A script that connects to remote hosts needs a credential. If that credential is a password embedded in the script, it is visible to anyone who can read the script, and it is likely to be committed to version control. Key-based authentication with an agent or a dedicated key file is the standard solution. If a secret must be passed to a remote command, piping it over the SSH channel is safer than embedding it in the command line, because the command line is visible in the process list .
a. Key-based authentication and batch mode
The prerequisite for an unattended SSH loop is key-based authentication. The script’s user must have a private key, and the corresponding public key must be in the authorized_keys file of the target user on every host.
# Generate a key pair if one does not exist
ssh-keygen -t ed25519 -f ~/.ssh/automation_key -N ""
# Copy the public key to each target host
ssh-copy-id -i ~/.ssh/automation_key.pub user@host1
ssh-copy-id -i ~/.ssh/automation_key.pub user@host2The -N "" flag creates the key without a passphrase, which is what allows the script to use it without an agent. A passphrase-protected key requires ssh-agent or keychain to unlock it, which adds a dependency the script must manage.
Once the key is in place, the script invokes ssh with options that make it safe for batch execution:
ssh -o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new \
-i ~/.ssh/automation_key user@host1 'uptime'BatchMode=yes prevents any prompt. ConnectTimeout=10 fails after ten seconds if the host is unreachable. StrictHostKeyChecking=accept-new accepts a host key on first connection but rejects a changed key on subsequent connections, which is the right balance for automation .
b. The for loop and while read patterns
The simplest loop iterates over an array of hostnames:
#!/bin/bash
set -euo pipefail
hosts=("web01" "web02" "db01")
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new)
for host in "${hosts[@]}"; do
echo "=== $host ==="
if ssh "${SSH_OPTS[@]}" "user@$host" 'hostname && uptime'; then
echo "OK: $host"
else
status=$?
if [[ $status -eq 255 ]]; then
echo "CONNECTION FAILED: $host" >&2
else
echo "COMMAND FAILED: $host (status $status)" >&2
fi
fi
doneThe if ssh ...; then construct captures the exit status. A zero status means both the connection and the command succeeded. A non-zero status requires the 255 check to distinguish the two failure modes .
For a host list stored in a file, the while read pattern is more appropriate:
while IFS= read -r host; do
[[ -z "$host" || "$host" == \#* ]] && continue
ssh "${SSH_OPTS[@]}" "user@$host" 'command' </dev/null
done < hosts.txtThe </dev/null redirect is important. Without it, ssh may consume the input that the while read loop is reading from, causing the loop to skip hosts . The IFS= read -r preserves leading and trailing whitespace in the hostname, and the comment check skips blank lines and comments.
c. Distinguishing connection failure from command failure
The 255 convention is the only reliable way to distinguish a transport failure from a command failure. A script that needs to report the actual remote exit status must capture it before any other command overwrites $?.
output=$(ssh "${SSH_OPTS[@]}" "user@$host" 'some_command' 2>&1)
status=$?
if [[ $status -eq 255 ]]; then
echo "SSH connection to $host failed" >&2
elif [[ $status -ne 0 ]]; then
echo "Command failed on $host with status $status" >&2
else
echo "Command succeeded on $host: $output"
fiIf the remote command itself can exit with 255, the script cannot use this convention. In that case, the remote command must print its own exit status to stdout, and the script must parse it from the output . This is rare, but it is the only way to handle the collision.
d. Connection multiplexing with ControlMaster
For a loop that runs multiple commands against the same host, ControlMaster eliminates the repeated handshake. The first connection creates a master socket, and subsequent connections reuse it.
#!/bin/bash
set -euo pipefail
host="web01"
socket_dir=$(mktemp -d)
control_socket="$socket_dir/ssh-%C"
cleanup() {
ssh -o ControlPath="$control_socket" -O exit "user@$host" 2>/dev/null || true
rm -rf "$socket_dir"
}
trap cleanup EXIT
# Open the master connection in the background
ssh -o ControlMaster=yes -o ControlPath="$control_socket" \
-o ControlPersist=60 -o BatchMode=yes \
-fN "user@$host"
# Verify the master is alive
ssh -o ControlPath="$control_socket" -O check "user@$host"
# Commands reuse the master connection
ssh -o ControlPath="$control_socket" "user@$host" 'command1'
ssh -o ControlPath="$control_socket" "user@$host" 'command2'The -fN flags start the master in the background (-f) without running a remote command (-N). The ControlPersist=60 keeps the master alive for 60 seconds after the last client disconnects. The trap cleanup EXIT ensures the master is terminated and the socket directory removed even if the script is interrupted .
The %C in the ControlPath is a hash of the connection parameters, which keeps the socket name short and avoids the Unix socket path length limit . The socket is placed in a mktemp -d directory rather than a predictable /tmp path, which prevents another user from hijacking the socket.
For a loop over multiple hosts, the pattern becomes a function that opens a master per host, runs the commands, and closes the master before moving to the next host.
Complete Example Session
# ============================================
# PART 1: KEY-BASED AUTHENTICATION
# ============================================
# Generate a dedicated key for automation
ssh-keygen -t ed25519 -f ~/.ssh/automation_key -N ""
# Copy the public key to each target
ssh-copy-id -i ~/.ssh/automation_key.pub user@web01
ssh-copy-id -i ~/.ssh/automation_key.pub user@web02# ============================================
# PART 2: BATCH MODE OPTIONS
# ============================================
SSH_OPTS=(
-o BatchMode=yes
-o ConnectTimeout=10
-o StrictHostKeyChecking=accept-new
-i ~/.ssh/automation_key
)
# BatchMode prevents prompts.
# ConnectTimeout bounds the wait.# ============================================
# PART 3: THE BASIC LOOP
# ============================================
hosts=("web01" "web02" "db01")
for host in "${hosts[@]}"; do
echo "=== $host ==="
ssh "${SSH_OPTS[@]}" "user@$host" 'hostname && uptime'
done# ============================================
# PART 4: DISTINGUISHING FAILURE TYPES
# ============================================
for host in "${hosts[@]}"; do
if ssh "${SSH_OPTS[@]}" "user@$host" 'true' 2>/dev/null; then
echo "OK: $host"
else
status=$?
if [[ $status -eq 255 ]]; then
echo "CONNECTION FAILED: $host" >&2
else
echo "COMMAND FAILED: $host (status $status)" >&2
fi
fi
done# ============================================
# PART 5: WHILE READ OVER A HOST FILE
# ============================================
while IFS= read -r host; do
[[ -z "$host" || "$host" == \#* ]] && continue
ssh "${SSH_OPTS[@]}" "user@$host" 'command' </dev/null
done < hosts.txt
# </dev/null prevents ssh from consuming the host list.# ============================================
# PART 6: CAPTURING OUTPUT AND STATUS
# ============================================
output=$(ssh "${SSH_OPTS[@]}" "user@$host" 'some_command' 2>&1)
status=$?
if [[ $status -eq 255 ]]; then
echo "Connection failed" >&2
elif [[ $status -ne 0 ]]; then
echo "Command failed: $status" >&2
else
echo "Success: $output"
fi# ============================================
# PART 7: CONTROLMASTER SETUP
# ============================================
socket_dir=$(mktemp -d)
control_socket="$socket_dir/ssh-%C"
ssh -o ControlMaster=yes -o ControlPath="$control_socket" \
-o ControlPersist=60 -o BatchMode=yes \
-fN "user@web01"# ============================================
# PART 8: REUSING THE MASTER
# ============================================
ssh -o ControlPath="$control_socket" "user@web01" 'command1'
ssh -o ControlPath="$control_socket" "user@web01" 'command2'
# Both reuse the master connection.
# No new handshake for command2.# ============================================
# PART 9: CLEANUP WITH TRAP
# ============================================
cleanup() {
ssh -o ControlPath="$control_socket" -O exit "user@web01" 2>/dev/null || true
rm -rf "$socket_dir"
}
trap cleanup EXIT
# The master is terminated and the socket removed
# even if the script fails or is interrupted.# ============================================
# PART 10: COMPLETE SCRIPT
# ============================================
#!/bin/bash
set -euo pipefail
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new)
hosts=("web01" "web02")
for host in "${hosts[@]}"; do
if ssh "${SSH_OPTS[@]}" "user@$host" 'uptime' </dev/null; then
echo "OK: $host"
else
status=$?
[[ $status -eq 255 ]] && echo "CONN FAIL: $host" >&2 || echo "CMD FAIL: $host ($status)" >&2
fi
doneThe ten parts covered key generation, batch options, the basic loop, failure distinction, while read, output capture, ControlMaster setup, reuse, cleanup, and the complete script.
Quick Reference
SSH options for automation
| Option | Purpose |
|---|---|
BatchMode=yes | Disable all interactive prompts |
ConnectTimeout=10 | Fail after 10 seconds if unreachable |
StrictHostKeyChecking=accept-new | Accept first key, reject changes |
-i keyfile | Specify the identity file |
ControlMaster=yes | Create a master connection |
ControlPath=socket | Shared socket path |
ControlPersist=60 | Keep master alive for 60 seconds |
-fN | Background, no remote command |
Exit status convention
| Status | Meaning |
|---|---|
| 0 | Remote command succeeded |
| 1–254 | Remote command failed with that status |
| 255 | SSH connection failed (any reason) |
Loop patterns
| Pattern | Use case |
|---|---|
for host in "${hosts[@]}" | Inline host array |
while IFS= read -r host; do ... done < file | Host file |
ssh ... </dev/null | Prevent SSH from consuming stdin |
if ssh ...; then ... else ... fi | Capture status |
ControlMaster lifecycle
| Step | Command |
|---|---|
| Open | ssh -o ControlMaster=yes -fN host |
| Verify | ssh -O check host |
| Reuse | ssh -o ControlPath=socket host cmd |
| Close | ssh -O exit host |
Best Practices
✅ Do This:
# Use BatchMode to prevent hangs
ssh -o BatchMode=yes host command # ✅# Check for 255 separately
if [[ $? -eq 255 ]]; then echo "connection failed"; fi # ✅# Redirect stdin from /dev/null in loops
ssh host command </dev/null # ✅# Use a trap to clean up ControlMaster sockets
trap 'ssh -O exit host; rm -rf "$socket_dir"' EXIT # ✅❌ Don’t Do This:
# Embed passwords in the script
sshpass -p 'secret' ssh host command # ❌# Treat every non-zero status as connection failure
if ! ssh host cmd; then echo "host down"; fi # ❌ (command may have failed)# Forget </dev/null in while read loops
while read host; do ssh host cmd; done < hosts.txt # ❌ (skips hosts)# Use a predictable ControlPath in /tmp
ControlPath=/tmp/ssh-%r@%h:%p # ❌ (hijack risk)Common Pitfalls
| Pitfall | Why It Happens | Fix |
|---|---|---|
| Script hangs waiting for password | BatchMode not set | Add -o BatchMode=yes |
| Host appears unreachable when command failed | 255 not checked separately | Check [[ $? -eq 255 ]] |
| Loop skips hosts | ssh consumed stdin | Add </dev/null |
| Host key prompt hangs script | StrictHostKeyChecking not set | Use accept-new |
| ControlMaster socket left behind | No cleanup trap | Add trap ... EXIT with -O exit |
| Permission denied on key | Key file permissions too open | chmod 600 ~/.ssh/automation_key |
Real-World Examples
1. Basic loop
for host in web01 web02; do ssh "$host" 'uptime'; done2. Batch mode options
ssh -o BatchMode=yes -o ConnectTimeout=10 user@host 'command'3. Status check
ssh host 'cmd'; status=$?; [[ $status -eq 255 ]] && echo "conn fail"4. While read
while read -r host; do ssh "$host" 'cmd' </dev/null; done < hosts.txt5. Output capture
output=$(ssh host 'cmd' 2>&1); status=$?6. ControlMaster open
ssh -o ControlMaster=yes -o ControlPath=/tmp/sock -fN host7. ControlMaster reuse
ssh -o ControlPath=/tmp/sock host 'cmd'8. ControlMaster close
ssh -o ControlPath=/tmp/sock -O exit host9. Key with no passphrase
ssh-keygen -t ed25519 -f ~/.ssh/auto -N ""10. Trap cleanup
trap 'ssh -O exit host 2>/dev/null; rm -rf "$dir"' EXITVisual
The SSH loop flow
┌─────────────────────────────────────────────────────────────┐
│ SSH LOOP │
│ │
│ for host in hosts: │
│ │ │
│ ▼ │
│ ssh -o BatchMode=yes -o ConnectTimeout=10 host cmd │
│ │ │
│ ├── exit 0 ──► command succeeded │
│ │ │
│ ├── exit 1-254 ──► command failed (remote status) │
│ │ │
│ └── exit 255 ──► connection failed │
│ │
│ The three outcomes require different handling. │
└─────────────────────────────────────────────────────────────┘ControlMaster multiplexing
┌─────────────────────────────────────────────────────────────┐
│ WITHOUT CONTROLMASTER │
│ │
│ ssh host cmd1 ──► TCP handshake ──► auth ──► cmd1 │
│ ssh host cmd2 ──► TCP handshake ──► auth ──► cmd2 │
│ ssh host cmd3 ──► TCP handshake ──► auth ──► cmd3 │
│ │
│ 3 handshakes, 3 auths. │
│ │
│ ───────────────────────────────────────────────────────────│
│ │
│ WITH CONTROLMASTER │
│ │
│ ssh -fN host ──► TCP handshake ──► auth ──► master socket │
│ ssh -o ControlPath=sock host cmd1 ──► reuse ──► cmd1 │
│ ssh -o ControlPath=sock host cmd2 ──► reuse ──► cmd2 │
│ │
│ 1 handshake, 1 auth. │
└─────────────────────────────────────────────────────────────┘Exit status decision
┌─────────────────────────────────────────────────────────────┐
│ EXIT STATUS │
│ │
│ status=$? │
│ │ │
│ ├── 0 ──► success │
│ │ │
│ ├── 255 ──► connection failure │
│ │ (DNS, TCP, auth, host key) │
│ │ │
│ └── other ──► remote command failure │
│ (the command's own exit code) │
└─────────────────────────────────────────────────────────────┘ControlMaster lifecycle
┌─────────────────────────────────────────────────────────────┐
│ CONTROLMASTER LIFECYCLE │
│ │
│ mktemp -d ──► socket_dir │
│ │ │
│ ▼ │
│ ssh -fN -o ControlMaster=yes -o ControlPath=sock host │
│ │ │
│ ▼ │
│ Master running in background │
│ │ │
│ ├── ssh -O check host ──► verify │
│ │ │
│ ├── ssh -o ControlPath=sock host cmd ──► reuse │
│ │ │
│ └── trap EXIT ──► ssh -O exit host ──► terminate │
│ │ │
│ ▼ │
│ rm -rf socket_dir │
└─────────────────────────────────────────────────────────────┘Summary
| Item | Value |
|---|---|
| Prompt prevention | BatchMode=yes |
| Connection timeout | ConnectTimeout=10 |
| Host key policy | StrictHostKeyChecking=accept-new |
| Connection failure code | 255 |
| Command failure code | 1–254 (remote command’s status) |
| Stdin protection | </dev/null in loops |
| Multiplexing | ControlMaster + ControlPath |
| Master open | ssh -fN -o ControlMaster=yes |
| Master close | ssh -O exit |
| Cleanup | trap ... EXIT |
Key takeaways:
BatchMode=yesis mandatory for unattended loops. Without it,sshprompts for a password or host-key confirmation, and the script hangs until it times out.ConnectTimeoutbounds the wait for an unreachable host .- Exit code 255 is the only connection-failure indicator. Any other non-zero status is the remote command’s own exit code. A loop that reports “host down” for a command that returned 1 is misdiagnosing the failure .
</dev/nullprevents SSH from consuming the host list. In awhile readloop,sshmay read from the same stdin that the loop uses, causing hosts to be skipped .ControlMastereliminates repeated handshakes. For multiple commands against the same host, the first connection becomes a master and subsequent connections reuse it. The socket must be cleaned up with atrapto prevent orphaned masters .- Keys should be dedicated and unprotected only when necessary. A passphrase-protected key requires
ssh-agent. An unprotected key is simpler for automation but must be stored with restrictive permissions and not committed to a repository. - Secrets should not be embedded in the script. If a secret must be passed to a remote command, pipe it over the SSH channel rather than putting it on the command line, where it is visible in the process list .
Remember: An SSH loop is a reliability problem disguised as a simple for construct. The loop must authenticate without a terminal, handle unreachable hosts without hanging, distinguish connection failures from command failures, and avoid re-negotiating the connection for every command. BatchMode, ConnectTimeout, the 255 check, </dev/null, and ControlMaster are the five mechanisms that make the difference between a loop that works and a loop that fails silently or hangs indefinitely. Get the exit-status handling right first: a script that cannot tell why a host failed cannot report what happened.
Stop using slow, ad-bloated tool sites! 🤮
🔎 Search “KandZ Tools” on Google to use many professional utilities for free.
KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)
⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free
🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!