Bash Scripting 17 🐧 Pipes & Process Substitution — Chaining commands and temporary named pipes - KandZ

⏱️ Estimated read: 14 min | |

A pipe connects the standard output of one command to the standard input of another. The shell creates the connection in the kernel, and the two commands run concurrently: the writer produces data, and the reader consumes it as it arrives. The previous chapter covered redirection, which connects a command’s streams to files; this chapter covers the pipe, which connects a command’s streams to another command’s streams, and the process substitution, which exposes the output of a command as a filename argument.

Quick How-To Guides & Fast FixesNeed a quick fix or step-by-step solution? Browse our fast answers on HowTo.KandZ.me:

Operating Systems CLI →Dev Environment Languages →Web Network Tools →Security Accounts →Security Access Tech →Device Phone Basics →Social Messaging →Digital Files Media →

The pipe is the mechanism that makes the Unix philosophy practical. A command that does one thing well can be combined with another command that does a different thing well, and the combination is more capable than either command alone. The ls | grep x example is the canonical illustration: ls lists the files, and grep x filters the lines that contain the letter x . The | operator is the syntax, and the kernel’s pipe buffer is the mechanism. The process substitution is the extension: the <(command) form exposes the command’s output as a filename, which allows a command that expects a file argument to read from a command’s output .

Key point: A pipeline runs each command in a separate subshell, and the exit status of the pipeline is the exit status of the last command unless set -o pipefail is enabled . The | operator connects stdout to stdin, but it does not connect stderr; the errors are printed on the terminal and do not flow through the pipe unless the 2>&1 redirection is added . The <(command) process substitution creates a temporary file descriptor or a FIFO and passes the filename to the command, which allows diff <(cmd1) <(cmd2) to compare two command outputs without intermediate files .

Why pipes and process substitution require understanding the subshell

The subshell problem.

Each command in a pipeline runs in its own subshell, which is a separate process. A variable that is assigned inside a pipeline is not visible outside the pipeline, because the assignment happened in the subshell . The while read loop that is fed by a pipe runs in a subshell, and the variables it modifies are lost when the pipeline ends. The process substitution is the alternative that keeps the loop in the current shell, because the <(command) form is an argument, not a pipe.

The exit-status problem.

The default exit status of a pipeline is the status of its last command. A pipeline that fails in the first command but succeeds in the last command returns zero, which makes the failure invisible to the script’s error handling. The set -o pipefail option changes this: the pipeline returns the status of the last command that failed, which is the correct behavior for a script that checks the status . The PIPESTATUS array is the alternative that records the status of every command in the pipeline.

The stderr problem.

A pipe connects stdout, not stderr. A command that writes its errors to stderr does not send them through the pipe, and the downstream command does not see them. The errors appear on the terminal, and the pipeline continues. The 2>&1 redirection merges stderr into stdout before the pipe, which is the mechanism for sending both streams through the pipeline .

a. The pipe operator and the pipeline status

The | operator connects the stdout of the left command to the stdin of the right command. The commands run concurrently, and the shell waits for all of them to finish.

#!/usr/bin/env bash
# List the files and filter the lines
ls -l | grep ".txt"

# Count the lines in the output
cat /etc/passwd | wc -l

# Sort the output and remove the duplicates
cat file.txt | sort | uniq

The ls -l | grep ".txt" pipeline sends the output of ls to grep, which prints only the lines that contain .txt . The cat /etc/passwd | wc -l pipeline counts the lines in the password file. The cat file.txt | sort | uniq pipeline sorts the lines and removes the duplicates.

The |& operator is the shorthand for 2>&1 |, which merges the stderr into the stdout before the pipe.

# Send both stdout and stderr through the pipe
command_with_errors |& grep "error"

The |& operator is the Bash 4 extension, and it is equivalent to 2>&1 | . The pipe receives both the normal output and the error messages, and the downstream command can filter them together.

The pipeline status is the status of the last command by default. The pipefail option changes the behavior.

# Without pipefail: the status is the last command's status
false | true
echo "$?"    # 0

# With pipefail: the status is the first failing command's status
set -o pipefail
false | true
echo "$?"    # 1

The set -o pipefail option makes the pipeline return the status of the last command that failed, which is the correct behavior for a script that checks the status . The PIPESTATUS array is the alternative that records the status of every command in the pipeline.

# Check the status of each command
false | true | false
echo "${PIPESTATUS[@]}"    # 1 0 1

The PIPESTATUS array is reset after every pipeline, and it must be captured immediately after the pipeline finishes.

b. The named pipe and the FIFO

A named pipe, also called a FIFO, is a special file that exists in the filesystem and behaves like a pipe. The mkfifo command creates it, and two unrelated processes can communicate through it by opening the file for reading and writing .

# Create a named pipe
mkfifo /tmp/mypipe

# Terminal 1: write to the pipe
ls -l > /tmp/mypipe

# Terminal 2: read from the pipe
cat < /tmp/mypipe

The mkfifo /tmp/mypipe command creates the FIFO. The ls -l > /tmp/mypipe command writes to the pipe, and the cat < /tmp/mypipe command reads from it . The writer blocks until a reader opens the pipe, and the reader blocks until a writer opens it. The FIFO is the mechanism that allows two processes that were not designed to work together to communicate through a named file.

The FIFO is useful when the pipeline is not sufficient, such as when the two processes need to run in different terminals or when the intermediate data needs to be available to a process that is not in the pipeline.

# Use a FIFO as an intermediate buffer
mkfifo /tmp/buffer
tar -czf - /home/user > /tmp/buffer &
gzip -d < /tmp/buffer | tar -xf - -C /backup

The tar command writes the compressed archive to the FIFO, and the gzip and tar commands read from it. The & runs the writer in the background, which allows the reader to open the FIFO . The FIFO is the intermediate buffer that decouples the two processes.

The FIFO should be removed when it is no longer needed, and the trap command can ensure that the cleanup happens even if the script exits early.

# Clean up the FIFO on exit
trap 'rm -f /tmp/buffer' EXIT

The trap command runs the rm when the script exits, which prevents the FIFO from being left behind .

c. The process substitution

The process substitution exposes the output of a command as a filename. The <(command) form is for reading, and the >(command) form is for writing .

#!/usr/bin/env bash
# Compare the output of two commands
diff <(ls /dir1) <(ls /dir2)

# Count the words in a command's output
wc -l <(grep "pattern" file.txt)

# Feed a command's output to a loop without a subshell
while IFS= read -r line; do
    echo "Line: $line"
done < <(grep "pattern" file.txt)

The diff <(ls /dir1) <(ls /dir2) command compares the output of the two ls commands without intermediate files . The wc -l <(grep ...) command counts the lines in the grep output. The while ... done < <(grep ...) pattern feeds the loop from the process substitution, and the loop runs in the current shell, not a subshell . The <(command) form is the mechanism that keeps the loop’s variable assignments visible after the loop ends.

The >(command) form is the write direction: the command’s stdin is connected to the file, and the file is the argument.

# Compress the output of tar without an intermediate file
tar cf >(bzip2 -c > archive.tar.bz2) /home/user

# Tee the output to two commands
command > >(tee >(gzip > out.gz) >(wc -l > count.txt))

The tar cf >(bzip2 -c > archive.tar.bz2) command writes the tar archive to the process substitution, and the bzip2 command compresses it . The tee example splits the output into two commands, one that compresses and one that counts, without an intermediate file.

The process substitution requires a space between the < or > and the (. The < (command) form is a syntax error, and the <(command) form is the correct one . The process substitution is supported on systems that have the /dev/fd filesystem or the named pipes, and it is not a POSIX feature, so a script that targets /bin/sh cannot use it.

Complete Example Session

#!/usr/bin/env bash
# ============================================
# PART 1: THE BASIC PIPE
# ============================================
ls -l | grep ".txt"
# Output: the lines that contain .txt

# ============================================
# PART 2: THE PIPELINE STATUS
# ============================================
false | true
echo "Status: $?"    # 0

set -o pipefail
false | true
echo "Status: $?"    # 1

# ============================================
# PART 3: THE PIPESTATUS ARRAY
# ============================================
false | true | false
echo "Statuses: ${PIPESTATUS[@]}"    # 1 0 1

# ============================================
# PART 4: THE STDERR PIPE
# ============================================
ls /nonexistent |& grep "No such file"
# Output: the error message

# ============================================
# PART 5: THE NAMED PIPE
# ============================================
mkfifo /tmp/mypipe
ls -l > /tmp/mypipe &
cat < /tmp/mypipe
rm /tmp/mypipe

# ============================================
# PART 6: THE PROCESS SUBSTITUTION FOR DIFF
# ============================================
diff <(ls /etc) <(ls /usr/etc)
# Output: the difference between the two directory listings

# ============================================
# PART 7: THE PROCESS SUBSTITUTION FOR A LOOP
# ============================================
while IFS= read -r line; do
    echo "Line: $line"
done < <(grep "root" /etc/passwd)

# ============================================
# PART 8: THE PROCESS SUBSTITUTION FOR TAR
# ============================================
tar cf >(bzip2 -c > /tmp/archive.tar.bz2) /etc/hostname
ls -l /tmp/archive.tar.bz2

# ============================================
# PART 9: THE CLEANUP TRAP
# ============================================
trap 'rm -f /tmp/mypipe' EXIT
mkfifo /tmp/mypipe
# ... use the pipe ...
# The trap removes it on exit

The nine parts demonstrate the pipe and process substitution surface: the basic pipe, the pipeline status, the PIPESTATUS array, the stderr pipe, the named pipe, the process substitution for diff, the process substitution for a loop, the process substitution for tar, and the cleanup trap. The critical distinction is between Parts 2 and 3: the pipefail option changes the pipeline’s status, and the PIPESTATUS array records the status of every command.

Quick Reference

Pipe Operators

Operator	Purpose	Example
`\|`	Connect stdout to stdin	`cmd1 \| cmd2`
`\|&`	Connect stdout and stderr to stdin	`cmd1 \|& cmd2`
`2>&1 \|`	The portable form of `\|&`	`cmd1 2>&1 \| cmd2`

Pipeline Status

Option/Array	Purpose
Default	Status of the last command
`set -o pipefail`	Status of the first failing command
`${PIPESTATUS[@]}`	Status of every command

Named Pipe Commands

Command	Purpose
`mkfifo <name>`	Create a named pipe
`rm <name>`	Remove the named pipe
`trap 'rm -f <name>' EXIT`	Clean up on exit

Process Substitution

Form	Purpose
`<(command)`	Read from the command’s output
`>(command)`	Write to the command’s input
`diff <(cmd1) <(cmd2)`	Compare two command outputs
`while ... done < <(cmd)`	Loop without a subshell

Best Practices

✅ Do This:

# Use pipefail in scripts that check the status
set -o pipefail                                               # ✅

# Use the process substitution for the loop
while IFS= read -r line; do ...; done < <(grep ...)           # ✅

# Use the named pipe for unrelated processes
mkfifo /tmp/pipe; writer > /tmp/pipe & reader < /tmp/pipe     # ✅

# Clean up the named pipe on exit
trap 'rm -f /tmp/pipe' EXIT                                   # ✅

# Use the |& for both streams
command |& grep "error"                                       # ✅

# Use the process substitution for diff
diff <(ls dir1) <(ls dir2)                                    # ✅

❌ Don’t Do This:

# Use a pipe for a loop that modifies variables
cat file | while read line; do count=$((count+1)); done
# count is lost after the pipeline                              # ❌

# Forget the space in the process substitution
diff <(ls dir1) < (ls dir2)    # Syntax error                  # ❌

# Use the process substitution in a POSIX script
#!/bin/sh
diff <(ls dir1) <(ls dir2)    # Not supported                 # ❌

# Forget to remove the named pipe
mkfifo /tmp/pipe    # Left behind                              # ❌

# Assume the pipeline status is the first command's status
cmd1 | cmd2    # The status is cmd2's status                 # ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
Variable lost after the loop	The loop ran in a subshell	Use `< <(command)`
Pipeline failure not detected	Default status is the last command	Use `set -o pipefail`
stderr not piped	The pipe connects stdout only	Use `2>&1 \|` or `\|&`
Process substitution syntax error	Space between `<` and `(`	Use `<(` with no space
Named pipe blocks	The reader has not opened the pipe	Start the reader first or use `&`

Real-World Examples

1. Basic pipe

ls -l | grep ".txt"

2. Pipe with both streams

command |& grep "error"

3. Pipefail

set -o pipefail; false | true; echo "$?"

4. PIPESTATUS

false | true | false; echo "${PIPESTATUS[@]}"

5. Named pipe

mkfifo /tmp/pipe; ls > /tmp/pipe & cat < /tmp/pipe

6. Process substitution for diff

diff <(ls dir1) <(ls dir2)

7. Process substitution for a loop

while IFS= read -r line; do echo "$line"; done < <(grep "root" /etc/passwd)

8. Process substitution for tar

tar cf >(bzip2 -c > archive.tar.bz2) /home/user

9. Cleanup trap

trap 'rm -f /tmp/pipe' EXIT

10. Compare command outputs

diff <(sort file1) <(sort file2)

Visual

The Pipe and the Subshell

┌──────────────────────────────────────────────────────────────┐
│  EACH COMMAND IN A PIPELINE RUNS IN ITS OWN SUBSHELL         │
│                                                              │
│  cmd1 | cmd2 | cmd3                                          │
│       │       │       │                                      │
│       ▼       ▼       ▼                                      │
│  ┌─────────┐ ┌─────────┐ ┌─────────┐                         │
│  │ cmd1    │ │ cmd2    │ │ cmd3    │                         │
│  │ subshell│ │ subshell│ │ subshell│                         │
│  └────┬────┘ └────┬────┘ └────┬────┘                         │
│       │           │           │                              │
│       └───────────┴───────────┘                              │
│              stdout → stdin                                  │
│                                                              │
│  A variable modified in cmd2 is not visible after the        │
│  pipeline finishes.                                          │
└──────────────────────────────────────────────────────────────┘

The Process Substitution

┌──────────────────────────────────────────────────────────────┐
│  <(command) EXPOSES THE OUTPUT AS A FILENAME                 │
│                                                              │
│  diff <(ls dir1) <(ls dir2)                                  │
│       │             │                                        │
│       ▼             ▼                                        │
│  ┌─────────┐   ┌─────────┐                                   │
│  │ ls dir1 │   │ ls dir2 │                                   │
│  └────┬────┘   └────┬────┘                                   │
│       │             │                                        │
│       ▼             ▼                                        │
│  /dev/fd/63    /dev/fd/62                                    │
│       │             │                                        │
│       └──────┬──────┘                                        │
│              ▼                                               │
│  diff compares the two file descriptors                      │
│                                                              │
│  No intermediate files are created.                          │
└──────────────────────────────────────────────────────────────┘

The Named Pipe

┌──────────────────────────────────────────────────────────────┐
│  mkfifo CREATES A FILE THAT BEHAVES LIKE A PIPE              │
│                                                              │
│  Terminal 1:                                                 │
│  ┌────────────────────────────────────────────────────────┐  │
│  │ ls -l > /tmp/pipe                                      │  │
│  │ → blocks until a reader opens the pipe                 │  │
│  └────────────────────────────────────────────────────────┘  │
│                                                              │
│  Terminal 2:                                                 │
│  ┌────────────────────────────────────────────────────────┐  │
│  │ cat < /tmp/pipe                                        │  │
│  │ → reads the output of ls                               │  │
│  └────────────────────────────────────────────────────────┘  │
│                                                              │
│  The pipe is a file in the filesystem, and the two           │
│  processes do not need to be related.                        │
└──────────────────────────────────────────────────────────────┘

Summary

Item	Value
Pipe operator	`\|`
Both-stream pipe	`\|&` or `2>&1 \|`
Pipeline status	Last command by default
Pipefail	`set -o pipefail`
PIPESTATUS	Array of every command’s status
Named pipe	`mkfifo <name>`
Process substitution (read)	`<(command)`
Process substitution (write)	`>(command)`
Loop without subshell	`while ... done < <(command)`
Cleanup	`trap 'rm -f <name>' EXIT`

Key takeaways:

A pipe connects the stdout of one command to the stdin of another. The commands run concurrently, and each one runs in its own subshell. The | operator is the syntax, and the kernel’s pipe buffer is the mechanism .
The pipeline’s exit status is the last command’s status by default. The set -o pipefail option changes it to the first failing command’s status, and the PIPESTATUS array records the status of every command .
A pipe does not connect stderr. The errors are printed on the terminal and do not flow through the pipe unless the 2>&1 redirection is added. The |& operator is the shorthand for the 2>&1 | form .
A named pipe is a file that behaves like a pipe. The mkfifo command creates it, and two unrelated processes can communicate through it by opening it for reading and writing. The writer blocks until a reader opens the pipe, and the reader blocks until a writer opens it .
The process substitution exposes a command’s output as a filename. The <(command) form is for reading, and the >(command) form is for writing. The diff <(cmd1) <(cmd2) pattern compares two command outputs without intermediate files .
The process substitution keeps a loop in the current shell. The while ... done < <(command) pattern feeds the loop from the process substitution, and the variables that the loop modifies remain visible after the loop ends .

Remember: Pipes and process substitution are the mechanisms that connect commands. The | operator sends stdout to stdin, and the |& operator sends both streams. The pipefail option makes the pipeline’s status meaningful, and the PIPESTATUS array records every command’s status. The mkfifo command creates a named pipe for unrelated processes. The <(command) and >(command) forms expose a command’s output as a filename, which allows diff, while, and tar to work with command output without intermediate files. The result is a script that composes small commands into a workflow.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!