LFCS 26 🐧 Pipes — Combining Commands
Every command on Linux has three streams: the standard input, the standard output, and the standard error. The pipe is the operator that connects the standard output of one command to the standard input of the next. The ls -l | grep '.txt' takes the listing from the ls, feeds it to the grep, and the grep filters the lines. The two commands run concurrently, the kernel buffers the data between them, and the result is the composition. The pipe is the fundamental Unix composition primitive, and the LFCS exam tests it as part of the “Use streams, pipes, and redirects” objective. The understanding of the pipe is the prerequisite for the text-processing pipelines, the log analysis, and the shell scripting.
The pipe has the specific behavior. It creates the unidirectional channel between the two processes. The left command writes to the pipe, and the right command reads from it. The two run in the separate processes, and the kernel schedules them. The data flows through the kernel buffer, and the two commands see each other only through the stream. The pipe is the anonymous, and it disappears when the commands finish. The named pipe, the mkfifo, is the persistent and the filesystem-based, and it is the separate topic.
The pipelines compose. The command1 | command2 | command3 chains the three commands, and the data flows from the left to the right. The pipeline’s exit status is the exit status of the last command, unless the pipefail option is set. The set -o pipefail makes the pipeline return the first non-zero exit, which is the important for the scripts. The PIPESTATUS array in the bash holds the exit status of the each command in the pipeline.
This chapter covers three areas. First, why the pipes exist — the composition problem and the Unix philosophy. Second, the pipe syntax and the behavior — the |, the |&, the exit status, and the pipefail. Third, the common pipelines — the grep, the cut, the sort, the uniq, the head, the tail, the wc, the xargs, and the tee. The chapter ends with a complete example session, a quick reference, best practices, common pitfalls, real-world examples, and diagrams showing the pipeline flow.
Key point: The pipe connects the standard output of one command to the standard input of the next. The command1 | command2 runs both concurrently, and the data flows from the left to the right. The pipeline’s exit status is the last command’s, unless the set -o pipefail is set. The PIPESTATUS array holds the exit status of the each command. The pipe is the fundamental Unix composition primitive.
Why the pipes exist
The composition problem. A single command does one thing. The ls lists the files, the grep filters the lines, the sort orders the lines, and the wc counts the lines. Each is the simple, and the combination is the powerful. The pipe is the glue: the output of one becomes the input of the next, and the pipeline is the composition. The Unix philosophy is the small tools that do one thing well, and the pipes are the way they combine.
The streaming problem. The pipe is the streaming, not the batch. The left command writes the data as it produces it, and the right command reads it as it arrives. The two run concurrently, and the memory usage is the bounded by the buffer. The pipeline can process the huge file without loading it into the memory, and the two commands overlap the execution.
The concurrency problem. The two commands in the pipeline run in the separate processes. The kernel schedules them, and the data flows through the kernel buffer. The left command does not wait for the right to finish, and the right does not wait for the left to finish. The pipeline is the concurrent by design.
The exit-status problem. The pipeline’s exit status is the last command’s, by default. The command1 | command2 returns the command2‘s exit status, even if the command1 failed. The set -o pipefail changes this: the pipeline returns the first non-zero exit, and the script can detect the failure. The PIPESTATUS array holds the each command’s status.
The |& problem. The | connects the standard output only. The |& connects both the standard output and the standard error. The command |& grep 'error' filters the both streams, and the command 2>&1 | grep 'error' is the portable equivalent.
The trade-off. The pipe adds the concurrency and the composition, and it hides the intermediate. The debugging is the harder, because the intermediate output is not visible. The tee is the way to inspect the intermediate, and the set -x is the way to trace the commands.
a. The pipe syntax and the behavior
The pipe is the | character. The command1 | command2 runs both concurrently, and the data flows from the left to the right.
ls -l | grep '.txt'
cat file.txt | sort | uniq -c | sort -rn | head -10
ps aux | grep nginx | awk '{print $2}'The ls -l writes the listing to the standard output, and the grep '.txt' reads it from the standard input. The two run in the separate processes, and the kernel buffers the data between them.
The |& pipe. The |& connects both the standard output and the standard error:
command |& grep 'error'
command 2>&1 | grep 'error'The two are the equivalent, and the |& is the bash shorthand.
The exit status. The pipeline’s exit status is the last command’s, by default:
ls /nonexistent | grep 'txt'
echo $? # the grep's exit status, not the ls'sThe ls fails, and the grep runs on the empty input. The pipeline’s exit status is the grep‘s, which is the 1 (the no match).
The pipefail. The set -o pipefail makes the pipeline return the first non-zero exit:
set -o pipefail
ls /nonexistent | grep 'txt'
echo $? # the ls's exit status (the 2)The pipefail is the important for the scripts, because the failed command in the middle is the otherwise hidden.
The PIPESTATUS. The PIPESTATUS array holds the exit status of the each command:
ls /nonexistent | grep 'txt'
echo "${PIPESTATUS[@]}" # the "2 1" (the ls and the grep)The PIPESTATUS is the bash-specific, and the zsh has the pipestatus.
The pipeline in the pipeline. The pipeline can be the part of the larger command, and the tee is the way to branch:
command | tee output.txt | grep 'error'The tee writes to the file and the standard output, and the grep reads the standard output.
b. The common pipeline commands
The grep. The grep filters the lines:
cat file.txt | grep 'pattern'
grep 'pattern' file.txt # the equivalent, the no pipe neededThe grep is the line-oriented, and the pipe is the way to feed the other commands.
The cut. The cut extracts the fields:
cat file.txt | cut -d: -f1
echo 'a:b:c' | cut -d: -f1,3The -d is the delimiter, and the -f is the field.
The sort. The sort orders the lines:
cat file.txt | sort
cat file.txt | sort -n
cat file.txt | sort -rn
cat file.txt | sort -k2The -n is the numeric, the -r is the reverse, and the -k is the key.
The uniq. The uniq removes the adjacent duplicates:
cat file.txt | sort | uniq
cat file.txt | sort | uniq -c
cat file.txt | sort | uniq -dThe uniq requires the sorted input, and the -c counts the occurrences, and the -d shows the duplicates only.
The head and the tail. The head shows the first lines, and the tail shows the last:
cat file.txt | head -10
cat file.txt | tail -10
cat file.txt | tail -f # the followThe -f follows the file, and the -n sets the count.
The wc. The wc counts the lines, the words, and the characters:
cat file.txt | wc -l
cat file.txt | wc -w
cat file.txt | wc -cThe -l is the lines, the -w is the words, and the -c is the bytes.
The xargs. The xargs converts the standard input to the arguments:
find . -name '*.log' | xargs rm
find . -name '*.log' -print0 | xargs -0 rm
echo 'file1 file2 file3' | xargs -n 1 ls -lThe -0 is the null-separated, and the -n is the batch size, and the -I {} is the placeholder.
The tee. The tee writes to the standard output and the file:
command | tee output.txt
command | tee -a output.txt
command | tee output.txt | grep 'error'The -a appends, and the tee is the T-splitter.
The awk. The awk is the field-processing and the programming:
ps aux | awk '{print $2, $11}'
ps aux | awk '$3 > 50 {print $2}'The awk is the field-oriented, and the pipe is the way to feed it.
The sed. The sed is the stream editor:
cat file.txt | sed 's/old/new/g'
cat file.txt | sed -n '/pattern/p'The sed is the substitution and the deletion.
c. The advanced patterns
The pv. The pv monitors the pipeline:
cat file.txt | pv | gzip > file.txt.gzThe pv shows the progress, the throughput, and the ETA.
The sponge. The sponge absorbs the standard input before the write:
grep 'pattern' file.txt | sponge file.txtThe sponge is the moreutils, and it is the way to write to the same file.
The process substitution. The <() and the >() are the process substitution:
diff <(ls dir1) <(ls dir2)
comm <(sort file1) <(sort file2)
tee >(gzip > out.gz) >(wc -l > count.txt) < input.txtThe process substitution is the bash feature, and it is the way to use the command output where the file is expected.
The named pipe. The mkfifo creates the named pipe:
mkfifo mypipe
command > mypipe &
other_command < mypipeThe named pipe is the filesystem-based, and it is the way for the unrelated processes to communicate.
The coproc. The coproc is the bash coprocess:
coproc myproc { command; }
echo 'input' >&"${myproc[1]}"
read -r line <&"${myproc[0]}"The coproc is the advanced, and it is the way for the bidirectional communication.
Complete Example Session
# ============================================
# PART 1: THE BASIC PIPE
# ============================================
ls -l | grep '.txt'
cat file.txt | sort
cat file.txt | wc -l
# ============================================
# PART 2: THE MULTI-STAGE PIPELINE
# ============================================
cat file.txt | grep 'error' | sort | uniq -c | sort -rn | head -10
# The top 10 error messages.
# ============================================
# PART 3: THE EXIT STATUS
# ============================================
ls /nonexistent | grep 'txt'
echo $? # the grep's status, not the ls's
set -o pipefail
ls /nonexistent | grep 'txt'
echo $? # the ls's status (the 2)
# ============================================
# PART 4: THE PIPESTATUS
# ============================================
ls /nonexistent | grep 'txt'
echo "${PIPESTATUS[@]}" # the "2 1"
# ============================================
# PART 5: THE |&
# ============================================
command |& grep 'error'
command 2>&1 | grep 'error'
# ============================================
# PART 6: THE TEE
# ============================================
command | tee output.txt
command | tee output.txt | grep 'error'
command | tee -a output.txt
# ============================================
# PART 7: THE XARGS
# ============================================
find . -name '*.log' | xargs rm
find . -name '*.log' -print0 | xargs -0 rm
echo 'file1 file2' | xargs -n 1 ls -l
# ============================================
# PART 8: THE PROCESS SUBSTITUTION
# ============================================
diff <(ls dir1) <(ls dir2)
comm <(sort file1) <(sort file2)
# ============================================
# PART 9: THE AWK AND THE SED
# ============================================
ps aux | awk '{print $2, $11}'
ps aux | awk '$3 > 50 {print $2}'
cat file.txt | sed 's/old/new/g'
# ============================================
# PART 10: THE LFCS PIPELINE
# ============================================
# The top 10 IP addresses in the access log
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -10
# The usernames with the /bin/bash shell
grep '/bin/bash' /etc/passwd | cut -d: -f1 | sort > /root/bash-users.txt
# The running processes sorted by the memory
ps aux | sort -k4 -rn | head -10
# The files larger than 100M
find / -type f -size +100M 2>/dev/null | xargs ls -lh
# The error count in the log
grep -c 'ERROR' /var/log/app.log
# The live log follow with the filter
tail -f /var/log/app.log | grep --line-buffered 'ERROR'
# The capture and the display
make 2>&1 | tee build.log
# The pipeline with the pipefail
set -o pipefail
grep 'pattern' file.txt | wc -l || echo 'The grep failed'The ten parts show the basic pipe, the multi-stage pipeline, the exit status, the PIPESTATUS, the |&, the tee, the xargs, the process substitution, the awk and the sed, and the LFCS pipeline.
Quick Reference
The Pipe Operators
| The operator | The purpose |
|---|---|
| | The stdout to the next stdin |
|& | The stdout and the stderr to the next stdin |
2>&1 | | The portable |& |
tee | The stdout to the stdout and the file |
The Exit Status
| The setting | The pipeline status |
|---|---|
| The default | The last command’s |
set -o pipefail | The first non-zero |
${PIPESTATUS[@]} | The each command’s |
The Common Commands
| The command | The purpose |
|---|---|
grep | The filter |
cut | The field extraction |
sort | The ordering |
uniq | The adjacent dedup |
head | The first lines |
tail | The last lines |
wc | The count |
awk | The field processing |
sed | The stream editing |
The xargs Options
| The option | The purpose |
|---|---|
-0 | The null-separated |
-n N | The N items per command |
-I {} | The placeholder |
-P N | The parallel |
The Advanced
| The tool | The purpose |
|---|---|
pv | The progress monitor |
sponge | The absorb the stdin |
<() | The process substitution (the read) |
>() | The process substitution (the write) |
mkfifo | The named pipe |
coproc | The coprocess |
Best Practices
✅ Do This:
# Use the pipe for the composition
cat file.txt | grep 'pattern' | sort | uniq -c # ✅
# Use the pipefail in the scripts
set -o pipefail # ✅
# Use the -print0 with the xargs
find . -name '*.log' -print0 | xargs -0 rm # ✅
# Use the tee for the capture
command | tee output.txt | grep 'error' # ✅
# Use the process substitution for the diff
diff <(ls dir1) <(ls dir2) # ✅
# Use the --line-buffered for the live follow
tail -f log | grep --line-buffered 'ERROR' # ✅❌ Don’t Do This:
# Don't parse the ls output
ls | grep 'pattern' # the ls is for the humans // ❌
# Don't forget the pipefail in the scripts
command1 | command2 # the command1 failure is hidden // ❌
# Don't use the xargs without the -0 for the spaces
find . -name '*.log' | xargs rm # the space split // ❌
# Don't use the useless cat
cat file.txt | grep 'pattern' # the grep file.txt is the simpler // ❌
# Don't pipeline the interactive commands
vi file.txt | grep 'pattern' # the vi is the interactive // ❌
# Don't forget the buffer
command | grep 'pattern' # the output may be delayed // ✅ (the grep is the buffered)Common Pitfalls
| The pitfall | The why | The fix |
|---|---|---|
| The failed command is hidden | The default exit status | Use the pipefail |
| The space split in the xargs | The no -0 | Use the -print0 and the -0 |
| The output is delayed | The block buffering | Use the stdbuf or the --line-buffered |
| The interactive command hangs | The pipeline | Do not pipe the interactive |
The uniq does not dedup | The unsorted input | Sort first |
The head causes the SIGPIPE | The early exit | The expected |
The tee overwrites | The no -a | Use the -a |
Real-World Examples
1. The Basic Pipe
ls -l | grep '.txt'2. The Top 10
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -103. The Users
grep '/bin/bash' /etc/passwd | cut -d: -f1 | sort4. The Memory Sort
ps aux | sort -k4 -rn | head -105. The Large Files
find / -type f -size +100M 2>/dev/null | xargs ls -lh6. The Capture
make 2>&1 | tee build.log7. The Live Follow
tail -f app.log | grep --line-buffered 'ERROR'8. The Diff
diff <(ls dir1) <(ls dir2)9. The Parallel xargs
find . -name '*.jpg' -print0 | xargs -0 -P 4 -n 1 convert10. The Complete
set -o pipefail
grep 'pattern' file.txt | wc -lVisual
The Pipeline Flow
┌──────────────────────────────────────────────────────────────┐
│ THE PIPELINE FLOW │
│ │
│ command1 | command2 | command3 │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │command1 │ │command2 │ │command3 │ │
│ │ │ │ │ │ │ │
│ │ the │ │ the │ │ the │ │
│ │ stdout ──┼───►│ stdin │ │ stdin │ │
│ │ │ │ stdout ──┼───►│ stdout │ │
│ │ the │ │ │ │ │ │
│ │ stderr ──┼──► │ the │ │ the │ │
│ │ (the │ │ stderr ──┼──► │ stderr │ │
│ │ terminal)│ │ (the │ │ (the │ │
│ │ │ │ terminal)│ │ terminal)│ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ │
│ The stdout flows from the left to the right. │
│ The stderr goes to the terminal (unless the |&). │
│ The three commands run concurrently. │
│ │
└──────────────────────────────────────────────────────────────┘The Exit Status
┌──────────────────────────────────────────────────────────────┐
│ THE EXIT STATUS │
│ │
│ THE DEFAULT: │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ command1 | command2 │ │
│ │ → the exit status is the command2's │ │
│ │ │ │
│ │ The command1 failure is the HIDDEN. │ │
│ └──────────────────────────────────────────────────────┘ │
│ │
│ THE PIPEFAIL: │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ set -o pipefail │ │
│ │ command1 | command2 │ │
│ │ → the exit status is the first non-zero │ │
│ │ │ │
│ │ The command1 failure is the VISIBLE. │ │
│ └──────────────────────────────────────────────────────┘ │
│ │
│ THE PIPESTATUS: │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ command1 | command2 │ │
│ │ echo "${PIPESTATUS[@]}" │ │
│ │ → the each command's status │ │
│ └──────────────────────────────────────────────────────┘ │
│ │
└──────────────────────────────────────────────────────────────┘The tee
┌──────────────────────────────────────────────────────────────┐
│ THE tee │
│ │
│ command | tee output.txt | grep 'error' │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │command │ │tee │ │grep │ │
│ │ │ │ │ │ │ │
│ │ the │ │ the │ │ the │ │
│ │ stdout ──┼───►│ stdin │ │ stdin │ │
│ │ │ │ │ │ │ │
│ │ │ │ output ──┼───►│ stdout │ │
│ │ │ │ .txt │ │ │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ │
│ The tee splits the stream: the file and the stdout. │
│ │
└──────────────────────────────────────────────────────────────┘The xargs
┌──────────────────────────────────────────────────────────────┐
│ THE xargs │
│ │
│ find . -name '*.log' -print0 | xargs -0 rm │
│ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │find │ │xargs │ │
│ │ │ │ │ │
│ │ the paths ───────┼───►│ the args ────────┼──► rm │
│ │ (the null- │ │ │ │
│ │ separated) │ │ │ │
│ └──────────────────┘ └──────────────────┘ │
│ │
│ The xargs converts the stdin to the arguments. │
│ The -0 handles the spaces in the file names. │
│ │
└──────────────────────────────────────────────────────────────┘Summary
| Item | Value |
|---|---|
| The pipe | | connects the stdout to the stdin |
The |& | The stdout and the stderr |
| The exit status | The last command’s by default |
The pipefail | The first non-zero |
The PIPESTATUS | The each command’s |
The grep | The filter |
The cut | The field extraction |
The sort | The ordering |
The uniq | The adjacent dedup |
The xargs | The stdin to the args |
The tee | The split the stream |
Key takeaways:
- The pipe connects the stdout of one command to the stdin of the next. The
command1 | command2runs both concurrently, and the data flows from the left to the right. The pipe is the anonymous and the kernel-buffered. - The pipeline’s exit status is the last command’s by default. The
set -o pipefailmakes the pipeline return the first non-zero exit, and the${PIPESTATUS[@]}holds the each command’s status. - The
|&connects both the stdout and the stderr. Thecommand |& grep 'error'filters the both, and the2>&1 |is the portable. - The
grep, thecut, thesort, theuniq, thehead, thetail, and thewcare the common pipeline commands. Theawkand thesedare the field-processing and the stream-editing. - The
xargsconverts the stdin to the arguments. The-0handles the null-separated input, and the-nbatches the items, and the-I {}is the placeholder. - The
teesplits the stream. Thecommand | tee output.txt | grep 'error'captures the output and the continues the pipeline. - The process substitution presents the command output as the file. The
<()and the>()are the bash feature, and they are the way for thediff, thecomm, and thetee. - The pipe is the fundamental Unix composition. The small tools that do the one thing well, and the pipes that combine them. The result is the powerful text-processing pipeline, which is the foundation of the Linux administration.
Remember: The pipe connects the stdout of one command to the stdin of the next. The command1 | command2 runs both concurrently, and the data flows from the left to the right. The pipeline’s exit status is the last command’s, unless the set -o pipefail is set. The result is the composition of the small tools, which is the foundation of the Unix philosophy.
Stop using slow, ad-bloated tool sites! 🤮
🔎 Search “KandZ Tools” on Google to use many professional utilities for free.
KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)
⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free
🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!