Bash Scripting 20 🐧 Text Processing Sed — Stream editing and automated regex substitutions - KandZ

⏱️ Estimated read: 13 min | |

sed is the stream editor: it reads the input line by line, applies the editing commands to each line, and writes the result to the output. The previous chapter covered cut and awk, which extract and process the fields; this chapter covers sed, which transforms the text in place or in a stream. The sed command is the tool for the find-and-replace, the deletion of the matching lines, the insertion of the text before or after a pattern, and the substitution with the captured groups. It is the mechanism that makes the automated text transformation possible without a full programming language.

Quick How-To Guides & Fast FixesNeed a quick fix or step-by-step solution? Browse our fast answers on HowTo.KandZ.me:

Operating Systems CLI →Dev Environment Languages →Web Network Tools →Security Accounts →Security Access Tech →Device Phone Basics →Social Messaging →Digital Files Media →

The sed syntax is sed [options] 'script' [file...]. The script is a sequence of commands, each of which is an address followed by a command letter and the command’s arguments. The s command is the substitution, the d command is the deletion, the p command is the printing, and the a, i, and c commands are the append, insert, and change. The -i option edits the file in place, and the -E or -r option enables the extended regular expressions .

Key point: The substitution command is s/pattern/replacement/flags. The pattern is a basic regular expression unless the -E option is used, and the replacement can contain the & for the whole match and the \1 through \9 for the captured groups. The g flag replaces all the occurrences on the line, and the absence of the g replaces only the first . The -i option edits the file in place, and it is the only way to modify the file without a temporary file; the -i.bak form creates a backup before the edit.

Why sed requires understanding the address and the flags

The address problem.

A sed command can have an address that restricts it to the matching lines. The address can be a line number, a pattern, or a range. The sed '5d' deletes the fifth line, and the sed '/error/d' deletes the lines that contain “error”. The sed '1,5d' deletes the first five lines, and the sed '/start/,/end/d' deletes the lines from the one that matches “start” to the one that matches “end” . The address is the mechanism that applies the command to the specific lines rather than to every line.

The flag problem.

The s command’s flags determine the scope of the substitution. The g flag replaces all the occurrences on the line, and the absence of the g replaces only the first. The i flag makes the pattern case-insensitive, and the p flag prints the line if the substitution was made . The sed 's/foo/bar/g' replaces every “foo” on every line, and the sed 's/foo/bar/' replaces only the first “foo” on each line. The sed -n 's/foo/bar/p' prints only the lines where the substitution was made, because the -n suppresses the default printing and the p flag prints the matching lines.

The in-place problem.

The sed command writes to the stdout by default, and the -i option edits the file in place. The -i.bak form creates a backup file with the .bak suffix before the edit, and the -i'' form (on macOS) edits the file in place without a backup . The -i option is not available in the POSIX sed, and the script that needs to be portable must use a temporary file and the mv command.

a. The substitution command and the captured groups

The s command is the most common sed command. It replaces the pattern with the replacement, and the flags control the scope.

#!/usr/bin/env bash
# Replace the first occurrence on each line
sed 's/foo/bar/' file.txt

# Replace all occurrences on each line
sed 's/foo/bar/g' file.txt

# Case-insensitive replacement
sed 's/foo/bar/i' file.txt

# Replace only the lines that match the address
sed '/error/s/warning/critical/' file.txt

The s/foo/bar/ replaces the first “foo” on each line, and the s/foo/bar/g replaces every “foo”. The i flag makes the pattern case-insensitive. The /error/s/warning/critical/ command applies the substitution only to the lines that contain “error”, which is the address and the substitution combined .

The replacement can contain the & for the whole match and the \1 through \9 for the captured groups.

# Wrap the matched text in brackets
sed 's/error/[&]/g' file.txt

# Swap the two words
sed -E 's/([a-z]+) ([a-z]+)/\2 \1/' file.txt

# Format a date from YYYY-MM-DD to DD/MM/YYYY
sed -E 's/([0-9]{4})-([0-9]{2})-([0-9]{2})/\3\/\2\/\1/' file.txt

The & in the replacement is the whole match, so the s/error/[&]/g wraps each “error” in brackets . The \1 and \2 are the captured groups, so the s/([a-z]+) ([a-z]+)/\2 \1/ swaps the two words. The date example captures the year, the month, and the day, and the replacement reorders them.

The delimiter does not have to be the slash. The sed command uses the first character after the s as the delimiter, so the s|foo|bar| and the s#foo#bar# are the same as the s/foo/bar/ . The alternative delimiter is useful when the pattern or the replacement contains the slash, such as a file path.

# Replace a path without escaping the slashes
sed 's|/usr/local/bin|/opt/bin|g' file.txt

The s|/usr/local/bin|/opt/bin|g replaces the path without the need to escape the slashes, which is the reason for the alternative delimiter.

b. The deletion, printing, and insertion commands

The d command deletes the matching lines, the p command prints them, and the a, i, and c commands insert or change the text.

#!/usr/bin/env bash
# Delete the blank lines
sed '/^$/d' file.txt

# Delete the comment lines
sed '/^#/d' file.txt

# Print only the matching lines
sed -n '/error/p' file.txt

# Insert a line before the match
sed '/error/i\WARNING: error found' file.txt

# Append a line after the match
sed '/error/a\Check the logs' file.txt

# Change the matching line
sed '/old-config/c\new-config' file.txt

The /^$/d deletes the blank lines, and the /^#/d deletes the lines that start with the # . The -n '/error/p' prints only the lines that contain “error”, because the -n suppresses the default printing and the p prints the matching lines. The i\ inserts the text before the matching line, the a\ appends it after, and the c\ replaces the matching line.

The = command prints the line number, and the q command quits after the first match.

# Print the line number and the line
sed -n '/error/{=;p}' file.txt

# Quit after the first match
sed '/error/q' file.txt

The /error/{=;p} command prints the line number and the line for the first match, and the /error/q command quits after the first match . The q is useful for the large files where the first match is the only one that matters.

c. The in-place editing and the multiple commands

The -i option edits the file in place, and the -e option allows multiple commands on the same command line.

#!/usr/bin/env bash
# Edit the file in place
sed -i 's/foo/bar/g' file.txt

# Edit in place with a backup
sed -i.bak 's/foo/bar/g' file.txt

# Multiple commands with -e
sed -e 's/foo/bar/g' -e 's/baz/qux/g' file.txt

# Multiple commands in a single string
sed 's/foo/bar/g; s/baz/qux/g' file.txt

The sed -i 's/foo/bar/g' file.txt edits the file in place, and the sed -i.bak creates the file.txt.bak backup before the edit . The -e option allows the multiple commands, and the ; separates the commands in a single string. The multiple commands are applied in the order they are listed.

The -i option is not portable to the macOS sed, which requires an argument to the -i option. The portable form uses a temporary file and the mv command.

# Portable in-place editing
sed 's/foo/bar/g' file.txt > file.txt.tmp && mv file.txt.tmp file.txt

The sed ... > file.txt.tmp && mv file.txt.tmp file.txt form writes the output to the temporary file and renames it, which is the portable alternative to the -i option .

Complete Example Session

#!/usr/bin/env bash
# ============================================
# PART 1: THE BASIC SUBSTITUTION
# ============================================
echo "hello world" | sed 's/world/there/'
# Output: hello there

# ============================================
# PART 2: THE GLOBAL SUBSTITUTION
# ============================================
echo "foo foo foo" | sed 's/foo/bar/g'
# Output: bar bar bar

# ============================================
# PART 3: THE CASE-INSENSITIVE SUBSTITUTION
# ============================================
echo "Foo FOO foo" | sed 's/foo/bar/gi'
# Output: bar bar bar

# ============================================
# PART 4: THE CAPTURED GROUPS
# ============================================
echo "2025-01-15" | sed -E 's/([0-9]{4})-([0-9]{2})-([0-9]{2})/\3\/\2\/\1/'
# Output: 15/01/2025

# ============================================
# PART 5: THE ADDRESS
# ============================================
printf "error: disk\nwarning: cpu\nerror: memory\n" | sed '/error/s/error/critical/'
# Output: critical: disk
#         warning: cpu
#         critical: memory

# ============================================
# PART 6: THE DELETION
# ============================================
printf "line1\n\nline3\n" | sed '/^$/d'
# Output: line1
#         line3

# ============================================
# PART 7: THE INSERT AND APPEND
# ============================================
printf "config\n" | sed '/config/i\# Header'
# Output: # Header
#         config

# ============================================
# PART 8: THE PRINT WITH -n
# ============================================
printf "error: disk\ninfo: ok\nerror: cpu\n" | sed -n '/error/p'
# Output: error: disk
#         error: cpu

# ============================================
# PART 9: THE IN-PLACE EDIT
# ============================================
cp /etc/hostname /tmp/test.txt
sed -i 's/old/new/g' /tmp/test.txt

# ============================================
# PART 10: THE MULTIPLE COMMANDS
# ============================================
printf "foo baz\n" | sed -e 's/foo/bar/' -e 's/baz/qux/'
# Output: bar qux

The ten parts demonstrate the sed surface: the basic substitution, the global flag, the case-insensitive flag, the captured groups, the address, the deletion, the insert, the print with -n, the in-place edit, and the multiple commands. The critical distinction is between Parts 1 and 2: the absence of the g replaces only the first occurrence, and the g replaces every occurrence.

Quick Reference

sed Commands

Command	Purpose	Example
`s/old/new/`	Substitute	`sed 's/foo/bar/'`
`s/old/new/g`	Substitute all	`sed 's/foo/bar/g'`
`d`	Delete	`sed '/error/d'`
`p`	Print	`sed -n '/error/p'`
`a\`	Append after	`sed '/x/a\new line'`
`i\`	Insert before	`sed '/x/i\new line'`
`c\`	Change line	`sed '/x/c\replacement'`
`q`	Quit	`sed '/error/q'`
`=`	Print line number	`sed -n '/error/='`

sed Flags

Flag	Purpose
`g`	All occurrences
`i`	Case-insensitive
`p`	Print if substituted
`n`	Suppress default printing
`-i`	Edit in place
`-i.bak`	Edit in place with backup
`-E`	Extended regular expressions
`-e`	Multiple commands

sed Addresses

Address	Matches
`5`	Line 5
`$`	Last line
`/pattern/`	Lines that match the pattern
`1,5`	Lines 1 through 5
`/start/,/end/`	Lines from the start to the end
`!`	Negation

sed Replacement Specials

Special	Meaning
`&`	The whole match
`\1`–`\9`	Captured groups
`\n`	Newline
`\t`	Tab

Best Practices

✅ Do This:

# Use -E for extended regex
sed -E 's/([0-9]+)/[\1]/g' file.txt                            # ✅

# Use the alternative delimiter for paths
sed 's|/old/path|/new/path|g' file.txt                         # ✅

# Use -i.bak for a backup
sed -i.bak 's/foo/bar/g' file.txt                              # ✅

# Use -n with p to print only the matches
sed -n '/error/p' file.txt                                     # ✅

# Use NR > 1 to skip the header
sed '1d' data.csv                                              # ✅

# Use the portable form for the in-place edit
sed 's/foo/bar/g' file.txt > file.txt.tmp && mv file.txt.tmp file.txt # ✅

❌ Don’t Do This:

# Forget the g flag
sed 's/foo/bar/' file.txt    # Only the first occurrence       # ❌

# Use -i in a POSIX script
sed -i 's/foo/bar/' file.txt    # Not portable                  # ❌

# Forget the p flag with -n
sed -n '/error/' file.txt    # Prints nothing                   # ❌

# Use the same delimiter as the pattern
sed 's/foo/bar/' file.txt    # The slash must be escaped       # ❌

# Use sed for the complex calculations
# Use awk instead                                              # ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
Only the first occurrence replaced	Missing `g` flag	Use `s/old/new/g`
No output	`-n` without `p`	Add `p` or remove `-n`
Pattern not matching	Special regex characters	Use `-E` or escape
File not modified	Forgot `-i`	Use `-i`
macOS `-i` error	`-i` requires an argument	Use `-i ''` or a temp file

Real-World Examples

1. Basic substitution

sed 's/foo/bar/' file.txt

2. Global substitution

sed 's/foo/bar/g' file.txt

3. Case-insensitive

sed 's/foo/bar/gi' file.txt

4. Delete blank lines

sed '/^$/d' file.txt

5. Delete comments

sed '/^#/d' file.txt

6. Print matching lines

sed -n '/error/p' file.txt

7. In-place edit

sed -i 's/foo/bar/g' file.txt

8. In-place with backup

sed -i.bak 's/foo/bar/g' file.txt

9. Multiple commands

sed -e 's/foo/bar/' -e 's/baz/qux/' file.txt

10. Date format

sed -E 's/([0-9]{4})-([0-9]{2})-([0-9]{2})/\3\/\2\/\1/' file.txt

Visual

The Substitution

┌──────────────────────────────────────────────────────────────┐
│  s/pattern/replacement/flags                                 │
│                                                              │
│  Input: "foo foo foo"                                        │
│                                                              │
│  s/foo/bar/                                                  │
│  ┌────────────────────────────────────────────────────────┐  │
│  │ → "bar foo foo"    (only the first)                   │  │
│  └────────────────────────────────────────────────────────┘  │
│                                                              │
│  s/foo/bar/g                                                 │
│  ┌────────────────────────────────────────────────────────┐  │
│  │ → "bar bar bar"    (all)                              │  │
│  └────────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────┘

The Address

┌──────────────────────────────────────────────────────────────┐
│  THE ADDRESS RESTRICTS THE COMMAND TO THE MATCHING LINES     │
│                                                              │
│  /error/s/error/critical/                                    │
│  ┌────────────────────────────────────────────────────────┐  │
│  │ error: disk    → critical: disk                        │  │
│  │ warning: cpu   → warning: cpu    (not matched)         │  │
│  │ error: memory  → critical: memory                      │  │
│  └────────────────────────────────────────────────────────┘  │
│                                                              │
│  The address is the /error/ before the s command.            │
└──────────────────────────────────────────────────────────────┘

The Captured Groups

┌──────────────────────────────────────────────────────────────┐
│  \1 THROUGH \9 ARE THE CAPTURED GROUPS                       │
│                                                              │
│  Input: "2025-01-15"                                         │
│                                                              │
│  s/([0-9]{4})-([0-9]{2})-([0-9]{2})/\3\/\2\/\1/              │
│       │         │         │          │   │   │               │
│       │         │         │          │   │   └── \1: year   │
│       │         │         │          │   └────── \2: month  │
│       │         │         │          └────────── \3: day    │
│       │         │         └── group 3                        │
│       │         └── group 2                                  │
│       └── group 1                                            │
│                                                              │
│  Output: "15/01/2025"                                        │
└──────────────────────────────────────────────────────────────┘

Summary

Item	Value
Command syntax	`sed [options] 'script' [file...]`
Substitution	`s/pattern/replacement/flags`
Global flag	`g`
Case-insensitive flag	`i`
In-place editing	`-i` or `-i.bak`
Extended regex	`-E` or `-r`
Delete	`d`
Print	`p` with `-n`
Insert/append	`i\` / `a\`
Captured groups	`\1`–`\9`
Whole match	`&`
Address	Line number, `/pattern/`, or range

Key takeaways:

The s command is the substitution, and the flags control the scope. The g flag replaces all the occurrences on the line, and the i flag makes the pattern case-insensitive. The absence of the g replaces only the first occurrence .
The address restricts the command to the matching lines. The /error/s/error/critical/ applies the substitution only to the lines that contain “error”. The address can be a line number, a pattern, or a range, and the ! negates it .
The -i option edits the file in place. The -i.bak form creates a backup before the edit. The -i option is not portable to the macOS sed, and the portable form uses a temporary file and the mv command .
The captured groups are \1 through \9. The s/([0-9]{4})-([0-9]{2})-([0-9]{2})/\3\/\2\/\1/ reorders the date, and the & is the whole match. The -E option enables the extended regular expressions, which makes the groups and the quantifiers easier to write .
The d command deletes the matching lines, and the p command prints them. The sed '/^$/d' deletes the blank lines, and the sed -n '/error/p' prints only the lines that contain “error”. The -n suppresses the default printing, and the p prints the matching lines .
The multiple commands are separated by -e or ;. The sed -e 's/foo/bar/' -e 's/baz/qux/' applies the two substitutions in order, and the sed 's/foo/bar/; s/baz/qux/' does the same in a single string.

Remember: sed is the stream editor. The s command substitutes, the d command deletes, the p command prints, and the a, i, and c commands insert, append, and change. The address restricts the command to the matching lines, and the flags control the scope of the substitution. The -i option edits the file in place, the -E option enables the extended regular expressions, and the -e option allows the multiple commands. The captured groups and the & are the mechanism for the reordering and the reformatting. The result is a script that can transform the text in a stream or in a file, without a full programming language.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!