Bash Scripting 09 🐧 String Manipulation — Substring extraction, replacement, and length
Bash treats every variable as a string. Numbers, filenames, configuration values, and command output are all stored as sequences of characters. The operations that inspect and transform those sequences—measuring length, extracting a portion, replacing a pattern—are performed through parameter expansion, the same ${...} syntax that handles variable substitution. This is not a separate library or an external tool. It is built into the shell, and it executes without spawning a process.
The previous chapters covered arithmetic and conditional logic. String manipulation is the complement to those tools: arithmetic computes with numbers, and string operations manipulate the text that scripts read from files, arguments, and command output. The syntax is dense, but it follows a consistent pattern. The variable name comes first, followed by an operator that determines the operation, and optionally a pattern or offset that determines the specific result.
Key point: Bash string operations use parameter expansion, not external commands. ${#var} returns length, ${var:offset:length} extracts a substring, and ${var/pattern/replacement} replaces text. The substring offset is zero-based for ordinary variables but one-based for positional parameters, and negative offsets require a space after the colon to distinguish them from the default-value operator. Pattern removal with # and % is often more useful for filenames than substring extraction because it matches the structure of the string rather than a fixed position.
Why parameter expansion replaces external tools
The process-spawning problem.
A script that calls sed, awk, or cut for every string operation pays a process-creation cost each time. On a loop that processes thousands of filenames, that cost dominates the runtime. Parameter expansion performs the same operations inside the shell process, with no fork and no exec. The syntax is less readable than a sed command, but the performance difference is measurable in any script that operates on more than a handful of strings.
The quoting problem.
External tools require the string to be passed as an argument or piped through stdin, which introduces quoting and escaping issues. Parameter expansion operates directly on the variable’s value, without any intermediate serialization. The string does not need to be quoted a second time, and special characters in the value do not need to be escaped for the shell to parse them correctly. The expansion produces the result directly, and the result can be assigned to another variable or used in a command.
The positional-parameter problem.
Bash provides ${var:offset:length} for substrings and ${var#pattern} / ${var%pattern} for pattern removal. The pattern-removal operators are often more appropriate for filenames and paths because they match the structure of the string. ${path##*/} extracts the basename by removing everything up to the last slash, regardless of how many directories the path contains. ${file%.*} removes the extension by matching the last dot and everything after it. These operations express intent more clearly than counting characters to a fixed offset.
a. String length and substring extraction
The ${#var} expansion returns the number of characters in the variable’s value. This is the simplest operation and the foundation for validation and loop bounds.
#!/usr/bin/env bash
# Length of a string
name="deploy-production-v2"
echo "${#name}" # 21
# Validation: reject empty values
if [ "${#name}" -eq 0 ]; then
echo "ERROR: name cannot be empty" >&2
exit 1
fiThe length operator works on any variable, including positional parameters (${#1}) and arrays (${#array[@]} for the number of elements). It returns a number, so it can be used directly in arithmetic comparisons.
Substring extraction uses ${var:offset:length}. The offset is the starting position, and the length is the number of characters to extract. If the length is omitted, the extraction continues to the end of the string .
# Substring extraction
full="us-east-1-web-prod-01"
echo "${full:0:9}" # us-east-1 (position 0, 9 chars)
echo "${full:10}" # web-prod-01 (position 10 to end)
echo "${full: -8}" # prod-01 (last 8 chars; note the space)
echo "${full: -8:4}" # prod (last 8 chars, take 4)The space before the negative offset is mandatory. Without it, ${full:-8} is interpreted as the default-value operator, which returns the value of full if it is set, or 8 if it is not. The space disambiguates the negative offset from the default-value syntax .
For positional parameters, the indexing is one-based by default. ${@:2} starts from the second argument, not the third. This is a deliberate inconsistency that reflects the convention that $1 is the first argument .
b. Pattern replacement
The replacement syntax is ${var/pattern/replacement} for the first match and ${var//pattern/replacement} for all matches. The pattern is a glob, not a regular expression, and the replacement is a literal string .
#!/usr/bin/env bash
# Replace first occurrence
version="app-v1.2.3-beta"
echo "${version/beta/stable}" # app-v1.2.3-stable
# Replace all occurrences
echo "${version//-/_}" # app_v1.2.3_beta
# Anchor to start with #
path="/old/path/to/file"
echo "${path/#\/old/\/new}" # /new/path/to/file
# Anchor to end with %
filename="deploy.sh.bak"
echo "${filename/%.bak/}" # deploy.shThe # anchor matches only at the beginning of the string, and the % anchor matches only at the end . This is distinct from the # and % removal operators, which delete the matching portion rather than replacing it. The replacement syntax with an anchor is useful when the same pattern could appear elsewhere in the string and should not be modified in those positions.
If the replacement string is empty, the pattern matches are deleted. ${filename/%.bak/} removes the .bak suffix by replacing it with nothing.
c. Prefix and suffix removal
The removal operators are the most commonly used string operations in scripts that handle filenames and paths. ${var#pattern} removes the shortest matching prefix, and ${var##pattern} removes the longest. ${var%pattern} removes the shortest matching suffix, and ${var%%pattern} removes the longest .
#!/usr/bin/env bash
# Prefix removal
filepath="/home/user/documents/file.txt"
echo "${filepath#*/}" # home/user/documents/file.txt (shortest prefix)
echo "${filepath##*/}" # file.txt (longest prefix; basename)
# Suffix removal
filename="document.old.txt"
echo "${filename%.*}" # document.old (shortest suffix; last extension)
echo "${filename%%.*}" # document (longest suffix; all extensions)The ${filepath##*/} idiom is the standard way to extract a basename without calling basename. The ## removes the longest prefix matching */, which is everything up to and including the last slash. The ${filename%.*} idiom removes the last extension by matching the shortest suffix that starts with a dot .
The same operators work with variables containing patterns. ${var#$prefix} removes the value of $prefix from the beginning of $var, which is useful when the prefix is stored in a separate variable.
Complete Example Session
#!/usr/bin/env bash
# ============================================
# PART 1: LENGTH VALIDATION
# ============================================
input="config.yaml"
if [ "${#input}" -eq 0 ]; then
echo "Empty input" >&2
exit 1
fi
echo "Length: ${#input}" # 11
# ============================================
# PART 2: SUBSTRING FOR PREFIX EXTRACTION
# ============================================
timestamp="20250101-143022"
date_part="${timestamp:0:8}" # 20250101
time_part="${timestamp:9:6}" # 143022
echo "Date: $date_part, Time: $time_part"
# ============================================
# PART 3: NEGATIVE OFFSET FOR SUFFIX
# ============================================
logfile="application-error.log"
extension="${logfile: -3}" # log (last 3 chars)
echo "Extension: $extension"
# ============================================
# PART 4: REPLACE FORMATTING
# ============================================
raw="2025_01_01"
formatted="${raw//_/-}" # 2025-01-01
echo "Formatted: $formatted"
# ============================================
# PART 5: ANCHORED REPLACEMENT
# ============================================
url="http://example.com/api"
secure="${url/#http:/https:}" # https://example.com/api
echo "Secure: $secure"
# ============================================
# PART 6: BASENAME EXTRACTION
# ============================================
fullpath="/var/log/nginx/access.log"
basename="${fullpath##*/}" # access.log
echo "Basename: $basename"
# ============================================
# PART 7: DIRECTORY EXTRACTION
# ============================================
dirpath="${fullpath%/*}" # /var/log/nginx
echo "Directory: $dirpath"
# ============================================
# PART 8: EXTENSION REMOVAL
# ============================================
filename="archive.tar.gz"
no_ext="${filename%%.*}" # archive
only_last="${filename%.*}" # archive.tar
echo "No extension: $no_ext, Last removed: $only_last"
# ============================================
# PART 9: COMBINED OPERATIONS
# ============================================
# Remove directory, then remove extension
clean="${fullpath##*/}" # access.log
stem="${clean%.*}" # access
echo "Stem: $stem"
# ============================================
# PART 10: CASE CONVERSION (BASH 4+)
# ============================================
name="deploy-prod"
upper="${name^^}" # DEPLOY-PROD
lower="${upper,,}" # deploy-prod
echo "Upper: $upper, Lower: $lower"The ten parts demonstrate the operations in the order a script would use them: length for validation, substring for fixed-position extraction, negative offset for suffix, replacement for formatting, anchored replacement for URL schemes, prefix and suffix removal for paths, and case conversion for normalization. The path operations in Parts 6–9 are the ones that appear most frequently in real scripts.
Quick Reference
String Operations
| Operation | Syntax | Example |
|---|---|---|
| Length | ${#var} | ${#name} |
| Substring | ${var:offset:length} | ${path:0:4} |
| Substring to end | ${var:offset} | ${path:10} |
| Negative offset | ${var: -n} | ${path: -8} |
| Replace first | ${var/old/new} | ${v/beta/stable} |
| Replace all | ${var//old/new} | ${v//-/_} |
| Replace at start | ${var/#old/new} | ${url/#http:/https:} |
| Replace at end | ${var/%old/new} | ${f/%.bak/} |
| Remove prefix (short) | ${var#pattern} | ${path#*/} |
| Remove prefix (long) | ${var##pattern} | ${path##*/} |
| Remove suffix (short) | ${var%pattern} | ${f%.*} |
| Remove suffix (long) | ${var%%pattern} | ${f%%.*} |
| Uppercase | ${var^^} | Bash 4+ |
| Lowercase | ${var,,} | Bash 4+ |
Path Operations
| Goal | Syntax | Result for /a/b/c.txt |
|---|---|---|
| Basename | ${path##*/} | c.txt |
| Directory | ${path%/*} | /a/b |
| Extension | ${path##*.} | txt |
| Remove extension | ${path%.*} | /a/b/c |
Best Practices
✅ Do This:
# Use parameter expansion instead of external commands
basename="${path##*/}" # ✅
extension="${file##*.}" # ✅
# Space before negative offset
last_chars="${var: -3}" # ✅
# Quote expansions in assignments
result="${var//old/new}" # ✅
# Use ## for basename, % for directory
base="${path##*/}"; dir="${path%/*}" # ✅
# Validate length before processing
[ "${#input}" -gt 0 ] || exit 1 # ✅❌ Don’t Do This:
# Forget the space for negative offset
"${var:-3}" # Default-value operator, not substring # ❌
# Use regex syntax in patterns
"${var/[0-9]+/}" # + is literal, not regex # ❌
# Use sed for simple basename extraction
basename=$(echo "$path" | sed 's|.*/||') # Slow # ❌
# Assume ${var:0:4} works on arrays the same way
# Array substring uses different indexing # ❌
# Replace without anchoring when only prefix should change
"${url/http/https}" # Replaces anywhere, not just prefix # ❌Common Pitfalls
| Pitfall | Why It Happens | Fix |
|---|---|---|
| Negative offset returns default | No space before - | ${var: -3}, not ${var:-3} |
| Pattern does not match | Used regex instead of glob | Use *, ?, [] |
| Basename extraction wrong | Used # instead of ## | ${path##*/} for last slash |
| Extension removal leaves dot | Used # instead of % | ${file%.*} removes last dot |
| Replacement affects all occurrences | Used / instead of // | Single slash for first only |
Real-World Examples
1. Length check
[ "${#name}" -gt 0 ] || echo "Empty"2. First 8 characters (date)
date="${timestamp:0:8}"3. Last 4 characters (extension)
ext="${file: -4}"4. Replace underscores
clean="${raw//_/-}"5. Change URL scheme
secure="${url/#http:/https:}"6. Extract basename
base="${path##*/}"7. Extract directory
dir="${path%/*}"8. Remove extension
stem="${file%.*}"9. Remove all extensions
stem="${file%%.*}"10. Uppercase conversion
upper="${name^^}"Visual
Substring Indexing
┌──────────────────────────────────────────────────────────────┐
│ SUBSTRING OFFSETS ARE ZERO-BASED │
│ │
│ var="us-east-1-web-prod-01" │
│ 0123456789... │
│ │
│ ${var:0:9} │
│ ├── offset 0 (start) │
│ ├── length 9 │
│ └── result: "us-east-1" │
│ │
│ ${var:10} │
│ ├── offset 10 │
│ ├── no length → to end │
│ └── result: "web-prod-01" │
│ │
│ ${var: -8} │
│ ├── space before -8 (negative offset) │
│ ├── offset -8 → 8 chars from end │
│ └── result: "prod-01" │
└──────────────────────────────────────────────────────────────┘Prefix and Suffix Removal
┌──────────────────────────────────────────────────────────────┐
│ # REMOVES FROM START; % REMOVES FROM END │
│ │
│ path="/home/user/docs/file.txt" │
│ │
│ ${path#*/} │
│ ├── # = shortest prefix match │
│ ├── pattern = */ │
│ └── result: "home/user/docs/file.txt" │
│ │
│ ${path##*/} │
│ ├── ## = longest prefix match │
│ ├── pattern = */ │
│ └── result: "file.txt" │
│ │
│ ${path%/*} │
│ ├── % = shortest suffix match │
│ ├── pattern = /* │
│ └── result: "/home/user/docs" │
│ │
│ ${path%%/*} │
│ ├── %% = longest suffix match │
│ ├── pattern = /* │
│ └── result: "" (empty; first / is the match) │
└──────────────────────────────────────────────────────────────┘Replacement Anchors
┌──────────────────────────────────────────────────────────────┐
│ ANCHORED REPLACEMENT RESTRICTS WHERE THE MATCH OCCURS │
│ │
│ url="http://http.example.com/http" │
│ │
│ ${url/http/https} │
│ └── replaces FIRST occurrence anywhere │
│ result: "https://http.example.com/http" │
│ │
│ ${url//http/https} │
│ └── replaces ALL occurrences │
│ result: "https://https.example.com/https" │
│ │
│ ${url/#http:/https:} │
│ └── # = only at start │
│ result: "https://http.example.com/http" │
│ │
│ ${url/%http/https} │
│ └── % = only at end │
│ result: "http://http.example.com/https" │
└──────────────────────────────────────────────────────────────┘Summary
| Item | Value |
|---|---|
| Length | ${#var} |
| Substring | ${var:offset:length} |
| Negative offset | ${var: -n} (space required) |
| Replace first | ${var/old/new} |
| Replace all | ${var//old/new} |
| Replace at start | ${var/#old/new} |
| Replace at end | ${var/%old/new} |
| Remove prefix (short) | ${var#pattern} |
| Remove prefix (long) | ${var##pattern} |
| Remove suffix (short) | ${var%pattern} |
| Remove suffix (long) | ${var%%pattern} |
| Basename | ${path##*/} |
| Directory | ${path%/*} |
| Uppercase | ${var^^} (Bash 4+) |
Key takeaways:
- Parameter expansion is the built-in string library. Length, substring, replacement, and pattern removal are all performed by
${...}syntax. No external command is spawned, and no quoting or escaping is required beyond the standard shell rules . - The negative-offset space is not optional.
${var: -3}extracts the last three characters.${var:-3}is the default-value operator and returns the value ofvarif it is set, or3if it is not . #removes from the start;%removes from the end. The single-character forms remove the shortest match, and the doubled forms remove the longest.${path##*/}is the standard basename idiom;${file%.*}is the standard extension-removal idiom .- Replacement patterns are globs, not regex.
*,?, and[]are the supported wildcards. The#and%anchors restrict the match to the beginning or end of the string, and the doubled slashes replace all occurrences instead of only the first . - Substring indexing is zero-based for variables. The offset counts characters from the start of the string. For positional parameters, the indexing is one-based by default, which reflects the convention that
$1is the first argument . - The pattern operators express intent better than fixed offsets. Extracting a basename with
${path##*/}works regardless of how many directories the path contains. Counting characters to a fixed offset breaks when the path depth changes.
Remember: Bash strings are sequences of characters, and the operations that manipulate them are built into the shell’s expansion syntax. ${#var} measures, ${var:offset:length} extracts, ${var/old/new} replaces, and ${var##pattern} removes. The syntax is dense but consistent: the variable comes first, the operator determines the operation, and the pattern or offset determines the result. The negative-offset space is the one trap that produces a wrong answer instead of an error. The ## and %% forms are the ones that handle filenames and paths without counting characters. Use parameter expansion instead of sed or cut when the operation is a simple string transformation, and the script will be faster and more portable.
Stop using slow, ad-bloated tool sites! 🤮
🔎 Search “KandZ Tools” on Google to use many professional utilities for free.
KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)
⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free
🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!