Bash Scripting 19 🐧 Text Processing Cut & Awk — Column parsing and tabular data extraction - KandZ

⏱️ Estimated read: 14 min | |

Text processing is the operation that turns structured text—CSV files, log entries, command output—into the specific fields that a script needs. The previous chapters covered loops and redirection, which move data between commands; this chapter covers the two tools that extract the data from the lines: cut and awk. cut is the simple field extractor: it splits each line on a delimiter and prints the fields that the user specifies. awk is the pattern-action language: it splits each line into fields, evaluates a condition, and executes an action, which makes it the tool for filtering, calculating, and formatting tabular data.

Quick How-To Guides & Fast FixesNeed a quick fix or step-by-step solution? Browse our fast answers on HowTo.KandZ.me:

Operating Systems CLI →Dev Environment Languages →Web Network Tools →Security Accounts →Security Access Tech →Device Phone Basics →Social Messaging →Digital Files Media →

The distinction between the two tools is the distinction between extraction and processing. cut -d',' -f1,3 prints the first and third fields of a CSV, and it does nothing else. awk -F',' '$3 > 100 {print $1, $3}' prints the first and third fields only for the rows where the third field is greater than 100 . The cut command is the right choice for the simple extraction, and awk is the right choice when the extraction depends on a condition or when the output needs to be calculated or formatted.

Key point: cut uses the -d option for the delimiter and the -f option for the fields, and the fields are 1-indexed. It cannot reorder the fields; the output is always in the order the fields appear in the input . awk splits each line into fields $1, $2, and so on, with $0 being the whole line. The -F option sets the field separator, and the BEGIN and END blocks run before the first line and after the last line . The awk fields are 1-indexed, and the $NF is the last field.

Why cut and awk require different approaches

The reordering problem.

The cut command prints the fields in the order they appear in the input, regardless of the order in the -f list. The cut -f1,3 and cut -f3,1 produce the same output, because cut reads the line from left to right and prints the selected fields as it encounters them . The awk command prints the fields in the order they are listed in the print statement, so awk '{print $3, $1}' prints the third field before the first. The reordering is the reason to choose awk over cut when the output order matters.

The condition problem.

The cut command has no condition. It prints the selected fields for every line, including the header line of a CSV. The awk command can filter the lines with a pattern, so awk 'NR > 1' skips the first line, and awk '$3 > 100' prints only the rows where the third field is greater than 100 . The NR is the current record number, and the NR > 1 condition is the standard way to skip the header.

The calculation problem.

The cut command cannot calculate. It prints the fields as they are. The awk command can sum a column, count the occurrences of a value, and format the output with printf. The awk '{sum += $2} END {print sum}' sums the second column, and the awk '{count[$1]++} END {for (k in count) print k, count[k]}' counts the occurrences of each value in the first column . The calculation and the associative arrays are the features that make awk a programming language rather than a field extractor.

a. The cut command and the field extraction

The cut command extracts fields or characters from each line. The -d option sets the delimiter, and the -f option selects the fields.

#!/usr/bin/env bash
# Extract the first and third fields of a CSV
cut -d',' -f1,3 data.csv

# Extract a range of fields
cut -d',' -f2-4 data.csv

# Extract by character position
cut -c1-10 file.txt

# Extract the first field of the password file
cut -d':' -f1 /etc/passwd

The -d',' sets the comma as the delimiter, and the -f1,3 selects the first and third fields . The -f2-4 selects the fields from 2 to 4, and the -c1-10 selects the characters from 1 to 10 of each line. The cut -d':' -f1 /etc/passwd extracts the usernames from the password file.

The cut command does not handle the multiple consecutive delimiters well. The ps output, for example, uses multiple spaces to align the columns, and the cut -d' ' treats each space as a delimiter, which produces the empty fields . The tr -s ' ' command squeezes the repeated spaces into a single space before the cut, and the awk command handles the whitespace by default.

# Squeeze the spaces before cut
ps aux | tr -s ' ' | cut -d' ' -f1

# Or use awk, which handles the whitespace by default
ps aux | awk '{print $1}'

The tr -s ' ' is the workaround for the repeated delimiters, and the awk default field separator (whitespace) handles the same case without the extra command . The cut is the simpler tool, but the awk is the more robust choice for the output with the variable spacing.

b. The awk command and the field variables

The awk command splits each line into fields and executes the action for the lines that match the pattern. The -F option sets the field separator, and the $1, $2, and $NF are the fields.

#!/usr/bin/env bash
# Print the first and third fields
awk '{print $1, $3}' file.txt

# Use a comma as the field separator
awk -F',' '{print $1, $3}' data.csv

# Print the last field
awk '{print $NF}' file.txt

# Filter the rows by condition
awk '$3 > 100 {print $1, $3}' data.txt

The {print $1, $3} action prints the first and third fields, separated by the default output field separator (a space) . The -F',' sets the comma as the input field separator, which is the correct choice for the CSV. The $NF is the last field, and the $3 > 100 is the condition that filters the rows.

The BEGIN and END blocks run before the first line and after the last line. The BEGIN is the place to set the field separator or print a header, and the END is the place to print the total or the average.

# Print a header, filter the rows, and print a footer
awk 'BEGIN {print "Name:Score"} $2 > 80 {print $1 ":" $2} END {print "Done"}' scores.txt

The BEGIN {print "Name:Score"} prints the header before the first line is read, and the END {print "Done"} prints the footer after the last line . The $2 > 80 is the condition for the rows that are printed.

The awk built-in variables NR, NF, and FNR are useful for the line numbers and the field counts.

# Skip the header and print the line number with the first field
awk 'NR > 1 {print NR ": " $1}' data.csv

# Print the number of fields in each line
awk '{print NF}' file.txt

# Print the file name and the line number for the first line of each file
awk 'FNR == 1 {print FILENAME ": " $0}' file1.txt file2.txt

The NR is the total record number across all files, and the FNR is the record number within the current file . The NF is the number of fields in the current line. The NR > 1 skips the first line, and the FNR == 1 matches the first line of each file.

c. The awk associative arrays and the calculation

The awk associative arrays are the mechanism for the grouping and the counting. The array key is a string, and the value is the count or the sum. The for (key in array) loop iterates over the keys, and the order is not guaranteed .

#!/usr/bin/env bash
# Count the occurrences of each value in the first column
awk '{count[$1]++} END {for (key in count) print key, count[key]}' server.log

# Sum the third column grouped by the first column
awk -F',' '{sales[$1] += $3} END {for (cat in sales) print cat, sales[cat]}' sales.csv

# Calculate the average of the second column
awk '{sum += $2} END {print "Average:", sum/NR}' numbers.txt

The count[$1]++ increments the counter for each value in the first column, and the END block prints the counts . The sales[$1] += $3 sums the third column grouped by the first column. The sum/NR calculates the average, where the NR is the number of lines.

The awk output can be formatted with the printf function, which is the mechanism for the column alignment and the decimal places.

# Format the output with the printf
awk -F',' '{printf "%-20s %10s\n", $1, $2}' data.csv

The %-20s left-aligns the first field in a 20-character column, and the %10s right-aligns the second field in a 10-character column . The \n is the newline. The printf is the correct choice when the output needs to be aligned, and the print is the correct choice when the default formatting is sufficient.

Complete Example Session

#!/usr/bin/env bash
# ============================================
# PART 1: THE SIMPLE CUT
# ============================================
cut -d',' -f1,3 data.csv
# Output: the first and third fields of the CSV

# ============================================
# PART 2: THE CUT RANGE
# ============================================
cut -d',' -f2-4 data.csv
# Output: the fields from 2 to 4

# ============================================
# PART 3: THE AWK FIELD EXTRACTION
# ============================================
awk -F',' '{print $1, $3}' data.csv
# Output: the first and third fields

# ============================================
# PART 4: THE AWK REORDERING
# ============================================
awk -F',' '{print $3, $1}' data.csv
# Output: the third field before the first

# ============================================
# PART 5: THE AWK FILTER
# ============================================
awk -F',' '$3 > 100 {print $1, $3}' data.csv
# Output: only the rows where the third field is greater than 100

# ============================================
# PART 6: THE AWK BEGIN AND END
# ============================================
awk -F',' 'BEGIN {print "Name,Score"} $2 > 80 {print $1 ":" $2} END {print "Done"}' scores.csv

# ============================================
# PART 7: THE AWK SUM AND AVERAGE
# ============================================
awk '{sum += $2} END {print "Total:", sum, "Average:", sum/NR}' numbers.txt

# ============================================
# PART 8: THE AWK COUNT
# ============================================
awk '{count[$1]++} END {for (key in count) print key, count[key]}' server.log

# ============================================
# PART 9: THE AWK HEADER SKIP
# ============================================
awk -F',' 'NR > 1 {sum += $3} END {print sum}' data.csv
# The header is skipped because NR > 1

# ============================================
# PART 10: THE COLUMN ALIGNMENT
# ============================================
awk -F',' '{printf "%-20s %10s\n", $1, $2}' data.csv

The ten parts demonstrate the cut and awk surface: the simple cut, the cut range, the awk field extraction, the awk reordering, the awk filter, the awk BEGIN and END, the awk sum and average, the awk count, the awk header skip, and the column alignment. The critical distinction is between Parts 1 and 5: the cut prints the fields for every line, and the awk prints them only for the rows that match the condition.

Quick Reference

cut Options

Option	Purpose	Example
`-d`	Field delimiter	`-d','`
`-f`	Fields to print	`-f1,3` or `-f2-4`
`-c`	Characters to print	`-c1-10`
`--output-delimiter`	Output delimiter	`--output-delimiter=' '`

awk Built-in Variables

Variable	Meaning
`$0`	The whole line
`$1`–`$NF`	The fields
`NF`	Number of fields
`NR`	Total record number
`FNR`	Record number within the file
`FS`	Field separator
`OFS`	Output field separator
`FILENAME`	Current file name

awk Special Blocks

Block	When
`BEGIN`	Before the first line
`END`	After the last line
`pattern { action }`	For the lines that match the pattern

cut vs awk

Task	cut	awk
Extract fields	Yes	Yes
Reorder fields	No	Yes
Filter by condition	No	Yes
Calculate	No	Yes
Handle multiple spaces	No	Yes (default)

Best Practices

✅ Do This:

# Use cut for the simple extraction
cut -d',' -f1,3 data.csv                                       # ✅

# Use awk for the reordering
awk -F',' '{print $3, $1}' data.csv                            # ✅

# Use awk for the filtering
awk -F',' '$3 > 100 {print $1, $3}' data.csv                   # ✅

# Skip the header with NR > 1
awk -F',' 'NR > 1 {sum += $3} END {print sum}' data.csv        # ✅

# Use printf for the aligned output
awk -F',' '{printf "%-20s %10s\n", $1, $2}' data.csv           # ✅

# Use the default whitespace separator for the ps output
ps aux | awk '{print $1}'                                      # ✅

❌ Don’t Do This:

# Use cut with a single space on the variable-spaced output
ps aux | cut -d' ' -f1    # Empty fields for the aligned columns # ❌

# Use cut to reorder the fields
cut -f3,1 data.csv    # The output is still 1,3                # ❌

# Forget the header in the CSV sum
awk -F',' '{sum += $3} END {print sum}' data.csv    # Header is included # ❌

# Use awk without the single quotes
awk {print $1} file.txt    # The shell expands $1              # ❌

# Assume the awk array order is guaranteed
for (k in arr)    # The order is not guaranteed                # ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
cut produces empty fields	Multiple consecutive delimiters	Use `tr -s` or `awk`
cut does not reorder	The output follows the input order	Use `awk`
awk script expanded by the shell	No single quotes	Quote the script
CSV sum includes the header	No `NR > 1` condition	Add the condition
awk array order is random	The `for` loop does not sort	Pipe to `sort` or use `PROCINFO`

Real-World Examples

1. Extract the CSV fields

cut -d',' -f1,3 data.csv

2. Extract the characters

cut -c1-10 file.txt

3. Print the last field

awk '{print $NF}' file.txt

4. Filter the rows

awk -F',' '$3 > 100 {print $1, $3}' data.csv

5. Sum the column

awk '{sum += $2} END {print sum}' numbers.txt

6. Count the occurrences

awk '{count[$1]++} END {for (k in count) print k, count[k]}' server.log

7. Skip the header

awk -F',' 'NR > 1 {sum += $3} END {print sum}' data.csv

8. Print with a header and footer

awk 'BEGIN {print "Start"} {print $1} END {print "End"}' file.txt

9. Format the output

awk -F',' '{printf "%-20s %10s\n", $1, $2}' data.csv

10. Align the columns

ls -l | column -t

Visual

The cut and awk Field Model

┌──────────────────────────────────────────────────────────────┐
│  BOTH TOOLS SPLIT THE LINE INTO FIELDS                       │
│                                                              │
│  Line: "alice,30,New York"                                   │
│       │                                                      │
│       ▼                                                      │
│  cut -d',' -f1,3                                             │
│  ├── field 1: alice                                          │
│  ├── field 2: 30                                             │
│  └── field 3: New York                                       │
│       │                                                      │
│       ▼                                                      │
│  Output: alice,New York                                      │
│                                                              │
│  awk -F',' '{print $3, $1}'                                  │
│  ├── $1: alice                                               │
│  ├── $2: 30                                                  │
│  └── $3: New York                                            │
│       │                                                      │
│       ▼                                                      │
│  Output: New York alice                                      │
└──────────────────────────────────────────────────────────────┘

The awk BEGIN and END

┌──────────────────────────────────────────────────────────────┐
│  BEGIN RUNS BEFORE THE FIRST LINE; END RUNS AFTER THE LAST   │
│                                                              │
│  BEGIN {print "Start"}                                       │
│       │                                                      │
│       ▼                                                      │
│  Line 1 → pattern { action }                                 │
│  Line 2 → pattern { action }                                 │
│  Line 3 → pattern { action }                                 │
│       │                                                      │
│       ▼                                                      │
│  END {print "Done"}                                          │
│                                                              │
│  The BEGIN is for the headers and the setup.                 │
│  The END is for the totals and the summaries.                │
└──────────────────────────────────────────────────────────────┘

cut vs awk

┌──────────────────────────────────────────────────────────────┐
│  CUT EXTRACTS; AWK PROCESSES                                 │
│                                                              │
│  cut:                                                        │
│  ┌────────────────────────────────────────────────────────┐  │
│  │ Simple field extraction                                │  │
│  │ No reordering                                          │  │
│  │ No conditions                                          │  │
│  │ No calculations                                        │  │
│  └────────────────────────────────────────────────────────┘  │
│                                                              │
│  awk:                                                        │
│  ┌────────────────────────────────────────────────────────┐  │
│  │ Field extraction with reordering                       │  │
│  │ Conditions and filters                                 │  │
│  │ Calculations and associative arrays                    │  │
│  │ Formatted output with printf                           │  │
│  └────────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────┘

Summary

Item	Value
cut field delimiter	`-d`
cut fields	`-f`
cut characters	`-c`
cut field order	Follows the input
awk field separator	`-F` or `FS`
awk fields	`$1`–`$NF`, `$0` is the whole line
awk record number	`NR`, `FNR`
awk field count	`NF`
awk special blocks	`BEGIN`, `END`
awk filter	`pattern { action }`
awk calculation	`sum += $2`, `count[$1]++`
awk formatting	`printf`

Key takeaways:

cut is the simple field extractor. The -d option sets the delimiter, and the -f option selects the fields. The output follows the input order, and the tool has no conditions or calculations . It is the right choice for the straightforward extraction from the well-delimited text.
awk is the pattern-action language. The -F option sets the field separator, and the $1 through $NF are the fields. The pattern { action } form filters the lines, and the BEGIN and END blocks run before the first line and after the last line .
cut cannot reorder the fields, but awk can. The cut -f3,1 output is the same as the cut -f1,3, because cut prints the fields in the order they appear. The awk '{print $3, $1}' prints the third field before the first.
awk can filter and calculate. The $3 > 100 {print $1, $3} prints only the rows that match the condition, and the sum += $2 accumulates the sum. The count[$1]++ counts the occurrences of each value in the first column .
The NR > 1 condition skips the header. The awk -F',' 'NR > 1 {sum += $3} END {print sum}' sums the third column without including the header row. The NR is the total record number, and the FNR is the record number within the current file .
The printf function formats the output. The %-20s left-aligns the string in a 20-character column, and the %10s right-aligns it in a 10-character column. The printf is the correct choice when the output needs to be aligned, and the print is the correct choice when the default formatting is sufficient.

Remember: cut and awk are the two tools for the column parsing. cut extracts the fields with the -d and -f options, and it is the simple choice for the well-delimited text. awk splits the line into the fields, evaluates the pattern, and executes the action, and it is the choice for the filtering, the reordering, the calculation, and the formatting. The -F option sets the separator, the $1 through $NF are the fields, the NR and NF are the built-in variables, and the BEGIN and END blocks run before and after the input. The result is a script that can extract the columns from the CSV, filter the log entries, and summarize the data.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!