Linux CLI 42 🐧 sort and uniq commands - KandZ

cat fruit.txt
sort -r fruit.txt
sort fruit.txt
sort -o sorted.txt fruit.txt
ls | sort -r

uniq fruit.txt
uniq -c fruit.txt
uniq -d fruit.txt
uniq -u fruit.txt

sort and uniq are the classic pair for working with lists of text lines. sort arranges lines in order; uniq collapses repeated lines. They’re almost always used together, because uniq only works correctly on sorted input.

Key point: uniq compares adjacent lines only. If duplicates aren’t next to each other, uniq won’t remove them. That’s why the standard idiom is sort file | uniq — never uniq file alone unless the file is already sorted.

a – sort command

sort is used to sort text files line by line. By default it sorts alphabetically (lexicographically), but it can sort numerically, by field, case-insensitively, and more.

Syntax:

sort [options] [file...]

Common options:

Option	Purpose
`-n`	Sort numerically
`-r`	Sort in reverse order
`-o FILE`	Write output to a file (instead of stdout)
`-f`	Case-insensitive sorting
`-u`	Sort and remove duplicate lines
`-c`	Check if the file is already sorted
`-k N`	Sort by field N
`-t SEP`	Use SEP as the field separator
`-h`	Human-readable numbers (1K, 2M)
`-M`	Sort by month name
`-R`	Random sort

Examples:

# Look at the file first
$ cat fruit.txt
banana
apple
cherry
apple
date
banana
apple

# Default sort (alphabetical)
$ sort fruit.txt
apple
apple
apple
banana
banana
cherry
date

# Reverse sort
$ sort -r fruit.txt
date
cherry
banana
banana
apple
apple
apple

# Sort and save to a file
$ sort -o sorted.txt fruit.txt
$ cat sorted.txt
apple
apple
apple
banana
banana
cherry
date

# Sort and remove duplicates
$ sort -u fruit.txt
apple
banana
cherry
date

# Sort pipe output
$ ls | sort -r
todo.txt
notes.txt
documents
downloads
Desktop

# Case-insensitive
$ cat mixed.txt
banana
Apple
cherry
apple
Banana

$ sort -f mixed.txt
apple
Apple
banana
Banana
cherry

# Numeric sort
$ cat numbers.txt
10
2
33
4
1

$ sort numbers.txt       # alphabetical — WRONG for numbers
1
10
2
33
4

$ sort -n numbers.txt    # numeric — correct
1
2
4
10
33

# Sort by the second field (column)
$ cat scores.txt
alice 95
bob 87
charlie 92

$ sort -k 2 -n scores.txt
bob 87
charlie 92
alice 95

# Sort by field 2, using ':' as separator
$ cat /etc/passwd | sort -t: -k 3 -n | head -5
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
...

Checking if a file is already sorted:

$ sort -c sorted.txt
$ echo $?
0
# ✅ File is sorted

$ sort -c fruit.txt
sort: fruit.txt:3: disorder: cherry
$ echo $?
1
# ❌ File is not sorted — first disorder at line 3

The -o option — why it matters:

# ❌ Wrong — reads and writes the same file, may corrupt it
$ sort fruit.txt > fruit.txt

# ✅ Correct — sort handles the file safely
$ sort -o fruit.txt fruit.txt

# ✅ Also fine — write to a temp file, then move
$ sort fruit.txt > sorted.txt
$ mv sorted.txt fruit.txt

b – uniq command

uniq is used to report or filter repeated lines in a sorted file. It’s the companion to sort — sort groups duplicates together, uniq then collapses or counts them.

Syntax:

uniq [options] [inputfile]

Common options:

Option	Purpose
`-c`	Show the number of occurrences of each line
`-d`	Show only duplicate lines
`-u`	Show only unique lines
`-i`	Case-insensitive comparison
`-s N`	Skip first N characters in each line
`-f N`	Skip first N fields
`-w N`	Compare only the first N characters

Examples:

# File must be sorted first!
$ cat fruit.txt
banana
apple
cherry
apple
date
banana
apple

# ❌ uniq on unsorted input — does almost nothing
$ uniq fruit.txt
banana
apple
cherry
apple
date
banana
apple

# ✅ Sort first, then uniq
$ sort fruit.txt | uniq
apple
banana
cherry
date

# Count occurrences
$ sort fruit.txt | uniq -c
      3 apple
      2 banana
      1 cherry
      1 date

# Show only lines that appear more than once
$ sort fruit.txt | uniq -d
apple
banana

# Show only lines that appear exactly once
$ sort fruit.txt | uniq -u
cherry
date

# Case-insensitive
$ sort -f mixed.txt | uniq -i
Apple
banana
cherry

Why sorting matters — the adjacent-only rule:

┌──────────────────────────────────────────────┐
│              Unsorted input                  │
│                                              │
│  banana                                      │
│  apple    ← different from banana            │
│  cherry   ← different from apple             │
│  apple    ← different from cherry            │
│  date     ← different from apple             │
│  banana   ← different from date              │
│  apple    ← different from banana            │
│                                              │
│  uniq sees NO adjacent duplicates →          │
│  prints every line unchanged                 │
│                                              │
└──────────────────────────────────────────────┘

┌──────────────────────────────────────────────┐
│              Sorted input                    │
│                                              │
│  apple    ─┐                                 │
│  apple     │ adjacent duplicates!            │
│  apple    ─┘                                 │
│  banana   ─┐                                 │
│  banana   ─┘                                 │
│  cherry                                      │
│  date                                        │
│                                              │
│  uniq collapses them → 4 unique lines        │
│                                              │
└──────────────────────────────────────────────┘

Skipping characters or fields:

# Ignore the first 2 characters
$ cat codes.txt
AA123
BB123
CC123
DD456

$ uniq -s 2 codes.txt
AA123
DD456

# Ignore the first field
$ cat logs.txt
2024-01-15 ERROR disk full
2024-01-15 ERROR disk full
2024-01-16 WARN cpu high

$ uniq -f 1 logs.txt
2024-01-15 ERROR disk full
2024-01-16 WARN cpu high

Combining with sort — the classic pipeline:

# Count how many times each line appears
$ sort file.txt | uniq -c

# Sort by frequency (most common first)
$ sort file.txt | uniq -c | sort -rn

# Top 10 most frequent lines
$ sort file.txt | uniq -c | sort -rn | head -10

# Find duplicates in a list
$ sort list.txt | uniq -d

# Find lines that appear only once
$ sort list.txt | uniq -u

Complete Example Session

# ============================================
# PART 1: BASIC SORT
# ============================================

$ cat fruit.txt
banana
apple
cherry
apple
date
banana
apple

$ sort fruit.txt
apple
apple
apple
banana
banana
cherry
date

$ sort -r fruit.txt
date
cherry
banana
banana
apple
apple
apple

$ sort -o sorted.txt fruit.txt
$ cat sorted.txt
apple
apple
apple
banana
banana
cherry
date

# ============================================
# PART 2: SORT WITH PIPES
# ============================================

$ ls
todo.txt  notes.txt  documents  downloads  Desktop

$ ls | sort -r
todo.txt
notes.txt
documents
downloads
Desktop

# ============================================
# PART 3: NUMERIC AND FIELD SORT
# ============================================

$ cat numbers.txt
10
2
33
4
1

$ sort numbers.txt       # wrong
1
10
2
33
4

$ sort -n numbers.txt    # right
1
2
4
10
33

$ cat scores.txt
alice 95
bob 87
charlie 92

$ sort -k 2 -n scores.txt
bob 87
charlie 92
alice 95

# ============================================
# PART 4: CHECK IF SORTED
# ============================================

$ sort -c sorted.txt
$ echo $?
0

$ sort -c fruit.txt
sort: fruit.txt:3: disorder: cherry
$ echo $?
1

# ============================================
# PART 5: UNIQ BASICS
# ============================================

$ uniq fruit.txt       # unsorted — does nothing
banana
apple
cherry
apple
date
banana
apple

$ sort fruit.txt | uniq
apple
banana
cherry
date

# ============================================
# PART 6: UNIQ WITH COUNTS
# ============================================

$ sort fruit.txt | uniq -c
      3 apple
      2 banana
      1 cherry
      1 date

$ sort fruit.txt | uniq -d
apple
banana

$ sort fruit.txt | uniq -u
cherry
date

# ============================================
# PART 7: COMBINED PIPELINES
# ============================================

# Top 3 most frequent words
$ cat text.txt | tr ' ' '\n' | sort | uniq -c | sort -rn | head -3
     42 the
     31 and
     27 of

# Find duplicate IPs in a log
$ awk '{print $1}' access.log | sort | uniq -d

# Unique usernames in /etc/passwd
$ cut -d: -f1 /etc/passwd | sort | uniq

# Count logins per user
$ cut -d: -f1 /etc/passwd | sort | uniq -c | sort -rn | head -5

# ============================================
# PART 8: SKIP FIELDS / CHARACTERS
# ============================================

$ cat codes.txt
AA123
BB123
CC123
DD456

$ uniq -s 2 codes.txt
AA123
DD456

$ cat logs.txt
2024-01-15 ERROR disk full
2024-01-15 ERROR disk full
2024-01-16 WARN cpu high

$ uniq -f 1 logs.txt
2024-01-15 ERROR disk full
2024-01-16 WARN cpu high

Quick Reference

sort

Command	Purpose
`sort FILE`	Alphabetical sort
`sort -r FILE`	Reverse
`sort -n FILE`	Numeric
`sort -f FILE`	Case-insensitive
`sort -u FILE`	Sort + unique
`sort -o OUT IN`	Write to a file
`sort -c FILE`	Check if sorted
`sort -k N FILE`	Sort by field N
`sort -t SEP FILE`	Custom separator
`sort -h FILE`	Human-readable numbers
`sort -M FILE`	Sort by month
`sort -R FILE`	Random order

uniq

Command	Purpose
`uniq FILE`	Collapse adjacent duplicates
`uniq -c FILE`	Count occurrences
`uniq -d FILE`	Only duplicates
`uniq -u FILE`	Only unique lines
`uniq -i FILE`	Case-insensitive
`uniq -s N FILE`	Skip N chars
`uniq -f N FILE`	Skip N fields
`uniq -w N FILE`	Compare first N chars

Classic Pipelines

Command	Purpose
`sort FILE \| uniq`	Remove duplicates
`sort FILE \| uniq -c`	Count occurrences
`sort FILE \| uniq -d`	Find duplicates
`sort FILE \| uniq -u`	Find unique lines
`sort FILE \| uniq -c \| sort -rn`	Sort by frequency
`sort FILE \| uniq -c \| sort -rn \| head`	Top N

Best Practices

✅ Do This:

# Always sort before uniq
sort file.txt | uniq                   # ✅

# Use sort -u for deduplication
sort -u file.txt                       # ✅

# Use -o to write back to the same file
sort -o file.txt file.txt              # ✅

# Use -n for numeric sorts
sort -n numbers.txt                    # ✅

# Use -k with -t for field sorting
sort -t: -k 3 -n /etc/passwd           # ✅

# Check sorting with -c
sort -c file.txt                       # ✅

# Combine with head for top-N
sort file.txt | uniq -c | sort -rn | head -10  # ✅

# Use -f for case-insensitive
sort -f file.txt | uniq -i             # ✅

❌ Don’t Do This:

# Don't run uniq on unsorted data
uniq unsorted.txt                      # ❌ does nothing useful

# Don't redirect output to the input file
sort file.txt > file.txt               # ❌ truncates first!

# Don't sort numbers alphabetically
sort numbers.txt                       # ❌ 10 comes before 2

# Don't forget -n with sort -k
sort -k 2 file.txt                     # ❌ alphabetical field sort

# Don't use uniq for non-adjacent dedup
uniq file.txt                          # ❌ only adjacent

# Don't ignore the -c output format
sort file.txt | uniq -c                # ✅ counts padded with spaces

Common Pitfalls

Pitfall	Problem	Solution
`uniq` on unsorted data	Nothing removed	`sort` first
Numeric file sorted alphabetically	10 before 2	Use `-n`
Redirect to same file	Data loss	Use `-o` or temp file
Duplicates not adjacent	`uniq` misses them	Sort first
`sort -k` on wrong field	Wrong order	Verify with `cat`
Case differences	“Apple” ≠ “apple”	Use `-f` / `-i`
Custom separator ignored	Whole line sorted	Add `-t`
Locale affects order	Unexpected results	`LC_ALL=C sort`

Real-World Examples

1. Count Occurrences of Each Line

$ sort access.log | uniq -c
     42 192.168.1.100
     31 192.168.1.101
     27 192.168.1.102

2. Top 10 Most Frequent IPs in a Log

$ awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10
   1234 192.168.1.100
    987 192.168.1.101
    456 192.168.1.102
    ...

3. Find Duplicate Lines in a List

$ sort list.txt | uniq -d
apple
banana

4. Find Lines That Appear Only Once

$ sort list.txt | uniq -u
cherry
date

5. Remove Duplicates from a File

$ sort -u file.txt > unique.txt

6. Sort `/etc/passwd` by UID

$ sort -t: -k 3 -n /etc/passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
...

7. Sort Files by Size

$ ls -l | sort -k 5 -n
-rw-r--r-- 1 kronos kronos    123 Jan 15 10:00 small.txt
-rw-r--r-- 1 kronos kronos   4567 Jan 15 10:00 medium.txt
-rw-r--r-- 1 kronos kronos 123456 Jan 15 10:00 large.txt

8. Sort Human-Readable Sizes

$ du -h * | sort -h
4.0K  file1.txt
1.2M  file2.log
500M  video.mp4
2.3G  backup.tar.gz

9. Find Unique Users in a Log

$ awk '{print $3}' auth.log | sort | uniq
alice
bob
charlie

10. Count Logins per User

$ awk '/session opened/ {print $11}' auth.log | sort | uniq -c | sort -rn
     42 alice
     31 bob
     12 charlie

11. Word Frequency Count

$ cat book.txt | tr -s '[:space:]' '\n' | sort | uniq -c | sort -rn | head -20
   5432 the
   3210 and
   2987 of
   ...

12. Find Files with Duplicate Content

$ md5sum * | sort | uniq -w 32 -D
d41d8cd98f00b204e9800998ecf8427e  file1.txt
d41d8cd98f00b204e9800998ecf8427e  file2.txt

13. Sort by Month Name

$ sort -M months.txt
January
February
March
April
...

14. Randomize a List

$ sort -R file.txt

15. Case-Insensitive Deduplication

$ sort -f emails.txt | uniq -i
Alice@example.com
bob@example.com
Charlie@example.com

16. Skip a Prefix Before Comparing

$ cat timestamps.txt
2024-01-15 10:00:00 ERROR disk
2024-01-15 10:00:01 ERROR disk
2024-01-15 10:00:02 ERROR disk

# Ignore the first 20 characters (timestamp)
$ uniq -s 20 timestamps.txt
2024-01-15 10:00:00 ERROR disk

Visual: The sort | uniq Pipeline

┌──────────────────────────────────────────────┐
│              Input file                      │
│                                              │
│  banana                                      │
│  apple                                       │
│  cherry                                      │
│  apple                                       │
│  date                                        │
│  banana                                      │
│  apple                                       │
│                                              │
└─────────────────┬────────────────────────────┘
                  │
                  │  sort
                  ▼
┌──────────────────────────────────────────────┐
│           Sorted output                      │
│                                              │
│  apple      ─┐                               │
│  apple       │ duplicates adjacent!          │
│  apple      ─┘                               │
│  banana     ─┐                               │
│  banana     ─┘                               │
│  cherry                                      │
│  date                                        │
│                                              │
└─────────────────┬────────────────────────────┘
                  │
                  │  uniq -c
                  ▼
┌──────────────────────────────────────────────┐
│           Counted output                     │
│                                              │
│  3 apple                                     │
│  2 banana                                    │
│  1 cherry                                    │
│  1 date                                      │
│                                              │
└─────────────────┬────────────────────────────┘
                  │
                  │  sort -rn
                  ▼
┌──────────────────────────────────────────────┐
│           Frequency sorted                   │
│                                              │
│  3 apple                                     │
│  2 banana                                    │
│  1 cherry                                    │
│  1 date                                      │
│                                              │
└──────────────────────────────────────────────┘

Summary

Command	Purpose	Example
`sort FILE`	Alphabetical sort	`sort fruit.txt`
`sort -r FILE`	Reverse	`sort -r fruit.txt`
`sort -n FILE`	Numeric	`sort -n numbers.txt`
`sort -f FILE`	Case-insensitive	`sort -f mixed.txt`
`sort -u FILE`	Sort + dedupe	`sort -u fruit.txt`
`sort -o OUT IN`	Output to file	`sort -o sorted.txt fruit.txt`
`sort -c FILE`	Check sorted	`sort -c fruit.txt`
`sort -k N FILE`	Sort by field	`sort -k 2 -n scores.txt`
`sort -t SEP FILE`	Custom separator	`sort -t: -k 3 /etc/passwd`
`sort -h FILE`	Human sizes	`du -h * \| sort -h`
`sort -M FILE`	Month names	`sort -M months.txt`
`sort -R FILE`	Random	`sort -R file.txt`
`uniq FILE`	Collapse adjacent dupes	`sort f \| uniq`
`uniq -c FILE`	Count occurrences	`sort f \| uniq -c`
`uniq -d FILE`	Only duplicates	`sort f \| uniq -d`
`uniq -u FILE`	Only unique	`sort f \| uniq -u`
`uniq -i FILE`	Case-insensitive	`sort f \| uniq -i`
`uniq -s N FILE`	Skip N chars	`uniq -s 2 codes.txt`
`uniq -f N FILE`	Skip N fields	`uniq -f 1 logs.txt`

Key takeaways:

sort arranges lines; uniq collapses adjacent duplicates
uniq only works on sorted data — always sort first
Use sort -u to sort and dedupe in one step
Use sort -n for numbers (otherwise 10 sorts before 2)
Use -o to write back to the same file safely
Use -c to check whether a file is already sorted
Use -k N and -t SEP for field-based sorting
Use -f / -i for case-insensitive work
The classic pipeline: sort FILE | uniq -c | sort -rn → frequency-sorted counts
Add head for a top-N report
Use uniq -d to find duplicates, uniq -u to find singletons
Use -s N or -f N to ignore prefixes or fields when comparing

Remember: sort and uniq are the bread and butter of text processing on Linux. Almost every log analysis one-liner includes them. The golden rule: sort before uniq — otherwise uniq silently does nothing useful. Master sort -n, sort -k, sort -u, and uniq -c, and you can count, rank, dedupe, and filter any list of lines in a single pipeline. Combine them with grep, awk, and cut, and you have a full text-processing toolkit.

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!