關鍵限制:uniq 僅比較相鄰行。非相鄰的重複行不會被識別,因此實務上常搭配 sort 先排序再去重。
uniq [選項] [輸入檔] [輸出檔]
# 輸入用 - 表示標準輸入,輸出檔不可與輸入檔相同
| 選項 | 功能 | 典型場景 |
|---|---|---|
-c | 每行行首顯示重複次數 | 統計頻率分佈 |
-d | 僅顯示重複行(每組一次) | 找出重複項 |
-D | 顯示所有重複行(含所有副本) | 審計完整重複集合 |
-u | 僅顯示唯一行(不重複) | 找出單一出現項 |
-i | 忽略大小寫比較 | 不區分大小寫去重 |
-f N | 忽略前 N 個欄位(以空白分隔) | 按特定欄位去重 |
-s N | 忽略前 N 個字符 | 按字符位置去重 |
-w N | 比較前 N 個字符 | 前綴匹配去重 |
boy took bat home
boy took bat home
girl took bat home
boy took bat home
boy took bat home
dog brought hat home
dog brought hat home
dog brought hat home
$ uniq test.txt
boy took bat home
girl took bat home
boy took bat home
dog brought hat home
注意:第 1-2 行合併,第 3 行獨立,第 4-5 行合併,第 6-8 行合併。非相鄰的 "boy..." 仍保留兩組。
$ uniq -c test.txt
2 boy took bat home
1 girl took bat home
2 boy took bat home
3 dog brought hat home
$ sort test.txt | uniq -c
4 boy took bat home
3 dog brought hat home
1 girl took bat home
黃金組合:sort | uniq -c | sort -nr 可得出頻率降序排行。
# 僅顯示有重複的行(每組一次)
$ uniq -d test.txt
boy took bat home
dog brought hat home
# 顯示所有重複行副本
$ uniq -D test.txt
boy took bat home
boy took bat home
boy took bat home
boy took bat home
dog brought hat home
dog brought hat home
dog brought hat home
# 僅顯示不重複的行
$ uniq -u test.txt
girl took bat home
# 忽略前 1 個欄位(boy/girl/dog),比較 "took bat home..."
$ uniq -f 1 test.txt
boy took bat home
dog brought hat home
# 忽略前 2 個欄位 + 忽略第 3 個欄位首字符
$ uniq -f 2 -s 2 test.txt
boy took bat home
# 比較內容實為 "at home",故全部視為重複
# 提取錯誤行 -> 排序 -> 計數 -> 取前 20
grep "ERROR" /var/log/app.log | sort | uniq -c | sort -nr | head -20
# 假設日誌格式:IP 方法 URL 狀態碼
# 統計獨立 IP 訪問次數
awk '{print $1}' access.log | sort | uniq -c | sort -nr
# 統計每個 URL 的獨立訪客數
awk '{print $3, $1}' access.log | sort -u | awk '{print $1}' | sort | uniq -c | sort -nr
# 保留順序同時去重(awk 方案)
awk '!seen[$0]++' config.conf > config.conf.dedup
# 或排序後去重(不保留原順序)
sort config.conf | uniq > config.conf.dedup
# 以第 2 欄位(用戶ID)去重,保留首次出現
sort -t',' -k2,2 users.csv | uniq -f 1
| 需求 | 命令管道 |
|---|---|
| 統計單詞頻率 | |
| 找出重複行並顯示行號 | |
| 比較兩檔案差異行 | |
| 統計空行數量 |
sort -u 替代 sort | uniq(內建去重,少一個管道)LC_ALL=C 可大幅加速排序:LC_ALL=C sort bigfile | uniq -csort -T /tmp 指定臨時目錄