從原始字串分詞、標點移除、小寫正規化、建立詞頻字典、排序輸出,完整展示文字探勘前處理管線核心步驟。
str1 = """Man who run in front of car, get tired.
Man who run behind car, get exhausted."""
# 1. 分詞 (以空白分隔)
word_list1 = str1.split()
# 2. 移除尾部標點、小寫正規化
word_list2 = []
punct = {',', '.', '!', '?', ';'}
for w in word_list1:
if w[-1] in punct:
w = w[:-1]
word_list2.append(w.lower())
# 3. 統計詞頻
freq = {}
for w in word_list2:
freq[w] = freq.get(w, 0) + 1
# 4. 排序輸出
for w in sorted(freq):
print(f'{w:<10} {freq[w]}')
behind 1 car 2 exhausted 1 front 1 get 2 man 2 of 1 run 2 tired 1 who 2
import re
from collections import Counter
text = """Man who run in front of car, get tired.
Man who run behind car, get exhausted."""
# 正則分詞:保留字母數字下劃線,其餘作為分隔
words = re.findall(r'\w+', text.lower())
freq = Counter(words)
for w, c in freq.most_common():
print(f'{w:<10} {c}')
jieba、pkuseg;英文 nltk.word_tokenize。nltk.corpus.stopwords 或自訂集合。nltk.PorterStemmer / WordNetLemmatizer。sklearn.feature_extraction.text.TfidfVectorizer。flatMap -> map -> reduceByKey。Count-Min Sketch、HyperLogLog (基數估計)。