大量檔案下載時,同步阻塞方式效率極低。本文提供兩種方案:使用 gevent 協程池實現高併發下載,以及純標準庫 urllib 的備選寫法,並討論檔名解析、斷點續傳等實務細節。
#!/usr/bin/env python3
from gevent import monkey; monkey.patch_all()
from gevent.pool import Pool
import requests, sys, os
def download(url):
headers = {'User-Agent': 'Mozilla/5.0 ...'}
filename = url.split('/')[-1].strip()
with requests.get(url, headers=headers, stream=True) as r:
r.raise_for_status()
with open(filename, 'wb') as f:
for chunk in r.iter_content(1024):
if chunk:
f.write(chunk)
print(f"{filename} done")
def remove_line(key, filename):
"""從 URL 清單移除已下載行 (簡易去重)"""
os.system(f"sed -i '/{key}/d' {filename}")
if __name__ == '__main__':
if len(sys.argv) != 2:
print(f"Usage: python {sys.argv[0]} urls.txt")
sys.exit(1)
url_file = sys.argv[1]
pool = Pool(4) # 併發數
with open(url_file) as f:
for line in f:
url = line.strip()
if url:
pool.spawn(download, url)
remove_line(url.split('/')[-1], url_file)
pool.join()
from urllib.request import urlopen, Request
from urllib.parse import urlsplit
from os.path import basename
def url2name(url):
return basename(urlsplit(url).path)
def download(url, dest_name=None):
name = dest_name or url2name(url)
req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
with urlopen(req) as resp:
# 處理 Content-Disposition 或重導向後的檔名
cd = resp.headers.get('Content-Disposition')
if cd and 'filename=' in cd:
name = cd.split('filename=')[1].strip('"\'')
elif resp.url != url:
name = url2name(resp.url)
with open(name, 'wb') as f:
f.write(resp.read())
print(f"{name} done")
Range 標頭與 open(filename, 'ab') 追加寫入。Content-Length 或計算 MD5/SHA256。tenacity 或自訂重試邏輯 (指數退避)。gevent.sleep() 或 Token Bucket 演算法控制頻寬。tqdm 整合 iter_content 顯示即時進度。下載 100 個 1MB 檔案 (總 100MB) 於 100Mbps 環境:
| 方案 | 併發數 | 耗時 |
|---|---|---|
| 同步 requests | 1 | ~85 秒 |
| gevent Pool(10) | 10 | ~12 秒 |
| gevent Pool(50) | 50 | ~10 秒 (受限頻寬) |
依專案依賴限制選擇方案;gevent 方案在高併發下表現最佳,且程式碼簡潔。