Python 網路請求生態歷經 `urllib`/`urllib2` (Python 2)、`urllib.request` (Python 3 標準庫)、`requests` (事實標準) 到現代異步 `httpx`/`aiohttp`。本文系統對比各庫特性,提供從同步到異步的完整實戰範本。
| 庫 | 類型 | Python 版本 | 特點 | 推薦場景 |
urllib.request | 同步/標準庫 | 3.0+ | 零依賴、API 繁瑣、無連線池 | 極簡環境、無法安裝第三方包 |
| requests | 同步/第三方 | 2.7+/3.5+ | API 友善、連線池、自動解碼、生態豐富 | 絕大多數同步專案首選 |
httpx | 同步/異步 | 3.6+ | API 與 requests 一致、原生 async、HTTP/2 | 新專案、需異步、FastAPI/Starlette |
aiohttp | 異步/第三方 | 3.5+ | 成熟異步、WebSocket、服務端支援 | 高併發爬蟲、WebSocket 客戶端 |
requests 核心用法 (同步標準)
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
# 會話 + 連線池 + 重試策略 (生產環境標配)
session = requests.Session()
retry = Retry(total=3, backoff_factor=0.5, status_forcelist=[500, 502, 503, 504])
adapter = HTTPAdapter(max_retries=retry, pool_connections=10, pool_maxsize=20)
session.mount('https://', adapter)
session.mount('https://', adapter)
# 基本請求
resp = session.get('https://api.example.com/users', params={'page': 1}, timeout=10)
resp.raise_for_status() # 非 2xx 拋異常
data = resp.json()
# POST JSON
resp = session.post('https://api.example.com/users',
json={'name': 'Alice'},
headers={'Authorization': 'Bearer token'},
timeout=(5, 30)) # (連線超時, 讀取超時)
# 串流下載大檔案
with session.get('https://example.com/large.zip', stream=True) as r:
r.raise_for_status()
with open('large.zip', 'wb') as f:
for chunk in r.iter_content(chunk_size=8192):
f.write(chunk)
# 檔案上傳
files = {'file': open('report.pdf', 'rb')}
session.post('https://api.example.com/upload', files=files)
httpx 異步用法 (現代標準)
import httpx
import asyncio
async def fetch_all(urls: list):
# 連線池限制並發
limits = httpx.Limits(max_connections=100, max_keepalive_connections=20)
timeout = httpx.Timeout(10.0, connect=5.0)
async with httpx.AsyncClient(limits=limits, timeout=timeout) as client:
tasks = [client.get(url) for url in urls]
responses = await asyncio.gather(*tasks, return_exceptions=True)
results = []
for url, resp in zip(urls, responses):
if isinstance(resp, Exception):
results.append({'url': url, 'error': str(resp)})
else:
resp.raise_for_status()
results.append({'url': url, 'data': resp.json()})
return results
# 執行
results = asyncio.run(fetch_all([
'https://api.github.com/users/octocat',
'https://api.github.com/users/torvalds',
]))
進階模式
認證與攔截器
# requests: AuthBase 子類
class TokenAuth(requests.auth.AuthBase):
def __init__(self, token): self.token = token
def __call__(self, r):
r.headers['Authorization'] = f'Bearer {self.token}'
return r
session.auth = TokenAuth('my-token')
# httpx: event hooks
async def auth_hook(request):
request.headers['Authorization'] = 'Bearer my-token'
client = httpx.AsyncClient(event_hooks={'request': [auth_hook]})
代理設定
# requests
proxies = {'http': 'https://proxy:8080', 'https': 'https://proxy:8080'}
session.get(url, proxies=proxies)
# httpx
client = httpx.AsyncClient(proxy='https://proxy:8080')
# 或環境變數 HTTP_PROXY/HTTPS_PROXY
Cookie 持久化
# requests: Session 自動處理
session.get('https://site.com/login', data={'user': 'x', 'pass': 'y'})
session.get('https://site.com/profile') # 自動攜帶 Cookie
# httpx: 同樣
client = httpx.AsyncClient()
await client.post('/login', data={...})
await client.get('/profile')
SSL/憑證處理
# 忽略驗證 (僅測試/內網)
session.get(url, verify=False) # requests
client = httpx.AsyncClient(verify=False) # httpx
# 指定 CA 憑證
session.get(url, verify='/path/to/ca-bundle.crt')
# 客戶端憑證 (mTLS)
session.get(url, cert=('/path/client.crt', '/path/client.key'))
urllib.request 基礎用法 (無依賴環境)
import urllib.request, urllib.parse, json, ssl
# 忽略 SSL 驗證
ssl._create_default_https_context = ssl._create_unverified_context
# GET
url = 'https://api.example.com/users?page=1'
req = urllib.request.Request(url, headers={'User-Agent': 'MyApp/1.0'})
with urllib.request.urlopen(req, timeout=10) as resp:
data = json.load(resp)
# POST JSON
url = 'https://api.example.com/users'
data = json.dumps({'name': 'Alice'}).encode('utf-8')
req = urllib.request.Request(url, data=data, headers={
'Content-Type': 'application/json',
'User-Agent': 'MyApp/1.0'
}, method='POST')
with urllib.request.urlopen(req) as resp:
result = json.load(resp)
# 代理
proxy = urllib.request.ProxyHandler({'http': 'https://proxy:8080'})
opener = urllib.request.build_opener(proxy)
urllib.request.install_opener(opener)
效能優化清單
- **連線複用**:`requests.Session` / `httpx.Client` / `AsyncClient` 重用 TCP 連線 (Keep-Alive)。
- **連線池調整**:`pool_connections` (池數量)、`pool_maxsize` (每池連線數) 據併發量設定。
- **超時務必設定**:連線超時 + 讀取超時,防止卡死。
- **壓縮支援**:自動 `Accept-Encoding: gzip, deflate`,確保 `Content-Encoding` 解壓。
- **並發控制**:`asyncio.Semaphore` 限制同時請求數,避免觸發限流或耗盡檔案描述符。
- **快取**:`requests-cache` 或 `httpx-cache` 透明快取 GET 請求。
錯誤處理模式
import requests
from requests.exceptions import RequestException, Timeout, ConnectionError, HTTPError
try:
resp = session.get(url, timeout=10)
resp.raise_for_status()
except Timeout:
log("請求超時")
except ConnectionError:
log("連線失敗 (DNS/TCP/SSL)")
except HTTPError as e:
log(f"HTTP 錯誤: {e.response.status_code}")
except RequestException as e:
log(f"其他請求異常: {e}")
總結建議
- **新專案**:直接用 `httpx` (同步 `httpx.Client`、異步 `httpx.AsyncClient`),API 現代、原生異步、HTTP/2 支援。
- **既有專案**:`requests` 穩定成熟,無需強行遷移。
- **無依賴限制**:`urllib.request` 配合手工封裝。
- **高併發爬蟲**:`aiohttp` + `asyncio` + `Semaphore` + `aiofiles` 異步寫檔。
copyright ©
萬盛學電腦網 all rights reserved