萬盛學電腦網

您现在的位置: 萬盛學電腦網 >> 程式設計 >  >> 網路程式設計 >> 程式語言綜合

Python 網路請求完整指南:urllib、urllib2、requests、httpx 選型與實戰

Python 網路請求生態歷經 `urllib`/`urllib2` (Python 2)、`urllib.request` (Python 3 標準庫)、`requests` (事實標準) 到現代異步 `httpx`/`aiohttp`。本文系統對比各庫特性,提供從同步到異步的完整實戰範本。

庫選型指南

requests 核心用法 (同步標準)


import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

# 會話 + 連線池 + 重試策略 (生產環境標配)
session = requests.Session()
retry = Retry(total=3, backoff_factor=0.5, status_forcelist=[500, 502, 503, 504])
adapter = HTTPAdapter(max_retries=retry, pool_connections=10, pool_maxsize=20)
session.mount('https://', adapter)
session.mount('https://', adapter)

# 基本請求
resp = session.get('https://api.example.com/users', params={'page': 1}, timeout=10)
resp.raise_for_status()  # 非 2xx 拋異常
data = resp.json()

# POST JSON
resp = session.post('https://api.example.com/users', 
                    json={'name': 'Alice'}, 
                    headers={'Authorization': 'Bearer token'},
                    timeout=(5, 30))  # (連線超時, 讀取超時)

# 串流下載大檔案
with session.get('https://example.com/large.zip', stream=True) as r:
    r.raise_for_status()
    with open('large.zip', 'wb') as f:
        for chunk in r.iter_content(chunk_size=8192):
            f.write(chunk)

# 檔案上傳
files = {'file': open('report.pdf', 'rb')}
session.post('https://api.example.com/upload', files=files)

httpx 異步用法 (現代標準)


import httpx
import asyncio

async def fetch_all(urls: list):
    # 連線池限制並發
    limits = httpx.Limits(max_connections=100, max_keepalive_connections=20)
    timeout = httpx.Timeout(10.0, connect=5.0)
    
    async with httpx.AsyncClient(limits=limits, timeout=timeout) as client:
        tasks = [client.get(url) for url in urls]
        responses = await asyncio.gather(*tasks, return_exceptions=True)
    
    results = []
    for url, resp in zip(urls, responses):
        if isinstance(resp, Exception):
            results.append({'url': url, 'error': str(resp)})
        else:
            resp.raise_for_status()
            results.append({'url': url, 'data': resp.json()})
    return results

# 執行
results = asyncio.run(fetch_all([
    'https://api.github.com/users/octocat',
    'https://api.github.com/users/torvalds',
]))

進階模式

認證與攔截器


# requests: AuthBase 子類
class TokenAuth(requests.auth.AuthBase):
    def __init__(self, token): self.token = token
    def __call__(self, r):
        r.headers['Authorization'] = f'Bearer {self.token}'
        return r

session.auth = TokenAuth('my-token')

# httpx: event hooks
async def auth_hook(request):
    request.headers['Authorization'] = 'Bearer my-token'

client = httpx.AsyncClient(event_hooks={'request': [auth_hook]})

代理設定


# requests
proxies = {'http': 'https://proxy:8080', 'https': 'https://proxy:8080'}
session.get(url, proxies=proxies)

# httpx
client = httpx.AsyncClient(proxy='https://proxy:8080')
# 或環境變數 HTTP_PROXY/HTTPS_PROXY

Cookie 持久化


# requests: Session 自動處理
session.get('https://site.com/login', data={'user': 'x', 'pass': 'y'})
session.get('https://site.com/profile')  # 自動攜帶 Cookie

# httpx: 同樣
client = httpx.AsyncClient()
await client.post('/login', data={...})
await client.get('/profile')

SSL/憑證處理


# 忽略驗證 (僅測試/內網)
session.get(url, verify=False)  # requests
client = httpx.AsyncClient(verify=False)  # httpx

# 指定 CA 憑證
session.get(url, verify='/path/to/ca-bundle.crt')

# 客戶端憑證 (mTLS)
session.get(url, cert=('/path/client.crt', '/path/client.key'))

urllib.request 基礎用法 (無依賴環境)


import urllib.request, urllib.parse, json, ssl

# 忽略 SSL 驗證
ssl._create_default_https_context = ssl._create_unverified_context

# GET
url = 'https://api.example.com/users?page=1'
req = urllib.request.Request(url, headers={'User-Agent': 'MyApp/1.0'})
with urllib.request.urlopen(req, timeout=10) as resp:
    data = json.load(resp)

# POST JSON
url = 'https://api.example.com/users'
data = json.dumps({'name': 'Alice'}).encode('utf-8')
req = urllib.request.Request(url, data=data, headers={
    'Content-Type': 'application/json',
    'User-Agent': 'MyApp/1.0'
}, method='POST')
with urllib.request.urlopen(req) as resp:
    result = json.load(resp)

# 代理
proxy = urllib.request.ProxyHandler({'http': 'https://proxy:8080'})
opener = urllib.request.build_opener(proxy)
urllib.request.install_opener(opener)

效能優化清單

  • **連線複用**:`requests.Session` / `httpx.Client` / `AsyncClient` 重用 TCP 連線 (Keep-Alive)。
  • **連線池調整**:`pool_connections` (池數量)、`pool_maxsize` (每池連線數) 據併發量設定。
  • **超時務必設定**:連線超時 + 讀取超時,防止卡死。
  • **壓縮支援**:自動 `Accept-Encoding: gzip, deflate`,確保 `Content-Encoding` 解壓。
  • **並發控制**:`asyncio.Semaphore` 限制同時請求數,避免觸發限流或耗盡檔案描述符。
  • **快取**:`requests-cache` 或 `httpx-cache` 透明快取 GET 請求。

錯誤處理模式


import requests
from requests.exceptions import RequestException, Timeout, ConnectionError, HTTPError

try:
    resp = session.get(url, timeout=10)
    resp.raise_for_status()
except Timeout:
    log("請求超時")
except ConnectionError:
    log("連線失敗 (DNS/TCP/SSL)")
except HTTPError as e:
    log(f"HTTP 錯誤: {e.response.status_code}")
except RequestException as e:
    log(f"其他請求異常: {e}")

總結建議

  • **新專案**:直接用 `httpx` (同步 `httpx.Client`、異步 `httpx.AsyncClient`),API 現代、原生異步、HTTP/2 支援。
  • **既有專案**:`requests` 穩定成熟,無需強行遷移。
  • **無依賴限制**:`urllib.request` 配合手工封裝。
  • **高併發爬蟲**:`aiohttp` + `asyncio` + `Semaphore` + `aiofiles` 異步寫檔。
copyright © 萬盛學電腦網 all rights reserved
庫類型Python 版本特點推薦場景
urllib.request同步/標準庫3.0+零依賴、API 繁瑣、無連線池極簡環境、無法安裝第三方包
requests同步/第三方2.7+/3.5+API 友善、連線池、自動解碼、生態豐富絕大多數同步專案首選
httpx同步/異步3.6+API 與 requests 一致、原生 async、HTTP/2新專案、需異步、FastAPI/Starlette
aiohttp異步/第三方3.5+成熟異步、WebSocket、服務端支援高併發爬蟲、WebSocket 客戶端