萬盛學電腦網

您现在的位置: 萬盛學電腦網 >> 程式設計 >  >> 網路程式設計 >> 程式語言綜合

Python urllib2 獲取網路資源完整指南:Request/Response、表單資料、Headers、異常處理與 Proxy

前言:urllib2 定位

urllib2 是 Python 2 標準庫的高階 HTTP 客戶端,**Python 3 已整合為 urllib.request**。本文以 Python 2 語法為主,並標註 Python 3 對應寫法。現代專案建議直接使用 **requests** 庫,但理解 urllib2 有助於閱讀舊代碼與無依賴環境。

最簡單請求

# Python 2
import urllib2
response = urllib2.urlopen('https://python.org/')
html = response.read()

# Python 3
import urllib.request
with urllib.request.urlopen('https://python.org/') as response:
    html = response.read().decode('utf-8')

Request 物件:控制 Header、Method、Data

# Python 2
req = urllib2.Request('https://www.example.com/api')
req.add_header('User-Agent', 'MyBot/1.0')
req.add_header('Accept', 'application/json')
response = urllib2.urlopen(req)

# Python 3
req = urllib.request.Request('https://www.example.com/api', headers={
    'User-Agent': 'MyBot/1.0',
    'Accept': 'application/json'
})
with urllib.request.urlopen(req) as response:
    data = response.read()

POST 表單資料 (application/x-www-form-urlencoded)

# Python 2
import urllib, urllib2
url = 'https://www.example.com/login'
values = {'username': 'user', 'password': 'pass', 'remember': '1'}
data = urllib.urlencode(values)  # 編碼為 key=val&key=val
req = urllib2.Request(url, data) # 有 data 參數自動變 POST
response = urllib2.urlopen(req)

# Python 3
import urllib.parse, urllib.request
data = urllib.parse.urlencode(values).encode('utf-8') # 必須 bytes
req = urllib.request.Request(url, data=data, method='POST')
with urllib.request.urlopen(req) as response:
    html = response.read().decode('utf-8')

自訂 Header:模擬瀏覽器、API Token、Referer

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
    'Referer': 'https://www.example.com/',
    'Authorization': 'Bearer YOUR_TOKEN',
    'Content-Type': 'application/json'
}
# Python 2: req = urllib2.Request(url, data, headers)
# Python 3: req = urllib.request.Request(url, data=data, headers=headers, method='POST')

異常處理體系

例外類別繼承觸發情境關鍵屬性
URLErrorEnvironmentError網路不通、DNS 失敗、連線拒絕、逾時reason (錯誤碼、訊息)
HTTPErrorURLErrorHTTP 狀態碼 4xx/5xxcode (int), read() 取得錯誤頁面, headers
# Python 2
try:
    response = urllib2.urlopen(req)
except urllib2.HTTPError as e:
    print('HTTP 錯誤:', e.code)
    print('錯誤內容:', e.read())
except urllib2.URLError as e:
    print('網路錯誤:', e.reason)
except Exception as e:
    print('其他錯誤:', e)

# Python 3
try:
    with urllib.request.urlopen(req) as response:
        html = response.read()
except urllib.error.HTTPError as e:
    print(f'HTTP {e.code}: {e.reason}')
    print(e.read().decode('utf-8', 'ignore'))
except urllib.error.URLError as e:
    print(f'網路錯誤: {e.reason}')

Response 物件實用方法

  • read():讀取全部內容 (bytes)。
  • readline(), readlines():逐行讀取。
  • info():回傳 httplib.HTTPMessage,可 headers.get('Content-Type')。
  • geturl():最終 URL (經過重導向後)。
  • getcode():HTTP 狀態碼 (200, 302, 404...)。

進階:Handler 與 Opener — Proxy、Cookie、Basic Auth、SSL

# Python 2: 建立支援 Proxy + Cookie + HTTP Basic Auth 的 Opener
import urllib2, cookielib

# 1. Cookie 容器
cj = cookielib.CookieJar()
cookie_handler = urllib2.HTTPCookieProcessor(cj)

# 2. Proxy
proxy_handler = urllib2.ProxyHandler({'http': 'https://proxy.example.com:8080'})

# 3. Basic Auth
auth_handler = urllib2.HTTPBasicAuthHandler()
auth_handler.add_password(realm='Protected', uri='https://example.com', user='usr', passwd='pwd')

# 4. 建立 Opener
opener = urllib2.build_opener(proxy_handler, cookie_handler, auth_handler)
urllib2.install_opener(opener) # 設為全域預設

# 後續 urlopen 自動攜帶 Cookie、走 Proxy、處理 Basic Auth
response = urllib2.urlopen('https://example.com/protected')
Python 3 對應:urllib.request.ProxyHandler, http.cookiejar.CookieJar, urllib.request.HTTPBasicAuthHandler, build_opener。

HTTPS 與 SSL 驗證

  • Python 2.7.9+ / 3.4+ **預設驗證憑證**。
  • 自簽憑證/測試環境需跳過驗證 (不建議生產):
    import ssl
    ctx = ssl.create_default_context()
    ctx.check_hostname = False
    ctx.verify_mode = ssl.CERT_NONE
    response = urllib2.urlopen('https://self-signed.example.com', context=ctx) # Py2
    # Py3: urllib.request.urlopen(req, context=ctx)

超時控制

# Python 2.6+ / 3 全支援 timeout 參數 (秒)
response = urllib2.urlopen(req, timeout=10)
# Python 3
response = urllib.request.urlopen(req, timeout=10)

從 urllib2 遷移到 requests (強烈建議)

# pip install requests
import requests

# GET
r = requests.get('https://example.com/api', params={'q': 'python'}, timeout=10, headers={'User-Agent': 'MyBot'})

# POST JSON
r = requests.post('https://example.com/api', json={'key': 'value'}, timeout=10)

# Session (自動 Cookie、連線池)
s = requests.Session()
s.headers.update({'Authorization': 'Bearer token'})
r = s.get('https://example.com/user')

# 異常處理
try:
    r.raise_for_status() # 非 2xx 拋出 HTTPError
except requests.HTTPError as e:
    print(e.response.status_code, e.response.text)

# 串流下載大檔案
with requests.get(url, stream=True) as r:
    r.raise_for_status()
    with open('large.zip', 'wb') as f:
        for chunk in r.iter_content(chunk_size=8192):
            f.write(chunk)

結語

urllib2/urllib.request 提供**零依賴**的 HTTP 客戶端能力,核心在於 **Request 物件封裝 Header/Data/Method**、**Opener 鏈組合 Handler (Proxy/Cookie/Auth)**、**分層異常處理**。現代開發請優先採用 **requests**,API 更人性化、支援 HTTP/2、連線池、串流、自動解碼、完整測試生態。

copyright © 萬盛學電腦網 all rights reserved