萬盛學電腦網

您现在的位置: 萬盛學電腦網 >> 程式設計 >  >> 網路程式設計 >> 程式語言綜合

C# 代理 IP 完整實戰指南:HttpClient、WebClient、HttpWebRequest、WebService 三大場景統一封裝

前言

爬蟲、API 整合、跨區域測試常需透過代理 IP 發送 HTTP 請求。.NET 提供 `WebProxy` 類別統一代理設定,但不同 HTTP 客戶端 (`HttpClient`、`HttpWebRequest`、`WebClient`、WCF/WebService) 套用方式各異。本文提供**統一封裝策略**、四大客戶端完整範例、驗證處理、效能優化與陷阱避雷。

核心類別:WebProxy

var proxy = new WebProxy
{
    Address = new Uri("https://proxy.example.com:8080"),
    BypassList = new[] { "localhost", "192.168.*", "*.internal.corp" }, // 繞過清單
    UseDefaultCredentials = false,
    Credentials = new NetworkCredential("username", "password", "domain") // 或 CredentialCache.DefaultNetworkCredentials
};

四大 HTTP 客戶端代理套用方式

1. HttpClient (現代首選 .NET Core 2.1+ / .NET 5+)

// 靜態單例模式 (官方推薦,避免 Socket 耗盡)
private static readonly HttpClient _httpClient = CreateHttpClient();

private static HttpClient CreateHttpClient() {
    var handler = new HttpClientHandler {
        Proxy = new WebProxy("https://proxy:8080") {
            Credentials = new NetworkCredential("user", "pwd"),
            BypassProxyOnLocal = true
        },
        UseProxy = true,
        AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate,
        // .NET 6+ 支援連線池限流
        MaxConnectionsPerServer = 100
    };
    var client = new HttpClient(handler) { Timeout = TimeSpan.FromSeconds(30) };
    return client;
}

// 使用
var response = await _httpClient.GetAsync("https://api.target.com/data");
var json = await response.Content.ReadAsStringAsync();

2. HttpWebRequest (舊版相容、精細控制)

var request = (HttpWebRequest)WebRequest.Create("https://api.target.com/data");
request.Method = "GET";
request.Proxy = new WebProxy("https://proxy:8080") {
    Credentials = new NetworkCredential("user", "pwd")
};
request.ServicePoint.ConnectionLimit = 100; // 連線池

using (var response = (HttpWebResponse)await request.GetResponseAsync())
using (var stream = response.GetResponseStream())
using (var reader = new StreamReader(stream, Encoding.UTF8)) {
    var content = await reader.ReadToEndAsync();
}

3. WebClient (簡易、同步、舊專案常見)

using (var wc = new WebClient()) {
    wc.Proxy = new WebProxy("https://proxy:8080") {
        Credentials = new NetworkCredential("user", "pwd")
    };
    wc.Encoding = Encoding.UTF8;
    wc.Headers[HttpRequestHeader.UserAgent] = "MyBot/1.0";
    string html = wc.DownloadString("https://example.com");
    // 或非同步
    // string html = await wc.DownloadStringTaskAsync(url);
}

4. WebService / WCF Proxy (動態編譯 WSDL 產生的 Proxy 類別)

// 動態編譯 WSDL 產生 Proxy 後,強制轉型為 HttpWebClientProtocol 設定 Proxy
var proxyInstance = Activator.CreateInstance(proxyType); // 從組件取得型別
if (proxyInstance is System.Web.Services.Protocols.HttpWebClientProtocol wsProxy) {
    wsProxy.Proxy = new WebProxy("https://proxy:8080") {
        Credentials = new NetworkCredential("user", "pwd")
    };
    wsProxy.Timeout = 60000; // ms
    // 呼叫方法
    var result = wsProxy.GetType().GetMethod("MyMethod").Invoke(wsProxy, args);
}

統一封裝:Proxy-Aware HttpClientFactory

public interface IProxyPolicy {
    WebProxy GetProxy(Uri requestUri);
}

public class FixedProxyPolicy : IProxyPolicy {
    private readonly WebProxy _proxy;
    public FixedProxyProxy(string address, string user, string pass) {
        _proxy = new WebProxy(address) { Credentials = new NetworkCredential(user, pass) };
    }
    public WebProxy GetProxy(Uri _) => _proxy;
}

public class RotatingProxyPolicy : IProxyPolicy {
    private readonly string[] _proxies;
    private int _index;
    public RotatingProxyPolicy(IEnumerable<string> proxies) { _proxies = proxies.ToArray(); }
    public WebProxy GetProxy(Uri _) {
        var p = _proxies[Interlocked.Increment(ref _index) % _proxies.Length];
        return new WebProxy(p) { Credentials = CredentialCache.DefaultNetworkCredentials };
    }
}

// 擴充 HttpClientFactory
services.AddHttpClient("WithProxy")
    .ConfigurePrimaryHttpMessageHandler(() => new HttpClientHandler {
        Proxy = appServices.GetRequiredService<IProxyPolicy>().GetProxy(null), // 需在請求時動態決定
        UseProxy = true
    });

// 更佳:DelegatingHandler 動態注入 Proxy
public class ProxyDelegatingHandler : DelegatingHandler {
    private readonly IProxyPolicy _policy;
    public ProxyDelegatingHandler(IProxyPolicy policy) { _policy = policy; }
    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken ct) {
        if (request.RequestUri != null) {
            var proxy = _policy.GetProxy(request.RequestUri);
            // HttpClientHandler 不支援每請求動態 Proxy,需自行實作或用 HttpRequestMessage.Properties
            // 實際專案建議:多個 Named HttpClient,各綁定固定 Proxy
        }
        return await base.SendAsync(request, ct);
    }
}

驗證模式完整對照

場景Credentials 設定備註
基本驗證 (Basic Auth)`new NetworkCredential("user", "pwd")`HTTP Header `Authorization: Basic base64(user:pwd)`
NTLM / Kerberos (Windows 整合)`CredentialCache.DefaultNetworkCredentials`自動使用當前登入使用者憑證
指定 Domain`new NetworkCredential("user", "pwd", "DOMAIN")`公司內部 Proxy 常需 Domain
Proxy 鏈 (Chain)不支援原生,需中介轉發 (如 Nginx、Squid、Caddy)架構層面解決

常見坑點與避雷

HttpClient 單例但 Proxy 需輪換
HttpClientHandler 的 Proxy **建構時固定**,無法每請求變更。解法:建立多個 Named HttpClient (`AddHttpClient("proxy1", ...)`),或自訂 `DelegatingHandler` 在 `SendAsync` 中透過 `HttpRequestMessage.Properties["proxy"]` 配合自訂 `HttpMessageHandler` 實現 (複雜)。建議:Proxy 池外層輪換,內層固定。
WebClient 非同步 `DownloadStringTaskAsync` 在 .NET Core 行為異常
`WebClient` 在 .NET Core 為相容性封裝,**強烈建議全面遷移至 `HttpClient`**。
SSL/TLS 憑證驗證失敗 (自簽憑證代理)
`HttpClientHandler.ServerCertificateCustomValidationCallback = (msg, cert, chain, errors) => true;` (僅測試環境)。
代理 IP 失效/封鎖導致請求掛起
設定 `Timeout`、`HttpClientHandler.Proxy` 無回應時無法自動切換。需實作**健康檢查 + 熔斷器 (Polly)** 自動移除失效代理。
免費代理 IP 極不穩定
生產環境**務必使用付費代理服務** (Bright Data, Oxylabs, Smartproxy, ScraperAPI) 或自建代理池。

效能優化清單

  1. 啟用連線池:`HttpClientHandler.MaxConnectionsPerServer` (預設 2,改 100+)
  2. 啟用自動解壓縮:`AutomaticDecompression = GZip | Deflate | Brotli`
  3. 重用 `HttpClient` / `HttpClientHandler` (單例)
  4. 非同步所有 I/O (`await`),避免阻塞執行緒
  5. 大量並發時使用 `SemaphoreSlim` 限制並行度
  6. 回應內容大時使用 `Stream` 串流處理,避免全載入記憶體

結語

C# 使用代理 IP 的核心在於 **`WebProxy` 物件建構** 與 **注入到對應 HTTP 客戶端**。現代 .NET 專案請**統一遷移至 `HttpClient` + `IHttpClientFactory` + `DelegatingHandler` 架構**,集中管理代理策略、重試、熔斷、度量。舊版 `WebClient`/`HttpWebRequest`/`WebService` 專案可參考本文對應範例最小改動接入。記住:**代理品質決定爬蟲成敗**,免費代理僅適合學習測試。

copyright © 萬盛學電腦網 all rights reserved