| 需求 | 推薦方案 |
|---|---|
| 簡單 GET/POST | file_get_contents() (需 allow_url_fopen=On)、cURL |
| 複雜請求 (Header、Cookie、Proxy、重導向、並發) | GuzzleHttp/Client (Composer 套件) |
| HTML 解析 | DOMDocument + DOMXPath (內建)、symfony/dom-crawler, paquettg/php-html-parser |
| 非同步/高併發 | amphp/http-client, react/http-client, Swoole 協程 |
| JavaScript 渲染頁面 (SPA) | spatie/browsershot (Headless Chrome)、puppeteer-php |
composer require guzzlehttp/guzzle symfony/dom-crawler symfony/css-selector
<?php
require 'vendor/autoload.php';
use GuzzleHttp\Client;
use Symfony\Component\DomCrawler\Crawler;
$client = new Client([
'timeout' => 10,
'headers' => ['User-Agent' => 'Mozilla/5.0 (compatible; MyBot/1.0)'],
'cookies' => true, // 自動處理 Cookie
]);
try {
// 1. 取得列表頁
$resp = $client->get('https://example.com/books');
$html = $resp->getBody()->getContents();
// 2. 解析列表連結
$crawler = new Crawler($html);
$links = $crawler->filter('.book-item > a')->links(); // CSS 選擇器
foreach ($links as $link) {
$detailUrl = $link->getUri();
echo "採集: $detailUrl\n";
// 3. 取得詳情頁
$detailResp = $client->get($detailUrl);
$detailHtml = $detailResp->getBody()->getContents();
$detail = new Crawler($detailHtml);
// 4. 提取結構化資料
$data = [
'title' => $detail->filter('h1.title')->text(),
'author' => $detail->filter('.author')->text(),
'price' => $detail->filter('.price')->text(),
'content' => $detail->filter('#content')->html(), // 保留 HTML
'source_url' => $detailUrl,
'crawled_at' => date('c'),
];
// 5. 儲存 (示範存 JSON Lines)
file_put_contents('books.jsonl', json_encode($data, JSON_UNESCAPED_UNICODE) . "\n", FILE_APPEND);
// 禮貌延遲
usleep(500_000); // 0.5 秒
}
} catch (GuzzleHttp\Exception\RequestException $e) {
error_log("請求失敗: " . $e->getMessage());
}
?>
robots.txt、Crawl-delay、頻率限制,**合法合規採集**。html_entity_decode)。mb_convert_encoding)。robots.txt、服務條款 (ToS)。大型採集專案建議採用:
cron + supervisor 或 Laravel Scheduler / Symfony Messenger。