Python 標準庫 concurrent.futures.ThreadPoolExecutor 已經足夠強大,但在教學、嵌入式環境或需要深度定制調度邏輯(如優先級隊列、動態伸縮、任務熔斷)時,理解底層機制並能手寫一個輕量級線程池極具價值。本文通過一個精簡且完整的實現,拆解「任務隊列 + 工作線程 + 同步原語」的核心模型。
TaskInfo 承載可調用對象與參數。TaskList 內部用 Lock 保護列表,用 Semaphore 實現「生產-消費」信號量同步。WorkerThread 循環 fetch -> execute,守護線程模式隨主進程退出。ThreadPool 批量啟動/管理工作線程。import threading
import time
import signal
import os
from typing import Callable, Any, Optional
class TaskInfo:
"""任務容器:封裝函數與最多三個位置參數"""
__slots__ = ("func", "arg0", "arg1", "arg2")
def __init__(self, func: Callable, *args):
self.func = func
self.arg0 = args[0] if len(args) > 0 else None
self.arg1 = args[1] if len(args) > 1 else None
self.arg2 = args[2] if len(args) > 2 else None
class TaskQueue:
"""線程安全任務隊列:Lock + Semaphore 經典組合"""
def __init__(self):
self._queue = []
self._lock = threading.Lock()
self._sem = threading.Semaphore(0)
def put(self, task: TaskInfo):
with self._lock:
self._queue.append(task)
self._sem.release()
def get(self) -> TaskInfo:
self._sem.acquire()
with self._lock:
return self._queue.pop(0)
def __len__(self):
with self._lock:
return len(self._queue)
class WorkerThread(threading.Thread):
"""工作線程:持續從隊列取任務並執行"""
def __init__(self, queue: TaskQueue, name: str = None):
super().__init__(name=name, daemon=True)
self._queue = queue
def run(self):
while True:
task = self._queue.get()
try:
task.func(task.arg0, task.arg1, task.arg2)
except Exception as e:
print(f"[Worker {self.name}] Task failed: {e}")
class ThreadPool:
"""線程池:生命週期管理"""
def __init__(self, worker_count: int, queue: TaskQueue):
self._workers = [WorkerThread(queue, f"Worker-{i}") for i in range(worker_count)]
def start(self):
for w in self._workers:
w.start()
def join(self, timeout: float = None):
for w in self._workers:
w.join(timeout)
# -------------------- Demo --------------------
def demo_task(idx: int, tag: str = "", extra: Any = None):
print(f"[{threading.current_thread().name}] Task {idx} {tag} {extra}")
def shutdown(signum, frame):
print(f"\nReceived signal {signum}, exiting...")
os._exit(0)
if __name__ == "__main__":
signal.signal(signal.SIGINT, shutdown)
signal.signal(signal.SIGTERM, shutdown)
q = TaskQueue()
pool = ThreadPool(6, q)
pool.start()
cnt = 0
try:
while True:
q.put(TaskInfo(demo_task, cnt, "demo", time.time()))
cnt += 1
time.sleep(2)
except KeyboardInterrupt:
shutdown(signal.SIGINT, None)
Semaphore 語義更清晰:release() 對應「生產」,acquire() 對應「消費」。若需支援「有界隊列」或「優先級」,可改用 Condition 配合 heapq 實現。
工作線程內部 try/except 確保單個任務崩潰不影響線程存活,生產環境建議接入結構化日誌(如 logging + traceback)。
示例採用 os._exit 硬退出;工程化方案應引入「毒丸任務」或 threading.Event 通知線程退出循環,並等待隊列清空。
concurrent.futures.Future 兼容標準庫 API。contextvars 實現請求級 Trace ID 透傳。不到 80 行核心代碼實現了生產級線程池雛形,理解其「隊列+信號量+線程循環」範式,能幫你在面對高併發、異步化改造時遊刃有餘。下一步嘗試加入「任務超時控制」與「指標暴露」吧!