Pythonmultiprocessing模組提供Pipe(管道)與Queue(隊列)兩種進程間通信(IPC)機制。Pipe適合雙向點對點通信,Queue支援多生產者多消費者模型。本文深度對比兩者特性、API、適用場景並提供完整示例。
from multiprocessing import Pipe, Process
def sender(conn):
for i in range(5):
conn.send(f'msg-{i}')
conn.close()
def receiver(conn):
while True:
try:
print(f'Received: {conn.recv()}')
except EOFError:
break
if __name__ == '__main__':
parent_conn, child_conn = Pipe(duplex=False) # 单向:parent只收,child只发
p1 = Process(target=sender, args=(child_conn,))
p2 = Process(target=receiver, args=(parent_conn,))
p1.start(); p2.start()
p1.join(); p2.join()
duplex=True(默認)雙向;False單向(conn1只收,conn2只發)。Connection對象,支持send()/recv()/poll()。socketpair或命名管道)。from multiprocessing import Queue, Process, Lock
import os, time
def producer(q, id):
for i in range(3):
q.put(f'P{id}-{i}')
time.sleep(0.1)
def consumer(q, lock):
while True:
item = q.get()
if item is None: break # 毒丸結束信號
with lock:
print(f'C{os.getpid()}: {item}')
if __name__ == '__main__':
q = Queue(maxsize=5)
lock = Lock()
producers = [Process(target=producer, args=(q, i)) for i in range(2)]
consumers = [Process(target=consumer, args=(q, lock)) for _ in range(2)]
for p in producers + consumers: p.start()
for p in producers: p.join()
for _ in consumers: q.put(None) # 發送結束信號
for c in consumers: c.join()
maxsize限制,put()/get()可阻塞/超時/非阻塞(put_nowait/get_nowait)。qsize()/empty()/full()在多進程下不可靠(僅供參考)。close()/join_thread()管理後台傳輸線程。| 場景 | 推薦 | 理由 |
|---|---|---|
| 兩進程高頻雙向通信 | Pipe | 延遲最低、開銷最小 |
| 多生產者/消費者、任務分發 | Queue | 內建同步、緩衝、毒丸機制 |
| 跨機器通信 | 都不適合 | 改用Redis/RabbitMQ/gRPC |
| 共享大量數據 | SharedMemory / mmap | 避免序列化拷貝開銷 |
put阻塞,消費者未啟動或異常退出 → 設置timeout或使用put_nowait捕獲Full異常。spawn啟動方式:主模組代碼會被重新導入,務必將啟動代碼放在if __name__ == '__main__':保護塊中。pickle,含鎖、連接、生成器的對象不可直接傳遞。合理選擇Pipe或Queue,能構建高效穩定的多進程架構。