Python 多進程繞過 GIL 實現真並行。本文從 Unix fork() 系統調用切入,系統講解跨平台 multiprocessing 模組:Process、Pool、Queue/Pipe 進程間通訊、同步原語、共享記憶體,並對比執行緒模型優缺點。
import os
print('Process (%s) start...' % os.getpid())
pid = os.fork()
if pid == 0:
print('I am child process (%s) and my parent is %s.' % (os.getpid(), os.getppid()))
else:
print('I (%s) just created a child process (%s).' % (os.getpid(), pid))
輸出:
Process (876) start...
I (876) just created a child process (877).
I am child process (877) and my parent is 876.
限制:Windows 無 fork,需用 multiprocessing。
from multiprocessing import Process
import os
def run_proc(name):
print('Run child process %s (%s)...' % (name, os.getpid()))
if __name__ == '__main__':
print('Parent process %s.' % os.getpid())
p = Process(target=run_proc, args=('test',))
print('Process will start.')
p.start()
p.join() # 等待子進程結束
print('Process end.')
from multiprocessing import Pool
import os, time, random
def long_time_task(name):
print('Run task %s (%s)...' % (name, os.getpid()))
start = time.time()
time.sleep(random.random() * 3)
end = time.time()
print('Task %s runs %0.2f seconds.' % (name, end - start))
if __name__ == '__main__':
print('Parent process %s.' % os.getpid())
p = Pool() # 預設大小 = CPU 核心數
for i in range(5):
p.apply_async(long_time_task, args=(i,))
print('Waiting for all subprocesses done...')
p.close() # 不再接受新任務
p.join() # 等待所有完成
print('All subprocesses done.')
輸出顯示:前 4 個任務並發,第 5 個等待空閒 Worker。
from multiprocessing import Process, Queue
import os, time, random
def write(q):
for value in ['A', 'B', 'C']:
print('Put %s to queue...' % value)
q.put(value)
time.sleep(random.random())
def read(q):
while True:
value = q.get(True)
print('Get %s from queue.' % value)
if __name__ == '__main__':
q = Queue()
pw = Process(target=write, args=(q,))
pr = Process(target=read, args=(q,))
pw.start()
pr.start()
pw.join()
pr.terminate() # 讀進程無限迴圈,需強制結束
pr.join()
| 原語 | 用途 | 對應 threading |
|---|---|---|
Lock | 互斥鎖 | threading.Lock |
RLock | 可重入鎖 | threading.RLock |
Semaphore | 信號量 | threading.Semaphore |
Event | 事件通知 | threading.Event |
Condition | 條件變數 | threading.Condition |
Barrier | 屏障同步 | threading.Barrier (3.2+) |
from multiprocessing import Process, Value, Array
def worker(n, a):
n.value = 3.14
for i in range(len(a)):
a[i] = -a[i]
if __name__ == '__main__':
num = Value('d', 0.0) # 'd' = double
arr = Array('i', range(10)) # 'i' = signed int
p = Process(target=worker, args=(num, arr))
p.start()
p.join()
print(num.value) # 3.14
print(arr[:]) # [0, -1, -2, ..., -9]
| 維度 | multiprocessing | threading |
|---|---|---|
| GIL 影響 | 無(獨立解譯器) | 受限(CPU 密集無並行) |
| 記憶體開銷 | 高(獨立記憶體空間) | 低(共享記憶體) |
| 啟動開銷 | 大(進程建立) | 小 |
| 資料共享 | 需 IPC/共享記憶體 | 直接共享(需鎖) |
| 崩潰隔離 | 進程崩潰不影響其他 | 執行緒崩潰殺死進程 |
| 適用場景 | CPU 密集、隔離性、繞過 GIL | I/O 密集、高頻共享狀態 |
from concurrent.futures import ProcessPoolExecutor
with ProcessPoolExecutor(max_workers=4) as executor:
futures = [executor.submit(long_time_task, i) for i in range(5)]
for f in futures:
print(f.result())
更高層級抽象,支援 Future 介面、as_completed、異常傳播。