萬盛學電腦網

您现在的位置: 萬盛學電腦網 >> 程式設計 >  >> 網路程式設計 >> 程式語言綜合

Python 多進程程式設計完整解析:fork、multiprocessing.Process、Pool、進程間通訊 Queue

Python 多進程繞過 GIL 實現真並行。本文從 Unix fork() 系統調用切入,系統講解跨平台 multiprocessing 模組:Process、Pool、Queue/Pipe 進程間通訊、同步原語、共享記憶體,並對比執行緒模型優缺點。

Unix fork() 基礎

import os
print('Process (%s) start...' % os.getpid())
pid = os.fork()
if pid == 0:
    print('I am child process (%s) and my parent is %s.' % (os.getpid(), os.getppid()))
else:
    print('I (%s) just created a child process (%s).' % (os.getpid(), pid))

輸出:

Process (876) start...
I (876) just created a child process (877).
I am child process (877) and my parent is 876.

限制:Windows 無 fork,需用 multiprocessing。

multiprocessing.Process:跨平台進程

from multiprocessing import Process
import os

def run_proc(name):
    print('Run child process %s (%s)...' % (name, os.getpid()))

if __name__ == '__main__':
    print('Parent process %s.' % os.getpid())
    p = Process(target=run_proc, args=('test',))
    print('Process will start.')
    p.start()
    p.join()  # 等待子進程結束
    print('Process end.')

Pool 進程池:批量並發

from multiprocessing import Pool
import os, time, random

def long_time_task(name):
    print('Run task %s (%s)...' % (name, os.getpid()))
    start = time.time()
    time.sleep(random.random() * 3)
    end = time.time()
    print('Task %s runs %0.2f seconds.' % (name, end - start))

if __name__ == '__main__':
    print('Parent process %s.' % os.getpid())
    p = Pool()  # 預設大小 = CPU 核心數
    for i in range(5):
        p.apply_async(long_time_task, args=(i,))
    print('Waiting for all subprocesses done...')
    p.close()  # 不再接受新任務
    p.join()   # 等待所有完成
    print('All subprocesses done.')

輸出顯示:前 4 個任務並發,第 5 個等待空閒 Worker。

進程間通訊:Queue

from multiprocessing import Process, Queue
import os, time, random

def write(q):
    for value in ['A', 'B', 'C']:
        print('Put %s to queue...' % value)
        q.put(value)
        time.sleep(random.random())

def read(q):
    while True:
        value = q.get(True)
        print('Get %s from queue.' % value)

if __name__ == '__main__':
    q = Queue()
    pw = Process(target=write, args=(q,))
    pr = Process(target=read, args=(q,))
    pw.start()
    pr.start()
    pw.join()
    pr.terminate()  # 讀進程無限迴圈,需強制結束
    pr.join()

同步原語與共享狀態

原語用途對應 threading
Lock互斥鎖threading.Lock
RLock可重入鎖threading.RLock
Semaphore信號量threading.Semaphore
Event事件通知threading.Event
Condition條件變數threading.Condition
Barrier屏障同步threading.Barrier (3.2+)

共享記憶體:Value / Array

from multiprocessing import Process, Value, Array

def worker(n, a):
    n.value = 3.14
    for i in range(len(a)):
        a[i] = -a[i]

if __name__ == '__main__':
    num = Value('d', 0.0)      # 'd' = double
    arr = Array('i', range(10)) # 'i' = signed int
    p = Process(target=worker, args=(num, arr))
    p.start()
    p.join()
    print(num.value)  # 3.14
    print(arr[:])     # [0, -1, -2, ..., -9]

進程 vs 執行緒 選型指南

維度multiprocessingthreading
GIL 影響無(獨立解譯器)受限(CPU 密集無並行)
記憶體開銷高(獨立記憶體空間)低(共享記憶體)
啟動開銷大(進程建立)小
資料共享需 IPC/共享記憶體直接共享(需鎖)
崩潰隔離進程崩潰不影響其他執行緒崩潰殺死進程
適用場景CPU 密集、隔離性、繞過 GILI/O 密集、高頻共享狀態

現代替代:concurrent.futures

from concurrent.futures import ProcessPoolExecutor

with ProcessPoolExecutor(max_workers=4) as executor:
    futures = [executor.submit(long_time_task, i) for i in range(5)]
    for f in futures:
        print(f.result())

更高層級抽象,支援 Future 介面、as_completed、異常傳播。

copyright © 萬盛學電腦網 all rights reserved