Python多進程機制實例詳解

 　　本文實例講述了Python多進程機制。分享給大家供大家參考。具體如下：

　　在以前只是接觸過PYTHON的多線程機制，今天搜了一下多進程，相關文章好像不是特別多。看了幾篇，小試了一把。程序如下，主要內容就是通過PRODUCER讀一個本地文件，一行一行的放到隊列中去。然後會有相應的WORKER從隊列中取出這些行。

　　?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 import multiprocessing import os import sys import Queue import time def writeQ(q,obj): q.put(obj,True,None) print "put size: ",q.qsize() def readQ(q): ret = q.get(True,1) print "get size: ",q.qsize() return ret def producer(q): time.sleep(5) #讓進行休息幾秒 方便ps命令看到相關內容 pid = os.getpid() handle_file = '/home/dwapp/joe.wangh/test/multiprocess/datafile' with open(handle_file,'r') as f: #with...as... 這個用法今天也是第一次看到的 for line in f: print "producer <" ,pid , "> is doing: ",line writeQ(q,line.strip()) q.close() def worker(q): time.sleep(5) #讓進行休息幾秒 方便ps命令看到相關內容 pid = os.getpid() empty_count = 0 while True: try: task = readQ(q) print "worker <" , pid , "> is doing: " ,task ''' 如果這裡不休眠的話 一般情況下所有行都會被同一個子進程讀取到 為了使實驗效果更加清楚 在這裡讓每個進程讀取完 一行內容時候休眠5s 這樣就可以讓其他的進程到隊列中進行讀取 ''' time.sleep(5) except Queue.Empty: empty_count += 1 if empty_count == 3: print "queue is empty, quit" q.close() sys.exit(0) def main(): concurrence = 3 q = multiprocessing.Queue(10) funcs = [producer , worker] for i in range(concurrence-1): funcs.append(worker) for item in funcs: print str(item) nfuncs = range( len(funcs) ) processes = [] for i in nfuncs: p = multiprocessing.Process(target=funcs[i] , args=(q,)) processes.append(p) print "concurrence worker is : ",concurrence," working start" for i in nfuncs: processes[i].start() for i in nfuncs: processes[i].join() print "all DONE" if __name__ == '__main__': main()

　　實驗結果如下：

　　?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 dwapp@pttest1:/home/dwapp/joe.wangh/test/multiprocess>python 1.py <function producer at 0xb7b9141c> <function worker at 0xb7b91454> <function worker at 0xb7b91454> <function worker at 0xb7b91454> concurrence worker is : 3 working start producer < 28320 > is doing: line 1 put size: 1 producer < 28320 > is doing: line 2 put size: 2 producer < 28320 > is doing: line 3 put size: 3 producer < 28320 > is doing: line 4 put size: 3 producer < 28320 > is doing: line 5 get size: 3 put size: 4 worker < 28321 > is doing: line 1 get size: 3 worker < 28322 > is doing: line 2 get size: 2 worker < 28323 > is doing: line 3 get size: 1 worker < 28321 > is doing: line 4 get size: 0 worker < 28322 > is doing: line 5 queue is empty, quit queue is empty, quit queue is empty, quit all DONE

　　程序運行期間在另外一個窗口進行ps命令 可以觀測到一些進程的信息

　　?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 dwapp@pttest1:/home/dwapp/joe.wangh/test/multiprocess>ps -ef | grep python dwapp 13735 11830 0 Nov20 pts/12 00:00:05 python dwapp 28319 27481 8 14:04 pts/0 00:00:00 python 1.py dwapp 28320 28319 0 14:04 pts/0 00:00:00 python 1.py dwapp 28321 28319 0 14:04 pts/0 00:00:00 python 1.py dwapp 28322 28319 0 14:04 pts/0 00:00:00 python 1.py dwapp 28323 28319 0 14:04 pts/0 00:00:00 python 1.py dwapp 28325 27849 0 14:04 pts/13 00:00:00 grep python dwapp@pttest1:/home/dwapp/joe.wangh/test/multiprocess>ps -ef | grep python dwapp 13735 11830 0 Nov20 pts/12 00:00:05 python #此時28320進程 也就是PRODUCER進程已經結束 dwapp 28319 27481 1 14:04 pts/0 00:00:00 python 1.py dwapp 28321 28319 0 14:04 pts/0 00:00:00 python 1.py dwapp 28322 28319 0 14:04 pts/0 00:00:00 python 1.py dwapp 28323 28319 0 14:04 pts/0 00:00:00 python 1.py dwapp 28328 27849 0 14:04 pts/13 00:00:00 grep python dwapp@pttest1:/home/dwapp/joe.wangh/test/multiprocess>ps -ef | grep python dwapp 13735 11830 0 Nov20 pts/12 00:00:05 python dwapp 28319 27481 0 14:04 pts/0 00:00:00 python 1.py dwapp 28321 28319 0 14:04 pts/0 00:00:00 python 1.py dwapp 28322 28319 0 14:04 pts/0 00:00:00 python 1.py dwapp 28323 28319 0 14:04 pts/0 00:00:00 [python] <defunct> #這裡應該是代表28323進程（WORKER）已經運行結束了 dwapp 28331 27849 0 14:04 pts/13 00:00:00 grep python dwapp@pttest1:/home/dwapp/joe.wangh/test/multiprocess>ps -ef | grep python