multiprocessing模塊的多進程與進程池
阿新 • • 發佈:2018-12-21
一個 回調函數 mon close nag users cal targe ini multiprocessing模塊的Process方法
可以利用Proces方法在一個主進程中創建幾個子進程
from multiprocessing import Process import time def f1(name): time.sleep(2) print(‘Hell %s‘ % name) def f2(age): time.sleep(2) print(‘Hell %s‘ % age) if __name__ == "__main__": p = Process(target=f1,args=(‘ayu‘,)) p.daemon = True #將daemon設置為True,則主進程不等待子進程,主進程結束,則整個進程結束 p.start() p = Process(target=f2,args=(‘22‘,)) p.daemon = True p.start() print(‘All Done‘) #子進程結束後會輸出
###進程間的內存是不共享的
from multiprocessing import Process li = [] def ad(i): li.append(i) print(li) if __name__ == "__main__": for i in range(10): p = Process(target=ad,args=(i)) p.start() /Users/wuxiangyu-pc/.conda/envs/test_all/bin/python /Users/wuxiangyu-pc/Documents/spider/test_all/fork_process.py [0] [1] [2] [3] [4] [5] [6] [7] [8] [9]
說明各個進程間,內存是不能共享的
但是線程之間內存是可以共享的,所以可以使用threading操作
from threading import Thread li = [] def ad(i): li.append(i) print(li) if __name__ == "__main__": for i in range(10): p = Thread(target=ad,args=(i,)) p.start() /Users/wuxiangyu-pc/.conda/envs/test_all/bin/python /Users/wuxiangyu-pc/Documents/spider/test_all/fork_process.py [0] [0, 1] [0, 1, 2] [0, 1, 2, 3] [0, 1, 2, 3, 4] [0, 1, 2, 3, 4, 5] [0, 1, 2, 3, 4, 5, 6] [0, 1, 2, 3, 4, 5, 6, 7] [0, 1, 2, 3, 4, 5, 6, 7, 8] [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] Process finished with exit code 0
要實現進程間的內存共享,可以使用Manager方法
from multiprocessing import Process,Manager
def ad(i,li):
li.append(i)
print(li)
if __name__ == "__main__":
manager = Manager()
li = manager.li()
for i in range(10):
p = Process(target=ad,args=(i,li))
p.start()
p.join()
/Users/wuxiangyu-pc/.conda/envs/test_all/bin/python /Users/wuxiangyu-pc/Documents/spider/test_all/fork_process.py
[0]
[0, 1]
[0, 1, 2]
[0, 1, 2, 3]
[0, 1, 2, 3, 4]
[0, 1, 2, 3, 4, 5]
[0, 1, 2, 3, 4, 5, 6]
[0, 1, 2, 3, 4, 5, 6, 7]
[0, 1, 2, 3, 4, 5, 6, 7, 8]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Process finished with exit code 0
##multiprocessing模塊的Pool進程池
Pool.apply方法可以實現多個子進程排序依次執行
from multiprocessing import Pool
import time
def f0(name):
time.sleep(2)
print(‘i am %s‘ % name)
if __name__ == "__main__":
p = Pool(5)
for i in range(5):
p.apply(func=f0,args=(i,))
print(‘Hello World‘)
p.close()
p.join()
Pool.apply_async實現多線程異步,比apply多一個回調函數
from multiprocessing import Pool
def f1(num):
i = num + 20
return i
def f1(i):
print(‘i am %s‘ % i)
if __name__ == "__main__":
p = Pool(5)
for i in range(5):
p.apply_async(func=f1,args=(i,),callback=f1)
p.close()
p.join()
multiprocessing模塊的多進程與進程池