3種方式實現python多執行緒併發處理
阿新 • • 發佈:2018-12-22
標籤: python奇淫技巧
最優執行緒數
- Ncpu=CPU的數量
- Ucpu=目標CPU使用率
- W/C=等待時間與計算時間的比率
為保持處理器達到期望的使用率,最優的執行緒池的大小等於
$$Nthreads=Ncpu*Ucpu*(1+W/C$$
- cpu密集型任務,即$W<<C$,則$W/C≈0$,則$Nthreads=Ncpu*Ucpu$
如果希望CPU利用率為100%,則$Nthreads=Ncpu$
- IO密集型任務,即系統大部分時間在跟I/O互動,而這個時間執行緒不會佔用CPU來處理,即在這個時間範圍內,可以由其他執行緒來使用CPU,因而可以多配置一些執行緒。
- 混合型任務,二者都佔有一定的時間
線城池
對於任務數量不斷增加的程式,每有一個任務就生成一個執行緒,最終會導致執行緒數量的失控。對於任務數量不端增加的程式,固定執行緒數量的執行緒池是必要的。
方法一:使用threadpool模組
threadpool是一個比較老的模組了,支援py2 和 py3 。
import threadpool import time def sayhello (a): print("hello: "+a) time.sleep(2) def main(): global result seed=["a","b","c"] start=time.time() task_pool=threadpool.ThreadPool(5) requests=threadpool.makeRequests(sayhello,seed) for req in requests: task_pool.putRequest(req) task_pool.wait() end=time.time() time_m = end-start print("time: "+str(time_m)) start1=time.time() for each in seed: sayhello(each) end1=time.time() print("time1: "+str(end1-start1)) if __name__ == '__main__': main(
方法二:使用concurrent.futures模組
from concurrent.futures import ThreadPoolExecutor import time import time from concurrent.futures import ThreadPoolExecutor, wait, as_completed ll = [] def sayhello(a): print("hello: "+a) ll.append(a) time.sleep(0.8) def main(): seed=["a","b","c","e","f","g","h"] start1=time.time() for each in seed: sayhello(each) end1=time.time() print("time1: "+str(end1-start1)) start2=time.time() with ThreadPoolExecutor(2) as executor: for each in seed: executor.submit(sayhello,each) end2=time.time() print("time2: "+str(end2-start2)) def main2(): seed = ["a", "b", "c", "e", "f", "g", "h"] executor = ThreadPoolExecutor(max_workers=10) f_list = [] for each in seed: future = executor.submit(sayhello, each) f_list.append(future) wait(f_list) print(ll) print('主執行緒結束') def main3(): seed = ["a", "b", "c", "e", "f", "g", "h"] with ThreadPoolExecutor(max_workers=2) as executor: f_list = [] for each in seed: future = executor.submit(sayhello, each) f_list.append(future) wait(f_list,return_when='ALL_COMPLETED') print(ll) print('主執行緒結束') if __name__ == '__main__': main3()
方法三:使用vthread模組
參考:https://pypi.org/project/vthr...
demo1
import vthread
@vthread.pool(6)
def some(a,b,c):
import time;time.sleep(1)
print(a+b+c)
for i in range(10):
some(i,i,i)
demo2:分組執行緒池
import vthread
pool_1 = vthread.pool(5,gqueue=1) # open a threadpool with 5 threads named 1
pool_2 = vthread.pool(2,gqueue=2) # open a threadpool with 2 threads named 2
@pool_1
def foolfunc1(num):
time.sleep(1)
print(f"foolstring1, test3 foolnumb1:{num}")
@pool_2
def foolfunc2(num):
time.sleep(1)
print(f"foolstring2, test3 foolnumb2:{num}")
@pool_2
def foolfunc3(num):
time.sleep(1)
print(f"foolstring3, test3 foolnumb3:{num}")
for i in range(10): foolfunc1(i)
for i in range(4): foolfunc2(i)
for i in range(2): foolfunc3(i)
來源:https://segmentfault.com/a/1190000017324613