haskell – 有效的並行策略

我試圖圍繞平行策略.我想我明白了每個組合器的作用,但是每次嘗試使用超過1個核心的程式時,程式會顯著減慢.

例如,一段時間後,我嘗試從〜700個文件中計算直方圖(並從它們獨特的單詞).我以為使用檔案級粒度會好的.與-N4我得到1.70工作平衡.然而與-N1相比,它的執行時間要比使用-N4的時間長一半.我不知道這個問題是什麼,但是我想知道如何決定何時/何時/如何並行並獲得一些理解.這將如何並行化,以便速度隨著核心而不是減少而增加？

import Data.Map (Map)
import qualified Data.Map as M
import System.Directory
import Control.Applicative
import Data.Vector (Vector)
import qualified Data.Vector as V
import qualified Data.Text as T
import qualified Data.Text.IO as TI
import Data.Text (Text)
import System.FilePath ((</>))
import Control.Parallel.Strategies
import qualified Data.Set as S
import Data.Set (Set)
import GHC.Conc (pseq, numCapabilities)
import Data.List (foldl')

mapReduce stratm m stratr r xs = let
mapped = parMap stratm m xs
reduced = r mapped `using` stratr
in mapped `pseq` reduced

type Histogram = Map Text Int

rootDir = "/home/masse/Documents/text_conversion/"

finnishStop = ["minä", "sinä", "hän", "kuitenkin", "jälkeen", "mukaanlukien", "koska", "mutta", "jos", "kuitenkin", "kun", "kunnes", "sanoo", "sanoi", "sanoa", "miksi", "vielä", "sinun"]
englishStop = ["a","able","about","across","after","all","almost","also","am","among","an","and","any","are","as","at","be","because","been","but","by","can","cannot","could","dear","did","do","does","either","else","ever","every","for","from","get","got","had","has","have","he","her","hers","him","his","how","however","i","if","in","into","is","it","its","just","least","let","like","likely","may","me","might","most","must","my","neither","no","nor","not","of","off","often","on","only","or","other","our","own","rather","said","say","says","she","should","since","so","some","than","that","the","their","them","then","there","these","they","this","tis","to","too","twas","us","wants","was","we","were","what","when","where","which","while","who","whom","why","will","with","would","yet","you","your"]
isStopWord :: Text -> Bool
isStopWord x = x `elem` (finnishStop ++ englishStop)

textFiles :: IO [FilePath]
textFiles = map (rootDir </>) . filter (not . meta) <$> getDirectoryContents rootDir
where meta "." = True
meta ".." = True
meta _ = False

histogram :: Text -> Histogram
histogram = foldr (\k -> M.insertWith' (+) k 1) M.empty . filter (not . isStopWord) . T.words

wordList = do
files <- mapM TI.readFile =<< textFiles
return $mapReduce rseq histogram rseq reduce files
where
reduce = M.unions

main = do
list <- wordList
print $M.size list

對於文字檔案,我使用的pdf轉換為文字檔案,所以我不能提供它們,但為了這個目的,幾乎任何書/書從古騰堡專案應該做.

編輯：新增匯入指令碼

實際上,讓並聯組合器進行規模化可能是困難的.

其他人已經提到使你的程式碼更嚴格,以確保你實際上

並行工作,這絕對是重要的.

兩個可以真正殺死效能的東西是大量的記憶體遍歷

垃圾收集.即使你沒有生產大量的垃圾,很多

記憶體遍歷給CPU快取帶來更多的壓力,最終在你的

記憶體匯流排成為瓶頸.您的isStopWord功能執行很多

的字串比較,並且必須遍歷一個相當長的連結串列才能這樣做.

你可以使用內建的Set型別來節省大量的工作,甚至可以節省大量的工作

來自無序容器包的HashSet型別(由於重複的字串

比較可能是昂貴的,特別是如果他們共享公共字首).

importData.HashSet(HashSet)
import qualified Data.HashSetas S

...

finnishStop :: [Text]
finnishStop = ["minä", "sinä", "hän", "kuitenkin", "jälkeen", "mukaanlukien", "koska", "mutta", "jos", "kuitenkin", "kun", "kunnes", "sanoo", "sanoi", "sanoa", "miksi", "vielä", "sinun"]
englishStop :: [Text]
englishStop = ["a","able","about","across","after","all","almost","also","am","among","an","and","any","are","as","at","be","because","been","but","by","can","cannot","could","dear","did","do","does","either","else","ever","every","for","from","get","got","had","has","have","he","her","hers","him","his","how","however","i","if","in","into","is","it","its","just","least","let","like","likely","may","me","might","most","must","my","neither","no","nor","not","of","off","often","on","only","or","other","our","own","rather","said","say","says","she","should","since","so","some","than","that","the","their","them","then","there","these","they","this","tis","to","too","twas","us","wants","was","we","were","what","when","where","which","while","who","whom","why","will","with","would","yet","you","your"]

stopWord :: HashSet Text
stopWord = S.fromList (finnishStop ++ englishStop)

isStopWord :: Text -> Bool
isStopWord x = x `S.member` stopWord

使用此版本替換您的isStopWord功能執行得更好

並且更好的縮放(儘管絕對不是1-1).你也可以考慮

使用HashMap(來自同一個包)而不是Map的原因相同,

但是我沒有得到明顯的改變.

另一個選擇是增加預設的堆大小來取得一些

壓力GC,並給它更多的空間移動的東西.給予

編譯程式碼一個預設的堆大小為1GB(-H1G標誌),我得到一個GC平衡

在4核心上約為50％,而我只有〜25％沒有(它也跑〜30％

更快).

有了這兩個變化,四個核心的平均執行時間(在我的機器上)

從〜10.5s降至〜3.5s.可以說,有改進的餘地

GC統計數字(仍然只花58％的時間做生產性工作),

但要做得更好,可能需要更加劇烈的改變

你的演算法