因此,當 wTx的估計值大於等於閾值0時,SVM對資料點標記為1,否則標記為0(其中閾值是SVM可以自適應的模型引數)。

SVM的損失函式被稱為合頁損失,定義為:

                                                                                             

SVM是一個最大間隔分類器,它試圖訓練一個使得類別儘可能分開的權重向量。在很多分類任務中,SVM不僅表現得效能突出,而且對大資料集的擴充套件是線性變化的。

在下圖中,基於原先的二分類簡單樣例,我們畫出了關於邏輯迴歸(藍線)和線性

SVM(紅線)的決策函式:

從圖中可以看出SVM可以有效定位到最靠近決策函式的資料點(間隔線用紅色的虛線表示)。

1.2 樸素貝葉斯模型

樸素貝葉斯是一個概率模型,通過計算給定資料點在某個類別的概率來進行預測。樸素貝葉斯模型假設每個特徵分配到某個類別的概率是獨立分佈的（假定各個特徵之間條件獨立），這也是樸素貝葉斯叫法的原因。

基於這個假設,屬於某個類別的概率表示為若干概率乘積的函式,其中這些概率包括某個特徵在給定某個類別的條件下出現的概率(條件概率),以及該類別的概率(先驗概率)。這樣使得模型訓練非常直接且易於處理。類別的先驗概率和特徵的條件概率可以通過資料的頻率估計得到。分類過程就是在給定特徵和類別概率的情況下選擇最可能的類別。

另外還有一個關於特徵分佈的假設,即引數的估計來自資料。MLlib實現了多項樸素貝葉斯(multinomial naïve Bayes),其中假設特徵分佈是多項分佈,用以表示特徵的非負頻率統計。

上述假設非常適合二元特徵(比如1-of-k,k維特徵向量中只有1維為1,其他為0中介紹的詞袋模型是一個典型的二元特徵表示)。

下圖展示了樸素貝葉斯在二分類樣本上的決策函式：

1.3 決策樹

決策樹是一個強大的非概率模型,它可以表達複雜的非線性模式和特徵相互關係。決策樹在很多工上表現出的效能很好,相對容易理解和解釋,可以處理類屬或者數值特徵,同時不要求輸入資料歸一化或者標準化。決策樹非常適合應用整合方法(

ensemble method),比如多個決策樹的整合,稱為決策樹森林。

決策樹模型就好比一棵樹,葉子代表值為0或1的分類,樹枝代表特徵。如圖5-6所示,二元輸出分別是“待在家裡”和“去海灘”,特徵則是天氣。

決策樹演算法是一種自上而下始於根節點(或特徵)的方法,在每一個步驟中通過評估特徵分裂的資訊增益,最後選出分割資料集最優的特徵。資訊增益通過計算節點不純度(即節點標籤不相似或不同質的程度)減去分割後的兩個子節點不純度的加權和。對於分類任務,這裡有兩個評估方法用於選擇最好的分割：基尼係數和熵。

2 從資料中抽取合適的特徵

,可以發現大部分機器學習模型以特徵向量的形式處理數值資料。另外,對於分類和迴歸等監督學習方法,需要將目標變數(或者多類別情況下的變數)和特徵向量放在一起。

MLlib中的分類模型通過LabeledPoint物件操作,其中封裝了目標變數(標籤)和特徵向量:

                LabeledPoint(label: Double, features: Vector) 

雖然在使用分類模型的很多樣例中會碰到向量格式的資料集,但在實際工作中,通常還需要從原始資料中抽取特徵 ，包括封裝數值特徵、歸一或者正則化特徵,以及使用1-of-k編碼表示類屬特徵 

從Kaggle/StumbleUpon evergreen分類資料集中抽取特徵 

考慮到上一篇博文推薦模型中的MovieLens資料集和分類問題無關,本章將使用另外一個數據集。這個資料集源自Kaggle比賽,由StumbleUpon提供。比賽的問題涉及網頁中推薦的頁面是短暫(短暫存在,很快就不流行了)還是長久(長時間流行)。 

為了讓Spark更好地操作資料,我們需要刪除檔案第一行的列頭名稱。先進入到資料檔案所在目錄，然後執行命令：

sed 1d train.tsv > train_noheader.tsv

讀取資料：

rawData = sc.textFile('/Users/youwei.tan/Downloads/train_noheader.tsv')
records = rawData.map(lambda x: x.split('\t'))
print records.first()

輸出結果：

[u'"http://www.bloomberg.com/news/2010-12-23/ibm-predicts-holographic-calls-air-breathing-batteries-by-2015.html"', u'"4042"', u'"{""title"":""IBM Sees Holographic Calls Air Breathing Batteries ibm sees holographic calls, air-breathing batteries"",""body"":""A sign stands outside the International Business Machines Corp IBM Almaden Research Center campus in San Jose California Photographer Tony Avelar Bloomberg Buildings stand at the International Business Machines Corp IBM Almaden Research Center campus in the Santa Teresa Hills of San Jose California Photographer Tony Avelar Bloomberg By 2015 your mobile phone will project a 3 D image of anyone who calls and your laptop will be powered by kinetic energy At least that s what International Business Machines Corp sees in its crystal ball The predictions are part of an annual tradition for the Armonk New York based company which surveys its 3 000 researchers to find five ideas expected to take root in the next five years IBM the world s largest provider of computer services looks to Silicon Valley for input gleaning many ideas from its Almaden research center in San Jose California Holographic conversations projected from mobile phones lead this year s list The predictions also include air breathing batteries computer programs that can tell when and where traffic jams will take place environmental information generated by sensors in cars and phones and cities powered by the heat thrown off by computer servers These are all stretch goals and that s good said Paul Saffo managing director of foresight at the investment advisory firm Discern in San Francisco In an era when pessimism is the new black a little dose of technological optimism is not a bad thing For IBM it s not just idle speculation The company is one of the few big corporations investing in long range research projects and it counts on innovation to fuel growth Saffo said Not all of its predictions pan out though IBM was overly optimistic about the spread of speech technology for instance When the ideas do lead to products they can have broad implications for society as well as IBM s bottom line he said Research Spending They have continued to do research when all the other grand research organizations are gone said Saffo who is also a consulting associate professor at Stanford University IBM invested 5 8 billion in research and development last year 6 1 percent of revenue While that s down from about 10 percent in the early 1990s the company spends a bigger share on research than its computing rivals Hewlett Packard Co the top maker of personal computers spent 2 4 percent last year At Almaden scientists work on projects that don t always fit in with IBM s computer business The lab s research includes efforts to develop an electric car battery that runs 500 miles on one charge a filtration system for desalination and a program that shows changes in geographic data IBM rose 9 cents to 146 04 at 11 02 a m in New York Stock Exchange composite trading The stock had gained 11 percent this year before today Citizen Science The list is meant to give a window into the company s innovation engine said Josephine Cheng a vice president at IBM s Almaden lab All this demonstrates a real culture of innovation at IBM and willingness to devote itself to solving some of the world s biggest problems she said Many of the predictions are based on projects that IBM has in the works One of this year s ideas that sensors in cars wallets and personal devices will give scientists better data about the environment is an expansion of the company s citizen science initiative Earlier this year IBM teamed up with the California State Water Resources Control Board and the City of San Jose Environmental Services to help gather information about waterways Researchers from Almaden created an application that lets smartphone users snap photos of streams and creeks and report back on conditions The hope is that these casual observations will help local and state officials who don t have the resources to do the work themselves Traffic Predictors IBM also sees data helping shorten commutes in the next five years Computer programs will use algorithms and real time traffic information to predict which roads will have backups and how to avoid getting stuck Batteries may last 10 times longer in 2015 than today IBM says Rather than using the current lithium ion technology new models could rely on energy dense metals that only need to interact with the air to recharge Some electronic devices might ditch batteries altogether and use something similar to kinetic wristwatches which only need to be shaken to generate a charge The final prediction involves recycling the heat generated by computers and data centers Almost half of the power used by data centers is currently spent keeping the computers cool IBM scientists say it would be better to harness that heat to warm houses and offices In IBM s first list of predictions compiled at the end of 2006 researchers said instantaneous speech translation would become the norm That hasn t happened yet While some programs can quickly translate electronic documents and instant messages and other apps can perform limited speech translation there s nothing widely available that acts like the universal translator in Star Trek Second Life The company also predicted that online immersive environments such as Second Life would become more widespread While immersive video games are as popular as ever Second Life s growth has slowed Internet users are flocking instead to the more 2 D environments of Facebook Inc and Twitter Inc Meanwhile a 2007 prediction that mobile phones will act as a wallet ticket broker concierge bank and shopping assistant is coming true thanks to the explosion of smartphone applications Consumers can pay bills through their banking apps buy movie tickets and get instant feedback on potential purchases all with a few taps on their phones The nice thing about the list is that it provokes thought Saffo said If everything came true they wouldn t be doing their job To contact the reporter on this story Ryan Flinn in San Francisco at rflinn bloomberg net To contact the editor responsible for this story Tom Giles at tgiles5 bloomberg net by 2015, your mobile phone will project a 3-d image of anyone who calls and your laptop will be powered by kinetic energy. at least that\\u2019s what international business machines corp. sees in its crystal ball."",""url"":""bloomberg news 2010 12 23 ibm predicts holographic calls air breathing batteries by 2015 html""}"', u'"business"', u'"0.789131"', u'"2.055555556"', u'"0.676470588"', u'"0.205882353"', u'"0.047058824"', u'"0.023529412"', u'"0.443783175"', u'"0"', u'"0"', u'"0.09077381"', u'"0"', u'"0.245831182"', u'"0.003883495"', u'"1"', u'"1"', u'"24"', u'"0"', u'"5424"', u'"170"', u'"8"', u'"0.152941176"', u'"0.079129575"', u'"0"']

資料集頁面中的資料簡介已經告訴我們。前四列分別指的是URL、頁面的ID、原始的文字內容和分配給頁面的類別。接下來22列包含各種各樣的數值或者類屬特徵。最後一列為目標值,1為長久, 0為短暫。 

我們將用簡單的方法直接對數值特徵做處理。因為每個類屬變數是二元的,對這些變數已有一個用1-of-k編碼的特徵,於是不需要額外提取特徵。 

由於資料格式的問題,我們做一些資料清理的工作,在處理過程中把額外的(")去掉。資料集中還有一些用"?"代替的缺失資料,本例中,我們直接用0替換那些缺失資料: 

from pyspark.mllib.regression import LabeledPoint
from pyspark.mllib.linalg import Vectors
trimmed = records.map(lambda x: [xx.replace('\"','') for xx in x])
label = trimmed.map(lambda x : x[-1])

data = trimmed.map(lambda x : (x[-1],x[4:-1])).\
               map(lambda (x,y):(x.replace('\"',''),[0.0 if yy=='\"?\"' else yy.replace('\"','') for yy in y])).\
               map(lambda (x,y):(x.replace("\"",""),[0.0 if yy =='?' else yy.replace("\"","") for yy in y])).\
               map(lambda (x,y):(int(x), [float(yy) for yy in y])).\
               map(lambda (x,y):LabeledPoint(x,Vectors.dense(y)))
data.take(5)

輸出結果：

[LabeledPoint(0.0, [0.789131,2.055555556,0.676470588,0.205882353,0.047058824,0.023529412,0.443783175,0.0,0.0,0.09077381,0.0,0.245831182,0.003883495,1.0,1.0,24.0,0.0,5424.0,170.0,8.0,0.152941176,0.079129575]),
 LabeledPoint(1.0, [0.574147,3.677966102,0.50802139,0.288770053,0.213903743,0.144385027,0.468648998,0.0,0.0,0.098707403,0.0,0.203489628,0.088652482,1.0,1.0,40.0,0.0,4973.0,187.0,9.0,0.181818182,0.125448029]),
 LabeledPoint(1.0, [0.996526,2.382882883,0.562015504,0.321705426,0.120155039,0.042635659,0.525448029,0.0,0.0,0.072447859,0.0,0.22640177,0.120535714,1.0,1.0,55.0,0.0,2240.0,258.0,11.0,0.166666667,0.057613169]),
 LabeledPoint(1.0, [0.801248,1.543103448,0.4,0.1,0.016666667,0.0,0.480724749,0.0,0.0,0.095860566,0.0,0.265655744,0.035343035,1.0,0.0,24.0,0.0,2737.0,120.0,5.0,0.041666667,0.100858369]),
 LabeledPoint(0.0, [0.719157,2.676470588,0.5,0.222222222,0.12345679,0.043209877,0.446143274,0.0,0.0,0.024908425,0.0,0.228887247,0.050473186,1.0,1.0,14.0,0.0,12032.0,162.0,10.0,0.098765432,0.082568807])]

對資料進行快取，同時統計資料樣本的數目：

data.cache()
numData = data.count()
print numData

輸出結果：

7395

在對資料集做進一步處理之前,我們發現數值資料中包含負的特徵值。我們知道,樸素貝葉斯模型要求特徵值非負,否則碰到負的特徵值程式會丟擲錯誤。因此,需要為樸素貝葉斯模型構建一份輸入特徵向量的資料,將負特徵值設為0: 

nbdata = trimmed.map(lambda x : (x[-1],x[4:-1])).\
                 map(lambda (x,y):(x.replace('\"',''),[0.0 if yy=='\"?\"' else yy.replace('\"','') for yy in y])).\
                 map(lambda (x,y):(x.replace("\"",""),[0.0 if yy =='?' else yy.replace("\"","") for yy in y])).\
                 map(lambda (x,y):(int(x), [float(yy) for yy in y])).\
                 map(lambda (x,y):(x, [0.0 if yy<0 else yy for yy in y])).\
                 map(lambda (x,y):LabeledPoint(x,Vectors.dense(y)))
nbdata.take(2)

輸出結果：

[LabeledPoint(0.0, [0.789131,2.055555556,0.676470588,0.205882353,0.047058824,0.023529412,0.443783175,0.0,0.0,0.09077381,0.0,0.245831182,0.003883495,1.0,1.0,24.0,0.0,5424.0,170.0,8.0,0.152941176,0.079129575]),
 LabeledPoint(1.0, [0.574147,3.677966102,0.50802139,0.288770053,0.213903743,0.144385027,0.468648998,0.0,0.0,0.098707403,0.0,0.203489628,0.088652482,1.0,1.0,40.0,0.0,4973.0,187.0,9.0,0.181818182,0.125448029])]

3 訓練分類模型 

現在我們已經從資料集中提取了基本的特徵並且建立了RDD,接下來開始訓練各種模型吧。為了比較不同模型的效能,我們將訓練邏輯迴歸、SVM、樸素貝葉斯和決策樹。你會發現每個模型的訓練方法幾乎一樣,不同的是每個模型都有著自己特定可配置的模型引數。MLlib大多數情況下會設定明確的預設值,但實際上,最好的引數配置需要通過評估技術來選擇,這個我們會在後續博文中進行討論。 

#匯入相應的類
from pyspark.mllib.classification import LogisticRegressionWithSGD
from pyspark.mllib.classification import SVMWithSGD
from pyspark.mllib.classification import NaiveBayes
from pyspark.mllib.tree import DecisionTree

numIteration = 10   #迭代次數
maxTreeDepth = 5    #樹的深度

numClass = label.distinct().count()     #類別數
print '類別數：',numClass

#訓練邏輯迴歸、支援向量機、樸素貝葉斯和決策樹模型
lrModel = LogisticRegressionWithSGD.train(data, numIteration)
svmModel = SVMWithSGD.train(data, numIteration)
nbModel = NaiveBayes.train(nbdata)
dtModel = DecisionTree.trainClassifier(data,numClass,{},impurity='entropy', maxDepth=maxTreeDepth)
print '邏輯迴歸模型引數：',lrModel
print '支援向量機模型引數：',svmModel
print '樸素貝葉斯模型引數：',nbModel
print '決策樹模型引數：',dtModel

輸出結果：

類別數： 2
邏輯迴歸模型引數： (weights=[-0.110216274454,-0.493200344739,-0.0712665620384,-0.0214744216778,0.00276706475384,0.00246385887598,-1.33300460292,0.0525232672351,0.0,-0.0320576776,-0.00653638798541,-0.0613702511674,-0.14975863133,-0.13648187383,-0.121161700009,-15.6451616669,-0.0177690355464,745.987958686,-7.73567729685,-1.38587998188,-0.0355600416613,-0.0352085128613], intercept=0.0)
支援向量機模型引數： (weights=[-0.122188386978,-0.527510758159,-0.0742371782434,-0.0206667449306,0.00546395033577,0.00409811283781,-1.54824523474,0.0607028905087,0.0,-0.037008323802,-0.007374037142,-0.067970375864,-0.172289581054,-0.148716595522,-0.129369384966,-18.0315472516,-0.0202704220321,1025.48043141,-5.05188911633,-1.54111193167,-0.038689478606,-0.0397619278886], intercept=0.0)
樸素貝葉斯模型引數： <pyspark.mllib.classification.NaiveBayesModel object at 0x11291ab90>
決策樹模型引數： DecisionTreeModel classifier of depth 5 with 61 nodes

4 使用分類模型 

現在我們有四個在輸入標籤和特徵下訓練好的模型,接下來看看如何使用這些模型進行預測。目前,我們將使用同樣的訓練資料來解釋每個模型的預測方法。 此處以邏輯迴歸模型為例，其他模型的使用方法類似。

dataPoint = data.first()
prediction = lrModel.predict(dataPoint.features)
print '預測的類別：',prediction
print '真實的類別：',int(dataPoint.label)

輸出結果：

預測的類別： 1
真實的類別： 0

可以看到對於訓練資料中第一個樣本,模型預測值為

相關推薦