安裝前置條件

1.         系統需要安裝the Oracle Java Development Kit(not OpenJDK),安裝jdk1.7以上，下載目錄：http://www.oracle.com/technetwork/java/javase/downloads/index-jsp-138363.html?ssSourceSiteId=ocomen。

  檢查方法：java -version

2.         安裝python2.7 以上，並配置環境變數和PATH，到https://www.python.org/ftp/python/2.7.3/Python-2.7.3.tar.bz2下載64位版本

  檢查方法：python -version

3.         在叢集所有機器中，安裝scala 2.10以上，並配置環境變數和PATH，下載目錄：http://downloads.typesafe.com/scala/2.10.5/scala-2.10.5.tgz

  檢查方法：scala命令可使用

4.         安裝rsync元件；

  檢查方法：scp命令可使用

5.         安裝ssh並在e3base使用者下完成叢集之間的無密碼訪問配置

6.         叢集之間保證ntp時鐘同步。

        檢查方法：service ntp status、service ntpd restart

7.         叢集各主機一定要永久關閉防火牆和selinux。

        檢查方法：service iptables status、service iptables stop

8.         將叢集中所有的主機名及ip對應關係都新增到每臺主機的/etc/hosts中。

Scala安裝

解壓安裝包

tar -xzvf scala-2.10.6.tgz

mv scala-2.10.6 scala

配置環境變數

exportSCALA_HOME=/home/hadoop/cdh5.5.0/scala

export PATH=$SCALA_HOME/bin:$PATH

使環境變數生效

source ~/.bash_profile

驗證是否安裝成功

執行如下命令：

scala –version

出現如下資訊則安裝成功

Spark叢集部署

安裝

解壓壓縮包

tar -xzvf spark-1.5.0-cdh5.5.0.tar.gz

mv spark-1.5.0-cdh5.5.0 spark

配置環境變數

vi ~/.bash_profile

exportSPARK_HOME=/home/hadoop/cdh5.5.0/spark

export PATH=$SPARK_HOME/bin:$PATH

使環境變數生效

source ~/.bash_profile

配置

spark-env.sh

cp spark-env.sh.template spark-env.sh

寫入如下資訊

vi spark-env.sh

slaves

cp slaves.template slaves

將Worker主機寫入slaves

啟動

將配置copy到Worker主機

cd $SPARK_HOME/sbin

sh start-all.sh

（注意：啟動之前保證Hadoop已經啟動，否則Master會啟動失敗）

Master HA配置

Master在從standby狀態到active狀態時，這個過程會影響新程式的提交，已經執行的程式不受影響。

export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER-Dspark.deploy.zookeeper.url=worker1:2181,worker2:2181,worker3:2181-Dspark.deploy.zookeeper.dir=/spark"

引數：

spark.deploy.recoveryMode=ZOOKEEPER

spark.deploy.zookeeper.url=worker1:2181,worker2:2181,worker3:2181

spark.deploy.zookeeper.dir=/spark

spark.deploy.recoveryMode

恢復模式（Master重新啟勱的模式），有三種：1.ZooKeeper,2. FileSystem, 3 NONE

spark.deploy.zookeeper.url

ZooKeeper的Server地址

spark.deploy.zookeeper.dir

/spark，ZooKeeper 儲存叢集元資料資訊的檔案目錄，包括Worker，Driver和Application。

問題解決

啟動失敗

問題描述

Exception in thread "main"java.lang.NoClassDefFoundError: org/slf4j/Logger

       at java.lang.Class.getDeclaredMethods0(Native Method)

       at java.lang.Class.privateGetDeclaredMethods(Class.java:2625)

       at java.lang.Class.getMethod0(Class.java:2866)

       at java.lang.Class.getMethod(Class.java:1676)

       at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)

       at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)

Caused by:java.lang.ClassNotFoundException: org.slf4j.Logger

       at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

       at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

       at java.security.AccessController.doPrivileged(Native Method)

       at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

       at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

       at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

問題解決

在spark-env.sh配置檔案中增加slf4j相關jar包，如下：

for f in$HADOOP_HOME/share/hadoop/common/lib/*.jar; do

  if[ "$SPARK_CLASSPATH" ]; then

   export SPARK_CLASSPATH=$SPARK_CLASSPATH:$f

 else

   export SPARK_CLASSPATH=$f

  fi

done

for f in$HADOOP_HOME/share/hadoop/common/*.jar; do

  if[ "$SPARK_CLASSPATH" ]; then

   export SPARK_CLASSPATH=$SPARK_CLASSPATH:$f

 else

   export SPARK_CLASSPATH=$f

  fi

done

啟動成功後一會自動shut down

問題描述

17/04/06 23:11:59 ERROR ActorSystemImpl:Uncaught fatal error from thread [sparkMaster-akka.actor.default-dispatcher-4]shutting down ActorSystem [sparkMaster]

java.lang.NoClassDefFoundError:com/fasterxml/jackson/databind/Module

       at java.lang.Class.forName0(Native Method)

       at java.lang.Class.forName(Class.java:278)

       at org.apache.spark.util.Utils$.classForName(Utils.scala:173)

       atorg.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:190)

       at org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:186)

       atscala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)

       atscala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)

       atscala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)

       at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)

       at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)

       at org.apache.spark.metrics.MetricsSystem.registerSinks(MetricsSystem.scala:186)

       at org.apache.spark.metrics.MetricsSystem.start(MetricsSystem.scala:100)

       at org.apache.spark.deploy.master.Master.onStart(Master.scala:152)

       at org.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1$$anonfun$preStart$1.apply$mcV$sp(AkkaRpcEnv.scala:100)

       atorg.apache.spark.rpc.akka.AkkaRpcEnv.org$apache$spark$rpc$akka$AkkaRpcEnv$$safelyCall(AkkaRpcEnv.scala:197)

       at org.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1.preStart(AkkaRpcEnv.scala:99)

       at akka.actor.ActorCell.create(ActorCell.scala:562)

       at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:425)

       at akka.actor.ActorCell.systemInvoke(ActorCell.scala:447)

       at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:262)

       at akka.dispatch.Mailbox.run(Mailbox.scala:218)

       atakka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)

       at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)

       at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)

       atscala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)

       atscala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Caused by:java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.Module

       at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

       at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

       at java.security.AccessController.doPrivileged(Native Method)

       at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

       at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

       at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

       ... 26 more

17/04/06 23:11:59 ERROR ErrorMonitor:Uncaught fatal error from thread [sparkMaster-akka.actor.default-dispatcher-4]shutting down ActorSystem [sparkMaster]

java.lang.NoClassDefFoundError:com/fasterxml/jackson/databind/Module

       at java.lang.Class.forName0(Native Method)

       at java.lang.Class.forName(Class.java:278)

       at org.apache.spark.util.Utils$.classForName(Utils.scala:173)

       atorg.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:190)

       atorg.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:186)

       atscala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)

       at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)

       atscala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)

       at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)

       at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)

       atorg.apache.spark.metrics.MetricsSystem.registerSinks(MetricsSystem.scala:186)

       at org.apache.spark.metrics.MetricsSystem.start(MetricsSystem.scala:100)

       at org.apache.spark.deploy.master.Master.onStart(Master.scala:152)

       atorg.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1$$anonfun$preStart$1.apply$mcV$sp(AkkaRpcEnv.scala:100)

       atorg.apache.spark.rpc.akka.AkkaRpcEnv.org$apache$spark$rpc$akka$AkkaRpcEnv$$safelyCall(AkkaRpcEnv.scala:197)

       atorg.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1.preStart(AkkaRpcEnv.scala:99)

       at akka.actor.ActorCell.create(ActorCell.scala:562)

       at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:425)

        atakka.actor.ActorCell.systemInvoke(ActorCell.scala:447)

       at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:262)

       at akka.dispatch.Mailbox.run(Mailbox.scala:218)

       at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)

       at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)

       atscala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)

       at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)

       atscala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Caused by:java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.Module

       at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

       at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

       at java.security.AccessController.doPrivileged(Native Method)

       at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

       at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

       at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

       ... 26 more

17/04/06 23:12:00 WARN MetricsSystem:Stopping a MetricsSystem that is not running

17/04/06 23:12:00 ERROR AkkaRpcEnv: Ignoreerror: null

java.lang.NullPointerException

       at org.apache.spark.deploy.master.Master.onStop(Master.scala:198)

       at org.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1$$anonfun$postStop$1.apply$mcV$sp(AkkaRpcEnv.scala:143)

       atorg.apache.spark.rpc.akka.AkkaRpcEnv.org$apache$spark$rpc$akka$AkkaRpcEnv$$safelyCall(AkkaRpcEnv.scala:197)

       at org.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1.postStop(AkkaRpcEnv.scala:142)

       atakka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:201)

       atakka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:163)

       at akka.actor.ActorCell.terminate(ActorCell.scala:338)

       at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:431)

       at akka.actor.ActorCell.systemInvoke(ActorCell.scala:447)

       at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:262)

       at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:240)

       at akka.dispatch.Mailbox.run(Mailbox.scala:219)

       at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)

       at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)

       atscala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)

       at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)

       atscala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

17/04/06 23:12:00 INFORemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.

17/04/06 23:12:00 INFORemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceedingwith flushing remote transports.

17/04/06 23:12:01 INFO Remoting: Remotingshut down

17/04/06 23:12:01 INFORemoteActorRefProvider$RemotingTerminator: Remoting shut down.

問題解決

CLASSPATH中缺少jackson包

for f in$HADOOP_HOME/share/hadoop/mapreduce*/lib/*.jar; do

  if[ "$SPARK_CLASSPATH" ]; then

   export SPARK_CLASSPATH=$SPARK_CLASSPATH:$f

 else

   export SPARK_CLASSPATH=$f

  fi

done

Spark Streaming除錯

使用sparkexample自帶的streaming用例進行測試：計算從資料伺服器監聽TCP套接字接收的文字資料字數。

1、  啟動netcat作為伺服器

nc –lk 9999

2、  啟動Streaming的例項

run-example streaming.NetworkWordCount 172.21.3.60 9999

3、  在終端netcat伺服器傳送資料