一.需要的軟體：

• eclipse
• 相應版本的scalaIDE
• 與叢集一樣的spark安裝包，主要是要用到spark中的jar包
• 與叢集一樣的hadoop安裝包
• 與hadoop版本對應的winutil.exe，hadoop.dll（只要版本差距不大不一樣也沒關係）

二.步驟
（一）在eclipse中安裝對應版本的ScalaIDE，具體安裝見網上。

（二）複製winutil.exe，hadoop.dll到hadoop的bin檔案下，並設定hadoop的環境變數，具體設定方法也可以在網上找到。

（三）在eclipse中建立scala工程，與java工程類似，在src下建立package和相應的scala object檔案。

程式碼示例：

import org.apache.spark.SparkContext._
import org.apache.spark.{SparkConf,SparkContext}

object RemoteDebug {
  def main(args: Array[String]) {
    val conf = new SparkConf().setAppName("Spark Pi").setMaster("spark://your-hadoop-master:7077")
      .setJars(List("file:///E:/scalatest.jar"))
      System.setProperty("hadoop.home.dir"
 
, "D:/Program Files/hadoop-2.6.5");
    val spark = new SparkContext(conf)
    val slices = if (args.length > 0) args(0).toInt else 2
    val n = 100000 * slices
    val count = spark.parallelize(1 to n, slices).map { i =>
      val x = Math.random * 2 - 1
      val y = Math.random * 2 - 1
      if (x * x + y * y < 1
 
) 1 else 0
    }.reduce(_ + _)
    println("Pi is roughly " + 4.0 * count / n)
    spark.stop()
  }
}

（四）build path

將spark安裝包下的jars資料夾下的jar包匯入scala project中。

（五）run as Scala Application

執行結果：

（六）出現的錯誤：

錯誤一：

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 5, 222.31.67.83): java.io.IOException: No FileSystem for scheme: E
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
    at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1822)
    at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:665)
    at org.apache.spark.util.Utils$.fetchFile(Utils.scala:449)
    at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:488)
    at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:480)
    at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
    at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
	at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
    at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
    at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
    at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
    at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
    at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:480)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:252)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

原因：使用file:///指定為本地路徑

錯誤二：

17/01/05 22:16:00 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, 222.31.67.133): java.lang.RuntimeException: Stream '/jars/scalatest.jar' was not found.
    at org.apache.spark.network.client.TransportResponseHandler.handle(TransportResponseHandler.java:222)
    at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:121)
    at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51)
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
    at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
    at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
    at java.lang.Thread.run(Thread.java:745)

原因：找不到jar包，是因為在執行之前沒有把我們的專案先導成jar包放到我們指定的路徑下。因此，只需要匯出jar包就可以了。

錯誤三：

系統變數設定了HADOOP_HOME也不行，找不到winutil.exe。

可以在程式中直接設定環境變數

  System.setProperty("hadoop.home.dir", "D:/Program Files/hadoop-2.6.5");