本文將介紹Apache Spark 1.6.1在單機的部署，與在叢集中部署的步驟基本一致，只是少了一些master和slave檔案的配置。

http://blog.csdn.net/u011513853/article/details/52865076     Spark在Windows下的環境搭建

1、Spark的安裝準備

Spark runs on Java 7+, Python 2.6+ and R 3.1+. For the Scala API, Spark 1.6.1 usesScala 2.10. You will need to use a compatible Scala version (2.10.x).

下載spark

點選開啟連結    http://spark.apache.org/downloads.html

備註：我選擇的是基於hadoop2.6的spark1.6.1的編譯版本

2.安裝jdk

cd /usr/app
$ tar -xzvf jdk-7u79-linux-x64.tar.gz
$ sudo vim /etc/profile

export JAVA_HOME=/usr/app/jdk1.7
export JRE_HOME=/usr/app/jdk1.7/jre
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
 

$ source /etc/profile

$ java -version

3.配置ssh localhost

確保安裝好ssh：

$ sudo apt-get update
$ sudo apt-get install openssh-server
$ sudo /etc/init.d/ssh start

$ ssh-keygen -t rsa
$ cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys
$ chmod 777 /root/.ssh/authorized_keys

$ ssh localhost
$ exit

4.安裝hadoop2.6.0

解壓hadoop2.6.0到任意目錄：

$ cd /usr/app
$ wget http://apache.claz.org/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz
$ tar -xzvf hadoop-2.6.0.tar.gz

export HADOOP_HOME=/usr/app/hadoop-2.6.0
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

$ vim $HADOOP_HOME/etc/hadoop/hadoop-env.sh

export JAVA_HOME=/usr/app/jdk1.7

$ cd $HADOOP_HOME/etc/hadoop

修改core-site.xml：

注意：hdfs://為你的主機名或者IP:9000

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://hostname or ip:9000</value>
</property>
</configuration>

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>

<property>
<name>dfs.name.dir</name>
<value>file:///home/hadoopdata/hdfs/namenode</value>
</property>

<property>
<name>dfs.data.dir</name>
<value>file:///home/hadoopdata/hdfs/datanode</value>
</property>
</configuration>

第一個是dfs的備份數目，單機用1份就行，後面兩個是namenode和datanode的目錄。

修改mapred-site.xml：

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

<configuration>
 <property>
  <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
 </property>
</configuration>

$ hdfs namenode -format

啟動

$ $HADOOP_HOME/sbin/start-all.sh

$ $HADOOP_HOME/sbin/stop-all.sh

• port 8088: cluster and all applications

• port 50070: Hadoop NameNode

• port 50090: Secondary NameNode

• port 50075: DataNode

hadoop執行後可使用jps命令檢視,得到結果：

[[email protected] app]# jps
  3558 NodeManager
  2907 SecondaryNameNode
  3228 DataNode
  3471 ResourceManager
  5061 Jps
  2737 NameNode

5.安裝scala

解壓scala安裝包到任意目錄：

  $ cd /usr/app
  $ tar -xzvf scala-2.10.4.tgz
  $ sudo vim /etc/profile

  export SCALA_HOME=/usr/app/scala-2.10.4
  export PATH=$SCALA_HOME/bin:$PATH

 $ source /etc/profile

 $ scala -version

6.安裝Spark

解壓spark安裝包到任意目錄：

  $ cd /usr/app
  $ tar -xzvf spark-1.6.0-bin-hadoop2.6.tgz
  $ mv spark-1.6.0-bin-hadoop2.6 spark-1.6.0
  $ sudo vim /etc/profile

  export SPARK_HOME=/usr/app/spark-1.6.0
  export PATH=$SPARK_HOME/bin:$PATH

  $ source /etc/profile

  $ cp spark-env.sh.template spark-env.sh
  $ vim spark-env.sh

在spark-env.sh中新增：

注意SPARK_MASTER_IP為你的主名或者ip

  export JAVA_HOME=/usr/app/jdk1.7
  export SCALA_HOME=/usr/app/scala-2.10.4
  export SPARK_MASTER_IP=hostname or ip</span>
  export SPARK_WORKER_MEMORY=4G

$ $SPARK_HOME/sbin/start-all.sh

$ $SPARK_HOME/sbin/stop-all.sh

$ $SPARK_HOME/bin/run-example SparkPi

Pi is roughly 3.14716