spark 2.0.0集群安裝與hive on spark配置
阿新 • • 發佈：2019-02-09
3.6 temp format info port urn mysql5.7 del cnblogs
1. 環境準備:
JDK1.8
hive 2.3.4
hadoop 2.7.3
hbase 1.3.3
scala 2.11.12
mysql5.7
2. 下載spark2.0.0
cd /home/worksapce/software wget https://archive.apache.org/dist/spark/spark-2.0.0/spark-2.0.0-bin-hadoop2.7.tgz tar -xzvf spark-2.0.0-bin-hadoop2.7.tgz mv spark-2.0.0-bin-hadoop2.7 spark-2.0.0
3. 配置系統環境變量
vim /etc/profile
末尾添加
#spark export SPARK_HOME=/home/workspace/software/spark-2.0.0 export PATH=:$PATH:$SPARK_HOME/bin
4. 配置spark-env.sh
cd /home/workspace/software/spark-2.0.0/conf cp spark-env.sh.template spark-env.sh vim spark-env.sh
末尾添加:
export JAVA_HOME=/usr/java/jdk1.8.0_172-amd64 export SCALA_HOME=/home/workspace/software/scala-2.11.12 export HADOOP_HOME=/home/workspace/hadoop-2.7.3 export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop export SPARK_HOME=/home/workspace/software/spark-2.0.0 export SPARK_DIST_CLASSPATH=$(/home/workspace/hadoop-2.7.3/bin/hadoop classpath) export SPARK_LIBRARY_PATH=$SPARK_HOME/lib export SPARK_LAUNCH_WITH_SCALA=0 export SPARK_WORKER_DIR=$SPARK_HOME/work export SPARK_LOG_DIR=$SPARK_HOME/logs export SPARK_PID_DIR=$SPARK_HOME/run export SPARK_MASTER_IP=192.168.1.101 export SPARK_MASTER_HOST=192.168.1.101 export SPARK_MASTER_WEBUI_PORT=18080 export SPARK_MASTER_PORT=7077 export SPARK_LOCAL_IP=192.168.1.101 export SPARK_WORKER_CORES=4 export SPARK_WORKER_PORT=7078 export SPARK_WORKER_MEMORY=4g export SPARK_DRIVER_MEMORY=4g export SPARK_EXECUTOR_MEMORY=4g
5. 配置spark-defaults.conf
cd /home/workspace/software/spark-2.0.0/conf cp spark-defaults.conf.template spark-defaults.conf vim spark-defaults.conf
末尾添加
spark.master spark://192.168.1.101:7077 spark.eventLog.enabled true spark.eventLog.dir hdfs://192.168.1.101:9000/spark-log spark.serializer org.apache.spark.serializer.KryoSerializer spark.executor.memory 4g spark.driver.memory 4g spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"
6. 配置slaves
cd /home/workspace/software/spark-2.0.0/conf cp slaves.template slaves vim slaves
末尾添加
192.168.1.101 192.168.1.102 192.168.1.103
7. 創建相關目錄（在spark-env.sh中定義)
hdfs dfs -mkdir -p /spark-log hdfs dfs -chmod 777 /spark-log mkdir -p $SPARK_HOME/work $SPARK_HOME/logs $SPARK_HOME/run
mkdir -p $HIVE_HOME/logs
8.修改hive-site.xml
vim $HIVE_HOME/conf/hive-site.xml
把文件內容修改為
<?xml version="1.0" encoding="UTF-8" standalone="no"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>hive.metastore.schema.verification</name> <value>false</value> </property> <property> <name>hive.metastore.warehouse.dir</name> <value>/hive/warehouse</value> <description>location of default database for the warehouse</description> </property> <property> <name>hive.exec.scratchdir</name> <value>/hive/tmp</value> <description>Scratch space for Hive jobs</description> </property> <property> <name>hive.querylog.location</name> <value>/hive/log</value> </property> <property> <name>hive.metastore.uris</name> <value>thrift://192.168.1.103:9083</value> </property> <!--hive server2 settings--> <property> <name>hive.server2.thrift.bind.host</name> <value>192.168.1.103</value> </property> <property> <name>hive.server2.thrift.port</name> <value>10000</value> </property> <property> <name>hive.server2.webui.host</name> <value>192.168.1.103</value> </property> <property> <name>hive.server2.webui.host.port</name> <value>10002</value> </property> <property> <name>hive.server2.long.polling.timeout</name> <value>5000</value> </property> <property> <name>hive.server2.enable.doAs</name> <value>true</value> </property> <!--metadata database connection string settings--> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://192.168.1.103:3308/hive?createDatabaseIfNotExist=true</value> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>hive</value> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>hive</value> </property> <property> <name>datanucleus.autoCreateSchema </name> <value>false</value> <description>creates necessary schema on a startup if one doesn‘t exist. set this to false, after creating it once</description> </property> <property> <name>datanucleus.fixedDatastore</name> <value>true</value> </property> <!-- hive on mr--> <!-- <property> <name>mapred.job.tracker</name> <value>http://192.168.1.101:9001</value> </property> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> --> <!--hive on spark or spark on yarn --> <property> <name>hive.execution.engine</name> <value>spark</value> </property> <property> <name>spark.home</name> <value>/home/workspace/software/spark-2.0.0</value> </property> <property> <name>spark.master</name> <value>spark://192.168.1.101:7077</value> <!-- 或者yarn-cluster/yarn-client --> </property> <property> <name>spark.submit.deployMode</name> <value>client</value> </property> <property> <name>spark.eventLog.enabled</name> <value>true</value> </property> <property> <name>spark.eventLog.dir</name> <value>hdfs://192.168.1.101:9000/spark-log</value> </property> <property> <name>spark.serializer</name> <value>org.apache.spark.serializer.KryoSerializer</value> </property> <property> <name>spark.executor.memeory</name> <value>4g</value> </property> <property> <name>spark.driver.memeory</name> <value>4g</value> </property> <property> <name>spark.executor.extraJavaOptions</name> <value>-XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"</value> </property> <!--concurrency support--> <property> <name>hive.support.concurrency</name> <value>true</value> <description>Whether hive supports concurrency or not. A zookeeper instance must be up and running for the default hive lock manager to support read-write locks.</description> </property> <property> <name>hive.exec.dynamic.partition.mode</name> <value>nonstrict</value> </property> <!--transaction support--> <property> <name>hive.txn.manager</name> <value>org.apache.hadoop.hive.ql.lockmgr.DbTxnManager</value> </property> <property> <name>hive.compactor.initiator.on</name> <value>true</value> </property> <property> <name>hive.compactor.worker.threads</name> <value>1</value> </property> <property> <name>hive.stats.autogather</name> <value>true</value> <description>A flag to gather statistics automatically during the INSERT OVERWRITE command.</description> </property> <!--hive web interface settings, I think this is useless,so comment it--> <!-- <property> <name>hive.hwi.listen.host</name> <value>192.168.1.131</value> </property> <property> <name>hive.hwi.listen.port</name> <value>9999</value> </property> <property> <name>hive.hwi.war.file</name> <value>lib/hive-hwi-2.1.1.war</value> </property> --> </configuration>
9. 拷貝hive-site.xml到spark/conf下
cp $HIVE_HOME/conf/hive-site.xml $SPARK_HOME/conf
10 分發到192.168.1.102，192.168.1.103
cd /home/workspace/software/ scp -r spark-2.0.0 192.168.1.102:/home/workspace/software scp -r spark-2.0.0 192.168.1.103:/home/workspace/software
修改102，103上的SPARK_LOCAL_IP值
vim /home/workspace/software/spark-2.0.0/conf/spark-env.sh
將SPARK_LOCAL_IP分別改為192.168.1.102，192.168.1.103
11 將mysql jar包復制到$SPARK_HOME/lib目錄下（每臺機器都要做)
cp $HIVE_HOME/lib/mysql-connector-java-5.1.47.jar $SPARK_HOME/lib
註：本例中之前已經安裝好hive，如果沒有，請到mysql官網網站下載對應的jdbc jar包
12. 啟動spark集群
在spark master節點上（本例為192.168.1.101)執行下面語句
$SPARK_HOME/sbin/start-all.sh
192.168.1.101
192.168.1.102:
192.168.1.103:
瀏覽器打開http:192.168.1.101:18080
13.測試使用
[druid@palo101 apache-maven-3.6.0]$ hive /tmp/druid Logging initialized using configuration in file:/home/workspace/software/apache-hive-2.3.4/conf/hive-log4j2.properties Async: true hive> use kylin_flat_db; OK Time taken: 1.794 seconds hive> desc kylin_sales; OK trans_id bigint part_dt date Order Date lstg_format_name string Order Transaction Type leaf_categ_id bigint Category ID lstg_site_id int Site ID slr_segment_cd smallint price decimal(19,4) Order Price item_count bigint Number of Purchased Goods seller_id bigint Seller ID buyer_id bigint Buyer ID ops_user_id string System User ID ops_region string System User Region Time taken: 0.579 seconds, Fetched: 12 row(s) hive> select trans_id, sum(price) as total, count(seller_id) as cnt from kylin_sales group by trans_id order by cnt desc limit 10; Query ID = druid_20190209000716_9676460c-1a76-456d-9bd6-b6f557d5e02c Total jobs = 1 Launching Job 1 out of 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Spark Job = 72720bf1-750d-4f6f-bf9c-5cffa0e4c73b Query Hive on Spark job[0] stages: [0, 1, 2] Status: Running (Hive on Spark job[0]) -------------------------------------------------------------------------------------- STAGES ATTEMPT STATUS TOTAL COMPLETED RUNNING PENDING FAILED -------------------------------------------------------------------------------------- Stage-0 ........ 0 FINISHED 1 1 0 0 0 Stage-1 ........ 0 FINISHED 1 1 0 0 0 Stage-2 ........ 0 FINISHED 1 1 0 0 0 -------------------------------------------------------------------------------------- STAGES: 03/03 [==========================>>] 100% ELAPSED TIME: 10.12 s -------------------------------------------------------------------------------------- Status: Finished successfully in 10.12 seconds OK 8621 33.4547 1 384 15.4188 1 7608 88.6492 1 9166 40.4308 1 9215 63.5407 1 4551 59.2537 1 7041 79.8884 1 522 18.3204 1 5618 78.6241 1 9831 5.8088 1 Time taken: 21.788 seconds, Fetched: 10 row(s) hive>
13 FAQ:
13.1 如果在使用過程中遇到類似下面的錯誤
Exception in thread "main" java.lang.NoSuchFieldError: SPARK_RPC_SERVER_ADDRESS
通過查看hive的日誌文件（在/tmp/{user}/hive.log),這是因為默認使用的spark安裝包是繼承了hive的包，名字為spark-xxx-bin-hadoopxx.xx.tgz都是繼承了hive的包，在hive on spark模式下，會出現沖突，解決辦法有兩個：
1） 手動編譯spark不包含hive的包，具體請參見本人的博文Spark2.0.0源碼編譯,編譯指令為：
./make-distribution.sh --name "hadoop2.7.3-without-hive" --tgz -Dhadoop.version=2.7.3 -Dscala-2.11 -Phadoop-2.7 -Pyarn -DskipTests clean package
用編譯出來的包來安裝。
2） 刪除預編譯包中hive的jar包，具體操作為：
cd $SPARK_HOME/jars rm -f hive-* rm -rf spark-hive_* #刪除下面6個文件 # hive-beeline-1.2.1.spark2.jar # hive-cli-1.2.1.spark2.jar # hive-exec-1.2.1.spark2.jar # hive-jdbc-1.2.1.spark2.jar # hive-metastore-1.2.1.spark2.jar # spark-hive_2.11-2.0.0.jar # spark-hive-thriftserver_2.11-2.0.0.jar
註意：每臺機器都要做.
13.2 如果出現類似下面的錯誤
Exception in thread "main" java.lang.NoClassDefFoundError: scala/collection/Iterable at org.apache.hadoop.hive.ql.optimizer.spark.SetSparkReducerParallelism.getSparkMemoryAndCores(SetSparkReducerParallelism.java:236) at org.apache.hadoop.hive.ql.optimizer.spark.SetSparkReducerParallelism.process(SetSparkReducerParallelism.java:173) at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89) at org.apache.hadoop.hive.ql.lib.PreOrderWalker.walk(PreOrderWalker.java:56) at org.apache.hadoop.hive.ql.lib.PreOrderWalker.walk(PreOrderWalker.java:61) at org.apache.hadoop.hive.ql.lib.PreOrderWalker.walk(PreOrderWalker.java:61) at org.apache.hadoop.hive.ql.lib.PreOrderWalker.walk(PreOrderWalker.java:61) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120) at org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.runSetReducerParallelism(SparkCompiler.java:288) at org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.optimizeOperatorPlan(SparkCompiler.java:122) at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:140) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11273) at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: java.lang.ClassNotFoundException: scala.collection.Iterable at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
這是因為hive無法加載spark的jar包，解決辦法為：
$HIVE_HOME/bin/hive
在執行hive之前添加下面的語句，把spark的jar包添加到hive的class path中
SPARK_HOME=/home/workspace/software/spark-2.0.0 for f in ${SPARK_HOME}/jars/*.jar; do CLASSPATH=${CLASSPATH}:$f; done
本人添加的位置為：
或者直接把$SPARK_HOME/jars/spark*復制到$HIVE_HOME/lib下，
cp $SPARK_HOME/jars/spark* $HIVE_HOME/lib
個人感覺修改hive啟動腳本更好一些。
spark 2.0.0集群安裝與hive on spark配置