對不於不習慣在linux 操作的同學，往往想在windows 環境下編輯程式碼，然後利用遠端linux 伺服器的spark群集來提交任務，下面提供一個可行的方案

1 、設定環境變數

vim /etc/profile
新增：
export PYTHONPATH=$SPARK_HOME/python/:$SPARK_HOME/python/lib/py4j-0.10.4-src.zip

執行 sourse /etc/profile 生效
（注意py4j-0.10.4-src.zip要跟自己電腦上一致,如果不是root使用者,設定相應使用者環境變數（vim ~/.bashrc））

2 、在linux上的python安裝py4j包

下載地址：https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/linux-64/
具體安裝方法自己百度

3、配置Dployment

Tools > Dployment > Configuration >connection
按要求填寫
Tools > Dployment > Configuration >mappings
第一個填本地路徑，如：E:\pc_workplace\spark
第二個linux 伺服器路徑：/root/yinxiong/spark
ools > Dployment > automatic upload (勾選)
 

4、 python interpreter 配置

settings>project>projec interpreter>
點選右上邊配置按鍵，add remote 
選擇ssh,然後根據實際情況填寫

５,run配置

script：E:\pc_workplace\spark 
path mapping :E:\pc_workplace\spark=/root/yinxiong/spark

6 測試：

import os
import sys
os.environ['SPARK_HOME'] = "/opt/spark-2.2.0-bin-hadoop2.7"
 

sys.path.append("/opt/spark-2.2.0-bin-hadoop2.7/python")

try:
    from pyspark import SparkContext
    from pyspark import SparkConf

    print ("Successfully imported Spark Modules")

except ImportError as e:
    print ("Can not import Spark Modules", e)
    sys.exit(1)

conf = SparkConf().setAppName('myFirstAPP').setMaster('local') #連線spark
sc = SparkContext(conf = conf) ##生成SparkContext 物件

rdd = sc.textFile('/spark/wc.txt')
rdd_result = rdd.faltMap(lambda x:x.split()).Map(lambda x:(x,1)).reduceByKey(lambda x,y:x+y)
print(rdd_result.collect())
#wordcount.cache()

執行結果：

ssh://[email protected]:22/opt/anaconda3/bin/python3 -u /root/yinxiong/spark/demo1.py
Successfully imported Spark Modules
Setting default log level to “WARN”.
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
18/01/03 00:13:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
18/01/03 00:13:06 WARN util.Utils: Service ‘SparkUI’ could not bind on port 4040. Attempting port 4041.
[‘hello inx’, ‘hello inx’, ‘hello crystal’, ‘hello crystal’, ‘hello crystal’, ‘hello spark’]
說明配置成功