參考文章：https://apacheignite-fs.readme.io/docs/installation-deployment

Spark application deployment model allows dynamic jar distribution during application start. This model, however, has some drawbacks:

  • Spark dynamic class loader does not implement getResource methods, so you will not be able to access resources located in jar files.
  • Java logger uses application class loader (not the context class loader) to load log handlers which results in ClassNotFoundException when using Java logging in Ignite.

There is a way to alter default Spark classpath for each launched application (this should be done on each machine of the Spark cluster, including master, worker and driver nodes).

  1. Locate the $SPARK_HOME/conf/spark-env.sh file. If this file does not exist, create it from template using $SPARK_HOME/conf/spark-env.sh.template
  2. Add the following lines to the end of the spark-env.sh file (uncomment the line settingIGNITE_HOME in case if you do not have it globally set):
# Optionally set IGNITE_HOME here.

# IGNITE_HOME=/path/to/ignite

IGNITE_LIBS="${IGNITE_HOME}/libs/*"

for file in ${IGNITE_HOME}/libs/*

do

    if [ -d ${file} ] && [ "${file}" != "${IGNITE_HOME}"/libs/optional ]; then

        IGNITE_LIBS=${IGNITE_LIBS}:${file}/*

    fi

done

export SPARK_CLASSPATH=$IGNITE_LIBS
 

You can verify that the Spark classpath is changed by running bin/spark-shell and typing a simple import statement:

scala> import org.apache.ignite.configuration._

import org.apache.ignite.configuration._

官方給出的以上解決方案，在spark2.1上是可以的，但是在spark2.2中不起作用，對比了一下spark2.1和spark2.2中的spark-env.sh指令碼，發現其中一些變數，在spark2.2中不存在了：