1. 程式人生 > >SparkStreaming任務保持執行,定時任務監控程序

SparkStreaming任務保持執行,定時任務監控程序

cron任務:每隔1分鐘啟動指令碼,檢查程序是否執行。

*/1 * * * * /data1/spark-1.6.0/startEnginea.sh

檢查程序,如果程序掛掉,重新啟動Spark任務:

#!/bin/sh

is_Engine_exist=$(ps aux | grep LbsStreamingEngineTJ | grep -v grep | wc -l)

if [ $is_Engine_exist = 0 ];then

        echo 'Process Engine is down'

        echo 'Bring Engine up'

        strDate=`date +%Y%m%d%H%M%S`

        strStart="start Engine ${strDate}"

        echo "${strStart}" >> /data1/log.txt

        nohup /data1/spark-1.6.0/bin/spark-submit --master  spark://localhost:7077 --name LbsStreamingEngineTJ --class com.datafactory.streaming.LbsStreamingEngineTJ --executor-memory 512m --total-executor-cores 2 /data1/work/datafactory-0.1.0-SNAPSHOT1023.jar "some${strDate}" 10 15000 &

        echo 'Bring Engine finished '

else

        strDate=`date +%Y%m%d%H%M%S`

        strRun="running ${strDate}"

        echo "${strRun}" >> /data1/log.txt

fi