1. 程式人生 > >解決Input path does not exist:file://

解決Input path does not exist:file://

任務提交指令:

spark-submit  --class orderinversioninspark.RelativeFrequency ~/soft/temp/bigdataAnalysis.jar 2 /input/inputfile.txt /outputfi

異常部分資訊如下:

Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file://input/inputfile.txt
    at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:285)

        at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:228)

        at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:313)

按理說,預設應該尋找HDFS目錄下的檔案,現在卻找的是本地檔案。具體原因想了很久也沒想明白,待續。

附上自己的解決方案:

spark-submit --class orderinversioninspark.RelativeFrequency ~/soft/temp/bigdataAnalysis.jar 2 hdfs://master:8020

/input/inputfile.txt /outputfi0

即新增完整的hdfs URL。

另外,若要使用本地檔案的話那麼URL改為:file:///home/centos/soft/files/inputfile.txt