解決Input path does not exist:file://
任務提交指令:
spark-submit --class orderinversioninspark.RelativeFrequency ~/soft/temp/bigdataAnalysis.jar 2 /input/inputfile.txt /outputfi
異常部分資訊如下:
Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file://input/inputfile.txt
at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:285)
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:228)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:313)
按理說,預設應該尋找HDFS目錄下的檔案,現在卻找的是本地檔案。具體原因想了很久也沒想明白,待續。
附上自己的解決方案:
spark-submit --class orderinversioninspark.RelativeFrequency ~/soft/temp/bigdataAnalysis.jar 2 hdfs://master:8020
即新增完整的hdfs URL。
另外,若要使用本地檔案的話那麼URL改為:file:///home/centos/soft/files/inputfile.txt