寫在前面：

本想使用hive呼叫python指令碼實現統計分析movielens資料，但是最後一步呼叫指令碼的地方不成功沒找到問題所在，於是將過程中的一些經驗寫出來，非常詳盡，對新手來說應該挺有用的。

另外呼叫指令碼的程式和報錯我會貼出來，應該是指令碼寫的有問題，後面找到問題或者有人告訴我我會更新。

還拿hive與movie lens資料說事兒。

1、首先進入hive資料庫建立基表：

2、在linxu檔案工作資料夾下下載資料資源並且解壓，我的目錄是opt/jskp/jinjiwei：

wget http://files.grouplens.org/datasets/movielens/ml-100
 
k.zip

3、在hdfs上新建自己的工作資料夾，我的是hdfs dfs -mkdir 檔名（JJW）

4、將本地解壓的檔案上傳到hdfs：

hdfs dfs -put /opt/jskp/jinjiwei/ml-100k /JJW（hdfs目錄）

在hdfs上面檢視上傳結果：

5、將ml-100k檔案下的u.data檔案載入到hive資料庫前面建的基表JJW中：

可以看到我第一次載入檔案路徑是本地路徑是錯誤的，第二次是hdfs上面路徑，結果正確，下面驗證載入結果：

可以在hive中及進行一些簡單的統計如：

6、建立子表JJW_new，用於把基表JJW資料匯入子表（因為我呼叫python指令碼不成功，這裡就直接匯入了）

CREATE TABLE JJW_new (
  userid INT,
  movieid INT,
  rating INT,
  weekday INT)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t';

7、編寫python指令碼，功能僅僅將unix時間改為正常時間戳：

import sys
import datetime

for line in sys.stdin:
  line = line.strip()
  userid, movieid, rating, unixtime = line.split('\t')
  weekday=datetime.datetime.fromtimestamp(float(unixtime)).isoweekday()
  print '\t'
 
.join([userid, movieid, rating, str(weekday)])

8、新增本地python指令碼如下圖，路徑為本地絕對路徑：

9、最終不呼叫python指令碼方法：

INSERT OVERWRITE TABLE JJW_new
SELECT
  userid, movieid, ratingid, unixtime
FROM JJW;

驗證：

10、引用指令碼方法：

INSERT OVERWRITE TABLE JJW_new
SELECT
  TRANSFORM (userid, movieid, ratingid, unixtime)
  USING 'python weekday_mapper.py'
  AS (userid, movieid, rating, weekday)
FROM JJW;

報錯：

hive> INSERT OVERWRITE TABLE JJW_new
    > SELECT
    >   TRANSFORM (userid, movieid, ratingid, unixtime)
    >   USING 'python tansform.py'
    >   AS (userid, movieid, rating, weekday) 
    > FROM JJW;
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1526968712310_2578, Tracking URL = http://hm:8088/proxy/application_1526968712310_2578/
Kill Command = /opt/software/hadoop/hadoop-2.6.4/bin/hadoop job  -kill job_1526968712310_2578
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2018-06-28 13:00:12,907 Stage-1 map = 0%,  reduce = 0%
2018-06-28 13:00:42,417 Stage-1 map = 100%,  reduce = 0%
Ended Job = job_1526968712310_2578 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1526968712310_2578_m_000000 (and more) from job job_1526968712310_2578

Task with the most failures(4): 
-----
Task ID:
  task_1526968712310_2578_m_000000

URL:
  http://hm:8088/taskdetails.jsp?jobid=job_1526968712310_2578&tipid=task_1526968712310_2578_m_000000
-----
Diagnostic Messages for this Task:
Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"userid":47,"movieid":324,"ratingid":3,"unixtime":"879439078"}
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:195)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"userid":47,"movieid":324,"ratingid":3,"unixtime":"879439078"}
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:550)
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
    ... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: [Error 20001]: An error occurred while reading or writing to your custom script. It may have crashed with an error.
    at org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp(ScriptOperator.java:410)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
    at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
    at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:540)
    ... 9 more
Caused by: java.io.IOException: Stream closed
    at java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433)
    at java.io.OutputStream.write(OutputStream.java:116)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
    at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
    at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
    at java.io.DataOutputStream.write(DataOutputStream.java:107)
    at org.apache.hadoop.hive.ql.exec.TextRecordWriter.write(TextRecordWriter.java:53)
    at org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp(ScriptOperator.java:378)
    ... 15 more


FAILED: Execution Error, return code 20001 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. An error occurred while reading or writing to your custom script. It may have crashed with an error.
MapReduce Jobs Launched: 
Job 0: Map: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

11、使用到的命令總結一波：

本地資料載入到hive表（可覆蓋表內容）

LOAD DATA LOCAL INPATH '/opt/jskp/jinjiwei/ml-100k/u.data’  
OVERWRITE INTO TABLE jjw; 

本地資料載入到hive表（不可覆蓋表內容）

LOAD DATA INPATH '/opt/jskp/jinjiwei/ml-100k/u.data' INTO table testkv;

本地資料上傳到hdfs

hdfs dfs -put /opt/jskp/jinjiwei/ml-100k.zip /JJW

修改hdfs的一個檔案：

獲取
hdfs dfs -get JJW/transform.py
修改
vi transform.py
上傳
hdfs dfs -put -f test.txt yourHdfsPath/test.txt

基本hadoop dfs與hdfs dfs可互換,後面跟的引數大多為linux命令，如hdfs dfs -ls , hdfs dfs -mkdir等。
另外hive與linux互動：可在hive環境下用！linux命令如！find等。
其他一些命令可以參考這裡（不是我寫的哦）：HDFS 常用檔案操作命令