1. 程式人生 > >hive運維記錄20111117

hive運維記錄20111117

1)執行hive cli客戶端時報錯:

Mkdirs failed to create /tmp/hadoop-test/hadoop-unjar15513
    錯誤原因:hive客戶端所在伺服器/tmp目錄已滿

    解決辦法:a)清空/tmp目錄;b)修改hive的臨時目錄

2)hive提交的作業始終是map 0% red 0%的狀態:

    作業圖示如下:

  

   發現該job分配的map數為零,最初還以為是jobtracker的問題(因為剛上了fair scheduler,且恰巧昨天該job執行時,jobtracker也丟擲了異常),但是今天覆現時,發現其他的mapred job都能夠正常執行,所以在仔細查看了日誌後發現:

2011-11-17 10:17:36,071 INFO org.apache.hadoop.mapred.JobInProgress: Input size for job job_201111170937_0037 = 0. Number of splits = 0
2011-11-17 10:17:36,071 INFO org.apache.hadoop.mapred.JobInProgress: Job job_201111170937_0037 initialized successfully with 0 map tasks and 1 reduce tasks.
  該作業的輸入資料分片為0,所以jobtracker分配了零個map,一個red槽位,從而導致了該作業始終是map 0% red 0%的狀態,不知道這算不算是mapred的一個bug

   3)在安裝新的hive服務時,必須要在hosts檔案中新增hadoop叢集伺服器的ip hostname資訊,且還需要新增本機的ip hostname資訊,否則會丟擲以下錯誤:

java.net.UnknownHostException: test: test
        at java.net.InetAddress.getLocalHost(InetAddress.java:1354)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:815)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:791)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:791)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:765)
        at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:657)
        at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:123)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:130)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1063)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:900)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:748)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:456)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Job Submission failed with exception 'java.net.UnknownHostException(test: test)'

     4)hive記憶體分配錯誤資訊:
Exception in thread "pool-1-thread-4345" java.lang.OutOfMemoryError: Java heap space

    5)hive檢視hadoop配置:

set -v 可得到所有環境變數。如果沒有-v引數,只顯示與hadoop不同的配置。