hive日常報錯之 serious problem serious problem,block 丟失
阿新 • • 發佈：2019-01-26
https://community.hortonworks.com/questions/71589/after-upgrading-to-hdp-253-using-beeline-with-a-si.html
報錯如下 參考連結如上 感覺是 orc的檔案優化導致的這個錯誤 解決辦法是
set hive.exec.orc.split.strategy=BI;
加上這麼一個引數即可
加上之後不報這個錯了 改成block missingVertex failed, vertexName=Map 5, vertexId=vertex_1507250665075_0317_1_00, diagnostics=[Vertex vertex_1507250665075_0317_1_00 [Map 5] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: erp_orders_union0124 initializer failed, vertex=vertex_1507250665075_0317_1_00 [Map 5], java.lang.RuntimeException: serious problem 09-10-2017 08:52:55 CST append ERROR - at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1172) 09-10-2017 08:52:55 CST append ERROR - at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1199) 09-10-2017 08:52:55 CST append ERROR - at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:307) 09-10-2017 08:52:55 CST append ERROR - at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:409) 09-10-2017 08:52:55 CST append ERROR - at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:155) 09-10-2017 08:52:55 CST append ERROR - at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:273) 09-10-2017 08:52:55 CST append ERROR - at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:266) 09-10-2017 08:52:55 CST append ERROR - at java.security.AccessController.doPrivileged(Native Method) 09-10-2017 08:52:55 CST append ERROR - at javax.security.auth.Subject.doAs(Subject.java:422) 09-10-2017 08:52:55 CST append ERROR - at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) 09-10-2017 08:52:55 CST append ERROR - at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:266) 09-10-2017 08:52:55 CST append ERROR - at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253) 09-10-2017 08:52:55 CST append ERROR - at java.util.concurrent.FutureTask.run(FutureTask.java:266) 09-10-2017 08:52:55 CST append ERROR - at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 09-10-2017 08:52:55 CST append ERROR - at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 09-10-2017 08:52:55 CST append ERROR - at java.lang.Thread.run(Thread.java:745) 09-10-2017 08:52:55 CST append ERROR - Caused by: java.util.concurrent.ExecutionException: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1265061168-192.168.5.206-1490093965591:blk_1074553674_813300 file=/apps/hive/warehouse/erp2.db/erp_orders_union0124/000000_0 09-10-2017 08:52:55 CST append ERROR - at java.util.concurrent.FutureTask.report(FutureTask.java:122) 09-10-2017 08:52:55 CST append ERROR - at java.util.concurrent.FutureTask.get(FutureTask.java:192) 09-10-2017 08:52:55 CST append ERROR - at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1167) 09-10-2017 08:52:55 CST append ERROR - ... 15 more 09-10-2017 08:52:55 CST append ERROR - Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1265061168-192.168.5.206-1490093965591:blk_1074553674_813300 file=/apps/hive/warehouse/erp2.db/erp_orders_union0124/000000_0 09-10-2017 08:52:55 CST append ERROR - at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:1004) 09-10-2017 08:52:55 CST append ERROR - at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:648) 09-10-2017 08:52:55 CST append ERROR - at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:898) 09-10-2017 08:52:55 CST append ERROR - at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:955) 09-10-2017 08:52:55 CST append ERROR - at java.io.DataInputStream.readFully(DataInputStream.java:195) 09-10-2017 08:52:55 CST append ERROR - at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:370) 09-10-2017 08:52:55 CST append ERROR - at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.(ReaderImpl.java:317) 09-10-2017 08:52:55 CST append ERROR - at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:238) 09-10-2017 08:52:55 CST append ERROR - at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.populateAndCacheStripeDetails(OrcInputFormat.java:1057) 09-10-2017 08:52:55 CST append ERROR - at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:969) 09-10-2017 08:52:55 CST append ERROR - at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.access$1800(OrcInputFormat.java:810) 09-10-2017 08:52:55 CST append ERROR - at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:960) 09-10-2017 08:52:55 CST append ERROR - at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:957) 09-10-2017 08:52:55 CST append ERROR - at java.security.AccessController.doPrivileged(Native Method) 09-10-2017 08:52:55 CST append ERROR - at javax.security.auth.Subject.doAs(Subject.java:422) 09-10-2017 08:52:55 CST append ERROR - at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) 09-10-2017 08:52:55 CST append ERROR - at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:957) 09-10-2017 08:52:55 CST append ERROR - at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:810) 09-10-2017 08:52:55 CST append ERROR - ... 4 more 09-10-2017 08:52:55 CST append ERROR - ] 09-10-2017 08:52:55 CST append ERROR - Vertex killed, vertexName=Reducer 4, vertexId=vertex_1507250665075_0317_1_03, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1507250665075_0317_1_03 [Reducer 4] killed/failed due to:OTHER_VERTEX_FAILURE] 09-10-2017 08:52:55 CST append ERROR - Vertex killed, vertexName=Reducer 3, vertexId=vertex_1507250665075_0317_1_02, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1507250665075_0317_1_02 [Reducer 3] killed/failed due to:OTHER_VERTEX_FAILURE] 09-10-2017 08:52:55 CST append ERROR - Vertex killed, vertexName=Map 1, vertexId=vertex_1507250665075_0317_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:87, Vertex vertex_1507250665075_0317_1_01 [Map 1] killed/failed due to:OTHER_VERTEX_FAILURE] 09-10-2017 08:52:55 CST append ERROR - DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:3
之後想起來 最近hdfs死過一回 應該就是那一回導致block missing
然後百度 hdfs fsck /
之後 hdfs fsck -delete / 刪除丟失的塊(因為我這兒的資料不怎麼重要 所以直接刪除)
再次執行 hive sql語句 ok 搞定