hbase常見問題及解決方案(二)
阿新 • • 發佈：2019-01-09
hbase常見問題總結二
之前介紹了工作中遇到的一些比較基礎的問題,下面介紹一些看起來沒有那麼簡單的問題
1. 類找不見的問題!(自己寫的類找不見的問題!)
出現該問題的情形: hbase和hadoop的hdfs,mapreduce整合使用的時候:
18/04/16 18:25:06 INFO mapreduce.JobSubmitter: Cleaning up the staging area /user/mingtong/.staging/job_1522546194099_223330
Exception in thread "main" java.lang.RuntimeException Configuration conf = HBaseConfiguration.create();
只要你的configuration使用的是hbase的,而且後面mapReduce的job用到這個conf,就會報這個問題!解決方法: 乖乖的使用Configuration conf = new Configuration(); 來建立conf吧
但是這種方法建立的conf,不會去載入hbase-site.xml配置檔案,
hbase-site.xml裡面重要的引數需要手動set!!
否則就無法正確的連線到Hbase! ```
由於上面介紹的問題還會引發下面的報錯:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /wangyou/mingtong/mt_wlyh/Data/hbase_bulkload/output/4503/inin (inode 1964063475): File does not exist. Holder DFSClient_NONMAPREDUCE_-769553346_1 does not have any open files.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3521)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3611)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:3578)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:905)
按照上述方法改進後,該問題就得到解決!
```
2.執行MapReduce遇到的問題：檔案租約超期異常.
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException
這個問題實際上就是data stream操作過程中檔案被刪掉了。之前也遇到過，通常是因為Mapred多個task操作同一個檔案，一個task完成後刪掉檔案導致,將可能造成這種情況的程式碼進行修改即可
我遇到這種問題的另一種情形 就是: 因為mapReduce之前的一些錯誤,job一直報錯... 到後面導致的這個問題,這種情況下,不要理會這個報錯,只需要解決前面的問題這個問題就迎刃而解3.連線Hbase時, 明明hbase.zookeeper.quorum 和hbase.zookeeper.property.clientPort的設定都是正確的,卻總是報錯 INFO client.ZooKeeperRegistry: ClusterId read in ZooKeeper is null
首先,這種情況出現在: 使用的configuration 是 new configuration這種方式獲得的
這裡: 涉及到一個關鍵的配置:
zookeeper.znode.parent --> 這個值的預設值是/hbase
但是如果叢集裡面設定的值不是這個的話,就會丟擲這個異常!比如說我們的叢集:
因為使用 new Configuration()獲得的configuration物件是不會讀取Hbase的配置檔案hbase-site.xml檔案的
在程式碼中將該配置按照hbase-site.xml裡面配置的新增進來即可
conf.set("zookeeper.znode.parent", "/hbase-unsecure");
這樣,該問題得到解決!
4.使用bulkload入庫遇到的另外一個問題!
報錯資訊如下所示:
Exception in thread "main" java.lang.IllegalArgumentException: Can not create a Path from a null string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:122)
at org.apache.hadoop.fs.Path.<init>(Path.java:134)
at org.apache.hadoop.fs.Path.<init>(Path.java:88)
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configurePartitioner(HFileOutputFormat2.java:596)
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configureIncrementalLoad(HFileOutputFormat2.java:445)
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configureIncrementalLoad(HFileOutputFormat2.java:410)
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configureIncrementalLoad(HFileOutputFormat2.java:372)
at mastercom.cn.hbase.helper.AddPaths.addUnCombineConfigJob(AddPaths.java:272)
at mastercom.cn.hbase.config.HbaseBulkloadConfigMain.CreateJob(HbaseBulkloadConfigMain.java:129)
at mastercom.cn.hbase.config.HbaseBulkloadConfigMain.main(HbaseBulkloadConfigMain.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)```
由報錯資訊上可以看出來:是在HFileOutputFormat2類裡面出現的錯誤
這個類是使用bulkload方式進行入庫的很關鍵的類
我們接下來一步一步的去定位錯誤:
丟擲來的錯誤資訊是來自於path類的這個方法:
private void checkPathArg( String path ) throws IllegalArgumentException {
// disallow construction of a Path from an empty string
if ( path == null ) {
throw new IllegalArgumentException(
"Can not create a Path from a null string");
}
if( path.length() == 0 ) {
throw new IllegalArgumentException(
"Can not create a Path from an empty string");
}
}
根據介面上的報錯結合一下: 可以得到path是一個null,
那麼這個空是從何而來,我們繼續看原始碼
static void configurePartitioner(Job job, List<ImmutableBytesWritable> splitPoints)
throws IOException {
Configuration conf = job.getConfiguration();
// create the partitions file
FileSystem fs = FileSystem.get(conf);
Path partitionsPath = new Path(conf.get("hbase.fs.tmp.dir"), "partitions_" + UUID.randomUUID());
fs.makeQualified(partitionsPath);
writePartitions(conf, partitionsPath, splitPoints);
fs.deleteOnExit(partitionsPath);
// configure job to use it
job.setPartitionerClass(TotalOrderPartitioner.class);
TotalOrderPartitioner.setPartitionFile(conf, partitionsPath);
}分析上面的原始碼,能夠產生null的又和path相關的,顯然是這行程式碼:
Path(conf.get("hbase.fs.tmp.dir"), "partitions_" + UUID.randomUUID());
我們不妨測試一下,在獲得conf物件後,列印一下hbase.fs.tmp.dir的值,果然為空!
那麼問題已經確認,只需要在程式碼裡面加上這行!
conf.set("hbase.fs.tmp.dir", "/wangyou/mingtong/mt_wlyh/tmp/hbase-staging");
問題便得到解決,入庫工作得以正常執行!5.gz壓縮檔案損壞導致入庫失敗的問題
ERROR hdfs.DFSClient: Failed to close inode 16886732
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /hbase_bulkload/output/inin (inode 16886732): File does not exist. Holder DFSClient_NONMAPREDUCE_1351255084_1 does not have any open files.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3431)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3521)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:3488)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:785)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:536)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
at org.apache.hadoop.ipc.Client.call(Client.java:1476)
at org.apache.hadoop.ipc.Client.call(Client.java:1407)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy9.complete(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:462)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy10.complete(Unknown Source)
at org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2257)
at org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:2238)
at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2204)
at org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:951)
at org.apache.hadoop.hdfs.DFSClient.closeOutputStreams(DFSClient.java:983)
at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:1076)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2744)
at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2761)
at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)該問題的場景是在對大量的小的.gz壓縮檔案進行入庫的時候,個別壓縮檔案損壞導致的,解決的方法就是找到那些出錯的.gz檔案刪除掉
我當時使用的方法: 1. 首先去介面檢視相應的job執行的日誌,日誌裡有可能會有出錯的.gz檔案的id資訊,找到將其刪除
2. 將入庫的資料夾下面的檔案按照檔案大小進行排序,一般來說,大小為0KB的都是有問題的.. 將其get下來,檢視能否解壓,不能正常解壓就幹掉
3. 可以使用命令: hdfs fsck path -openforwrite
檢測某個資料夾下面檔案是否正常6.查詢hbase的時候報錯:
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.util.ByteStringer
at org.apache.hadoop.hbase.protobuf.RequestConverter.buildRegionSpecifier(RequestConverter.java:989)
at org.apache.hadoop.hbase.protobuf.RequestConverter.buildScanRequest(RequestConverter.java:485)
at org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:195)
at org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:181)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
... 6 more
java.lang.NullPointerException
at mastercom.cn.bigdata.util.hbase.HbaseDBHelper.qureyAsList(HbaseDBHelper.java:86)
at conf.config.CellBuildInfo.loadCellBuildHbase(CellBuildInfo.java:150)
at mro.loc.MroXdrDeal.init(MroXdrDeal.java:200)
at mapr.mro.loc.MroLableFileReducers$MroDataFileReducers.reduce(MroLableFileReducers.java:80)
at mapr.mro.loc.MroLableFileReducers$MroDataFileReducers.reduce(MroLableFileReducers.java:1)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.util.ByteStringer
at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:229)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:140)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:310)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:291)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.util.ByteStringer
at org.apache.hadoop.hbase.protobuf.RequestConverter.buildRegionSpecifier(RequestConverter.java:989)
at org.apache.hadoop.hbase.protobuf.RequestConverter.buildScanRequest(RequestConverter.java:485)
at org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:195)
at org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:181)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
... 6 more
java.lang.NullPointerException
at mastercom.cn.bigdata.util.hbase.HbaseDBHelper.qureyAsList(HbaseDBHelper.java:86)
at conf.config.CellBuildInfo.loadCellBuildHbase(CellBuildInfo.java:150)
at mro.loc.MroXdrDeal.init(MroXdrDeal.java:200)
at mapr.mro.loc.MroLableFileReducers$MroDataFileReducers.reduce(MroLableFileReducers.java:80)
at mapr.mro.loc.MroLableFileReducers$MroDataFileReducers.reduce(MroLableFileReducers.java:1)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)下面是出現問題的程式碼:
/**
* 根據行鍵進行查詢,返回的結果是一個List集合
*
* @param tableName
* @param rowKey
* @param conn
* @return
*/
public List<String> qureyAsList(String tableName, String rowKey, Connection conn) {
getList = new ArrayList<Get>();
valueList = new ArrayList<String>();
try {
table = conn.getTable(TableName.valueOf(tableName));
} catch (IOException e) {
LOGHelper.GetLogger().writeLog(LogType.error,"get table error" + e.getMessage());
}
// 把rowkey加到get裡，再把get裝到list中
Get get = new Get(Bytes.toBytes(rowKey));
getList.add(get);
try {
results = table.get(getList);
} catch (IOException e) {
LOGHelper.GetLogger().writeLog(LogType.error,"can't get results" + e.getMessage());
}
for (Result result : results) {
for (Cell kv : result.rawCells()) {
String value = Bytes.toString(CellUtil.cloneValue(kv));
valueList.add(value);
}
}
return valueList;
}
這個空指標異常也是挺噁心的,
我已經正常連線到了hbase,而且表名也是正常的...
原來是程式碼不夠嚴謹: 在一些情況下,根據行鍵進行查詢,可能得到的結果集是null,但是我的程式碼裡並沒有加上對可能出現的空指標異常進行處理的機制,然後使用for迴圈遍歷這個空的結果集
for (Result result : results)
遍歷一個空的結果集當然會報錯啦!
解決方法: 前面加上一個判斷,就解決了!