phoenix-hbase 服務頻繁掛掉問題排查

阿新 • • 發佈：2019-01-03

call(Client.java:1475) at org.apache.hadoop.ipc.Client.call(Client.java:1408) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) at com.sun.proxy.$Proxy9.transitionToStandby(Unknown Source) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.transitionToStandby(HAServiceProtocolClientSideTranslatorPB.java:112

) at org.apache.hadoop.ha.FailoverController.tryGracefulFence(FailoverController.java:172) at org.apache.hadoop.ha.ZKFailoverController.doFence(ZKFailoverController.java:511) at org.apache.hadoop.ha.ZKFailoverController.fenceOldActive(ZKFailoverController.java:502) at org.apache.hadoop.ha.ZKFailoverController.access$1100

(ZKFailoverController.java:60) at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.fenceOldActive(ZKFailoverController.java:888) at org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:909) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:808

) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:417) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:599) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: java.net.ConnectException: 拒絕連線 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:713) at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1524) at org.apache.hadoop.ipc.Client.call(Client.java:1447) ... 14 more 2017-08-03 05:31:33,927 INFO org.apache.hadoop.ha.NodeFencer: ====== Beginning Service Fencing Process... ====== 2017-08-03 05:31:33,927 INFO org.apache.hadoop.ha.NodeFencer: Trying method 1/1: org.apache.hadoop.ha.SshFenceByTcpPort(null) 2017-08-03 05:31:34,092 INFO org.apache.hadoop.ha.SshFenceByTcpPort: Connecting to hadoop171... 2017-08-03 05:31:34,096 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Connecting to hadoop171 port 22 2017-08-03 05:31:34,104 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Connection established 2017-08-03 05:31:34,122 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Remote version string: SSH-2.0-OpenSSH_6.6.1 2017-08-03 05:31:34,122 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Local version string: SSH-2.0-JSCH-0.1.42 2017-08-03 05:31:34,123 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: CheckCiphers: aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des-ctr,arcfour,arcfour128,arcfour256 2017-08-03 05:31:35,373 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: aes256-ctr is not available. 2017-08-03 05:31:35,374 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: aes192-ctr is not available. 2017-08-03 05:31:35,374 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: aes256-cbc is not available. 2017-08-03 05:31:35,374 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: aes192-cbc is not available. 2017-08-03 05:31:35,374 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: arcfour256 is not available. 2017-08-03 05:31:35,376 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: SSH_MSG_KEXINIT sent 2017-08-03 05:31:35,376 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: SSH_MSG_KEXINIT received 2017-08-03 05:31:35,377 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: kex: server->client aes128-ctr hmac-md5 none 2017-08-03 05:31:35,377 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: kex: client->server aes128-ctr hmac-md5 none 2017-08-03 05:31:35,430 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: SSH_MSG_KEXDH_INIT sent 2017-08-03 05:31:35,431 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: expecting SSH_MSG_KEXDH_REPLY 2017-08-03 05:31:35,447 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: ssh_rsa_verify: signature true 2017-08-03 05:31:35,450 WARN org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Permanently added 'hadoop171' (RSA) to the list of known hosts. 2017-08-03 05:31:35,451 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: SSH_MSG_NEWKEYS sent 2017-08-03 05:31:35,451 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: SSH_MSG_NEWKEYS received 2017-08-03 05:31:35,456 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: SSH_MSG_SERVICE_REQUEST sent 2017-08-03 05:31:35,457 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: SSH_MSG_SERVICE_ACCEPT received 2017-08-03 05:31:35,459 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Authentications that can continue: gssapi-with-mic,publickey,keyboard-interactive,password 2017-08-03 05:31:35,460 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Next authentication method: gssapi-with-mic 2017-08-03 05:31:35,468 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Authentications that can continue: publickey,keyboard-interactive,password 2017-08-03 05:31:35,468 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Next authentication method: publickey 2017-08-03 05:31:35,628 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Authentication succeeded (publickey). 2017-08-03 05:31:35,629 INFO org.apache.hadoop.ha.SshFenceByTcpPort: Connected to hadoop171 2017-08-03 05:31:35,629 INFO org.apache.hadoop.ha.SshFenceByTcpPort: Looking for process running on port 9000 2017-08-03 05:31:35,840 WARN org.apache.hadoop.ha.SshFenceByTcpPort: PATH=$PATH:/sbin:/usr/sbin fuser -v -k -n tcp 9000 via ssh: bash: fuser: 未找到命令 2017-08-03 05:31:35,844 INFO org.apache.hadoop.ha.SshFenceByTcpPort: rc: 127 2017-08-03 05:31:35,844 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Disconnecting from hadoop171 port 22 2017-08-03 05:31:35,847 WARN org.apache.hadoop.ha.NodeFencer: Fencing method org.apache.hadoop.ha.SshFenceByTcpPort(null) was unsuccessful. 2017-08-03 05:31:35,847 ERROR org.apache.hadoop.ha.NodeFencer: Unable to fence service by any configured method. 2017-08-03 05:31:35,847 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Caught an exception, leaving main loop due to Socket closed 2017-08-03 05:31:35,905 WARN org.apache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of election java.lang.RuntimeException: Unable to fence NameNode at hadoop171/172.16.31.171:9000 at org.apache.hadoop.ha.ZKFailoverController.doFence(ZKFailoverController.java:530) at org.apache.hadoop.ha.ZKFailoverController.fenceOldActive(ZKFailoverController.java:502) at org.apache.hadoop.ha.ZKFailoverController.access$1100(ZKFailoverController.java:60) at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.fenceOldActive(ZKFailoverController.java:888) at org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:909) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:808) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:417) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:599) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) 2017-08-03 05:31:35,906 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2017-08-03 05:31:35,967 INFO org.apache.zookeeper.ZooKeeper: Session: 0x35d9be43dc1019c closed 2017-08-03 05:31:36,968 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop171:2181,hadoop172:2181,hadoop173:2181 sessionTimeout=5000 watcher[email protected]6562a9e9 2017-08-03 05:31:36,973 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop173/172.16.31.173:2181. Will not attempt to authenticate using SASL (unknown error) 2017-08-03 05:31:37,731 INFO org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /172.16.31.172:52192, server: hadoop173/172.16.31.173:2181 2017-08-03 05:31:37,952 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop173/172.16.31.173:2181, sessionid = 0x35d9be43dc1021b, negotiated timeout = 5000 2017-08-03 05:31:37,955 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2017-08-03 05:31:37,956 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2017-08-03 05:31:38,047 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2017-08-03 05:31:38,054 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a0362656812036e6e311a096861646f6f7031373120a84628d33e 2017-08-03 05:31:38,056 INFO org.apache.hadoop.ha.ZKFailoverController: Should fence: NameNode at hadoop171/172.16.31.171:9000 2017-08-03 05:31:39,061 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop171/172.16.31.171:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS) 2017-08-03 05:31:39,064 WARN org.apache.hadoop.ha.FailoverController: Unable to gracefully make NameNode at hadoop171/172.16.31.171:9000 standby (unable to connect)

phoenix-hbase 服務頻繁掛掉問題排查

phoenix-hbase 服務頻繁掛掉問題排查

問題定位分享（1）HBase RegionServer頻繁掛掉

Hbase regionserver 逐個掛掉的問題分析

Java進程突然掛掉排查分析手記（jvm bug）

RK3399使用乙太網pppoe撥號導致系統服務全部掛掉。

記一次ss程序莫名其妙掛掉排查

簡單利用Apache Logs Viewer工具分析錯誤日誌和頻繁掛掉原因

使用start-hbase.sh 啟動服務然後HMaster 和 HRegionServer 相繼掛掉

Redis服務掛掉後,重啟時閃退

JIRA應用的記憶體引數設定不當+容器沒有對資源進行限制導致服務掛掉的例子

為什麼spring cloud服務啟動之後回到命令列會自動掛掉

如何實現Spring Cloud微服務掛掉之後自動重啟

hbase GC時間過程導致程序掛掉問題

解決服務不斷重啟掛掉問題

JVM服務進程掛掉問題定位查詢思路

【Azure微服務 Service Fabric 】因證書過期導致Service Fabric叢集掛掉（升級無法完成，節點不可用）

【Azure Cloud Services】雲服務頻繁發生伺服器崩潰的排查方案

phxsql問題 - 17000掛掉了

MongoDB的mongos實例因無法分配mlock內存掛掉

12.11-xshell遠程連接服務器以及故障排查

phoenix-hbase 服務頻繁掛掉問題排查

相關推薦