HBASE-1.1.12叢集啟動之後Hmaster，HregionServer程序會很快死掉問題
一、檢視日誌資訊如下
2017-11-25 10:37:25,646 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/usr/local/hadoop-2.7.4/lib/native
2017-11-25 10:37:25,646 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2017-11-25 10:37:25,646 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2017-11-25 10:37:25,647 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
2017-11-25 10:37:25,647 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2017-11-25 10:37:25,647 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=3.10.0-514.el7.x86_64
2017-11-25 10:37:25,647 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=root
2017-11-25 10:37:25,647 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/root
2017-11-25 10:37:25,648 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/root
2017-11-25 10:37:25,650 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=master:2181,slave1:2181,slave2:2181 sessionTimeout=90000 watcher=master:600000x0, quorum=master:2181,slave1:2181,slave2:2181, baseZNode=/hbase
2017-11-25 10:37:25,859 INFO [main-SendThread(slave2:2181)] zookeeper.ClientCnxn: Opening socket connection to server slave2/172.16.169.12:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-25 10:37:25,927 INFO [main-SendThread(slave2:2181)] zookeeper.ClientCnxn: Socket connection established to slave2/172.16.169.12:2181, initiating session
2017-11-25 10:37:26,096 INFO [main-SendThread(slave2:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2017-11-25 10:37:26,240 WARN [main] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=master:2181,slave1:2181,slave2:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
2017-11-25 10:37:26,871 INFO [main-SendThread(master:2181)] zookeeper.ClientCnxn: Opening socket connection to server master/172.16.169.10:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-25 10:37:26,873 INFO [main-SendThread(master:2181)] zookeeper.ClientCnxn: Socket connection established to master/172.16.169.10:2181, initiating session
2017-11-25 10:37:26,905 INFO [main-SendThread(master:2181)] zookeeper.ClientCnxn: Session establishment complete on server master/172.16.169.10:2181, sessionid = 0x15ff10795050002, negotiated timeout = 40000
2017-11-25 10:37:27,708 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: starting
2017-11-25 10:37:27,731 INFO [RpcServer.listener,port=60000] ipc.RpcServer: RpcServer.listener,port=60000: starting
2017-11-25 10:37:29,361 INFO [main] mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2017-11-25 10:37:29,412 INFO [main] http.HttpRequestLog: Http request log for http.requests.master is not defined
2017-11-25 10:37:29,585 INFO [main] http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter)
2017-11-25 10:37:29,636 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context master
2017-11-25 10:37:29,638 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2017-11-25 10:37:29,639 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
2017-11-25 10:37:29,860 INFO [main] http.HttpServer: Jetty bound to port 60010
2017-11-25 10:37:29,860 INFO [main] mortbay.log: jetty-6.1.26
2017-11-25 10:37:34,181 INFO [main] mortbay.log: Started
2017-11-25 10:37:34,210 INFO [main] master.HMaster: hbase.rootdir=hdfs://master:9000/hbase, hbase.cluster.distributed=true
2017-11-25 10:37:34,375 INFO [main] master.HMaster: Adding backup master ZNode /hbase/backup-masters/master,60000,1511577430262
2017-11-25 10:37:34,997 INFO [master:60000.activeMasterManager] master.ActiveMasterManager: Deleting ZNode for /hbase/backup-masters/master,60000,1511577430262 from backup master directory
2017-11-25 10:37:35,154 INFO [master:60000.activeMasterManager] master.ActiveMasterManager: Registered Active Master=master,60000,1511577430262
2017-11-25 10:37:36,813 INFO [master/master/172.16.169.10:60000] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x6ffbec93 connecting to ZooKeeper ensemble=master:2181,slave1:2181,slave2:2181
2017-11-25 10:37:36,835 INFO [master/master/172.16.169.10:60000] zookeeper.ZooKeeper: Initiating client connection, connectString=master:2181,slave1:2181,slave2:2181 sessionTimeout=90000 watcher=hconnection-0x6ffbec930x0, quorum=master:2181,slave1:2181,slave2:2181,
baseZNode=/hbase
2017-11-25 10:37:36,932 INFO [master/master/172.16.169.10:60000-SendThread(master:2181)] zookeeper.ClientCnxn: Opening socket connection to server master/172.16.169.10:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-25 10:37:36,936 INFO [master/master/172.16.169.10:60000-SendThread(master:2181)] zookeeper.ClientCnxn: Socket connection established to master/172.16.169.10:2181, initiating session
2017-11-25 10:37:37,017 INFO [master/master/172.16.169.10:60000-SendThread(master:2181)] zookeeper.ClientCnxn: Session establishment complete on server master/172.16.169.10:2181, sessionid = 0x15ff10795050004, negotiated timeout = 40000
2017-11-25 10:37:37,431 INFO [master/master/172.16.169.10:60000] regionserver.HRegionServer: ClusterId : 54fafe3b-fc82-40f5-9791-47ef66133d4f
2017-11-25 10:37:43,571 INFO [master:60000.activeMasterManager] fs.HFileSystem: Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
2017-11-25 10:37:43,649 INFO [master:60000.activeMasterManager] coordination.SplitLogManagerCoordination: Found 0 orphan tasks and 0 rescan nodes
2017-11-25 10:37:43,932 INFO [master:60000.activeMasterManager] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x157f9d2f connecting to ZooKeeper ensemble=master:2181,slave1:2181,slave2:2181
2017-11-25 10:37:43,933 INFO [master:60000.activeMasterManager] zookeeper.ZooKeeper: Initiating client connection, connectString=master:2181,slave1:2181,slave2:2181 sessionTimeout=90000 watcher=hconnection-0x157f9d2f0x0, quorum=master:2181,slave1:2181,slave2:2181,
baseZNode=/hbase
2017-11-25 10:37:43,949 INFO [master:60000.activeMasterManager-SendThread(slave1:2181)] zookeeper.ClientCnxn: Opening socket connection to server slave1/172.16.169.11:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-25 10:37:43,956 INFO [master:60000.activeMasterManager-SendThread(slave1:2181)] zookeeper.ClientCnxn: Socket connection established to slave1/172.16.169.11:2181, initiating session
2017-11-25 10:37:43,964 INFO [master:60000.activeMasterManager-SendThread(slave1:2181)] zookeeper.ClientCnxn: Session establishment complete on server slave1/172.16.169.11:2181, sessionid = 0x25ff10795560003, negotiated timeout = 40000
2017-11-25 10:37:44,063 INFO [master:60000.activeMasterManager] balancer.StochasticLoadBalancer: loading config
2017-11-25 10:37:44,365 INFO [master:60000.activeMasterManager] master.HMaster: Server active/primary master=master,60000,1511577430262, sessionid=0x15ff10795050002, setting cluster-up flag (Was=true)
2017-11-25 10:37:44,609 INFO [master:60000.activeMasterManager] procedure.ZKProcedureUtil: Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort
2017-11-25 10:37:44,732 INFO [master:60000.activeMasterManager] procedure.ZKProcedureUtil: Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort
2017-11-25 10:37:45,007 INFO [master:60000.activeMasterManager] master.MasterCoprocessorHost: System coprocessor loading is enabled
2017-11-25 10:37:45,189 INFO [master:60000.activeMasterManager] procedure2.ProcedureExecutor: Starting procedure executor threads=5
2017-11-25 10:37:45,191 INFO [master:60000.activeMasterManager] wal.WALProcedureStore: Starting WAL Procedure Store lease recovery
2017-11-25 10:37:45,207 INFO [master:60000.activeMasterManager] util.FSHDFSUtils: Recovering lease on dfs file hdfs://master:9000/hbase/MasterProcWALs/state-00000000000000000004.log
2017-11-25 10:37:45,308 INFO [master:60000.activeMasterManager] util.FSHDFSUtils: recoverLease=false, attempt=0 on file=hdfs://master:9000/hbase/MasterProcWALs/state-00000000000000000004.log after 100ms
2017-11-25 10:37:49,317 INFO [master:60000.activeMasterManager] util.FSHDFSUtils: recoverLease=true, attempt=1 on file=hdfs://master:9000/hbase/MasterProcWALs/state-00000000000000000004.log after 4109ms
2017-11-25 10:37:49,414 WARN [master:60000.activeMasterManager] wal.WALProcedureStore: Unable to read tracker for hdfs://master:9000/hbase/MasterProcWALs/state-00000000000000000004.log - Invalid Trailer version. got 202 expected 1
2017-11-25 10:37:49,503 INFO [master:60000.activeMasterManager] wal.WALProcedureStore: Lease acquired for flushLogId: 5
2017-11-25 10:37:49,646 WARN [master:60000.activeMasterManager] wal.ProcedureWALFormatReader: nothing left to decode. exiting with missing EOF
2017-11-25 10:37:49,785 INFO [master:60000.activeMasterManager] zookeeper.RecoverableZooKeeper: Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=master:2181,slave1:2181,slave2:2181
2017-11-25 10:37:49,785 INFO [master:60000.activeMasterManager] zookeeper.ZooKeeper: Initiating client connection, connectString=master:2181,slave1:2181,slave2:2181 sessionTimeout=90000 watcher=replicationLogCleaner0x0, quorum=master:2181,slave1:2181,slave2:2181,
baseZNode=/hbase
2017-11-25 10:37:49,829 INFO [master:60000.activeMasterManager-SendThread(slave1:2181)] zookeeper.ClientCnxn: Opening socket connection to server slave1/172.16.169.11:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-25 10:37:49,841 INFO [master:60000.activeMasterManager-SendThread(slave1:2181)] zookeeper.ClientCnxn: Socket connection established to slave1/172.16.169.11:2181, initiating session
2017-11-25 10:37:49,864 INFO [master:60000.activeMasterManager-SendThread(slave1:2181)] zookeeper.ClientCnxn: Session establishment complete on server slave1/172.16.169.11:2181, sessionid = 0x25ff10795560004, negotiated timeout = 40000
2017-11-25 10:37:49,899 INFO [master:60000.activeMasterManager] master.ServerManager: Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2017-11-25 10:37:51,441 INFO [master:60000.activeMasterManager] master.ServerManager: Waiting for region servers count to settle; currently checked in 0, slept for 1542 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500
ms.
2017-11-25 10:37:52,378 INFO [PriorityRpcServer.handler=5,queue=1,port=60000] master.ServerManager: Registering server=slave1,16020,1511577422976
2017-11-25 10:37:52,399 INFO [PriorityRpcServer.handler=3,queue=1,port=60000] master.ServerManager: Registering server=slave2,16020,1511577423313
2017-11-25 10:37:52,399 INFO [PriorityRpcServer.handler=1,queue=1,port=60000] master.ServerManager: Registering server=master,16020,1511577430553
2017-11-25 10:37:52,430 INFO [master:60000.activeMasterManager] master.ServerManager: Waiting for region servers count to settle; currently checked in 3, slept for 2531 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500
ms.
2017-11-25 10:37:52,567 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Remove log: hdfs://master:9000/hbase/MasterProcWALs/state-00000000000000000004.log
2017-11-25 10:37:52,568 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Removed logs: [hdfs://master:9000/hbase/MasterProcWALs/state-00000000000000000005.log, hdfs://master:9000/hbase/MasterProcWALs/state-00000000000000000006.log]
2017-11-25 10:37:52,599 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Remove log: hdfs://master:9000/hbase/MasterProcWALs/state-00000000000000000005.log
2017-11-25 10:37:52,600 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Removed logs: [hdfs://master:9000/hbase/MasterProcWALs/state-00000000000000000006.log]
2017-11-25 10:37:53,933 INFO [master:60000.activeMasterManager] master.ServerManager: Waiting for region servers count to settle; currently checked in 3, slept for 4034 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500
ms.
2017-11-25 10:37:54,415 INFO [master:60000.activeMasterManager] master.ServerManager: Finished waiting for region servers count to settle; checked in 3, slept for 4516 ms, expecting minimum of 1, maximum of 2147483647, master is running
2017-11-25 10:37:54,470 INFO [master:60000.activeMasterManager] master.MasterFileSystem: Log folder hdfs://master:9000/hbase/WALs/master,16020,1511532189674 doesn't belong to a known region server, splitting
2017-11-25 10:37:54,528 INFO [master:60000.activeMasterManager] master.MasterFileSystem: Log folder hdfs://master:9000/hbase/WALs/master,16020,1511577430553 belongs to an existing region server
2017-11-25 10:37:54,581 INFO [master:60000.activeMasterManager] master.MasterFileSystem: Log folder hdfs://master:9000/hbase/WALs/slave1,16020,1511532206980 doesn't belong to a known region server, splitting
2017-11-25 10:37:54,618 INFO [master:60000.activeMasterManager] master.MasterFileSystem: Log folder hdfs://master:9000/hbase/WALs/slave2,16020,1511532219830 doesn't belong to a known region server, splitting
2017-11-25 10:37:54,805 INFO [master:60000.activeMasterManager] master.SplitLogManager: dead splitlog workers [slave2,16020,1511532219830]
2017-11-25 10:37:54,855 INFO [master:60000.activeMasterManager] master.SplitLogManager: started splitting 1 logs in [hdfs://master:9000/hbase/WALs/slave2,16020,1511532219830-splitting] for [slave2,16020,1511532219830]
2017-11-25 10:37:55,654 INFO [master,60000,1511577430262_splitLogManager__ChoreService_1] master.SplitLogManager: total tasks = 1 unassigned = 1 tasks={/hbase/splitWAL/WALs%2Fslave2%2C16020%2C1511532219830-splitting%2Fslave2%252C16020%252C1511532219830..meta.1511532416907.meta=last_update
= -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0}
2017-11-25 10:38:00,482 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/WALs%2Fslave2%2C16020%2C1511532219830-splitting%2Fslave2%252C16020%252C1511532219830..meta.1511532416907.meta acquired by master,16020,1511577430553
2017-11-25 10:38:00,658 INFO [master,60000,1511577430262_splitLogManager__ChoreService_1] master.SplitLogManager: total tasks = 1 unassigned = 0 tasks={/hbase/splitWAL/WALs%2Fslave2%2C16020%2C1511532219830-splitting%2Fslave2%252C16020%252C1511532219830..meta.1511532416907.meta=last_update
= 1511577480483 last_version = 1 cur_worker_name = master,16020,1511577430553 status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0}
2017-11-25 10:38:06,250 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/WALs%2Fslave2%2C16020%2C1511532219830-splitting%2Fslave2%252C16020%252C1511532219830..meta.1511532416907.meta entered state: DONE master,16020,1511577430553
2017-11-25 10:38:06,464 INFO [main-EventThread] wal.WALSplitter: Archived processed log hdfs://master:9000/hbase/WALs/slave2,16020,1511532219830-splitting/slave2%2C16020%2C1511532219830..meta.1511532416907.meta to hdfs://master:9000/hbase/oldWALs/slave2%2C16020%2C1511532219830..meta.1511532416907.meta
2017-11-25 10:38:06,471 INFO [main-EventThread] coordination.SplitLogManagerCoordination: Done splitting /hbase/splitWAL/WALs%2Fslave2%2C16020%2C1511532219830-splitting%2Fslave2%252C16020%252C1511532219830..meta.1511532416907.meta
2017-11-25 10:38:06,545 WARN [master:60000.activeMasterManager] master.SplitLogManager: returning success without actually splitting and deleting all the log files in path hdfs://master:9000/hbase/WALs/slave2,16020,1511532219830-splitting
2017-11-25 10:38:06,545 INFO [master:60000.activeMasterManager] master.SplitLogManager: finished splitting (more than or equal to) 83 bytes in 1 log files in [hdfs://master:9000/hbase/WALs/slave2,16020,1511532219830-splitting] in 11690ms
2017-11-25 10:38:06,929 INFO [master:60000.activeMasterManager] zookeeper.MetaTableLocator: Failed verification of hbase:meta,,1 at address=slave2,16020,1511532219830, exception=org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is not
online on slave2,16020,1511577423313
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2915)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:979)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionInfo(RSRpcServices.java:1259)
at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22233)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2145)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:748)
2017-11-25 10:38:06,941 INFO [master:60000.activeMasterManager] master.SplitLogManager: dead splitlog workers [slave2,16020,1511532219830]
2017-11-25 10:38:06,951 INFO [master:60000.activeMasterManager] master.SplitLogManager: hdfs://master:9000/hbase/WALs/slave2,16020,1511532219830-splitting is empty dir, no logs to split
2017-11-25 10:38:06,951 INFO [master:60000.activeMasterManager] master.SplitLogManager: started splitting 0 logs in [hdfs://master:9000/hbase/WALs/slave2,16020,1511532219830-splitting] for [slave2,16020,1511532219830]
2017-11-25 10:38:06,977 WARN [master:60000.activeMasterManager] master.SplitLogManager: returning success without actually splitting and deleting all the log files in path hdfs://master:9000/hbase/WALs/slave2,16020,1511532219830-splitting
2017-11-25 10:38:06,977 INFO [master:60000.activeMasterManager] master.SplitLogManager: finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://master:9000/hbase/WALs/slave2,16020,1511532219830-splitting] in 26ms
2017-11-25 10:38:06,978 INFO [master:60000.activeMasterManager] zookeeper.MetaTableLocator: Deleting hbase:meta region location in ZooKeeper
2017-11-25 10:38:07,292 INFO [master:60000.activeMasterManager] master.AssignmentManager: Setting node as OFFLINED in ZooKeeper for region {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}
2017-11-25 10:38:07,385 INFO [master:60000.activeMasterManager] master.AssignmentManager: Assigning hbase:meta,,1.1588230740 to slave2,16020,1511577423313
2017-11-25 10:38:07,385 INFO [master:60000.activeMasterManager] master.RegionStates: Transition {1588230740 state=OFFLINE, ts=1511577487292, server=null} to {1588230740 state=PENDING_OPEN, ts=1511577487385, server=slave2,16020,1511577423313}
2017-11-25 10:38:07,595 INFO [master:60000.activeMasterManager] master.ServerManager: AssignmentManager hasn't finished failover cleanup; waiting
2017-11-25 10:38:07,837 INFO [AM.ZK.Worker-pool2-t1] master.RegionStates: Transition {1588230740 state=PENDING_OPEN, ts=1511577487385, server=slave2,16020,1511577423313} to {1588230740 state=OPENING, ts=1511577487837, server=slave2,16020,1511577423313}
2017-11-25 10:38:12,722 INFO [AM.ZK.Worker-pool2-t2] master.RegionStates: Transition {1588230740 state=OPENING, ts=1511577487837, server=slave2,16020,1511577423313} to {1588230740 state=OPEN, ts=1511577492722, server=slave2,16020,1511577423313}
2017-11-25 10:38:12,729 INFO [AM.ZK.Worker-pool2-t2] coordination.ZkOpenRegionCoordination: Handling OPENED of 1588230740 from master,60000,1511577430262; deleting unassigned node
2017-11-25 10:38:12,882 INFO [master:60000.activeMasterManager] master.HMaster: hbase:meta with replicaId 0 assigned=1, rit=false, location=slave2,16020,1511577423313
2017-11-25 10:38:13,691 INFO [master:60000.activeMasterManager] hbase.MetaMigrationConvertingToPB: META already up-to date with PB serialization
2017-11-25 10:38:13,878 INFO [master:60000.activeMasterManager] master.RegionStates: Transition {e6bff28294ecdbdc5fadca964ceafb26 state=OPEN, ts=1511577493878, server=slave1,16020,1511532206980} to {e6bff28294ecdbdc5fadca964ceafb26 state=OFFLINE, ts=1511577493878,
server=slave1,16020,1511532206980}
2017-11-25 10:38:13,899 INFO [master:60000.activeMasterManager] master.RegionStates: Transition {1d3b7b34795a72f8fb15b68d64c56e53 state=OPEN, ts=1511577493899, server=master,16020,1511532189674} to {1d3b7b34795a72f8fb15b68d64c56e53 state=OFFLINE, ts=1511577493899,
server=master,16020,1511532189674}
2017-11-25 10:38:13,994 INFO [master:60000.activeMasterManager] master.AssignmentManager: Found regions out on cluster or in RIT; presuming failover
2017-11-25 10:38:14,105 INFO [master:60000.activeMasterManager] master.AssignmentManager: Joined the cluster in 414ms, failover=true
2017-11-25 10:38:14,334 INFO [MASTER_SERVER_OPERATIONS-master:60000-1] handler.ServerShutdownHandler: Splitting logs for slave1,16020,1511532206980 before assignment; region count=1
2017-11-25 10:38:14,367 INFO [MASTER_SERVER_OPERATIONS-master:60000-0] handler.ServerShutdownHandler: Splitting logs for slave2,16020,1511532219830 before assignment; region count=0
2017-11-25 10:38:14,424 INFO [MASTER_SERVER_OPERATIONS-master:60000-2] handler.ServerShutdownHandler: Splitting logs for master,16020,1511532189674 before assignment; region count=1
2017-11-25 10:38:14,483 INFO [MASTER_SERVER_OPERATIONS-master:60000-1] master.SplitLogManager: dead splitlog workers [slave1,16020,1511532206980]
2017-11-25 10:38:14,506 INFO [MASTER_SERVER_OPERATIONS-master:60000-1] master.SplitLogManager: started splitting 1 logs in [hdfs://master:9000/hbase/WALs/slave1,16020,1511532206980-splitting] for [slave1,16020,1511532206980]
2017-11-25 10:38:14,559 INFO [MASTER_SERVER_OPERATIONS-master:60000-0] master.SplitLogManager: dead splitlog workers [slave2,16020,1511532219830]
2017-11-25 10:38:14,617 INFO [MASTER_SERVER_OPERATIONS-master:60000-0] master.SplitLogManager: started splitting 1 logs in [hdfs://master:9000/hbase/WALs/slave2,16020,1511532219830-splitting] for [slave2,16020,1511532219830]
2017-11-25 10:38:14,655 INFO [master,60000,1511577430262_splitLogManager__ChoreService_1] master.SplitLogManager: total tasks = 2 unassigned = 2 tasks={/hbase/splitWAL/WALs%2Fslave2%2C16020%2C1511532219830-splitting%2Fslave2%252C16020%252C1511532219830.default.1511532372089=last_update
= -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0, /hbase/splitWAL/WALs%2Fslave1%2C16020%2C1511532206980-splitting%2Fslave1%252C16020%252C1511532206980.default.1511532378995=last_update
= 1511577494615 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0}
2017-11-25 10:38:14,693 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/WALs%2Fslave1%2C16020%2C1511532206980-splitting%2Fslave1%252C16020%252C1511532206980.default.1511532378995 acquired by slave2,16020,1511577423313
2017-11-25 10:38:14,818 INFO [MASTER_SERVER_OPERATIONS-master:60000-2] master.SplitLogManager: dead splitlog workers [master,16020,1511532189674]
2017-11-25 10:38:14,907 INFO [MASTER_SERVER_OPERATIONS-master:60000-2] master.SplitLogManager: started splitting 1 logs in [hdfs://master:9000/hbase/WALs/master,16020,1511532189674-splitting] for [master,16020,1511532189674]
2017-11-25 10:38:15,171 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/WALs%2Fslave2%2C16020%2C1511532219830-splitting%2Fslave2%252C16020%252C1511532219830.default.1511532372089 acquired by master,16020,1511577430553
2017-11-25 10:38:15,337 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/WALs%2Fmaster%2C16020%2C1511532189674-splitting%2Fmaster%252C16020%252C1511532189674.default.1511532381539 acquired by slave1,16020,1511577422976
2017-11-25 10:38:20,646 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/WALs%2Fslave1%2C16020%2C1511532206980-splitting%2Fslave1%252C16020%252C1511532206980.default.1511532378995 entered state: DONE slave2,16020,1511577423313
2017-11-25 10:38:20,654 INFO [master,60000,1511577430262_splitLogManager__ChoreService_1] master.SplitLogManager: total tasks = 3 unassigned = 0 tasks={/hbase/splitWAL/WALs%2Fmaster%2C16020%2C1511532189674-splitting%2Fmaster%252C16020%252C1511532189674.default.1511532381539=last_update
= 1511577496132 last_version = 2 cur_worker_name = slave1,16020,1511577422976 status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0, /hbase/splitWAL/WALs%2Fslave2%2C16020%2C1511532219830-splitting%2Fslave2%252C16020%252C1511532219830.default.1511532372089=last_update
= 1511577495619 last_version = 2 cur_worker_name = master,16020,1511577430553 status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0, /hbase/splitWAL/WALs%2Fslave1%2C16020%2C1511532206980-splitting%2Fslave1%252C16020%252C1511532206980.default.1511532378995=last_update
= 1511577500561 last_version = 2 cur_worker_name = slave2,16020,1511577423313 status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0}
2017-11-25 10:38:20,905 INFO [main-EventThread] wal.WALSplitter: Archived processed log hdfs://master:9000/hbase/WALs/slave1,16020,1511532206980-splitting/slave1%2C16020%2C1511532206980.default.1511532378995 to hdfs://master:9000/hbase/oldWALs/slave1%2C16020%2C1511532206980.default.1511532378995
2017-11-25 10:38:20,909 INFO [main-EventThread] coordination.SplitLogManagerCoordination: Done splitting /hbase/splitWAL/WALs%2Fslave1%2C16020%2C1511532206980-splitting%2Fslave1%252C16020%252C1511532206980.default.1511532378995
2017-11-25 10:38:20,920 INFO [MASTER_SERVER_OPERATIONS-master:60000-1] master.SplitLogManager: finished splitting (more than or equal to) 83 bytes in 1 log files in [hdfs://master:9000/hbase/WALs/slave1,16020,1511532206980-splitting] in 6414ms
2017-11-25 10:38:20,927 INFO [MASTER_SERVER_OPERATIONS-master:60000-1] handler.ServerShutdownHandler: Reassigning 1 region(s) that slave1,16020,1511532206980 was carrying (and 0 regions(s) that were opening on this server)
2017-11-25 10:38:20,928 INFO [MASTER_SERVER_OPERATIONS-master:60000-1] handler.ServerShutdownHandler: Best effort in SSH to retain assignment of 1 regions from the dead server slave1,16020,1511532206980
2017-11-25 10:38:21,241 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/WALs%2Fslave2%2C16020%2C1511532219830-splitting%2Fslave2%252C16020%252C1511532219830.default.1511532372089 entered state: DONE master,16020,1511577430553
2017-11-25 10:38:21,262 INFO [MASTER_SERVER_OPERATIONS-master:60000-1] balancer.BaseLoadBalancer: Reassigned 1 regions. 1 retained the pre-restart assignment.
2017-11-25 10:38:21,263 INFO [MASTER_SERVER_OPERATIONS-master:60000-1] master.AssignmentManager: Bulk assigning 1 region(s) across 3 server(s), retainAssignment=true
2017-11-25 10:38:21,332 INFO [main-EventThread] wal.WALSplitter: Archived processed log hdfs://master:9000/hbase/WALs/slave2,16020,1511532219830-splitting/slave2%2C16020%2C1511532219830.default.1511532372089 to hdfs://master:9000/hbase/oldWALs/slave2%2C16020%2C1511532219830.default.1511532372089
2017-11-25 10:38:21,342 INFO [main-EventThread] coordination.SplitLogManagerCoordination: Done splitting /hbase/splitWAL/WALs%2Fslave2%2C16020%2C1511532219830-splitting%2Fslave2%252C16020%252C1511532219830.default.1511532372089
2017-11-25 10:38:21,361 INFO [MASTER_SERVER_OPERATIONS-master:60000-0] master.SplitLogManager: finished splitting (more than or equal to) 83 bytes in 1 log files in [hdfs://master:9000/hbase/WALs/slave2,16020,1511532219830-splitting] in 6744ms
2017-11-25 10:38:21,362 INFO [MASTER_SERVER_OPERATIONS-master:60000-0] handler.ServerShutdownHandler: Reassigning 0 region(s) that slave2,16020,1511532219830 was carrying (and 0 regions(s) that were opening on this server)
2017-11-25 10:38:21,362 INFO [MASTER_SERVER_OPERATIONS-master:60000-0] handler.ServerShutdownHandler: Best effort in SSH to retain assignment of 0 regions from the dead server slave2,16020,1511532219830
2017-11-25 10:38:21,363 INFO [MASTER_SERVER_OPERATIONS-master:60000-0] handler.ServerShutdownHandler: Finished processing of shutdown of slave2,16020,1511532219830
2017-11-25 10:38:21,424 INFO [master,60000,1511577430262-GeneralBulkAssigner-1] master.AssignmentManager: Assigning 1 region(s) to slave1,16020,1511577422976
2017-11-25 10:38:21,631 INFO [master,60000,1511577430262-GeneralBulkAssigner-1] master.RegionStates: Transition {e6bff28294ecdbdc5fadca964ceafb26 state=OFFLINE, ts=1511577501524, server=slave1,16020,1511532206980} to {e6bff28294ecdbdc5fadca964ceafb26 state=PENDING_OPEN,
ts=1511577501631, server=slave1,16020,1511577422976}
2017-11-25 10:38:22,777 INFO [MASTER_SERVER_OPERATIONS-master:60000-1] master.AssignmentManager: Bulk assigning done
2017-11-25 10:38:22,777 INFO [MASTER_SERVER_OPERATIONS-master:60000-1] handler.ServerShutdownHandler: Finished processing of shutdown of slave1,16020,1511532206980
2017-11-25 10:38:23,054 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/WALs%2Fmaster%2C16020%2C1511532189674-splitting%2Fmaster%252C16020%252C1511532189674.default.1511532381539 entered state: DONE slave1,16020,1511577422976
2017-11-25 10:38:23,068 INFO [AM.ZK.Worker-pool2-t5] master.RegionStates: Transition {e6bff28294ecdbdc5fadca964ceafb26 state=PENDING_OPEN, ts=1511577501631, server=slave1,16020,1511577422976} to {e6bff28294ecdbdc5fadca964ceafb26 state=OPENING, ts=1511577503068,
server=slave1,16020,1511577422976}
2017-11-25 10:38:23,137 INFO [main-EventThread] wal.WALSplitter: Archived processed log hdfs://master:9000/hbase/WALs/master,16020,1511532189674-splitting/master%2C16020%2C1511532189674.default.1511532381539 to hdfs://master:9000/hbase/oldWALs/master%2C16020%2C1511532189674.default.1511532381539
2017-11-25 10:38:23,142 INFO [main-EventThread] coordination.SplitLogManagerCoordination: Done splitting /hbase/splitWAL/WALs%2Fmaster%2C16020%2C1511532189674-splitting%2Fmaster%252C16020%252C1511532189674.default.1511532381539
2017-11-25 10:38:23,177 INFO [MASTER_SERVER_OPERATIONS-master:60000-2] master.SplitLogManager: finished splitting (more than or equal to) 83 bytes in 1 log files in [hdfs://master:9000/hbase/WALs/master,16020,1511532189674-splitting] in 8270ms
2017-11-25 10:38:23,180 INFO [MASTER_SERVER_OPERATIONS-master:60000-2] handler.ServerShutdownHandler: Reassigning 1 region(s) that master,16020,1511532189674 was carrying (and 0 regions(s) that were opening on this server)
2017-11-25 10:38:23,181 INFO [MASTER_SERVER_OPERATIONS-master:60000-2] handler.ServerShutdownHandler: Best effort in SSH to retain assignment of 1 regions from the dead server master,16020,1511532189674
2017-11-25 10:38:23,245 INFO [MASTER_SERVER_OPERATIONS-master:60000-2] balancer.BaseLoadBalancer: Reassigned 1 regions. 1 retained the pre-restart assignment.
2017-11-25 10:38:23,258 INFO [MASTER_SERVER_OPERATIONS-master:60000-2] master.AssignmentManager: Bulk assigning 1 region(s) across 3 server(s), retainAssignment=true
2017-11-25 10:38:23,284 INFO [master,60000,1511577430262-GeneralBulkAssigner-0] master.AssignmentManager: Assigning 1 region(s) to master,16020,1511577430553
2017-11-25 10:38:23,395 INFO [master,60000,1511577430262-GeneralBulkAssigner-0] master.RegionStates: Transition {1d3b7b34795a72f8fb15b68d64c56e53 state=OFFLINE, ts=1511577503339, server=master,16020,1511532189674} to {1d3b7b34795a72f8fb15b68d64c56e53 state=PENDING_OPEN,
ts=1511577503395, server=master,16020,1511577430553}
2017-11-25 10:38:24,489 INFO [AM.ZK.Worker-pool2-t7] master.RegionStates: Transition {1d3b7b34795a72f8fb15b68d64c56e53 state=PENDING_OPEN, ts=1511577503395, server=master,16020,1511577430553} to {1d3b7b34795a72f8fb15b68d64c56e53 state=OPENING, ts=1511577504489,
server=master,16020,1511577430553}
2017-11-25 10:38:29,242 INFO [AM.ZK.Worker-pool2-t8] master.RegionStates: Transition {e6bff28294ecdbdc5fadca964ceafb26 state=OPENING, ts=1511577503068, server=slave1,16020,1511577422976} to {e6bff28294ecdbdc5fadca964ceafb26 state=OPEN, ts=1511577509242, server=slave1,16020,1511577422976}
2017-11-25 10:38:29,372 INFO [AM.ZK.Worker-pool2-t10] master.RegionStates: Offlined e6bff28294ecdbdc5fadca964ceafb26 from slave1,16020,1511532206980
2017-11-25 10:38:34,136 INFO [master:60000.activeMasterManager] master.HMaster: Master has completed initialization
2017-11-25 10:38:34,219 INFO [master:60000.activeMasterManager] quotas.MasterQuotaManager: Quota support disabled
2017-11-25 10:38:34,264 INFO [master:60000.activeMasterManager] zookeeper.ZooKeeperWatcher: not a secure deployment, proceeding
2017-11-25 10:38:34,891 INFO [AM.ZK.Worker-pool2-t11] master.RegionStates: Transition {1d3b7b34795a72f8fb15b68d64c56e53 state=OPENING, ts=1511577504489, server=master,16020,1511577430553} to {1d3b7b34795a72f8fb15b68d64c56e53 state=OPEN, ts=1511577514891, server=master,16020,1511577430553}
2017-11-25 10:38:34,915 INFO [AM.ZK.Worker-pool2-t13] master.RegionStates: Offlined 1d3b7b34795a72f8fb15b68d64c56e53 from master,16020,1511532189674
2017-11-25 10:38:34,917 INFO [MASTER_SERVER_OPERATIONS-master:60000-2] master.AssignmentManager: Bulk assigning done
2017-11-25 10:38:34,917 INFO [MASTER_SERVER_OPERATIONS-master:60000-2] handler.ServerShutdownHandler: Finished processing of shutdown of master,16020,1511532189674
二、在網上查看了很多的介紹，感覺與我的情況不太相符
於是我查看了叢集中的程序資訊發現zookeeper叢集的的程序沒有了，重啟zookeeper叢集，在啟動HBASE就行
使用netstat -nltp 命令檢視埠資訊
60010埠已經正常了，問題解決。
相關推薦
HBASE-1.1.12叢集啟動之後Hmaster，HregionServer程序會很快死掉問題
一、檢視日誌資訊如下 2017-11-25 10:37:25,646 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/usr/local/hadoop-2.7.4/lib/n
hadoop叢集啟動之後，datanode程序未啟動解決辦法
這種情況一般是由於自己進行hadoop格式化的時候沒有事先結束所有程序，或者在開啟一次hadoop叢集，發現配置不對，這時重新 hadoop namenode -format 就會出問題 因為此時你format之後， datanode的clusterID 和 namenode的clu
啟動hbase時，Hmaster、HRegionServer程序有時候起不來
它出現的原因可能有是以下兩點： 第一，可能是各個主機的時間不同步 【1】在各個主機上執行：date 命令，看看時間是否同步 【2】如果時間不同步的話 格式如下： # ntpdate -u&nbs
第1天 | 12天搞定Python，告訴你有什麼用？
掌握多一門程式語言，多一種選擇，多一份機遇，更何況學的是人見人愛，花見花開的Python語言。它目前可佔據程式語言排行榜的第3名，是名副其實的“探花郎”，無論用它做什麼(網路爬蟲、人工智慧、Web應用等) ，咱都不虧，開篇有益。 有趣的話，多說兩句，”無趣”的就總結概述。本章總結概述一下Python的發展歷史
Ubuntu和Win7雙系統，ubuntu被刪，重新啟動之後顯示，no such partition
div cmd命令 class ubuntu 終端 oot windows7 data- 重新啟動 準備一張windows7的系統安裝盤。從光盤啟動電腦，在光盤啟動完畢之後。按shift+F10，調出cmd命令終端，在終端輸入：bootrec/fixmbr OK。重新啟
叢集啟動使用Hadoop，執行後沒有NameNode節點
叢集啟動Hadoop後，執行後有SecondNameNode和dataNode,但是沒有NameNode節點？ 解決方法： 1、先執行stop-all.sh 2、執行hadoop namenode
父程序退出之後，子程序會發生什麼？
在linux中，所有程序都有一個共同的父程序systemd，如果父程序退出了，子程序還沒執行結束，子程序會被stsremd收養 下面用一個小程式來驗證一下： #include <cstdio> #include <sys/types.h> #inc
zookeepe啟動顯示成功，但是程序沒有（實際沒有啟動成功）
使用命令/zookeeper/bin/zkServer.sh start啟動zookeeper 顯示Starting zookeeper 。。。 STARTED 但是通過ps命令和telnet localhost 2181發現無服務啟動 通過echo $JAVA_HOME檢視，
kube-apiserver進程無法啟動了，k8s 1.7.12
k8s kubernetes kube-apiserver 服務 無法啟動 [root@k8s_m3:~]# journalctl -fu kube-apiserver-- Logs begin at Fri 2018-03-23 05:28:27 CST. --Mar 23 11:21:
Hadoop ha CDH5.15.1-hadoop叢集啟動後，兩個namenode都是standby模式
Hadoop ha CDH5.15.1-hadoop叢集啟動後，兩個namenode都是standby模式 作者：尹正傑 版權宣告：原創作品，謝絕轉載！否則將追究法律責任。 &nb
使用開源Breeze工具部署Kubernetes 1.12.1高可用叢集_Kubernetes中文社群
Breeze專案是深圳睿雲智合所開源的Kubernetes圖形化部署工具，大大簡化了Kubernetes部署的步驟，其最大亮點在於支援全離線環境的部署，且不需要翻牆獲取Google的相應資源包，尤其適合某些不便訪問網際網路的伺服器場景。（專案地址 https://github.com/wise
用kubeadm在centos7下搭建k8s 1.12叢集
在搭建的過程中遇到一個難題 本地將下載的映象包，匯入到本地並打好tag後，執行 kubeadm init命令，還會一直去拉取映象，不知道原因？ [[email protected] log]# kubeadm init I1113 15:31:40.17659
解決 win 8.1 + Ubuntu 12.04 安裝雙系統後啟動，無法進入ubuntu的問題
本著不折騰不舒服的原則，今天將一塊嶄新的SSD安裝了win8.1 + ubuntu 12.04的雙系統。之前win8+ubuntu的組合不知道裝了多少遍了，沒想到今天裝完win8.1 + ubuntu啟動後，卻不能進入grub介面，而是直接進入win8.1。 前置條件
MongoDB分片叢集啟動過程中Failed to connect to 127.0.0.1:27017錯誤解決辦法
（1）進行MongoDB分片叢集部署過後，進入MongoDB的目錄下，輸入bin/mongo -port:30000，居然彈出錯誤： MongoDB shell version: 3.0.7 connecting to: test 2015-10-30T10:34:27.2
安裝12.1.0.2 叢集GRID/GI, 執行root.sh 指令碼失敗的案例
在Linux系統上安裝12.1.0.2 叢集GRID/GI軟體，節點2執行root.sh失敗，螢幕的錯誤資訊： OLR initialization - successful 2015/12/15 13:16:55 CLSRSC-507: The root script c
Hadoop-2.6.0+Zookeeper-3.4.6+Spark-1.5.0+Hbase-1.1.2+Hive-1.2.0叢集搭建
前言 本部落格目的在於跟大家分享大資料平臺搭建過程，是筆者半年的結晶。在大資料搭建過程中,希望能給大家提過一些幫助，這也是本部落格的
搭建kafka2.12-1.0.0+storm1.1.1叢集
各應用程式及版本CentOS65jdk1.8.0_121storm1.1.1zookeeper3.4.10kafka2.12-1.0.0redis4.0.5storm、zookeeper、kafka可從apache.org網站獲取相應版本，建議版本與本文件一致；否則後續的示例
（13）HBase 1.2.6 叢集部署
1.引言 在前面的部落格中我們介紹HBase的偽分佈安裝，在本篇部落格中就主要介紹一下HBase的叢集部署，由於HBase是一款Hadoop之上的NoSQL資料庫，因此我們在安裝
cool-2018-10-22-centos7-hive-1.2叢集+整合hbase-1.1.3叢集
hive與hbase整合版本相容說明 HIVE 0.9.0，HBASE整合至少需要HBASE 0.92，HIVE的早期版本與HBase 0.89/0.90一起使用。 hive1.X將保持與HBase 0.98.x和更低版本相容。hive2x將與HBASE 1 x
2017.7.1 mysql安裝與啟動（已驗證可以使用）
界面 all avi 配置 install 服務 毫無 壓力 cnblogs 之前一直用解壓版安裝，啟動mysql服務的時候總是失敗，這次用mysql installer安裝一遍，終於成功啟動。 1.下載mysql installer 下載的32位的，在64位系統運行也