1. 程式人生 > >hadoop-2.5.0-cdh5.3.0 HA線上升級

hadoop-2.5.0-cdh5.3.0 HA線上升級



本文升級到hadoop2.6
1、hadoop升級前準備,namenode元資料和配置備份等資訊參考上篇文章:http://blog.csdn.net/linux_ja/article/details/51908466


2、備份資訊:
hdfs dfsadmin -rollingUpgrade prepare
hdfs dfsadmin -rollingUpgrade query


3、進入standby nn節點執行如下四個命令:
sudo rm -rf /opt/hadoop
sudo ln -s /home/hadoop/platform/hadoop-2.6.0-cdh5.4.0 /opt/hadoop
hadoop-daemon.sh stop namenode
nohup hdfs namenode -rollingUpgrade started &


注意,目前hdfs namenode -rollingUpgrade started這個命令先以後臺執行,過一段時間以正常命令啟動nn,
4、進入active nn節點重複執行步驟2的命令,這樣就完成nn的線上升級


5、在nn節點上以此執行所有要升級的datanode,我的四個節點(hdfs.beta1,hdfs.beta2,hdfs.beta3,hdfs.beta4)
 a、hdfs dfsadmin -getDatanodeInfo hdfs.beta1:50020
 b、hdfs dfsadmin -shutdownDatanode hdfs.beta1:50020 upgrade
  c、到所要升級的datanode上執行,hadoop-daemon.sh start datanode
以此重複執行abc步驟,這就完成dn的線上升級步驟


6、如果允許一段時間沒問題,可以執行:
hdfs dfsadmin -rollingUpgrade finalize,執行後就不能回滾


7、test:
hadoop jar /opt/hadoop/share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.4.0.jar randomwriter /rand 
hadoop jar /opt/hadoop/share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.4.0.jar sort rand /sort-rand 
hadoop jar /opt/hadoop/share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.4.0.jar wordcount  /wordcount/a.txt /wordcount/temp


8、升級之後問題:



org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: 
got premature end-of-file at txid 445259; expected file to go up to 4243581

nn和standby nn的editlog不一致

執行:hdfs namenode -bootstrapStandby