Hadoop 2.3.0-cdh5.0.2升級Hadoop 2.5.0-cdh5.3.1
阿新 • • 發佈:2019-01-04
一、準備升級包
for line in `cat /home/hadoop/platform/hadoop.list|awk '{print $1}'`; do echo $line;ssh $line "/bin/mkdir /home/hadoop/platform";done
for line in `cat /home/hadoop/platform/hadoop.list|awk '{print $1}'`; do echo $line;scp /home/hadoop/platform/hadoop-2.5.0-cdh5.3.3.tar.gz [email protected]$line:/home/hadoop/platform;done
for line in `cat /home/hadoop/platform/hadoop.list|awk '{print $1}'`; do echo $line;ssh $line "/bin/tar -zvxf /home/hadoop/platform/hadoop-2.5.0-cdh5.3.3.tar.gz -C /home/hadoop/platform/";done
二、做好檔案備份資訊:
a、備份namenode資訊,即etc/hadoop/hdfs-site.xml下的dfs.namenode.name.dir所指向的資料夾
cp -r /data/tmp/dfs/name .
b、備份config檔案:hadoop-env.sh,yarn-env.sh,core-site.xml,mapred-site.xml,hdfs-site.xml,yarn-site.xml
cp -r $HADOOP_CONF_DIR .
c、檔案資訊 hadoop fsck / -files -blocks -locations > dfs-v-old-fsck-1.log hadoop dfs -lsr / > dfs-v-old-lsr-1.log hadoop dfsadmin -report > dfs-v-old-report-1.log 三、將原來的hadoop備份一下,防止升級失敗回滾 for line in `cat /home/hadoop/platform/hadoop.list|awk '{print $1}'`; do echo $line;ssh $line "/bin/cp -r /opt/hadoop /opt/hadoop2.3.0";done 四,刪除原來的軟連線,新建軟連線到新版本的hadoop 注意:刪除軟連線 /opt/hadoop而不是 /opt/hadoop/, for line in `cat /home/hadoop/platform/hadoop.list|awk '{print $1}'`; do echo $line;ssh $line "/bin/rm -rf /opt/hadoop";done for line in `cat /home/hadoop/platform/hadoop.list|awk '{print $1}'`; do echo $line;ssh $line "/bin/ln -s /data/platform/hadoop-2.5.0-cdh5.3.1 /opt/hadoop";done 五、建立一個FsImage
hdfs dfsadmin -rollingUpgrade prepare
hdfs dfsadmin -rollingUpgrade query檢查滾動Image的狀態。等一會接著重新執行這個命令直到“Proceed
with rolling upgrade”訊息出現。
六、執行升級命令
a、關閉standby nn1:
hadoop-daemon.sh stop namenode
b、hdfs namenode -rollingUpgrade started c、關閉active nn2:
hadoop-daemon.sh stop namenode
d、hdfs namenode -rollingUpgrade started e、啟動所yostart-all.sh
遇到如下異常:
File system image contains an old layout version -55.
An upgrade to version -59 is required.
Please restart NameNode with the "-rollingUpgrade started"
option if a rolling upgrade is already started;
or restart NameNode with the "-upgrade" option to start a new upgrade.
修改:
/data/platform/hadoopupgrade/name/current/VERSION
layoutVersion=-55
改成layoutVersion=-59
nohup hdfs namenode -rollingUpgrade started &
datanode啟動不了
最好執行回滾:
start-dfs.sh -rollback
參考資料:
http://blog.csdn.net/shubingzhuoxue/article/details/50085407
https://www.zybuluo.com/layor/note/161370
cp -r /data/tmp/dfs/name .
b、備份config檔案:hadoop-env.sh,yarn-env.sh,core-site.xml,mapred-site.xml,hdfs-site.xml,yarn-site.xml
cp -r $HADOOP_CONF_DIR .
c、檔案資訊 hadoop fsck / -files -blocks -locations > dfs-v-old-fsck-1.log hadoop dfs -lsr / > dfs-v-old-lsr-1.log hadoop dfsadmin -report > dfs-v-old-report-1.log 三、將原來的hadoop備份一下,防止升級失敗回滾 for line in `cat /home/hadoop/platform/hadoop.list|awk '{print $1}'`; do echo $line;ssh $line "/bin/cp -r /opt/hadoop /opt/hadoop2.3.0";done 四,刪除原來的軟連線,新建軟連線到新版本的hadoop 注意:刪除軟連線 /opt/hadoop而不是 /opt/hadoop/, for line in `cat /home/hadoop/platform/hadoop.list|awk '{print $1}'`; do echo $line;ssh $line "/bin/rm -rf /opt/hadoop";done for line in `cat /home/hadoop/platform/hadoop.list|awk '{print $1}'`; do echo $line;ssh $line "/bin/ln -s /data/platform/hadoop-2.5.0-cdh5.3.1 /opt/hadoop";done 五、建立一個FsImage
hadoop-daemon.sh stop namenode
b、hdfs namenode -rollingUpgrade started c、關閉active nn2:
hadoop-daemon.sh stop namenode
d、hdfs namenode -rollingUpgrade started e、啟動所yostart-all.sh
遇到如下異常:
https://www.zybuluo.com/layor/note/161370