1. 程式人生 > >基於Gitlab+Jenkins的測試環境自動構建和生產多環境手動釋出方案

基於Gitlab+Jenkins的測試環境自動構建和生產多環境手動釋出方案

需求說明:

專案和生產環境越來越多,專案的測試釋出和線上釋出任務繁重
本方案使用Gitlab+Jenkins實現測試環境自動構建和生產多環境手動控制釋出

實驗主機列表和功能:

192.168.77.100 CentOS7 gitlab
192.168.77.130 CentOS7 jenkins+nginx
192.168.77.200 CentOS6 測試環境主機
192.168.77.211 CentOS6 生產環境主機1
192.168.77.212 CentOS6 生產環境主機2

Gitlab環境搭建:

依據《CentOS7實驗機模板搭建部署》克隆實驗機,IP:192.168.77.100,部署Gitlab並裝入測試用專案:

HOSTNAME=gitlab
hostnamectl set-hostname "$HOSTNAME"
echo "$HOSTNAME">/etc/hostname
echo "$(grep -E '127|::1' /etc/hosts)">/etc/hosts
echo "$(ip a|grep "inet "|grep -v 127|awk -F'[ /]' '{print $6}') $HOSTNAME">>/etc/hosts

cd /tmp
wget https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/gitlab-ce-10.8.6-ce.0.el7.x86_64.rpm
yum -y localinstall gitlab-ce-10.8.6-ce.0.el7.x86_64.rpm
# 目前生產使用版本為10.8.6,不使用更新版本
gitlab-ctl reconfigure systemctl is-enabled gitlab-runsvdir.service sed -i "s|http://gitlab.example.com|http://$(hostname -i)|g" /etc/gitlab/gitlab.rb sed -i "s/^.*backup_keep_time.*$/gitlab_rails['backup_keep_time'] = 604800/g" /etc/gitlab/gitlab.rb crontab -l>/tmp/crontab.tmp echo '# GitLab Backup Job'>>
/tmp/crontab.tmp echo '30 0 * * * /opt/gitlab/bin/gitlab-rake gitlab:backup:create'>>/tmp/crontab.tmp cat /tmp/crontab.tmp |crontab rm -rf /tmp/crontab.tmp cat >>/etc/gitlab/gitlab.rb<<EOF # mail alert setup gitlab_rails['smtp_enable'] = true gitlab_rails['smtp_address'] = 'smtp.126.com' gitlab_rails['smtp_port'] = 25 gitlab_rails['smtp_user_name'] = '[email protected]' gitlab_rails['smtp_password'] = 'xxxx' gitlab_rails['smtp_authentication'] = 'login' gitlab_rails['smtp_enable_starttls_auto']= true gitlab_rails['gitlab_email_from']= '[email protected]' gitlab_rails['gitlab_email_reply_to']= '[email protected]' EOF gitlab-ctl reconfigure gitlab-ctl restart

root賬號瀏覽器登陸,建立普通開發使用者vincent
匯入測試用的專案:http://vincent:[email protected]/vincent/shareprofit.git
該專案是一個maven專案,根目錄下的pom.xml可以構建出三個子專案的war包
該專案的master分支是受保護的,不能直接push到master分支,而需要提交merge請求合併更新到master分支

Jenkins環境搭建:

依據《CentOS7實驗機模板搭建部署》克隆實驗機,IP:192.168.77.130,使用使用者deploy部署Jenkins

HOSTNAME=jenkins
hostnamectl set-hostname "$HOSTNAME"
echo "$HOSTNAME">/etc/hostname
echo "$(grep -E '127|::1' /etc/hosts)">/etc/hosts
echo "$(ip a|grep "inet "|grep -v 127|awk -F'[ /]' '{print $6}') $HOSTNAME">>/etc/hosts

cd /usr/local/
tar -xf /tmp/jdk-8u172-linux-x64.tar.gz
echo 'export JAVA_HOME=/usr/local/jdk1.8.0_172'>>/etc/profile
echo 'export CLASSPATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib'>>/etc/profile
echo 'export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH'>>/etc/profile
source /etc/profile
java -version

yum -y install git

cd /usr/local/
unzip /tmp/apache-maven-3.5.2-bin.zip
echo 'export MAVEN_HOME=/usr/local/apache-maven-3.5.2'>>/etc/profile
echo 'export PATH=$PATH:$MAVEN_HOME/bin'>>/etc/profile
source /etc/profile
mvn --version

cd /tmp
wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key
yum -y install jenkins

useradd deploy
echo deploy|passwd --stdin deploy

sed -i 's|^\(JENKINS_JAVA_CMD=\).*$|\1"/usr/local/jdk1.8.0_172/bin/java"|g' /etc/sysconfig/jenkins
sed -i 's|^\(JENKINS_PORT=\).*$|\1"18080"|g' /etc/sysconfig/jenkins
sed -i 's|JENKINS_USER="jenkins"|JENKINS_USER="deploy"|g' /etc/sysconfig/jenkins
chown -R deploy: /var/log/jenkins
chown -R deploy: /var/lib/jenkins
chown -R deploy: /var/cache/jenkins
chkconfig jenkins on
/etc/init.d/jenkins start

參見《CentOS7 Jenkins部署 Maven專案構建測試》的網頁配置部分繼續網頁配置

進一步做Jenkins和Gitlab對接,使用deploy使用者做ssh key對接:

su - deploy
ssh-keygen -t rsa
ssh-copy-id 127.0.0.1
ssh 127.0.0.1 date
ssh -o StrictHostKeyChecking=no $(hostname) date
cat ~/.ssh/authorized_keys
# 記錄公鑰認證檔案內容,瀏覽器使用root使用者登陸gitlab,新增ssh key
# User Settings——>SSH Keys——>貼入id_rsa.pub內容——>Add key

# 簡單測試:
cd /tmp/
mkdir GitLabTest
cd GitLabTest
git clone [email protected]:root/shareprofit.git

Nginx環境搭建:

將nginx部署在jenkins主機192.168.77.130之上:

cat >/etc/yum.repos.d/nginx.repo<<EOF
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/\$basearch/
gpgcheck=0
enabled=1
EOF
yum -y install nginx
systemctl enable nginx
systemctl start nginx
chown -R deploy: /usr/share/nginx/html/

測試環境主機和生產環境主機1以及生產環境主機2搭建:

依據《CentOS6實驗機模板搭建部署》克隆部署三臺試驗機,IP:192.168.77.200、192.168.77.211和192.168.77.212
依據《CentOS6u8 java和tomcat多版本模板部署搭建配置》配置jdk8/tomcat8專案執行環境
建立執行三個tomcat專案:

yum -y install rsync
su - web_pro
cd /web/checkTOMCAT
bash pro_deploy.sh -n shareprofit-backport -j java_1.8 -t tomcat8
bash pro_deploy.sh -n shareprofit-merchantport -j java_1.8 -t tomcat8
bash pro_deploy.sh -n shareprofit-soaport -j java_1.8 -t tomcat8

Jenkins主機的deploy使用者需要直接遠端管控這三臺機器,因此使用192.168.77.130進行部署:

su - deploy
ssh-copy-id [email protected]
ssh-copy-id [email protected]
ssh-copy-id [email protected]
ssh -o StrictHostKeyChecking=no [email protected] date
ssh -o StrictHostKeyChecking=no [email protected] date
ssh -o StrictHostKeyChecking=no [email protected] date

測試環境自動構建功能的實現:

配置部署gitlab的webhook,當有新merge請求確定時,自動觸發jenkins的構建
並將構建出的各個子專案war包同步到測試環境之上,重啟對應的tomcat

  • Jenkins新增外掛:Gitlab Hook

  • 建立構建job,構建觸發器標籤,勾選Build when a change is pushed to GitLab
    勾選Accepted Merge Request Events,其他都不勾選
    高階選項卡,Secret token,Generate,生成token,儲存下來,同時儲存觸發該job的URL

  • 繼續配置原始碼、構建保留策略等

  • gitlab,Admin Area,Settings,Outbound requests,Allow requests to the local network from hooks and services 開啟

  • gitlab上的專案,settings,Integrations,webhook,Merge request events請求觸發
    填寫儲存的URL和token,測試,返回碼為200時配置成功

  • 手動修改gitlab上的原始碼,生成一個merge請求並確認,檢視是否自動觸發jenkins的自動構建

  • 修改構建job,新增Post Steps,在Run only if build succeeds執行指令碼,將構建出來的war包同步到77.200之上,並重啟tomcat

#!/bin/bash
source ~/.bash_profile
set +x
for i in $(seq 50);do echo -n '#';done;echo
cd ${WORKSPACE}/
for war in $(find ./ -type f -name "*.war")
do
  echo "${war} will be deployment"
  RootDir=$(echo ${war}|awk -F'/' '{print $2}')
  echo "${RootDir} project will be update"
  /bin/cp -av ${war} /tmp/ROOT.war
  echo "/bin/rsync -avz /tmp/ROOT.war [email protected]:/web/project/*${RootDir}/"
  /bin/ssh [email protected] rm -rf /web/project/*${RootDir}/*
  /bin/rsync -avz /tmp/ROOT.war [email protected]:/web/project/*${RootDir}/
  /bin/rm -rvf /tmp/ROOT.war
  echo "From 192.168.77.200 restart ${RootDir} project"
  Pid=$(/bin/ssh [email protected] ps -ef|grep ${RootDir}/bin|grep -v grep|awk '{print $2}')
  /bin/ssh [email protected] kill -9 ${Pid}
  echo "Restart ${RootDir} project done"
done
for i in $(seq 50);do echo -n '#';done;echo
set -x

生產多環境手動控制釋出功能的實現:

生產多環境手動控制釋出功能需求細節:
能夠將新版本釋出到生產主機
能夠將上一個版本釋出到生產主機
釋出的方式為全量和金絲雀
包釋出後,tomcat主機重啟方式為全部重啟和一定時間的間隔重啟

  • 建立新的jenkins的job作為生產上線用,為每個子專案單獨建立一個job

  • 在${WORKSPACE}目錄下配置環境檔案:

cat deployOS.lst
# IP:ProjectName:WarPath:SingleDeployFlag
192.168.77.211:/web/tomcat8_8081_shareprofit-merchantport/bin:/web/project/tomcat8_8081_shareprofit-merchantport:True
192.168.77.212:/web/tomcat8_8081_shareprofit-merchantport/bin:/web/project/tomcat8_8081_shareprofit-merchantport:False

該檔案配置了IP指向專案所在的主機,ProjectName用於殺死專案,依據《CentOS6u8 java和tomcat多版本模板部署搭建配置》建立的專案能夠自動拉起
WarPath指向war包所在的目錄,使用者最終的war包分發,SingleDeployFlag用於標識該主機是否為金絲雀釋出主機
該配置檔案較為重要,建議加鎖,該配置檔案儲存在${WORKSPACE}目錄之下,需要建立job之後再配置

  • job相關配置點1:
    引數化構建過程 ——> 選項引數 ——> Option ——> Update;Rollback
    引數化構建過程 ——> 選項引數 ——> Range ——> All;Single
    引數化構建過程 ——> 選項引數 ——> Interval ——> Same time;One by one

  • job相關配置點2:
    Pre Steps,執行shell

#!/bin/bash
source ~/.bash_profile
set +x

# war包版本號
Release=$(date +%Y%m%d.%H%M%S)
# war包同步伺服器
Web=192.168.77.130:80
# 預設釋出間隔
Sleep_time=1

# 判斷髮布的主機列表,是否為單機發布
osCheck(){
if [ ${Range} == 'All' ]
then
  OS_list=$(cat ${WORKSPACE}/deployOS.lst |awk -F':' '{if($1!~/^#/&&$1!~/^$/) print $1}')
else
  OS_list=$(cat ${WORKSPACE}/deployOS.lst |awk -F':' '{if($1!~/^#/&&$1!~/^$/&&$4=="True") print $1}')  
fi
echo -e "[+] $(date +%F_%T) 釋出的環境為:"
echo "${OS_list}"|awk '{print "\t"$1}'
}

# 判斷髮布的方式,是否為同時釋出,設定釋出間隔
intervalCheck(){
if [ "${Interval}" != 'Same time' ]
then
  Sleep_time=60
fi
echo "[+] $(date +%F_%T) 釋出的方式為: ${Interval}"
echo "[+] $(date +%F_%T) Tomcat重啟時間間隔: ${Sleep_time} s"
}

# 同步war包到jenkins主機的nginx目錄中,並清理歷史,保留5個war包
warMoveAndClear(){
WarFile=${JOB_NAME}-${Release}.war
mkdir -p /usr/share/nginx/html/${JOB_NAME}
cd ${WORKSPACE}/${JOB_NAME}/target/
cp -a ${JOB_NAME}*.war /usr/share/nginx/html/${JOB_NAME}/${WarFile}
WarFileNumber=$(ls /usr/share/nginx/html/${JOB_NAME}/*.war|wc -l)
if [ "${WarFileNumber}" -ge 6 ]
then
  StandFile=$(ls -t /usr/share/nginx/html/${JOB_NAME}/*.war|head -6|tail -1)
  find /usr/share/nginx/html/${JOB_NAME}/ -type f -name "*.war" -not -newer ${StandFile} -exec rm -f {} \;
fi
echo "[+] $(date +%F_%T) 本次釋出為升級釋出"
echo "[+] $(date +%F_%T) 釋出war包為: ${WarFile}"
}

# 回滾功能的包名定位
warFileFind(){
WarFile=$(ls -t /usr/share/nginx/html/${JOB_NAME}/*.war|head -2|tail -1|awk -F'/' '{print $NF}')
echo "[+] $(date +%F_%T) 本次釋出為回滾釋出"
echo "[+] $(date +%F_%T) 釋出war包為: ${WarFile}"
}

# 釋出環境下載war包,該包可能是回滾包,也可能是釋出包
warDownload(){
for IP in ${OS_list}
do
  ssh [email protected]${IP} "rm -vf /tmp/${JOB_NAME}.war"
  echo "[+] $(date +%F_%T) ${IP} 釋出包 ${WarFile} 下載開始"
  ssh [email protected]${IP} "wget http://${Web}/${JOB_NAME}/${WarFile} -O /tmp/${JOB_NAME}.war -o /dev/null"
done
}

# 釋出功能
restartTomcat(){
TomHost=${1}
WarPath=${2}
Project=${3}
ssh [email protected]${TomHost} "rm -rf ${WarPath}/*"
ssh [email protected]${TomHost} "mv /tmp/${JOB_NAME}.war ${WarPath}/ROOT.war"
TomPid=$(ssh [email protected]${TomHost} ps -ef|grep ${Project}|grep -v grep|awk '{print $2}')
ssh [email protected]${TomHost} kill -9 ${TomPid}
echo "[+] $(date +%F_%T) 休眠${Sleep_time}s"
sleep ${Sleep_time}
}

# 最終釋出
finalRestart(){
for IP in ${OS_list}
do
  TomHost=$(grep ${IP} ${WORKSPACE}/deployOS.lst|awk -F':' '{print $1}')
  WarPath=$(grep ${IP} ${WORKSPACE}/deployOS.lst|awk -F':' '{print $3}')
  Project=$(grep ${IP} ${WORKSPACE}/deployOS.lst|awk -F':' '{print $2}')
  echo "[+] $(date +%F_%T) 釋出開始 ${TomHost} ${WarPath} ${Project}"
  restartTomcat ${TomHost} ${WarPath} ${Project}
done
}

# 根據Option來確定是升級還是回滾,升級和回滾的區別在於同步的包是否是構建的包或者歷史版本包
# 當執行回滾之時,返回碼為非0,表示指令碼執行失敗,那麼放到Pre Steps,執行回滾時,就不會繼續做構建的操作了
if [ "${Option}" == 'Rollback' ]
then
  for i in $(seq 50);do echo -n '#';done;echo
  osCheck
  intervalCheck
  warFileFind
  warDownload
  finalRestart
  for i in $(seq 50);do echo -n '#';done;echo
  exit 1
fi
set -x
  • job相關配置點3:
    Post Steps,Run only if build succeeds,執行shell
#!/bin/bash
source ~/.bash_profile
set +x

# war包版本號
Release=$(date +%Y%m%d.%H%M%S)
# war包同步伺服器
Web=192.168.77.130:80
# 預設釋出間隔
Sleep_time=1

# 判斷髮布的主機列表,是否為單機發布
osCheck(){
if [ ${Range} == 'All' ]
then
  OS_list=$(cat ${WORKSPACE}/deployOS.lst |awk -F':' '{if($1!~/^#/&&$1!~/^$/) print $1}')
else
  OS_list=$(cat ${WORKSPACE}/deployOS.lst |awk -F':' '{if($1!~/^#/&&$1!~/^$/&&$4=="True") print $1}')  
fi
echo -e "[+] $(date +%F_%T) 釋出的環境為:"
echo "${OS_list}"|awk '{print "\t"$1}'
}

# 判斷髮布的方式,是否為同時釋出,設定釋出間隔
intervalCheck(){
if [ "${Interval}" != 'Same time' ]
then
  Sleep_time=60
fi
echo "[+] $(date +%F_%T) 釋出的方式為: ${Interval}"
echo "[+] $(date +%F_%T) Tomcat重啟時間間隔: ${Sleep_time} s"
}

# 同步war包到jenkins主機的nginx目錄中,並清理歷史,保留5個war包
warMoveAndClear(){
WarFile=${JOB_NAME}-${Release}.war
mkdir -p /usr/share/nginx/html/${JOB_NAME}
cd ${WORKSPACE}/${JOB_NAME}/target/
cp -a ${JOB_NAME}*.war /usr/share/nginx/html/${JOB_NAME}/${WarFile}
WarFileNumber=$(ls /usr/share/nginx/html/${JOB_NAME}/*.war|wc -l)
if [ "${WarFileNumber}" -ge 6 ]
then
  StandFile=$(ls -t /usr/share/nginx/html/${JOB_NAME}/*.war|head -6|tail -1)
  find /usr/share/nginx/html/${JOB_NAME}/ -type f -name "*.war" -not -newer ${StandFile} -exec rm -f {} \;
fi
echo "[+] $(date +%F_%T) 本次釋出為升級釋出"
echo "[+] $(date +%F_%T) 釋出war包為: ${WarFile}"
}

# 回滾功能的包名定位
warFileFind(){
WarFile=$(ls -t /usr/share/nginx/html/${JOB_NAME}/*.war|head -2|tail -1|awk -F'/' '{print $NF}')
echo "[+] $(date +%F_%T) 本次釋出為回滾釋出"
echo "[+] $(date +%F_%T) 釋出war包為: ${WarFile}"
}

# 釋出環境下載war包,該包可能是回滾包,也可能是釋出包
warDownload(){
for IP in ${OS_list}
do
  ssh [email protected]${IP} "rm -vf /tmp/${JOB_NAME}.war"
  echo "[+] $(date +%F_%T) ${IP} 釋出包 ${WarFile} 下載開始"
  ssh [email protected]${IP} "wget http://${Web}/${JOB_NAME}/${WarFile} -O /tmp/${JOB_NAME}.war -o /dev/null"
done
}

# 釋出功能
restartTomcat(){
TomHost=${1}
WarPath=${2}
Project=${3}
ssh [email protected]${TomHost} "rm -rf ${WarPath}/*"
ssh [email protected]${TomHost} "mv /tmp/${JOB_NAME}.war ${WarPath}/ROOT.war"
TomPid=$(ssh [email protected]${TomHost} ps -ef|grep ${Project}|grep -v grep|awk '{print $2}')
ssh [email protected]${TomHost} kill -9 ${TomPid}
echo "[+] $(date +%F_%T) 休眠${Sleep_time}s"
sleep ${Sleep_time}
}

# 最終釋出
finalRestart(){
for IP in ${OS_list}
do
  TomHost=$(grep ${IP} ${WORKSPACE}/deployOS.lst|awk -F':' '{print $1}')
  WarPath=$(grep ${IP} ${WORKSPACE}/deployOS.lst|awk -F':' '{print $3}')
  Project=$(grep ${IP} ${WORKSPACE}/deployOS.lst|awk -F':' '{print $2}')
  echo "[+] $(date +%F_%T) 釋出開始 ${TomHost} ${WarPath} ${Project}"
  restartTomcat ${TomHost} ${WarPath} ${Project}
done
}

# 根據Option來確定是升級還是回滾,升級和回滾的區別在於同步的包是否是構建的包或者歷史版本包
if [ "${Option}" == 'Update' ]
then
  for i in $(seq 50);do echo -n '#';done;echo
  osCheck
  intervalCheck
  warMoveAndClear
  warDownload
  finalRestart
  for i in $(seq 50);do echo -n '#';done;echo
fi
set -x
  • 需要收集的資訊:
    gitlab地址,子專案名
    對應的生產環境主機IP,金絲雀釋出用的IP,每個IP上的war包目錄和重啟用的專案名
    指令碼中配置的war包同步伺服器,也就是搭建的nginx伺服器IP地址需要根據情況修改
    jenkins主機的deploy使用者能夠直接操控測試環境主機和所有的生產環境主機做部署和重啟tomcat

  • 第一次構建使用Rollback,生成${WORKSPACE}目錄,配置deployOS.lst檔案

後期優化建議:

配置使用ansible來操縱各個環境主機
將jenkins的job改成pipeline型別,更加精準的控制
tomcat重啟後檢測catalina.out日誌來確認重啟是否成功
ELK生產多環境日誌監控

[TOC]