1. 程式人生 > >Nginx+keepalived+tomcat+負載均衡+反向代理

Nginx+keepalived+tomcat+負載均衡+反向代理

需求說明

實現nginx的keepalived和負載均衡

環境說明

ip 伺服器型別
172.16.11.19 nginx負載均衡伺服器(master)
172.16.11.21 nginx負載均衡伺服器(backup)
172.16.11.18 tomcat
172.16.11.20 tomcat

①.關閉防火牆

[
[email protected]
~]# systemctl stop firewalld.service

②.關閉selinux

[[email protected] ~]# setenforce 0
[[email protected] ~]# sed -ri 's/^(SELINUX=).*/\1disabled/g' /etc/selinux/config 

③.配置網路源

[[email protected] ~]# curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
[
[email protected]
~]# sed -i 's/$releasever/7/g' /etc/yum.repos.d/CentOS-Base.repo

操作步驟

  • 172.16.11.19

①.安裝nginx和keepalived

[[email protected] ~]# yum -y install epel-release
[[email protected] ~]# yum -y install nginx keepalived

②.配置nginx的歡迎主頁,用於區分兩個節點的nginx

[[email protected]
~]# cd /usr/share/nginx/html/ [[email protected] html]# mv index.html{,.bak} [[email protected] html]# echo 'master' > index.html

③.啟動nginx服務測試nginx

[[email protected] ~]# systemctl start nginx
[[email protected] ~]# systemctl enable nginx
Created symlink from /etc/systemd/system/multi-user.target.wants/nginx.service to /usr/lib/systemd/system/nginx.service.
[[email protected] ~]# ss -antl
State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              
LISTEN     0      128           *:80                        *:*                  
LISTEN     0      128           *:22                        *:*                  
LISTEN     0      100    127.0.0.1:25                        *:*                  
LISTEN     0      128          :::80                       :::*                  
LISTEN     0      128          :::22                       :::*                  
LISTEN     0      100         ::1:25                       :::*  

在這裡插入圖片描述

④.配置keepalived

[[email protected] ~]# cd /etc/keepalived/
[[email protected] keepalived]# cp keepalived.conf keepalived.conf.bak          //備份主配置檔案
[[email protected] keepalived]# vim keepalived.conf
! Configuration File for keepalived

global_defs {           //全域性配置
// keepalived 自帶的郵件提醒需要開啟 sendmail 服務。 建議用獨立的監控或第三方 SMTP
   router_id lb_01     //標識節點的字串,同區域網內唯一
}

vrrp_instance VI_1 {           //例項,定義虛擬路由
    state MASTER        //節點的初始狀態,主為master,備為backup
    interface eth0        //繫結虛擬ip的網路介面,必須與本地ip所在介面相同
    virtual_router_id 51           //虛擬路由id號,兩個節點必須一樣
    priority 100           //節點優先順序,master要比backup高
    advert_int 1        //兩個節點發送組播資訊間隔時間,主備需要設定相同的時間,預設為秒
    authentication {          //配置認證
        auth_type PASS             //PASS即為密碼認證
        auth_pass 1111          //設定密碼,可自定義
    }
    virtual_ipaddress {        //設定虛擬ip(VIP),兩個節點需要一樣
        172.16.11.200
    }
}

virtual_server 172.16.11.200 80 {         //虛擬伺服器配置
    delay_loop 6      //健康檢查的時間間隔
    lb_algo rr          //lvs排程演算法
    lb_kind NAT        //lvs模式
    persistence_timeout 50           //持久化超時時間,單位為秒
    protocol TCP               //OSI的四層協議,即網路層

    real_server 172.16.11.19 80 {         //真實處理請求的伺服器配置
        weight 1        //指定權重
        TCP_CHECK {
            connect_timeout 3            //連線超時時間
            nb_get_retry 3       //重試次數
            delay_before_retry 3         //在重試之前延遲多長時間
        }
    }
    real_server 172.16.11.21 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }

}

⑤.啟動keepalived

[[email protected] keepalived]# systemctl start keepalived.service 
[[email protected] keepalived]# systemctl enable keepalived.service 
Created symlink from /etc/systemd/system/multi-user.target.wants/keepalived.service to /usr/lib/systemd/system/keepalived.service.

  • 172.16.11.21

①.安裝nginx和keepalived

[[email protected] ~]# yum -y install epel-release
[[email protected] ~]# yum -y install nginx keepalived

②.配置nginx的歡迎主頁,用於區分兩個節點的nginx

[[email protected] ~]# cd /usr/share/nginx/html/
[[email protected] html]# mv index.html{,.bak}
[[email protected] html]# echo 'backup' > index.html

③.啟動nginx服務測試nginx

[[email protected] ~]# systemctl start nginx
[[email protected] ~]# systemctl enable nginx
Created symlink from /etc/systemd/system/multi-user.target.wants/nginx.service to /usr/lib/systemd/system/nginx.service.
[[email protected] ~]# ss -antl
State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              
LISTEN     0      128           *:80                        *:*                  
LISTEN     0      128           *:22                        *:*                  
LISTEN     0      100    127.0.0.1:25                        *:*                  
LISTEN     0      128          :::80                       :::*                  
LISTEN     0      128          :::22                       :::*                  
LISTEN     0      100         ::1:25                       :::*  

在這裡插入圖片描述
④.配置keepalived

[[email protected] ~]# cd /etc/keepalived/
[[email protected] keepalived]# cp keepalived.conf keepalived.conf.bak          //備份主配置檔案
[[email protected] keepalived]# vim keepalived.conf
! Configuration File for keepalived

global_defs {           //全域性配置
// keepalived 自帶的郵件提醒需要開啟 sendmail 服務。 建議用獨立的監控或第三方 SMTP
   router_id lb_02    //標識節點的字串,同區域網內唯一
}

vrrp_instance VI_1 {           //例項,定義虛擬路由
    state BACKUP        //節點的初始狀態,主為master,備為backup
    interface ens33        //繫結虛擬ip的網路介面,必須與本地ip所在介面相同
    virtual_router_id 51           //虛擬路由id號,兩個節點必須一樣
    priority 90           //節點優先順序,master要比backup高
    advert_int 1        //兩個節點發送組播資訊間隔時間,主備需要設定相同的時間,預設為秒
    authentication {          //配置認證
        auth_type PASS             //PASS即為密碼認證
        auth_pass 1111          //設定密碼,可自定義
    }
    virtual_ipaddress {        //設定虛擬ip(VIP),兩個節點需要一樣
        172.16.11.200
    }
}

virtual_server 172.16.11.200 80 {         //虛擬伺服器配置
    delay_loop 6      //健康檢查的時間間隔
    lb_algo rr          //lvs排程演算法
    lb_kind NAT        //lvs模式
    persistence_timeout 50           //持久化超時時間,單位為秒
    protocol TCP               //OSI的四層協議,即網路層

    real_server 172.16.11.19 80 {         //真實處理請求的伺服器配置
        weight 1        //指定權重
        TCP_CHECK {
            connect_timeout 3            //連線超時時間
            nb_get_retry 3       //重試次數
            delay_before_retry 3         //在重試之前延遲多長時間
        }
    }
    real_server 172.16.11.21 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }

}

⑤.啟動keepalived

[[email protected] keepalived]# systemctl start keepalived.service 
[[email protected] keepalived]# systemctl enable keepalived.service 
Created symlink from /etc/systemd/system/multi-user.target.wants/keepalived.service to /usr/lib/systemd/system/keepalived.service.

分別在兩臺機器上檢視ip

  • 172.16.11.19
[[email protected] keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:35:dd:b8 brd ff:ff:ff:ff:ff:ff
    inet 172.16.11.19/24 brd 172.16.11.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 172.16.11.200/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe35:ddb8/64 scope link 
       valid_lft forever preferred_lft forever
  • 172.16.11.21
[[email protected] keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:72:aa:10 brd ff:ff:ff:ff:ff:ff
    inet 172.16.11.21/24 brd 172.16.11.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::c6eb:d9f9:32e9:f3df/64 scope link 
       valid_lft forever preferred_lft forever

可以看出來,在主master伺服器上,出現了虛擬ip,這時候停掉主master伺服器的nginx和keepalived,再來看看

  • 172.16.11.19
[[email protected] keepalived]# systemctl stop nginx
[[email protected] keepalived]# systemctl stop keepalived.service 
[[email protected] keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:35:dd:b8 brd ff:ff:ff:ff:ff:ff
    inet 172.16.11.19/24 brd 172.16.11.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe35:ddb8/64 scope link 
       valid_lft forever preferred_lft forever
  • 172.16.11.21
[[email protected] keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:72:aa:10 brd ff:ff:ff:ff:ff:ff
    inet 172.16.11.21/24 brd 172.16.11.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 172.16.11.200/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::c6eb:d9f9:32e9:f3df/64 scope link 
       valid_lft forever preferred_lft forever

可以看到虛擬ip漂移到了備backup伺服器上面,現在備backup伺服器成為了主master


為了防止出現主master伺服器nginx掛了,然後keeplived沒有停掉,導致出現虛擬ip還是在掛掉的nginx伺服器上面,但是因為nginx掛掉了無法訪問後端伺服器,需要寫一個指令碼來監控nginx的狀態並自動控制keeplived,這個指令碼放在主master伺服器上面,

  • 172.16.11.19
    ①.寫監控nginx的指令碼
[[email protected] ~]# mkdir /etc/keepalived/scripts
[[email protected] ~]# cd /etc/keepalived/scripts
[[email protected] scripts]# vim check_n.sh
#!/bin/bash

status=`ps -ef | grep '\bnginx\b' | grep -Ev "grep|${0}" | wc -l`

if [ $status -eq 0 ];then
     systemctl stop keepalived
fi
[[email protected] scripts]# chmod +x check_n.sh 

②.修改master的配置檔案,在global_defs全域性配置後新增

[[email protected] scripts]# vim /etc/keepalived/keepalived.conf
vrrp_script nginx_check {
    script "/etc/keeplived/scripts/check_n.sh"         //要執行的指令碼的路徑
    interval 1          //執行指令碼間隔時間,單位為秒,預設1秒
    weight -20     //調整優先順序
}

然後寫一個指令碼,通過這個指令碼來使當主master伺服器的nginx掛掉,然後backup變為master的時候傳送郵件,並自動啟動nginx,當原master伺服器被修復以後,原backup伺服器恢復原樣,並停掉nginx.(注:只能有一邊的nginx啟動才能通過虛擬ip訪問,不然兩邊都啟動nginx,即使你的虛擬ip在master上也訪問不到)

發郵件的伺服器上面需要安裝mailx
yum -y install mailx

  • 172.16.11.19

①.在master上編寫指令碼

[[email protected] scripts]# vim notify.sh
#!/bin/bash

VIP=$2
sendmail (){
        subject="${VIP} ip address drift"        content="`date +'%F-%H%M%S'`:`ip a | grep eth0 | grep inet | head -1 | awk -F '[ /]+' '{print $3}'` change to master"
        echo $conter | mail -s "$subject" [email protected]
}

case "$1" in
  master)
            systemctl start nginx

;;
  backup)        
            systemctl stop nginx

;;
  *)
        echo "Usage:$0 master|backup $VIP"
;;
esac
~     
[[email protected] scripts]# chmod +x notify.sh 

②.修改配置檔案,在vrrp_instance裡面最後新增

[[email protected] scripts]# vim /etc/keepalived/keepalived.conf
vrrp_instance VI_1 {
......
    track_script {
        nginx_check
    }
    notify_master "/etc/keeplived/scripts/notify.sh master 172.16.11.200"
    notify_backup "/etc/keeplived/scripts/notify.sh backup 172.16.11.200"

}
  • 172.16.11.21

backup無需檢測nginx是否正常,當升級為master時啟動nginx,當降級為backup時關閉

①.在backup上編寫指令碼

[[email protected] ~]# mkdir /etc/keepalived/scripts
[[email protected] ~]# cd /etc/keepalived/scripts
[[email protected] scripts]# vim notify.sh
#!/bin/bash

VIP=$2
sendmail (){
        subject="${VIP} ip address drift"
        content="`date +'%F-%H%M%S'`:`ip a | grep ens33 | grep inet | head -1 | awk -F '[ /]+' '{print $3}'` change to master"
        echo $conter | mail -s "$subject" [email protected]
}

case "$1" in
            systemctl start nginx
;;
  backup)
            systemctl stop nginx
;;
  *)
        echo "Usage:$0 master|backup $VIP"
;;
esac
[[email protected] scripts]# chmod +X notify.sh 
[[email protected] scripts]# yum -y install mailx      //安裝郵箱命令

②.修改配置檔案,在vrrp_instance裡面最後新增

[[email protected] scripts]# vim /etc/keepalived/keepalived.conf
vrrp_instance VI_1 {
......

    notify_master "/etc/keeplived/scripts/notify.sh master 172.16.11.200"
    notify_backup "/etc/keeplived/scripts/notify.sh backup 172.16.11.200"

}

到這裡就可以nginx的keepalived就配置完成了,接下來就可以來配置負載均衡和反向代理了

  • 172.16.11.18
    ①.安裝jdk開發環境
[[email protected] ~]# yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel

②.到官網下載tomcat包

 [[email protected] ~]# wget https://archive.apache.org/dist/tomcat/tomcat-9/v9.0.8/bin/apache-tomcat-9.0.8.tar.gz

③.解壓tomcat,並配置網頁

[[email protected] ~]# tar -xf apache-tomcat-9.0.8.tar.gz -C /usr/local/
[[email protected] ~]# ln -s /usr/local/apache-tomcat-9.0.8/ /usr/local/tomcat
[[email protected] tomcat]# cd /usr/local/tomcat/webapps/
[[email protected] webapps]# mkdir test
[[email protected] webapps]# vim test/index.jsp
<html>
<head>
        <title>test page</title>
</head>
<body>
<       %
            out.println("Hellow World");
        %>
</body>
</html>
[[email protected] webapps]# /usr/local/tomcat/bin/catalina.sh start
 
  • 172.16.11.20
    ①.安裝jdk開發環境
[[email protected] ~]# yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel

②.到官網下載tomcat包

 [[email protected] ~]# wget https://archive.apache.org/dist/tomcat/tomcat-9/v9.0.8/bin/apache-tomcat-9.0.8.tar.gz

③.解壓tomcat,並配置網頁

[[email protected] ~]# tar -xf apache-tomcat-9.0.8.tar.gz -C /usr/local/
[[email protected] ~]# ln -s /usr/local/apache-tomcat-9.0.8/ /usr/local/tomcat
[[email protected] tomcat]# cd /usr/local/tomcat/webapps/
[[email protected] webapps]# mkdir test
[[email protected] webapps]# vim test/index.jsp
<html>
<head>
        <title>test page</title>
</head>
<body>
<       %
            out.println("Hellow World too");
        %>
</body>
</html>
[[email protected] webapps]# /usr/local/tomcat/bin/catalina.sh start
 

在nginx上配置負載均衡和反向代理

  • 172.16.11.19
[[email protected] scripts]# vim /etc/nginx/nginx.conf
    upstream web.com {                    //新增在http段
        server 172.16.11.18:8080;
        server 172.16.11.20:8080
    }
        location ~ \.jsp {                        //新增在server段
            proxy_pass http://web.com;
        }


  • 172.16.11.21
[[email protected] scripts]# vim /etc/nginx/nginx.conf
    upstream web.com {                    //新增在http段
        server 172.16.11.18:8080;
        server 172.16.11.20:8080;
    }
        location ~ \.jsp {                        //新增在server段
            proxy_pass http://web.com;
        }


測試

當master的nginx和keepalived都啟動的時候,訪問虛擬ip

  • 172.16.11.19
[[email protected] ~]# systemctl start nginx
[[email protected] ~]# systemctl start keepalived.service 
[[email protected] ~]# ip add show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:35:dd:b8 brd ff:ff:ff:ff:ff:ff
    inet 172.16.11.19/24 brd 172.16.11.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 172.16.11.200/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe35:ddb8/64 scope link 
       valid_lft forever preferred_lft forever

在這裡插入圖片描述
在這裡插入圖片描述

可以看到訪問到後端的tomcat伺服器上面並實現了負載均衡


然後停掉master的nginx,可以看到keepalived也自動停掉了,並且原backup成為了master,虛擬ip漂移,也傳送了郵件

  • 172.16.11.19
[[email protected] ~]# systemctl stop nginx
[[email protected] ~]# systemctl status keepalived.service 
● keepalived.service - LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Thu 2018-11-01 23:59:10 CST; 17s ago
  Process: 40968 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 40969 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/keepalived.service

Nov 01 23:49:57 lizihan Keepalived_vrrp[40971]: VRRP_Instance(VI_1) Sending/qu...0
Nov 01 23:49:57 lizihan Keepalived_vrrp[40971]: Sending gratuitous ARP on eth0...0
Nov 01 23:49:57 lizihan Keepalived_vrrp[40971]: Sending gratuitous ARP on eth0...0
Nov 01 23:49:57 lizihan Keepalived_vrrp[40971]: Sending gratuitous ARP on eth0...0
Nov 01 23:49:57 lizihan Keepalived_vrrp[40971]: Sending gratuitous ARP on eth0...0
Nov 01 23:59:09 lizihan Keepalived[40969]: Stopping
Nov 01 23:59:09 lizihan systemd[1]: Stopping LVS and VRRP High Availability M.....
Nov 01 23:59:09 lizihan Keepalived_vrrp[40971]: VRRP_Instance(VI_1) sent 0 pri...y
Nov 01 23:59:09 lizihan Keepalived_vrrp[40971]: VRRP_Instance(VI_1) removing p....
Nov 01 23:59:10 lizihan systemd[1]: Stopped LVS and VRRP High Availability Mo...r.
Hint: Some lines were ellipsized, use -l to show in full.
[[email protected] ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:35:dd:b8 brd ff:ff:ff:ff:ff:ff
    inet 172.16.11.19/24 brd 172.16.11.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe35:ddb8/64 scope link 
       valid_lft forever preferred_lft forever

  • 172.16.11.21
[[email protected] ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:72:aa:10 brd ff:ff:ff:ff:ff:ff
    inet 172.16.11.21/24 brd 172.16.11.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 172.16.11.200/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::c6eb:d9f9:32e9:f3df/64 scope link 
       valid_lft forever preferred_lft forever

在這裡插入圖片描述

這時候再訪問虛擬ip
在這裡插入圖片描述

在這裡插入圖片描述