利用saltstack一鍵部署高可用負載均衡叢集
實驗環境:
Server1 172.25.254.1 maseter/minion keepalived/haproxy
Server2 172.25.254.2 minion httpd
Server3 172.25.254.3 minion nginx
Server4 172.25.254.4 minion keepalived/haproxy
配置一臺新的server4進行配置salt-minion:
[[email protected] ~]# yum install salt-minion -y ## 配置完整yum源
[[email protected] ~]# cd /etc/salt/
[[email protected] salt]# ls
cloud cloud.maps.d master minion.d proxy.d
cloud.conf.d cloud.profiles.d master.d pki roster
cloud.deploy.d cloud.providers.d minion proxy
[[email protected] salt]# vim minion 填寫master端的IP
master: 172.25.254.1
[ [email protected] salt]# /etc/init.d/salt-minion start 開啟服務
Starting salt-minion:root:server4 daemon: OK
交換公鑰,新增server4到server1master端
[[email protected] files]# salt-key -A 新增公鑰匹配
The following keys are going to be accepted:
Unaccepted Keys:
server4
Proceed? [n/Y] y
Key for minion server4 accepted.
[ [email protected] files]# salt-key -L 查詢新增成功
Accepted Keys:
server1
server2
server3
server4
Denied Keys:
Unaccepted Keys:
Rejected Keys:
配置高可用模組keepalived:
[[email protected] ~]# cd /srv/salt/
[[email protected] salt]# ls
_grains haproxy httpd nginx pkgs top.sls users
[[email protected] salt]# mkdir keepalived
[[email protected] salt]# cd keepalived/
[[email protected] keepalived]# ls
[[email protected] keepalived]# vim install.sls
[[email protected] keepalived]# cat install.sls
kp-install:
file.managed:
- name: /mnt/keepalived-2.0.6.tar.gz
- source: salt://keepalived/files/keepalived-2.0.6.tar.gz
[[email protected] keepalived]# mkdir files
[[email protected] keepalived]# cd files/
[[email protected] files]# ls
[[email protected] files]# pwd
/srv/salt/keepalived/files
[[email protected] files]# ls
keepalived-2.0.6.tar.gz
在server1上推送keepalived
[[email protected] files]# salt server4 state.sls keepalived.install
推送成功結果:在server4的/mnt/目錄下可以看到推送過來的安裝包
[[email protected] salt]# cd /mnt/
[[email protected] mnt]# ls
keepalived-2.0.6.tar.gz
在server1完善keepalived的安裝指令碼進行推送:
[[email protected] keepalived]# cd ..
[[email protected] salt]# ls
_grains haproxy httpd keepalived nginx pkgs top.sls users
[[email protected] salt]# cd pkgs/
[[email protected] pkgs]# ls
make.sls
[[email protected] pkgs]# cat make.sls ## 原始碼編譯一些依賴性的軟體包
make-gcc:
pkg.installed:
- pkgs:
- pcre-devel
- openssl-devel
- gcc
[[email protected] pkgs]# cd ..
[[email protected] salt]# cd keepalived/
[[email protected] keepalived]# ls
files install.sls
[[email protected] keepalived]# vim install.sls
[[email protected] keepalived]# cat install.sls
include:
- pkgs.make 匯入工具包
kp-install:
file.managed:
- name: /mnt/keepalived-2.0.6.tar.gz
- source: salt://keepalived/files/keepalived-2.0.6.tar.gz
cmd.run: 進行原始碼編譯不顯示過程資訊
- name: cd /mnt && tar zxf keepalived-2.0.6.tar.gz && cd keepalived-2.0.6 && ./configure --prefix=/usr/local/keepalived --with-init=SYSV &> /dev/null && make &> /dev/null && make install &> /dev/null
- creates: /usr/local/keepalived
[[email protected] keepalived]# salt server4 state.sls keepalived.install 進行推送在server4安裝keepalived
安裝過程中在server4可以檢視相應程序
[[email protected] mnt]# ps ax
在server4的對應安裝位置上已經有了keepalived
[[email protected] mnt]# ls
keepalived-2.0.6 keepalived-2.0.6.tar.gz
[[email protected] mnt]# ll /usr/local/keepalived/
total 16
drwxr-xr-x 2 root root 4096 Aug 18 10:42 bin
drwxr-xr-x 5 root root 4096 Aug 18 10:42 etc
drwxr-xr-x 2 root root 4096 Aug 18 10:42 sbin
drwxr-xr-x 5 root root 4096 Aug 18 10:42 share
配置Keepalived的高可用
將server4的keepalived的配置檔案/usr/local/keepalived/etc/keepalived/keepalived.conf和啟動指令碼/usr/local/keepalived/etc/rc.d/init.d/keepalived傳送到server1的/srv/salt/keepalived/files目錄下
[[email protected] mnt]# cd /usr/local/keepalived/etc/rc.d/init.d/
[[email protected] init.d]# scp keepalived server1:/srv/salt/keepalived/files ## keepalived的啟動指令碼
[[email protected] ~]# cd /usr/local/keepalived/etc/keepalived/
[[email protected] keepalived]# ls
keepalived.conf samples
[[email protected] keepalived]# scp keepalived.conf server1:/srv/salt/keepalived/files ## 配置檔案
在server1配置安裝指令碼製作相應的軟連結:
[[email protected] ~ ]# cd /srv/salt/keepalived
[[email protected] keepalived]# ls
files install.sls
[[email protected] keepalived]# vim install.sls
[[email protected] keepalived]# cat install.sls
include:
- pkgs.make
kp-install:
file.managed:
- name: /mnt/keepalived-2.0.6.tar.gz
- source: salt://keepalived/files/keepalived-2.0.6.tar.gz
cmd.run:
- name: cd /mnt && tar zxf keepalived-2.0.6.tar.gz && cd keepalived-2.0.6 && ./configure --prefix=/usr/local/keepalived --with-init=SYSV &> /dev/null && make &> /dev/null && make install &> /dev/null
- creates: /usr/local/keepalived
/etc/keepalived:
file.directory:
- mode: 755 指令碼的許可權
/etc/sysconfig/keepalived: 製作軟連結
file.symlink:
- target: /usr/local/keepalived/etc/sysconfig/keepalived
/sbin/keepalived: 製作軟連結
file.symlink:
- target: /usr/local/keepalived/sbin/keepalived
[[email protected] keepalived]# salt server4 state.sls keepalived.install 進行推送
在server4可以看到對應的軟連結代表推送成功:
在serevr1安裝開啟keepalived服務的指令碼:
[[email protected] keepalived]# pwd
/srv/salt/keepalived
[[email protected] keepalived]# vim service.sls
[[email protected] keepalived]# cat service.sls
include:
- keepalived.install 匯入安裝指令碼
/etc/keepalived/keepalived.conf:
file.managed:
- source: salt://keepalived/files/keepalived.conf
kp-service:
file.managed:
- name: /etc/init.d/keepalived
- source: salt://keepalived/files/keepalived
- mode: 755
service.running:
- name: keepalived
- reload: True
- watch:
- file: /etc/keepalived/keepalived.conf
[[email protected] keepalived]# cd /srv/pillar/ 進去pillar模組
[[email protected] pillar]# ls
top.sls web
[[email protected] pillar]# cd web/
[[email protected] web]# ls
install.sls
[[email protected] web]# vim install.sls
[[email protected] web]# cat install.sls
{% if grains['fqdn'] == 'server2' %} 如果主機名字為server2
webserver: httpd
bind: 172.25.254.2
port: 80
{% elif grains['fqdn'] == 'server3' %}
webserver: nginx
{% endif %}
[[email protected] web]# cd ..
[[email protected] pillar]# ls
top.sls web
[[email protected] pillar]# mkdir keepalived
[[email protected] pillar]# cd keepalived/
[[email protected] keepalived]# ls
[[email protected] keepalived]# cp ../web/install.sls .
[[email protected] keepalived]# vim install.sls
[[email protected] keepalived]# cat install.sls
{% if grains['fqdn'] == 'server1' %}
state: MASTER 寫入狀態MASTER還是BACKUP測試高可用
vrid: 38
priority: 100 寫入優先順序
{% elif grains['fqdn'] == 'server4' %}
state: BACKUP
vrid: 38
priority: 50
{% endif %}
[[email protected] keepalived]# ls
install.sls
[[email protected] keepalived]# cd ..
[[email protected] pillar]# ls
keepalived top.sls web
[[email protected] pillar]# vim top.sls
[[email protected] pillar]# cat top.sls
base:
'*':
- web.install
- keepalived.install
[[email protected] pillar]# cd ..
[[email protected] srv]# cd salt/
[[email protected] salt]# ls
_grains haproxy apache keepalived nginx pkgs top.sls users
[[email protected] salt]# cd keepalived/
[[email protected] keepalived]# ls
files install.sls service.sls
配置keppalived的一鍵安裝並開啟服務加入優先順序:
[[email protected] keepalived]# vim service.sls
[[email protected] keepalived]# cat service.sls
include:
- keepalived.install
/etc/keepalived/keepalived.conf:
file.managed:
- source: salt://keepalived/files/keepalived.conf:
- template: jinja
- context:
STATE: {{ pillar['state'] }}
VRID: {{ pillar['vrid'] }}
PRIORITY: {{ pillar['priority'] }}
kp-service:
file.managed:
- name: /etc/init.d/keepalived
- source: salt://keepalived/files/keepalived
- mode: 755
service.running:
- name: keepalived
- reload: True
- watch:
- file: /etc/keepalived/keepalived.conf
[[email protected] keepalived]# vim files/keepalived
[[email protected] keepalived]# ls
files install.sls service.sls
[[email protected] keepalived]# cd files/
[[email protected] files]# ls
keepalived keepalived-2.0.6.tar.gz keepalived.conf
[[email protected] files]# vim keepalived.conf 編輯主配置檔案寫入虛擬IP
[[email protected] keepalived]# salt server4 state.sls keepalived.service 推送service服務:
在server4可以看到對應的配置檔案還有指令碼檢視有相應程序:
在server4檢視已經接管虛擬IP:
[[email protected] keepalived]# ip addr
2: eth0:
inet 172.25.254.4/24 brd 172.25.254.255 scope global eth0
inet 172.25.254.100/32 scope global eth0
在server1編寫top.sls指令碼在對應節點安裝服務:
[[email protected] keepalived]# pwd
/srv/salt/keepalived
[[email protected] keepalived]# cd ..
[[email protected] salt]# ls
_grains haproxy apache keepalived nginx pkgs top.sls users
[[email protected] salt]# vim top.sls
[[email protected] salt]# cat top.sls
base:
'server1':
- haproxy.install
- keepalived.service
'server4':
- haproxy.install
- keepalived.service
'roles:apache':
- match: grain
- apache.service
'roles:nginx':
- match: grain
- nginx.service
[[email protected] salt]# salt '*' state.highstate
在server4檢視已經有相應的haproxy和keepalived的程序:
[[email protected] keepalived]# ps ax
[[email protected] keepalived]# ip addr 虛擬IP已經轉換到server1
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 52:54:00:52:c1:00 brd ff:ff:ff:ff:ff:ff
inet 172.25.254.4/24 brd 172.25.254.255 scope global eth0
inet6 fe80::5054:ff:fe52:c100/64 scope link
valid_lft forever preferred_lft forever
在網頁測試後端可以負載均衡:
將server1的keepalived服務關閉:
[[email protected] salt]# /etc/init.d/keepalived stop
Stopping keepalived: [ OK ]
在網頁再次測試後端依舊可以負載均衡
server4接管虛擬IP成為新的master:
[[email protected] keepalived]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 52:54:00:52:c1:00 brd ff:ff:ff:ff:ff:ff
inet 172.25.254.4/24 brd 172.25.254.255 scope global eth0
inet 172.25.254.100/32 scope global eth0
inet6 fe80::5054:ff:fe52:c100/64 scope link
valid_lft forever preferred_lft forever
server1再次開啟keepalived會搶回來master的身份:
[[email protected] salt]# /etc/init.d/keepalived start
Starting keepalived: [ OK ]
[[email protected] salt]# ip addr 已經接管虛擬IP
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 52:54:00:58:4d:1f brd ff:ff:ff:ff:ff:ff
inet 172.25.38.1/24 brd 172.25.38.255 scope global eth0
inet 172.25.38.100/32 scope global eth0
inet6 fe80::5054:ff:fe58:4d1f/64 scope link
valid_lft forever preferred_lft forever
呼叫指令碼控制高可用
[[email protected] files]# cd /opt/
[[email protected] opt]# ls
[[email protected] opt]# vim check_haproxy.sh
[[email protected] opt]# cat check_haproxy.sh 指令碼意思就是當haproxy服務出現故障時就停止keepalived
#!/bin/bash
/etc/init.d/haproxy status &> /dev/null || /etc/init.d/haproxy restart &> /dev/null
if [ $? -ne 0 ];then
/etc/init.d/keepalived stop &> /dev/null
fi
[[email protected] opt]# chmod +x check_haproxy.sh 給指令碼賦予執行許可權
[[email protected] opt]# /etc/init.d/haproxy status
haproxy (pid 2877) is running...
[[email protected] opt]# echo $?
0
[[email protected] opt]# cd /etc/keepalived/
[[email protected] keepalived]# ls
keepalived.conf
[[email protected] opt]#scp check_haproxy.sh server4:/opt/ 將指令碼傳遞到server4,必須新增可執行許可權不然指令碼沒辦法執行
[[email protected] keepalived]# vim /srv/salt/keepalived/files/keepalived.conf
[[email protected] keepalived]# cat /srv/salt/keepalived/files/keepalived.conf
! Configuration File for keepalived
vrrp_script check_haproxy {
script "/opt/check_haproxy.sh" 同樣新增指令碼的呼叫
interval 2
weight 2
}
global_defs {
notification_email {
[email protected]
}
notification_email_from [email protected]
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_DEVEL
vrrp_skip_check_adv_addr
# vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_instance VI_1 {
state {{ STATE }}
interface eth0
virtual_router_id {{ VRID }}
priority {{ PRIORITY }}
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.25.254.100
}
track_script{
check_haproxy
}
}
進行推送測試:
[[email protected] keepalived]# salt server4 state.sls keepalived.service
在server1減去haproxy指令碼的執行許可權,指令碼生效相應的keepalived服務會關閉:
[[email protected] ~]# cd /etc/init.d/
[[email protected] init.d]# ls
auditd halt keepalived netconsole rdisc rsyslog saslauthd
blk-availability haproxy killall netfs restorecond salt-master single
crond ip6tables lvm2-lvmetad network rhnsd salt-minion sshd
functions iptables lvm2-monitor postfix rhsmcertd sandbox udev-post
[[email protected] init.d]# chmod -x haproxy
[[email protected] init.d]# /etc/init.d/keepalived status
keepalived is stopped
MASTER會轉換到serevr4,直接的效果就是server4接管虛擬IP
[[email protected] opt]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 52:54:00:52:c1:00 brd ff:ff:ff:ff:ff:ff
inet 172.25..254.4/24 brd 172.25.254.255 scope global eth0
inet 172.25.254.100/32 scope global eth0
inet6 fe80::5054:ff:fe52:c100/64 scope link
valid_lft forever preferred_lft forever
驗證高可用
[[email protected] init.d]# ls
auditd halt keepalived netconsole rdisc rsyslog saslauthd
blk-availability haproxy killall netfs restorecond salt-master single
crond ip6tables lvm2-lvmetad network rhnsd salt-minion sshd
functions iptables lvm2-monitor postfix rhsmcertd sandbox udev-post
[[email protected] init.d]# chmod +x haproxy 重新增加可執行許可權
[[email protected] init.d]# ll haproxy
-rwxr-xr-x 1 root root 2298 Jul 10 2013 haproxy
[[email protected] init.d]# /etc/init.d/keepalived status
keepalived is stopped
[[email protected] init.d]# /etc/init.d/keepalived start 開啟keepalived服務
Starting keepalived: [ OK ]
[[email protected] init.d]# ip addr server1依舊會搶回來MASTER的角色進行接管虛擬IP
[[email protected] init.d]# salt server1 service.start keepalived 開啟keepalived服務
server1:
True
[[email protected] init.d]# ls
auditd halt keepalived netconsole rdisc rsyslog saslauthd
blk-availability haproxy killall netfs restorecond salt-master single
crond ip6tables lvm2-lvmetad network rhnsd salt-minion sshd
functions iptables lvm2-monitor postfix rhsmcertd sandbox udev-post
[[email protected] init.d]# cd
[[email protected] ~]# salt-cp '*' /etc/passwd /tmp 將/etc/passwd檔案同步到所有主機的/tmp目錄下
在server2檢視已經推送成功,拿server2舉例子,不一一進行檢視後端:
[[email protected] ~]# cd /tmp/
[[email protected] tmp]# ls
passwd yum.log yum_save_tx-2018-08-17-09-30GSgtBm.yumtx
[[email protected] ~]# salt '*' cmd.run 'rm -f /tmp/passwd' 呼叫salt命令刪除傳遞過去的passwd檔案
server4:
server2:
server3:
server1:
[[email protected] ~]# cd /tmp/ 在server1檢視已經被刪除
[[email protected] tmp]# ls
yum.log
在server2檢視passwd檔案已經被刪除:
[[email protected] tmp]# ls
yum.log yum_save_tx-2018-08-17-09-30GSgtBm.yumtx
[[email protected] tmp]# salt server3 state.single pkg.installed httpd 直接呼叫模組命令安裝apache
在serevr3檢視httpd已經被安裝:
[[email protected] ~]# rpm -q httpd
httpd-2.2.15-29.el6_4.x86_64