4、keepalived高可用nginx負載均衡
keepalived：
HTTP_GET //使用keepalived獲取後端real server健康狀態檢測
SSL_GET(https) //這裡以為這後端使用的是http協議
TCP_CHECK
下面演示基於TCP_CHECK做檢測
# man keepalived //檢視TCP_CHECK配置段
# TCP healthchecker
TCP_CHECK
{
# ======== generic connection options
# Optional IP address to connect to.
# The default is the realserver IP //預設使用real server的IP
connect_ip <IP ADDRESS> //可省略
# Optional port to connect to
# The default is the realserver port
connect_port <PORT> //可省略
# Optional interface to use to
# originate the connection
bindto <IP ADDRESS>
# Optional source port to
# originate the connection from
bind_port <PORT>
# Optional connection timeout in seconds.
# The default is 5 seconds
connect_timeout <INTEGER>
# Optional fwmark to mark all outgoing
# checker packets with
fwmark <INTEGER>
# Optional random delay to start the initial check
# for maximum N seconds.
# Useful to scatter multiple simultaneous
# checks to the same RS. Enabled by default, with
# the maximum at delay_loop. Specify 0 to disable
warmup <INT>
# Retry count to make additional checks if check
# of an alive server fails. Default: 1
retry <INT>
# Delay in seconds before retrying. Default: 1
delay_before_retry <INT>
} #TCP_CHECK
# cd /etc/keepalived
# vim keepalived.conf //兩臺keepalived都要設定
1 virtual_server 192.168.184.150 80 { //這裡可以合併 2 delay_loop 6 3 lb_algo wrr 4 lb_kind DR 5 net_mask 255.255.0.0 6 protocol TCP 7 sorry_server 127.0.0.1 80 8 9 real_server 192.168.184.143 80 { 10 weight 1 11 TCP_CHECK { 12 connect_timeout 3 13 } 14 } 15 16 real_server 192.168.184.144 80 { 17 weight 2 18 TCP_CHECK { 19 connect_timeout 3 20 } 21 } 22 }
systemctl restart keepalived
# systemctl status keepalived
1 ● keepalived.service - LVS and VRRP High Availability Monitor 2 Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled) 3 Active: active (running) since Thu 2018-12-13 23:11:06 CST; 1min 32s ago 4 Process: 6233 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS) 5 Main PID: 6234 (keepalived) 6 CGroup: /system.slice/keepalived.service 7 ├─6234 /usr/sbin/keepalived -D 8 ├─6235 /usr/sbin/keepalived -D 9 └─6236 /usr/sbin/keepalived -D 10 11 Dec 13 23:11:11 node1 Keepalived_healthcheckers[6235]: Check on service [192.168.184.144]:80 failed after 1 retry. 12 Dec 13 23:11:11 node1 Keepalived_healthcheckers[6235]: Removing service [192.168.184.144]:80 from VS [192.168.184.150]:80 13 Dec 13 23:11:11 node1 Keepalived_healthcheckers[6235]: Remote SMTP server [127.0.0.1]:25 connected. 14 Dec 13 23:11:11 node1 Keepalived_healthcheckers[6235]: SMTP alert successfully sent. 15 Dec 13 23:11:14 node1 Keepalived_vrrp[6236]: Sending gratuitous ARP on eth0 for 192.168.184.150 16 Dec 13 23:11:14 node1 Keepalived_vrrp[6236]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth0 for 192.168.184.150 17 Dec 13 23:11:14 node1 Keepalived_vrrp[6236]: Sending gratuitous ARP on eth0 for 192.168.184.150 18 Dec 13 23:11:14 node1 Keepalived_vrrp[6236]: Sending gratuitous ARP on eth0 for 192.168.184.150 19 Dec 13 23:11:14 node1 Keepalived_vrrp[6236]: Sending gratuitous ARP on eth0 for 192.168.184.150 20 Dec 13 23:11:14 node1 Keepalived_vrrp[6236]: Sending gratuitous ARP on eth0 for 192.168.184.150 //傳送廣播地址已經新增 21 You have new mail in /var/spool/mail/root
示例：
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
TCP_CHECK {
connect_timeout 3
}
HA Services:
nginx
100: -25
96: -20 79 --> 99 --> 79
部落格作業：
keepalived 高可用 ipvs
nginx
active/active
Linux HA Cluster
LB, HA, HP, hadoop
LB:
傳輸層：lvs
應用層：nginx, haproxy, httpd, perlbal, ats, varnish
HA:
vrrp: keepalived
AIS: heartbeat, OpenAIS, corosync/pacemaker, cman/rgmanager(conga) RHCS
HA:
故障場景：
硬體故障：
設計缺陷
使用過久自然損壞
人為故障
…… ……
軟體故障
設計缺陷
bug
人為誤操作
……
A=MTBF/(MTBF+MTTR)
MTBF: Mean Time Between Failure
MTTR: Mean Time To Repair
0<A<1: 百分比
90%, 95%, 99%
99.9%, 99.99%, 99.999%
提供冗餘：
network partition： vote system
隔離：
STONITH：shoot the other node on the head 節點級別隔離
Fence: 資源級別的隔離
failover domain：
fda: node1, node5
fdb: node2, node5
fdc: node3, node5
fdd: node4, node5
資源的約束性：
位置約束：資源對節點的傾向性；
排列約束：資源彼此間是否能運行於同一節點的傾向性；
順序約束：多個資源啟動順序依賴關係；
vote system:
少數服從多數：quorum
> total/2
with quorum: 擁有法定票數
without quorum: 不擁有法定票數
兩個節點(偶數個節點)：
Ping node
qdisk
failover
failback
Messaging Layer:
heartbeat
v1
v2
v3
corosync
cman
Cluster Resource Manager(CRM):
heartbeat v1 haresources (配置介面：配置檔案haresources)
heartbeat v2 crm (在每個節點執行一個crmd(5560/tcp)守護程序，有命令列介面crmsh; GUI: hb_gui)
heartbeat v3, pacemaker (配置介面：crmsh, pcs; GUI: hawk(suse), LCMC, pacemaker-gui)
rgmanager (配置介面：cluster.conf, system-config-cluster, conga(webgui), cman_tool, clustat)
組合方式：
heartbeat v1 (haresources)
heartbeat v2 (crm)
heartbeat v3 + pacemaker
corosync + pacemaker
corosync v1 + pacemaker (plugin)
corosync v2 + pacemaker (standalone service)
cman + rgmanager
corosync v1 + cman + pacemaker
RHCS: Red Hat Cluster Suite
RHEL5: cman + rgmanager + conga (ricci/luci)
RHEL6: cman + rgmanager + conga (ricci/luci)
corosync + pacemaker
corosync + cman + pacemaker
RHEL7: corosync + pacemaker
Resource Agent：
service: /etc/ha.d/haresources.d/目錄下的指令碼；
LSB: /etc/rc.d/init.d/目錄下的指令碼；
OCF：Open Cluster Framework
provider:
STONITH:
Systemd:
資源型別：
primitive：主資源，原始資源；在叢集中只能執行一個例項；
clone：克隆資源，在叢集中可執行多個例項；
匿名克隆、全域性惟一克隆、狀態克隆(主動、被動)
multi-state(master/slave)：克隆資源的特殊實現；多狀態資源；
group: 組資源；
啟動或停止；
資源監視
相關性：
資源屬性：
priority: 優先順序；
target-role：started, stopped, master;
is-managed: 是否允許叢集管理此資源；
resource-stickiness: 資源粘性；
allow-migrate: 是否允許遷移；
約束：score
位置約束：資源對節點的傾向性；
(-oo, +oo):
任何值+無窮大=無窮大
任何值+負無窮大=負無窮大
無窮大+負無窮大=負無窮大
排列約束：資源彼此間是否能運行於同一節點的傾向性；
(-oo, +oo)
順序約束：多個資源啟動順序依賴關係；
(-oo, +oo)
Mandatory
安裝配置：
CentOS 7： corosync v2 + pacemaker
corosync v2: vote system
pacemaker: 獨立服務
叢集的全生命週期管理工具：
pcs: agent(pcsd)
crmsh: agentless (pssh)
配置叢集的前提：
(1) 時間同步；
(2) 基於當前正在使用的主機名互相訪問；
(3) 是否會用到仲裁裝置；
web serivce：
vip: 172.16.100.91
httpd
回顧：AIS HA
Messaging Layer:
heartbeat v1, v2, v3
corosync v1, v2(votequorum)
OpenAIS
CRM:
pacemaker
配置介面：crmsh (agentless), pssh
pcs (agent), pcsd
conga(ricci/luci)
group, constraint
rgmanager(cman)
resource group:
failover domain
配置：
全域性屬性：property, stonith-enable等等；
高可用服務：資源，通過RA
RA:
LSB: /etc/rc.d/init.d/
systemd：/etc/systemd/system/multi-user.wants
處於enable狀態的服務；
OCF: [provider]
heartbeat
pacemaker
linbit
service
stonith
高可用叢集的可用方案：
heartbeat v1
heartbeat v2
heartbeat v3 + pacemaker X
corosync + pacemaker
cman + rgmanager
corosync + cman + pacemaker
corosync + pacemaker
keepalived
HA Cluster(2)
Heartbeat資訊傳遞：
Unicast, udpu
Mutlicast, udp
Broadcast
組播地址：用於標識一個IP組播域；IANA把D類地址留給組播使用：224.0.0.0-239.255.255.255
永久組播地址：224.0.0.0-224.0.0.255
臨時組播地址：224.0.1.0-238.255.255.255
本地組播地址：239.0.0.0-239.255.255.255
示例配置檔案：
totem {
version: 2
crypto_cipher: aes128
crypto_hash: sha1
secauth: on
interface {
ringnumber: 0
bindnetaddr: 172.16.0.0
mcastaddr: 239.185.1.31
mcastport: 5405
ttl: 1
}
}
nodelist {
node {
ring0_addr: 172.16.100.67
nodeid: 1
}
node {
ring0_addr: 172.16.100.68
nodeid: 2
}
node {
ring0_addr: 172.16.100.69
nodeid: 3
}
}
logging {
fileline: off
to_stderr: no
to_logfile: yes
logfile: /var/log/cluster/corosync.log
to_syslog: no
debug: off
timestamp: on
logger_subsys {
subsys: QUORUM
debug: off
}
}
quorum {
provider: corosync_votequorum
}
HA Web Service:
vip: 172.16.100.92, ocf:heartbeat:IPaddr
httpd: systemd
nfs shared storage: ocf:heartbeat:Filesystem
HA Cluster工作模型:
A/P：兩節點叢集; active/passive;
without-quorum-policy={stop|ignore|suicide|freeze}
A/A：雙主模型
N-M: N個節點，M個服務，N>M;
N-N: N個節點，N個服務；
network partition:
brain-split：塊級別的共享儲存時，非常危險；
vote quorum:
with quorum > total/2
without quorum <= total/2
stop
ignore
suicide
freeze
CAP:
C: consistency
A: availiability
P: partition tolerance
webip, webstore, webserver
node1: 100 + 0 + 0
node2: 0 + 0 + 0
node3: 0 + 0 + 0
node2: 50+50+50
A --> B --> C
C --> B --> A
pcs：
cluster
auth
setup
resource
describe
list
create
delete
constraint
colocation
order
location
property
list
set
status
config
部落格作業：
(1) 手動配置，多播：corosync+pacemaker+crmsh, 配置高可用的mysql叢集，datadir指向的路徑為nfs匯出路徑；
(2) pcs/pcsd，單播：corosync+pacemaker， 配置高可用的web叢集；
單播配置示例：
某些環境中可能不支援組播。這時應該配置 Corosync 使用單播，下面是使用單播的 Corosync 配置檔案的一部分：
totem {
#...
interface {
ringnumber: 0
bindnetaddr: 192.168.42.0
broadcast: yes
mcastport: 5405
}
interface {
ringnumber: 1
bindnetaddr: 10.0.42.0
broadcast: yes
mcastport: 5405
}
transport: udpu
}
nodelist {
node {
ring0_addr: 192.168.42.1
ring1_addr: 10.0.42.1
nodeid: 1
}
node {
ring0_addr: 192.168.42.2
ring1_addr: 10.0.42.2
nodeid: 2
}
}
如果將 broadcast 設定為 yes ，叢集心跳將通過廣播實現。設定該引數時，不能設定 mcastaddr 。
transport 配置項決定叢集通訊方式。要完全禁用組播，應該配置單播傳輸引數 udpu 。這要求將所有的節點伺服器資訊寫入 nodelist ，也就是需要在配署 HA 叢集之前確定節點組成。配認配置是 udp 。通訊方式型別還支援 udpu 和 iba 。
在 nodelist 之下可以為某一節點設定只與該節點相關的資訊，這些設定項只能包含在 node 之中，即只能對屬於叢集的節點伺服器進行設定，而且只應包括那些與預設設定不同的引數。每臺伺服器都必須配置 ring0_addr 。