PXC5.7(Percona XtraDB Cluster)+HAproxy集群部署
阿新 • • 發佈：2018-04-18
PXChaproxymysqlPXC是Percona XtraDB Cluster的縮寫，是一種具有高可用性和高擴展性的MySQL開源集群。它集成了Percona Server和Percona XtraBackup，同時采用了Codership Galera庫。
優點：
1.準同步復制
2.多個可同時讀寫節點，可實現寫擴展，較分片方案更進一步
3.自動節點管理
4.數據嚴格一致
5.服務高可用
缺點：
1.只支持innodb引擎
2.所有表都要有主鍵
3.所有的寫操作都將發生在所有節點上，存在寫擴大問題
4.加入新節點，開銷大。需要復制完整的數據。
部署準備工作
| 主機名 | IP |
| pxc-linux-28 | 192.168.253.28 |
| pxc-linux-29 | 192.168.253.29 |
| pxc-linux-30 | 192.168.253.30 |
| haproxy-linux-31 | 192.168.253.31 |
三個PXC節點分別配置防火墻
[root@pxc-linux-28 ~]# firewall-cmd --add-port=3306/tcp --permanent [root@pxc-linux-28 ~]# firewall-cmd --add-port=4567/tcp --permanent [root@pxc-linux-28 ~]# firewall-cmd --add-port=4568/tcp --permanent [root@pxc-linux-28 ~]# firewall-cmd --add-port=4444/tcp --permanent [root@pxc-linux-28 ~]# firewall-cmd --reload
安裝相關yum源
rpm -Uvh https://www.percona.com/downloads/percona-release/redhat/latest/percona-release-0.1-4.noarch.rpm
安裝pxc
yum install Percona-XtraDB-Cluster-57 -y
分別在三個節點創建數據存儲目錄
mkdir -p /data/local/percona-xtradb-cluster/data chown -R mysql:mysql /data/local/percona-xtradb-cluster/data mkdir -p /data/local/percona-xtradb-cluster/run chown -R mysql:mysql /data/local/percona-xtradb-cluster/run mkdir -p /data/logs/mysql chown -R mysql:mysql /data/logs/mysql
修改/etc/my.cnf配置文件，其他兩臺節點需要修改server_id和wsrep_node_address
pxc-linux-28:
#cat /etc/my.cnf [client] port = 3306 socket = /data/local/percona-xtradb-cluster/run/mysql.sock default-character-set = utf8mb4 [mysqld] user = mysql innodb_buffer_pool_size = 1024M character_set_server = utf8mb4 datadir = /data/local/percona-xtradb-cluster/data port = 3306 server_id = 28 socket = /data/local/percona-xtradb-cluster/run/mysql.sock pid-file = /data/local/percona-xtradb-cluster/run/mysql.pid log-error = /data/logs/mysql/error.log log_warnings = 2 slow_query_log_file = /data/logs/mysql/slow.log long_query_time = 2 log_timestamps=SYSTEM lower_case_table_names = 1 key_buffer_size = 1344M event_scheduler=ON sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES wsrep_provider=/usr/lib64/libgalera_smm.so wsrep_cluster_address=gcomm://192.168.253.28,192.168.253.29,192.168.253.30 binlog_format=ROW pxc_strict_mode=PERMISSIVE max_connect_errors=1000 max_allowed_packet = 1024M default_storage_engine=InnoDB #Innodb innodb_flush_method = O_DIRECT innodb_log_files_in_group = 5 innodb_lock_wait_timeout = 50 innodb_log_file_size = 1024M innodb_flush_log_at_trx_commit = 1 innodb_file_per_table = 1 innodb_thread_concurrency = 8 innodb_buffer_pool_size = 5G innodb_read_io_threads = 24 innodb_write_io_threads = 24 log_bin_trust_function_creators=1 innodb_autoinc_lock_mode=2 innodb_locks_unsafe_for_binlog=1 # CACHES AND LIMITS # tmp_table_size = 32M max_heap_table_size = 32M max_connections = 1000 thread_cache_size = 50 open_files_limit = 65535 table_definition_cache = 4096 table_open_cache = 5000 #wsrep wsrep_retry_autocommit=1 wsrep_auto_increment_control=1 wsrep_node_name = pxc-linux-28 wsrep_node_address=192.168.253.28 wsrep_sst_method=xtrabackup-v2 wsrep_cluster_name=test-pxc wsrep_sst_auth="sync_rep:abc123"
pxc-linux-29:
[client] port = 3306 socket = /data/local/percona-xtradb-cluster/run/mysql.sock default-character-set = utf8mb4 [mysqld] user = mysql innodb_buffer_pool_size = 1024M character_set_server = utf8mb4 datadir = /data/local/percona-xtradb-cluster/data port = 3306 server_id = 29 socket = /data/local/percona-xtradb-cluster/run/mysql.sock pid-file = /data/local/percona-xtradb-cluster/run/mysql.pid log-error = /data/logs/mysql/error.log log_warnings = 2 slow_query_log_file = /data/logs/mysql/slow.log long_query_time = 2 log_timestamps=SYSTEM log_bin_trust_function_creators=1 lower_case_table_names=1 key_buffer_size = 1344M event_scheduler=ON sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES wsrep_provider=/usr/lib64/libgalera_smm.so wsrep_cluster_address=gcomm://192.168.253.28,192.168.253.29,192.168.253.30 binlog_format=ROW pxc_strict_mode=PERMISSIVE max_connect_errors=1000 max_allowed_packet = 1024M default_storage_engine=InnoDB #Innodb innodb_flush_method = O_DIRECT innodb_log_files_in_group = 5 innodb_lock_wait_timeout = 50 innodb_log_file_size = 1024M innodb_flush_log_at_trx_commit = 1 innodb_file_per_table = 1 innodb_thread_concurrency = 8 innodb_buffer_pool_size = 5G innodb_read_io_threads = 24 innodb_write_io_threads = 24 log_bin_trust_function_creators=1 innodb_autoinc_lock_mode=2 innodb_locks_unsafe_for_binlog=1 # CACHES AND LIMITS # tmp_table_size = 32M max_heap_table_size = 32M max_connections = 1000 thread_cache_size = 50 open_files_limit = 65535 table_definition_cache = 4096 table_open_cache = 5000 #wsrep wsrep_retry_autocommit=1 wsrep_auto_increment_control=1 wsrep_node_name =pxc-linux-29 wsrep_node_address=192.168.253.29 wsrep_sst_method=xtrabackup-v2 wsrep_cluster_name=test-pxc wsrep_sst_auth="sync_rep:abc123"
pxc-linux-30:
[client] port = 3306 socket = /data/local/percona-xtradb-cluster/run/mysql.sock default-character-set = utf8mb4 [mysqld] user = mysql innodb_buffer_pool_size = 1024M character_set_server = utf8mb4 datadir = /data/local/percona-xtradb-cluster/data port = 3306 server_id = 30 socket = /data/local/percona-xtradb-cluster/run/mysql.sock pid-file = /data/local/percona-xtradb-cluster/run/mysql.pid log-error = /data/logs/mysql/error.log log_warnings = 2 slow_query_log_file = /data/logs/mysql/slow.log long_query_time = 2 log_timestamps=SYSTEM lower_case_table_names = 1 key_buffer_size = 1344M event_scheduler=ON sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES wsrep_provider=/usr/lib64/libgalera_smm.so wsrep_cluster_address=gcomm://192.168.253.28,192.168.253.29,192.168.253.30 binlog_format=ROW pxc_strict_mode=PERMISSIVE max_connect_errors=1000 max_allowed_packet = 1024M default_storage_engine=InnoDB #Innodb innodb_flush_method = O_DIRECT innodb_log_files_in_group = 5 innodb_lock_wait_timeout = 50 innodb_log_file_size = 1024M innodb_flush_log_at_trx_commit = 1 innodb_file_per_table = 1 innodb_thread_concurrency = 8 innodb_buffer_pool_size = 5G innodb_read_io_threads = 24 innodb_write_io_threads = 24 log_bin_trust_function_creators=1 innodb_autoinc_lock_mode=2 innodb_locks_unsafe_for_binlog=1 # CACHES AND LIMITS # tmp_table_size = 32M max_heap_table_size = 32M max_connections = 1000 thread_cache_size = 50 open_files_limit = 65535 table_definition_cache = 4096 table_open_cache = 5000 #wsrep wsrep_retry_autocommit=1 wsrep_auto_increment_control=1 wsrep_node_name = pxc-linux-30 wsrep_node_address=192.168.253.30 wsrep_sst_method=xtrabackup-v2 wsrep_cluster_name=test-pxc wsrep_sst_auth="sync_rep:abc123"
pxc-linux-28啟動
systemctl start mysql@bootstrape
備註：謹記，只要是啟動集群的第一個Node（首次搭建集群或者集群全部關閉），都要用此命令
pxc-linux-29和pxc-linux-30啟動
systemctl start mysql
備註：謹記，只要集群有一個Node啟動，其余節點都是用此命令
修改密碼
mysql5.7版本日誌均在error.log 裏面生成
grep "temporary password" /data/logs/mysql/error.log
使用改密碼登陸MySQL，修改成自己想要的密碼
mysql> alter user 'root'@'localhost' idnetified by '123456';
在pxc-linux-28MySQL中創建SST同步用戶
mysql> GRANT PROCESS,RELOAD,LOCK TABLES,REPLICATION CLIENT ON *.* TO 'sync_rep'@'192.168.%.%' IDENTIFIED BY 'abc123'; mysql> flush privileges;
State Snapshot Transfer 是可以讓one node (donor) 通過數據拷貝升級至joining node (joiner)。
一個新節點加入至集群中，需要同步數據，那麽SST可以傳輸數據，讓新節點成為集群的一部分。
SST的三個有效方法：mysqldump, rsync andxtrabackup。不過多解釋，目前最好用的是xtrabackup
查看集群狀態
集群驗證
[root@pxc-linux-28 ~]# mysql -uroot -p mysql> show variables like 'version'; +---------------+--------------+ | Variable_name | Value | +---------------+--------------+ | version | 5.7.19-17-57 | +---------------+--------------+ 1 row in set (0.00 sec) mysql> create database pxcdb; mysql> use pxcdb; mysql> create table t1(id tinyint,ename varchar(20)); mysql> insert into t1 values(1,'Leshami'); 在29上進行驗證 [root@pxc-linux-29 ~]# mysql -uroot -p mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | pxcdb | | sys | +--------------------+ 5 rows in set (0.00 sec) mysql> show variables like 'server_id'; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | server_id | 29 | +---------------+-------+ 1 row in set (0.01 sec) mysql> select * from pxcdb.t1; +------+---------+ | id | ename | +------+---------+ | 1 | Leshami | +------+---------+ 1 row in set (0.00 sec) --查看集群狀態 mysql> show status like '%wsrep_clust%'; +--------------------------+--------------------------------------+ | Variable_name | Value | +--------------------------+--------------------------------------+ | wsrep_cluster_conf_id | 13 | | wsrep_cluster_size | 3 | | wsrep_cluster_state_uuid | aeb87793-ebb2-11e7-b33e-eeaf4988bbe4 | | wsrep_cluster_status | Primary | +--------------------------+--------------------------------------+ 4 rows in set (0.00 sec) mysql> show status like 'wsrep_connected'; +-----------------+-------+ | Variable_name | Value | +-----------------+-------+ | wsrep_connected | ON | +-----------------+-------+ 1 row in set (0.00 sec)
openssl版本過低導致的錯誤
2017-12-28T09:23:19.605353Z 0 [ERROR] WSREP: wsrep_load(): dlopen(): /usr/lib64/galera3/libgalera_smm.so: symbol SSL_COMP_free_compression_methods, version libssl.so.10 not defined in file libssl.so.10 with link time reference 2017-12-28T09:23:19.605379Z 0 [ERROR] WSREP: Failed to load wsrep_provider (/usr/lib64/galera3/libgalera_smm.so). Error: Invalid argument (code: 22). Reverting to no provider. 2017-12-28T09:23:19.605386Z 0 [Note] WSREP: Setting wsrep_ready to false [root@pxc-linux-29 ~]# rpm -qa|grep openssl openssl-1.0.1e-42.el7.9.x86_64 openssl-libs-1.0.1e-42.el7.9.x86_64 [root@pxc-linux-29 ~]# yum update openssl -y
需要升級openssl，建議全部升級後再啟動集群，這問題在CentOS 7.4.1708不存在即openssl版本較新
Haproxy負載均衡
Haproxy是一個反向代理負載均衡解決方案，支持4層和7層模式，提供後端服務器健康檢查，非常穩定。淘寶前期也使用Haproxy作為CDN系統負載均衡器。
安裝haproxy
yum -y install haproxy
簡單配置文件：
cat /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Example configuration for a possible web application. See the
# full configuration options online.
#
# http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend pxc-front
bind *:3306
mode tcp
default_backend pxc-back
frontend statas-front
bind *:8088
mode http
default_backend stats-back
#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
backend pxc-back
mode tcp
balance leastconn
option httpchk
server pxc-linux-28 192.168.253.28:3306 check port 9200 inter 12000 rise 3 fall 3
server pxc-linux-29 192.168.253.29:3306 check port 9200 inter 12000 rise 3 fall 3
server pxc-linux-30 192.168.253.30:3306 check port 9200 inter 12000 rise 3 fall 3
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend stats-back
mode http
balance roundrobin
stats uri /haproxy/stats
stats auth pxcstats:xxwzopop在cluster的MySQL上創建用戶
(一個節點創建，會被復制到其它節點)
監控用帳號：
grant usage on *.* to 'pxc-monitor'@'%' identified by 'testpxc';
服務測試帳號：
grant all privileges on *.* to 'zxw'@'%' identified by 'xxwzopop';
更改用戶名和密碼（三個節點都得修改）
#vim /usr/bin/clustercheck MYSQL_USERNAME="pxc-monitor" MYSQL_PASSWORD="testpxc"
測試
[root@pxc-linux-28 ~]# clustercheck HTTP/1.1 200 OK Content-Type: text/plain Connection: close Content-Length: 40 Percona XtraDB Cluster Node is synced.
更改/etc/services添加mysqlchk的端口號：
mysqlchk 9200/tcp #mysqlchk
配置xinetd
vi /etc/xinetd.d/mysqlchk
該文件保持默認即可
測試
#for i in `seq 1 1000`;do mysql -h 192.168.253.31 -P3306 -upxc -pabc123 -e "select @@hostname;";done mysql: [Warning] Using a password on the command line interface can be insecure. +--------------+ | @@hostname | +--------------+ | pxc-linux-30 | +--------------+ mysql: [Warning] Using a password on the command line interface can be insecure. +--------------+ | @@hostname | +--------------+ | pxc-linux-28 | +--------------+ mysql: [Warning] Using a password on the command line interface can be insecure. +--------------+ | @@hostname | +--------------+ | pxc-linux-29 | +--------------+
可以看到負載了
查看haproxy狀態
http://192.168.253.31:8088/haproxy/stats
帳號密碼是：pxcstats:xxwzopop
/etc/haproxy/haproxy.cfg 指定的。
PXC5.7(Percona XtraDB Cluster)+HAproxy集群部署