MySQL-Cluster三機叢集+HA高可用+負載均衡配置手冊
阿新 • • 發佈：2018-12-25
本文中的配置已經在實驗室進行過效能測試、可靠性測試驗證。
一、介紹
這篇文件旨在介紹如何安裝配置基於3臺伺服器的MySQL叢集。並且實現任意一臺伺服器出現問題或宕機時MySQL叢集依然能夠繼續執行。
1. MySQL-Cluster簡介
MySQL-Cluster主要有三種類型節點：
NDB node，資料節點將儲存資料庫，它會自動複製所有的資料節點；
SQL node，守護節點相當於是SQL資料庫和客戶端之間的介面，它提供從資料節點查詢等操作，類似於"閘道器"；
Management node，管理節點，用以監控和管理整個叢集。
-------------------------------此處插圖----------------------------
2.安裝環境及軟體包
3臺華為RH2285伺服器——其中兩臺（16顆CPU、96G記憶體、500G磁碟陣列）作為NDB節點和SQL節點伺服器，另一臺伺服器16顆CPU、32G記憶體作為管理節點
軟體包：MySQL-Cluster-client-gpl-7.2.13-1.rhel5.x86_64.rpm
MySQL-Cluster-server-gpl-7.2.13-1.rhel5.x86_64.rpm
MySQL-Cluster-devel-gpl-7.2.13-1.rhel5.x86_64.rpm
MySQL-Cluster-embedded-gpl-7.2.13-1.rhel5.x86_64.rpm
MySQL-Cluster-shared-compat-gpl-7.2.13-1.rhel5.x86_64.rpm
MySQL-Cluster-shared-gpl-7.2.13-1.rhel5.x86_64.rpm
作業系統：Redhat.Enterprise.Linux.v5.UPDATE.7.X86_64.DVD-HOTiSO.iso
伺服器hostname和IP地址:
db1 172.16.1.74
db2 172.16.1.75
demo 172.16.1.78
資料庫規劃：
資料庫管理節點（ndb_mgmd）：172.16.1.78
資料庫資料節點（NDB）：172.16.1.74，172.16.1.75
資料庫SQL節點(API節點)：172.16.1.74，172.16.1.75
二、在db1和db2上安裝MySQL-cluster
以下步驟需要在db1和db2上各做一次
1.首先檢查linux作業系統中是否安裝了自帶的mysql資料庫
一般linux自帶mysql版本都比較低一些，因此要安裝比較新的版本需要先將自帶的低版本解除安裝。
檢查作業系統中是否安裝了mysql的命令：
rpm -qa |grep -i mysql
解除安裝mysql的命令：rpm -e mysqlXXXX
或者使用yum -remove mysqlXXX來刪除，這樣可以避免解除安裝過程中由於各個包之間的依賴造成解除安裝停止
2.建立mysql使用者和組
groupadd mysql
useradd -g mysql mysql
usermod -d /home/mysql mysql
3.將MySQL-Cluster上傳到/home/mysql/目錄下，開始安裝MySQL-Cluster
[
Preparing... ########################################### [100%]
1:MySQL-Cluster-server-gp########################################### [100%]
PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following commands:
/usr/bin/mysqladmin -u root password 'new-password'
/usr/bin/mysqladmin -u root -h db1 password 'new-password'
Alternatively you can run:
/usr/bin/mysql_secure_installation
which will also give you the option of removing the test
databases and anonymous user created by default. This is
strongly recommended for production servers.
See the manual for more instructions.
Please report any problems with the /usr/bin/mysqlbug script!
[
Preparing... ########################################### [100%]
1:MySQL-Cluster-client-gp ########################################### [100%]
依次完成MySQL-Cluster-devel-gpl-7.2.13-1.rhel5.x86_64.rpm
MySQL-Cluster-embedded-gpl-7.2.13-1.rhel5.x86_64.rpm
MySQL-Cluster-shared-compat-gpl-7.2.13-1.rhel5.x86_64.rpm
MySQL-Cluster-shared-gpl-7.2.13-1.rhel5.x86_64.rpm
的安裝
4.安裝完成之後建立連線
ln -s /usr/sbin/ndbd /usr/bin
ln -s /usr/sbin/ndb_mgmd /usr/bin
ln -s /usr/sbin/mysqld /usr/bin
#ln -s /usr/local/lib/mysqlmanager /usr/bin ---這個在7.2.13版本上不需要
管理節點伺服器demo（172.16.1.78）上只安裝MySQL-Cluster-server-gpl-7.2.13-1.rhel5.x86_64.rpm即可
三、安裝並配置節點
1.配置管理節點（172.16.1.78）配置檔案
# mkdir /var/lib/mysql-cluster
# cd /var/lib/mysql-cluster
# vi config.ini
在config.ini中新增如下內容：
[ndbd default]
NoOfReplicas= 2
MaxNoOfConcurrentOperations= 500000
MaxNoOfLocalOperations=500000
MaxNoOfConcurrentTransactions=500000
MaxNoOfConcurrentIndexOperations=50000
# Amount of memory required=(SizeofDatabase * NumberOfReplicas * 1.1 ) / NumberOfDataNodes
DataMemory= 49152M
IndexMemory= 6144M
TimeBetweenWatchDogCheck= 30000
DataDir=/var/lib/mysql/data
MaxNoOfAttributes=500000
MaxNoOfTables=1000
MaxNoOfOrderedIndexes= 512
#StartPartialTimeout=100
#StartPartitionedTimeout=100
#ArbitrationTimeout=5000
#MaxNoOfFiredTriggers=8000
#TransactionBufferMemory=5M
TransactionDeadlockDetectionTimeout=90000
HeartbeatIntervalDbDb=5000
#StopOnError=1
[ndb_mgmd default]
DataDir=/var/lib/mysql/data
[ndb_mgmd]
NodeId=1
HostName= 172.16.1.78
[ndbd]
NodeId= 2
HostName= 172.16.1.74
[ndbd]
NodeId= 3
HostName= 172.16.1.75
[mysqld]
ArbitrationRank=2
[mysqld]
ArbitrationRank=2
[mysqld]
[mysqld]
[mysqld]
[mysqld]
[tcp default]
portnumber=2279
以下步驟需要在db1和db2上各做一次
2.配置通用my.cnf檔案（SQL節點及ndb節點均使用此檔案）
# vi /etc/my.cnf
在my.cnf中新增如下內容：
[mysqld]
datadir=/var/lib/mysql/data/
socket=/var/lib/mysql/mysql.sock
lower_case_table_names=1
ndb_cluster_connection_pool=2
tmp_table_size=512M
max_heap_table_size=512M
max_connections =400
thread_stack=1M
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=1
default-storage-engine=ndbcluster
ndbcluster
ndb-connectstring=172.16.1.78
[ndbd]
connect-string=172.16.1.78
[ndb_mgm]
connect-string=172.16.1.78
[ndb_mgmd]
config-file=/var/lib/mysql-cluster/config.ini
[mysql_cluster]
ndb-connectstring=172.16.1.78
[mysql.server]
user=mysql
basedir=/usr
[mysqld_safe]
log-error=/var/log/mysqld.log
#pid-file=/var/run/mysqld/mysqld.pid
[mysql]
#socket=/usr/local/var/mysql.sock
[mysqladmin]
#socket=/usr/local/var/mysql.sock
[ndb_restore default]
四、啟動管理節點
MySQL安裝目錄初始化
執行/usr/bin/mysql_install_db --user=mysql --datadir=/usr/local/mysql/data即可。注意這裡的datadir要和config.ini和my.cnf中配置的datadir一致
[
MySQL Cluster Management Server mysql-5.5.31 ndb-7.2.13
2013-10-29 15:45:36 [MgmtSrvr] INFO -- The default config directory '/usr/mysql-cluster' does not exist. Trying to create it...
2013-10-29 15:45:36 [MgmtSrvr] INFO -- Sucessfully created config directory
2013-10-29 15:45:36 [MgmtSrvr] WARNING -- at line 38: [tcp] portnumber is deprecated
2013-10-29 15:45:36 [MgmtSrvr] WARNING -- at line 38: Cluster configuration warning:
arbitrator with id 1 and db node with id 2 on same host 172.16.1.74
arbitrator with id 2 and db node with id 3 on same host 172.16.1.75
arbitrator with id 4 has no hostname specified
arbitrator with id 5 has no hostname specified
Running arbitrator on the same host as a database node may
cause complete cluster shutdown in case of host failure.
注:在啟動時幾個警告提示
2013-10-29 15:45:36 [MgmtSrvr] WARNING -- at line 38: [tcp] portnumber is deprecated這個警告提示可以不必理會，不影響使用；
2013-10-29 15:45:36 [MgmtSrvr] WARNING -- at line 38: Cluster configuration warning這個警告提示說節點1和3，2和4的arbitrator一樣，可能引起整個叢集失敗，由於是雙管理節點所示此警告可以不用放在心上
[[email protected] sbin]# netstat -lntpu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:1186 0.0.0.0:* LISTEN 9484/ndb_mgmd
看到1186埠開放了說明啟動是正常的。
五、初始化叢集
在db1中
[[email protected] sbin]# ndbd --ndb_nodeid=2 --initial
2013-10-30 09:37:07 [ndbd] INFO -- Angel connected to '172.16.1.78:1186'
2013-10-30 09:37:07 [ndbd] INFO -- Angel allocated nodeid: 2
執行netstat –lntpu，如果列表中有埠為2279，表示初始化成功
[[email protected] mysql-cluster]# netstat -lntpu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:1186 0.0.0.0:* LISTEN 13713/ndb_mgmd
tcp 0 0 172.16.1.74:2279 0.0.0.0:* LISTEN 13735/ndbd
在db2中
[[email protected] mysql]# ndbd --ndb_nodeid=3 --initial
2013-10-30 09:37:49 [ndbd] INFO -- Angel connected to '172.16.1.78:1186'
2013-10-30 09:37:49 [ndbd] INFO -- Angel allocated nodeid: 3
初始化叢集也可以直接輸入ndbd –initial命令，而不必輸入節點號
注：只有在第一次啟動ndbd時或者對config.ini進行改動後才需要使用--initial引數！資料庫維護過程中慎用initial引數，
如果資料庫中已經存在大量資料，並且在所有資料節點上都執行了 ndbd --initial，那麼所有資料庫中的表和資料都會消失。
檢查叢集工作狀態
在db1或者db2任意一臺伺服器上啟動管理終端：
[[email protected] mysql]# ndb_mgm -e show
Connected to Management Server at: 172.16.1.78:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=2 @172.16.1.74 (mysql-5.5.31 ndb-7.2.13, Nodegroup: 0, Master)
id=3 @172.16.1.75 (mysql-5.5.31 ndb-7.2.13, Nodegroup: 0)
[ndb_mgmd(MGM)] 1 node(s)
id=1 @172.16.1.78 (mysql-5.5.31 ndb-7.2.13)
[mysqld(API)] 4 node(s)
id=4 (not connected, accepting connect from any host)
id=5 (not connected, accepting connect from any host)
id=6 (not connected, accepting connect from any host)
id=7 (not connected, accepting connect from any host)
id=8 (not connected, accepting connect from any host)
id=9 (not connected, accepting connect from any host)
如果上面沒有問題，現在開始加入mysqld(API)
六、啟動SQL節點
在db1 中：
[[email protected] sbin]# mysqld_safe --ndb_nodeid=4 --user=mysql &
[1] 12495
[[email protected] sbin]# 131030 10:26:00 mysqld_safe Logging to '/var/log/mysqld.log'.
131030 10:26:00 mysqld_safe Starting mysqld daemon with databases from /usr/local/mysql/data/
在db2 中：
[[email protected] mysql]# mysqld_safe --ndb_nodeid=6 --user=mysql &
[1] 11756
[[email protected] sbin]#131030 10:18:56 mysqld_safe Logging to '/var/log/mysqld.log'.
131030 10:18:57 mysqld_safe Starting mysqld daemon with databases from /usr/local/mysql/data/
出現131110 0:37:19 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.host' doesn't exist
錯誤時怎樣解決問題
/usr/bin/mysql_install_db --user=mysql --datadir=/var/lib/mysql/data/
再次檢查叢集工作狀態，看SQL節點是否加入成功，出現下面的提示資訊表示新增成功
[[email protected] sbin]# ndb_mgm -e show
Connected to Management Server at: 172.16.1.78:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=2 @172.16.1.74 (mysql-5.5.31 ndb-7.2.13, Nodegroup: 0, Master)
id=3 @172.16.1.75 (mysql-5.5.31 ndb-7.2.13, Nodegroup: 0)
[ndb_mgmd(MGM)] 1 node(s)
id=1 @172.16.1.78 (mysql-5.5.31 ndb-7.2.13)
[mysqld(API)] 4 node(s)
id=4 @172.16.1.74 (mysql-5.5.31 ndb-7.2.13)
id=5 @172.16.1.74 (mysql-5.5.31 ndb-7.2.13)
id=6 @172.16.1.75 (mysql-5.5.31 ndb-7.2.13)
id=7 @172.16.1.75 (mysql-5.5.31 ndb-7.2.13)
id=8 (not connected, accepting connect from any host)
id=9 (not connected, accepting connect from any host)
此步也可以通過執行mysql伺服器啟動命令來代替
[[email protected] mysql-cluster]# service mysql start
Starting MySQL.......................[確定]......................
設定開機自動啟動
將一下內容加入到/etc/rc.d/rc.local檔案的最後一行，就可以使得ndb_mgmd和ndbd開機自動啟動了
管理節點伺服器172.16.1.78上新增ndb_mgmd -f /var/lib/mysql-cluster/config.ini
資料和SQL節點伺服器172.16.1.74、172.16.1.75上新增ndbd
注：mysqld服務會自動重啟，不需要新增
七、設定資料庫的使用者名稱和密碼
執行如下命令：
./bin/mysqladmin -u root password 'huawei'
./bin/mysqladmin -u root -h db2 password 'huawei'
新增一個webapp使用者，並且配置許可權
mysql>GRANT ALL ON *.* TO 'webapp'@'%' IDENTIFIED BY 'huawei';
mysql>GRANT ALL ON *.* TO 'webapp'@'localhost' IDENTIFIED BY 'huawei';
mysql>GRANT ALL ON *.* TO 'webapp'@'db1' IDENTIFIED BY 'huawei';
mysql>GRANT ALL ON *.* TO 'webapp'@'db2' IDENTIFIED BY 'huawei';
八、功能測試
到管理節點檢視下相關服務狀態
# ndb_mgm
ndb_mgm> show
Connected to Management Server at: 172.16.1.78:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=2 @172.16.1.74 (mysql-5.5.31 ndb-7.2.13, Nodegroup: 0, Master)
id=3 @172.16.1.75 (mysql-5.5.31 ndb-7.2.13, Nodegroup: 0)
[ndb_mgmd(MGM)] 1 node(s)
id=1 @172.16.1.78 (mysql-5.5.31 ndb-7.2.13)
[mysqld(API)] 6 node(s)
id=4 @172.16.1.74 (mysql-5.5.31 ndb-7.2.13)
id=5 @172.16.1.74 (mysql-5.5.31 ndb-7.2.13)
id=6 @172.16.1.75 (mysql-5.5.31 ndb-7.2.13)
id=7 @172.16.1.75 (mysql-5.5.31 ndb-7.2.13)
id=8 (not connected, accepting connect from any host)
id=9 (not connected, accepting connect from any host)
可以看到這裡的資料節點、管理節點、sql節點都是正常的。現在我們在其中一個數據節點上進行相關資料庫的建立,然後到另外一個數據節點上看看資料是否同步。
[[email protected] usr]# mysql -u root
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.5.31-ndb-7.2.13-cluster-gpl MySQL Cluster Community Server (GPL)
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| ndb_4_fs |
| ndbinfo |
| performance_schema |
| test |
+--------------------+
6 rows in set (0.08 sec)
mysql> create database aa;
Query OK, 1 row affected (3.56 sec)
mysql> use aa
Database changed
mysql> CREATE TABLE ctest2 (i INT) ;
Query OK, 0 rows affected (24.71 sec)
mysql> INSERT INTO ctest2 () VALUES (1);
Query OK, 1 row affected (0.54 sec)
mysql> SELECT * FROM ctest2;
+------+
| i |
+------+
| 1 |
+------+
1 row in set (0.01 sec)
現在到另外一個數據節點db1檢視下aa資料庫是否同步過來了。
[[email protected] mysql-cluster]# mysql -u root
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| aa |
| mysql |
| ndb_3_fs |
| ndbinfo |
| performance_schema |
| test |
+--------------------+
7 rows in set (4.80 sec)
mysql> use aa
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> select * from ctest2;
+------+
| i |
+------+
| 1 |
+------+
1 row in set (0.01 sec)
從上面可以看到資料已經同步了，mysql叢集環境已經搭建完成。
九、破壞性測試
在上面可以看到172.16.1.74作為主的資料節點，現在把172.16.1.74這臺機器關閉，有如下的結果：
[[email protected] mysql-cluster]# shutdown -h now
Broadcast message from root (pts/2) (Thu Oct 31 14:00:56 2013):
The system is going down for system halt NOW!
[[email protected] ~]# ndb_mgm -e show
Connected to Management Server at: 172.16.1.75:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=2 (not connected, accepting connect from 172.16.1.74)
id=3 @172.16.1.75 (mysql-5.5.31 ndb-7.2.13, Nodegroup: 0, Master)
[ndb_mgmd(MGM)] 1 node(s)
id=1 @172.16.1.78 (mysql-5.5.31 ndb-7.2.13)
[mysqld(API)] 4 node(s)
id=4 (not connected, accepting connect from any host)
id=5 (not connected, accepting connect from any host)
id=6 @172.16.1.75 (mysql-5.5.31 ndb-7.2.13)
id=7 @172.16.1.75 (mysql-5.5.31 ndb-7.2.13)
id=8 (not connected, accepting connect from any host)
id=9 (not connected, accepting connect from any host)
從上面可以發現現在172.16.1.74這臺機器的資料節點和sql節點已經連線不上了，172.16.1.75成為了主資料節點，現在在172.16.1.75資料節點上建立一個表，然後恢復172.16.1.74的資料節點，檢視是否能夠把資料同步過來了。
先在172.16.1.75資料節點做如下操作：
mysql> create table ctest3(id int(11)) ;
mysql> show tables;
mysql> create table ctest3(id int(11)) ;
Query OK, 0 rows affected (0.20 sec)
mysql> show tables;
+--------------+
| Tables_in_aa |
+--------------+
| ctest2 |
| ctest3 |
+--------------+
2 rows in set (0.00 sec)
mysql> insert into ctest3 values(1);
Query OK, 1 row affected (0.02 sec)
mysql> select * from ctest3;
+------+
| id |
+------+
| 1 |
+------+
1 row in set (0.01 sec)
然後我們恢復172.16.1.74資料節點，檢視下ctest3資料是否同步過來了。
[[email protected] ~]# mysql -u root
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| aa |
| mysql |
| ndb_3_fs |
| ndbinfo |
| performance_schema |
| test |
+--------------------+
7 rows in set (0.19 sec)
mysql> use aa
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;
+--------------+
| Tables_in_aa |
+--------------+
| ctest2 |
| ctest3 |
+--------------+
2 rows in set (0.00 sec)
mysql> select * from ctest3;
+------+
| id |
+------+
| 1 |
+------+
1 row in set (0.06 sec)
可以看到172.16.1.74資料節點已經把172.16.1.75資料節點的資料同步過來了，說明mysql叢集是沒有問題的。
HA高可用+負載均衡的配置
十、實現高可用和負載均衡(還需要再完善整理)
ldirectord+heartbeat介紹
ldirectord的作用
在Apache伺服器的前端，我們要放置一臺伺服器專門來做負載排程的任務（為了稱呼簡單和便於理解，我們將這樣的負載排程伺服器簡稱為“導演”），
用來把訪問需求分發給兩臺Apache伺服器。這個“導演”的任務，正是由ldirectord來完成的。
“ldirectord”（Linux Director Daemon）可以對服務和物理伺服器進行監測，被廣泛地用於http和https等服務。
它是專門為LVS（Linux Virtual Server）監控而編寫的，
不僅能從heartbeat的配置檔案/etc/ha.d/xxx.cf中讀取所有有關IPVS（IP Virtul Server）路由表配置的資訊，
還可以方便地被heartbeat管理（比如由heartbeat來啟動和停止ldirectord服務）。
heartbeat的工作原理
因為我們要提供“高可用性”，所以要考慮到“導演”突然罷工的情形；因此我們要安排兩個導演，也就是要有兩個排程伺服器節點。
這兩個節點的地位不同，其中一個是主節點，另外一個是輔節點（可以看成是一個“主導演”和一個“副導演”）。
這兩個節點正是用heartbeat來互相監測對方的。heartbeat可以通過乙太網（或者序列介面）來監控節點的“健康”狀況。
如果有多個heartbeat節點（heartbeat 2.0及後續版本已經能夠支援兩個以上節點），我們既可以使用序列線又可以使用乙太網連線它們，
這樣將大大提高系統的可用性。heartbeat的核心功能有兩個部分：心跳監測和資源接管。通過心跳監測，節點之間相互“打招呼”（傳送報文）
來告訴對方自己當前的狀態；如果在指定的時間內沒“聽”到對方“打招呼”（沒收到報文），那麼就認為對方罷工了，
這時heartbeat會自動啟動資源接管模組，執行相關的shell指令碼來接管執行在對方主機上的資源或者服務。
注意：heartbeat+ldirectord要發揮作用，必須保證承擔負載均衡的兩臺伺服器時間是同步的。
具體到我的配置裡就要求172.16.1.74與172.16.1.75伺服器時間同步。
檢視核心是否支援IPVS
[[email protected] init.d]# modprobe -l |grep ipvs
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_dh.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_ftp.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_lblc.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_lblcr.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_lc.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_nq.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_rr.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_sed.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_sh.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_wlc.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_wrr.ko
[[email protected] ipvs]# modprobe -l |grep ipvs
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_dh.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_ftp.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_lblc.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_lblcr.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_lc.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_nq.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_rr.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_sed.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_sh.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_wlc.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_wrr.ko
啟用核心的包轉發功能
修改/etc/sysctl.conf
#net.ipv4.ip_forward = 0
net.ipv4.ip_forward = 1
[[email protected] etc]# sysctl -p //使其馬上生效
net.ipv4.ip_forward = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
安裝heartbeat和lidirectord軟體
[[email protected] heartbeat2_x86_64]# ls -l
-rw-r--r-- 1 root root 1691714 11-10 11:31 heartbeat-2.1.4-11.el5.x86_64.rpm
-rw-r--r-- 1 root root 111941 11-10 12:08 heartbeat-ldirectord-2.1.4-11.el5.x86_64.rpm
-rw-r--r-- 1 root root 93297 11-10 11:57 heartbeat-pils-2.1.4-11.el5.x86_64.rpm
-rw-r--r-- 1 root root 190686 11-10 11:56 heartbeat-stonith-2.1.4-11.el5.x86_64.rpm
-rw-r--r-- 1 root root 33839 11-10 12:10 ipvsadm-1.24-12.el5.x86_64.rpm
-rw-r--r-- 1 root root 62324 11-10 11:44 libnet-1.1.6-2.el5.pp.x86_64.rpm
-rw-r--r-- 1 root root 38792 2011-09-10 libtool-ltdl-1.5.22-7.el5_4.x86_64.rpm
-rw-r--r-- 1 root root 171524 2011-09-10 openhpi-libs-2.14.0-5.el5.x86_64.rpm
-rw-r--r-- 1 root root 224092 11-10 12:18 perl-DBD-mysql-4.005-1.el5.rf.x86_64.rpm
-rw-r--r-- 1 root root 860170 11-10 12:19 perl-DBI-1.58-2.el5.rf.x86_64.rpm
-rw-r--r-- 1 root root 93196 2011-09-10 perl-MailTools-1.77-1.el5.centos.noarch.rpm
-rw-r--r-- 1 root root 45282 11-10 12:26 perl-Net-Daemon-0.39-1.el5.rf.noarch.rpm
-rw-r--r-- 1 root root 33187 11-10 12:25 perl-PlRPC-0.2018-1.2.el5.rf.noarch.rpm
-rw-r--r-- 1 root root 32736 2007-07-02 perl-TimeDate-1.16-5.el5.noarch.rpm
[[email protected] heartbeat2_x86_64]# rpm -ivh *.rpm
[[email protected] heartbeat2_x86_64]# ls
heartbeat-2.1.4-11.el5.x86_64.rpm ipvsadm-1.24-12.el5.x86_64.rpm perl-DBD-mysql-4.005-1.el5.rf.x86_64.rpm perl-PlRPC-0.2018-1.2.el5.rf.noarch.rpm
heartbeat-ldirectord-2.1.4-11.el5.x86_64.rpm libnet-1.1.6-2.el5.pp.x86_64.rpm perl-DBI-1.58-2.el5.rf.x86_64.rpm perl-TimeDate-1.16-5.el5.noarch.rpm
heartbeat-pils-2.1.4-11.el5.x86_64.rpm libtool-ltdl-1.5.22-7.el5_4.x86_64.rpm perl-MailTools-1.77-1.el5.centos.noarch.rpm
heartbeat-stonith-2.1.4-11.el5.x86_64.rpm openhpi-libs-2.14.0-5.el5.x86_64.rpm perl-Net-Daemon-0.39-1.el5.rf.noarch.rpm
[[email protected] heartbeat2_x86_64]# rpm -ivh *.rpm
配置heartbeat
[[email protected] etc]# vi /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
172.16.1.75 db2.test.com db2
172.16.1.74 db1.test.com db1
172.16.1.78 demo.test.com demo
172.16.1.73 db.test.com db #後面要專門設定這樣一個虛擬IP地址
[[email protected] ~]# vi /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
172.16.1.74 db1.test.com db1
172.16.1.75 db2.test.com db2
172.16.1.78 demo.test.com demo
172.16.1.73 db.test.com db #後面要專門設定這樣一個虛擬IP地址
複製一些檔案
[[email protected] doc]# cd /usr/share/doc/heartbeat-2.1.4/
[[email protected] heartbeat-3.0.3]# ls
apphbd.cf authkeys AUTHORS ChangeLog COPYING COPYING.LGPL ha.cf haresources README
[[email protected] heartbeat-3.0.3]# cp ha.cf /etc/ha.d/
[[email protected] heartbeat-3.0.3]# cp haresources /etc/ha.d/
[[email protected] heartbeat-3.0.3]# cp authkeys /etc/ha.d/
[[email protected] heartbeat-3.0.3]# ls
apphbd.cf authkeys AUTHORS ChangeLog COPYING COPYING.LGPL ha.cf haresources README
[[email protected] heartbeat-3.0.3]# cp ha.cf /etc/ha.d/
[[email protected] heartbeat-3.0.3]# cp haresources /etc/ha.d/
[[email protected] heartbeat-3.0.3]# cp authkeys /etc/ha.d/
vi /etc/ha.d/ha.cf新增如下內容
debugfile /var/log/ha.log
logfile /var/log/ha.log
logfacility local0
bcast eth0 # Linux
mcast eth0 225.0.0.1 694 1 0
auto_failback off
node db1
node db2
respawn hacluster /usr/lib/heartbeat/ipfail #64位系統此處應該是/usr/lib64/heartbeat/ipfail
apiauth ipfail gid=haclient uid=hacluster
keepalive 1
deadtime 10
#warntime 10
initdead 120
#udpport 694
#baud 19200
64位系統需要執行下面的操作：
[[email protected] lib]# mkdir heartbeat
[[email protected] lib]# ln -s /usr/lib64/heartbeat/ipfail /usr/lib/heartbeat/ipfail
vi /etc/ha.d/haresources 設定虛擬IP地址
db1.test.com ldirectord::ldirectord.cf LVSSyncDaemonSwap::master IPaddr::172.16.1.73/24/eth0/172.16.1.255
vi /etc/ha.d/authkeys
auth 3
3 md5 Hello!
[[email protected] ha.d]# chmod 600 authkeys
[[email protected] ha.d]# chmod 600 authkeys
上述三個檔案的配置db1、db2節點要相同
配置ldirectord
[[email protected] ha.d]# find / -name ldirectord.cf
/usr/share/doc/ldirectord-1.0.4/ldirectord.cf
[[email protected] ha.d]# cd /usr/share/doc/ldirectord-1.0.4/
[[email protected] ldirectord-1.0.4]# cp ldirectord.cf /etc/ha.d/
[[email protected] ldirectord-1.0.4]# vi /etc/ha.d/ldirectord.cf
配置如下的內容：
# Global Directives
checktimeout=10
checkinterval=2
#fallback=127.0.0.1:80
#fallback6=[::1]:80
autoreload=yes
logfile="/var/log/ldirectord.log"
#logfile="local0"
#emailalert="[email protected]"
#emailalertfreq=3600
#emailalertstatus=all
quiescent=yes
# Sample for an http virtual service
virtual=172.16.1.73:3306
real=172.16.1.74:3306 gate
real=172.16.1.75:3306 gate
# real=192.168..: gate
# fallback=127.0.0.1:3306 gate
service=mysql
scheduler=wrr
#persistent=600
#netmask=255.255.255.255
protocol=tcp
checktype=negotiate
# checkport=80
login="ldirectord"
passwd="ldirectord"
database="ldirectordb"
request="select * from ldirectordtest"
receive="ldirectord is OK"
# virtualhost=www.x.y.z
阻止ldirectord服務開機啟動，推遲heartbeat啟動的操作怎麼弄呢？讓hearbeat來控制ldirectord
/sbin/chkconfig --level 2345 heartbeat on
/sbin/chkconfig --del ldirectord
還有怎樣才能修改heartbeat啟動時候的優先級別呢？
目前的方法是/etc/rc.d/init.d/目錄下的heatbeat的指令碼的註釋
# chkconfig: - 75 05
# description: Startup script high-availability services.
# processname: heartbeat
# pidfile: /var/run/heartbeat.pid
# config: /etc/ha.d/ha.cf
建立資料庫和測試表
mysql>GRANT ALL ON ldirectordb.* TO 'ldirectord'@'%' IDENTIFIED BY 'ldirectord';
mysql>GRANT ALL ON ldirectordb.* TO 'ldirectord'@'localhost' IDENTIFIED BY 'ldirectord';
mysql>GRANT ALL ON ldirectordb.* TO 'ldirectord'@'db1' IDENTIFIED BY 'ldirectord';
mysql>GRANT ALL ON ldirectordb.* TO 'ldirectord'@'db2' IDENTIFIED BY 'ldirectord';
mysql>FLUSH PRIVILEGES;
mysql>CREATE DATABASE ldirectordb;
mysql>USE ldirectordb;
mysql>CREATE TABLE ldirectordtest (Status INT) ;
mysql>INSERT INTO ldirectordtest() VALUES (1);
mysql>quit
先設定IP路由
檢查是否已經安裝了iproute包
[[email protected] mysql]# rpm -qa |grep iproute
iproute-2.6.18-11.el5
修改/etc/sysctl.conf檔案
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.eth0.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.eth0.arp_announce = 2
sysctl -p
配置虛擬IP地址，虛擬IP應該使用lo:0來配置，不應該使用eth0:0來配置，使用eth0:0來配置會導致虛擬IP地址在區域網內衝突，使用lo:0
不會有這樣情況發生。
/etc/sysconfig/network-scripts/
db1伺服器lo:0的設定（虛擬IP地址）
DEVICE=lo:0
IPADDR=172.16.1.73
NETMASK=255.255.255.255
ONBOOT=yes
db2上設定lo:0的IP地址（虛擬IP地址）
DEVICE=lo:0
IPADDR=172.16.1.73
NETMASK=255.255.255.255
ONBOOT=yes
[[email protected] network-scripts]# ifup ifcfg-lo:0
db2.test.com上執行同樣的配置和操作即可。
高可用性和負載均衡測試
[[email protected] init.d]# ldirectord stop
[[email protected] init.d]# heartbeat start
執行上面命令若沒有報錯，就表示一切正常
重啟兩臺伺服器db1和db2
ldirectord狀態檢查
[[email protected] ~]# ldirectord ldirectord.cf status
ldirectord for /etc/ha.d/ldirectord.cf is running with pid: 6203
[[email protected] ~]# ldirectord ldirectord.cf status
ldirectord is stopped for /etc/ha.d/ldirectord.cf
heartbeat狀態檢查
[[email protected] log]# service heartbeat status
heartbeat OK [pid 5750 et al] is running on db1.test.com [db1.test.com]...
[[email protected] log]# service heartbeat status
heartbeat OK [pid 5796 et al] is running on db2.test.com [db2.test.com]...
----錯誤問題解決
[[email protected] init.d]# ./ldirectord status
ldirectord stale pid file /var/run/ldirectord.ldirectord.pid for /etc/ha.d/ldirectord.cf
ldirectord is stopped for /etc/ha.d/ldirectord.cf
此問題已經解決了，是因為缺少包造成的。缺少perl-DBI-mysql、perl-DBD-mysql
虛擬IP狀態檢查
[[email protected] log]# ip addr sh eth0
4: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 90:e2:ba:01:fe:24 brd ff:ff:ff:ff:ff:ff
inet 172.16.1.75/24 brd 172.16.1.255 scope global eth0
inet 172.16.1.73/24 brd 172.16.1.255 scope global secondary eth0:0
inet6 fe80::92e2:baff:fe01:fe24/64 scope link
valid_lft forever preferred_lft forever
[[email protected] log]# ip addr sh eth0
4: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 90:e2:ba:01:fa:fa brd ff:ff:ff:ff:ff:ff
inet 172.16.1.74/24 brd 172.16.1.255 scope global eth0
inet6 fe80::92e2:baff:fe01:fafa/64 scope link
valid_lft forever preferred_lft forever
IPVS狀態檢查
[[email protected] log]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.16.1.73:3306 wrr
-> 172.16.1.74:3306 Route 1 0 0
-> 172.16.1.75:3306 Local 0 0 0
[[email protected] log]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
db1上執行./LVSSyncDaemonSwap master status
[[email protected] resource.d]# ./LVSSyncDaemonSwap master status
master running
(ipvs_syncmaster pid: 4043)
db1上執行./LVSSyncDaemonSwap master status
[[email protected] ha.d]# resource.d/LVSSyncDaemonSwap master status
master stopped
[[email protected] log]# mysql -u ldirectord -p -h 172.16.1.73
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 492
Server version: 5.5.31-ndb-7.2.13-cluster-gpl MySQL Cluster Community Server (GPL)
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| ldirectordb |
| test |
+--------------------+
3 rows in set (0.00 sec)
mysql> use ldirectordb;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> select * from ldirectordtest;
+--------+
| Status |
+--------+
| 1 |
+--------+
1 row in set (0.00 sec)
mysql>
[[email protected] log]# mysql -u ldirectord -p -h 172.16.1.73
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 492
Server version: 5.5.31-ndb-7.2.13-cluster-gpl MySQL Cluster Community Server (GPL)
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| ldirectordb |
| test |
+--------------------+
3 rows in set (0.00 sec)
mysql> use ldirectordb;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> select * from ldirectordtest;
+--------+
| Status |
+--------+
| 1 |
+--------+
1 row in set (0.00 sec)
參考文件：
http://wenku.baidu.com/view/0054090e7cd184254b353546.html
http://blog.csdn.net/sdd00001/article/details/21464713
http://www.linuxfly.org/post/531/
http://blog.csdn.net/tongle_deng/article/details/7297597
一、介紹
這篇文件旨在介紹如何安裝配置基於3臺伺服器的MySQL叢集。並且實現任意一臺伺服器出現問題或宕機時MySQL叢集依然能夠繼續執行。
1. MySQL-Cluster簡介
MySQL-Cluster主要有三種類型節點：
NDB node，資料節點將儲存資料庫，它會自動複製所有的資料節點；
SQL node，守護節點相當於是SQL資料庫和客戶端之間的介面，它提供從資料節點查詢等操作，類似於"閘道器"；
Management node，管理節點，用以監控和管理整個叢集。
-------------------------------此處插圖----------------------------
2.安裝環境及軟體包
3臺華為RH2285伺服器——其中兩臺（16顆CPU、96G記憶體、500G磁碟陣列）作為NDB節點和SQL節點伺服器，另一臺伺服器16顆CPU、32G記憶體作為管理節點
軟體包：MySQL-Cluster-client-gpl-7.2.13-1.rhel5.x86_64.rpm
MySQL-Cluster-server-gpl-7.2.13-1.rhel5.x86_64.rpm
MySQL-Cluster-devel-gpl-7.2.13-1.rhel5.x86_64.rpm
MySQL-Cluster-embedded-gpl-7.2.13-1.rhel5.x86_64.rpm
MySQL-Cluster-shared-compat-gpl-7.2.13-1.rhel5.x86_64.rpm
MySQL-Cluster-shared-gpl-7.2.13-1.rhel5.x86_64.rpm
作業系統：Redhat.Enterprise.Linux.v5.UPDATE.7.X86_64.DVD-HOTiSO.iso
伺服器hostname和IP地址:
db1 172.16.1.74
db2 172.16.1.75
demo 172.16.1.78
資料庫規劃：
資料庫管理節點（ndb_mgmd）：172.16.1.78
資料庫資料節點（NDB）：172.16.1.74，172.16.1.75
資料庫SQL節點(API節點)：172.16.1.74，172.16.1.75
二、在db1和db2上安裝MySQL-cluster
以下步驟需要在db1和db2上各做一次
1.首先檢查linux作業系統中是否安裝了自帶的mysql資料庫
一般linux自帶mysql版本都比較低一些，因此要安裝比較新的版本需要先將自帶的低版本解除安裝。
檢查作業系統中是否安裝了mysql的命令：
rpm -qa |grep -i mysql
解除安裝mysql的命令：rpm -e mysqlXXXX
或者使用yum -remove mysqlXXX來刪除，這樣可以避免解除安裝過程中由於各個包之間的依賴造成解除安裝停止
2.建立mysql使用者和組
groupadd mysql
useradd -g mysql mysql
usermod -d /home/mysql mysql
3.將MySQL-Cluster上傳到/home/mysql/目錄下，開始安裝MySQL-Cluster
[
Preparing... ########################################### [100%]
1:MySQL-Cluster-server-gp########################################### [100%]
PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following commands:
/usr/bin/mysqladmin -u root password 'new-password'
/usr/bin/mysqladmin -u root -h db1 password 'new-password'
Alternatively you can run:
/usr/bin/mysql_secure_installation
which will also give you the option of removing the test
databases and anonymous user created by default. This is
strongly recommended for production servers.
See the manual for more instructions.
Please report any problems with the /usr/bin/mysqlbug script!
[
Preparing... ########################################### [100%]
1:MySQL-Cluster-client-gp ########################################### [100%]
依次完成MySQL-Cluster-devel-gpl-7.2.13-1.rhel5.x86_64.rpm
MySQL-Cluster-embedded-gpl-7.2.13-1.rhel5.x86_64.rpm
MySQL-Cluster-shared-compat-gpl-7.2.13-1.rhel5.x86_64.rpm
MySQL-Cluster-shared-gpl-7.2.13-1.rhel5.x86_64.rpm
的安裝
4.安裝完成之後建立連線
ln -s /usr/sbin/ndbd /usr/bin
ln -s /usr/sbin/ndb_mgmd /usr/bin
ln -s /usr/sbin/mysqld /usr/bin
#ln -s /usr/local/lib/mysqlmanager /usr/bin ---這個在7.2.13版本上不需要
管理節點伺服器demo（172.16.1.78）上只安裝MySQL-Cluster-server-gpl-7.2.13-1.rhel5.x86_64.rpm即可
三、安裝並配置節點
1.配置管理節點（172.16.1.78）配置檔案
# mkdir /var/lib/mysql-cluster
# cd /var/lib/mysql-cluster
# vi config.ini
在config.ini中新增如下內容：
[ndbd default]
NoOfReplicas= 2
MaxNoOfConcurrentOperations= 500000
MaxNoOfLocalOperations=500000
MaxNoOfConcurrentTransactions=500000
MaxNoOfConcurrentIndexOperations=50000
# Amount of memory required=(SizeofDatabase * NumberOfReplicas * 1.1 ) / NumberOfDataNodes
DataMemory= 49152M
IndexMemory= 6144M
TimeBetweenWatchDogCheck= 30000
DataDir=/var/lib/mysql/data
MaxNoOfAttributes=500000
MaxNoOfTables=1000
MaxNoOfOrderedIndexes= 512
#StartPartialTimeout=100
#StartPartitionedTimeout=100
#ArbitrationTimeout=5000
#MaxNoOfFiredTriggers=8000
#TransactionBufferMemory=5M
TransactionDeadlockDetectionTimeout=90000
HeartbeatIntervalDbDb=5000
#StopOnError=1
[ndb_mgmd default]
DataDir=/var/lib/mysql/data
[ndb_mgmd]
NodeId=1
HostName= 172.16.1.78
[ndbd]
NodeId= 2
HostName= 172.16.1.74
[ndbd]
NodeId= 3
HostName= 172.16.1.75
[mysqld]
ArbitrationRank=2
[mysqld]
ArbitrationRank=2
[mysqld]
[mysqld]
[mysqld]
[mysqld]
[tcp default]
portnumber=2279
以下步驟需要在db1和db2上各做一次
2.配置通用my.cnf檔案（SQL節點及ndb節點均使用此檔案）
# vi /etc/my.cnf
在my.cnf中新增如下內容：
[mysqld]
datadir=/var/lib/mysql/data/
socket=/var/lib/mysql/mysql.sock
lower_case_table_names=1
ndb_cluster_connection_pool=2
tmp_table_size=512M
max_heap_table_size=512M
max_connections =400
thread_stack=1M
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=1
default-storage-engine=ndbcluster
ndbcluster
ndb-connectstring=172.16.1.78
[ndbd]
connect-string=172.16.1.78
[ndb_mgm]
connect-string=172.16.1.78
[ndb_mgmd]
config-file=/var/lib/mysql-cluster/config.ini
[mysql_cluster]
ndb-connectstring=172.16.1.78
[mysql.server]
user=mysql
basedir=/usr
[mysqld_safe]
log-error=/var/log/mysqld.log
#pid-file=/var/run/mysqld/mysqld.pid
[mysql]
#socket=/usr/local/var/mysql.sock
[mysqladmin]
#socket=/usr/local/var/mysql.sock
[ndb_restore default]
四、啟動管理節點
MySQL安裝目錄初始化
執行/usr/bin/mysql_install_db --user=mysql --datadir=/usr/local/mysql/data即可。注意這裡的datadir要和config.ini和my.cnf中配置的datadir一致
[
MySQL Cluster Management Server mysql-5.5.31 ndb-7.2.13
2013-10-29 15:45:36 [MgmtSrvr] INFO -- The default config directory '/usr/mysql-cluster' does not exist. Trying to create it...
2013-10-29 15:45:36 [MgmtSrvr] INFO -- Sucessfully created config directory
2013-10-29 15:45:36 [MgmtSrvr] WARNING -- at line 38: [tcp] portnumber is deprecated
2013-10-29 15:45:36 [MgmtSrvr] WARNING -- at line 38: Cluster configuration warning:
arbitrator with id 1 and db node with id 2 on same host 172.16.1.74
arbitrator with id 2 and db node with id 3 on same host 172.16.1.75
arbitrator with id 4 has no hostname specified
arbitrator with id 5 has no hostname specified
Running arbitrator on the same host as a database node may
cause complete cluster shutdown in case of host failure.
注:在啟動時幾個警告提示
2013-10-29 15:45:36 [MgmtSrvr] WARNING -- at line 38: [tcp] portnumber is deprecated這個警告提示可以不必理會，不影響使用；
2013-10-29 15:45:36 [MgmtSrvr] WARNING -- at line 38: Cluster configuration warning這個警告提示說節點1和3，2和4的arbitrator一樣，可能引起整個叢集失敗，由於是雙管理節點所示此警告可以不用放在心上
[[email protected] sbin]# netstat -lntpu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:1186 0.0.0.0:* LISTEN 9484/ndb_mgmd
看到1186埠開放了說明啟動是正常的。
五、初始化叢集
在db1中
[[email protected] sbin]# ndbd --ndb_nodeid=2 --initial
2013-10-30 09:37:07 [ndbd] INFO -- Angel connected to '172.16.1.78:1186'
2013-10-30 09:37:07 [ndbd] INFO -- Angel allocated nodeid: 2
執行netstat –lntpu，如果列表中有埠為2279，表示初始化成功
[[email protected] mysql-cluster]# netstat -lntpu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:1186 0.0.0.0:* LISTEN 13713/ndb_mgmd
tcp 0 0 172.16.1.74:2279 0.0.0.0:* LISTEN 13735/ndbd
在db2中
[[email protected] mysql]# ndbd --ndb_nodeid=3 --initial
2013-10-30 09:37:49 [ndbd] INFO -- Angel connected to '172.16.1.78:1186'
2013-10-30 09:37:49 [ndbd] INFO -- Angel allocated nodeid: 3
初始化叢集也可以直接輸入ndbd –initial命令，而不必輸入節點號
注：只有在第一次啟動ndbd時或者對config.ini進行改動後才需要使用--initial引數！資料庫維護過程中慎用initial引數，
如果資料庫中已經存在大量資料，並且在所有資料節點上都執行了 ndbd --initial，那麼所有資料庫中的表和資料都會消失。
檢查叢集工作狀態
在db1或者db2任意一臺伺服器上啟動管理終端：
[[email protected] mysql]# ndb_mgm -e show
Connected to Management Server at: 172.16.1.78:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=2 @172.16.1.74 (mysql-5.5.31 ndb-7.2.13, Nodegroup: 0, Master)
id=3 @172.16.1.75 (mysql-5.5.31 ndb-7.2.13, Nodegroup: 0)
[ndb_mgmd(MGM)] 1 node(s)
id=1 @172.16.1.78 (mysql-5.5.31 ndb-7.2.13)
[mysqld(API)] 4 node(s)
id=4 (not connected, accepting connect from any host)
id=5 (not connected, accepting connect from any host)
id=6 (not connected, accepting connect from any host)
id=7 (not connected, accepting connect from any host)
id=8 (not connected, accepting connect from any host)
id=9 (not connected, accepting connect from any host)
如果上面沒有問題，現在開始加入mysqld(API)
六、啟動SQL節點
在db1 中：
[[email protected] sbin]# mysqld_safe --ndb_nodeid=4 --user=mysql &
[1] 12495
[[email protected] sbin]# 131030 10:26:00 mysqld_safe Logging to '/var/log/mysqld.log'.
131030 10:26:00 mysqld_safe Starting mysqld daemon with databases from /usr/local/mysql/data/
在db2 中：
[[email protected] mysql]# mysqld_safe --ndb_nodeid=6 --user=mysql &
[1] 11756
[[email protected] sbin]#131030 10:18:56 mysqld_safe Logging to '/var/log/mysqld.log'.
131030 10:18:57 mysqld_safe Starting mysqld daemon with databases from /usr/local/mysql/data/
出現131110 0:37:19 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.host' doesn't exist
錯誤時怎樣解決問題
/usr/bin/mysql_install_db --user=mysql --datadir=/var/lib/mysql/data/
再次檢查叢集工作狀態，看SQL節點是否加入成功，出現下面的提示資訊表示新增成功
[[email protected] sbin]# ndb_mgm -e show
Connected to Management Server at: 172.16.1.78:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=2 @172.16.1.74 (mysql-5.5.31 ndb-7.2.13, Nodegroup: 0, Master)
id=3 @172.16.1.75 (mysql-5.5.31 ndb-7.2.13, Nodegroup: 0)
[ndb_mgmd(MGM)] 1 node(s)
id=1 @172.16.1.78 (mysql-5.5.31 ndb-7.2.13)
[mysqld(API)] 4 node(s)
id=4 @172.16.1.74 (mysql-5.5.31 ndb-7.2.13)
id=5 @172.16.1.74 (mysql-5.5.31 ndb-7.2.13)
id=6 @172.16.1.75 (mysql-5.5.31 ndb-7.2.13)
id=7 @172.16.1.75 (mysql-5.5.31 ndb-7.2.13)
id=8 (not connected, accepting connect from any host)
id=9 (not connected, accepting connect from any host)
此步也可以通過執行mysql伺服器啟動命令來代替
[[email protected] mysql-cluster]# service mysql start
Starting MySQL.......................[確定]......................
設定開機自動啟動
將一下內容加入到/etc/rc.d/rc.local檔案的最後一行，就可以使得ndb_mgmd和ndbd開機自動啟動了
管理節點伺服器172.16.1.78上新增ndb_mgmd -f /var/lib/mysql-cluster/config.ini
資料和SQL節點伺服器172.16.1.74、172.16.1.75上新增ndbd
注：mysqld服務會自動重啟，不需要新增
七、設定資料庫的使用者名稱和密碼
執行如下命令：
./bin/mysqladmin -u root password 'huawei'
./bin/mysqladmin -u root -h db2 password 'huawei'
新增一個webapp使用者，並且配置許可權
mysql>GRANT ALL ON *.* TO 'webapp'@'%' IDENTIFIED BY 'huawei';
mysql>GRANT ALL ON *.* TO 'webapp'@'localhost' IDENTIFIED BY 'huawei';
mysql>GRANT ALL ON *.* TO 'webapp'@'db1' IDENTIFIED BY 'huawei';
mysql>GRANT ALL ON *.* TO 'webapp'@'db2' IDENTIFIED BY 'huawei';
八、功能測試
到管理節點檢視下相關服務狀態
# ndb_mgm
ndb_mgm> show
Connected to Management Server at: 172.16.1.78:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=2 @172.16.1.74 (mysql-5.5.31 ndb-7.2.13, Nodegroup: 0, Master)
id=3 @172.16.1.75 (mysql-5.5.31 ndb-7.2.13, Nodegroup: 0)
[ndb_mgmd(MGM)] 1 node(s)
id=1 @172.16.1.78 (mysql-5.5.31 ndb-7.2.13)
[mysqld(API)] 6 node(s)
id=4 @172.16.1.74 (mysql-5.5.31 ndb-7.2.13)
id=5 @172.16.1.74 (mysql-5.5.31 ndb-7.2.13)
id=6 @172.16.1.75 (mysql-5.5.31 ndb-7.2.13)
id=7 @172.16.1.75 (mysql-5.5.31 ndb-7.2.13)
id=8 (not connected, accepting connect from any host)
id=9 (not connected, accepting connect from any host)
可以看到這裡的資料節點、管理節點、sql節點都是正常的。現在我們在其中一個數據節點上進行相關資料庫的建立,然後到另外一個數據節點上看看資料是否同步。
[[email protected] usr]# mysql -u root
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.5.31-ndb-7.2.13-cluster-gpl MySQL Cluster Community Server (GPL)
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| ndb_4_fs |
| ndbinfo |
| performance_schema |
| test |
+--------------------+
6 rows in set (0.08 sec)
mysql> create database aa;
Query OK, 1 row affected (3.56 sec)
mysql> use aa
Database changed
mysql> CREATE TABLE ctest2 (i INT) ;
Query OK, 0 rows affected (24.71 sec)
mysql> INSERT INTO ctest2 () VALUES (1);
Query OK, 1 row affected (0.54 sec)
mysql> SELECT * FROM ctest2;
+------+
| i |
+------+
| 1 |
+------+
1 row in set (0.01 sec)
現在到另外一個數據節點db1檢視下aa資料庫是否同步過來了。
[[email protected] mysql-cluster]# mysql -u root
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| aa |
| mysql |
| ndb_3_fs |
| ndbinfo |
| performance_schema |
| test |
+--------------------+
7 rows in set (4.80 sec)
mysql> use aa
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> select * from ctest2;
+------+
| i |
+------+
| 1 |
+------+
1 row in set (0.01 sec)
從上面可以看到資料已經同步了，mysql叢集環境已經搭建完成。
九、破壞性測試
在上面可以看到172.16.1.74作為主的資料節點，現在把172.16.1.74這臺機器關閉，有如下的結果：
[[email protected] mysql-cluster]# shutdown -h now
Broadcast message from root (pts/2) (Thu Oct 31 14:00:56 2013):
The system is going down for system halt NOW!
[[email protected] ~]# ndb_mgm -e show
Connected to Management Server at: 172.16.1.75:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=2 (not connected, accepting connect from 172.16.1.74)
id=3 @172.16.1.75 (mysql-5.5.31 ndb-7.2.13, Nodegroup: 0, Master)
[ndb_mgmd(MGM)] 1 node(s)
id=1 @172.16.1.78 (mysql-5.5.31 ndb-7.2.13)
[mysqld(API)] 4 node(s)
id=4 (not connected, accepting connect from any host)
id=5 (not connected, accepting connect from any host)
id=6 @172.16.1.75 (mysql-5.5.31 ndb-7.2.13)
id=7 @172.16.1.75 (mysql-5.5.31 ndb-7.2.13)
id=8 (not connected, accepting connect from any host)
id=9 (not connected, accepting connect from any host)
從上面可以發現現在172.16.1.74這臺機器的資料節點和sql節點已經連線不上了，172.16.1.75成為了主資料節點，現在在172.16.1.75資料節點上建立一個表，然後恢復172.16.1.74的資料節點，檢視是否能夠把資料同步過來了。
先在172.16.1.75資料節點做如下操作：
mysql> create table ctest3(id int(11)) ;
mysql> show tables;
mysql> create table ctest3(id int(11)) ;
Query OK, 0 rows affected (0.20 sec)
mysql> show tables;
+--------------+
| Tables_in_aa |
+--------------+
| ctest2 |
| ctest3 |
+--------------+
2 rows in set (0.00 sec)
mysql> insert into ctest3 values(1);
Query OK, 1 row affected (0.02 sec)
mysql> select * from ctest3;
+------+
| id |
+------+
| 1 |
+------+
1 row in set (0.01 sec)
然後我們恢復172.16.1.74資料節點，檢視下ctest3資料是否同步過來了。
[[email protected] ~]# mysql -u root
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| aa |
| mysql |
| ndb_3_fs |
| ndbinfo |
| performance_schema |
| test |
+--------------------+
7 rows in set (0.19 sec)
mysql> use aa
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;
+--------------+
| Tables_in_aa |
+--------------+
| ctest2 |
| ctest3 |
+--------------+
2 rows in set (0.00 sec)
mysql> select * from ctest3;
+------+
| id |
+------+
| 1 |
+------+
1 row in set (0.06 sec)
可以看到172.16.1.74資料節點已經把172.16.1.75資料節點的資料同步過來了，說明mysql叢集是沒有問題的。
HA高可用+負載均衡的配置
十、實現高可用和負載均衡(還需要再完善整理)
ldirectord+heartbeat介紹
ldirectord的作用
在Apache伺服器的前端，我們要放置一臺伺服器專門來做負載排程的任務（為了稱呼簡單和便於理解，我們將這樣的負載排程伺服器簡稱為“導演”），
用來把訪問需求分發給兩臺Apache伺服器。這個“導演”的任務，正是由ldirectord來完成的。
“ldirectord”（Linux Director Daemon）可以對服務和物理伺服器進行監測，被廣泛地用於http和https等服務。
它是專門為LVS（Linux Virtual Server）監控而編寫的，
不僅能從heartbeat的配置檔案/etc/ha.d/xxx.cf中讀取所有有關IPVS（IP Virtul Server）路由表配置的資訊，
還可以方便地被heartbeat管理（比如由heartbeat來啟動和停止ldirectord服務）。
heartbeat的工作原理
因為我們要提供“高可用性”，所以要考慮到“導演”突然罷工的情形；因此我們要安排兩個導演，也就是要有兩個排程伺服器節點。
這兩個節點的地位不同，其中一個是主節點，另外一個是輔節點（可以看成是一個“主導演”和一個“副導演”）。
這兩個節點正是用heartbeat來互相監測對方的。heartbeat可以通過乙太網（或者序列介面）來監控節點的“健康”狀況。
如果有多個heartbeat節點（heartbeat 2.0及後續版本已經能夠支援兩個以上節點），我們既可以使用序列線又可以使用乙太網連線它們，
這樣將大大提高系統的可用性。heartbeat的核心功能有兩個部分：心跳監測和資源接管。通過心跳監測，節點之間相互“打招呼”（傳送報文）
來告訴對方自己當前的狀態；如果在指定的時間內沒“聽”到對方“打招呼”（沒收到報文），那麼就認為對方罷工了，
這時heartbeat會自動啟動資源接管模組，執行相關的shell指令碼來接管執行在對方主機上的資源或者服務。
注意：heartbeat+ldirectord要發揮作用，必須保證承擔負載均衡的兩臺伺服器時間是同步的。
具體到我的配置裡就要求172.16.1.74與172.16.1.75伺服器時間同步。
檢視核心是否支援IPVS
[[email protected] init.d]# modprobe -l |grep ipvs
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_dh.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_ftp.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_lblc.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_lblcr.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_lc.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_nq.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_rr.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_sed.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_sh.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_wlc.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_wrr.ko
[[email protected] ipvs]# modprobe -l |grep ipvs
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_dh.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_ftp.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_lblc.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_lblcr.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_lc.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_nq.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_rr.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_sed.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_sh.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_wlc.ko
/lib/modules/2.6.18-274.el5/kernel/net/ipv4/ipvs/ip_vs_wrr.ko
啟用核心的包轉發功能
修改/etc/sysctl.conf
#net.ipv4.ip_forward = 0
net.ipv4.ip_forward = 1
[[email protected] etc]# sysctl -p //使其馬上生效
net.ipv4.ip_forward = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
安裝heartbeat和lidirectord軟體
[[email protected] heartbeat2_x86_64]# ls -l
-rw-r--r-- 1 root root 1691714 11-10 11:31 heartbeat-2.1.4-11.el5.x86_64.rpm
-rw-r--r-- 1 root root 111941 11-10 12:08 heartbeat-ldirectord-2.1.4-11.el5.x86_64.rpm
-rw-r--r-- 1 root root 93297 11-10 11:57 heartbeat-pils-2.1.4-11.el5.x86_64.rpm
-rw-r--r-- 1 root root 190686 11-10 11:56 heartbeat-stonith-2.1.4-11.el5.x86_64.rpm
-rw-r--r-- 1 root root 33839 11-10 12:10 ipvsadm-1.24-12.el5.x86_64.rpm
-rw-r--r-- 1 root root 62324 11-10 11:44 libnet-1.1.6-2.el5.pp.x86_64.rpm
-rw-r--r-- 1 root root 38792 2011-09-10 libtool-ltdl-1.5.22-7.el5_4.x86_64.rpm
-rw-r--r-- 1 root root 171524 2011-09-10 openhpi-libs-2.14.0-5.el5.x86_64.rpm
-rw-r--r-- 1 root root 224092 11-10 12:18 perl-DBD-mysql-4.005-1.el5.rf.x86_64.rpm
-rw-r--r-- 1 root root 860170 11-10 12:19 perl-DBI-1.58-2.el5.rf.x86_64.rpm
-rw-r--r-- 1 root root 93196 2011-09-10 perl-MailTools-1.77-1.el5.centos.noarch.rpm
-rw-r--r-- 1 root root 45282 11-10 12:26 perl-Net-Daemon-0.39-1.el5.rf.noarch.rpm
-rw-r--r-- 1 root root 33187 11-10 12:25 perl-PlRPC-0.2018-1.2.el5.rf.noarch.rpm
-rw-r--r-- 1 root root 32736 2007-07-02 perl-TimeDate-1.16-5.el5.noarch.rpm
[[email protected] heartbeat2_x86_64]# rpm -ivh *.rpm
[[email protected] heartbeat2_x86_64]# ls
heartbeat-2.1.4-11.el5.x86_64.rpm ipvsadm-1.24-12.el5.x86_64.rpm perl-DBD-mysql-4.005-1.el5.rf.x86_64.rpm perl-PlRPC-0.2018-1.2.el5.rf.noarch.rpm
heartbeat-ldirectord-2.1.4-11.el5.x86_64.rpm libnet-1.1.6-2.el5.pp.x86_64.rpm perl-DBI-1.58-2.el5.rf.x86_64.rpm perl-TimeDate-1.16-5.el5.noarch.rpm
heartbeat-pils-2.1.4-11.el5.x86_64.rpm libtool-ltdl-1.5.22-7.el5_4.x86_64.rpm perl-MailTools-1.77-1.el5.centos.noarch.rpm
heartbeat-stonith-2.1.4-11.el5.x86_64.rpm openhpi-libs-2.14.0-5.el5.x86_64.rpm perl-Net-Daemon-0.39-1.el5.rf.noarch.rpm
[[email protected] heartbeat2_x86_64]# rpm -ivh *.rpm
配置heartbeat
[[email protected] etc]# vi /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
172.16.1.75 db2.test.com db2
172.16.1.74 db1.test.com db1
172.16.1.78 demo.test.com demo
172.16.1.73 db.test.com db #後面要專門設定這樣一個虛擬IP地址
[[email protected] ~]# vi /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
172.16.1.74 db1.test.com db1
172.16.1.75 db2.test.com db2
172.16.1.78 demo.test.com demo
172.16.1.73 db.test.com db #後面要專門設定這樣一個虛擬IP地址
複製一些檔案
[[email protected] doc]# cd /usr/share/doc/heartbeat-2.1.4/
[[email protected] heartbeat-3.0.3]# ls
apphbd.cf authkeys AUTHORS ChangeLog COPYING COPYING.LGPL ha.cf haresources README
[[email protected] heartbeat-3.0.3]# cp ha.cf /etc/ha.d/
[[email protected] heartbeat-3.0.3]# cp haresources /etc/ha.d/
[[email protected] heartbeat-3.0.3]# cp authkeys /etc/ha.d/
[[email protected] heartbeat-3.0.3]# ls
apphbd.cf authkeys AUTHORS ChangeLog COPYING COPYING.LGPL ha.cf haresources README
[[email protected] heartbeat-3.0.3]# cp ha.cf /etc/ha.d/
[[email protected] heartbeat-3.0.3]# cp haresources /etc/ha.d/
[[email protected] heartbeat-3.0.3]# cp authkeys /etc/ha.d/
vi /etc/ha.d/ha.cf新增如下內容
debugfile /var/log/ha.log
logfile /var/log/ha.log
logfacility local0
bcast eth0 # Linux
mcast eth0 225.0.0.1 694 1 0
auto_failback off
node db1
node db2
respawn hacluster /usr/lib/heartbeat/ipfail #64位系統此處應該是/usr/lib64/heartbeat/ipfail
apiauth ipfail gid=haclient uid=hacluster
keepalive 1
deadtime 10
#warntime 10
initdead 120
#udpport 694
#baud 19200
64位系統需要執行下面的操作：
[[email protected] lib]# mkdir heartbeat
[[email protected] lib]# ln -s /usr/lib64/heartbeat/ipfail /usr/lib/heartbeat/ipfail
vi /etc/ha.d/haresources 設定虛擬IP地址
db1.test.com ldirectord::ldirectord.cf LVSSyncDaemonSwap::master IPaddr::172.16.1.73/24/eth0/172.16.1.255
vi /etc/ha.d/authkeys
auth 3
3 md5 Hello!
[[email protected] ha.d]# chmod 600 authkeys
[[email protected] ha.d]# chmod 600 authkeys
上述三個檔案的配置db1、db2節點要相同
配置ldirectord
[[email protected] ha.d]# find / -name ldirectord.cf
/usr/share/doc/ldirectord-1.0.4/ldirectord.cf
[[email protected] ha.d]# cd /usr/share/doc/ldirectord-1.0.4/
[[email protected] ldirectord-1.0.4]# cp ldirectord.cf /etc/ha.d/
[[email protected] ldirectord-1.0.4]# vi /etc/ha.d/ldirectord.cf
配置如下的內容：
# Global Directives
checktimeout=10
checkinterval=2
#fallback=127.0.0.1:80
#fallback6=[::1]:80
autoreload=yes
logfile="/var/log/ldirectord.log"
#logfile="local0"
#emailalert="[email protected]"
#emailalertfreq=3600
#emailalertstatus=all
quiescent=yes
# Sample for an http virtual service
virtual=172.16.1.73:3306
real=172.16.1.74:3306 gate
real=172.16.1.75:3306 gate
# real=192.168..: gate
# fallback=127.0.0.1:3306 gate
service=mysql
scheduler=wrr
#persistent=600
#netmask=255.255.255.255
protocol=tcp
checktype=negotiate
# checkport=80
login="ldirectord"
passwd="ldirectord"
database="ldirectordb"
request="select * from ldirectordtest"
receive="ldirectord is OK"
# virtualhost=www.x.y.z
阻止ldirectord服務開機啟動，推遲heartbeat啟動的操作怎麼弄呢？讓hearbeat來控制ldirectord
/sbin/chkconfig --level 2345 heartbeat on
/sbin/chkconfig --del ldirectord
還有怎樣才能修改heartbeat啟動時候的優先級別呢？
目前的方法是/etc/rc.d/init.d/目錄下的heatbeat的指令碼的註釋
# chkconfig: - 75 05
# description: Startup script high-availability services.
# processname: heartbeat
# pidfile: /var/run/heartbeat.pid
# config: /etc/ha.d/ha.cf
建立資料庫和測試表
mysql>GRANT ALL ON ldirectordb.* TO 'ldirectord'@'%' IDENTIFIED BY 'ldirectord';
mysql>GRANT ALL ON ldirectordb.* TO 'ldirectord'@'localhost' IDENTIFIED BY 'ldirectord';
mysql>GRANT ALL ON ldirectordb.* TO 'ldirectord'@'db1' IDENTIFIED BY 'ldirectord';
mysql>GRANT ALL ON ldirectordb.* TO 'ldirectord'@'db2' IDENTIFIED BY 'ldirectord';
mysql>FLUSH PRIVILEGES;
mysql>CREATE DATABASE ldirectordb;
mysql>USE ldirectordb;
mysql>CREATE TABLE ldirectordtest (Status INT) ;
mysql>INSERT INTO ldirectordtest() VALUES (1);
mysql>quit
先設定IP路由
檢查是否已經安裝了iproute包
[[email protected] mysql]# rpm -qa |grep iproute
iproute-2.6.18-11.el5
修改/etc/sysctl.conf檔案
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.eth0.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.eth0.arp_announce = 2
sysctl -p
配置虛擬IP地址，虛擬IP應該使用lo:0來配置，不應該使用eth0:0來配置，使用eth0:0來配置會導致虛擬IP地址在區域網內衝突，使用lo:0
不會有這樣情況發生。
/etc/sysconfig/network-scripts/
db1伺服器lo:0的設定（虛擬IP地址）
DEVICE=lo:0
IPADDR=172.16.1.73
NETMASK=255.255.255.255
ONBOOT=yes
db2上設定lo:0的IP地址（虛擬IP地址）
DEVICE=lo:0
IPADDR=172.16.1.73
NETMASK=255.255.255.255
ONBOOT=yes
[[email protected] network-scripts]# ifup ifcfg-lo:0
db2.test.com上執行同樣的配置和操作即可。
高可用性和負載均衡測試
[[email protected] init.d]# ldirectord stop
[[email protected] init.d]# heartbeat start
執行上面命令若沒有報錯，就表示一切正常
重啟兩臺伺服器db1和db2
ldirectord狀態檢查
[[email protected] ~]# ldirectord ldirectord.cf status
ldirectord for /etc/ha.d/ldirectord.cf is running with pid: 6203
[[email protected] ~]# ldirectord ldirectord.cf status
ldirectord is stopped for /etc/ha.d/ldirectord.cf
heartbeat狀態檢查
[[email protected] log]# service heartbeat status
heartbeat OK [pid 5750 et al] is running on db1.test.com [db1.test.com]...
[[email protected] log]# service heartbeat status
heartbeat OK [pid 5796 et al] is running on db2.test.com [db2.test.com]...
----錯誤問題解決
[[email protected] init.d]# ./ldirectord status
ldirectord stale pid file /var/run/ldirectord.ldirectord.pid for /etc/ha.d/ldirectord.cf
ldirectord is stopped for /etc/ha.d/ldirectord.cf
此問題已經解決了，是因為缺少包造成的。缺少perl-DBI-mysql、perl-DBD-mysql
虛擬IP狀態檢查
[[email protected] log]# ip addr sh eth0
4: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 90:e2:ba:01:fe:24 brd ff:ff:ff:ff:ff:ff
inet 172.16.1.75/24 brd 172.16.1.255 scope global eth0
inet 172.16.1.73/24 brd 172.16.1.255 scope global secondary eth0:0
inet6 fe80::92e2:baff:fe01:fe24/64 scope link
valid_lft forever preferred_lft forever
[[email protected] log]# ip addr sh eth0
4: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 90:e2:ba:01:fa:fa brd ff:ff:ff:ff:ff:ff
inet 172.16.1.74/24 brd 172.16.1.255 scope global eth0
inet6 fe80::92e2:baff:fe01:fafa/64 scope link
valid_lft forever preferred_lft forever
IPVS狀態檢查
[[email protected] log]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.16.1.73:3306 wrr
-> 172.16.1.74:3306 Route 1 0 0
-> 172.16.1.75:3306 Local 0 0 0
[[email protected] log]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
db1上執行./LVSSyncDaemonSwap master status
[[email protected] resource.d]# ./LVSSyncDaemonSwap master status
master running
(ipvs_syncmaster pid: 4043)
db1上執行./LVSSyncDaemonSwap master status
[[email protected] ha.d]# resource.d/LVSSyncDaemonSwap master status
master stopped
[[email protected] log]# mysql -u ldirectord -p -h 172.16.1.73
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 492
Server version: 5.5.31-ndb-7.2.13-cluster-gpl MySQL Cluster Community Server (GPL)
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| ldirectordb |
| test |
+--------------------+
3 rows in set (0.00 sec)
mysql> use ldirectordb;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> select * from ldirectordtest;
+--------+
| Status |
+--------+
| 1 |
+--------+
1 row in set (0.00 sec)
mysql>
[[email protected] log]# mysql -u ldirectord -p -h 172.16.1.73
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 492
Server version: 5.5.31-ndb-7.2.13-cluster-gpl MySQL Cluster Community Server (GPL)
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| ldirectordb |
| test |
+--------------------+
3 rows in set (0.00 sec)
mysql> use ldirectordb;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> select * from ldirectordtest;
+--------+
| Status |
+--------+
| 1 |
+--------+
1 row in set (0.00 sec)
參考文件：
http://wenku.baidu.com/view/0054090e7cd184254b353546.html
http://blog.csdn.net/sdd00001/article/details/21464713
http://www.linuxfly.org/post/531/
http://blog.csdn.net/tongle_deng/article/details/7297597