Keepalived
由於在生產環境使用了mysqlcluster,需要實現高可用負載均衡,這裡提供了keepalived+haproxy來實現.
keepalived主要功能是實現真實機器的故障隔離及負載均衡器間的失敗切換.可在第3,4,5層交換.它通過VRRPv2(Virtual Router Redundancy Protocol) stack實現的.
Layer3:Keepalived會定期向服務器群中的服務器.發送一個ICMP的數據包(既我們平時用的Ping程序),如果發現某台服務的IP地址沒有激活,Keepalived便報告這台服務器失效,並將它從服務器群中剔除,這種情況的典型例子是某台服務器被非法關機。Layer3的方式是以服務器的IP地址是否有效作為服務器工作正常與否的標准。
Layer4:主要以TCP端口的狀態來決定服務器工作正常與否。如web server的服務端口一般是80,如果Keepalived檢測到80端口沒有啟動,則Keepalived將把這台服務器從服務器群中剔除。
Layer5:在網絡上占用的帶寬也要大一些。Keepalived將根據用戶的設定檢查服務器程序的運行是否正常,如果與用戶的設定不相符,則Keepalived將把服務器從服務器群中剔除。
Software Design
keepalived啟動後會有單個進程
8352 ? Ss 0:00 /usr/sbin/keepalived 8353 ? S 0:00 \_ /usr/sbin/keepalived 8356 ? S 0:01 \_ /usr/sbin/keepalived
父進程:內存管理,子進程管理等等
子進程:VRRP子進程
子進程:Healthchecking 子進程
實例
2台mysqlcluster 10.1.6.203 master 10.1.6.205 backup
vip 10.1.6.173
目的訪問10.1.6.173 3366端口 分別輪詢通過haproxy轉發到10.1.6.203 3306 和10.1.6.205 3306
mysqlcluster搭建參照之前博客,這裡在2台機上安裝keepalived
[email protected]:~# apt-get install keepalived [email protected]:~# cat /etc/keepalived/keepalived.conf vrrp_script chk_haproxy { script "killall -0 haproxy" # verify the pid existance interval 2 # check every 2 seconds weight -2 # add 2 points of prio if OK } vrrp_instance VI_1 { interface eth1 # interface to monitor state MASTER virtual_router_id 51 # Assign one ID for this route priority 101 # 101 on master, 100 on backup nopreempt debug virtual_ipaddress { 10.1.6.173 } track_script { #注意大括號空格 chk_haproxy } notify_master /etc/keepalived/scripts/start_haproxy.sh #表示當切換到master狀態時,要執行的腳本 notify_fault /etc/keepalived/scripts/stop_keepalived.sh #故障時執行的腳本 notify_stop /etc/keepalived/scripts/stop_haproxy.sh #keepalived停止運行前運行notify_stop指定的腳本 }
VRRPD配置包括三個類:
這裡使用了 VRRP實例, VRRP腳本
注意配置選項:
stat:指定instance(Initial)的初始狀態,就是說在配置好後,這台服務器的初始狀態就是這裡指定的,但這裡指定的不算,還是得要通過競選通過優先級來確定,裡如果這裡設置為master,但如若他的優先級不及另外一台,那麼這台在發送通告時,會發送自己的優先級,另外一台發現優先級不如自己的高,那麼他會就回搶占為master
vrrp_script chk_haproxy { script "killall -0 haproxy" # verify the pid existance interval 2 # check every 2 seconds 腳本執行間隔 weight -2 # add 2 points of prio if OK 腳本結果導致的優先級變更:2表示優先級+2;-2則表示優先級-2 }
然後在實例(vrrp_instance)裡面引用,有點類似腳本裡面的函數引用一樣:先定義,後引用函數名
track_script { chk_haproxy }
注意:VRRP腳本(vrrp_script)和VRRP實例(vrrp_instance)屬於同一個級別
[email protected]:scripts# cat start_haproxy.sh #!/bin/bash sleep 5 get=`ip addr |grep 10.1.6.173 |wc -l` echo $get >> /etc/keepalived/scripts/start_ha.log if [ $get -eq 1 ] then echo "`date +%c` success to get vip" >> /etc/keepalived/scripts/start_ha.log /usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg else echo "`date +%c` can not get vip" >> /etc/keepalived/scripts/start_ha.log fi [email protected]:scripts# cat stop_keepalived.sh #!/bin/bash pid=`pidof keepalived` if [ $pid == "" ] then echo "`date +%c` no keepalived process id" >> /etc/keepalived/scripts/stop_keep.log else echo "`date +%c` will stop keepalived " >> /etc/keepalived/scripts/stop_keep.log /etc/init.d/keepalived stop fi /etc/init.d/keepalived stop [email protected]:scripts# cat stop_haproxy.sh #!/bin/bash pid=`pidof haproxy` echo "`date +%c` stop haproxy" >> /etc/keepalived/scripts/stop_ha.log kill -9 $pid
同理配置10.1.6.205
[email protected]:~# cat /etc/keepalived/keepalived.conf vrrp_script chk_haproxy { script "killall -0 haproxy" # verify the pid existance interval 2 # check every 2 seconds weight 2 # add 2 points of prio if OK } vrrp_instance VI_1 { interface eth1 # interface to monitor state BACKUP virtual_router_id 51 # Assign one ID for this route priority 100 # 101 on master, 100 on backup virtual_ipaddress { 10.1.6.173 } track_script { chk_haproxy } notify_master /etc/keepalived/scripts/start_haproxy.sh notify_fault /etc/keepalived/scripts/stop_keepalived.sh notify_stop /etc/keepalived/scripts/stop_haproxy.sh }
HAProxy
下面再介紹下haproxy
HAProxy是一款基於TCP(第四層)和HTTP(第七層)應用的代理軟件,它也可作為負載均衡器.可以支持數以萬計的並發連接.同時可以保護服務器不暴露到網絡上,通過端口映射.它還自帶監控服務器狀態的頁面.
安裝haproxy
wget -O/tmp/haproxy-1.4.22.tar.gz http://haproxy.1wt.eu/download/1.4/src/haproxy-1.4.22.tar.gz tar xvfz /tmp/haproxy-1.4.22.tar.gz -C /tmp/ cd /tmp/haproxy-1.4.22 make TARGET=linux26 make install
haproxy需要對每一個mysqlcluster服務器進行健康檢查
1.在2台主機分別配置haproxy.cfg
[email protected]:scripts# cat /etc/haproxy/haproxy.cfg global maxconn 51200 #默認最大連接數 #uid 99 #gid 99 daemon #以後台形式運行haproxy #quiet nbproc 1 #進程數量(可以設置多個進程提高性能) pidfile /etc/haproxy/haproxy.pid #haproxy的pid存放路徑,啟動進程的用戶必須有權限訪問此文件 defaults mode tcp #所處理的類別 (#7層 http;4層tcp ) option redispatch #serverId對應的服務器掛掉後,強制定向到其他健康的服務器 option abortonclose #當服務器負載很高的時候,自動結束掉當前隊列處理比較久的連接 timeout connect 5000s #連接超時 timeout client 50000s #客戶端超時 timeout server 50000s #服務器超時 log 127.0.0.1 local0 #錯誤日志記錄 balance roundrobin #默認的負載均衡的方式,輪詢方式 listen proxy bind 10.1.6.173:3366 #監聽端口 mode tcp #http的7層模式 option httpchk #心跳檢測的文件 server db1 10.1.6.203:3306 weight 1 check port 9222 inter 12000 rise 3 fall 3 #服務器定義,check inter 12000是檢測心跳頻率 rise 3是3次正確認為服務器可用, fall 3是3次失敗認為服務器不可用,weight代表權重 server db2 10.1.6.205:3306 weight 1 check port 9222 inter 12000 rise 3 fall 3 listen haproxy_stats mode http bind 10.1.6.173:8888 option httplog stats refresh 5s stats uri /status #網站健康檢測URL,用來檢測HAProxy管理的網站是否可以用,正常返回200,不正常返回503 stats realm Haproxy Manager stats auth admin:p@a1SZs24 #賬號密碼 [email protected]:~$ cat /etc/haproxy/haproxy.cfg global maxconn 51200 #uid 99 #gid 99 daemon #quiet nbproc 1 pidfile /etc/haproxy/haproxy.pid defaults mode tcp option redispatch option abortonclose timeout connect 5000s timeout client 50000s timeout server 50000s log 127.0.0.1 local0 balance roundrobin listen proxy bind 10.1.6.173:3366 mode tcp option httpchk server db1 10.1.6.203:3306 weight 1 check port 9222 inter 12000 rise 3 fall 3 server db2 10.1.6.205:3306 weight 1 check port 9222 inter 12000 rise 3 fall 3 listen haproxy_stats mode http bind 10.1.6.173:8888 option httplog stats refresh 5s stats uri /status stats realm Haproxy Manager stats auth admin:p@a1SZs24
2.安裝xinetd
[email protected]:~# apt-get install xinetd
3.在每個節點添加xinetd服務腳本和mysqlchk端口號
[email protected]:~# vim /etc/xinetd.d/mysqlchk # default: on # description: mysqlchk service mysqlchk #需要在servive定義 { flags = REUSE socket_type = stream port = 9222 wait = no user = nobody server = /opt/mysqlchk log_on_failure += USERID disable = no per_source = UNLIMITED bind = 10.1.6.173 } [email protected]:~# vim /etc/services mysqlchk 9222/tcp # mysqlchk
4.編寫mysqlchk監控服務腳本
[email protected]:~# ls -l /opt/mysqlchk -rwxr--r-- 1 nobody root 1994 2013-09-17 11:27 /opt/mysqlchk [email protected]:~# cat /opt/mysqlchk #!/bin/bash # # This script checks if a mysql server is healthy running on localhost. It will # return: # "HTTP/1.x 200 OK\r" (if mysql is running smoothly) # - OR - # "HTTP/1.x 500 Internal Server Error\r" (else) # # The purpose of this script is make haproxy capable of monitoring mysql properly # MYSQL_HOST="localhost" MYSQL_SOCKET="/var/run/mysqld/mysqld.sock" MYSQL_USERNAME="mysqlchkusr" #該賬戶密碼需要在mysql裡添加 MYSQL_PASSWORD="secret" MYSQL_OPTS="-N -q -A" TMP_FILE="/dev/shm/mysqlchk.$$.out" ERR_FILE="/dev/shm/mysqlchk.$$.err" FORCE_FAIL="/dev/shm/proxyoff" MYSQL_BIN="/opt/mysqlcluster/mysql-cluster-gpl-7.2.6-linux2.6-x86_64/bin/mysql" CHECK_QUERY="select 1" preflight_check() { for I in "$TMP_FILE" "$ERR_FILE"; do if [ -f "$I" ]; then if [ ! -w $I ]; then echo -e "HTTP/1.1 503 Service Unavailable\r\n" echo -e "Content-Type: Content-Type: text/plain\r\n" echo -e "\r\n" echo -e "Cannot write to $I\r\n" echo -e "\r\n" exit 1 fi fi done } return_ok() { echo -e "HTTP/1.1 200 OK\r\n" echo -e "Content-Type: text/html\r\n" echo -e "Content-Length: 43\r\n" echo -e "\r\n" echo -e "<html><body>MySQL is running.</body></html>\r\n" echo -e "\r\n" rm $ERR_FILE $TMP_FILE exit 0 } return_fail() { echo -e "HTTP/1.1 503 Service Unavailable\r\n" echo -e "Content-Type: text/html\r\n" echo -e "Content-Length: 42\r\n" echo -e "\r\n" echo -e "<html><body>MySQL is *down*.</body></html>\r\n" sed -e 's/\n$/\r\n/' $ERR_FILE echo -e "\r\n" rm $ERR_FILE $TMP_FILE exit 1 } preflight_check if [ -f "$FORCE_FAIL" ]; then echo "$FORCE_FAIL found" > $ERR_FILE return_fail; fi $MYSQL_BIN $MYSQL_OPTS --host=$MYSQL_HOST --socket=$MYSQL_SOCKET --user=$MYSQL_USERNAME --password=$MYSQL_PASSWORD -e "$CHECK_QUERY" > $TMP_FILE 2> $ERR_FILE if [ $? -ne 0 ]; then return_fail; fi return_ok;
測試
2個節點開啟keepalived(主節點會獲得vip,自動拉起haproxy),xinetd
[email protected]:~# ip add 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo 2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000 link/ether 00:26:b9:36:0f:81 brd ff:ff:ff:ff:ff:ff inet 211.151.105.186/26 brd 211.151.105.191 scope global eth0 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:26:b9:36:0f:83 brd ff:ff:ff:ff:ff:ff inet 10.1.6.203/24 brd 10.1.6.255 scope global eth1 inet 10.1.6.173/32 scope global eth1 4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 link/ether 00:26:b9:36:0f:85 brd ff:ff:ff:ff:ff:ff 5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 link/ether 00:26:b9:36:0f:87 brd ff:ff:ff:ff:ff:ff [email protected]:~# netstat -tunlp | grep ha tcp 0 0 10.1.6.173:3366 0.0.0.0:* LISTEN 1042/haproxy tcp 0 0 10.1.6.203:8888 0.0.0.0:* LISTEN 1042/haproxy udp 0 0 0.0.0.0:56562 0.0.0.0:* 1042/haproxy [email protected]:~# netstat -tunlp | grep xine tcp 0 0 10.1.6.203:9222 0.0.0.0:* LISTEN 30897/xinetd [email protected]:~# ps -ef | grep haproxy root 1042 1 0 Sep17 ? 00:00:00 /usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg
測試:
通過vip10.1.6.173 3366訪問cluster數據庫(注意賬戶dave權限需要加3個ip10.1.6.203,10.1.6.205,10.1.6.173)
[email protected]:mgm# mysql -udave -p -h 10.1.6.173 -P 3366 Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 1344316 Server version: 5.5.22-ndb-7.2.6-gpl-log MySQL Cluster Community Server (GPL) Type 'help;' or '\h' for help. Type '\c' to clear the buffer. mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | dave | | test | +--------------------+ 3 rows in set (0.01 sec) mysql>
手動分別使keepalive,haproxy,數據庫掛掉.vip10.1.6.173會自動漂到10.1.6.205從上,並不影響vip的訪問
通過vip,haproxy查看各節點狀態
http://10.1.6.173:8888/status