程式師世界 >> 數據庫知識 >> MYSQL數據庫 >> 關於MYSQL數據庫 >> Heartbeat + Drbd +Mysql 構建高可用的MYSQL數據庫服務

Heartbeat + Drbd +Mysql 構建高可用的MYSQL數據庫服務

編輯：關於MYSQL數據庫

1，什麼是DRBD
是由內核模塊和相關腳本而構成，用以構建高可用性的集群。其實現方式是通過網絡來鏡像整個設備。您可以把它看作是一種網絡RAID1Drbd 負責接收數據，把數據寫到本地磁盤，然後發送給另一個主機。另一個主機再將數據存到自己的磁盤中。其他所需的組件有集群成員服務，如TurboHA 或心跳連接，以及一些能在塊設備上運行的應用程序
環境
CentOS 5.2
Eth0：192.168.0.251
Eth1：192.168.254.4
CentOS 5.2
Eth0：192.168.0.252
Eth1：192.168.254.52. /etc/hosts 文件中新增加的內容，兩邊一樣ha1:192.168.254.4
ha2:192.168.254.5OS:CentOS 5.2
軟件：drbd8.2 kmod-8.2
安裝drbd 及drbd內核模塊#

yum -y install drbd82 kmod-drbd82 復制代碼

3.修改配置文件
#mv /etc/drbd.conf /etc/drbd.conf.bak 備份原配置文件
#vim /etc/drbd.conf

global { usage-count yes; }

common { syncer { rate 10M; } } //如果是100M網絡，就寫100

resource r0 {

protocol C;

net {

cram-hMac-alg sha1;

shared-secret "FooFunFactory";

}

on ha1 {

device /dev/drbd0;

disk /dev/sdb1;

address 192.168.254.4:7898;

meta-disk internal;

}

on ha2 {

device /dev/drbd0;

disk /dev/sdb1;

address 192.168.254.5:7898;

meta-disk internal;

}

復制代碼

我的是很簡單的，為了測試，還需要大家根據自己的需要自己修改~好了兩台機器都寫完配置文件後，兩邊的drbd.conf 配置一模一樣在ha1和ha2上分別輸入命令#

drbdadm create-md r0 復制代碼

“r0"是我們在drbd.conf裡定義
彈出的信息，敲回車，如果drbd.conf配置好的話，有的信息是不會彈出的。
HA1上[root@ha1 etc]#

drbdadm create-md r0 復制代碼

v08 Magic number not found
v07 Magic number not found
v07 Magic number not found
v08 Magic number not found
Writing meta data...
initialising activity log
NOT initialized bitmap
New drbd meta data block sucessfully created.現在我們可以啟動DRBD了,分別在兩台主機上執行:
[root@ha1 etc]# /etc/init.d/drbd start
Starting DRBD resources: [ d(r0) s(r0) n(r0) ].
..........
***************************************************************
DRBD's startup script waits for the peer node(s) to appear.
- In case this node was already a degraded cluster before the
reboot the timeout is 0 seconds. [degr-wfc-timeout]
- If the peer was available before the reboot the timeout will
expire after 0 seconds. [wfc-timeout]
(These values are for resource 'r0'; 0 sec -> wait forever)
To abort waiting enter 'yes' [ 23]: //在這個時候你在HA2 上啟動服務，就會馬上成功啟動服務，因為他在等待另一個節點
[root@ha2 etc]# /etc/init.d/drbd start
Starting DRBD resources: [ d(r0) s(r0) n(r0) ].
[root@ha1 etc]# cat /proc/drbd
version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-i386-build, 2008-10-03 11:42:32
0: cs:Connected st:Secondary/Secondary ds:Inconsistent/Inconsistent C r---
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 oos:8385604
[root@ha2 etc]# cat /proc/drbd
version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-i386-build, 2008-10-03 11:42:32
0: cs:Connected st:Secondary/Secondary ds:Inconsistent/Inconsistent C r---
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 oos:8385604

"/proc/drbd"中顯示了drbd當前的狀態.第一行的st表示兩台主機的狀態,都是"備機"狀態.
ds是磁盤狀態,都是"不一致"狀態.
這是由於,DRBD無法判斷哪一方為主機,以哪一方的磁盤數據作為標准數據.所以,我們需要初始化一個主機.在drbd1上執行:
[root@drbd1 local]#

drbdsetup /dev/drbd0 primary -o 復制代碼

主備機狀態分別是"主/備",主機磁盤狀態是"實時",備機狀態是"不一致".
在第3行,可以看到數據正在同步中,即主機正在將磁盤上的數據,傳遞到備機上.現在的進度是[>...................] sync'ed: 0.4% (1040316/1040316)K
稍等一會,在數據同步完後,再查看一下ha1的DRBD狀態:
磁盤狀態都是"實時",表示數據同步完成了.
你現在可以把主機上的DRBD設備掛載到一個目錄上進行使用.備機的DRBD設備無法被掛載,因為它是
用來接收主機數據的,由DRBD負責操作.
在drbd1執行
root@ha1:~#

mkfs.ext3 /dev/drbd0 復制代碼

//網上這裡有很多不同的版本，具體drbd深入我還沒有研究，我現在安裝的版本是只要在第一台上面格式化文件系統，他會自動傳過去。如果你在ha2上格式化，他會提示出錯，而且格式化要在上面相應工作做完以後才可以到這一步。root@ha1:~#

mount /dev/drbd0 /mnt 復制代碼

root@ha1:~#

cd /mnt 復制代碼

root@ha1:/mnt#

touch huzi 復制代碼

root@dha1:/mnt# ls
huzi lost+found可以看到,在主機drbd1上產生的文件huzi,也完整的保存在備機drBD2的DRBD分區上.
這就是DRBD的網絡RAID-1功能. 在主機上的任何操作,都會被同步到備機的相應磁盤分區上,達到數據備份的效果.DRBD的主備機切換有時,你需要將DRBD的主備機互換一下.可以執行下面的操作:
在主機上,先要卸載掉DRBD設備
root@drbd1~#

umount /mnt 復制代碼

將主機降級為"備機"
root@ha1~#

drbdadm secondary r0 復制代碼

[root@ha1 ~]#

cat /proc/drbd 復制代碼

version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-i386-build, 2008-10-03 11:42:32
0: cs:Connected st:Secondary/Primary ds:UpToDate/UpToDate C r---
ns:8650688 nr:24 dw:265108 dr:8385749 al:105 bm:584 lo:0 pe:0 ua:0 ap:0 oos:0
現在,兩台主機都是"備機".
在備機drBD2上,將它升級為"主機". [root@ha2 mnt]#

drbdadm primary r0 復制代碼

[root@ha2 etc]#

cat /proc/drbd 復制代碼

version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-i386-build, 2008-10-03 11:42:32
0: cs:Connected strimary/Secondary ds:UpToDate/UpToDate C r---
ns:0 nr:32 dw:32 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 oos:0
[root@ha2 etc]#

mount /dev/drbd0 /mnt 復制代碼

[root@ha2 etc]#

cd /mnt 復制代碼

[root@ha2 mnt]#

ls 復制代碼

huzi lost+found
[root@ha2 mnt]#

ll 復制代碼

total 16
-rw-r--r-- 1 root root 0 Mar 19 11:02 huzi
drwx------ 2 root root 16384 Mar 19 11:01 lost+found
現在drBD2成為主機了。
DRBD相關切換命令DRBD切換 drbdadm secondary r0 //把主機切換成備機drbdadm primary r0 //把主機切換成主機
注意，在我測試的情況。DRBD只有主機可以讀寫，備機不能夠掛載，也就是說不能夠讀。可能是我的drbd.conf文件配置方面的原因，我會繼續找一些文章來讀，這樣對於數據庫是可以滿足的，如果想做WEB。可以有些台只是用來讀取的，不過我們這裡只討論數據庫
------------------------------------------------------------------上面的DRBD已經成功配置完成，可以正常切換，現在我們讓兩台成為高可用，所謂高可用，就是雙機熱備，互備。這裡我不解釋太多，相應搞過HA的人都清楚，不清楚的上上面看看文檔。主機ha1和ha2上安裝heartbeat

yum Cy install heartbeat && yum Cy install heartbeat 復制代碼

//這裡我也不清楚為什麼要運行兩次他才給我安裝heartbeat ，嘿嘿，哪位知道告訴我為什麼，我是每次都要暈兩次才可以我安裝在主機ha1和ha2 上安裝mysql MySQL-server

yum Cy install mysql MySQL-server 復制代碼

以下只在ha1上進行操作
在ha1上啟動MySQL

service MySQLd start 復制代碼

//初始化數據庫數據文件再停止數據庫

service MySQLd stop 復制代碼

mount /dev/drbd0 /mnt 復制代碼

cp /var/lib/MySQL/* /mnt -ar 復制代碼

umount /mnt 復制代碼

配置heartbeat 拷呗 cp /usr/share/doc/heartbeat-2.1.3/ 下面的 ha.cf,authkeys,haresources 到/etc/ha.d 目錄下面

cp /usr/share/doc/heartbeat-2.1.3/haresources /etc/ha.d

cp /usr/share/doc/heartbeat-2.1.3/ha.cf /etc/ha.d

cp /usr/share/doc/heartbeat-2.1.3/authkeys /etc/ha.d 復制代碼

cd /etc/ha.d

chmod 600 authkeys 復制代碼

authkeys文件中下面兩行前面的#號去掉
vi authkeys

auth 1

1 crc 復制代碼

配置ha.cf 內容如下，我就不一一解釋了，有英語注釋，注意的地方我會加注釋

debugfile /var/log/ha-debug

logfile /var/log/ha-log

logfacility local0

keepalive 2

deadtime 10

warntime 5

initdead 120

ucast eth1 192.168.254.5 //這裡是對方的ip 主要用於HA的相互檢測網絡接口要注意了，我用的是專用接口，ha2主機改為192.168.254.4

auto_failback on

watchdog /dev/watchdog

node ha1

node ha2

ping 192.168.254.1

respawn hacluster /usr/lib/heartbeat/ipfail

apiauth ipfail gid=haclIEnt uid=hacluster 復制代碼

添加自動加載drbd 及持載文件系統的腳本，放在/etc/ha.d/resource.d 目錄下面vi MySQLd_umount

#!/bin/sh

# High-Availability Pre-Startup/Shutdown Script

# Description: Runs on Startup or shutdown of heartbeat (not resource based).

# Also runs before start, after start or before stop,

# and after stop.

# Author: Matthew Soffen

# Support: [email protected]

# License: GNU Lesser General Public License (LGPL)

unset LC_ALL; export LC_ALL

unset LANGUAGE; export LANGUAGE

prefix=/usr

exec_prefix=/usr

. /etc/ha.d/shellfuncs

case "$1" in

'start')

#/sbin/drbdadm -- --do-what-I-say primary all

/sbin/drbdadm primary all

#drbdsetup /dev/drbd1 primary -o

/bin/mount /dev/drbd0 /var/lib/MySQL

;;

'pre-start')

;;

'post-start')

;;

'stop')

/bin/umount /var/lib/MySQL

/sbin/drbdadm secondary all

;;

'pre-stop')

;;

'post-stop')

;;

echo "Usage: $0 { start | pre-start | post-start | stop | pre-stop | post-stop }"

;;

esac

exit 0 復制代碼

配置haresource添加如下一行

ha1 IPaddr::192.168.0.253/24/eth0:1 mysqld_umount MySQLd 復制代碼

工作完成,把/etc/ha.d目錄全部覆蓋到ha2上面去

scp -r /etc/ha.d 192.168.254.5:/etc 復制代碼

提示輸入對方root用戶的密碼，全部轉輸完以後，注意修改/etc/ha.d/ha.cf 中的ucast eth1 192.168.254.5 為對方的IP也就是HA1主機的ETH1的IP 改為ucast eth1 192.168.254.4
兩邊同時啟動heartbeat 觀察日志 HA1上 HA的日志heartbeat[4039]: 2009/03/19_15:11:25 info: heartbeat: version 2.1.3
heartbeat[4039]: 2009/03/19_15:11:25 info: Heartbeat generation: 1237438269
heartbeat[4039]: 2009/03/19_15:11:25 info: glib: ucast: write socket priority set to IPTOS_LOWDELAY on eth1
heartbeat[4039]: 2009/03/19_15:11:25 info: glib: ucast: bound send socket to device: eth1
heartbeat[4039]: 2009/03/19_15:11:25 info: glib: ucast: bound receive socket to device: eth1
heartbeat[4039]: 2009/03/19_15:11:25 info: glib: ucast: started on port 694 interface eth1 to 192.168.254.5
heartbeat[4039]: 2009/03/19_15:11:25 info: glib: ping heartbeat started.
heartbeat[4039]: 2009/03/19_15:11:25 info: G_main_add_TriggerHandler: Added signal manual handler
heartbeat[4039]: 2009/03/19_15:11:25 info: G_main_add_TriggerHandler: Added signal manual handler
heartbeat[4039]: 2009/03/19_15:11:25 notice: Using watchdog device: /dev/watchdog
heartbeat[4039]: 2009/03/19_15:11:25 info: G_main_add_SignalHandler: Added signal handler for signal 17
heartbeat[4039]: 2009/03/19_15:11:25 info: Local status now set to: 'up'
heartbeat[4039]: 2009/03/19_15:11:26 info: Link ha2:eth1 up.
heartbeat[4039]: 2009/03/19_15:11:26 info: Status update for node ha2: status active
heartbeat[4039]: 2009/03/19_15:11:26 info: Link 192.168.254.1:192.168.254.1 up.
heartbeat[4039]: 2009/03/19_15:11:26 info: Status update for node 192.168.254.1: status ping
harc[4048]:    2009/03/19_15:11:26 info: Running /etc/ha.d/rc.d/status status
heartbeat[4039]: 2009/03/19_15:11:27 info: Comm_now_up(): updating status to active
heartbeat[4039]: 2009/03/19_15:11:27 info: Local status now set to: 'active'
heartbeat[4039]: 2009/03/19_15:11:27 info: Starting child clIEnt "/usr/lib/heartbeat/ipfail" (498,496)
heartbeat[4065]: 2009/03/19_15:11:27 info: Starting "/usr/lib/heartbeat/ipfail" as uid 498  gid 496 (pid 4065)
heartbeat[4039]: 2009/03/19_15:11:27 info: remote resource transition completed.
heartbeat[4039]: 2009/03/19_15:11:27 info: remote resource transition completed.
heartbeat[4039]: 2009/03/19_15:11:27 info: Local Resource acquisition completed. (none)
heartbeat[4039]: 2009/03/19_15:11:28 info: ha2 wants to go standby [foreign]
heartbeat[4039]: 2009/03/19_15:11:30 info: standby: acquire [foreign] resources from ha2
heartbeat[4068]: 2009/03/19_15:11:30 info: acquire local HA resources (standby).
ResourceManager[4081]:  2009/03/19_15:11:30 info: Acquiring resource group: ha1 IPaddr::192.168.0.253/24/eth0:1 mysqld_umount MySQLd
IPaddr[4108]: 2009/03/19_15:11:31 INFO:  Resource is stopped
ResourceManager[4081]:  2009/03/19_15:11:31 info: Running /etc/ha.d/resource.d/IPaddr 192.168.0.253/24/eth0:1 start
IPaddr[4206]: 2009/03/19_15:11:31 INFO: Using calculated netmask for 192.168.0.253: 255.255.255.0
IPaddr[4206]: 2009/03/19_15:11:32 INFO: eval ifconfig eth0:0 192.168.0.253 netmask 255.255.255.0 broadcast 192.168.0.255
IPaddr[4177]: 2009/03/19_15:11:32 INFO:  Success
ResourceManager[4081]:  2009/03/19_15:11:32 info: Running /etc/ha.d/resource.d/MySQLd_umount  start
ResourceManager[4081]:  2009/03/19_15:11:33 info: Running /etc/ha.d/resource.d/MySQLd  start
ipfail[4065]: 2009/03/19_15:11:34 info: Ping node count is balanced.
heartbeat[4068]: 2009/03/19_15:11:35 info: local HA resource acquisition completed (standby).
heartbeat[4039]: 2009/03/19_15:11:35 info: Standby resource acquisition done [foreign].
heartbeat[4039]: 2009/03/19_15:11:35 info: Initial resource acquisition complete (auto_failback)
heartbeat[4039]: 2009/03/19_15:11:35 info: remote resource transition completed.通過以上觀察，資源全部加載成功，我們來驗證一下網絡加載
eth0:0 Link encap:Ethernet  HWaddr 00:0C:29:39:A8:2B
      inet addr:192.168.0.253  Bcast:192.168.0.255  Mask:255.255.255.0
      UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
      Interrupt:169 Base address:0x2000 文件系統加載

[root@ha1 resource.d]#

mount 復制代碼

/dev/sda3 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda5 on /data type ext3 (rw)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/dev/drbd0 on /var/lib/MySQL type ext3 (rw)
[root@ha1 resource.d]version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-i386-build, 2008-10-03 11:42:32
0: cs:Connectedst:Primary/Secondaryds:UpToDate/UpToDate C r---
ns:512 nr:1008 dw:1520 dr:5950 al:9 bm:8 lo:0 pe:0 ua:0 ap:0 oos:0DRBD是在主機狀態，MYSQL我就不貼出來，驗證MySQL就沒有必要了，大家自己去驗證吧，我已經驗證過了，其實大家都知道，文件系統都過來了，還驗證個啥子資源的狀態，HA處在備機狀態

heartbeat[4412]: 2009/03/19_13:30:43 info: foreign HA resource release completed (standby).
heartbeat[3795]: 2009/03/19_13:30:43 info: Local standby process completed [foreign].
ipfail[3820]: 2009/03/19_13:30:44 info: No giveup timer to abort.
heartbeat[3795]: 2009/03/19_13:30:45 WARN: 1 lost packet(s) for [ha1] [16:18]
heartbeat[3795]: 2009/03/19_13:30:45 info: remote resource transition completed.
heartbeat[3795]: 2009/03/19_13:30:45 info: No pkts missing from ha1!
heartbeat[3795]: 2009/03/19_13:30:45 info: Other node completed standby takeover of foreign resources.
[root@ha2 ha.d]#

[root@ha2 ha.d]#

cat /proc/drbd 復制代碼

version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-i386-build, 2008-10-03 11:42:32
0: cs:Connected st:Secondary/Primary ds:UpToDate/UpToDate C r---
ns:392 nr:832 dw:1224 dr:2989 al:9 bm:8 lo:0 pe:0 ua:0 ap:0 oos:0
切換測試
HA1上停止HA服務[root@ha1 resource.d]#

service heartbeat stop 復制代碼

Stopping High-Availability services:
                                                         [  OK  ]
[root@ha1 resource.d]# 在HA2上觀察[root@ha2 ha.d]# tail /var/log/ha-log -f
heartbeat[3795]: 2009/03/19_13:55:21 info: Standby resource acquisition done [foreign].
harc[4636]:    2009/03/19_13:55:21 info: Running /etc/ha.d/rc.d/status status
Mach_down[4652]:       2009/03/19_13:55:21 info: Taking over resource group IPaddr::192.168.0.253/24/eth0:1
ResourceManager[4678]:  2009/03/19_13:55:21 info: Acquiring resource group: ha1 IPaddr::192.168.0.253/24/eth0:1 mysqld_umount MySQLd
IPaddr[4705]: 2009/03/19_13:55:22 INFO:  Resource is stopped
ResourceManager[4678]:  2009/03/19_13:55:22 info: Running /etc/ha.d/resource.d/IPaddr 192.168.0.253/24/eth0:1 start
IPaddr[4803]: 2009/03/19_13:55:22 INFO: Using calculated netmask for 192.168.0.253: 255.255.255.0
IPaddr[4803]: 2009/03/19_13:55:22 INFO: eval ifconfig eth0:0 192.168.0.253 netmask 255.255.255.0 broadcast 192.168.0.255
IPaddr[4774]: 2009/03/19_13:55:23 INFO:  Success
ResourceManager[4678]:  2009/03/19_13:55:23 info: Running /etc/ha.d/resource.d/MySQLd_umount  start
ResourceManager[4678]:  2009/03/19_13:55:23 info: Running /etc/ha.d/resource.d/MySQLd  start
mach_down[4652]:       2009/03/19_13:55:25 info: /usr/share/heartbeat/Mach_down: nice_failback: foreign resources acquired
mach_down[4652]:       2009/03/19_13:55:25 info: Mach_down takeover complete for node ha1.
heartbeat[3795]: 2009/03/19_13:55:25 info: Mach_down takeover complete.驗證不貼出來了，通過上面已經正常切換了，在10秒內服務啟動的順序，drbd讓系統自動加載，heartbeat 放在/etc/rc.local裡面，不然會出錯，為什麼自己去想，因為heartbeat 在drbd前面啟動，你可以調整他們兩的啟動順序。MySQLD不要開機自動加載啟動，這裡我把注意的地方寫一下，自己快半年沒弄HA啦。今天做起來有點問題，HA通信的IP是相互對方的IP地址，fence ip 一定要存在，因為我是虛擬機測試的，開始沒注意，花了點時間來查找問題
這個對比AB復制，備機不可以用來讀，但保證了高可用性。DRBD在生產上已經有人使用，如果想達到AB復制效果，再加一台做復制就行了。這個相信大家可以加一台做復制Ｓ惺裁次侍馇敫