redis master-slave 切換演練
redis master-slave 切換演練
環境IP地址端口屬性192.168.31.2086379master192.168.31.208
6378
slave192.168.31.209
6379
master
192.168.31.209
6378
slave192.168.31.210
6379
master
192.168.31.210
6378
slave
環境變量
PATH=$PATH:$HOME/cpprelease/redis-3.0.2/src/:$HOME/bin
BASE_PATH=/home/beidou_soa/cpprelease/
export PATH
alias redisstart1='cd ~/redis/ && redis-server $BASE_PATH/cfg/redis/redis1.conf && cd -'
alias redisstart2='cd ~/redis/ && redis-server $BASE_PATH/cfg/redis/redis2.conf && cd -'
alias redisstop1='cd ~/redis/ && redis-cli -p 6379 shutdown && cd -'
alias redisstop2='cd ~/redis/ && redis-cli -p 6378 shutdown && cd -'
1.檢查環境[beidou_soa@localhost~]$redis-trib.rbcheck192.168.31.208:6379Connectingtonode192.168.31.208:6379:OKConnectingtonode192.168.31.210:6378:OKConnectingtonode192.168.31.209:6378:OKConnectingtonode192.168.31.210:6379:OKConnectingtonode192.168.31.209:6379:OKConnectingtonode192.168.31.208:6378:OK>>>PerformingClusterCheck(usingnode192.168.31.208:6379)M:273aa3c0416e7d1795ce678d56bd2db148613f7e192.168.31.208:6379slots:10923-16383(5461slots)master1additionalreplica(s)S:08f61dcd66389dae5c39e375d4f52e1defa77ec1192.168.31.210:6378slots:(0slots)slavereplicates40cecda23f32cb3b8ff60752c00514f2d7d9c3d0S:d8e8369bfdbf4f9e0bebd04911181785e3ee1129192.168.31.209:6378slots:(0slots)slavereplicates4f11d4265178d72e0ccf7edf0ddabf835e9c56dfM:40cecda23f32cb3b8ff60752c00514f2d7d9c3d0192.168.31.210:6379slots:0-5460(5461slots)master1additionalreplica(s)M:4f11d4265178d72e0ccf7edf0ddabf835e9c56df192.168.31.209:6379slots:5461-10922(5462slots)master1additionalreplica(s)S:4b6e2b13b1be1a081db2153dc4beaf430b489605192.168.31.208:6378slots:(0slots)slavereplicates273aa3c0416e7d1795ce678d56bd2db148613f7e[OK]Allnodesagreeaboutslotsconfiguration.>>>Checkforopenslots...>>>Checkslotscoverage...[OK]All16384slotscovered.
[beidou_soa@localhost~]$redis-cli-c-p6379127.0.0.1:6379>127.0.0.1:6379>127.0.0.1:6379>gettestkey001(nil)127.0.0.1:6379>127.0.0.1:6379>127.0.0.1:6379>settestkey002testvalue002->Redirectedtoslot[401]locatedat192.168.31.210:6379OK192.168.31.210:6379>192.168.31.210:6379>gettestkey002"testvalue002"192.168.31.210:6379>settestkey003testvalue003OK
2.准備關閉208的master[beidou_soa@localhost~]$redisstop1
3.檢查集群狀態查看slave的日志[beidou_soa@localhostredis]$vimredis-6378.log6874:S30Dec15:28:04.755#ErrorconditiononsocketforSYNC:Connectionrefused6874:S30Dec15:28:05.758*ConnectingtoMASTER192.168.31.208:63796874:S30Dec15:28:05.758*MASTER<->SLAVEsyncstarted6874:S30Dec15:28:05.759#ErrorconditiononsocketforSYNC:Connectionrefused6874:S30Dec15:28:06.647*FAILmessagereceivedfrom40cecda23f32cb3b8ff60752c00514f2d7d9c3d0about273aa3c0416e7d1795ce678d56bd2db148613f7e6874:S30Dec15:28:06.647#Clusterstatechanged:fail6874:S30Dec15:28:06.662#Startofelectiondelayedfor842milliseconds(rank#0,offset105547).6874:S30Dec15:28:06.762*ConnectingtoMASTER192.168.31.208:63796874:S30Dec15:28:06.762*MASTER<->SLAVEsyncstarted6874:S30Dec15:28:06.763#ErrorconditiononsocketforSYNC:Connectionrefused6874:S30Dec15:28:07.565#Startingafailoverelectionforepoch4.6874:S30Dec15:28:07.567#Failoverelectionwon:I'mthenewmaster.6874:S30Dec15:28:07.567#configEpochsetto4aftersuccessfulfailover6874:M30Dec15:28:07.567*Discardingpreviouslycachedmasterstate.6874:M30Dec15:28:07.567#Clusterstatechanged:ok
查看集群狀態[beidou_soa@localhost~]$redis-trib.rbcheck192.168.31.208:6378Connectingtonode192.168.31.208:6378:OKConnectingtonode192.168.31.209:6378:OKConnectingtonode192.168.31.209:6379:OKConnectingtonode192.168.31.210:6378:OKConnectingtonode192.168.31.210:6379:OK>>>PerformingClusterCheck(usingnode192.168.31.208:6378)M:4b6e2b13b1be1a081db2153dc4beaf430b489605192.168.31.208:6378slots:10923-16383(5461slots)master0additionalreplica(s)S:d8e8369bfdbf4f9e0bebd04911181785e3ee1129192.168.31.209:6378slots:(0slots)slavereplicates4f11d4265178d72e0ccf7edf0ddabf835e9c56dfM:4f11d4265178d72e0ccf7edf0ddabf835e9c56df192.168.31.209:6379slots:5461-10922(5462slots)master1additionalreplica(s)S:08f61dcd66389dae5c39e375d4f52e1defa77ec1192.168.31.210:6378slots:(0slots)slavereplicates40cecda23f32cb3b8ff60752c00514f2d7d9c3d0M:40cecda23f32cb3b8ff60752c00514f2d7d9c3d0192.168.31.210:6379slots:0-5460(5461slots)master1additionalreplica(s)[OK]Allnodesagreeaboutslotsconfiguration.>>>Checkforopenslots...>>>Checkslotscoverage...[OK]All16384slotscovered.
4.關閉slave[beidou_soa@localhost~]$redisstop1
5.檢查集群狀態[beidou_soa@localhost~]$redis-trib.rbcheck192.168.31.209:6379Connectingtonode192.168.31.209:6379:OKConnectingtonode192.168.31.210:6379:OKConnectingtonode192.168.31.210:6378:OKConnectingtonode192.168.31.209:6378:OKConnectingtonode192.168.31.208:6378:OK>>>PerformingClusterCheck(usingnode192.168.31.209:6379)M:4f11d4265178d72e0ccf7edf0ddabf835e9c56df192.168.31.209:6379slots:5461-10922(5462slots)master1additionalreplica(s)M:40cecda23f32cb3b8ff60752c00514f2d7d9c3d0192.168.31.210:6379slots:0-5460(5461slots)master1additionalreplica(s)S:08f61dcd66389dae5c39e375d4f52e1defa77ec1192.168.31.210:6378slots:(0slots)slavereplicates40cecda23f32cb3b8ff60752c00514f2d7d9c3d0S:d8e8369bfdbf4f9e0bebd04911181785e3ee1129192.168.31.209:6378slots:(0slots)slavereplicates4f11d4265178d72e0ccf7edf0ddabf835e9c56dfM:4b6e2b13b1be1a081db2153dc4beaf430b489605192.168.31.208:6378slots:10923-16383(5461slots)master0additionalreplica(s)[OK]Allnodesagreeaboutslotsconfiguration.>>>Checkforopenslots...>>>Checkslotscoverage...[OK]All16384slotscovered.
6.關閉3台slave[beidou_soa@localhost~]$redisstop2
[beidou_soa@localhost~]$redisstop2
[beidou_soa@localhost~]$redisstop2
7.檢查集群狀態[beidou_soa@localhost~]$redis-trib.rbcheck192.168.31.209:6379Connectingtonode192.168.31.209:6379:OKConnectingtonode192.168.31.210:6379:OKConnectingtonode192.168.31.208:6378:OK>>>PerformingClusterCheck(usingnode192.168.31.209:6379)M:4f11d4265178d72e0ccf7edf0ddabf835e9c56df192.168.31.209:6379slots:5461-10922(5462slots)master0additionalreplica(s)M:40cecda23f32cb3b8ff60752c00514f2d7d9c3d0192.168.31.210:6379slots:0-5460(5461slots)master0additionalreplica(s)M:4b6e2b13b1be1a081db2153dc4beaf430b489605192.168.31.208:6378slots:10923-16383(5461slots)master0additionalreplica(s)[OK]Allnodesagreeaboutslotsconfiguration.>>>Checkforopenslots...>>>Checkslotscoverage...[OK]All16384slotscovered.
8.在關閉slave的情況下關閉一台master[beidou_soa@localhost~]$redisstop1
9.檢查集群狀態[beidou_soa@localhost~]$redis-trib.rbcheck192.168.31.209:6379Connectingtonode192.168.31.209:6379:OKConnectingtonode192.168.31.210:6379:OK>>>PerformingClusterCheck(usingnode192.168.31.209:6379)M:4f11d4265178d72e0ccf7edf0ddabf835e9c56df192.168.31.209:6379slots:5461-10922(5462slots)master0additionalreplica(s)M:40cecda23f32cb3b8ff60752c00514f2d7d9c3d0192.168.31.210:6379slots:0-5460(5461slots)master0additionalreplica(s)[OK]Allnodesagreeaboutslotsconfiguration.>>>Checkforopenslots...>>>Checkslotscoverage...[ERR]Notall16384slotsarecoveredbynodes.
集群進入fail狀態，不可用
10.在集群down的狀態開啟一台slave[beidou_soa@localhost~]$redisstar2
11.檢查集群狀態[beidou_soa@localhost~]$redis-trib.rbcheck192.168.31.209:6379Connectingtonode192.168.31.209:6379:OKConnectingtonode192.168.31.210:6379:OKConnectingtonode192.168.31.208:6379:OK***WARNING:192.168.31.208:6379claimstobeslaveofunknownnodeID4b6e2b13b1be1a081db2153dc4beaf430b489605.>>>PerformingClusterCheck(usingnode192.168.31.209:6379)M:4f11d4265178d72e0ccf7edf0ddabf835e9c56df192.168.31.209:6379slots:5461-10922(5462slots)master0additionalreplica(s)M:40cecda23f32cb3b8ff60752c00514f2d7d9c3d0192.168.31.210:6379slots:0-5460(5461slots)master0additionalreplica(s)S:273aa3c0416e7d1795ce678d56bd2db148613f7e192.168.31.208:6379slots:(0slots)slavereplicates4b6e2b13b1be1a081db2153dc4beaf430b489605[OK]Allnodesagreeaboutslotsconfiguration.>>>Checkforopenslots...>>>Checkslotscoverage...[ERR]Notall16384slotsarecoveredbynodes
集群在fail狀態，無法自動選舉master，集群不可用
12開啟master，檢查集群狀態[beidou_soa@localhost~]$redis-trib.rbcheck192.168.31.209:6379Connectingtonode192.168.31.209:6379:OKConnectingtonode192.168.31.210:6379:OKConnectingtonode192.168.31.208:6379:OKConnectingtonode192.168.31.208:6378:OK>>>PerformingClusterCheck(usingnode192.168.31.209:6379)M:4f11d4265178d72e0ccf7edf0ddabf835e9c56df192.168.31.209:6379slots:5461-10922(5462slots)master0additionalreplica(s)M:40cecda23f32cb3b8ff60752c00514f2d7d9c3d0192.168.31.210:6379slots:0-5460(5461slots)master0additionalreplica(s)S:273aa3c0416e7d1795ce678d56bd2db148613f7e192.168.31.208:6379slots:(0slots)slavereplicates4b6e2b13b1be1a081db2153dc4beaf430b489605M:4b6e2b13b1be1a081db2153dc4beaf430b489605192.168.31.208:6378slots:10923-16383(5461slots)master1additionalreplica(s)[OK]Allnodesagreeaboutslotsconfiguration.>>>Checkforopenslots...>>>Checkslotscoverage...[OK]All16384slotscovered.
13.關閉一台master[beidou_soa@localhost~]$redisstop1
14.檢查集群狀態[beidou_soa@localhost~]$redis-trib.rbcheck192.168.31.209:6379Connectingtonode192.168.31.209:6379:OKConnectingtonode192.168.31.210:6379:^[[AOKConnectingtonode192.168.31.208:6379:OK***WARNING:192.168.31.208:6379claimstobeslaveofunknownnodeID4b6e2b13b1be1a081db2153dc4beaf430b489605.>>>PerformingClusterCheck(usingnode192.168.31.209:6379)M:4f11d4265178d72e0ccf7edf0ddabf835e9c56df192.168.31.209:6379slots:5461-10922(5462slots)master0additionalreplica(s)M:40cecda23f32cb3b8ff60752c00514f2d7d9c3d0192.168.31.210:6379slots:0-5460(5461slots)master0additionalreplica(s)S:273aa3c0416e7d1795ce678d56bd2db148613f7e192.168.31.208:6379slots:(0slots)slavereplicates4b6e2b13b1be1a081db2153dc4beaf430b489605[OK]Allnodesagreeaboutslotsconfiguration.>>>Checkforopenslots...>>>Checkslotscoverage...[ERR]Notall16384slotsarecoveredbynodes.
預計1分鐘之後
[beidou_soa@localhost~]$redis-trib.rbcheck192.168.31.209:6379Connectingtonode192.168.31.209:6379:OKConnectingtonode192.168.31.210:6379:OKConnectingtonode192.168.31.208:6379:OK>>>PerformingClusterCheck(usingnode192.168.31.209:6379)M:4f11d4265178d72e0ccf7edf0ddabf835e9c56df192.168.31.209:6379slots:5461-10922(5462slots)master0additionalreplica(s)M:40cecda23f32cb3b8ff60752c00514f2d7d9c3d0192.168.31.210:6379slots:0-5460(5461slots)master0additionalreplica(s)M:273aa3c0416e7d1795ce678d56bd2db148613f7e192.168.31.208:6379slots:10923-16383(5461slots)master0additionalreplica(s)[OK]Allnodesagreeaboutslotsconfiguration.>>>Checkforopenslots...>>>Checkslotscoverage...[OK]All16384slotscovered.
選舉完成。redis集群恢復。
redis架構
架構細節:
(1)所有的redis節點彼此互聯(PING-PONG機制),內部使用二進制協議優化傳輸速度和帶寬.
(2)節點的fail是通過集群中超過半數的節點檢測失效時才生效.
(3)客戶端與redis節點直連,不需要中間proxy層.客戶端不需要連接集群所有節點,連接集群中任何一個可用節點即可
(4)redis-cluster把所有的物理節點映射到[0-16383]slot上,cluster 負責維護node<->slot<->value
2) redis-cluster選舉:容錯
(1)領著選舉過程是集群中所有master參與,如果半數以上master節點與master節點通信超過(cluster-node-timeout),認為當前master節點掛掉.
(2):什麼時候整個集群不可用(cluster_state:fail),當集群不可用時,所有對集群的操作做都不可用，收到((error) CLUSTERDOWN The cluster is down)錯誤
a:如果集群任意master掛掉,且當前master沒有slave.集群進入fail狀態,也可以理解成進群的slot映射[0-16383]不完成時進入fail狀態.
b:如果進群超過半數以上master掛掉，無論是否有slave集群進入fail狀態.
