Redis3.0.7集群部署完整版

Redis3.0.7集群部署完整版

Redis3.0.7集群安裝部署
Redis集群沒有出來前一直使用Codis集群，現在測試下Redis集群的性能。 一，架構
Centos6.5 64位
redis1redis1:6379主redis3:6380從
redis2redis2:6379主redis1:6380從
redis3redis3:6379主redis2:6380從

二，部署Redis實例
1，安裝依賴
yum -y install tcl-devel
2，下載
wget http://download.redis.io/releases/redis-3.0.7.tar.gz
3，編譯
cd /usr/local/src/
tar zxvf redis-3.0.7.tar.gz
cd redis-3.0.7
make
4，測試
make test
5，安裝二進制程序到/usr/local/bin/目錄
make install
ls /usr/local/bin/ -lh
6，安裝服務
sh ./utils/install_server.sh
Welcome to the redis service installer
This script will help you easily set up a running redis server


Please select the redis port for this instance: [6379]
Selecting default: 6379
Please select the redis config file name [/etc/redis/6379.conf]
Selected default - /etc/redis/6379.conf
Please select the redis log file name [/var/log/redis_6379.log] /data/redis/redis_6379.log
Please select the data directory for this instance [/var/lib/redis/6379] /data/redis/6379
Please select the redis executable path [/usr/local/bin/redis-server]
Selected config:
Port : 6379
Config file : /etc/redis/6379.conf
Log file : /data/redis/redis_6379.log
Data dir : /data/redis/6379
Executable : /usr/local/bin/redis-server
Cli Executable : /usr/local/bin/redis-cli
Is this ok? Then press ENTER to go on or Ctrl-C to abort.
Copied /tmp/6379.conf => /etc/init.d/redis_6379
Installing service...
Successfully added to chkconfig!
Successfully added to runlevels 345!
Starting Redis server...
Installation successful!
7，驗證服務
/etc/init.d/redis_6379 status
8，其他節點類似安裝


三，部署Cluster環境
1，集群主從節點關系
redis1redis1:6379主redis3:6380從
redis2redis2:6379主redis1:6380從
redis3redis3:6379主redis2:6380從
2，集群環境准備
這是源碼安裝
tar -xf ruby-2.2.3.tar.gz
cd ruby-2.2.3
./configure --disable-install-rdoc
make && make install
/usr/local/bin/ruby
/usr/local/bin/gem
YUM安裝
yum install ruby ruby-devel rubygems


設置ruby gem源為淘寶,安裝gems更快。
# gem sources --add https://ruby.taobao.org/ --remove https://rubygems.org/
查看gem源
# gem sources -l
安裝ruby的redis模塊（依賴接口）
# gem install redis（注意這地方要選定版本排錯時會提到）




3，配置文件(其他實例相應修改端口）
cat /etc/redis/6379.conf |grep -Ev "^#|^$"
daemonize yes
pidfile /var/run/redis_6379.pid
port 6379
timeout 0
logfile /data/redis/redis_6379.log
dbfilename dump.rdb
dir /data/redis/6379
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
取消如下注釋，讓Redis在集群模式下運行
cluster-enabled yes 啟動cluster模式
cluster-config-file nodes-6379.conf 集群信息文件名，由redis自己維護
cluster-node-timeout 5000 5秒中聯系不到對方node，即認為對方有故障可能


4，重啟redis (其他實例相應修改端口）
/etc/init.d/redis_6379 restart
會在/data/redis/6379/下面多出一個文件nodes-6379.conf


5，利用腳本工具建立集群
首先查看幫助看redis-trib腳本功能
注意編譯安裝時redis-trib.rb沒有復制過來
cp ./src/redis-trib.rb /usr/local/bin/


ruby redis-trib.rb help
create：創建集群
check：檢查集群
info：查看集群信息
fix：修復集群
reshard：在線遷移slot
rebalance：平衡集群節點slot數量
add-node：將新節點加入集群
del-node：從集群中刪除節點
set-timeout：設置集群節點間心跳連接的超時時間
call：在集群全部節點上執行命令
import：將外部redis數據導入集群
開始創建集群注意前三個為主後三個為從（但是不能指定主從關系，後面可以在Cluster裡面操作，目的是主從實例不能在統一節點服務器上）， 選項 --replicas 1 表示集群中的每個主節點創建一個從節點。
/usr/local/bin/redis-trib.rb create --replicas 1 192.168.188.18:6379 192.168.188.28:6379 192.168.188.38:6379 192.168.188.18:6380 192.168.188.28:6380 192.168.188.38:6380
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.188.38:6379
192.168.188.28:6379
192.168.188.18:6379
Adding replica 192.168.188.28:6380 to 192.168.188.38:6379
Adding replica 192.168.188.38:6380 to 192.168.188.28:6379
Adding replica 192.168.188.18:6380 to 192.168.188.18:6379
M: 53c202561155d141a8e24d42bb8eb2907998fce9 192.168.188.18:6379
slots:10923-16383 (5461 slots) master
M: 654c0a7853724fe33683f54e8df665993ea704ed 192.168.188.28:6379
slots:5461-10922 (5462 slots) master
M: b4cc74e0ac6ec9b634d93e86fdb376b4d89dc318 192.168.188.38:6379
slots:0-5460 (5461 slots) master
S: ee3d5f85034e5a3c9d9369357b545d0a92751728 192.168.188.18:6380
replicates 53c202561155d141a8e24d42bb8eb2907998fce9
S: 10b558217d6618093b87f75a453ca39679bbcdf4 192.168.188.28:6380
replicates b4cc74e0ac6ec9b634d93e86fdb376b4d89dc318
S: 5dc55e90d3d3b56502a319ec9845e901102cfa6c 192.168.188.38:6380
replicates 654c0a7853724fe33683f54e8df665993ea704ed
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join......
>>> Performing Cluster Check (using node 192.168.188.18:6379)
M: 53c202561155d141a8e24d42bb8eb2907998fce9 192.168.188.18:6379
slots:10923-16383 (5461 slots) master
M: 654c0a7853724fe33683f54e8df665993ea704ed 192.168.188.28:6379
slots:5461-10922 (5462 slots) master
M: b4cc74e0ac6ec9b634d93e86fdb376b4d89dc318 192.168.188.38:6379
slots:0-5460 (5461 slots) master
M: ee3d5f85034e5a3c9d9369357b545d0a92751728 192.168.188.18:6380
slots: (0 slots) master
replicates 53c202561155d141a8e24d42bb8eb2907998fce9
M: 10b558217d6618093b87f75a453ca39679bbcdf4 192.168.188.28:6380
slots: (0 slots) master
replicates b4cc74e0ac6ec9b634d93e86fdb376b4d89dc318
M: 5dc55e90d3d3b56502a319ec9845e901102cfa6c 192.168.188.38:6380
slots: (0 slots) master
replicates 654c0a7853724fe33683f54e8df665993ea704ed
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.


四，測試驗證
調整主從狀態
登錄redis集群：
設置192.168.188.18:6380為192.168.188.28:6379的從
redis-cli -c -p 6380 -h 192.168.188.18
192.168.188.18:6380> cluster replicate 654c0a7853724fe33683f54e8df665993ea704ed
OK
設置192.168.188.38:6380為192.168.188.18:6379的從
redis-cli -c -p 6380 -h 192.168.188.38
192.168.188.38:6380> cluster replicate 53c202561155d141a8e24d42bb8eb2907998fce9
OK
最後保存配置到硬盤
192.168.188.38:6380> cluster saveconfig
OK


檢查集群狀態：
#redis-trib.rb check 192.168.188.18:6379
>>> Performing Cluster Check (using node 192.168.188.18:6379)
M: 53c202561155d141a8e24d42bb8eb2907998fce9 192.168.188.18:6379
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: 5dc55e90d3d3b56502a319ec9845e901102cfa6c 192.168.188.38:6380
slots: (0 slots) slave
replicates 53c202561155d141a8e24d42bb8eb2907998fce9
M: 654c0a7853724fe33683f54e8df665993ea704ed 192.168.188.28:6379
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 10b558217d6618093b87f75a453ca39679bbcdf4 192.168.188.28:6380
slots: (0 slots) slave
replicates b4cc74e0ac6ec9b634d93e86fdb376b4d89dc318
S: ee3d5f85034e5a3c9d9369357b545d0a92751728 192.168.188.18:6380
slots: (0 slots) slave
replicates 654c0a7853724fe33683f54e8df665993ea704ed
M: b4cc74e0ac6ec9b634d93e86fdb376b4d89dc318 192.168.188.38:6379
slots:0-5460 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.


使用方法
集群
CLUSTER INFO 打印集群的信息
CLUSTER NODES 列出集群當前已知的所有節點（node），以及這些節點的相關信息。
節點
CLUSTER MEET 將 ip 和 port 所指定的節點添加到集群當中，讓它成為集群的一份子。
CLUSTER FORGET 從集群中移除 node_id 指定的節點。
CLUSTER REPLICATE 將當前節點設置為 node_id 指定的節點的從節點。
CLUSTER SAVECONFIG 將節點的配置文件保存到硬盤裡面。
槽(slot)
CLUSTER ADDSLOTS [slot ...] 將一個或多個槽（slot）指派（assign）給當前節點。
CLUSTER DELSLOTS [slot ...] 移除一個或多個槽對當前節點的指派。
CLUSTER FLUSHSLOTS 移除指派給當前節點的所有槽，讓當前節點變成一個沒有指派任何槽的節點。
CLUSTER SETSLOT NODE 將槽 slot 指派給 node_id 指定的節點，如果槽已經指派給另一個節點，那麼先讓另一個節點刪除該槽>，然後再進行指派。
CLUSTER SETSLOT MIGRATING 將本節點的槽 slot 遷移到 node_id 指定的節點中。
CLUSTER SETSLOT IMPORTING 從 node_id 指定的節點中導入槽 slot 到本節點。
CLUSTER SETSLOT STABLE 取消對槽 slot 的導入（import）或者遷移（migrate）。
鍵
CLUSTER KEYSLOT 計算鍵 key 應該被放置在哪個槽上。
CLUSTER COUNTKEYSINSLOT 返回槽 slot 目前包含的鍵值對數量。
CLUSTER GETKEYSINSLOT 返回 count 個 slot 槽中的鍵。




五，排錯優化
報錯
/usr/lib/ruby/gems/1.8/gems/redis-3.0.7/lib/redis/client.rb:97:in `call': ERR Slot 16011 is already busy (Redis::CommandError)
from /usr/lib/ruby/gems/1.8/gems/redis-3.0.7/lib/redis.rb:2432:in `method_missing'
from /usr/lib/ruby/gems/1.8/gems/redis-3.0.7/lib/redis.rb:37:in `synchronize'
from /usr/lib/ruby/1.8/monitor.rb:242:in `mon_synchronize'
from /usr/lib/ruby/gems/1.8/gems/redis-3.0.7/lib/redis.rb:37:in `synchronize'
from /usr/lib/ruby/gems/1.8/gems/redis-3.0.7/lib/redis.rb:2431:in `method_missing'
from /usr/local/bin/redis-trib.rb:212:in `flush_node_config'
from /usr/local/bin/redis-trib.rb:775:in `flush_nodes_config'
from /usr/local/bin/redis-trib.rb:774:in `each'
from /usr/local/bin/redis-trib.rb:774:in `flush_nodes_config'
from /usr/local/bin/redis-trib.rb:1295:in `create_cluster_cmd'
from /usr/local/bin/redis-trib.rb:1695:in `send'
from /usr/local/bin/redis-trib.rb:1695
處理
rm -rf /data/redis/6379/nodes-6379.conf
rm -rf /data/redis/6380/nodes-6380.conf
/etc/init.d/redis_6379 restart
/etc/init.d/redis_6380 restart
gem list
gem uninstall redis --version 3.3.2
gem install redis --version 3.0.7
gem list


警告
WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
7997:M 08 Dec 15:04:59.760 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
優化
vm.overcommit_memory = 1
echo never > /sys/kernel/mm/transparent_hugepage/enabled