Redis高可用集群部署-Cluster

Posted on In Views: Symbols count in article: 10k Reading time ≈ 9 mins.
Docker Swarm部署Redis-Cluster+Proxy实现容灾

Redis cluster是redis的官方集群方案，但是要求客户端自己做重定向，所以连接单机redis和集群redis的客户端会有些不同，连接集群版redis的时候要客户端连接6个redis实例。

官方为了屏蔽这种差异，做了一个redis-cluster-proxy，经过这个proxy的代理后，连接redis集群就和连接单机redis一样了。

RedisLabs/redis-cluster-proxy: A proxy for Redis clusters. (github.com)

image-20230203142147518

系统环境

集群规划

搭建3master+3slave+1proxy，共三个节点，如下：

IP hostname 部署实例
192.168.2.213 manager redis1、redis2、proxy(1实例)
192.168.2.214 worker1 redis3、redis4
192.168.2.215 worker2 redis5、redis5

查看系统版本

1
2
3
4
[root@manager ~]# uname -a
Linux manager 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@manager ~]# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)

Docker

查看Docker版本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
[root@manager ~]# docker version
Client: Docker Engine - Community
 Version:           20.10.23
 API version:       1.41
 Go version:        go1.18.10
 Git commit:        7155243
 Built:             Thu Jan 19 17:36:21 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.23
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.10
  Git commit:       6051f14
  Built:            Thu Jan 19 17:34:26 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.15
  GitCommit:        5b842e528e99d4d4c1686467debf2bd4b88ecd86
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

防火墙

所有节点执行以下命令，打开集群初始化所需端口

1
2
3
4
5
6
# 所有node
firewall-cmd --zone=public --add-port=2377/tcp --permanent
firewall-cmd --zone=public --add-port=7946/tcp --permanent
firewall-cmd --zone=public --add-port=7946/udp --permanent
firewall-cmd --zone=public --add-port=4789/tcp --permanent
firewall-cmd --zone=public --add-port=4789/udp --permanent

重启防火墙以及Docker

1
2
3
# 所有node
firewall-cmd --reload
systemctl restart docker

Docker Swarm

创建3个manager节点，如下：

image-20230203101937613

manager节点初始化集群：

1
docker swarm init --advertise-addr 192.168.2.213

manager节点获取加入manager token

1
docker swarm join-token manager

worker节点以manager身份加入集群

docker swarm join \
--token SWMTKN-1-3pu6hszjas19xyp7ghgosyx9k8atbfcr8p2is99znpy26u2lkl-7p73s1dx5in4tatdymyhg9hu2 \
192.168.2.213:2377

系统部署

redis_cluster

部署准备

创建目录

1
2
mkdir redis-cluster
cd redis-cluster

docker-compose.yml

编写docker-compose.yml文件，需要根据实际IP地址做调整

1
vi docker-compose.yml

docker-compose.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
version: '3.7'
services:
  redis-node1:
    image: redis:6.2.4
    hostname: redis-node1
    ports:
      - 7001:6379
      - 17001:16379
    networks:
      - redis-swarm
    volumes:
      - "redis-node1-data:/data"
    command: 
      - "redis-server"
      - "--appendonly yes"
      - "--cluster-enabled yes"
      - "--cluster-config-file nodes.conf"
      - "--requirepass bb123456"
      - "--masterauth bb123456"
      - "--cluster-announce-ip 192.168.2.213"
      - "--cluster-announce-port 7001"
      - "--cluster-announce-bus-port 17001"
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.hostname == manager

  redis-node2:
    image: redis:6.2.4
    hostname: redis-node2
    ports:
      - 7002:6379
      - 17002:16379
    networks:
      - redis-swarm
    volumes:
      - "redis-node2-data:/data"
    command: 
      - "redis-server"
      - "--appendonly yes"
      - "--cluster-enabled yes"
      - "--cluster-config-file nodes.conf"
      - "--requirepass bb123456"
      - "--masterauth bb123456"
      - "--cluster-announce-ip 192.168.2.213"
      - "--cluster-announce-port 7002"
      - "--cluster-announce-bus-port 17002"
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.hostname == manager

  redis-node3:
    image: redis:6.2.4
    hostname: redis-node3
    ports:
      - 7003:6379
      - 17003:16379
    networks:
      - redis-swarm
    volumes:
      - "redis-node3-data:/data"
    command: 
      - "redis-server"
      - "--appendonly yes"
      - "--cluster-enabled yes"
      - "--cluster-config-file nodes.conf"
      - "--requirepass bb123456"
      - "--masterauth bb123456"
      - "--cluster-announce-ip 192.168.2.214"
      - "--cluster-announce-port 7003"
      - "--cluster-announce-bus-port 17003"
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.hostname == worker1

  redis-node4:
    image: redis:6.2.4
    hostname: redis-node4
    ports:
      - 7004:6379
      - 17004:16379
    networks:
      - redis-swarm
    volumes:
      - "redis-node4-data:/data"
    command: 
      - "--appendonly yes"
      - "--cluster-enabled yes"
      - "--cluster-config-file nodes.conf"
      - "--requirepass bb123456"
      - "--masterauth bb123456"
      - "--cluster-announce-ip 192.168.2.214"
      - "--cluster-announce-port 7004"
      - "--cluster-announce-bus-port 17004"
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.hostname == worker1

  redis-node5:
    image: redis:6.2.4
    hostname: redis-node5
    ports:
      - 7005:6379
      - 17005:16379
    networks:
      - redis-swarm
    volumes:
      - "redis-node5-data:/data"
    command: 
      - "--appendonly yes"
      - "--cluster-enabled yes"
      - "--cluster-config-file nodes.conf"
      - "--requirepass bb123456"
      - "--masterauth bb123456"
      - "--cluster-announce-ip 192.168.2.215"
      - "--cluster-announce-port 7005"
      - "--cluster-announce-bus-port 17005"
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.hostname == worker2

  redis-node6:
    image: redis:6.2.4
    hostname: redis-node6
    ports:
      - 7006:6379
      - 17006:16379
    networks:
      - redis-swarm
    volumes:
      - "redis-node6-data:/data"
    command: 
      - "--appendonly yes"
      - "--cluster-enabled yes"
      - "--cluster-config-file nodes.conf"
      - "--requirepass bb123456"
      - "--masterauth bb123456"
      - "--cluster-announce-ip 192.168.2.215"
      - "--cluster-announce-port 7006"
      - "--cluster-announce-bus-port 17006"
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.hostname == worker2

  redis-cluster:
    image: redis:6.2.4
    hostname: redis-cluster
    networks:
      - redis-swarm
    depends_on:
      - redis-node1
      - redis-node2
      - redis-node3
      - redis-node4
      - redis-node5
      - redis-node6
    command: 'redis-cli --cluster create
    192.168.2.213:7001 192.168.2.213:7002 192.168.2.214:7003
    192.168.2.214:7004 192.168.2.215:7005 192.168.2.215:7006
    --cluster-yes --cluster-replicas 1 -a bb123456'
    deploy:
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 5
      placement:
        constraints:
          - node.hostname == manager

  redis-proxy:
    image: jontymax/redis-cluster-proxy:dev
    hostname: redis-proxy
    ports: 
      - "7777:7777"
    networks:
      - redis-swarm
    depends_on:
      - redis-cluster
    command: 'redis-cluster-proxy --auth bb123456 192.168.2.213:7001 192.168.2.213:7002 192.168.2.214:7003 192.168.2.214:7004 192.168.2.215:7005 192.168.2.215:7006'
    deploy:
      mode: global

networks:
  redis-swarm:
    driver: overlay

volumes:
  redis-node1-data:
  redis-node2-data:
  redis-node3-data:
  redis-node4-data:
  redis-node5-data:
  redis-node6-data:

服务部署

启动服务

1
docker stack deploy -c docker-compose.yml redis-cluster-proxy

image-20230203145125983

查看服务

1
docker service ls | grep redis-cluster

image-20230203145230510

redis-cluster服务用于初始化集群，启动完成日志如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.2.214:7004 to 192.168.2.213:7001
Adding replica 192.168.2.215:7006 to 192.168.2.214:7003
Adding replica 192.168.2.213:7002 to 192.168.2.215:7005
M: 9ee907ba218b782c264be77e4dd5ca53a79b7350 192.168.2.213:7001
   slots:[0-5460] (5461 slots) master
S: 567cbca6ba03a2c4d1f4461882cf64a369daaafb 192.168.2.213:7002
   replicates 98719ee5734cdc8b9b06519e2ebcf23b66041266
M: c912e5bc90cdaed6159a5ca83725848556a8e68d 192.168.2.214:7003
   slots:[5461-10922] (5462 slots) master
S: 257d31f2f29f658103e9032584de8c2803bfb7b9 192.168.2.214:7004
   replicates 9ee907ba218b782c264be77e4dd5ca53a79b7350
M: 98719ee5734cdc8b9b06519e2ebcf23b66041266 192.168.2.215:7005
   slots:[10923-16383] (5461 slots) master
S: de5dbc1a51252d1c2ccf863470a85e5bd9e168bd 192.168.2.215:7006
   replicates c912e5bc90cdaed6159a5ca83725848556a8e68d
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
......
>>> Performing Cluster Check (using node 192.168.2.213:7001)
M: 9ee907ba218b782c264be77e4dd5ca53a79b7350 192.168.2.213:7001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: c912e5bc90cdaed6159a5ca83725848556a8e68d 192.168.2.214:7003
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 567cbca6ba03a2c4d1f4461882cf64a369daaafb 192.168.2.213:7002
   slots: (0 slots) slave
   replicates 98719ee5734cdc8b9b06519e2ebcf23b66041266
S: 257d31f2f29f658103e9032584de8c2803bfb7b9 192.168.2.214:7004
   slots: (0 slots) slave
   replicates 9ee907ba218b782c264be77e4dd5ca53a79b7350
S: de5dbc1a51252d1c2ccf863470a85e5bd9e168bd 192.168.2.215:7006
   slots: (0 slots) slave
   replicates c912e5bc90cdaed6159a5ca83725848556a8e68d
M: 98719ee5734cdc8b9b06519e2ebcf23b66041266 192.168.2.215:7005
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

redis-proxy服务用于代理redis-cluster，等待redis-cluster初始化完成，启动完成日志如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[2023-02-03 06:28:08.955/M] Redis Cluster Proxy v999.999.999 (unstable)
[2023-02-03 06:28:08.955/M] Commit: (ac83840d/0)
[2023-02-03 06:28:08.955/M] Git Branch: unstable
[2023-02-03 06:28:08.955/M] PID: 1
[2023-02-03 06:28:08.955/M] OS: Linux 3.10.0-957.el7.x86_64 x86_64
[2023-02-03 06:28:08.955/M] Bits: 64
[2023-02-03 06:28:08.955/M] Log level: info
[2023-02-03 06:28:08.955/M] Connections pool size: 10 (respawn 2 every 50ms if below 10)
[2023-02-03 06:28:08.955/M] The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
[2023-02-03 06:28:08.955/M] Listening on *:7777
[2023-02-03 06:28:08.955/M] Starting 8 threads...
[2023-02-03 06:28:08.955/M] Fetching cluster configuration...
[2023-02-03 06:28:08.962/M] Cluster Address: 192.168.2.213:7001
[2023-02-03 06:28:08.962/M] Cluster has 3 masters and 3 replica(s)
[2023-02-03 06:28:09.057/M] All thread(s) started!

如果初始化或代理启动失败，可以尝试移除服务重新创建

移除服务

1
docker stack rm redis-cluster

服务连接

连接到集群

连接任意节点IP:700x可连接到集群

image-20230203145831336

可正常使用

image-20230203150113829

查看集群节点信息

image-20230203150222570

连接到代理

image-20230203145814034

image-20230203150312909

Related Posts