Redis高可用集群部署-Sentinel

Posted on In Views: Symbols count in article: 15k Reading time ≈ 14 mins.
Docker Swarm部署Redis主从+哨兵Sentinel+HAProxy实现容灾

redis-sentinel-haproxy

Redis使用使用哨兵模式进行组网，哨兵负责主节点的故障转移。
HAProxy作为Redis集群的代理(HAProxy工作在TCP层)，屏蔽底层redis的组网细节，对上层应用来看就是单节点的redis。

关于Redis集群方案介绍+优缺点对比

系统环境

集群规划

搭建1主2从3哨兵，共三个节点，如下：

IP hostname 部署实例
192.168.2.213 manager redis1、sentinel1、Haproxy(1实例)
192.168.2.214 worker1 redis2、sentinel2
192.168.2.215 worker2 redis3、sentinel3

查看系统版本

1
2
3
4
[root@manager ~]# uname -a
Linux manager 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@manager ~]# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)

Docker

查看Docker版本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
[root@manager ~]# docker version
Client: Docker Engine - Community
 Version:           20.10.23
 API version:       1.41
 Go version:        go1.18.10
 Git commit:        7155243
 Built:             Thu Jan 19 17:36:21 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.23
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.10
  Git commit:       6051f14
  Built:            Thu Jan 19 17:34:26 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.15
  GitCommit:        5b842e528e99d4d4c1686467debf2bd4b88ecd86
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

防火墙

所有节点执行以下命令，打开集群初始化所需端口

1
2
3
4
5
6
# 所有node
firewall-cmd --zone=public --add-port=2377/tcp --permanent
firewall-cmd --zone=public --add-port=7946/tcp --permanent
firewall-cmd --zone=public --add-port=7946/udp --permanent
firewall-cmd --zone=public --add-port=4789/tcp --permanent
firewall-cmd --zone=public --add-port=4789/udp --permanent

重启防火墙以及Docker

1
2
3
# 所有node
firewall-cmd --reload
systemctl restart docker

Docker Swarm

创建3个manager节点，如下：

image-20230203101937613

manager节点初始化集群：

1
docker swarm init --advertise-addr 192.168.2.213

manager节点获取加入manager token

1
docker swarm join-token manager

worker节点以manager身份加入集群

docker swarm join \
--token SWMTKN-1-3pu6hszjas19xyp7ghgosyx9k8atbfcr8p2is99znpy26u2lkl-7p73s1dx5in4tatdymyhg9hu2 \
192.168.2.213:2377

系统部署

部署架构图

redis-sentinel

部署准备

创建目录

1
2
3
mkdir redis-sentinel
cd redis-sentinel
mkdir haproxy

docker-compose.yml

编写docker-compose.yml文件，需要根据实际IP地址做调整

1
vi docker-compose.yml

docker-compose.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
version: '3.8'
services:
  redis1:
    image: redis:6.2.4
    environment:
      TZ: "Asia/Shanghai"
    ports:
      - target: 6379
        published: 6379
        protocol: tcp
        mode: host
    volumes:
      - redis-data:/data
      - redis-conf:/conf
    networks:
      - redis-sentinel
    deploy:
      mode: replicated
      resources:
        limits:
          # cpus: '0.001'
          memory: 5120M
        reservations:
          # cpus: '0.001'
          memory: 512M
      replicas: 1
      placement:
        constraints:
          - node.hostname == manager
    command: >
      bash -c "if [ ! -f /conf/redis.conf ];then
      echo 'port 6379' > /conf/redis.conf ;
      echo 'bind 0.0.0.0' >> /conf/redis.conf ;
      echo 'slave-announce-ip 192.168.2.213' >> /conf/redis.conf ;
      echo 'slave-announce-port 6379' >> /conf/redis.conf ;
      echo 'masterauth bb123456' >> /conf/redis.conf;
      echo 'requirepass bb123456' >> /conf/redis.conf ;
      echo 'appendonly yes' >> /conf/redis.conf ; fi &&
      redis-server /conf/redis.conf"

  redis2:
    image: redis:6.2.4
    environment:
      TZ: "Asia/Shanghai"
    ports:
      - target: 6379
        published: 6379
        protocol: tcp
        mode: host
    volumes:
      - redis-data:/data
      - redis-conf:/conf
    networks:
      - redis-sentinel
    depends_on:
      - redis1
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.hostname == worker1
    command: >
      bash -c "if [ ! -f /conf/redis.conf ];then
      echo 'port 6379' > /conf/redis.conf ;
      echo 'bind 0.0.0.0' >> /conf/redis.conf ;
      echo 'slave-announce-ip 192.168.2.214' >> /conf/redis.conf ;
      echo 'slave-announce-port 6379' >> /conf/redis.conf ;
      echo 'masterauth bb123456' >> /conf/redis.conf;
      echo 'requirepass bb123456' >> /conf/redis.conf ;
      echo 'replicaof 192.168.2.213 6379' >> /conf/redis.conf ;
      echo 'appendonly yes' >> /conf/redis.conf ; fi &&
      redis-server /conf/redis.conf"

  redis3:
    image: redis:6.2.4
    environment:
      TZ: "Asia/Shanghai"
    ports:
      - target: 6379
        published: 6379
        protocol: tcp
        mode: host
    volumes:
      - redis-data:/data
      - redis-conf:/conf
    networks:
      - redis-sentinel
    depends_on:
      - redis1
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.hostname == worker2
    command: >
      bash -c "if [ ! -f /conf/redis.conf ];then
      echo 'port 6379' > /conf/redis.conf ;
      echo 'bind 0.0.0.0' >> /conf/redis.conf ;
      echo 'slave-announce-ip 192.168.2.215' >> /conf/redis.conf ;
      echo 'slave-announce-port 6379' >> /conf/redis.conf ;
      echo 'masterauth bb123456' >> /conf/redis.conf;
      echo 'requirepass bb123456' >> /conf/redis.conf ;
      echo 'replicaof 192.168.2.213 6379' >> /conf/redis.conf ;
      echo 'appendonly yes' >> /conf/redis.conf ; fi &&
      redis-server /conf/redis.conf"
      
  sentinel1:
    image: redis:6.2.4
    environment:
      TZ: "Asia/Shanghai"
    ports:
      - target: 26379
        published: 26379
        protocol: tcp
        mode: host
    depends_on:
      - redis1
      - redis2
      - redis3
    volumes:
      - redis-data:/data
      - redis-conf:/conf
    networks:
      - redis-sentinel
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.hostname == manager
    command: >
      bash -c "if [ ! -f /conf/sentinel.conf ];then
      echo 'port 26379' > /conf/sentinel.conf ;
      echo 'sentinel announce-ip 192.168.2.213' >> /conf/sentinel.conf ;
      echo 'sentinel monitor mymaster 192.168.2.213 6379 2' >> /conf/sentinel.conf ;
      echo 'sentinel auth-pass mymaster bb123456' >> /conf/sentinel.conf ; fi &&
      redis-server /conf/sentinel.conf --sentinel"

  sentinel2:
    image: redis:6.2.4
    environment:
      TZ: "Asia/Shanghai"
    ports:
      - target: 26379
        published: 26379
        protocol: tcp
        mode: host
    depends_on:
      - redis1
      - redis2
      - redis3
    volumes:
      - redis-data:/data
      - redis-conf:/conf
    networks:
      - redis-sentinel
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.hostname == worker1
    command: >
      bash -c "if [ ! -f /conf/sentinel.conf ];then
      echo 'port 26379' > /conf/sentinel.conf ;
      echo 'sentinel announce-ip 192.168.2.214' >> /conf/sentinel.conf ;
      echo 'sentinel monitor mymaster 192.168.2.213 6379 2' >> /conf/sentinel.conf ;
      echo 'sentinel auth-pass mymaster bb123456' >> /conf/sentinel.conf ; fi &&
      redis-server /conf/sentinel.conf --sentinel"

  sentinel3:
    image: redis:6.2.4
    environment:
      TZ: "Asia/Shanghai"
    ports:
      - target: 26379
        published: 26379
        protocol: tcp
        mode: host
    depends_on:
      - redis1
      - redis2
      - redis3
    volumes:
      - redis-data:/data
      - redis-conf:/conf
    networks:
      - redis-sentinel
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.hostname == worker2
    command: >
      bash -c "if [ ! -f /conf/sentinel.conf ];then
      echo 'port 26379' > /conf/sentinel.conf ;
      echo 'sentinel announce-ip 192.168.2.215' >> /conf/sentinel.conf ;
      echo 'sentinel monitor mymaster 192.168.2.213 6379 2' >> /conf/sentinel.conf ;
      echo 'sentinel auth-pass mymaster bb123456' >> /conf/sentinel.conf ; fi &&
      redis-server /conf/sentinel.conf --sentinel"

  haproxy:
    image: haproxytech/haproxy-alpine:2.4
    depends_on:
      - sentinel1
      - sentinel2
      - sentinel3
    configs:
      - source: haproxy_conf
        target: /usr/local/etc/haproxy/haproxy.cfg
    networks:
      - redis-sentinel
    ports:
      - "8080:8080"
      - "16379:16379"
    deploy:
      mode: global

networks:
  redis-sentinel:
    driver: overlay

volumes:
  redis-data:
  redis-conf:

configs:
  haproxy_conf:
    file: ./haproxy/haproxy.cfg

haproxy.cfg

创建haproxy.cfg配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    # 
  log stdout format raw local0 info  
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    log global
    mode tcp
    retries 3
    option redispatch
    maxconn 20000
    timeout connect 600s
    timeout client 600s
    timeout server 600s
listen stats
    bind 0.0.0.0:8080 # 面板代理
    mode http 
    stats uri /haproxy-status # 面板地址
    stats auth admin:admin # 面板user:pwd
    stats hide-version
    stats refresh 30s
frontend redis16379 
    bind :16379 # 代理端口
    default_backend redis_16379_backend
backend redis_16379_backend
    option tcp-check
    tcp-check connect
    tcp-check send AUTH\ bb123456\r\n # redis 密码 按需调整
    tcp-check expect string +OK
    tcp-check send PING\r\n
    tcp-check expect string +PONG
    tcp-check send info\ replication\r\n
    tcp-check expect string role:master # 轮询 master节点
    tcp-check send QUIT\r\n
    tcp-check expect string +OK
    server redis1 192.168.2.213:6379 check inter 1s
    server redis2 192.168.2.214:6379 check inter 1s
    server redis3 192.168.2.215:6379 check inter 1s

文件目录如下：

image-20230203105159275

服务部署

启动服务

1
docker stack deploy -c docker-compose.yml redis-sentinel

image-20230203105502431

查看服务

1
docker service ls | grep redis-sentinel

image-20230203105609286

image-20230203110439408

服务日志

redis:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
1:S 03 Feb 2023 09:07:22.284 * MASTER <-> REPLICA sync: receiving 214 bytes from master to disk
1:S 03 Feb 2023 09:07:22.284 * MASTER <-> REPLICA sync: Flushing old data
1:S 03 Feb 2023 09:07:22.284 * MASTER <-> REPLICA sync: Loading DB in memory
1:S 03 Feb 2023 09:07:22.285 * Loading RDB produced by version 6.2.4
1:S 03 Feb 2023 09:07:22.285 * RDB age 0 seconds
1:S 03 Feb 2023 09:07:22.285 * RDB memory usage when created 1.85 Mb
1:S 03 Feb 2023 09:07:22.285 * MASTER <-> REPLICA sync: Finished with success
1:S 03 Feb 2023 09:07:22.286 * Background append only file rewriting started by pid 13
1:S 03 Feb 2023 09:07:22.308 * AOF rewrite child asks to stop sending diffs.
13:C 03 Feb 2023 09:07:22.308 * Parent agreed to stop sending diffs. Finalizing AOF...
13:C 03 Feb 2023 09:07:22.308 * Concatenating 0.00 MB of AOF diff received from parent.
13:C 03 Feb 2023 09:07:22.309 * SYNC append only file rewrite performed
13:C 03 Feb 2023 09:07:22.309 * AOF rewrite: 4 MB of memory used by copy-on-write
1:S 03 Feb 2023 09:07:22.348 * Background AOF rewrite terminated with success
1:S 03 Feb 2023 09:07:22.348 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
1:S 03 Feb 2023 09:07:22.348 * Background AOF rewrite finished successfully
1:S 03 Feb 2023 10:07:14.007 * 1 changes in 3600 seconds. Saving...
1:S 03 Feb 2023 10:07:14.009 * Background saving started by pid 15
15:C 03 Feb 2023 10:07:14.012 * DB saved on disk
15:C 03 Feb 2023 10:07:14.013 * RDB: 4 MB of memory used by copy-on-write
1:S 03 Feb 2023 10:07:14.109 * Background saving terminated with success

sentinel:

1
2
3
4
5
6
7
8
9
10
11
12
13
1:X 03 Feb 2023 09:07:24.787 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:X 03 Feb 2023 09:07:24.787 # Redis version=6.2.4, bits=64, commit=00000000, modified=0, pid=1, just started
1:X 03 Feb 2023 09:07:24.787 # Configuration loaded
1:X 03 Feb 2023 09:07:24.788 * monotonic clock: POSIX clock_gettime
1:X 03 Feb 2023 09:07:24.788 * Running mode=sentinel, port=26379.
1:X 03 Feb 2023 09:07:24.788 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:X 03 Feb 2023 09:07:24.789 # Sentinel ID is 49d6ef4a1346ec4a05bec6a6387e6fd66c2a177e
1:X 03 Feb 2023 09:07:24.789 # +monitor master mymaster 192.168.2.12 6379 quorum 2
1:X 03 Feb 2023 09:09:48.027 # +sdown sentinel 32c2d1980d2d054c811f474140f17bb60c1a6dde 192.168.2.214 26379 @ mymaster 192.168.2.215 6379
1:X 03 Feb 2023 09:09:48.099 # +sdown slave 192.168.2.214:6379 192.168.2.11 6379 @ mymaster 192.168.2.12 6379
1:X 03 Feb 2023 09:11:54.402 * +reboot slave 192.168.2.214:6379 192.168.2.11 6379 @ mymaster 192.168.2.12 6379
1:X 03 Feb 2023 09:11:54.502 # -sdown slave 192.168.2.214:6379 192.168.2.11 6379 @ mymaster 192.168.2.12 6379
1:X 03 Feb 2023 09:11:54.502 # -sdown sentinel 32c2d1980d2d054c811f474140f17bb60c1a6dde 192.168.2.11 26379 @ mymaster 192.168.2.215 6379

haproxy:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[NOTICE]   (1) : New worker #1 (8) forked
Connect from 10.0.0.2:3768 to 10.0.0.249:16379 (redis16379/TCP)
Connect from 10.0.0.2:3767 to 10.0.0.249:16379 (redis16379/TCP)
Connect from 10.0.0.2:3769 to 10.0.0.249:16379 (redis16379/TCP)
Server redis_16379_backend/redis1 is DOWN, reason: Layer7 timeout, info: " at step 7 of tcp-check (expect string 'role:master')", check duration: 1001ms. 2 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
[WARNING]  (8) : Server redis_16379_backend/redis1 is DOWN, reason: Layer7 timeout, info: " at step 7 of tcp-check (expect string 'role:master')", check duration: 1001ms. 2 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
[WARNING]  (8) : Server redis_16379_backend/redis2 is DOWN, reason: Layer7 timeout, info: " at step 7 of tcp-check (expect string 'role:master')", check duration: 1001ms. 1 active and 0 backup servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
Server redis_16379_backend/redis2 is DOWN, reason: Layer7 timeout, info: " at step 7 of tcp-check (expect string 'role:master')", check duration: 1001ms. 1 active and 0 backup servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
Connect from 10.0.0.2:3765 to 10.0.0.249:16379 (redis16379/TCP)
Connect from 10.0.0.2:3764 to 10.0.0.249:16379 (redis16379/TCP)
Connect from 10.0.0.2:3770 to 10.0.0.249:16379 (redis16379/TCP)
Connect from 10.0.0.2:3771 to 10.0.0.249:16379 (redis16379/TCP)
Connect from 10.0.0.2:3766 to 10.0.0.249:16379 (redis16379/TCP)
Connect from 10.0.0.2:3772 to 10.0.0.249:16379 (redis16379/TCP)
代理面板

访问IP:8080/haproxy-status

image-20230203112402342

停止服务

1
docker stack rm redis-sentinel

连接测试

使用redis连接工具测试连接

连接到Redis单实例

正常连接IP:6379即可

image-20230203110812313

连接到Redis哨兵

连接到IP:26379

image-20230203110947077

连接到Haproxy代理

连接到haproxy:16379端口

image-20230203111030197

测试数据同步

通过Haproxy添加新的键值：name:jonty

image-20230203111202530

新增成功

image-20230203111313941

查看哨兵连接：

可以看到已经同步成功

image-20230203111344072

查看同步日志

1
2
3
4
5
1:S 03 Feb 2023 11:11:55.079 * 1 changes in 3600 seconds. Saving...
1:S 03 Feb 2023 11:11:55.081 * Background saving started by pid 16
16:C 03 Feb 2023 11:11:55.083 * DB saved on disk
16:C 03 Feb 2023 11:11:55.084 * RDB: 4 MB of memory used by copy-on-write
1:S 03 Feb 2023 11:11:55.181 * Background saving terminated with success

容灾测试

将214也就是worker1停止服务

1
2
systemctl stop docker
systemctl stop docker.socket

image-20230203111954001

可以看到worker1节点状态为down

1
2
3
4
5
[root@manager redis-swarm]# docker node ls
ID                            HOSTNAME   STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
eyufo1vxn0m5xksslcns6gdau *   manager    Ready     Active         Reachable        20.10.23
xki7hv7tw44394vqyhidco14w     worker1    Down      Active         Unreachable      20.10.23
xv4fw7yzwg5iwypea65z9n9yn     worker2    Ready     Active         Leader           20.10.23

214实例已无法访问

image-20230203112527481

image-20230203112729076

连接HaProxy，Redis服务正常

image-20230203112443957

添加新的键值

image-20230203112635960

并重新恢复worker1，查看数据是否同步

1
2
[root@worker1 ~]# systemctl start docker
[root@worker1 ~]# systemctl start docker.socket

可以看到redis2服务已经恢复

image-20230203112739424

查看redis2日志，可以看到已经同步成功

1
2
3
4
5
6
7
8
9
10
1:S 03 Feb 2023 11:26:55.825 * MASTER <-> REPLICA sync: Finished with success
1:S 03 Feb 2023 11:26:55.825 * Background append only file rewriting started by pid 13
1:S 03 Feb 2023 11:26:55.850 * AOF rewrite child asks to stop sending diffs.
13:C 03 Feb 2023 11:26:55.850 * Parent agreed to stop sending diffs. Finalizing AOF...
13:C 03 Feb 2023 11:26:55.850 * Concatenating 0.00 MB of AOF diff received from parent.
13:C 03 Feb 2023 11:26:55.850 * SYNC append only file rewrite performed
13:C 03 Feb 2023 11:26:55.851 * AOF rewrite: 6 MB of memory used by copy-on-write
1:S 03 Feb 2023 11:26:55.927 * Background AOF rewrite terminated with success
1:S 03 Feb 2023 11:26:55.927 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
1:S 03 Feb 2023 11:26:55.927 * Background AOF rewrite finished successfully

使用连接工具查看，没问题~

image-20230203112848725

Related Posts