2. 程式人生 > >redis cluster 全部宕機後重啟會自動恢復叢集狀態

redis cluster 全部宕機後重啟會自動恢復叢集狀態

阿新 • • 發佈：2018-12-23

昨天 測試環境上3主3從的redis節點叢集 虛擬機器3臺全部宕機

(3主3從交叉部署在3臺虛機上)

重新啟動各個節點發現  叢集自動恢復了  本來以為要重新使用create  命令

猜測叢集是根據node的主從資訊檔案自己恢復的 利用心跳檢測 

節點關係的檔案node-7001.conf

197d2e50893487f54d3cbbf07f746abc1fa08318 10.166.15.36:[email protected] master - 0 1530527218141 2 connected 5461-10922
052b9df159281988af20d8b5d0690e5defa42c47 10.166.15.37:
 
[email protected] slave 197d2e50893487f54d3cbbf07f746abc1fa08318 0 1530527220145 4 connected
ad20d73e76c72579be410a458c74bde554bf85bc 10.166.15.36:[email protected] master - 0 1530527218000 7 connected 0-5460
6d6f6fbeff41f5a85a7a7ebe15565a448fb0a69f 10.166.15.37:[email protected] slave ad20d73e76c72579be410a458c74bde554bf85bc 0 1530527220431 7 connected
b98f368b250360dbb4d74a1b831d7fa939ea4f8d 10.166.15.35:
 
[email protected] myself,slave 1e7a4a6b7376a89726211ce04acdaf97afbe2f13 0 1530527217000 1 connected
1e7a4a6b7376a89726211ce04acdaf97afbe2f13 10.166.15.35:[email protected] master - 0 1530527219143 8 connected 10923-16383
vars currentEpoch 8 lastVoteEpoch 7

對於日誌的分析

1 查看了 一個slave節點10.166.15.35:7001的異常斷電重新啟動日誌資訊如下

4446:M 02 Jul 18:07:55.091 * Node configuration loaded, I'm b98f368b250360dbb4d74a1b831d7fa939ea4f8d
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 4.0.6 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in cluster mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 7001
 |    `-._   `._    /     _.-'    |     PID: 4446
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               


4446:M 02 Jul 18:07:55.092 # Server initialized
4446:M 02 Jul 18:07:55.092 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
4446:M 02 Jul 18:07:55.092 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
4446:M 02 Jul 18:07:55.716 * DB loaded from append only file: 0.624 seconds
4446:M 02 Jul 18:07:55.716 * Ready to accept connections
4446:S 02 Jul 18:07:55.717 * Before turning into a slave, using my master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
4446:S 02 Jul 18:07:55.717 # Cluster state changed: ok
4446:S 02 Jul 18:07:56.720 * Connecting to MASTER 10.166.15.35:7002  ---ps:說明本slave節點是10.166.15.35:7001 這裡開始連結master
4446:S 02 Jul 18:07:56.720 * MASTER <-> SLAVE sync started
4446:S 02 Jul 18:07:56.720 * Non blocking connect for SYNC fired the event.
4446:S 02 Jul 18:07:56.720 * Master replied to PING, replication can continue...
4446:S 02 Jul 18:07:56.721 * Trying a partial resynchronization (request eda94ebd6d85f2f9dfb8337358ab14bdaee896bd:1).
4446:S 02 Jul 18:07:56.723 * Full resync from master: 79be914765bfde6e102bfae1b106847c8dcaea36:0
4446:S 02 Jul 18:07:56.723 * Discarding previously cached master state.
4446:S 02 Jul 18:07:56.772 * MASTER <-> SLAVE sync: receiving 318259 bytes from master
4446:S 02 Jul 18:07:56.773 * MASTER <-> SLAVE sync: Flushing old data
4446:S 02 Jul 18:07:56.776 * MASTER <-> SLAVE sync: Loading DB in memory
4446:S 02 Jul 18:07:56.784 * MASTER <-> SLAVE sync: Finished with success
4446:S 02 Jul 18:07:56.785 * Background append only file rewriting started by pid 4451
4446:S 02 Jul 18:07:56.812 * AOF rewrite child asks to stop sending diffs.
4451:C 02 Jul 18:07:56.812 * Parent agreed to stop sending diffs. Finalizing AOF...
4451:C 02 Jul 18:07:56.812 * Concatenating 0.00 MB of AOF diff received from parent.
4451:C 02 Jul 18:07:56.812 * SYNC append only file rewrite performed
4451:C 02 Jul 18:07:56.812 * AOF rewrite: 6 MB of memory used by copy-on-write
4446:S 02 Jul 18:07:56.820 * Background AOF rewrite terminated with success
4446:S 02 Jul 18:07:56.820 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
4446:S 02 Jul 18:07:56.820 * Background AOF rewrite finished successfully
4446:S 02 Jul 18:08:10.776 # Cluster state changed: fail
4446:S 02 Jul 18:08:13.650 # Cluster state changed: ok

4446:S 02 Jul 18:08:29.586 * FAIL message received from 1e7a4a6b7376a89726211ce04acdaf97afbe2f13 about 052b9df159281988af20d8b5d0690e5defa42c47

說明:1e7a4a6b7376a89726211ce04acdaf97afbe2f13  就是10.166.15.35：7002

4446:S 02 Jul 18:08:29.586 * FAIL message received from 1e7a4a6b7376a89726211ce04acdaf97afbe2f13 about 6d6f6fbeff41f5a85a7a7ebe15565a448fb0a69f
4446:S 02 Jul 18:08:56.087 * 10000 changes in 60 seconds. Saving...
4446:S 02 Jul 18:08:56.088 * Background saving started by pid 4472
4472:C 02 Jul 18:08:56.106 * DB saved on disk
4472:C 02 Jul 18:08:56.107 * RDB: 6 MB of memory used by copy-on-write
4446:S 02 Jul 18:08:56.188 * Background saving terminated with success
4446:S 02 Jul 18:26:56.159 * Clear FAIL state for node 052b9df159281988af20d8b5d0690e5defa42c47: slave is reachable again.
4446:S 02 Jul 18:27:00.431 * Clear FAIL state for node 6d6f6fbeff41f5a85a7a7ebe15565a448fb0a69f: slave is reachable again.
4446:S 02 Jul 18:27:11.969 * 1 changes in 900 seconds. Saving...
4446:S 02 Jul 18:27:11.971 * Background saving started by pid 5248
5248:C 02 Jul 18:27:11.995 * DB saved on disk
5248:C 02 Jul 18:27:11.996 * RDB: 6 MB of memory used by copy-on-write
4446:S 02 Jul 18:27:12.071 * Background saving terminated with success
4446:S 02 Jul 18:42:13.029 * 1 changes in 900 seconds. Saving...
4446:S 02 Jul 18:42:13.031 * Background saving started by pid 6014
6014:C 02 Jul 18:42:13.051 * DB saved on disk
6014:C 02 Jul 18:42:13.051 * RDB: 8 MB of memory used by copy-on-write
4446:S 02 Jul 18:42:13.134 * Background saving terminated with success
4446:S 02 Jul 19:21:15.352 * 1 changes in 900 seconds. Saving...
4446:S 02 Jul 19:21:15.354 * Background saving started by pid 7287
7287:C 02 Jul 19:21:15.363 * DB saved on disk
7287:C 02 Jul 19:21:15.363 * RDB: 8 MB of memory used by copy-on-write
4446:S 02 Jul 19:21:15.455 * Background saving terminated with success
4446:S 02 Jul 19:26:16.093 * 10 changes in 300 seconds. Saving...
4446:S 02 Jul 19:26:16.094 * Background saving started by pid 7615
7615:C 02 Jul 19:26:16.113 * DB saved on disk
7615:C 02 Jul 19:26:16.114 * RDB: 8 MB of memory used by copy-on-write
4446:S 02 Jul 19:26:16.197 * Background saving terminated with success
4446:S 02 Jul 19:41:17.032 * 1 changes in 900 seconds. Saving...
4446:S 02 Jul 19:41:17.033 * Background saving started by pid 8074
8074:C 02 Jul 19:41:17.055 * DB saved on disk
8074:C 02 Jul 19:41:17.055 * RDB: 8 MB of memory used by copy-on-write
4446:S 02 Jul 19:41:17.134 * Background saving terminated with success
4446:S 02 Jul 22:03:58.651 * 1 changes in 900 seconds. Saving...
4446:S 02 Jul 22:03:58.653 * Background saving started by pid 13916
13916:C 02 Jul 22:03:58.667 * DB saved on disk
13916:C 02 Jul 22:03:58.667 * RDB: 8 MB of memory used by copy-on-write
4446:S 02 Jul 22:03:58.754 * Background saving terminated with success
4446:S 03 Jul 00:14:43.282 * 1 changes in 900 seconds. Saving...
4446:S 03 Jul 00:14:43.284 * Background saving started by pid 19046
19046:C 03 Jul 00:14:43.307 * DB saved on disk

2 給出1中slave節點對應的主節點10.166.15.35:7002的異常斷電後重新啟動資訊

4440:C 02 Jul 18:07:51.105 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
4440:C 02 Jul 18:07:51.109 # Redis version=4.0.6, bits=64, commit=00000000, modified=0, pid=4440, just started
4440:C 02 Jul 18:07:51.109 # Configuration loaded
4441:M 02 Jul 18:07:51.122 * Node configuration loaded, I'm 1e7a4a6b7376a89726211ce04acdaf97afbe2f13
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 4.0.6 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in cluster mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 7002
|    `-._   `._    /     _.-'    |     PID: 4441
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               


4441:M 02 Jul 18:07:51.127 # Server initialized
4441:M 02 Jul 18:07:51.127 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
4441:M 02 Jul 18:07:51.127 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
4441:M 02 Jul 18:07:51.752 * DB loaded from append only file: 0.625 seconds
4441:M 02 Jul 18:07:51.752 * Ready to accept connections
4441:M 02 Jul 18:07:53.759 # Cluster state changed: ok
4441:M 02 Jul 18:07:56.721 * Slave 10.166.15.35:7001 asks for synchronization
4441:M 02 Jul 18:07:56.721 * Partial resynchronization not accepted: Replication ID mismatch (Slave asked for 'eda94ebd6d85f2f9dfb8337358ab14bdaee896bd', my replication IDs are 'd64784b6f440cfc58eb490428541b43c1cc7eecf' and '0000000000000000000000000000000000000000')
4441:M 02 Jul 18:07:56.721 * Starting BGSAVE for SYNC with target: disk
4441:M 02 Jul 18:07:56.722 * Background saving started by pid 4450
4450:C 02 Jul 18:07:56.740 * DB saved on disk
4450:C 02 Jul 18:07:56.741 * RDB: 6 MB of memory used by copy-on-write
4441:M 02 Jul 18:07:56.771 * Background saving terminated with success
4441:M 02 Jul 18:07:56.772 * Synchronization with slave 10.166.15.35:7001 succeeded
4441:M 02 Jul 18:08:06.807 # Cluster state changed: fail
4441:M 02 Jul 18:08:18.647 # Cluster state changed: ok
4441:M 02 Jul 18:08:29.585 * Marking node 052b9df159281988af20d8b5d0690e5defa42c47 as failing (quorum reached).

4441:M 02 Jul 18:08:29.586 * Marking node 6d6f6fbeff41f5a85a7a7ebe15565a448fb0a69f as failing (quorum reached).

--說明本節點為主節點 聯合36虛擬啟動的兩個主節點 確認了37上兩個slave節點宕機（超過半數的主投票確認,37上兩個節點晚啟動20分鐘）

4441:M 02 Jul 18:26:56.159 * Clear FAIL state for node 052b9df159281988af20d8b5d0690e5defa42c47: slave is reachable again.

4441:M 02 Jul 18:27:00.431 * Clear FAIL state for node 6d6f6fbeff41f5a85a7a7ebe15565a448fb0a69f: slave is reachable again.

--說明本節點為主節點 聯合36虛擬啟動的兩個主節點 確認了37上兩個slave節點恢復 重新加入叢集（超過半數的主投票確認）

4441:M 02 Jul 18:27:11.969 * 1 changes in 900 seconds. Saving...
4441:M 02 Jul 18:27:11.970 * Background saving started by pid 5249
5249:C 02 Jul 18:27:11.979 * DB saved on disk
5249:C 02 Jul 18:27:11.979 * RDB: 6 MB of memory used by copy-on-write
4441:M 02 Jul 18:27:12.072 * Background saving terminated with success
4441:M 02 Jul 18:42:13.032 * 1 changes in 900 seconds. Saving...
4441:M 02 Jul 18:42:13.034 * Background saving started by pid 6015
6015:C 02 Jul 18:42:13.068 * DB saved on disk
6015:C 02 Jul 18:42:13.069 * RDB: 8 MB of memory used by copy-on-write
4441:M 02 Jul 18:42:13.135 * Background saving terminated with success
4441:M 02 Jul 19:21:15.353 * 1 changes in 900 seconds. Saving...
4441:M 02 Jul 19:21:15.354 * Background saving started by pid 7288
7288:C 02 Jul 19:21:15.363 * DB saved on disk
7288:C 02 Jul 19:21:15.363 * RDB: 8 MB of memory used by copy-on-write
4441:M 02 Jul 19:21:15.455 * Background saving terminated with success

4441:M 02 Jul 19:26:16.093 * 10 changes in 300 seconds. Saving...

參考: https://www.cnblogs.com/yjmyzz/p/redis-cluster-turotial.html

ps：2018.07.12add

以上這種是理想情況下的，可以自動恢復，極端情況（多發生在高併發場景）

可能出現主從資料不一致，

有一些公司遇到叢集宕機某些節點後，重啟無法恢復叢集初始狀態，甚至叢集不可用的

情況 

相關推薦

redis cluster 全部宕機後重啟會自動恢復叢集狀態

昨天 測試環境上3主3從的redis節點叢集 虛擬機器3臺全部宕機(3主3從交叉部署在3臺虛機上)重新啟動各個節點發現  叢集自動恢復了  本來以為要重新使用create  命令猜測叢集是根據node的主從資訊檔案自己恢復的 利用心跳檢測 節點關係的檔案node-7001.c

redis主庫宕機後重啟，主庫和從庫的資料丟失

redis主庫宕機後重啟，主庫和從庫的資料丟失   解決方法： 假如主庫127.0.0.1 6379，從庫127.0.0.1 6380 1.在從資料庫中執行SLAVEOF NO ONE命令，斷開主從關係並且提升為主庫繼續服務； 2

Redis叢集宕機後重啟

 Redis叢集宕機問題： 如果redis掛掉，或者伺服器突然斷電，要刪除以下節點下檔案 ；我的節點路徑是 /redis/src 1)將每個節點下appendonly.aof dump.rdb no

vCenter宕機，重啟出現fsck failed無法啟動

現象 突然發現VC無響應，控制檯看結果如圖： 網上查資料 主要參考其中的 Option 2: Press Control+D or reboot the vCSA using the vSphere Client. When the GRUB boot

Android 系統（126）---Android的宕機、重啟問題分析方法

Android的宕機、重啟問題分析方法當手機長時間無法再被使用者控制操作時，我們稱為宕機。在這裡我們強調長時間，如果是短時間，歸結為效能問題。* 使用者操作手機無任何響應，如觸控式螢幕幕，按鍵操作等。* 手機螢幕黑屏，無法點亮螢幕。* 手機介面顯示內容和使用者輸入不相干。1. 系統簡圖當用戶對手機進行操作時，

EBS-異常宕機，重啟服務登陸連線超時

錯誤資訊如下： 參考官方文件： (Doc ID 2339121.1) 造成問題原因： 但是沒有更新補丁，ben例中出現的情況是由於，EBS服務異常關閉（伺服器直接重啟）導致的。 因此參考解決方法： SOLUTION To resolve the iss

Redis Cluster節點伺服器宕機後導致叢集重啟失敗案例

這裡說下自己碰到的一種情況： redis cluster叢集由三個節點伺服器組成，一個6個redis例項，每個節點開啟2個埠，三主三從。reids部署目錄是/data/redis-4.0.1，叢集情況如下： 172.16.50.245:7000 master主節點 1

Redis服務掛掉後,重啟時閃退

重啟 class exe 管理 body 整理 bsp gpo 服務 這個時候去進程管理器裏找一個 redisservice.exe 的進程..殺死他 殺死他 殺死他!!! 整理領結,嘬口咖啡, 嗯... 然後再來啟動服務.. Redis服務掛掉後,重啟時閃退

簡訊貓宕機後AT指令遠端重啟動

在某寶上購買的簡訊貓長時間無操作後，會自動關機，AT指令正常，SIM卡會不可用，打不通號碼，原因不明，可以通過以下指令重新啟動SIM卡： AT+CFUN=0 （停止模組） AT+CFUN=1 （啟動模組）

Redis 安裝 啟動 連接 配置 重啟

apt-get space sed .cn 置配 服務 clas cal itl Linux下安裝 ]# wget http://download.redis.io/releases/redis-2.8.17.tar.gz ]# tar xzf redis-2.8.

Oracle斷電後重啟異常

第一篇 acl blog 技術 重啟 顯示 sda 機器 用戶 　　這是我的第一篇博客~ 　　2017-06-23，我所在的項目現成由於機房斷電，導致項目所在的一臺數據庫服務器掛掉了。早上重啟後發現切換Oracle用戶異常，顯示這個界面 　　初步分析是：用戶還在，但是掛載有

一例mysql主從數據庫，從庫宕機後無法啟動的解決方案

mysql starting 啟動時報錯信息： Starting MySQL... ERROR! The server quit without updating PID file (/usr/local/mysql/data/qkzhi-appzookeeper-1.novalocal.pid

Redis Cluster集群搭建後，客戶端的連接研究（Spring/Jedis）（待實踐）

turn ron 記錄 redis div println 刪除 clu name 說明：無論是否已經搭建好集群，還是使用什麽樣的客戶端去連接，都是必須把全部IP列表集成進去，然後隨機往其中一個IP寫。 這樣做的好處： 1、隨機IP寫入之後，Redis Cluster代

Linux下安裝Oracle後重啟無法登錄數據庫ORA-01034:ORACLE not available

http use username connected 無法 -- .aspx data ase Linux下安裝了數據庫，安裝完成後可以用，今天啟動就不能用了，提示Oracle not available，後來查找資料，據說是oracle服務沒有打開。如下方式可以解決問題

U盤裝完系統後 重啟起不來故障解決

stage1 很多 修改 vi命令 沒有 啟動引導 eboot onf term 這個是運維新手經常碰到的問題，我就這麽被坑了一下網上有很多解決此故障的文檔，我做一下歸整，從問題的發生，到解決問題的思路再到步驟那麽我們開始第一步，是哪兒出的問題： 如果是對Boot Load

weblogic清理緩存後重啟

security cts dmi pro servers domain 認證 ins clas 清理緩存步驟如下： 1、前置條件：停止服務 2、找到下面3個目錄，然後將裏面的文件刪除即可： ……/user_projects/domains/base_domain/serve

如何解決WIN10系統的電腦一開機就提示“你的電腦將在一分鐘後重啟”並無限循環

jsb 重裝 新浪 微信 依次 image out 界面 netsh 遇到好多次這個問題了，但是我給他們的通用解決方法是重！置！系！統！，然而誰也不想總重置，畢竟軟件還都得重裝，於是乎今天我中招了！那麽怎麽解決呢？其實很簡單。一、進入安全模式1、按住shift鍵後點擊重啟並

centos安裝後重啟需要license

enter post enter鍵 令行 命令 script clas live entos 安裝完成centos7後出現 需要接受license，才能重啟，而且還是命令行格式 解決方案： 輸入【1】，按Enter鍵閱讀許可協議， 輸入【2】，按Enter鍵接受許

oracle 宕機後文件損壞resetlogs後處理

ora-00600 13013數據庫意外宕機，歸檔開了，但是歸檔文件損壞，redo損壞，在強行拉起來之後UNDO報錯，設置_corrupted_rollback_segments 跳過不一致的ＵＮＤＯ，重建UNOD表空間，接著報錯：SMON encountered 100 out of maximum 100

配置完php.ini中的擴展庫後,重啟apache出現錯誤1067

ini 沒有 更改 一次 是把 需要 解決 dir 重裝 網上有很多解決辦法,比如更改環境變量,重裝apache等等,但沒有一個是符合我的.最後發現只是犯了一個低級錯誤,因為是第一次配置php.ini中的擴展庫,忘記配置擴展庫的路徑. 解決辦法:需要先加上擴展庫路徑: ex