手動創建binary log files和手動編輯binary log index file會有什麽影響
阿新 • • 發佈：2018-01-13
href 生成 view 我們 htm 不知道 ica 字節 oss 
一、了解Binary Log結構
1.1、High-Level Binary Log Structure and Contents
• Binlog包括binary log files和index file
• 每個binary log文件的前4字節是Magic Number，緊接著是一組描述數據修改的Events
• The magic number bytes are 0xfe 0x62 0x69 0x6e = 0xfe ‘b‘‘i‘‘n‘
• 每個Event包括header bytes和data bytes
header bytes：Event類型，什麽時候，由哪個server生成等信息data bytes：特定的數據修改
• 第一個Event是Description Event，描述文件的格式版本
• 其他Events按照binlog_version(1、3、4)進行記錄
• 最後一個Event是Log-rotation Event，指定下一個binary log的文件名稱
• index文件裏面是當前binary log的一個列表
1.2、Binary Log結構示意圖
# Binary Log結構示意圖(binlog_version=4) |<----------------------------------Events---------------------------------->| |<---------Fisrt Event--------->|<------Mid Events----->|<---Final Event---->| +--------------+-------------------------------+--------+-----+--------+--------------------+ | Magic Number | FDE(Format Description Event) | Event1 | ... | EventN | Log-rotation Event | +--------------+-------------------------------+--------+-----+--------+--------------------+ |<-event header->|<-event data->|
上圖是binlog_version=4(MySQL 5.0 and up)的示意圖
二、index file添加一行不存在的binary log記錄
查看當前binary log
# 查看當前binary log [root@ZST1 logs]# ll total 20 -rw-r-----. 1 mysql mysql 201 Jan 12 11:32 mysql-bin.000004 -rw-r-----. 1 mysql mysql 201 Jan 12 11:32 mysql-bin.000005 -rw-r-----. 1 mysql mysql 177 Jan 12View Code11:34 mysql-bin.000006 -rw-r-----. 1 mysql mysql 177 Jan 12 11:37 mysql-bin.000007 -rw-r-----. 1 mysql mysql 220 Jan 12 11:34 mysql-bin.index [root@ZST1 logs]# cat mysql-bin.index /data/mysql/mysql3308/logs/mysql-bin.000004 /data/mysql/mysql3308/logs/mysql-bin.000005 /data/mysql/mysql3308/logs/mysql-bin.000006 /data/mysql/mysql3308/logs/mysql-bin.000007 [root@ZST1 logs]#
mysql-bin.index 添加一行已經刪除的記錄/data/mysql/mysql3308/logs/mysql-bin.000001，重新啟動數據庫服務
# 查看error log [root@ZST1 ~]# more /data/mysql/mysql3308/data/error.log ... 2018-01-12T03:34:33.467474Z 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/mysql3308/data/ib_buffer_pool mysqld: File ‘/data/mysql/mysql3308/logs/mysql-bin.000001‘ not found (Errcode: 2 - No such file or directory) 2018-01-12T03:34:33.546276Z 0 [ERROR] Failed to open log (file ‘/data/mysql/mysql3308/logs/mysql-bin.000001‘, errno 2) 2018-01-12T03:34:33.546294Z 0 [ERROR] Could not open log file 2018-01-12T03:34:33.563485Z 0 [Warning] Failed to set up SSL because of the following SSL library error: SSL context is not usable without certificate and private key 2018-01-12T03:34:33.563528Z 0 [Note] Server hostname (bind-address): ‘*‘; port: 3308 數據庫服務最終啟動起來View Code
mysql-bin.index 添加一行未來的記錄/data/mysql/mysql3308/logs/mysql-bin.000017，重新啟動數據庫服務
# 查看error log [root@ZST1 ~]# more /data/mysql/mysql3308/data/error.log ... 2018-01-12T03:44:45.921370Z 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/mysql3308/data/ib_buffer_pool mysqld: File ‘/data/mysql/mysql3308/logs/mysql-bin.000017‘ not found (Errcode: 2 - No such file or directory) 2018-01-12T03:44:46.017490Z 0 [ERROR] Failed to open log (file ‘/data/mysql/mysql3308/logs/mysql-bin.000017‘, errno 2) 2018-01-12T03:44:46.017509Z 0 [ERROR] Could not open log file 2018-01-12T03:44:46.017532Z 0 [ERROR] Can not init tc log 2018-01-12T03:44:46.017545Z 0 [ERROR] Aborting 數據庫服務最終退出View Code
啟動過程會讀取index file中的第一條和最後一條記錄，然後檢查對應的文件是否存在。如果第一條不存在，error log中會有報錯，但服務正常啟動；如果最後一條不存在，error log中會有報錯，服務退出。即使你打亂index file中列表順序，或者寫入其他信息，它始終只驗證index file的第一條和最後一條~
三、手動創建一個binary log，但不添加到index file
如果touch_seq < max(exists_seq)，正常啟動，最新的new_seq=max(exists_seq)+1
如果max(exists_seq) < touch_seq < pow(2,31)-1，正常啟動，最新的new_seq=touch_seq+1，當new_seq > pow(2,31)-1001 時會往error log中寫入警告
如果touch_seq = pow(2,31)-1，mysqld退出
如果touch_seq > pow(2,31)-1，正常啟動，會往error log中寫入警告
3.1、MySQL binlog後面的編號最大是多大
最大編號參考文章：MySQL binlog後面的編號最大是多大
原文結論：
1、MySQL binlog的最大sequence是：pow(2,31)-1 = 2147483647
2、當 pow(2,31)-1001 < sequence < pow(2,31)-1 會往error log中寫入警告
3、binlog的sequence達到最大值(2147483647)時，不管有沒有mysql-bin.000001類似這樣的文件，mysqld都是退出
4、在MySQL產生binlog時會讀取當前日件文目錄下的log-bin的base name獲取下一個日誌文件的後面的Seq。所以日誌目錄下文件太多，會影響MySQL的啟動及日誌切換。這裏也有一個大的隱患，運行中給放一個較大的日誌文件，在下次日誌文件切換時有可能很快就接近於最大值，造成mysqld crash退出
5、一定要監控error log的輸出，並足夠重視
查看源碼
# 源碼信息 sql/binlog.cc #define MAX_LOG_UNIQUE_FN_EXT 0x7FFFFFFF #define LOG_WARN_UNIQUE_FN_EXT_LEFT 1000 /* check if reached the maximum possible extension number */ if (max_found == MAX_LOG_UNIQUE_FN_EXT) { sql_print_error("Log filename extension number exhausted: %06lu. \ Please fix this by archiving old logs and updating the index files.", max_found); error= 1; goto end; } /* print warning if reaching the end of available extensions. */ if ((next > (MAX_LOG_UNIQUE_FN_EXT - LOG_WARN_UNIQUE_FN_EXT_LEFT))) sql_print_warning("Next log extension: %lu. \ Remaining log filename extensions: %lu. Please consider archiving some logs.", next, (MAX_LOG_UNIQUE_FN_EXT - next)); mysql> select conv(‘7FFFFFFF‘,16,10); +------------------------+ | conv(‘7FFFFFFF‘,16,10) | +------------------------+ | 2147483647 | +------------------------+View Code
源碼邏輯正是上面的結論1、2、3。如果我們的binlog的sequence瘋漲到2147483647，那很不幸mysqld會退出。。。但如果我們在正常情況下，touch一個大於2147483647的binary log的文件會出現什麽情況呢？
3.2、MySQL binlog後面的編號最大到底是多大
首先我們從2147483648開始
# reset master mydba@192.168.85.132,3308 [replcrash]> reset master; Query OK, 0 rows affected (0.01 sec) # 查看binary log mydba@192.168.85.132,3308 [replcrash]> show binary logs; +------------------+-----------+ | Log_name | File_size | +------------------+-----------+ | mysql-bin.000001 | 154 | +------------------+-----------+ row in set (0.00 sec) # touch 2^31 [root@ZST1 logs]# touch mysql-bin.2147483648 [root@ZST1 logs]# chown -R mysql:mysql * # flush執行5次 mydba@192.168.85.132,3308 [replcrash]> flush binary logs; Query OK, 0 rows affected (0.01 sec) # 查看binary log mydba@192.168.85.132,3308 [replcrash]> show binary logs; +----------------------+-----------+ | Log_name | File_size | +----------------------+-----------+ | mysql-bin.000001 | 205 | | mysql-bin.2147483649 | 205 | | mysql-bin.2147483650 | 205 | | mysql-bin.2147483651 | 205 | | mysql-bin.2147483652 | 205 | | mysql-bin.2147483653 | 154 | +----------------------+-----------+ rows in set (0.00 sec) # 查看error log [root@ZST1 data]# cat error.log 2018-01-08T09:09:06.235149Z 3 [Warning] Next log extension: 2147483649. Remaining log filename extensions: 18446744073709551614. Please consider archiving some logs. 2018-01-08T09:09:23.868883Z 3 [Warning] Next log extension: 2147483650. Remaining log filename extensions: 18446744073709551613. Please consider archiving some logs. 2018-01-08T09:10:29.670811Z 3 [Warning] Next log extension: 2147483651. Remaining log filename extensions: 18446744073709551612. Please consider archiving some logs. 2018-01-08T09:10:30.401541Z 3 [Warning] Next log extension: 2147483652. Remaining log filename extensions: 18446744073709551611. Please consider archiving some logs. 2018-01-08T09:10:31.075702Z 3 [Warning] Next log extension: 2147483653. Remaining log filename extensions: 18446744073709551610. Please consider archiving some logs. [root@ZST1 logs]# # 再次reset master mydba@192.168.85.132,3308 [replcrash]> reset master; Query OK, 0 rows affected (0.02 sec) # 查看binary log mydba@192.168.85.132,3308 [replcrash]> show binary logs; +----------------------+-----------+ | Log_name | File_size | +----------------------+-----------+ | mysql-bin.2147483649 | 154 | +----------------------+-----------+ row in set (0.00 sec)View Code
可以看到，執行flush binary logs後，sequence一直往後+1(next)。error log中會有警告日誌，next > (MAX_LOG_UNIQUE_FN_EXT - LOG_WARN_UNIQUE_FN_EXT_LEFT)恒成立；Remaining log filename extensions後面的數值看著有點怪異~
第二次reset後的sequence從2147483649開始，並不是從000001開始，不知道什麽原因。
我們繼續加大sequence，直到接近9223372036854775808
# touch 2^63-2 [root@ZST1 logs]# touch mysql-bin.9223372036854775806 [root@ZST1 logs]# chown -R mysql:mysql * # flush執行5次 mydba@192.168.85.132,3308 [replcrash]> flush binary logs; Query OK, 0 rows affected (0.01 sec) # 查看binary log mydba@192.168.85.132,3308 [replcrash]> show binary logs; +-------------------------------+-----------+ | Log_name | File_size | +-------------------------------+-----------+ | mysql-bin.000001 | 214 | | mysql-bin.9223372036854775807 | 214 | | mysql-bin.9223372036854775808 | 154 | | mysql-bin.9223372036854775808 | 154 | | mysql-bin.9223372036854775808 | 154 | | mysql-bin.9223372036854775808 | 154 | +-------------------------------+-----------+ rows in set (0.00 sec) # 查看error log [root@ZST1 data]# cat error.log 2018-01-08T09:35:20.879612Z 3 [Warning] Next log extension: 9223372036854775807. Remaining log filename extensions: 9223372039002259456. Please consider archiving some logs. 2018-01-08T09:35:22.082530Z 3 [Warning] Next log extension: 9223372036854775808. Remaining log filename extensions: 9223372039002259455. Please consider archiving some logs. 2018-01-08T09:35:22.934636Z 3 [Warning] Next log extension: 9223372036854775808. Remaining log filename extensions: 9223372039002259455. Please consider archiving some logs. 2018-01-08T09:35:23.544604Z 3 [Warning] Next log extension: 9223372036854775808. Remaining log filename extensions: 9223372039002259455. Please consider archiving some logs. 2018-01-08T09:35:24.134664Z 3 [Warning] Next log extension: 9223372036854775808. Remaining log filename extensions: 9223372039002259455. Please consider archiving some logs. [root@ZST1 logs]#View Code
當sequence達到9223372036854775808後，再flush binary log，看不出產生新binary log~
[root@ZST1 logs]# mysqlbinlog -v --base64-output=decode-rows mysql-bin.9223372036854775808 /*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; # at 4 #180112 15:57:18 server id 1323308 end_log_pos 123 CRC32 0x798b200f Start: binlog v 4, server v 5.7.19-log created 180112 15:57:18 # Warning: this binlog is either in use or was not closed properly. # at 123 #180112 15:57:18 server id 1323308 end_log_pos 154 CRC32 0xf3252c32 Previous-GTIDs # [empty] # at 154 #180112 15:57:18 server id 1323308 end_log_pos 214 CRC32 0x6054f256 Rotate to mysql-bin.9223372036854775808 pos: 4 SET @@SESSION.GTID_NEXT= ‘AUTOMATIC‘ /* added by mysqlbinlog */ /*!*/; DELIMITER ; # End of log file /*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/; /*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/; [root@ZST1 logs]#View Code
解析binlog可以在末尾看到Rotate to mysql-bin.9223372036854775808 pos: 4
也就是說再怎麽切，也只能切到9223372036854775808，這個文件感覺有點不太正常，不受max_binlog_size限制，解析出的日誌莫名不完整~
恢復原始：手工刪除 mysql-bin.2147483647 這類大文件，然後修改 mysql-bin.index；或者直接清空logs文件夾，再重新啟動數據庫服務
四、參考文檔
MySQL binlog後面的編號最大是多大：http://wubx.net/mysql-binlog-max-sequence/
High-Level Binary Log Structure and Contents：https://dev.mysql.com/doc/internals/en/binary-log-structure-and-contents.html
Binary Log Versions：https://dev.mysql.com/doc/internals/en/binary-log-versions.html
Event Structure：https://dev.mysql.com/doc/internals/en/event-structure.html
手動創建binary log files和手動編輯binary log index file會有什麽影響