磁盤空間不夠導致mysql崩潰重啟
起因：
群裏有人提了句pt-ioprofile，我不知道，就查了查，想測一測，想以後可能會有幫助。
為了能看到效果，我選擇了我虛擬機上最大的壓測表Sbtest1，該表有100w數據，執行update sbtest1 set k=k+1;
並且通過pt-ioprofile查看到了想要的結果，然後就幹別的去了，下午了，看update sbtest1 set k=k+1;這個窗口的光標還閃著，以為還沒執行完，不停地回車，crtl c，各種不好用。過了一會兒，報錯了，並且提示mysql已經重啟了。
我去他嘞
報錯信息為：Binary logging not possible. Message: An error occurred during flush stage of the commit.‘binlog_error_action‘ is set to ‘ABORT_SERVER‘. Hence aborting the server.
給出這個，我也看不明白
查看錯誤日誌：
2018-11-01T11:25:54.493321+08:00 895 [Note] Access denied for user ‘root‘@‘localhost‘ (using password: NO)
2018-11-01T11:29:17.566331+08:00 896 [ERROR] Disk is full writing ‘/data/mysql/mysql-bin.000025‘ (Errcode: 16026912 - No space left on device). Waiting for someone to free space...
2018-11-01T11:29:17.566355+08:00 896 [ERROR] Retry in 60 secs. Message reprinted in 600 secs
2018-11-01T11:30:17.567705+08:00 896 [ERROR] Retry in 60 secs. Message reprinted in 600 secs
2018-11-01T11:40:17.641906+08:00 896 [ERROR] Disk is full writing ‘/data/mysql/mysql-bin.000025‘ (Errcode: 16026912 - No space left on device). Waiting for someone to free space...
2018-11-01T11:40:17.707079+08:00 896 [ERROR] Disk is full writing for someone to free space...
2018-11-01T11:40:17.642048+08:00 896 [ERROR] Retry in 60 secs. Message reprinted in 600 secs
2018-11-01T11:50:19.000088+08:00 896 [ERROR] /usr/local/mysql/bin/mysqld: Binary logging not possible. Message: An error occurred during flush stage of the commit. ‘binlog_error_action‘ is set to ‘ABORT_SERVER‘. Hence aborting the server.
06:08:19 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
key_buffer_size=134217728
read_buffer_size=524288
max_used_connections=3
max_threads=10000
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 15624665 K bytes of memory
Hope that‘s ok; if not, decrease some variables in the equation.
Thread pointer: 0x5aa4a70
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fa568c81e58 thread_stack 0x40000
/usr/local/mysql/bin/mysqld(my_print_stacktrace+0x35)[0xf4a495]
/usr/local/mysql/bin/mysqld(handle_fatal_signal+0x4a4)[0x7ce2f4]
/lib64/libpthread.so.0[0x3c9bc0f670]
/lib64/libc.so.6(gsignal+0x3e)[0x3c9b4322fe]
/lib64/libc.so.6(abort+0x175)[0x3c9b433745]
/usr/local/mysql/bin/mysqld[0xee249a]
/usr/local/mysql/bin/mysqld(_ZN13MYSQL_BIN_LOG33handle_binlog_flush_or_sync_errorEP3THDb+0x163)[0xef0bd3]
/usr/local/mysql/bin/mysqld(_ZN13MYSQL_BIN_LOG14ordered_commitEP3THDbb+0x3ca)[0xef106a]
/usr/local/mysql/bin/mysqld(_ZN13MYSQL_BIN_LOG6commitEP3THDb+0x585)[0xef1825]
/usr/local/mysql/bin/mysqld(_Z15ha_commit_transP3THDbb+0x174)[0x81f594]
/usr/local/mysql/bin/mysqld(_Z17trans_commit_stmtP3THD+0x32)[0xdd1272]
/usr/local/mysql/bin/mysqld(_Z21mysql_execute_commandP3THDb+0x707)[0xd161f7]
/usr/local/mysql/bin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x40d)[0xd1af7d]
/usr/local/mysql/bin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x119a)[0xd1c19a]
/usr/local/mysql/bin/mysqld(_Z10do_commandP3THD+0x194)[0xd1d044]
/usr/local/mysql/bin/mysqld(handle_connection+0x29c)[0xded7ac]
/usr/local/mysql/bin/mysqld(pfs_spawn_thread+0x174)[0xf707b4]
/lib64/libpthread.so.0[0x3c9bc06cea]
/lib64/libc.so.6(clone+0x6d)[0x3c9b4d7fad]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (5b066e0): update sbtest1 set k=k+1
Connection ID (thread ID): 896
Status: KILL_QUERY
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2018-11-01T14:08:19.463179+08:00 0 [Note] --secure-file-priv is set to NULL. Operations related to importing and exporting data are disabled
2018-11-01T14:08:19.463280+08:00 0 [Note] /usr/local/mysql/bin/mysqld (mysqld 5.7.21-log) starting as process 54682 ...
就現象看，即使磁盤空間滿了，mysql不會立刻宕機，而是每一分鐘做一次檢查，在600秒內將錯誤信息記錄到錯誤日誌。直到我瞎按，觸發了重啟。
由於不知所措，期間並沒有執行show full processlist查看，一切都是後知後覺
重新測試一下：空間寫滿
錯誤日誌開始循環打印信息
State query end
日誌不記錄了，手動ctrl c
日誌開始kill query
Mysql重啟
空間還原
Binlog之前記錄的也被清理了，重啟後生成個新的27
前：
後：
如果手動kill query是不是不會導致restart？
執行kill操作依然觸發了重啟
結論：所以還是要實時監控空間大小啊。
磁盤空間不夠導致mysql崩潰重啟