記一次mysql事故---紀念逝去的一上午
阿新 • • 發佈：2017-09-05
not 路徑 內部 oca oss ica relay its scala 
虛擬機關機後第二天mysql起不來，回想一下我關機前和關機後的操作發現：關機前沒關閉mysqld服務就直接init 0了，關機後將虛擬機內存由1G降到724M。筆者保證再也做過別的騷操作了。
2017-09-05 09:19:21 1940 [Note] Plugin ‘FEDERATED‘ is disabled. 2017-09-05 09:19:21 1940 [Note] InnoDB: Using atomics to ref count buffer pool pages 2017-09-05 09:19:21 1940 [Note] InnoDB: The InnoDB memory heap is disabled日誌2017-09-05 09:19:21 1940 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2017-09-05 09:19:21 1940 [Note] InnoDB: Memory barrier is not used 2017-09-05 09:19:21 1940 [Note] InnoDB: Compressed tables use zlib 1.2.3 2017-09-05 09:19:21 1940 [Note] InnoDB: Using CPU crc32 instructions 2017-09-05 09:19:21 1940 [Note] InnoDB: Initializing buffer pool, size = 128.0MInnoDB: mmap(137363456 bytes) failed; errno 12 2017-09-05 09:19:21 1940 [ERROR] InnoDB: Cannot allocate memory for the buffer pool 2017-09-05 09:19:21 1940 [ERROR] Plugin ‘InnoDB‘ init function returned error. 2017-09-05 09:19:21 1940 [ERROR] Plugin ‘InnoDB‘ registration as a STORAGE ENGINE failed. 2017-09-05 09:19:21 1940 [ERROR] Unknown/unsupported storage engine: InnoDB 2017-09-05 09:19:21 1940 [ERROR] Aborting 2017-09-05 09:19:21 1940 [Note] Binlog end 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘partition‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_SYS_DATAFILES‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_SYS_TABLESPACES‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_SYS_FOREIGN_COLS‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_SYS_FOREIGN‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_SYS_FIELDS‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_SYS_COLUMNS‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_SYS_INDEXES‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_SYS_TABLESTATS‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_SYS_TABLES‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_FT_INDEX_TABLE‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_FT_INDEX_CACHE‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_FT_CONFIG‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_FT_BEING_DELETED‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_FT_DELETED‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_FT_DEFAULT_STOPWORD‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_METRICS‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_BUFFER_POOL_STATS‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_BUFFER_PAGE_LRU‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_BUFFER_PAGE‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_CMP_PER_INDEX_RESET‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_CMP_PER_INDEX‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_CMPMEM_RESET‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_CMPMEM‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_CMP_RESET‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_CMP‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_LOCK_WAITS‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_LOCKS‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘INNODB_TRX‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘PERFORMANCE_SCHEMA‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘ARCHIVE‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘BLACKHOLE‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘MyISAM‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘MRG_MYISAM‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘CSV‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘MEMORY‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘sha256_password‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘mysql_old_password‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘mysql_native_password‘ 2017-09-05 09:19:21 1940 [Note] Shutting down plugin ‘binlog‘ 2017-09-05 09:19:21 1940 [Note] /usr/local/mysql/bin/mysqld: Shutdown complete
2017-09-05 09:19:21 1940 [ERROR] InnoDB: Cannot allocate memory for the buffer pool 2017-09-05 09:19:21 1940 [ERROR] Plugin ‘InnoDB‘ init function returned error. 2017-09-05 09:19:21 1940 [ERROR] Plugin ‘InnoDB‘ registration as a STORAGE ENGINE failed. 2017-09-05 09:19:21 1940 [ERROR] Unknown/unsupported storage engine: InnoDB 2017-09-05 09:19:21 1940 [ERROR] Aborting
貌似是innodb引擎錯誤，還有就是分配內存失敗，內存分配失敗會導致innodb錯誤嗎，還是斷電導致的？不得而知！！！
百度答案花樣百出，最後試著在/etc/my.cnf中加入下列一行再重啟
tmp_table_size=500M
***************************************
PS:下列谷歌出來的方法筆者沒試過，可以一試：
主要是innodb日誌大小不對造成的 innodb_buffer_pool_size=512M innodb_log_file_size=128M 把以上幾個參數調大點，然後把ibdata1，ib_logfile0，ib_logfile1這三個文件刪除後，再重啟服務就好了。 如果還是不行，在my.ini文件裏還需要加入tmpdir="usr/tmp"這樣的路徑，因為innodb還需要一個臨時的文件緩存區。沒試過的方法
****************************************
事實證明沒用，該句只是用來設置內部（內存中）臨時表的最大大小
重啟後仍然報錯，網上解決方法一般是：
問題描述： 非正常關閉mysql，同時更改了my.cnf 導致啟動時不支持innodb，出現如下錯誤： [ERROR] Plugin ‘InnoDB’ init function returned error. [ERROR] Plugin ‘InnoDB’ registration as a STORAGE ENGINE failed. [ERROR] Unknown/unsupported table type: InnoDB [ERROR] Aborting 解決辦法： 刪除log文件和data的初始化文件 重新運行mysql_install_db 再啟動網上某解決方案
筆者並沒有刪除文件，只是將/data/dbdata下的三個文件移動到/tmp下備份了，萬一以後用得著呢？
三個文件：ibdata1、ib_logfile0、ib_logfile1
再用mysql-install-db初始化
但是這一系列操作後mysqld仍然啟動不了，直接關機，將虛擬機內存調回1G，沒想到居然好了，可以成功啟動mysqld服務啦！
##########
但是這還不夠，進入數據庫後出現了奇怪的一幕：數據庫和表都在，卻打不開，什麽鬼
mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | binlog | | log | | mysql | | performance_schema | | test | | zabbix | +--------------------+ 7 rows in set (0.00 sec) mysql> use zabbix; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed mysql> show tables; +----------------------------+ | Tables_in_zabbix | +----------------------------+ | acknowledges | | actions | | alerts | | application_discovery | | application_prototype | | application_template | | applications | | auditlog | | auditlog_details | | autoreg_host | | conditions | | config | | corr_condition | | corr_condition_group | | corr_condition_tag | | corr_condition_tagpair | | corr_condition_tagvalue | | corr_operation | | correlation | | dbversion | | dchecks | | dhosts | | drules | | dservices | | escalations | | event_recovery | | event_tag | | events | | expressions | | functions | | globalmacro | | globalvars | | graph_discovery | | graph_theme | | graphs | | graphs_items | | group_discovery | | group_prototype | | groups | | history | | history_log | | history_str | | history_text | | history_uint | | host_discovery | | host_inventory | | hostmacro | | hosts | | hosts_groups | | hosts_templates | | housekeeper | | httpstep | | httpstepitem | | httptest | | httptestitem | | icon_map | | icon_mapping | | ids | | images | | interface | | interface_discovery | | item_application_prototype | | item_condition | | item_discovery | | items | | items_applications | | maintenances | | maintenances_groups | | maintenances_hosts | | maintenances_windows | | mappings | | media | | media_type | | opcommand | | opcommand_grp | | opcommand_hst | | opconditions | | operations | | opgroup | | opinventory | | opmessage | | opmessage_grp | | opmessage_usr | | optemplate | | problem | | problem_tag | | profiles | | proxy_autoreg_host | | proxy_dhistory | | proxy_history | | regexps | | rights | | screen_user | | screen_usrgrp | | screens | | screens_items | | scripts | | service_alarms | | services | | services_links | | services_times | | sessions | | slides | | slideshow_user | | slideshow_usrgrp | | slideshows | | sysmap_element_url | | sysmap_url | | sysmap_user | | sysmap_usrgrp | | sysmaps | | sysmaps_elements | | sysmaps_link_triggers | | sysmaps_links | | task | | task_close_problem | | timeperiods | | trends | | trends_uint | | trigger_depends | | trigger_discovery | | trigger_tag | | triggers | | users | | users_groups | | usrgrp | | valuemaps | +----------------------------+ 127 rows in set (0.00 sec) mysql> desc valuemaps; ERROR 1146 (42S02): Table ‘zabbix.valuemaps‘ doesn‘t exist mysql> desc usrgrp; ERROR 1146 (42S02): Table ‘zabbix.usrgrp‘ doesn‘t exist mysql> select * from users; ERROR 1146 (42S02): Table ‘zabbix.users‘ doesn‘t exist mysql> select user(); +----------------+ | user() | +----------------+ | [email protected] | +----------------+ 1 row in set (0.00 sec) mysql>數據庫操作
日誌文件中大量報錯：
2017-09-05 13:57:26 4817 [Warning] InnoDB: Cannot open table zabbix/dbversion from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem. 2017-09-05 13:57:31 4817 [Warning] InnoDB: Cannot open table zabbix/dbversion from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem. 2017-09-05 13:57:41 4817 [Warning] InnoDB: Cannot open table zabbix/dbversion from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem. 2017-09-05 13:57:49 4817 [Warning] InnoDB: Cannot open table mysql/innodb_index_stats from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem. 2017-09-05 13:57:49 4817 [Warning] InnoDB: Cannot open table mysql/innodb_table_stats from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem. 2017-09-05 13:57:49 4817 [Warning] InnoDB: Cannot open table mysql/slave_master_info from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem. 2017-09-05 13:57:49 4817 [Warning] InnoDB: Cannot open table mysql/slave_relay_log_info from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem. 2017-09-05 13:57:49 4817 [Warning] InnoDB: Cannot open table mysql/slave_worker_info from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem. 2017-09-05 13:57:51 4817 [Warning] InnoDB: Cannot open table zabbix/dbversion from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem. 2017-09-05 13:57:56 4817 [Warning] InnoDB: Cannot open table zabbix/dbversion from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.cannot open file
各種表都不能訪問，包括下列五個表：
innodb_table_stats innodb_index_stats slave_master_info slave_relay_log_info slave_worker_info
之前備份過舊的ibdata1文件
雖然重新初始化之後,數據庫會自動創建一個ibdata1文件,但是上述系統表也是innodb引擎,所以不能訪問了.
關閉mysql服務後，嘗試著將之前備份的ibdata1恢復。結果很神奇的解決了這個問題！數據庫能成功讀寫了，但查看日誌文件後發現還是有問題：
2017-09-05 12:33:11 3421 [Note] InnoDB: Using atomics to ref count buffer pool pages 2017-09-05 12:33:11 3421 [Note] InnoDB: The InnoDB memory heap is disabled 2017-09-05 12:33:11 3421 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2017-09-05 12:33:11 3421 [Note] InnoDB: Memory barrier is not used 2017-09-05 12:33:11 3421 [Note] InnoDB: Compressed tables use zlib 1.2.3 2017-09-05 12:33:11 3421 [Note] InnoDB: Using CPU crc32 instructions 2017-09-05 12:33:11 3421 [Note] InnoDB: Initializing buffer pool, size = 128.0M 2017-09-05 12:33:11 3421 [Note] InnoDB: Completed initialization of buffer pool 2017-09-05 12:33:11 3421 [Note] InnoDB: Highest supported file format is Barracuda. 2017-09-05 12:33:11 3421 [Warning] InnoDB: The log sequence number in the ibdata files is higher than the log sequence number in the ib_logfiles! Are you sure you are using the right ib_logfiles to start up the database. Log sequence number in the ib_logfiles is 1600617, logsequence numbers stamped to ibdata file headers are between 3116912313 and 3116912313. 2017-09-05 12:33:11 3421 [Note] InnoDB: The log sequence numbers 3116912313 and 3116912313 in ibdata files do not match the log sequence number 1600617 in the ib_logfiles! 2017-09-05 12:33:11 3421 [Note] InnoDB: Database was not shutdown normally! 2017-09-05 12:33:11 3421 [Note] InnoDB: Starting crash recovery. 2017-09-05 12:33:11 3421 [Note] InnoDB: Reading tablespace information from the .ibd files... 2017-09-05 12:33:11 3421 [Note] InnoDB: Restoring possible half-written data pages 2017-09-05 12:33:11 3421 [Note] InnoDB: from the doublewrite buffer... 2017-09-05 12:33:11 7f30b98c47e0 InnoDB: Error: page 7 log sequence number 3116899374 InnoDB: is in the future! Current system log sequence number 1600617. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: for more information. 2017-09-05 12:33:11 7f30b98c47e0 InnoDB: Error: page 2 log sequence number 3116912313 InnoDB: is in the future! Current system log sequence number 1600617. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: for more information. 2017-09-05 12:33:11 7f30b98c47e0 InnoDB: Error: page 4 log sequence number 3116705336 InnoDB: is in the future! Current system log sequence number 1600617. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: for more information. 。。。日誌超前
記一次mysql事故---紀念逝去的一上午