一 背景
某一創業的朋友的主機因為磁盤陣列損壞機器crash,重啟MySQL服務時 報如下錯誤:
復制代碼 代碼如下:
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 9120034833
150125 16:12:51 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 150125 16:12:51 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 5.5.37-MariaDB-log
key_buffer_size=268435456
read_buffer_size=1048576
max_used_connections=0
max_threads=1002
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2332093 K bytes of memory
41 Hope that.
二 分析
主要關注 mysqld got signal 11 的問題,從日志內容分析來看,數據庫在機器crash 導致日志文件損壞,重啟之後無法正常恢復,更無法正常對外提供服務。
三 解決
因為日志已經損壞，這裡采用非常規手段,首先修改innodb_force_recovery參數，使mysqld跳過恢復步驟，將mysqld 啟動,將數據導出來然後重建數據庫。
innodb_force_recovery可以設置為1-6,大的數字包含前面所有數字的影響。
1. (SRV_FORCE_IGNORE_CORRUPT):忽略檢查到的corrupt頁。
2. (SRV_FORCE_NO_BACKGROUND):阻止主線程的運行，如主線程需要執行full purge操作，會導致crash。
3. (SRV_FORCE_NO_TRX_UNDO):不執行事務回滾操作。
4. (SRV_FORCE_NO_IBUF_MERGE):不執行插入緩沖的合並操作。
5. (SRV_FORCE_NO_UNDO_LOG_SCAN):不查看重做日志，InnoDB存儲引擎會將未提交的事務視為已提交。
6. (SRV_FORCE_NO_LOG_REDO):不執行前滾的操作。
注意
a 當設置參數值大於0後，可以對表進行select,create,drop操作,但insert,update或者delete這類操作是不允許的。
b 當innodb_purge_threads 和 innodb_force_recovery一起設置會出現一種loop現象:
復制代碼 代碼如下:
150125 17:07:42 InnoDB: Waiting for the background threads to start
150125 17:07:43 InnoDB: Waiting for the background threads to start
150125 17:07:44 InnoDB: Waiting for the background threads to start
150125 17:07:45 InnoDB: Waiting for the background threads to start
150125 17:07:46 InnoDB: Waiting for the background threads to start
150125 17:07:47 InnoDB: Waiting for the background threads to start
在my.cnf中修改以下兩個參數
復制代碼 代碼如下:
innodb_force_recovery=6
innodb_purge_thread=0
重啟MySQL
復制代碼 代碼如下:
150125 17:10:47 [Note] Crash recovery finished.
150125 17:10:47 [Note] Server socket created on IP: '0.0.0.0'.
150125 17:10:47 [Note] Event Scheduler: Loaded 0 events
150125 17:10:47 [Note] /vdata/webserver/mysql/bin/mysqld: ready for connections.
Version: '5.5.37-MariaDB-log' socket: '/tmp/mysql.sock' port: 3306 Source distribution
立即對數據庫做邏輯導出 ，完成之後將innodb_force_recovery設置為0 ，innodb_purge_thread=1 ,然後重建數據庫 。
另外 MySQL 版本 5.5以及之前 ,當innodb_purge_threads =1，innodb_force_recovery >1 的情況會出現上文提到的循環報warning 問題（=1 沒有問題），
原因：
MySQL 的源代碼中顯示 當innodb_purge_threads 和 innodb_force_recovery一起設置會出現loop循環
復制代碼 代碼如下:
while (srv_shutdown_state == SRV_SHUTDOWN_NONE) {
if (srv_thread_has_reserved_slot(SRV_MASTER) == ULINT_UNDEFINED
|| (srv_n_purge_threads == 1
&& srv_thread_has_reserved_slot(SRV_WORKER)
== ULINT_UNDEFINED)) {
ut_print_timestamp(stderr);
fprintf(stderr, " InnoDB: Waiting for the background threads to start\n");
os_thread_sleep(1000000);
} else {
break;
}
}
所以當需要設置innodb_force_recovery>1的時候需要關閉 innodb_purge_threads，設置為0（默認）。
四 小結
MySQL crash 或者 MySQL 數據庫服務器 crash 會導致各種各樣的問題 ，比如主備之間的error 1594 (5.6 版本開啟crash-safe ，會最大程度上避免 error 1594的問題，以後會寫5.6新特性介紹該功能 )，error 1236， 日志損壞，數據文件損壞 ，等等，本案例只是其中的一種，細心從日志中找的相關錯誤提示，逐步解決即可。
