Mysql Crash Receovery

Revision as of 05:58, 13 August 2007 by Vagarwal (talk | contribs)
Verified Against: Zimbra Collaboration Suite 4.5.6 Date Created: 13/8/2007
Article ID: https://wiki.zimbra.com/index.php?title=Mysql_Crash_Receovery Date Modified: 2007-08-13



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »



http://bugzilla.zimbra.com/show_bug.cgi?id=15797


Mysql Crash recovery is performed when Mysql stops responding and the error in the /opt/zimbra/db/data/hostname.err is:-



Number of processes running now: 0
. 070725 16:24:00 mysqld restarted 070725 16:24:00 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... 070725 16:24:00 InnoDB: Starting log scan based on checkpoint at InnoDB: log sequence number 0 1787584935. InnoDB: Doing recovery: scanned up to log sequence number 0 1789698076 InnoDB: 1 transaction(s) which must be rolled back or cleaned up InnoDB: in total 125 row operations to undo InnoDB: Trx id counter is 0 2439168 070725 16:24:00 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed 070725 16:24:01 InnoDB: Started; log sequence number 0 1789698076 InnoDB: Starting in background the rollback of uncommitted transactions 070725 16:24:01 InnoDB: Rolling back trx with id 0 2438651, 125 rows to undo 070725 16:24:01 [Note] /opt/zimbra/mysql/libexec/mysqld: ready for connections. Version: '5.0.33-log' socket: '/opt/zimbra/db/mysql.sock' port: 7306 Source distribution InnoDB: Dump of the tablespace extent descriptor: len 40; hex 0000000000000002ffffffff0000000000000b1600000004ffffffbefaafffffffffbffeaaffffff; asc  ; InnoDB: Serious error! InnoDB is trying to free page 3348 InnoDB: though it is already marked as free in the tablespace! InnoDB: The tablespace free space info is corrupt. InnoDB: You may need to dump your InnoDB tables and recreate the whole InnoDB: database! InnoDB: Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: about forcing recovery. 070725 16:24:01InnoDB: Assertion failure in thread 2372672432 in file fsp0fsp.c line 2980 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: about forcing recovery. mysqld got signal 11; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.

key_buffer_size=8388600 read_buffer_size=1044480 max_used_connections=0 max_connections=21 threads_connected=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 51115 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.

thd=(nil) Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0x8d6c03a8, backtrace may not be correct. Stack range sanity check OK, backtrace follows: 0x816df93 0x832bb36 0x82ea91d 0x82ec69f 0x82d07a7 0x82c9701 0x82a9261 0x82fe620 0x4b7d4371 0x4b509ffe New value of fp=(nil) failed sanity check, terminating stack trace! Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved stack trace is much more helpful in diagnosing the problem, so please do resolve it The manual page at http://www.mysql.com/doc/en/Crashing.html contains information that should help you find out what is causing the crash.


Steps to perform Recovery:

  • Edit /opt/zimbra/conf/my.cnf: Put an entry :- innodb_force_recovery = 3
  • Dump the database:- mysqldump zimbra and mboxgroup DB's
  • Drop all Database's.
  • Create all Database's.
  • Restore dump data.

1) Edit the my.cnf file and put innodb_force_recovery = 3 : save the file and start mysql. [mysql.server start]


2) mysql -NB -e "show databases" | grep mbox > /tmp/mysql.db.list [Make a list of the existing database]


3) mkdir /tmp/mysql.sql


4) mysqldump zimbra -S /opt/zimbra/db/mysql.sock -u root -p AOJTGukheoAar080cauxqOyVt0khuUI > /tmp/mysql.sql/zimbra.sql


["AOJTGukheoAar080cauxqOyVt0khuUI" is the Mysql Root password]


5) Dumping Databases: Run the following command from the prompt:


for db in `< /tmp/mysql.db.list`; \do /opt/zimbra/mysql/bin/mysqldump $db -S /opt/zimbra/db/mysql.sock -u zimbra -ppvbaAlDCHQ8UtMcY2oqNo91kyyqdX0 >

/tmp/mysql.sql/$db.sql; \echo -e "Dumped $db\n"; \done;


6) Drp the Database:


for db in `< /tmp/mysql.db.list`; \ do mysql -e "drop database $db"; \echo -e "Dropped $db"; \done;


7) mysql -e "drop database zimbra"


8) Run Mysql in Non-Recovery Mode:


Remove the innodb_force_recovery = 3 from the my.cnf file.


Stop Mysql Server and Start again.


9) Create Database:


mysql -e "create database zimbra character set utf8"


10) Create the existed Database:


for db in `< /tmp/mysql.db.list`; \ do mysql -e "create database $db character set utf8"; \

echo -e "Created $db \n";done;  


11) mysql zimbra < /tmp/mysql.sql/zimbra.sql


12) Update Mboxgroup:


for sql in /tmp/mysql.sql/mbox*; \ do mysql `basename $sql .sql` < $sql; \ echo -e "Updated `basename $sql .sql` \n"; \done;


13) zmcontrol start


14) Check if everything is up and running.

Jump to: navigation, search