King0770-Notes-When innodb force recovery Fails

King Notes when InnoDB force recovery fails

   KB 3027        Last updated on 2015-07-12  

(0 votes)

Scenario: You just found out that mysql will not start up. In an attempt to do a mysqldump and reload, you edit the ~/my.cnf file to have the innodb_force_recovery setting. You tried values 1 through 6 for innodb_force_recovery, however, mysql will not start.

In this scenario, if mysql will not start, you will need to do a restore, and play the redo files to get the server back to it's current state.

1) zmcontrol stop

2) mv ~/db/data ~/db/data.OLD

3) source ~/bin/zmshutil ; zmsetvars

 /opt/zimbra/libexec/zmmyinit --sql_root_pw $mysql_root_password (this was done to initialize the data, i.e to have a clean state)

4) ldap start

5) zmconvertctl start

6) zmrestoreoffline -a all --excludeSearchIndex --excludeBlobs --excludeHsmBlobs --ignoreRedoErrors -c
6) zmrestoreoffline -a all --excludeSearchIndex --excludeBlobs --excludeHsmBlobs --ignoreRedoErrors -c -rf -lb <latest_full_backup_label>
** If restoring the full backup using "-rf" then you will need to run the redolog files manually as mentioned below.

While the restore is working, you will need to figure out which redo files to play. The best method is to open the session.xml file from the latest full backup and check the beginning and ending sequence numbers included in the backup.


minRedoSeq="638" maxRedoSeq="638"

**Note** minRedoSeq="638" maxRedoSeq="638"

In our example lets look at the redo files from the incremental backups.

-rw-r----- 1 zimbra zimbra   5079776 Sep  7 21:00 incr-20090908.010012.512/redologs/redo-20090908.010012.511-seq622.log
-rw-r----- 1 zimbra zimbra 117327479 Sep  8 21:00 incr-20090909.010013.946/redologs/redo-20090908.144204.162-seq623.log
-rw-r----- 1 zimbra zimbra 111285083 Sep  8 21:00 incr-20090909.010013.946/redologs/redo-20090908.144303.388-seq624.log
-rw-r----- 1 zimbra zimbra 105475888 Sep  8 21:00 incr-20090909.010013.946/redologs/redo-20090908.184251.115-seq625.log
-rw-r----- 1 zimbra zimbra  17637693 Sep  8 21:00 incr-20090909.010013.946/redologs/redo-20090909.010013.944-seq626.log
-rw-r----- 1 zimbra zimbra 118385579 Sep  9 21:00 incr-20090910.010014.770/redologs/redo-20090909.143443.347-seq627.log
-rw-r----- 1 zimbra zimbra 146039792 Sep  9 21:00 incr-20090910.010014.770/redologs/redo-20090909.143544.051-seq628.log
-rw-r----- 1 zimbra zimbra 219058845 Sep  9 21:01 incr-20090910.010014.770/redologs/redo-20090909.143623.940-seq629.log
-rw-r----- 1 zimbra zimbra 146032102 Sep  9 21:01 incr-20090910.010014.770/redologs/redo-20090909.143637.179-seq630.log
-rw-r----- 1 zimbra zimbra 146032140 Sep  9 21:01 incr-20090910.010014.770/redologs/redo-20090909.143637.223-seq631.log
-rw-r----- 1 zimbra zimbra 112605113 Sep  9 21:02 incr-20090910.010014.770/redologs/redo-20090909.144136.380-seq632.log
-rw-r----- 1 zimbra zimbra 104924041 Sep  9 21:02 incr-20090910.010014.770/redologs/redo-20090909.155319.532-seq633.log
-rw-r----- 1 zimbra zimbra  52520210 Sep  9 21:02 incr-20090910.010014.770/redologs/redo-20090910.010014.766-seq634.log
-rw-r----- 1 zimbra zimbra 103305944 Sep 10 21:00 incr-20090911.010012.175/redologs/redo-20090911.010012.173-seq635.log
-rw-r----- 1 zimbra zimbra 125025922 Sep 12 21:00 incr-20090913.010016.581/redologs/redo-20090911.194711.592-seq636.log
-rw-r----- 1 zimbra zimbra 134086559 Sep 12 21:01 incr-20090913.010016.581/redologs/redo-20090911.211919.123-seq637.log
-rw-r----- 1 zimbra zimbra 103718532 Sep 12 21:01 incr-20090913.010016.581/redologs/redo-20090913.010016.580-seq638.log
-rw-r----- 1 zimbra zimbra   4899300 Sep 13 21:00 incr-20090914.010015.716/redologs/redo-20090914.010015.715-seq639.log
-rw-r----- 1 zimbra zimbra 115999506 Sep 14 21:00 incr-20090915.010013.036/redologs/redo-20090914.145143.501-seq640.log
-rw-r----- 1 zimbra zimbra  92986896 Sep 14 21:00 incr-20090915.010013.036/redologs/redo-20090915.010012.700-seq641.log

Basically, you will only need to run the redo files from the latest full backup references, in our example, minRedoSeq="638" maxRedoSeq="638", and play the redo logs that are sequentially higher.

In our example we would run:

zmplayredo --logfiles redo-20090913.010016.580-seq638.log

zmplayredo --logfiles redo-20090914.010015.715-seq639.log

zmplayredo --logfiles redo-20090914.145143.501-seq640.log

zmplayredo --logfiles redo-20090915.010012.700-seq641.log

Also, don't forget to play the redo log in /opt/zimbra/redolog/archive/

-rw-r----- 1 zimbra zimbra 11607806 Sep 15 13:06 redo-20090915.170630.142-seq642.log

Again, note the sequence number.

Lastly, play the redo log from /opt/zimbra/redolog/

-rw-r----- 1 zimbra zimbra 6340 Sep 15 13:08 redo.log

It's possible this will cause some errors and some operations to be done twice, but we don't think there's any other way to avoid missing operations that may have arrived during the backup. Do not include any sequences prior to the full backup, as they will just throw errors and be redundant.

Note: It isn't necessary to do the full restore >=6.0. See,

Verified Against: Zimbra Collaboration 7.0 Date Created: 04/16/2014
Article ID: Date Modified: 2015-07-12

Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search