King0770-Notes-When innodb force recovery Fails

From Zimbra :: Wiki

Jump to: navigation, search
   Article-alert.png  - This article is a Work in Progress, and may be unfinished or missing sections.



Scenario: You just found out that mysql will not start up. In an attempt to do a mysqldump and reload, you edit the ~/my.cnf file to have the innodb_force_recovery setting. You tried values 1 through 6 for innodb_force_recovery, however, mysql will not start.

In this scenario, if mysql will not start, you will need to do a restore, and play the redo files to get the server back to it's current state.

1) zmcontrol stop

2) mv ~/db/data ~/db/data.OLD

3) ~/libexec/zmmyinit 
**Most likely your mysql passwords will change.
 See the following article to set them back, http://wiki.zimbra.com/index.php?title=Issues_with_mysql_and_logmysql_passwords#For_Mysql_Database**

4) ldap start

5) zmconvertctl start

6) zmrestoreoffline -a all --excludeSearchIndex --excludeBlobs --excludeHsmBlobs --ignoreRedoErrors -c
Or
6) zmrestoreoffline -a all --excludeSearchIndex --excludeBlobs --excludeHsmBlobs --ignoreRedoErrors -c -rf -lb <latest_full_backup_label>
** If restoring the full backup using "-rf" then you will need to run the redolog files manually as mentioned below.

While the restore is working, you will need to figure out which redo files to play. The best method is to open the session.xml file from the latest full backup and check the beginning and ending sequence numbers included in the backup.


Example:

minRedoSeq="638" maxRedoSeq="638"

**Note** minRedoSeq="638" maxRedoSeq="638"

In our example lets look at the redo files from the incremental backups.

-rw-r----- 1 zimbra zimbra   5079776 Sep  7 21:00 incr-20090908.010012.512/redologs/redo-20090908.010012.511-seq622.log
-rw-r----- 1 zimbra zimbra 117327479 Sep  8 21:00 incr-20090909.010013.946/redologs/redo-20090908.144204.162-seq623.log
-rw-r----- 1 zimbra zimbra 111285083 Sep  8 21:00 incr-20090909.010013.946/redologs/redo-20090908.144303.388-seq624.log
-rw-r----- 1 zimbra zimbra 105475888 Sep  8 21:00 incr-20090909.010013.946/redologs/redo-20090908.184251.115-seq625.log
-rw-r----- 1 zimbra zimbra  17637693 Sep  8 21:00 incr-20090909.010013.946/redologs/redo-20090909.010013.944-seq626.log
-rw-r----- 1 zimbra zimbra 118385579 Sep  9 21:00 incr-20090910.010014.770/redologs/redo-20090909.143443.347-seq627.log
-rw-r----- 1 zimbra zimbra 146039792 Sep  9 21:00 incr-20090910.010014.770/redologs/redo-20090909.143544.051-seq628.log
-rw-r----- 1 zimbra zimbra 219058845 Sep  9 21:01 incr-20090910.010014.770/redologs/redo-20090909.143623.940-seq629.log
-rw-r----- 1 zimbra zimbra 146032102 Sep  9 21:01 incr-20090910.010014.770/redologs/redo-20090909.143637.179-seq630.log
-rw-r----- 1 zimbra zimbra 146032140 Sep  9 21:01 incr-20090910.010014.770/redologs/redo-20090909.143637.223-seq631.log
-rw-r----- 1 zimbra zimbra 112605113 Sep  9 21:02 incr-20090910.010014.770/redologs/redo-20090909.144136.380-seq632.log
-rw-r----- 1 zimbra zimbra 104924041 Sep  9 21:02 incr-20090910.010014.770/redologs/redo-20090909.155319.532-seq633.log
-rw-r----- 1 zimbra zimbra  52520210 Sep  9 21:02 incr-20090910.010014.770/redologs/redo-20090910.010014.766-seq634.log
-rw-r----- 1 zimbra zimbra 103305944 Sep 10 21:00 incr-20090911.010012.175/redologs/redo-20090911.010012.173-seq635.log
-rw-r----- 1 zimbra zimbra 125025922 Sep 12 21:00 incr-20090913.010016.581/redologs/redo-20090911.194711.592-seq636.log
-rw-r----- 1 zimbra zimbra 134086559 Sep 12 21:01 incr-20090913.010016.581/redologs/redo-20090911.211919.123-seq637.log
-rw-r----- 1 zimbra zimbra 103718532 Sep 12 21:01 incr-20090913.010016.581/redologs/redo-20090913.010016.580-seq638.log
-rw-r----- 1 zimbra zimbra   4899300 Sep 13 21:00 incr-20090914.010015.716/redologs/redo-20090914.010015.715-seq639.log
-rw-r----- 1 zimbra zimbra 115999506 Sep 14 21:00 incr-20090915.010013.036/redologs/redo-20090914.145143.501-seq640.log
-rw-r----- 1 zimbra zimbra  92986896 Sep 14 21:00 incr-20090915.010013.036/redologs/redo-20090915.010012.700-seq641.log


Basically, you will only need to run the redo files from the latest full backup references, in our example, minRedoSeq="638" maxRedoSeq="638", and play the redo logs that are sequentially higher.

In our example we would run:

zmplayredo --logfiles redo-20090913.010016.580-seq638.log

zmplayredo --logfiles redo-20090914.010015.715-seq639.log

zmplayredo --logfiles redo-20090914.145143.501-seq640.log

zmplayredo --logfiles redo-20090915.010012.700-seq641.log

Also, don't forget to play the redo log in /opt/zimbra/redolog/archive/

-rw-r----- 1 zimbra zimbra 11607806 Sep 15 13:06 redo-20090915.170630.142-seq642.log


Again, note the sequence number.

Lastly, play the redo log from /opt/zimbra/redolog/

-rw-r----- 1 zimbra zimbra 6340 Sep 15 13:08 redo.log


It's possible this will cause some errors and some operations to be done twice, but we don't think there's any other way to avoid missing operations that may have arrived during the backup. Do not include any sequences prior to the full backup, as they will just throw errors and be redundant.

Note: It isn't necessary to do the full restore >=6.0. See, http://bugzilla.zimbra.com/show_bug.cgi?id=35278

Personal tools