https://wiki.zimbra.com/api.php?action=feedcontributions&user=Harley77&feedformat=atomZimbra :: Tech Center - User contributions [en]2024-03-29T07:56:23ZUser contributionsMediaWiki 1.39.0https://wiki.zimbra.com/index.php?title=Harley77-Notes&diff=12767Harley77-Notes2009-02-25T15:27:51Z<p>Harley77: /* Making this a central page to list my other wiki entries. */</p>
<hr />
<div>== Making this a central page to list my other wiki entries. ==<br />
<br />
<br />
<br />
'''Standby Backup Machine''' [http://wiki.zimbra.com/index.php?title=Harley77-Backup]<br />
<br />
'''Shell Scripts for Backup''' [http://wiki.zimbra.com/index.php?title=Harley77-Scripts]<br />
<br />
'''Mysql corruption, high cpu usage, and how I fixed it.''' [http://wiki.zimbra.com/index.php?title=Harley77-Mysqld]</div>Harley77https://wiki.zimbra.com/index.php?title=Harley77-Notes&diff=12766Harley77-Notes2009-02-25T15:27:08Z<p>Harley77: /* Making this a central page to list my other wiki entries. */</p>
<hr />
<div>== Making this a central page to list my other wiki entries. ==<br />
<br />
<br />
<br />
'''Standby Backup Machine''' [http://wiki.zimbra.com/index.php?title=Harley77-Backup]<br />
<br />
'''Shell Scripts for Backup''' [http://wiki.zimbra.com/index.php?title=Harley77-Scripts]<br />
<br />
'''Mysql corruption, high cpu usage, and how I fixed it.''' [http://wiki.zimbra.com/index.php?title=Harley77-Mysqld]<br />
<br />
<br />
[Harley77-Backup]</div>Harley77https://wiki.zimbra.com/index.php?title=Harley77-Notes&diff=12765Harley77-Notes2009-02-25T15:26:30Z<p>Harley77: /* Making this a central page to list my other wiki entries. */</p>
<hr />
<div>== Making this a central page to list my other wiki entries. ==<br />
<br />
<br />
<br />
'''Standby Backup Machine''' [http://wiki.zimbra.com/index.php?title=Harley77-Backup]<br />
<br />
'''Shell Scripts for Backup''' [http://wiki.zimbra.com/index.php?title=Harley77-Scripts]<br />
<br />
'''Mysql corruption, high cpu usage, and how I fixed it.''' [http://wiki.zimbra.com/index.php?title=Harley77-Mysqld]<br />
<br />
<br />
Harley77-Backup</div>Harley77https://wiki.zimbra.com/index.php?title=Harley77-Mysqld&diff=12713Harley77-Mysqld2009-02-18T19:29:45Z<p>Harley77: /* The solution (script style) */</p>
<hr />
<div>== UPDATE ==<br />
I've made a script that automates the process below. See the end of this page. <br />
<br />
<br />
<br />
== The Issue ==<br />
Hi. Has this ever happened to you?<br />
<br />
InnoDB: Error: page 0 log sequence number 0 933369315<br />
InnoDB: is in the future! Current system log sequence number 0 43685.<br />
InnoDB: Your database may be corrupt or you may have copied the InnoDB<br />
InnoDB: tablespace but not the InnoDB log files. See<br />
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html<br />
InnoDB: for more information.<br />
<br />
<br />
<br />
Is The mysqld process eating away every ounce of CPU time like it was free lobster?<br />
<br />
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br />
7590 zimbra 16 0 484m 402m 5036 S 98 11.3 2151:46 mysqld<br />
13033 zimbra 19 0 1770m 1.1g 46m S 13 32.8 47:28.14 java<br />
2148 root 15 0 2196 1132 856 R 2 0.0 0:00.03 top<br />
1 root 16 0 1564 528 460 S 0 0.0 0:01.80 init<br />
2 root RT 0 0 0 0 S 0 0.0 0:00.87 migration/0<br />
3 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0<br />
4 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0<br />
5 root RT 0 0 0 0 S 0 0.0 0:03.04 migration/1<br />
6 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1<br />
<br />
<br />
Well, I may have an answer. At least an answer that worked for me. <br />
<br />
<br />
Talking completely out of my butt, it has to do with ibdata1, at least, it did in my case. It happened after a server crash, syncing with a backup server and re-syncing back. Somewhere, it became "out of sorts". <br />
<br />
Now, After a nice email exchange with Zimbra tech support (a certian Chandrashekhar Bhosle was my helper) who pointed me in the right direction I finally achieved a solid procedure for curing this ailment. Using the backup server to do my testing, I could resync with the live server over and over so I could revert it back to it's mysql hogging state and retry various procedures until I nailed one down that worked every time. The following is the exact procedure, I recommend STRONGLY that you backup, backup, BACKUP before attempting any of the following. <br />
<br />
Also, My machine specs:<br />
OS: Ubuntu 6.02LTS 32-bit<br />
Zimbra: 5.0.13 Network Edition. <br />
<br />
== The solution (manual) ==<br />
<br />
1. Login as root then su Zimbra<br />
su zimbra<br />
<br />
2. Stop Zimbra<br />
zmcontrol stop<br />
<br />
3. Go back to root<br />
exit<br />
<br />
4. Edit your my.cnf file. <br />
pico /opt/zimbra/conf/my.cnf <br />
<br />
5. Add the follwing below the [mysql] section.<br />
innodb_force_recovery = 3<br />
<br />
6. Save and exit<br />
<br />
7. Login as zimbra<br />
su zimbra<br />
<br />
8. Start mysql server<br />
mysql.server start<br />
<br />
9. Dump out a list of databases to a temporary list. <br />
mysql -NB -e "show databases" | grep mbox > /tmp/mysql.db.list<br />
<br />
<br />
10. Make a temp directory to put stuff in. <br />
mkdir /tmp/mysql.sql <br />
<br />
<br />
11. Dump your zimbra database to a file. <br />
/opt/zimbra/mysql/bin/mysqldump zimbra -S /opt/zimbra/db/mysql.sock -u root / <br />
--password=''ROOT_SQL_PASSWORD'' > /tmp/mysql.sql/zimbra.sql <br />
<br />
12. Dump your mailbox databases to files. <br />
for db in `< /tmp/mysql.db.list`;<br />
do<br />
/opt/zimbra/mysql/bin/mysqldump $db -S /opt/zimbra/db/mysql.sock -u root /<br />
--password=ROOT_SQL_PASSWORD > /tmp/mysql.sql/$db.sql<br />
echo -e "Dumped $db\n"<br />
done<br />
<br />
13.Drop your mailbox databases <br />
for db in `< /tmp/mysql.db.list`<br />
do<br />
mysql -u root --password=''ROOT SQL PASSWORD'' -e "drop database $db"<br />
echo -e "Dropped $db"<br />
done<br />
<br />
14. Drop your zimbra database<br />
mysql -e "drop database zimbra" <br />
<br />
15. Stop mysql server<br />
mysql.server stop<br />
<br />
16. exit to root <br />
exit<br />
<br />
17. Edit my.conf, remove the innodb_force_recovert entry.<br />
pico /opt/zimbra/conf/my.cnf <br />
<br />
18. Save and exit.<br />
<br />
19. Remove the ibdata, ib_logfile0 and ib_logfile1<br />
rm /opt/zimbra/db/data/ibdata1<br />
rm /opt/zimbra/db/data/ib_*<br />
<br />
20. Login as zimbra<br />
su zimbra<br />
<br />
21. Start mysql server<br />
mysql.server start<br />
<br />
22. Create zimbra database<br />
mysql -e "create database zimbra character set utf8" <br />
<br />
23. Create Mailbox databases<br />
for db in `< /tmp/mysql.db.list`<br />
do<br />
mysql -e "create database $db character set utf8"<br />
echo -e "Created $db \n"<br />
done<br />
<br />
24. Import zimbra data<br />
mysql zimbra < /tmp/mysql.sql/zimbra.sql<br />
<br />
25. Import mailbox data <br />
for sql in /tmp/mysql.sql/mbox*<br />
do<br />
mysql `basename $sql .sql` < $sql<br />
echo -e "Updated `basename $sql .sql` \n"<br />
done<br />
<br />
26. Stop mysql server<br />
mysql.server stop<br />
<br />
27. Start zimbra<br />
zmcontrol start<br />
<br />
<br />
28. Check your mysql_error.log file for errors, make sure you can login, etc. etc. <br />
<br />
29. If its good, do the happy dance. If not, restore from backup and try something else.<br />
<br />
<br />
<br />
== The solution (script style) ==<br />
<br />
WARNING!!!!! There is virtually NO error checking in the script as it sits right now. So, if something screws up, it will happily keep right on going, potentially killing your database in the process. I'm just putting this here because it works great on my setup and I'm using it as a part of an hourly syncing setup for our cold standby server. <br />
<br />
You've been warned. <br />
<br />
<br />
#!/bin/bash<br />
# Dump and restore. Use with Rsynced Zimbra to resolve the out-of-sequence issues.<br />
# This should be run as root.<br />
<br />
# Remove the # below for debugging<br />
# set -x<br />
<br />
# Put your mySQL ROOT password below.<br />
<br />
SQLPASSWD="PUTPASSWORDHERE"<br />
<br />
clear<br />
<br />
echo "<br />
**** Starting mysql.server ****"<br />
su - zimbra -c mysql.server start<br />
<br />
echo ".......Waiting a bit for mysql to fully start."<br />
sleep 20<br />
<br />
<br />
if [ -f /tmp/mysql.db.list ]; then<br />
rm -Rf /tmp/mysql.db.list<br />
else<br />
echo " "<br />
fi<br />
<br />
echo "<br />
**** Creating database list ****"<br />
su - zimbra -c "/opt/zimbra/bin/mysql -NB -e 'show databases' | grep mbox > /tmp/mysql.db.list"<br />
<br />
<br />
if [ -d /tmp/mysql.sql ]; then<br />
echo " "<br />
else<br />
su - zimbra -c 'mkdir /tmp/mysql.sql'<br />
fi<br />
<br />
echo "<br />
**** Dumping Zimbra database ****"<br />
su - zimbra -c "/opt/zimbra/mysql/bin/mysqldump zimbra -S /opt/zimbra/db/mysql.sock -u root --password=$SQLPASSWD > /tmp/mysql.sql/zimbra.sql"<br />
<br />
<br />
echo "<br />
**** Dumping mailboxes ****"<br />
for db in `< /tmp/mysql.db.list`;<br />
do<br />
su - zimbra -c "/opt/zimbra/mysql/bin/mysqldump $db -S /opt/zimbra/db/mysql.sock -u root --password=$SQLPASSWD > /tmp/mysql.sql/$db.sql"<br />
echo -n "#"<br />
done<br />
<br />
echo -e " "<br />
echo "<br />
**** Dropping mailboxes ****"<br />
for db in `< /tmp/mysql.db.list`;<br />
do<br />
/opt/zimbra/bin/mysql -e "drop database $db"<br />
echo -n "#"<br />
done<br />
<br />
echo -e " "<br />
echo "<br />
**** Dropping zimbra database ****"<br />
/opt/zimbra/bin/mysql -e 'drop database zimbra'<br />
<br />
<br />
echo "<br />
**** Stopping Sql Server ****"<br />
su - zimbra "mysql.server stop"<br />
<br />
echo "<br />
**** Deleting ibdata1 and ib_log files ****"<br />
rm /opt/zimbra/db/data/ibdata1<br />
echo -n "Pew! "<br />
rm /opt/zimbra/db/data/ib_*<br />
echo -n "Pew! Pew!"<br />
echo "<br />
Files deleted"<br />
<br />
<br />
echo "<br />
**** Starting mysql.server ****"<br />
su - zimbra 'mysql.server start'<br />
<br />
echo "....... Waiting for mysql to fully start."<br />
sleep 20<br />
<br />
echo "<br />
**** Creating Zimbra database ****"<br />
/opt/zimbra/bin/mysql -e "create database zimbra character set utf8"<br />
<br />
<br />
echo "<br />
**** Creating Mailbox Databases ****"<br />
for db in `< /tmp/mysql.db.list`;<br />
do<br />
/opt/zimbra/bin/mysql -e "create database $db character set utf8"<br />
echo -n "#"<br />
done<br />
<br />
<br />
echo "<br />
**** Importing Zimbra Database ****"<br />
/opt/zimbra/bin/mysql zimbra < /tmp/mysql.sql/zimbra.sql<br />
<br />
<br />
<br />
echo "<br />
**** Importing Mailbox Databases ****"<br />
<br />
for sql in /tmp/mysql.sql/mbox*<br />
do<br />
/opt/zimbra/bin/mysql `basename $sql .sql` < $sql<br />
echo -n "#"<br />
done<br />
<br />
echo -e " "<br />
echo "<br />
**** Stopping mysql.server ****"<br />
su - zimbra 'mysql.server stop'<br />
<br />
echo "********************* All Finished! ***********************"</div>Harley77https://wiki.zimbra.com/index.php?title=Harley77-Mysqld&diff=12712Harley77-Mysqld2009-02-18T19:27:19Z<p>Harley77: /* UPDATE */</p>
<hr />
<div>== UPDATE ==<br />
I've made a script that automates the process below. See the end of this page. <br />
<br />
<br />
<br />
== The Issue ==<br />
Hi. Has this ever happened to you?<br />
<br />
InnoDB: Error: page 0 log sequence number 0 933369315<br />
InnoDB: is in the future! Current system log sequence number 0 43685.<br />
InnoDB: Your database may be corrupt or you may have copied the InnoDB<br />
InnoDB: tablespace but not the InnoDB log files. See<br />
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html<br />
InnoDB: for more information.<br />
<br />
<br />
<br />
Is The mysqld process eating away every ounce of CPU time like it was free lobster?<br />
<br />
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br />
7590 zimbra 16 0 484m 402m 5036 S 98 11.3 2151:46 mysqld<br />
13033 zimbra 19 0 1770m 1.1g 46m S 13 32.8 47:28.14 java<br />
2148 root 15 0 2196 1132 856 R 2 0.0 0:00.03 top<br />
1 root 16 0 1564 528 460 S 0 0.0 0:01.80 init<br />
2 root RT 0 0 0 0 S 0 0.0 0:00.87 migration/0<br />
3 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0<br />
4 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0<br />
5 root RT 0 0 0 0 S 0 0.0 0:03.04 migration/1<br />
6 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1<br />
<br />
<br />
Well, I may have an answer. At least an answer that worked for me. <br />
<br />
<br />
Talking completely out of my butt, it has to do with ibdata1, at least, it did in my case. It happened after a server crash, syncing with a backup server and re-syncing back. Somewhere, it became "out of sorts". <br />
<br />
Now, After a nice email exchange with Zimbra tech support (a certian Chandrashekhar Bhosle was my helper) who pointed me in the right direction I finally achieved a solid procedure for curing this ailment. Using the backup server to do my testing, I could resync with the live server over and over so I could revert it back to it's mysql hogging state and retry various procedures until I nailed one down that worked every time. The following is the exact procedure, I recommend STRONGLY that you backup, backup, BACKUP before attempting any of the following. <br />
<br />
Also, My machine specs:<br />
OS: Ubuntu 6.02LTS 32-bit<br />
Zimbra: 5.0.13 Network Edition. <br />
<br />
== The solution (manual) ==<br />
<br />
1. Login as root then su Zimbra<br />
su zimbra<br />
<br />
2. Stop Zimbra<br />
zmcontrol stop<br />
<br />
3. Go back to root<br />
exit<br />
<br />
4. Edit your my.cnf file. <br />
pico /opt/zimbra/conf/my.cnf <br />
<br />
5. Add the follwing below the [mysql] section.<br />
innodb_force_recovery = 3<br />
<br />
6. Save and exit<br />
<br />
7. Login as zimbra<br />
su zimbra<br />
<br />
8. Start mysql server<br />
mysql.server start<br />
<br />
9. Dump out a list of databases to a temporary list. <br />
mysql -NB -e "show databases" | grep mbox > /tmp/mysql.db.list<br />
<br />
<br />
10. Make a temp directory to put stuff in. <br />
mkdir /tmp/mysql.sql <br />
<br />
<br />
11. Dump your zimbra database to a file. <br />
/opt/zimbra/mysql/bin/mysqldump zimbra -S /opt/zimbra/db/mysql.sock -u root / <br />
--password=''ROOT_SQL_PASSWORD'' > /tmp/mysql.sql/zimbra.sql <br />
<br />
12. Dump your mailbox databases to files. <br />
for db in `< /tmp/mysql.db.list`;<br />
do<br />
/opt/zimbra/mysql/bin/mysqldump $db -S /opt/zimbra/db/mysql.sock -u root /<br />
--password=ROOT_SQL_PASSWORD > /tmp/mysql.sql/$db.sql<br />
echo -e "Dumped $db\n"<br />
done<br />
<br />
13.Drop your mailbox databases <br />
for db in `< /tmp/mysql.db.list`<br />
do<br />
mysql -u root --password=''ROOT SQL PASSWORD'' -e "drop database $db"<br />
echo -e "Dropped $db"<br />
done<br />
<br />
14. Drop your zimbra database<br />
mysql -e "drop database zimbra" <br />
<br />
15. Stop mysql server<br />
mysql.server stop<br />
<br />
16. exit to root <br />
exit<br />
<br />
17. Edit my.conf, remove the innodb_force_recovert entry.<br />
pico /opt/zimbra/conf/my.cnf <br />
<br />
18. Save and exit.<br />
<br />
19. Remove the ibdata, ib_logfile0 and ib_logfile1<br />
rm /opt/zimbra/db/data/ibdata1<br />
rm /opt/zimbra/db/data/ib_*<br />
<br />
20. Login as zimbra<br />
su zimbra<br />
<br />
21. Start mysql server<br />
mysql.server start<br />
<br />
22. Create zimbra database<br />
mysql -e "create database zimbra character set utf8" <br />
<br />
23. Create Mailbox databases<br />
for db in `< /tmp/mysql.db.list`<br />
do<br />
mysql -e "create database $db character set utf8"<br />
echo -e "Created $db \n"<br />
done<br />
<br />
24. Import zimbra data<br />
mysql zimbra < /tmp/mysql.sql/zimbra.sql<br />
<br />
25. Import mailbox data <br />
for sql in /tmp/mysql.sql/mbox*<br />
do<br />
mysql `basename $sql .sql` < $sql<br />
echo -e "Updated `basename $sql .sql` \n"<br />
done<br />
<br />
26. Stop mysql server<br />
mysql.server stop<br />
<br />
27. Start zimbra<br />
zmcontrol start<br />
<br />
<br />
28. Check your mysql_error.log file for errors, make sure you can login, etc. etc. <br />
<br />
29. If its good, do the happy dance. If not, restore from backup and try something else.<br />
<br />
<br />
<br />
== The solution (script style) ==<br />
<br />
#!/bin/bash<br />
# Dump and restore. Use with Rsynced Zimbra to resolve the out-of-sequence issues.<br />
# This should be run as root.<br />
<br />
# Remove the # below for debugging<br />
# set -x<br />
<br />
# Put your mySQL ROOT password below.<br />
<br />
SQLPASSWD="PUTPASSWORDHERE"<br />
<br />
clear<br />
<br />
echo "<br />
**** Starting mysql.server ****"<br />
su - zimbra -c mysql.server start<br />
<br />
echo ".......Waiting a bit for mysql to fully start."<br />
sleep 20<br />
<br />
<br />
if [ -f /tmp/mysql.db.list ]; then<br />
rm -Rf /tmp/mysql.db.list<br />
else<br />
echo " "<br />
fi<br />
<br />
echo "<br />
**** Creating database list ****"<br />
su - zimbra -c "/opt/zimbra/bin/mysql -NB -e 'show databases' | grep mbox > /tmp/mysql.db.list"<br />
<br />
<br />
if [ -d /tmp/mysql.sql ]; then<br />
echo " "<br />
else<br />
su - zimbra -c 'mkdir /tmp/mysql.sql'<br />
fi<br />
<br />
echo "<br />
**** Dumping Zimbra database ****"<br />
su - zimbra -c "/opt/zimbra/mysql/bin/mysqldump zimbra -S /opt/zimbra/db/mysql.sock -u root --password=$SQLPASSWD > /tmp/mysql.sql/zimbra.sql"<br />
<br />
<br />
echo "<br />
**** Dumping mailboxes ****"<br />
for db in `< /tmp/mysql.db.list`;<br />
do<br />
su - zimbra -c "/opt/zimbra/mysql/bin/mysqldump $db -S /opt/zimbra/db/mysql.sock -u root --password=$SQLPASSWD > /tmp/mysql.sql/$db.sql"<br />
echo -n "#"<br />
done<br />
<br />
echo -e " "<br />
echo "<br />
**** Dropping mailboxes ****"<br />
for db in `< /tmp/mysql.db.list`;<br />
do<br />
/opt/zimbra/bin/mysql -e "drop database $db"<br />
echo -n "#"<br />
done<br />
<br />
echo -e " "<br />
echo "<br />
**** Dropping zimbra database ****"<br />
/opt/zimbra/bin/mysql -e 'drop database zimbra'<br />
<br />
<br />
echo "<br />
**** Stopping Sql Server ****"<br />
su - zimbra "mysql.server stop"<br />
<br />
echo "<br />
**** Deleting ibdata1 and ib_log files ****"<br />
rm /opt/zimbra/db/data/ibdata1<br />
echo -n "Pew! "<br />
rm /opt/zimbra/db/data/ib_*<br />
echo -n "Pew! Pew!"<br />
echo "<br />
Files deleted"<br />
<br />
<br />
echo "<br />
**** Starting mysql.server ****"<br />
su - zimbra 'mysql.server start'<br />
<br />
echo "....... Waiting for mysql to fully start."<br />
sleep 20<br />
<br />
echo "<br />
**** Creating Zimbra database ****"<br />
/opt/zimbra/bin/mysql -e "create database zimbra character set utf8"<br />
<br />
<br />
echo "<br />
**** Creating Mailbox Databases ****"<br />
for db in `< /tmp/mysql.db.list`;<br />
do<br />
/opt/zimbra/bin/mysql -e "create database $db character set utf8"<br />
echo -n "#"<br />
done<br />
<br />
<br />
echo "<br />
**** Importing Zimbra Database ****"<br />
/opt/zimbra/bin/mysql zimbra < /tmp/mysql.sql/zimbra.sql<br />
<br />
<br />
<br />
echo "<br />
**** Importing Mailbox Databases ****"<br />
<br />
for sql in /tmp/mysql.sql/mbox*<br />
do<br />
/opt/zimbra/bin/mysql `basename $sql .sql` < $sql<br />
echo -n "#"<br />
done<br />
<br />
echo -e " "<br />
echo "<br />
**** Stopping mysql.server ****"<br />
su - zimbra 'mysql.server stop'<br />
<br />
echo "********************* All Finished! ***********************"</div>Harley77https://wiki.zimbra.com/index.php?title=Harley77-Mysqld&diff=12711Harley77-Mysqld2009-02-18T19:11:29Z<p>Harley77: </p>
<hr />
<div><br />
== UPDATE ==<br />
I've made a script that automates the process below. See the end of this page. <br />
<br />
<br />
Hi. Has this ever happened to you?<br />
<br />
InnoDB: Error: page 0 log sequence number 0 933369315<br />
InnoDB: is in the future! Current system log sequence number 0 43685.<br />
InnoDB: Your database may be corrupt or you may have copied the InnoDB<br />
InnoDB: tablespace but not the InnoDB log files. See<br />
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html<br />
InnoDB: for more information.<br />
<br />
<br />
<br />
Is The mysqld process eating away every ounce of CPU time like it was free lobster?<br />
<br />
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br />
7590 zimbra 16 0 484m 402m 5036 S 98 11.3 2151:46 mysqld<br />
13033 zimbra 19 0 1770m 1.1g 46m S 13 32.8 47:28.14 java<br />
2148 root 15 0 2196 1132 856 R 2 0.0 0:00.03 top<br />
1 root 16 0 1564 528 460 S 0 0.0 0:01.80 init<br />
2 root RT 0 0 0 0 S 0 0.0 0:00.87 migration/0<br />
3 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0<br />
4 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0<br />
5 root RT 0 0 0 0 S 0 0.0 0:03.04 migration/1<br />
6 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1<br />
<br />
<br />
Well, I may have an answer. At least an answer that worked for me. <br />
<br />
<br />
Talking completely out of my butt, it has to do with ibdata1, at least, it did in my case. It happened after a server crash, syncing with a backup server and re-syncing back. Somewhere, it became "out of sorts". <br />
<br />
Now, After a nice email exchange with Zimbra tech support (a certian Chandrashekhar Bhosle was my helper) who pointed me in the right direction I finally achieved a solid procedure for curing this ailment. Using the backup server to do my testing, I could resync with the live server over and over so I could revert it back to it's mysql hogging state and retry various procedures until I nailed one down that worked every time. The following is the exact procedure, I recommend STRONGLY that you backup, backup, BACKUP before attempting any of the following. <br />
<br />
Also, My machine specs:<br />
OS: Ubuntu 6.02LTS 32-bit<br />
Zimbra: 5.0.13 Network Edition. <br />
<br />
<br />
1. Login as root then su Zimbra<br />
su zimbra<br />
<br />
2. Stop Zimbra<br />
zmcontrol stop<br />
<br />
3. Go back to root<br />
exit<br />
<br />
4. Edit your my.cnf file. <br />
pico /opt/zimbra/conf/my.cnf <br />
<br />
5. Add the follwing below the [mysql] section.<br />
innodb_force_recovery = 3<br />
<br />
6. Save and exit<br />
<br />
7. Login as zimbra<br />
su zimbra<br />
<br />
8. Start mysql server<br />
mysql.server start<br />
<br />
9. Dump out a list of databases to a temporary list. <br />
mysql -NB -e "show databases" | grep mbox > /tmp/mysql.db.list<br />
<br />
<br />
10. Make a temp directory to put stuff in. <br />
mkdir /tmp/mysql.sql <br />
<br />
<br />
11. Dump your zimbra database to a file. <br />
/opt/zimbra/mysql/bin/mysqldump zimbra -S /opt/zimbra/db/mysql.sock -u root / <br />
--password=''ROOT_SQL_PASSWORD'' > /tmp/mysql.sql/zimbra.sql <br />
<br />
12. Dump your mailbox databases to files. <br />
for db in `< /tmp/mysql.db.list`;<br />
do<br />
/opt/zimbra/mysql/bin/mysqldump $db -S /opt/zimbra/db/mysql.sock -u root /<br />
--password=ROOT_SQL_PASSWORD > /tmp/mysql.sql/$db.sql<br />
echo -e "Dumped $db\n"<br />
done<br />
<br />
13.Drop your mailbox databases <br />
for db in `< /tmp/mysql.db.list`<br />
do<br />
mysql -u root --password=''ROOT SQL PASSWORD'' -e "drop database $db"<br />
echo -e "Dropped $db"<br />
done<br />
<br />
14. Drop your zimbra database<br />
mysql -e "drop database zimbra" <br />
<br />
15. Stop mysql server<br />
mysql.server stop<br />
<br />
16. exit to root <br />
exit<br />
<br />
17. Edit my.conf, remove the innodb_force_recovert entry.<br />
pico /opt/zimbra/conf/my.cnf <br />
<br />
18. Save and exit.<br />
<br />
19. Remove the ibdata, ib_logfile0 and ib_logfile1<br />
rm /opt/zimbra/db/data/ibdata1<br />
rm /opt/zimbra/db/data/ib_*<br />
<br />
20. Login as zimbra<br />
su zimbra<br />
<br />
21. Start mysql server<br />
mysql.server start<br />
<br />
22. Create zimbra database<br />
mysql -e "create database zimbra character set utf8" <br />
<br />
23. Create Mailbox databases<br />
for db in `< /tmp/mysql.db.list`<br />
do<br />
mysql -e "create database $db character set utf8"<br />
echo -e "Created $db \n"<br />
done<br />
<br />
24. Import zimbra data<br />
mysql zimbra < /tmp/mysql.sql/zimbra.sql<br />
<br />
25. Import mailbox data <br />
for sql in /tmp/mysql.sql/mbox*<br />
do<br />
mysql `basename $sql .sql` < $sql<br />
echo -e "Updated `basename $sql .sql` \n"<br />
done<br />
<br />
26. Stop mysql server<br />
mysql.server stop<br />
<br />
27. Start zimbra<br />
zmcontrol start<br />
<br />
<br />
28. Check your mysql_error.log file for errors, make sure you can login, etc. etc. <br />
<br />
29. If its good, do the happy dance. If not, restore from backup and try something else.</div>Harley77https://wiki.zimbra.com/index.php?title=Harley77-Notes&diff=12703Harley77-Notes2009-02-13T20:43:57Z<p>Harley77: /* Making this a central page to list my other wiki entries. */</p>
<hr />
<div>== Making this a central page to list my other wiki entries. ==<br />
<br />
<br />
<br />
'''Standby Backup Machine''' [http://wiki.zimbra.com/index.php?title=Harley77-Backup]<br />
<br />
'''Shell Scripts for Backup''' [http://wiki.zimbra.com/index.php?title=Harley77-Scripts]<br />
<br />
'''Mysql corruption, high cpu usage, and how I fixed it.''' [http://wiki.zimbra.com/index.php?title=Harley77-Mysqld]</div>Harley77https://wiki.zimbra.com/index.php?title=Harley77-Mysqld&diff=12702Harley77-Mysqld2009-02-13T20:26:01Z<p>Harley77: </p>
<hr />
<div>Hi. Has this ever happened to you?<br />
<br />
InnoDB: Error: page 0 log sequence number 0 933369315<br />
InnoDB: is in the future! Current system log sequence number 0 43685.<br />
InnoDB: Your database may be corrupt or you may have copied the InnoDB<br />
InnoDB: tablespace but not the InnoDB log files. See<br />
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html<br />
InnoDB: for more information.<br />
<br />
<br />
<br />
Is The mysqld process eating away every ounce of CPU time like it was free lobster?<br />
<br />
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br />
7590 zimbra 16 0 484m 402m 5036 S 98 11.3 2151:46 mysqld<br />
13033 zimbra 19 0 1770m 1.1g 46m S 13 32.8 47:28.14 java<br />
2148 root 15 0 2196 1132 856 R 2 0.0 0:00.03 top<br />
1 root 16 0 1564 528 460 S 0 0.0 0:01.80 init<br />
2 root RT 0 0 0 0 S 0 0.0 0:00.87 migration/0<br />
3 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0<br />
4 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0<br />
5 root RT 0 0 0 0 S 0 0.0 0:03.04 migration/1<br />
6 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1<br />
<br />
<br />
Well, I may have an answer. At least an answer that worked for me. <br />
<br />
<br />
Talking completely out of my butt, it has to do with ibdata1, at least, it did in my case. It happened after a server crash, syncing with a backup server and re-syncing back. Somewhere, it became "out of sorts". <br />
<br />
Now, After a nice email exchange with Zimbra tech support (a certian Chandrashekhar Bhosle was my helper) who pointed me in the right direction I finally achieved a solid procedure for curing this ailment. Using the backup server to do my testing, I could resync with the live server over and over so I could revert it back to it's mysql hogging state and retry various procedures until I nailed one down that worked every time. The following is the exact procedure, I recommend STRONGLY that you backup, backup, BACKUP before attempting any of the following. <br />
<br />
Also, My machine specs:<br />
OS: Ubuntu 6.02LTS 32-bit<br />
Zimbra: 5.0.13 Network Edition. <br />
<br />
<br />
1. Login as root then su Zimbra<br />
su zimbra<br />
<br />
2. Stop Zimbra<br />
zmcontrol stop<br />
<br />
3. Go back to root<br />
exit<br />
<br />
4. Edit your my.cnf file. <br />
pico /opt/zimbra/conf/my.cnf <br />
<br />
5. Add the follwing below the [mysql] section.<br />
innodb_force_recovery = 3<br />
<br />
6. Save and exit<br />
<br />
7. Login as zimbra<br />
su zimbra<br />
<br />
8. Start mysql server<br />
mysql.server start<br />
<br />
9. Dump out a list of databases to a temporary list. <br />
mysql -NB -e "show databases" | grep mbox > /tmp/mysql.db.list<br />
<br />
<br />
10. Make a temp directory to put stuff in. <br />
mkdir /tmp/mysql.sql <br />
<br />
<br />
11. Dump your zimbra database to a file. <br />
/opt/zimbra/mysql/bin/mysqldump zimbra -S /opt/zimbra/db/mysql.sock -u root / <br />
--password=''ROOT_SQL_PASSWORD'' > /tmp/mysql.sql/zimbra.sql <br />
<br />
12. Dump your mailbox databases to files. <br />
for db in `< /tmp/mysql.db.list`;<br />
do<br />
/opt/zimbra/mysql/bin/mysqldump $db -S /opt/zimbra/db/mysql.sock -u root /<br />
--password=ROOT_SQL_PASSWORD > /tmp/mysql.sql/$db.sql<br />
echo -e "Dumped $db\n"<br />
done<br />
<br />
13.Drop your mailbox databases <br />
for db in `< /tmp/mysql.db.list`<br />
do<br />
mysql -u root --password=''ROOT SQL PASSWORD'' -e "drop database $db"<br />
echo -e "Dropped $db"<br />
done<br />
<br />
14. Drop your zimbra database<br />
mysql -e "drop database zimbra" <br />
<br />
15. Stop mysql server<br />
mysql.server stop<br />
<br />
16. exit to root <br />
exit<br />
<br />
17. Edit my.conf, remove the innodb_force_recovert entry.<br />
pico /opt/zimbra/conf/my.cnf <br />
<br />
18. Save and exit.<br />
<br />
19. Remove the ibdata, ib_logfile0 and ib_logfile1<br />
rm /opt/zimbra/db/data/ibdata1<br />
rm /opt/zimbra/db/data/ib_*<br />
<br />
20. Login as zimbra<br />
su zimbra<br />
<br />
21. Start mysql server<br />
mysql.server start<br />
<br />
22. Create zimbra database<br />
mysql -e "create database zimbra character set utf8" <br />
<br />
23. Create Mailbox databases<br />
for db in `< /tmp/mysql.db.list`<br />
do<br />
mysql -e "create database $db character set utf8"<br />
echo -e "Created $db \n"<br />
done<br />
<br />
24. Import zimbra data<br />
mysql zimbra < /tmp/mysql.sql/zimbra.sql<br />
<br />
25. Import mailbox data <br />
for sql in /tmp/mysql.sql/mbox*<br />
do<br />
mysql `basename $sql .sql` < $sql<br />
echo -e "Updated `basename $sql .sql` \n"<br />
done<br />
<br />
26. Stop mysql server<br />
mysql.server stop<br />
<br />
27. Start zimbra<br />
zmcontrol start<br />
<br />
<br />
28. Check your mysql_error.log file for errors, make sure you can login, etc. etc. <br />
<br />
29. If its good, do the happy dance. If not, restore from backup and try something else.</div>Harley77https://wiki.zimbra.com/index.php?title=Harley77-Mysqld&diff=12701Harley77-Mysqld2009-02-13T20:23:31Z<p>Harley77: </p>
<hr />
<div>Hi. Has this ever happened to you?<br />
<br />
InnoDB: Error: page 0 log sequence number 0 933369315<br />
InnoDB: is in the future! Current system log sequence number 0 43685.<br />
InnoDB: Your database may be corrupt or you may have copied the InnoDB<br />
InnoDB: tablespace but not the InnoDB log files. See<br />
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html<br />
InnoDB: for more information.<br />
<br />
<br />
<br />
Is The mysqld process eating away every ounce of CPU time like it was free lobster?<br />
<br />
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br />
7590 zimbra 16 0 484m 402m 5036 S 98 11.3 2151:46 mysqld<br />
13033 zimbra 19 0 1770m 1.1g 46m S 13 32.8 47:28.14 java<br />
2148 root 15 0 2196 1132 856 R 2 0.0 0:00.03 top<br />
1 root 16 0 1564 528 460 S 0 0.0 0:01.80 init<br />
2 root RT 0 0 0 0 S 0 0.0 0:00.87 migration/0<br />
3 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0<br />
4 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0<br />
5 root RT 0 0 0 0 S 0 0.0 0:03.04 migration/1<br />
6 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1<br />
<br />
<br />
Well, I may have an answer. At least an answer that worked for me. <br />
<br />
<br />
Talking completely out of my butt, it has to do with ibdata1, at least, it did in my case. It happened after a server crash, syncing with a backup server and re-syncing back. Somewhere, it became "out of sorts". <br />
<br />
Now, After a nice email exchange with Zimbra tech support (a certian Chandrashekhar Bhosle was my helper) who pointed me in the right direction I finally achieved a solid procedure for curing this ailment. Using the backup server to do my testing, I could resync with the live server over and over so I could revert it back to it's mysql hogging state and retry various procedures until I nailed one down that worked every time. The following is the exact procedure, I recommend STRONGLY that you backup, backup, BACKUP before attempting any of the following. <br />
<br />
<br />
1. Login as root then su Zimbra<br />
su zimbra<br />
<br />
2. Stop Zimbra<br />
zmcontrol stop<br />
<br />
3. Go back to root<br />
exit<br />
<br />
4. Edit your my.cnf file. <br />
pico /opt/zimbra/conf/my.cnf <br />
<br />
5. Add the follwing below the [mysql] section.<br />
innodb_force_recovery = 3<br />
<br />
6. Save and exit<br />
<br />
7. Login as zimbra<br />
su zimbra<br />
<br />
8. Start mysql server<br />
mysql.server start<br />
<br />
9. Dump out a list of databases to a temporary list. <br />
mysql -NB -e "show databases" | grep mbox > /tmp/mysql.db.list<br />
<br />
<br />
10. Make a temp directory to put stuff in. <br />
mkdir /tmp/mysql.sql <br />
<br />
<br />
11. Dump your zimbra database to a file. <br />
/opt/zimbra/mysql/bin/mysqldump zimbra -S /opt/zimbra/db/mysql.sock -u root / <br />
--password=''ROOT_SQL_PASSWORD'' > /tmp/mysql.sql/zimbra.sql <br />
<br />
12. Dump your mailbox databases to files. <br />
for db in `< /tmp/mysql.db.list`;<br />
do<br />
/opt/zimbra/mysql/bin/mysqldump $db -S /opt/zimbra/db/mysql.sock -u root /<br />
--password=ROOT_SQL_PASSWORD > /tmp/mysql.sql/$db.sql<br />
echo -e "Dumped $db\n"<br />
done<br />
<br />
13.Drop your mailbox databases <br />
for db in `< /tmp/mysql.db.list`<br />
do<br />
mysql -u root --password=''ROOT SQL PASSWORD'' -e "drop database $db"<br />
echo -e "Dropped $db"<br />
done<br />
<br />
14. Drop your zimbra database<br />
mysql -e "drop database zimbra" <br />
<br />
15. Stop mysql server<br />
mysql.server stop<br />
<br />
16. exit to root <br />
exit<br />
<br />
17. Edit my.conf, remove the innodb_force_recovert entry.<br />
pico /opt/zimbra/conf/my.cnf <br />
<br />
18. Save and exit.<br />
<br />
19. Remove the ibdata, ib_logfile0 and ib_logfile1<br />
rm /opt/zimbra/db/data/ibdata1<br />
rm /opt/zimbra/db/data/ib_*<br />
<br />
20. Login as zimbra<br />
su zimbra<br />
<br />
21. Start mysql server<br />
mysql.server start<br />
<br />
22. Create zimbra database<br />
mysql -e "create database zimbra character set utf8" <br />
<br />
23. Create Mailbox databases<br />
for db in `< /tmp/mysql.db.list`<br />
do<br />
mysql -e "create database $db character set utf8"<br />
echo -e "Created $db \n"<br />
done<br />
<br />
24. Import zimbra data<br />
mysql zimbra < /tmp/mysql.sql/zimbra.sql<br />
<br />
25. Import mailbox data <br />
for sql in /tmp/mysql.sql/mbox*<br />
do<br />
mysql `basename $sql .sql` < $sql<br />
echo -e "Updated `basename $sql .sql` \n"<br />
done<br />
<br />
26. Stop mysql server<br />
mysql.server stop<br />
<br />
27. Start zimbra<br />
zmcontrol start<br />
<br />
<br />
28. Check your mysql_error.log file for errors, make sure you can login, etc. etc. <br />
<br />
29. If its good, do the happy dance. If not, restore from backup and try something else.</div>Harley77https://wiki.zimbra.com/index.php?title=Harley77-Mysqld&diff=12700Harley77-Mysqld2009-02-13T20:22:00Z<p>Harley77: </p>
<hr />
<div>Hi. Has this ever happened to you?<br />
<br />
InnoDB: Error: page 0 log sequence number 0 933369315<br />
InnoDB: is in the future! Current system log sequence number 0 43685.<br />
InnoDB: Your database may be corrupt or you may have copied the InnoDB<br />
InnoDB: tablespace but not the InnoDB log files. See<br />
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html<br />
InnoDB: for more information.<br />
<br />
<br />
<br />
Is The mysqld process eating away every ounce of CPU time like it was free lobster?<br />
<br />
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br />
7590 zimbra 16 0 484m 402m 5036 S 98 11.3 2151:46 mysqld<br />
13033 zimbra 19 0 1770m 1.1g 46m S 13 32.8 47:28.14 java<br />
2148 root 15 0 2196 1132 856 R 2 0.0 0:00.03 top<br />
1 root 16 0 1564 528 460 S 0 0.0 0:01.80 init<br />
2 root RT 0 0 0 0 S 0 0.0 0:00.87 migration/0<br />
3 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0<br />
4 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0<br />
5 root RT 0 0 0 0 S 0 0.0 0:03.04 migration/1<br />
6 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1<br />
<br />
<br />
Well, I may have an answer. At least an answer that worked for me. <br />
<br />
<br />
Talking completely out of my butt, it has to do with ibdata1, at least, it did in my case. It happened after a server crash, syncing with a backup server and re-syncing back. Somewhere, it became "out of sorts". <br />
<br />
Now, After a nice email exchange with Zimbra tech support (a certian Chandrashekhar Bhosle was my helper) who pointed me in the right direction I finally achieved a solid procedure for curing this ailment. Using the backup server to do my testing, I could resync with the live server over and over so I could revert it back to it's mysql hogging state and retry various procedures until I nailed one down that worked every time. The following is the exact procedure, I recommend STRONGLY that you backup, backup, BACKUP before attempting any of the following. <br />
<br />
<br />
1. Login as root then su Zimbra<br />
su zimbra<br />
<br />
2. Stop Zimbra<br />
zmcontrol stop<br />
<br />
3. Go back to root<br />
exit<br />
<br />
4. Edit your my.cnf file. <br />
pico /opt/zimbra/conf/my.cnf <br />
<br />
5. Add the follwing below the [mysql] section.<br />
innodb_force_recovery = 3<br />
<br />
6. Save and exit<br />
<br />
7. Login as zimbra<br />
su zimbra<br />
<br />
8. Start mysql server<br />
mysql.server start<br />
<br />
9. Dump out a list of databases to a temporary list. <br />
mysql -NB -e "show databases" | grep mbox > /tmp/mysql.db.list<br />
<br />
<br />
10. Make a temp directory to put stuff in. <br />
mkdir /tmp/mysql.sql <br />
<br />
<br />
11. Dump your zimbra database to a file. <br />
/opt/zimbra/mysql/bin/mysqldump zimbra -S /opt/zimbra/db/mysql.sock -u root / <br />
--password=''ROOT_SQL_PASSWORD'' > /tmp/mysql.sql/zimbra.sql <br />
<br />
12. Dump your mailbox databases to files. <br />
for db in `< /tmp/mysql.db.list`;<br />
do<br />
/opt/zimbra/mysql/bin/mysqldump $db -S /opt/zimbra/db/mysql.sock -u root /<br />
--password=ROOT_SQL_PASSWORD > /tmp/mysql.sql/$db.sql<br />
echo -e "Dumped $db\n"<br />
done<br />
<br />
13.Drop your mailbox databases <br />
for db in `< /tmp/mysql.db.list`<br />
do<br />
mysql -u root --password=''ROOT SQL PASSWORD'' -e "drop database $db"<br />
echo -e "Dropped $db"<br />
done<br />
<br />
14. Drop your zimbra database<br />
mysql -e "drop database zimbra" <br />
<br />
15. Stop mysql server<br />
mysql.server stop<br />
<br />
16. exit to root <br />
exit<br />
<br />
17. Edit my.conf, remove the innodb_force_recovert entry.<br />
pico /opt/zimbra/conf/my.cnf <br />
<br />
18. Save and exit.<br />
<br />
19. Remove the ibdata, ib_logfile0 and ib_logfile1<br />
rm /opt/zimbra/db/data/ibdata1<br />
rm /opt/zimbra/db/data/ib_*<br />
<br />
20. Login as zimbra<br />
su zimbra<br />
<br />
21. Start mysql server<br />
mysql.server start<br />
<br />
22. Create zimbra database<br />
mysql -e "create database zimbra character set utf8" <br />
<br />
23. Create Mailbox databases<br />
for db in `< /tmp/mysql.db.list`<br />
do<br />
mysql -e "create database $db character set utf8"<br />
echo -e "Created $db \n"<br />
done<br />
<br />
24. Import zimbra data<br />
mysql zimbra < /tmp/mysql.sql/zimbra.sql<br />
<br />
25. Import mailbox data <br />
for sql in /tmp/mysql.sql/mbox*<br />
do<br />
mysql `basename $sql .sql` < $sql<br />
echo -e "Updated `basename $sql .sql` \n"<br />
done<br />
<br />
26. Stop mysql server<br />
mysql.server stop<br />
<br />
27. Start zimbra<br />
zmcontrol start<br />
<br />
<br />
28. Check your mysql_error.log file for errors, make sure you can login, etc. etc. <br />
<br />
29. If its good, do the happy dance. If not, restore from backup and try something else.</div>Harley77https://wiki.zimbra.com/index.php?title=Harley77-Mysqld&diff=12698Harley77-Mysqld2009-02-13T16:36:18Z<p>Harley77: New page: This will be a page to document my fix for a mysql issue I experienced. Coming soon.</p>
<hr />
<div>This will be a page to document my fix for a mysql issue I experienced. Coming soon.</div>Harley77https://wiki.zimbra.com/index.php?title=Harley77-Notes&diff=12697Harley77-Notes2009-02-13T16:35:35Z<p>Harley77: Replacing page with '
== Making this a central page to list my other wiki entries. ==
'''Standby Backup Machine''' [http://wiki.zimbra.com/index.php?title=Harley77-Backup]
'''Shell Scripts for...'</p>
<hr />
<div><br />
== Making this a central page to list my other wiki entries. ==<br />
<br />
<br />
<br />
'''Standby Backup Machine''' [http://wiki.zimbra.com/index.php?title=Harley77-Backup]<br />
<br />
'''Shell Scripts for Backup''' [http://wiki.zimbra.com/index.php?title=Harley77-Scripts]<br />
<br />
'''mysql corruption, high cpu usage, and how I fixed it.''' [http://wiki.zimbra.com/index.php?title=Harley77-Mysqld]</div>Harley77https://wiki.zimbra.com/index.php?title=Harley77-Notes&diff=12647Harley77-Notes2009-02-05T17:35:31Z<p>Harley77: /* Installing Zimbra for Cold-Standby Operation */</p>
<hr />
<div>'''My Wonderful adventure in backing stuff Up'''<br />
<br />
This is a work in progress, it may and probably does contain errors, misspellings, bad code, and bad grammar. Be warned. <br />
<br />
Ok, so, I originally had a "Cold Backup Server" that I setup using some info gleaned from the forums. The solution that was provided didn't quite work for me. For one, it didn't provide a fast enough fail over AND due to misspellings and other steps left out of the script, it didn't work either.<br />
<br />
I decided to try something that was probably not the ideal way, but, it was effective. I used a Dell workstation with a big HDD and a decent amount of ram, installed ubuntu 6.02lts (same as our live server) and installed Zimbra. The following is from my own documentation of this process. <br />
<br />
<br />
<br />
== Install Ubuntu 6.0 LTS Server ==<br />
<br />
I'm not going into detail on how to install ubuntu, but, rest assured it is a required step. <br />
One thing to note however, the hostname should be identical to your LIVE server. <br />
<br />
After you have ubuntu installed, you may proceed to the next step. <br />
<br />
'''Add the “universe” depository to the apt-get updater''' <br />
<br />
*Open /etc/apt/sources.list – “pico /etc/apt/sources.list”<br />
*Remove the # in front of :<br />
deb http://us.archive.ubuntu.com/ubuntu/ dapper universe<br />
deb-src http://us.archive.ubuntu.com/ubuntu/ dapper universe<br />
*Ctrl-X and press Y to save.<br />
*Run <br />
apt-get update<br />
<br />
== Install dnsmasq ==<br />
<br />
Run the following commands<br />
apt-get install dnsmasq<br />
pico /etc/dnsmasq.conf<br />
<br />
*Add the following lines:<br />
address=/your.maildomain.com/X.X.X.X<br />
mx-host=maildomain.com,your.maildomain.com,10<br />
mx-host=your.maildomain.com,your.maildomain.com,15<br />
mx-target=your.maildomain.com<br />
*Press Ctrl+X and Y to save<br />
<br />
== Install Zimbra Prerequisites ==<br />
<br />
*Run <br />
apt-get install libidn11 fetchmail libxml2 libstdc++5 openssl libltdl3 perl<br />
<br />
#Set the DNS Resolver to point to itself.<br />
*Type <br />
mv /etc/resolv.conf resolv.conf.BU<br />
pico /etc/resolv.conf<br />
*Add the following:<br />
Nameservers X.X.X.X #where x is your local ip<br />
**Press “ctrl+X” then Y to save.<br />
<br />
<br />
<br />
== Install the zimbra version that matches what is currently running on the LIVE server. ==<br />
<br />
*Copy the .tgz file downloaded from Zimbra.com .<br />
*Copy the file to a directory on the server. I’ve been using /ZIMBRAINSTALL/<br />
*Decompress the file: “gunzip FILENAME”<br />
*Extract the resulting tar file: “tar –xvf FILENAME”<br />
*Go into the directory that was created. “cd FILENAME”<br />
<br />
<br />
Ok, I stop at this point because this is where my original and new methodologies begin to take different roads.<br />
<br />
To make a long story short, I was rsyncing the /opt/zimbra directory from my live server to the local system. It did this once every hour. I didn't shutdown anything, I just had it syncing. The standby server was not active (meaning, zimbra was not running). However, thanks to a modified version of the script provided by fisch09 in the forums, I could quickly switch it from cold standby to Hot mode with a single command.<br />
<br />
This past Christmas eve, our mail server crashed. I was 1500 miles away at the time and was able to remote into the backup machine, run the script, reboot and was back in action. So, it DOES work, but I've never been comfortable it. This fear was realized when we had a LDAP corruption and that corruption was synced with the backup server. This put an end to THAT particular setup. <br />
<br />
This method also meant that after my Zimbra install, I would essentially move it aside and sync over the existing install. Since that is no longer the case, the installing of zimbra now takes on a few more steps. For this, the Disaster recovery Wiki entry was very helpful. I will be detailing that below and adding in a few things that were not quite covered. So, I revisited fisch09's method. Basically, taking backups from the live server, syncing them over and restoring them on the backup system. So, now to continue our setup.<br />
<br />
<br />
==Installing Zimbra for Cold-Standby Operation==<br />
<br />
The process is pretty much identical to the method described for Disaster Recovery of the Network Edition. The following is essentially a cut and paste of the internal documentation I made for the company I work for. <br />
<br />
*Go to the extracted Zimbra install Directory and Run the installer <br />
./install.sh<br />
**Pay attention to any errors. <br />
**It will Confirm what packages you are installing. <br />
*Use whatever packages you use on the Live Server<br />
**After this, it will test to make sure the DNS is working properly<br />
If it isn’t you need to trouble shoot the dnsmasq settings and resolv.conf<br />
**Assuming all goes ok, it will stop at the config screen where you need to set Additional parameters. See next step. <br />
'''IMPORTANT – At this step, you need to set the system to match the EXISTING server your backing up.''' <br />
You need to set: <br />
#LDAP Passwords: There are 5 different ones. All of them are identical (typically).To find the existing passwords:<br />
*Log into the existing email server.<br />
su zimbra<br />
zmlocalconfig -s | grep -i pass<br />
*Write down the password used for the LDAP_ entries.<br />
#Admin Password<br />
#SPAM and HAM Email Addresses<br />
#Admin email address <br />
#wiki document email Address.<br />
11) Enable default backup schedule: no<br />
r) Start servers after configuration no<br />
#Apply the config and continue install. <br />
<br />
After it’s complete, we need to sync the backup directory and zimlets.<br />
<br />
==Sync the Backup Directory==<br />
<br />
#Establish a directory to place the backup files. I’m using an identical structure as the source server. <br />
*Type: mkdir /DRIVE2/backup<br />
*Type: chmod 777 /DRIVE2/backup<br />
#To sync the files, use the following command. <br />
rsync -e 'ssh -ax -c blowfish' -av 192.192.100.15:/DRIVE2/backup/ /DRIVE2/backup/<br />
#Enter the root password for the source server when prompted. This will take awhile if it’s the first time. <br />
<br />
==Sync the Zimlets==<br />
<br />
#We need to sync the zimlets on the source server with the new target server. Use the following command:<br />
rsync -e 'ssh -ax -c blowfish' -av 192.192.100.15:/opt/zimbra/mailboxd/webapps/service/zimlet <br />
/opt/zimbra/mailboxd/webapps/service/<br />
<br />
Since we haven’t yet, Lets reboot. It may not be absolutely needed, but It seems to solve issues I sometimes get after a clean install.. <br />
<br />
Once rebooted, <br />
su zimbra<br />
Now we are going to restore our LDAP to the latest incremental backup.<br />
Type:<br />
ldap start<br />
*in the list, will be a bunch of directories with a INCR prefix. Look for the newest one. Copy the whole directory name, this is our incremental backup label. <br />
For this example, we will use: incr-20090203.200008.925<br />
<br />
zmrestoreldap -lb incr-20090203.200008.925<br />
*This should show an output of the accounts being restored. Look for any errors. It will be done quickly. <br />
*LDAP Server should be started at this point. <br />
mysql.server start<br />
zmmailboxdctl start<br />
#Now to restore the main message store.<br />
zmrestore -restoreToIncrLabel incr-20090204.170011.969 -a "all" -t /Path/To/Backup<br />
#It won’t give you any feedback, so you need to open a second ssh session:<br />
#Login and Type: <br />
tail –f /opt/zimbra/log/mailbox.log<br />
#You can monitor its progress here. This will take a while. About 1 – 2 hours. <br />
<br />
Once the restore is complete:<br />
zmcontrol stop<br />
zmcontrol start<br />
#Use the IP address of the server to access it and login. Verify the data.</div>Harley77https://wiki.zimbra.com/index.php?title=Harley77-Notes&diff=12637Harley77-Notes2009-02-04T21:39:42Z<p>Harley77: New page: '''My Wonderful adventure in backing stuff Up''' This is a work in progress, it may and probably does contain errors, misspellings, bad code, and bad grammar. Be warned. Ok, so, I origi...</p>
<hr />
<div>'''My Wonderful adventure in backing stuff Up'''<br />
<br />
This is a work in progress, it may and probably does contain errors, misspellings, bad code, and bad grammar. Be warned. <br />
<br />
Ok, so, I originally had a "Cold Backup Server" that I setup using some info gleaned from the forums. The solution that was provided didn't quite work for me. For one, it didn't provide a fast enough fail over AND due to misspellings and other steps left out of the script, it didn't work either.<br />
<br />
I decided to try something that was probably not the ideal way, but, it was effective. I used a Dell workstation with a big HDD and a decent amount of ram, installed ubuntu 6.02lts (same as our live server) and installed Zimbra. The following is from my own documentation of this process. <br />
<br />
<br />
<br />
== Install Ubuntu 6.0 LTS Server ==<br />
<br />
I'm not going into detail on how to install ubuntu, but, rest assured it is a required step. <br />
One thing to note however, the hostname should be identical to your LIVE server. <br />
<br />
After you have ubuntu installed, you may proceed to the next step. <br />
<br />
'''Add the “universe” depository to the apt-get updater''' <br />
<br />
*Open /etc/apt/sources.list – “pico /etc/apt/sources.list”<br />
*Remove the # in front of :<br />
deb http://us.archive.ubuntu.com/ubuntu/ dapper universe<br />
deb-src http://us.archive.ubuntu.com/ubuntu/ dapper universe<br />
*Ctrl-X and press Y to save.<br />
*Run <br />
apt-get update<br />
<br />
== Install dnsmasq ==<br />
<br />
Run the following commands<br />
apt-get install dnsmasq<br />
pico /etc/dnsmasq.conf<br />
<br />
*Add the following lines:<br />
address=/your.maildomain.com/X.X.X.X<br />
mx-host=maildomain.com,your.maildomain.com,10<br />
mx-host=your.maildomain.com,your.maildomain.com,15<br />
mx-target=your.maildomain.com<br />
*Press Ctrl+X and Y to save<br />
<br />
== Install Zimbra Prerequisites ==<br />
<br />
*Run <br />
apt-get install libidn11 fetchmail libxml2 libstdc++5 openssl libltdl3 perl<br />
<br />
#Set the DNS Resolver to point to itself.<br />
*Type <br />
mv /etc/resolv.conf resolv.conf.BU<br />
pico /etc/resolv.conf<br />
*Add the following:<br />
Nameservers X.X.X.X #where x is your local ip<br />
**Press “ctrl+X” then Y to save.<br />
<br />
<br />
<br />
== Install the zimbra version that matches what is currently running on the LIVE server. ==<br />
<br />
*Copy the .tgz file downloaded from Zimbra.com .<br />
*Copy the file to a directory on the server. I’ve been using /ZIMBRAINSTALL/<br />
*Decompress the file: “gunzip FILENAME”<br />
*Extract the resulting tar file: “tar –xvf FILENAME”<br />
*Go into the directory that was created. “cd FILENAME”<br />
<br />
<br />
Ok, I stop at this point because this is where my original and new methodologies begin to take different roads.<br />
<br />
To make a long story short, I was rsyncing the /opt/zimbra directory from my live server to the local system. It did this once every hour. I didn't shutdown anything, I just had it syncing. The standby server was not active (meaning, zimbra was not running). However, thanks to a modified version of the script provided by fisch09 in the forums, I could quickly switch it from cold standby to Hot mode with a single command.<br />
<br />
This past Christmas eve, our mail server crashed. I was 1500 miles away at the time and was able to remote into the backup machine, run the script, reboot and was back in action. So, it DOES work, but I've never been comfortable it. This fear was realized when we had a LDAP corruption and that corruption was synced with the backup server. This put an end to THAT particular setup. <br />
<br />
This method also meant that after my Zimbra install, I would essentially move it aside and sync over the existing install. Since that is no longer the case, the installing of zimbra now takes on a few more steps. For this, the Disaster recovery Wiki entry was very helpful. I will be detailing that below and adding in a few things that were not quite covered. So, I revisited fisch09's method. Basically, taking backups from the live server, syncing them over and restoring them on the backup system. So, now to continue our setup.<br />
<br />
<br />
==Installing Zimbra for Cold-Standby Operation==<br />
<br />
The process is pretty much identical to the method described for Disaster Recovery of the Network Edition. The following is essentially a cut and paste of the internal documentation I made for the company I work for. <br />
<br />
*Go to the extracted Zimbra install Directory and Run the installer <br />
./install.sh<br />
**Pay attention to any errors. <br />
**It will Confirm what packages you are installing. <br />
*Use whatever packages you use on the Live Server<br />
**After this, it will test to make sure the DNS is working properly<br />
If it isn’t you need to trouble shoot the dnsmasq settings and resolv.conf<br />
**Assuming all goes ok, it will stop at the config screen where you need to set Additional parameters. See next step. <br />
'''IMPORTANT – At this step, you need to set the system to match the EXISTING server your backing up.''' <br />
You need to set: <br />
#LDAP Passwords: There are 5 different ones. All of them are identical (typically).To find the existing passwords:<br />
*Log into the existing email server.<br />
su zimbra<br />
zmlocalconfig -s | grep -i pass<br />
*Write down the password used for the LDAP_ entries.<br />
#Admin Password, (our typical admin password)<br />
#SPAM and HAM Email Addresses<br />
#Admin email address <br />
#wiki document email Address.<br />
11) Enable default backup schedule: no<br />
r) Start servers after configuration no<br />
#Apply the config and continue install. <br />
<br />
After it’s complete, we need to sync the backup directory and zimlets.<br />
<br />
==Sync the Backup Directory==<br />
<br />
#Establish a directory to place the backup files. I’m using an identical structure as the source server. <br />
*Type: mkdir /DRIVE2/backup<br />
*Type: chmod 777 /DRIVE2/backup<br />
#To sync the files, use the following command. <br />
rsync -e 'ssh -ax -c blowfish' -av 192.192.100.15:/DRIVE2/backup/ /DRIVE2/backup/<br />
#Enter the root password for the source server when prompted. This will take awhile if it’s the first time. <br />
<br />
==Sync the Zimlets==<br />
<br />
#We need to sync the zimlets on the source server with the new target server. Use the following command:<br />
rsync -e 'ssh -ax -c blowfish' -av 192.192.100.15:/opt/zimbra/mailboxd/webapps/service/zimlet <br />
/opt/zimbra/mailboxd/webapps/service/<br />
<br />
Since we haven’t yet, Lets reboot. It may not be absolutely needed, but It seems to solve issues I sometimes get after a clean install.. <br />
<br />
Once rebooted, <br />
su zimbra<br />
Now we are going to restore our LDAP to the latest incremental backup.<br />
Type:<br />
ldap start<br />
*in the list, will be a bunch of directories with a INCR prefix. Look for the newest one. Copy the whole directory name, this is our incremental backup label. <br />
For this example, we will use: incr-20090203.200008.925<br />
<br />
zmrestoreldap -lb incr-20090203.200008.925<br />
*This should show an output of the accounts being restored. Look for any errors. It will be done quickly. <br />
*LDAP Server should be started at this point. <br />
mysql.server start<br />
zmmailboxdctl start<br />
#Now to restore the main message store.<br />
zmrestore -restoreToIncrLabel incr-20090204.170011.969 -a "all" -t /Path/To/Backup<br />
#It won’t give you any feedback, so you need to open a second ssh session:<br />
#Login and Type: <br />
tail –f /opt/zimbra/log/mailbox.log<br />
#You can monitor its progress here. This will take a while. About 1 – 2 hours. <br />
<br />
Once the restore is complete:<br />
zmcontrol stop<br />
zmcontrol start<br />
#Use the IP address of the server to access it and login. Verify the data.</div>Harley77