Ajcody-Server-Topics
Server Topics
Actual Server Topics Homepage
Please see Ajcody-Server-Topics
Server Move
Please see Ajcody-Notes-ServerMove
Backup Plans And Cheap HA/DR Secondary Server
Please see Ajcody-Notes-BackupPlans
My Disaster Recovery Failed
Reference: Network_Edition_Disaster_Recovery#Restoring_to_the_new_server
First, make sure your TIME is set right! See Time_Zones_in_ZCS#The_server_OS
Though I'm still investigating why this is happening for our customers, the root issues seem to be resolved by the following. Some of the auth errors will be logged to /var/log/zimbra.log and /opt/zimbra/log/mailbox.log
Put the output of this command in a text file:
zmlocalconfig -s | grep password
These should match what's in /opt/zimbra/conf/localconfig.xml
Now, compare the passwords with what is in your restore. Put in your path of the restore specific directory.
vi /opt/zimbra/backup/sessions/YOUR_DIR/sys/localconfig.xml
Do a /password or the full variable in vi to see what the old passwords are. You'll need to adjust the following below. Remember to copy a backup.
cp /opt/zimbra/conf/localconfig.xml /opt/zimbra/conf/localconfig.xml.DR
- ldap_postfix_password
- vi /opt/zimbra/conf/localconfig.xml and put in the needed password
- ldap_amavis_password
- vi /opt/zimbra/conf/localconfig.xml and put in the needed password
- ldap_root_password
- http://wiki.zimbra.com/index.php?title=Resetting_LDAP_%26_MySQL_Passwords Note difference between 4.x and 5.x
- zimbra_ldap_password
- http://wiki.zimbra.com/index.php?title=Resetting_LDAP_%26_MySQL_Passwords Note difference between 4.x and 5.x
Hostname resolution issues and testing commands
Allot of failed installs are because administrators are skipping the steps to make sure resolution is working.
Check your /etc/hosts file. You should have the localhost similar to the one below. Your ZCS server name should be replaced with the zimbra.test.com and zimbra fields. The IP address should be the IP address bound to the network interface [ip addr or ifconfig -a to confirm].
127.0.0.1 localhost.localdomain localhost 192.168.0.1 zimbra.test.com zimbra
If your ZCS server is behind a firewall or is being NAT'd. Make sure that resolution for the hostname on the ZCS server returns the internal IP information rather than the external IP. To test:
hostname -f zimbra.test.com
or
domainname -f zimbra.test.com
As long as one of the above returns the full hostname, use the command for the following. The MX lookups depend on how you are doing your mail domain and server hostname.
host `hostname -f` zimbra.test.com has address 192.168.0.1 host -t MX `domainname -f` zimbra.test.com mail is handled by 10 zimbra.test.com. host -t MX `domainname -d` test.com mail is handled by 10 zimbra.test.com.
You should also have reverse records (PTR) as well. Replace 192.168.0.1 using your internal ip address, it should return something like:
host 192.168.0.1 1.0.168.192.in-addr.arpa domain name pointer zimbra.test.com
Please review wiki page on split-DNS :
http://wiki.zimbra.com/index.php?title=Split_DNS
and also the variable use of lmtp_host_lookup .
http://bugzilla.zimbra.com/show_bug.cgi?id=27988
What's my time and timezone?
I wrote up the steps to make sure the ZCS server is using the correct time and timezone here:
http://wiki.zimbra.com/index.php?title=Time_Zones_in_ZCS#The_server_OS
Disable LDAP Replica
This is a work in progress, please don't use for production servers at this time.
References:
http://wiki.zimbra.com/index.php?title=Promoting_Replica_to_LDAP_Master
http://www.zimbra.com/docs/os/latest/multi_server_install/LDAP%20Replication.6.1.html
For each server that you want to change:
Stop the Zimbra services on the server, zmcontrol stop.
Check the existing ldap_url value.
zmlocalconfig | grep ldap_url
Update the ldap_url value. Remove the replica LDAP server URL, below assumes you only had one replica.
zmlocalconfig -e ldap_url=”ldap:// ”
If other replica's exist, then the list typed is like:
zmlocalconfig -e ldap_url=”ldap:// ldap:// ldap:// ”
The hosts are tried in the order listed. The master URL must always be included and is listed last.
Additional Steps for MTA hosts.
After updating the ldap_url, rerun /opt/zimbra/libexe/zmmtainit. This rewrites the Postfix configuration with the updated ldap_url.
To stop the ldap service running on the ldap replica server.
su - zimbra ldap stop
To now disable ldap from running on the old ldap replica. The - in front of zimbraServiceEnabled is [off], rather than a + for [on].
zmprov ms -- -zimbraServiceEnabled ldap
Public Service Hostname
variable is zimbraPublicServiceHostname , as referenced in /opt/zimbra/openldap/etc/openldap/schema/zimbra.schema
Bugs about the url's for shares that use spaces (%20), the redirection drops the %20
http://bugzilla.zimbra.com/show_bug.cgi?id=27788
http://bugzilla.zimbra.com/show_bug.cgi?id=27477
Domain Rename Issues
We have a command to rename a "domain" and the resources within it - unfortunately it doesn't handle ALL of the different components for various reasons. Specifically Documents and Calendars [more down below].
Review this RFE that was done when they implemented the renameDomain command.
http://bugzilla.zimbra.com/show_bug.cgi?id=7655
Note comment 21 & 22 (confirming what QA'd). The syntax is
zmprov -l rd testold.com testnew.com
Problems that arise with Documents are explained here, comment #1 has workaround.
http://bugzilla.zimbra.com/show_bug.cgi?id=25873
Now, the very unfortunate part about calendars with a domainname change/move.
"Ability to change a username globally on all appointments"
http://bugzilla.zimbra.com/show_bug.cgi?id=26736
"Cant edit calendar entries after renameDomain"
http://bugzilla.zimbra.com/show_bug.cgi?id=27707
The root issue here's seems to be more about the calendar standards and practices with the use of the "Organizer" field and notifications. You'll see 3 "work arounds" in comment 2 of bug 26736.
Export/Importing of the calendar data is shown here:
http://wiki.zimbra.com/index.php?title=User_Migration
This part specifically :
http://wiki.zimbra.com/index.php?title=User_Migration#Copy_Calendar_From_One_Zimbra_User_to_Another
My Gal & LDAP Settings For A Domain
To see your setting, do the following - replacing with domainname with the domain in question.
zmprov gd domainname | egrep -i 'ldap|gal'
Zimlets
Location Of Zimlets
Zimlets should be already located on the zimbra server in one of these directories:
/opt/zimbra/zimlets/ /opt/zimbra/zimlets-admin-extra/ /opt/zimbra/zimlets-extra/ /opt/zimbra/zimlets-experimental/ /opt/zimbra/zimlets-network/
How To Deploy Zimlets
To deploy a zimlet, simply cd to the directory where the zimlet is located and issue this command:
/opt/zimbra/bin/zmzimletctl deploy <zimlet_name>
Archive & Discovery
References for A&D
The two main A&D references are :
http://wiki.zimbra.com/index.php?title=ZAD
http://www.zimbra.com/pdf/Zimbra%20Archiving%20and%20Discovery%20Release%20Notes.pdf
Multi-Server & New Mailstore A&D setup
See below for a very rough draft document I made for multi-server / new mailstore A&D setups.
http://wiki.zimbra.com/index.php?title=Ajcody-Notes-AD-mailstore
I also created a RFE for documentation on this.
http://bugzilla.zimbra.com/show_bug.cgi?id=25135
NO_SUCH_BLOB Errors
Actual No Such Blob Homepage
Please see Ajcody-Notes-No-Such-Blob
Other Resource Overviews
It would be good to review this wiki page before you start any "changes":
http://wiki.zimbra.com/index.php?title=Account_mailbox_database_structure
And great forum posting:
http://www.zimbra.com/forums/administrators/19811-solved-missing-blob-errors-zimbra-4-5-10-a.html
Check Permissions First
If you recently did a server move or similar type of operation, make sure zimbra:zimbra permissions are applied correctly to store directory. You could run the following to double check permissions (as root):
/opt/zimbra/libexec/zmfixperms --verbose --extended
Perl To Find MailboxID's That Have Problem
And down and dirty way to see what mailboxes have this error:
grep -B2 NO_SUCH_BLOB /opt/zimbra/log/mailbox.lo* |grep mailbox= |sed -r 's/.*mailbox=([0-9]*).*$/\1/' |sort -u
Mapping MailboxId to User Account
Remember, mailboxId is unique to the mailstore - it's not a global user variable that's unique.
Stolen from forum post:
http://www.zimbra.com/forums/administrators/16885-mailbox_id-email-address.html
su - zimbra mysql use zimbra; select comment from mailbox where id=257; And you'll get back: +----------------------+ | comment | +----------------------+ | user@domain.com | +----------------------+ 1 row in set (0.00 sec) If you wanted to see all the info for that account: select * from mailbox where id=257; And you'll get back: +-----+----------+--------------------------------------+-----------------+--------------------+---------------+-----------------+-------------------+---------------+---------------+----------------+----------------------+------------------+--------------+--------------------+ | id | group_id | account_id | index_volume_id | item_id_checkpoint | contact_count | size_checkpoint | change_checkpoint | tracking_sync | tracking_imap | last_backup_at | comment | last_soap_access | new_messages | idx_deferred_count | +-----+----------+--------------------------------------+-----------------+--------------------+---------------+-----------------+-------------------+---------------+---------------+----------------+----------------------+------------------+--------------+--------------------+ | 257 | 71 | xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxxxxxxx | 2 | 257 | 0 | 1832 | 100 | 0 | 0 | NULL | user@domain.com | 1209652908 | 0 | 0 | +-----+----------+--------------------------------------+-----------------+--------------------+---------------+-----------------+-------------------+---------------+---------------+----------------+----------------------+------------------+--------------+--------------------+ 1 row in set (0.00 sec)
Zmblobchk for 5.0.6+ Systems
On 5.0.6+ there's a script in /opt/zimbra/bin called zmblobchk . This will list out information about the NO_SUCH_BLOB errors and other inconsistencies.
zmblobchk - RFE's
zmblobchk currently just lists "issues" - for the most part. Please see the following RFE's, some will give details about resolutions based upon what zmblobchk reported.
- "blobchk: Add support for 'repair-mode' to the zmblobchk tool" Resolved for ZCS 6.0.6
- The initial RFE for the zmblobchk tool
- "tool to do consistency checks and repair for missing blob for ID x" Resolved for ZCS 5.0.6, "repair" part done in bug 27958.
Example Output On A System With No Issues
su - zimbra ls /opt/zimbra/zmblobchk.jar zmblobchk.jar java -jar zmblobchk.jar Retrieving volume information Retrieving mboxgroup list Spooling item list to /tmp/mailitems53650.lst Retrieving items from mboxgroup1 Retrieving items from mboxgroup2 Retrieving items from mboxgroup3 Retrieving items from mboxgroup4 Retrieving items from mboxgroup5 Retrieving items from mboxgroup6 Retrieving items from mboxgroup7 Retrieving items from mboxgroup8 Retrieving items from mboxgroup9 Retrieving items from mboxgroup10 / Processing BLOB store - /tmp/mailitems53650.lst: size 28141 Processed 329 items in 1223ms No inconsistencies found
Example Output On A System With Issues
Last login: Thu Oct 30 12:03:12 on ttys000 COLDMILE-LM-3:~ ajcody$ ssh root@mail3.internal.homeunix.com root@mail3.internal.homeunix.com's password: Last login: Sat Nov 1 14:25:50 2008 from 192.168.0.13 [root@mail3 ~]# su - zimbra [zimbra@mail3 ~]$ zmblobchk Retrieving volume information Retrieving mboxgroup list Spooling item list to /tmp/mailitems39690.lst Retrieving items from mboxgroup1 Retrieving items from mboxgroup2 Retrieving items from mboxgroup3 Retrieving items from mboxgroup4 Retrieving items from mboxgroup5 Retrieving items from mboxgroup6 Retrieving items from mboxgroup7 Retrieving items from mboxgroup8 Retrieving items from mboxgroup9 Retrieving items from mboxgroup10 Retrieving items from mboxgroup11 Retrieving items from mboxgroup12 Retrieving items from mboxgroup13 Retrieving items from mboxgroup14 Retrieving items from mboxgroup15 Retrieving items from mboxgroup16 Retrieving items from mboxgroup17 Retrieving items from mboxgroup18 Retrieving items from mboxgroup19 Retrieving items from mboxgroup20 - Processing BLOB store / /tmp/mailitems39690.lst: size 470853 Processed 5873 items in 18462ms 11844 inconsistencies found * MailboxGroup=1, mailbox=1, item=303: wrong volume, expected at /vol1/0/1/msg/0/303-2062.msg, found at /opt/zimbra/store/0/1/msg/0/303-2062.msg (move to correct volume) * MailboxGroup=1, mailbox=1, item=304: wrong volume, expected at /vol1/0/1/msg/0/304-2064.msg, found at /opt/zimbra/store/0/1/msg/0/304-2064.msg (move to correct volume) * MailboxGroup=1, mailbox=1, item=305: wrong volume, expected at /vol1/0/1/msg/0/305-2077.msg, found at /opt/zimbra/store/0/1/msg/0/305-2077.msg (move to correct volume) * MailboxGroup=1, mailbox=1, item=306: wrong volume, expected at /vol1/0/1/msg/0/306-2091.msg, found at /opt/zimbra/store/0/1/msg/0/306-2091.msg (move to correct volume) ## CUT OF 1000's OF SIMILAR LINES ## * MailboxGroup=1, mailbox=1, item=6480: file not found: /vol1/0/1/msg/1/6480-20100.msg (delete associated metadata) * MailboxGroup=3, mailbox=3, item=400: wrong volume, expected at /vol1/0/3/msg/0/400-11287.msg, found at /opt/zimbra/store/0/3/msg/0/400-11287.msg (move to correct volume) * MailboxGroup=3, mailbox=3, item=425: wrong volume, expected at /vol1/0/3/msg/0/425-12939.msg, found at /opt/zimbra/store/0/3/msg/0/425-12939.msg (move to correct volume) * MailboxGroup=3, mailbox=3, item=427: wrong volume, expected at /vol1/0/3/msg/0/427-12943.msg, found at /opt/zimbra/store/0/3/msg/0/427-12943.msg (move to correct volume) * MailboxGroup=3, mailbox=3, item=440: file not found: /vol1/0/3/msg/0/440-13400.msg (delete associated metadata) * /opt/zimbra/store/0/3/msg/0/363-6194.msg: no associated metadata (delete blob) * /opt/zimbra/store/0/3/msg/0/360-5000.msg: no associated metadata (delete blob) * /opt/zimbra/store/0/3/msg/0/350-3129.msg: no associated metadata (delete blob) * /opt/zimbra/store/0/3/msg/0/320-2272.msg: no associated metadata (delete blob) ## CUT OF 1000's OF SIMILAR LINES ## * /opt/zimbra/store/0/2/msg/0/304-6907.msg: no associated metadata (delete blob) * /opt/zimbra/store/0/2/msg/0/315-6918.msg: no associated metadata (delete blob) * /opt/zimbra/store/0/2/msg/0/312-6915.msg: no associated metadata (delete blob) * /opt/zimbra/store/0/2/msg/0/308-6911.msg: no associated metadata (delete blob) * /opt/zimbra/store/0/2/msg/0/303-6906.msg: no associated metadata (delete blob) * /opt/zimbra/store/0/2/msg/0/301-6904.msg: no associated metadata (delete blob) *** 11844 faults found Report saved to: /tmp/zmblobc39689.rpt [zimbra@mail3 ~]$
To see the reported output again:
zmblobchk -l /tmp/zmblobc39689.rpt
Error - Not Enough Memory To Run zmblobchk
If the "Processing BLOB store" part errors out with Java out of memory, you could try adjusting the variable for it. The default is set to 30%. This would require a mailstore/jetty restart though [zmmailboxdctl restart]. If the mailstore doesn't come back up, check /opt/zimbra/log/zmmailbox.log - you most likely exceeded the threshold for your box. Remember, this is a percentage and a 32bit machine with more than 4GB can cause issues with this setting since it could try to allocate more than it actually can to the thread.
# ZCS 5.0 and later $ zmlocalconfig -e mailboxd_java_heap_memory_percent=40
And zmmailboxdctl restart when it's appropriate.
BLOB Issue Script
I've commented out the two delete lines and also the "$msghash &= $fmask;" one.
If you're using HSM, this script isn't smart enough to detect it. It will flag and remove entries that are in the HSM store because it doesn't see them in the primary store. You can modify the output though to use the actual path to the HSM store. I had one case where this was the case and copying the missing blobs to the HSM path worked.
Comment from other support staff member on a case that used this.
Once you've gotten a list of the 'NOT OK' files generated by this tool, you'll want to check the store directories to see if the message blobs exist with an incorrect change number. That's the second number in the filename. If the first number matches the ID it's looking for but the second does not, you can rename the file to the new change number and get the message blob back without losing data. If not, then the blob is completely missing. If that's the case, we should try to figure out what happened to it by going through the mailbox.log files. You should be able to grep for the missing message ID and the user's account ID to find a log entry showing something happening to it. It's possible it was deleted and the deletion was never written to mysql for some reason.
#!/usr/bin/perl # On 5.0.6+ there's a script, /opt/zimbra/bin/zmblobchk , that you can use instead of this. # This script is compatible with Zimbra version 4.5.x and 5.x . # OK, there's 2 MAILBOX_*_BITS values in the VOLUME table. # Take the mailbox ID, right-shift it by MAILBOX_BITS, and take the lowest MAILBOX_GROUP_BITS of the result. # That's your mailbox hash. # Take the message ID, right-shift it by FILE_BITS, and take the lowest FILE_GROUP_BITS of the result. That's your msgid hash. # I think. # <mbx-hash>/<mbx-id>/msg/<msgid-hash>/<msgid>-<mod_content>.msg my ($fbits, $fgbits, $mbits, $mgbits, $basepath) = split (' ',`echo "select file_bits,file_group_bits,mailbox_bits,mailbox_group_bits,path from volume where type='1'" | mysql -N zimbra`); my $ARGV = shift; chomp $basepath; my $mbmask = "1" x $mgbits; my $fmask = "1" x $fgbits; foreach (`echo "select id, group_id, account_id, comment from mailbox where comment = '$ARGV'" | mysql -N zimbra`) { chomp; my $path = "$basepath/"; my ($id, $grid, $aid, $nm) = split; my $mbhash = $id >> $mbits; $mbhash &= $mbmask; $path .= $mbhash."/".$id."/msg/"; foreach my $msgstuff (`echo "select id, mod_content, type from mail_item where blob_digest is not null and mailbox_id=${id};" | mysql -N mboxgroup${grid}`) { chomp $msgstuff; next if $msgstuff eq ""; my ($msgid, $modContent, $type) = split (' ',$msgstuff); my $msghash = $msgid >> $fbits; # $msghash &= $fmask; my $nm = $msgid; if ($modContent) {$nm .= "-$modContent";} my $npath = $path.$msghash."/".$nm.".msg"; if (-e $npath) {print $npath."\t\tOK\n";} else { print $npath."\t\tNOT OK\n"; # not ok, remove the entry from the database so it is not a nuisance print "Delete from mail_item where MSGID is ${msgid} and MAILBOXID is ${id} \n"; print "Uncomment line below me in script to have me delete. \n"; # `echo "delete from mail_item where id=${msgid} and mailbox_id=${id}" | mysql -N mboxgroup${grid}`; if ($type eq "11") { print "Delete from appointment where MSGID is ${msgid} and MAILBOX_ID is ${id} \n"; print "Uncomment line below me in script to have me delete. \n"; # `echo "delete from appointment where item_id=${msgid} and mailbox_id=${id}" | mysql -N mboxgroup${grid}`; } } # print $npath."\n"; } }
If this doesn't run, make sure perl is installed and in the correct path.
You run this as the zimbra user.
What I did was the following.
Copied the script to /opt/zimbra/bin/ as root and called it blob-check.pl . chmod +x /opt/zimbra/bin/blob-check.pl
su - zimbra mkdir /tmp/BLOB-CHECKS for i in `zmprov gaa`; do blob-check.pl $i | grep "NOT OK" >> /tmp/BLOB-CHECKS/$i.txt; done ls -la /tmp/BLOB-CHECKS
The ls will help identify the more troubled accounts. You'll end up with something like this:
[zimbra@mail3 ~]$ ls -la /tmp/BLOB-CHECKS/ total 16 drwxr-x--- 2 zimbra zimbra 4096 Jul 10 23:40 . drwxrwxrwt 8 root root 4096 Jul 10 23:46 .. -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 admin@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 ajcody2@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 49 Jul 10 23:40 ajcody@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 forward@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 ham.bidiob2mm@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 secondary@secondary.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 spam.rormmtcyy@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 wiki@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 wiki@secondary.internal.homeunix.com.txt
And the output of one that shows some size:
$ cat /tmp/BLOB-CHECKS/ajcody@mail3.internal.homeunix.com.txt /opt/zimbra/store/0/3/msg/0/360-6070.msg NOT OK
Remember this from above, "you'll want to check the store directories to see if the message blobs exist with an incorrect change number. That's the second number in the filename. If the first number matches the ID it's looking for but the second does not, you can rename the file to the new change number and get the message blob back without losing data."
One can uncomment the 2 lines in the script with delete and run the script again. This will remove the reference to the blob.
To re-index the users mailbox [this can take start, cancel, status]:
zmprov rim user@domainname start
Running the script again shouldn't show "NOT OK" lines.
FYI - I still need to gather more information about this situation and how to guide one to make a decision to delete, re-index, and so forth.
BLOB Script To Copy From Restore
You might need to use these variables with the restore command. I need to find a way for someone to figure out what restore to goto for a particular blob:
-restoreToTime <arg> - Replay the redo logs until the time specified.
-restoreToIncreLabel <arg> - Replay redo logs up to and including this incremental backup.
-restoreToRedoSeq <arg> - Replay up to and including this redo log sequence
-br - Replays the redo logs in backup only, which excludes archived and current redo logs of the system.
-rf - Restores to the full backup only, does not include any incremental backups since that backup
This is a walk through to test the proceedure.
$ zmrestore -ca -a ajcody@mail3.internal.homeunix.com -pre restored- $ zmprov gmi restored-ajcody@mail3.internal.homeunix.com mailboxId: 12 quotaUsed: 251513 $ zmprov gmi ajcody@mail3.internal.homeunix.com mailboxId: 3 quotaUsed: 251512 $ diff /opt/zimbra/store/0/12/msg/0/257-25.msg /opt/zimbra/store/0/3/msg/0/257-25.msg $ rm /opt/zimbra/store/0/3/msg/0/257-25.msg rm: remove regular file `/opt/zimbra/store/0/3/msg/0/257-25.msg'? y $ cp /opt/zimbra/store/0/12/msg/0/257-25.msg /opt/zimbra/store/0/3/msg/0/257-25.msg $ grep Subject /opt/zimbra/store/0/3/msg/0/257-25.msg Log into main account and confirm I can see/use that mail.
The script to look at the blob-check.pl output and do the copies after a redirected restore is here:
#!/bin/bash #START OF VARIABLES# #CHANGEME to existing user account USER=ajcody@mail3.internal.homeunix.com #CHANGEME to the restored account name # zmrestore -ca -a ajcody@mail3.internal.homeunix.com -pre restored- RESTOREDUSER=restored-ajcody@mail3.internal.homeunix.com # To get FILE, blob-check.pl ajcody@mail3.internal.homeunix.com > /tmp/BLOB-CHECKS/ajcody.txt # blob-check.pl located at http://wiki.zimbra.com/index.php?title=Ajcody-Notes#BLOB_Issue_Script FILE=/tmp/BLOB-CHECKS/ajcody.txt #CHANGEME #CHANGEME, if needed, to the mailstore. /opt/zimbra/store is default STOREDIR=/opt/zimbra/store #END VARIABLES# USERUID=`zmprov gmi $USER | grep mailboxId | cut -f2 -d: | cut -c 2-10` USERGID=`perl -e 'print $USERUID >> 12 ; print "\n"'` USERPATH=`echo $USERGID/\$USERUID` RESTOREDUID=`zmprov gmi $RESTOREDUSER | grep mailboxId | cut -f2 -d: | cut -c 2-10` RESTOREDGID=`perl -e 'print $RESTOREDUID >> 12 ; print "\n"'` RESTOREDPATH=`echo $RESTOREDGID/\$RESTOREDUID` echo USERUID $USERUID echo USERGID $USERGID echo USERPATH $USERPATH echo RESTOREDUID $RESTOREDUID echo RESTOREDGID $RESTOREDGID echo RESTOREDPATH $RESTOREDPATH cd $STOREDIR echo "Run ECHO first to confirm copies look right and then uncomment COPY in script" for ROOTBLOBPATH in `grep "NOT OK" $FILE | cut -f7-9 -d/ | cut -f1` do # Comment out echo and uncomment copy after dry run echo $RESTOREDPATH/$ROOTBLOBPATH $USERPATH/$ROOTBLOBPATH #cp -uv $RESTOREDPATH/$ROOTBLOBPATH $USERPATH/$ROOTBLOBPATH done
BLOB Script To Copy From Restore (HSM Issue)
One customer ran into an issue that the blob references were really expecting them in the HSM mailstore path.
Example:
Default mailstore for PROD : /opt/zimbra/mail/
HSM mailstore path : /opt/zimbra/mail0 which was another partition mount, compared to /opt/zimbra/mail
So, what this script did was used the output from the blob-check.pl script which has the missing blobs formatted for the PROD path rather than the HSM path. The blob-check.pl script doesn't handle HSM data. The script then finds the blobs in the full backup directory of the user and copies them into the HSM path that the WEBCLIENT is expecting. This whole situation with this customer might of been a fluke. So don't consider these steps as applying to your situation unless you do your research. This case took 9 days to resolve and figure out.
Notice, if you use this script below you really need to review it in details and adjust it for your situation. There's more to adjust beyond the base variables.
#!/bin/bash # Steps # Make sure directory path exists for the copy command # Get mailboxId - zmprov ga USER@DOMAIN | grep zimbraId: # mkdir /opt/zimbra/mail0/0/[mailboxId #]/msg/{1..15} or how what ever # you need. # Change USEREMAIL # Get path for BACKUPFILES and uncomment for use # Run find command with echo and confirm. Then rerun using the find with cp rather than echo # Put in user's email for USEREMAIL USEREMAIL="USER@DOMAIN" ZIMBRAID=`zmprov ga $USEREMAIL | grep zimbraId: | awk '{print $2}'` # See below # ROOTBACKUPPATH="/opt/zimbra/mail0/BACKUP/sessions/full-20080707.004336.789" #The line below takes to long, see instructions underneath #BACKUPFILES=`find $ROOTBACKUPPATH -name *$ZIMBRAID*` # To run manually, which will increase speed of script if you then paste full path in for BACKUPFILES var # zmprov ga USER@DOMAIN | grep zimbraId: | awk '{print $2}' # Then place the out in this find command, leave the wildcards on each side of it # find /opt/zimbra/mail0/BACKUP/sessions/full-20080707.004336.789 -name *OUTPUTHERE* # This should give you path like the exampe below, paste it between the quotes and uncomment the variable BACKUPFILES #EXAMPLE FOR BACKUPFILES="/opt/zimbra/mail0/BACKUP/sessions/full-20080707.004336.789/accounts/a07/1b5/a071b5ad-b341-4de4-b6ee-4463f322873b" # BACKUPFILES="PASTE_IN_PATH_HER" for BLOBPATH in `blob-check.pl $USEREMAIL | grep "NOT OK" | awk '{print $1}' | sed 's/store/mail0/'` do MSGID=`echo $BLOBPATH | awk -F/ '{print $NF}'` # Run echo first to make sure copy path seems correct, ctrl-c to cancel. And then uncomment the other find command with the cp. Comment the echo one. find $BACKUPFILES -name *$MSGID* -exec echo {} $BLOBPATH \; #find $BACKUPFILES -name *$MSGID* -exec cp -uv {} $BLOBPATH \; done
The Really Bad Situation, Blob References Gone & Mail Still Missing
For now, this is all I have to offer.
Method One - zmlmtpinject
See forum post:
http://www.zimbra.com/forums/installation/12617-recover-data-mbox-folders.html#post64962
Method Two - Use IMAPSYNC
I haven't tested this and would just be guessing, but I believe IMAPSYNC could be used.
From the IMAPSYNC notes, "imapsync is the adequate tool because it reduces the amount of data transferred by not transferring a given message if it is already on both sides. Same headers, same message size and the transfer is done only once. All flags are preserved, unread will stay unread, read will stay read, deleted will stay deleted."
This would involve having imap setup on ZCS and then doing a redirected user restore (which would consume a license).
See the following on imapsync details:
- Ajcody-Server-Plan-Move-Migration-Upgrade-DR#Non-Zimbra_IMAP_Accounts_To_Zimbra
- Mail_Migration#using_imapsync_.28Recommended_Method.29
Imapsync During A Certain Time Frame
You might want to check out the imapsync options of --maxage and --minage
Proposed Steps
- Confirm these steps are necessary
- Install imapsync
- Do a redirected restore of the account
- This is an example format of the redirected restore:
zmrestore -ca -a prod-user@servername.com -pre restored- -rf -lb full-'date of full'
- The "new" account would be restored-userPROD@servername.com . The -rf flags should look only at the backup data in regards to the restore and not catch the "production" references to missing blobs and what not.
- This is an example format of the redirected restore:
- Confirm in webclient that msg's are there
- Do a IMAPSYNC
- Two ways, unsure what is the best method
- From PROD to NEW
imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates --host1 servername.com --user1 prod-user --password1 test123 -user2 restored-prod-user --host2 servername.com --password2 test123 --authmech1 PLAIN --authmech2 PLAIN
- From NEW to PROD
imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates --host1 servername.com --user1 restored-prod-user --password1 test123 -user2 prod-user --host2 servername.com --password2 test123 --authmech1 PLAIN --authmech2 PLAIN
- From PROD to NEW
- Two ways, unsure what is the best method
- Reindex restored-prod-user
- Rename prod-user to prod-user-old
zmprov ra prod-user@servername.com prod-user-old@servername.com
- Rename restored-prod-user to prod-user
zmprov ra restored-prod-user@servername.com prod-user@servername.com
- Do a full backup of each account
zmbackup -f - <servername.com> -a <newaccountname@servername.com>
Notes On - No Associated Metadata
Tentative Research At This Point - Haven't Proven Out Yet
This is for the errors you see from the blobchk like this:
"* /opt/zimbra/store/0/1303/msg/0/1059-1726.msg: no associated metadata (delete blob) "
You could use the following command to bring the blob back in and regenerate the metadata -- zmmailbox am along -d . The am flag details are : addMessage(am) [opts] {dest-folder-path} {filename-or-dir} [{filename-or-dir} ...] . But the -d flag states : -d/--date <arg> received date (msecs since epoch) . To get the epoch time in seconds using the same receive data, for example :
ls -l --time-style=+%s /opt/zimbra/store/0/1303/msg/0/1059-1726.msg
And then remember the -d requires milliseconds [a thousandth (1/1,000) of a second] so you'll need to convert it. Or, you could just use something like this below. This with the cut would work well for a for loop script against a number of msg files.
ls -l --time-style=+%s000 --si /opt/zimbra/store/0/1303/msg/0/1059-1726.msg | cut -d" " -f6 1272642001000 ls -l --time-style=+%s000 --si /opt/zimbra/store/0/1303/msg/0/1059-1726.msg -rw-r--r-- 1 zimbra zimbra 0 1272642001000
Another suggestion for scripting was:
date=$(stat -c +%Y /opt/zimbra/store/0/1303/msg/0/1059-1726.msg); echo $(($date*1000));
Mysql Queries To See What Is Shown For Missing Message ID
References
See also :
Getting User Variables we need to query MySQL
Please see Ajcody-Mysql-Topics#Getting_User_Variables_We_Need_To_Query_MySQL
Mysql Query For A Users Specific Blob-Data ID
Please see Useful Mysql Query Examples
Migration Issues
IMAPSYNC with admin login
Reference - http://wiki.zimbra.com/index.php?title=User_Migration
imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates \ --host1 server.gtds.lan --user1 yourAccount --password1 yourPassword \ --user2 yourZimbraAccount --authuser2 admin --password2 adminZimbraPassword --authmech2 LOGIN
I found this description in one of the imapsync files:
"You may authenticate as one user (typically an admin user), but be authorized as someone else, which means you don't need to know every user's personal password. Specify --authuser1 "adminuser" to enable this on host1. In this case, --authmech1 PLAIN will be used, but otherwise, --authmech1 CRAM-MD5 is the default. Same behavior with the --authuser2 option."
Mailing Lists And Mailman
If you are planning to use a distro list of over 60K members, you may be inclined to use a dedicated machine for this purpose. According to bug 19153, you have 2 options: "set up mailing list manager or change the value of virtual_alias_expansion_limit as a customization. We have never tested with > 1000 so this should be done carefully, and will pound LDAP for any messages with lots of recipients." Mailman integration is an option. There is a forum thread on how to integrate zcs and mailman. This would be the preferred fix to your issue, also noted in the bug report. We probably won't see Mailman integration in ZCS at least for another year or so, not until 6.0
Restricting Who Can Send To Mailing List
http://wiki.zimbra.com/index.php?title=RestrictPostfixRecipients
Problems Resolving Virtual Aliases For Members Of Large Distribution Lists
http://bugzilla.zimbra.com/show_bug.cgi?id=19153
Mailman Configuration
http://wiki.zimbra.com/index.php?title=ZCS_Mailman_Integration
http://bugzilla.zimbra.com/show_bug.cgi?id=8968
http://www.zimbra.com/forums/administrators/1380-solved-zimbra-mailman-howto.html