Ajcody-Server-Topics
Server Topics
Actual Server Topics Homepage
Please see Ajcody-Server-Topics
Server Move
Please see:
http://wiki.zimbra.com/index.php?title=Ajcody-Notes-ServerMove
Backup Plans And Cheap HA/DR Secondary Server
Please see:
http://wiki.zimbra.com/index.php?title=Ajcody-Notes-BackupPlans
My Disaster Recovery Failed
Reference: http://wiki.zimbra.com/index.php?title=Network_Edition_Disaster_Recovery#Restoring_to_the_new_server
First, make sure your TIME is set right! See Time_Zones_in_ZCS#The_server_OS
Though I'm still investigating why this is happening for our customers, the root issues seem to be resolved by the following. Some of the auth errors will be logged to /var/log/zimbra.log and /opt/zimbra/log/mailbox.log
Put the output of this command in a text file:
zmlocalconfig -s | grep password
These should match what's in /opt/zimbra/conf/localconfig.xml
Now, compare the passwords with what is in your restore. Put in your path of the restore specific directory.
vi /opt/zimbra/backup/sessions/YOUR_DIR/sys/localconfig.xml
Do a /password or the full variable in vi to see what the old passwords are. You'll need to adjust the following below. Remember to copy a backup.
cp /opt/zimbra/conf/localconfig.xml /opt/zimbra/conf/localconfig.xml.DR
- ldap_postfix_password
- vi /opt/zimbra/conf/localconfig.xml and put in the needed password
- ldap_amavis_password
- vi /opt/zimbra/conf/localconfig.xml and put in the needed password
- ldap_root_password
- http://wiki.zimbra.com/index.php?title=Resetting_LDAP_%26_MySQL_Passwords Note difference between 4.x and 5.x
- zimbra_ldap_password
- http://wiki.zimbra.com/index.php?title=Resetting_LDAP_%26_MySQL_Passwords Note difference between 4.x and 5.x
Hostname resolution issues and testing commands
Allot of failed installs are because administrators are skipping the steps to make sure resolution is working.
Check your /etc/hosts file. You should have the localhost similar to the one below. Your ZCS server name should be replaced with the zimbra.test.com and zimbra fields. The IP address should be the IP address bound to the network interface [ip addr or ifconfig -a to confirm].
127.0.0.1 localhost.localdomain localhost 192.168.0.1 zimbra.test.com zimbra
If your ZCS server is behind a firewall or is being NAT'd. Make sure that resolution for the hostname on the ZCS server returns the internal IP information rather than the external IP. To test:
hostname -f zimbra.test.com
or
domainname -f zimbra.test.com
As long as one of the above returns the full hostname, use the command for the following. The MX lookups depend on how you are doing your mail domain and server hostname.
host `hostname -f` zimbra.test.com has address 192.168.0.1 host -t MX `domainname -f` zimbra.test.com mail is handled by 10 zimbra.test.com. host -t MX `domainname -d` test.com mail is handled by 10 zimbra.test.com.
You should also have reverse records (PTR) as well. Replace 192.168.0.1 using your internal ip address, it should return something like:
host 192.168.0.1 1.0.168.192.in-addr.arpa domain name pointer zimbra.test.com
Please review wiki page on split-DNS :
http://wiki.zimbra.com/index.php?title=Split_DNS
and also the variable use of lmtp_host_lookup .
http://bugzilla.zimbra.com/show_bug.cgi?id=27988
What's my time and timezone?
I wrote up the steps to make sure the ZCS server is using the correct time and timezone here:
http://wiki.zimbra.com/index.php?title=Time_Zones_in_ZCS#The_server_OS
Disable LDAP Replica
This is a work in progress, please don't use for production servers at this time.
References:
http://wiki.zimbra.com/index.php?title=Promoting_Replica_to_LDAP_Master
http://www.zimbra.com/docs/os/latest/multi_server_install/LDAP%20Replication.6.1.html
For each server that you want to change:
Stop the Zimbra services on the server, zmcontrol stop.
Check the existing ldap_url value.
zmlocalconfig | grep ldap_url
Update the ldap_url value. Remove the replica LDAP server URL, below assumes you only had one replica.
zmlocalconfig -e ldap_url=”ldap:// ”
If other replica's exist, then the list typed is like:
zmlocalconfig -e ldap_url=”ldap:// ldap:// ldap:// ”
The hosts are tried in the order listed. The master URL must always be included and is listed last.
Additional Steps for MTA hosts.
After updating the ldap_url, rerun /opt/zimbra/libexe/zmmtainit. This rewrites the Postfix configuration with the updated ldap_url.
To stop the ldap service running on the ldap replica server.
su - zimbra ldap stop
To now disable ldap from running on the old ldap replica. The - in front of zimbraServiceEnabled is [off], rather than a + for [on].
zmprov ms -- -zimbraServiceEnabled ldap
Public Service Hostname
variable is zimbraPublicServiceHostname , as referenced in /opt/zimbra/openldap/etc/openldap/schema/zimbra.schema
Bugs about the url's for shares that use spaces (%20), the redirection drops the %20
http://bugzilla.zimbra.com/show_bug.cgi?id=27788
http://bugzilla.zimbra.com/show_bug.cgi?id=27477
Domain Rename Issues
We have a command to rename a "domain" and the resources within it - unfortunately it doesn't handle ALL of the different components for various reasons. Specifically Documents and Calendars [more down below].
Review this RFE that was done when they implemented the renameDomain command.
http://bugzilla.zimbra.com/show_bug.cgi?id=7655
Note comment 21 & 22 (confirming what QA'd). The syntax is
zmprov -l rd testold.com testnew.com
Problems that arise with Documents are explained here, comment #1 has workaround.
http://bugzilla.zimbra.com/show_bug.cgi?id=25873
Now, the very unfortunate part about calendars with a domainname change/move.
"Ability to change a username globally on all appointments"
http://bugzilla.zimbra.com/show_bug.cgi?id=26736
"Cant edit calendar entries after renameDomain"
http://bugzilla.zimbra.com/show_bug.cgi?id=27707
The root issue here's seems to be more about the calendar standards and practices with the use of the "Organizer" field and notifications. You'll see 3 "work arounds" in comment 2 of bug 26736.
Export/Importing of the calendar data is shown here:
http://wiki.zimbra.com/index.php?title=User_Migration
This part specifically :
http://wiki.zimbra.com/index.php?title=User_Migration#Copy_Calendar_From_One_Zimbra_User_to_Another
My Gal & LDAP Settings For A Domain
To see your setting, do the following - replacing with domainname with the domain in question.
zmprov gd domainname | egrep -i 'ldap|gal'
Zimlets
Location Of Zimlets
Zimlets should be already located on the zimbra server in one of these directories:
/opt/zimbra/zimlets/ /opt/zimbra/zimlets-admin-extra/ /opt/zimbra/zimlets-extra/ /opt/zimbra/zimlets-experimental/ /opt/zimbra/zimlets-network/
How To Deploy Zimlets
To deploy a zimlet, simply cd to the directory where the zimlet is located and issue this command:
/opt/zimbra/bin/zmzimletctl deploy <zimlet_name>
Archive & Discovery
References for A&D
The two main A&D references are :
http://wiki.zimbra.com/index.php?title=ZAD
http://www.zimbra.com/pdf/Zimbra%20Archiving%20and%20Discovery%20Release%20Notes.pdf
Multi-Server & New Mailstore A&D setup
See below for a very rough draft document I made for multi-server / new mailstore A&D setups.
http://wiki.zimbra.com/index.php?title=Ajcody-Notes-AD-mailstore
I also created a RFE for documentation on this.
http://bugzilla.zimbra.com/show_bug.cgi?id=25135
Finding The NO_SUCH_BLOB Errors
It would be good to review this wiki page before you start any "changes":
http://wiki.zimbra.com/index.php?title=Account_mailbox_database_structure
And great forum posting:
http://www.zimbra.com/forums/administrators/19811-solved-missing-blob-errors-zimbra-4-5-10-a.html
Check Permissions First
If you recently did a server move or similar type of operation, make sure zimbra:zimbra permissions are applied correctly to store directory. You could run the following to double check permissions (as root):
/opt/zimbra/libexec/zmfixperms --verbose --extended
Perl To Find MailboxID's That Have Problem
And down and dirty way to see what mailboxes have this error:
grep -B2 NO_SUCH_BLOB /opt/zimbra/log/mailbox.lo* |grep mailbox= |sed -r 's/.*mailbox=([0-9]*).*$/\1/' |sort -u
Mapping MailboxId to User Account
Remember, mailboxId is unique to the mailstore - it's not a global user variable that's unique.
Stolen from forum post:
http://www.zimbra.com/forums/administrators/16885-mailbox_id-email-address.html
su - zimbra mysql use zimbra; select comment from mailbox where id=257; And you'll get back: +----------------------+ | comment | +----------------------+ | user@domain.com | +----------------------+ 1 row in set (0.00 sec) If you wanted to see all the info for that account: select * from mailbox where id=257; And you'll get back: +-----+----------+--------------------------------------+-----------------+--------------------+---------------+-----------------+-------------------+---------------+---------------+----------------+----------------------+------------------+--------------+--------------------+ | id | group_id | account_id | index_volume_id | item_id_checkpoint | contact_count | size_checkpoint | change_checkpoint | tracking_sync | tracking_imap | last_backup_at | comment | last_soap_access | new_messages | idx_deferred_count | +-----+----------+--------------------------------------+-----------------+--------------------+---------------+-----------------+-------------------+---------------+---------------+----------------+----------------------+------------------+--------------+--------------------+ | 257 | 71 | xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxxxxxxx | 2 | 257 | 0 | 1832 | 100 | 0 | 0 | NULL | user@domain.com | 1209652908 | 0 | 0 | +-----+----------+--------------------------------------+-----------------+--------------------+---------------+-----------------+-------------------+---------------+---------------+----------------+----------------------+------------------+--------------+--------------------+ 1 row in set (0.00 sec)
Zmblobchk for 5.0.6+ Systems
On 5.0.6+ there's a script in /opt/zimbra/bin called zmblobchk . This will list out information about the NO_SUCH_BLOB errors.
On pre-5.0.6 systems, ask support for zmblobchk.jar file. To run it:
su - zimbra ls /opt/zimbra/zmblobchk.jar zmblobchk.jar java -jar zmblobchk.jar Retrieving volume information Retrieving mboxgroup list Spooling item list to /tmp/mailitems53650.lst Retrieving items from mboxgroup1 Retrieving items from mboxgroup2 Retrieving items from mboxgroup3 Retrieving items from mboxgroup4 Retrieving items from mboxgroup5 Retrieving items from mboxgroup6 Retrieving items from mboxgroup7 Retrieving items from mboxgroup8 Retrieving items from mboxgroup9 Retrieving items from mboxgroup10 / Processing BLOB store - /tmp/mailitems53650.lst: size 28141 Processed 329 items in 1223ms No inconsistencies found
If the "Processing BLOB store" part errors out with Java out of memory, you could try adjusting the variable for it. The default is set to 30%. This would require a mailstore/jetty restart though [zmmailboxdctl restart]. If the mailstore doesn't come back up, check /opt/zimbra/log/zmmailbox.log - you most likely exceeded the threshold for your box. Remember, this is a percentage and a 32bit machine with more than 4GB can cause issues with this setting since it could try to allocate more than it actually can to the thread.
# ZCS 5.0 and later $ zmlocalconfig -e mailboxd_java_heap_memory_percent=40
And zmmailboxdctl restart when it's appropriate.
BLOB Issue Script
I've commented out the two delete lines and also the "$msghash &= $fmask;" one.
If you're using HSM, this script isn't smart enough to detect it. It will flag and remove entries that are in the HSM store because it doesn't see them in the primary store. You can modify the output though to use the actual path to the HSM store. I had one case where this was the case and copying the missing blobs to the HSM path worked.
Comment from other support staff member on a case that used this.
Once you've gotten a list of the 'NOT OK' files generated by this tool, you'll want to check the store directories to see if the message blobs exist with an incorrect change number. That's the second number in the filename. If the first number matches the ID it's looking for but the second does not, you can rename the file to the new change number and get the message blob back without losing data. If not, then the blob is completely missing. If that's the case, we should try to figure out what happened to it by going through the mailbox.log files. You should be able to grep for the missing message ID and the user's account ID to find a log entry showing something happening to it. It's possible it was deleted and the deletion was never written to mysql for some reason.
#!/usr/bin/perl # This script compatible with Zimbra version 4.5.x only. Do not use with any other version. # OK, there's 2 MAILBOX_*_BITS values in the VOLUME table. # Take the mailbox ID, right-shift it by MAILBOX_BITS, and take the lowest MAILBOX_GROUP_BITS of the result. # That's your mailbox hash. # Take the message ID, right-shift it by FILE_BITS, and take the lowest FILE_GROUP_BITS of the result. That's your msgid hash. # I think. # <mbx-hash>/<mbx-id>/msg/<msgid-hash>/<msgid>-<mod_content>.msg my ($fbits, $fgbits, $mbits, $mgbits, $basepath) = split (' ',`echo "select file_bits,file_group_bits,mailbox_bits,mailbox_group_bits,path from volume where type='1'" | mysql -N zimbra`); my $ARGV = shift @ARGV; chomp $basepath; my $mbmask = sprintf "1" x $mgbits; my $fmask = sprintf "1" x $fgbits; foreach (`echo "select id, group_id, account_id, comment from mailbox where comment = '$ARGV'" | mysql -N zimbra`) { chomp; my $path = "$basepath/"; my ($id, $grid, $aid, $nm) = (split); my $mbhash = $id >> $mbits; $mbhash &= $mbmask; $path .= $mbhash."/".$id."/msg/"; foreach my $msgstuff (`echo "select id, mod_content, type from mail_item where blob_digest is not null and mailbox_id=${id};" | mysql -N mboxgroup${grid}`) { chomp $msgstuff; if ($msgstuff eq "") {next;} my ($msgid, $modContent, $type) = split (' ',$msgstuff); my $msghash = $msgid >> $fbits; # $msghash &= $fmask; my $nm = $msgid; if ($modContent) {$nm .= "-$modContent";} my $npath = $path.$msghash."/".$nm.".msg"; if (-e $npath) {print $npath."\t\tOK\n";} else { print $npath."\t\tNOT OK\n"; # not ok, remove the entry from the database so it is not a nuisance print "Delete from mail_item where MSGID is ${msgid} and MAILBOXID is ${id} \n"; print "Uncomment line below me in script to have me delete. \n"; # `echo "delete from mail_item where id=${msgid} and mailbox_id=${id}" | mysql -N mboxgroup${grid}`; if ($type eq "11") { print "Delete from appointment where MSGID is ${msgid} and MAILBOX_ID is ${id} \n"; print "Uncomment line below me in script to have me delete. \n"; # `echo "delete from appointment where item_id=${msgid} and mailbox_id=${id}" | mysql -N mboxgroup${grid}`; } } # print $npath."\n"; } }
If this doesn't run, make sure perl is installed and in the correct path.
You run this as the zimbra user.
What I did was the following.
Copied the script to /opt/zimbra/bin/ as root and called it blob-check.pl . chmod +x /opt/zimbra/bin/blob-check.pl
su - zimbra mkdir /tmp/BLOB-CHECKS for i in `zmprov gaa`; do blob-check.pl $i | grep "NOT OK" >> /tmp/BLOB-CHECKS/$i.txt; done ls -la /tmp/BLOB-CHECKS
The ls will help identify the more troubled accounts. You'll end up with something like this:
[zimbra@mail3 ~]$ ls -la /tmp/BLOB-CHECKS/ total 16 drwxr-x--- 2 zimbra zimbra 4096 Jul 10 23:40 . drwxrwxrwt 8 root root 4096 Jul 10 23:46 .. -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 admin@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 ajcody2@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 49 Jul 10 23:40 ajcody@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 forward@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 ham.bidiob2mm@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 secondary@secondary.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 spam.rormmtcyy@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 wiki@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 wiki@secondary.internal.homeunix.com.txt
And the output of one that shows some size:
$ cat /tmp/BLOB-CHECKS/ajcody@mail3.internal.homeunix.com.txt /opt/zimbra/store/0/3/msg/0/360-6070.msg NOT OK
Remember this from above, "you'll want to check the store directories to see if the message blobs exist with an incorrect change number. That's the second number in the filename. If the first number matches the ID it's looking for but the second does not, you can rename the file to the new change number and get the message blob back without losing data."
One can uncomment the 2 lines in the script with delete and run the script again. This will remove the reference to the blob.
To re-index the users mailbox [this can take start, cancel, status]:
zmprov rim user@domainname start
Running the script again shouldn't show "NOT OK" lines.
FYI - I still need to gather more information about this situation and how to guide one to make a decision to delete, re-index, and so forth.
BLOB Script To Copy From Restore
You might need to use these variables with the restore command. I need to find a way for someone to figure out what restore to goto for a particular blob:
-restoreToTime <arg> - Replay the redo logs until the time specified.
-restoreToIncreLabel <arg> - Replay redo logs up to and including this incremental backup.
-restoreToRedoSeq <arg> - Replay up to and including this redo log sequence
-br - Replays the redo logs in backup only, which excludes archived and current redo logs of the system.
-rf - Restores to the full backup only, does not include any incremental backups since that backup
This is a walk through to test the proceedure.
$ zmrestore -ca -a ajcody@mail3.internal.homeunix.com -pre restored- $ zmprov gmi restored-ajcody@mail3.internal.homeunix.com mailboxId: 12 quotaUsed: 251513 $ zmprov gmi ajcody@mail3.internal.homeunix.com mailboxId: 3 quotaUsed: 251512 $ diff /opt/zimbra/store/0/12/msg/0/257-25.msg /opt/zimbra/store/0/3/msg/0/257-25.msg $ rm /opt/zimbra/store/0/3/msg/0/257-25.msg rm: remove regular file `/opt/zimbra/store/0/3/msg/0/257-25.msg'? y $ cp /opt/zimbra/store/0/12/msg/0/257-25.msg /opt/zimbra/store/0/3/msg/0/257-25.msg $ grep Subject /opt/zimbra/store/0/3/msg/0/257-25.msg Log into main account and confirm I can see/use that mail.
The script to look at the blob-check.pl output and do the copies after a redirected restore is here:
#!/bin/bash #START OF VARIABLES# #CHANGEME to existing user account USER=ajcody@mail3.internal.homeunix.com #CHANGEME to the restored account name # zmrestore -ca -a ajcody@mail3.internal.homeunix.com -pre restored- RESTOREDUSER=restored-ajcody@mail3.internal.homeunix.com # To get FILE, blob-check.pl ajcody@mail3.internal.homeunix.com > /tmp/BLOB-CHECKS/ajcody.txt # blob-check.pl located at http://wiki.zimbra.com/index.php?title=Ajcody-Notes#BLOB_Issue_Script FILE=/tmp/BLOB-CHECKS/ajcody.txt #CHANGEME #CHANGEME, if needed, to the mailstore. /opt/zimbra/store is default STOREDIR=/opt/zimbra/store #END VARIABLES# USERUID=`zmprov gmi $USER | grep mailboxId | cut -f2 -d: | cut -c 2-10` USERGID=`perl -e 'print $USERUID >> 12 ; print "\n"'` USERPATH=`echo $USERGID/\$USERUID` RESTOREDUID=`zmprov gmi $RESTOREDUSER | grep mailboxId | cut -f2 -d: | cut -c 2-10` RESTOREDGID=`perl -e 'print $RESTOREDUID >> 12 ; print "\n"'` RESTOREDPATH=`echo $RESTOREDGID/\$RESTOREDUID` echo USERUID $USERUID echo USERGID $USERGID echo USERPATH $USERPATH echo RESTOREDUID $RESTOREDUID echo RESTOREDGID $RESTOREDGID echo RESTOREDPATH $RESTOREDPATH cd $STOREDIR echo "Run ECHO first to confirm copies look right and then uncomment COPY in script" for ROOTBLOBPATH in `cat $FILE | grep "NOT OK" | cut -f7-9 -d/ | cut -f1` do # Comment out echo and uncomment copy after dry run echo $RESTOREDPATH/$ROOTBLOBPATH $USERPATH/$ROOTBLOBPATH #cp -uv $RESTOREDPATH/$ROOTBLOBPATH $USERPATH/$ROOTBLOBPATH done
Migration Issues
IMAPSYNC with admin login
Reference - http://wiki.zimbra.com/index.php?title=User_Migration
imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates \ --host1 server.gtds.lan --user1 yourAccount --password1 yourPassword \ --user2 yourZimbraAccount --authuser2 admin --password2 adminZimbraPassword --authmech2 LOGIN
I found this description in one of the imapsync files:
"You may authenticate as one user (typically an admin user), but be authorized as someone else, which means you don't need to know every user's personal password. Specify --authuser1 "adminuser" to enable this on host1. In this case, --authmech1 PLAIN will be used, but otherwise, --authmech1 CRAM-MD5 is the default. Same behavior with the --authuser2 option."
Mailing Lists And Mailman
If you are planning to use a distro list of over 60K members, you may be inclined to use a dedicated machine for this purpose. According to bug 19153, you have 2 options: "set up mailing list manager or change the value of virtual_alias_expansion_limit as a customization. We have never tested with > 1000 so this should be done carefully, and will pound LDAP for any messages with lots of recipients." Mailman integration is an option. There is a forum thread on how to integrate zcs and mailman. This would be the preferred fix to your issue, also noted in the bug report. We probably won't see Mailman integration in ZCS at least for another year or so, not until 6.0
Restricting Who Can Send To Mailing List
http://wiki.zimbra.com/index.php?title=RestrictPostfixRecipients
Problems Resolving Virtual Aliases For Members Of Large Distribution Lists
http://bugzilla.zimbra.com/show_bug.cgi?id=19153
Mailman Configuration
http://wiki.zimbra.com/index.php?title=ZCS_Mailman_Integration
http://bugzilla.zimbra.com/show_bug.cgi?id=8968
http://www.zimbra.com/forums/administrators/1380-solved-zimbra-mailman-howto.html