Ajcody-Server-Topics

Server Topics

Actual Server Topics Homepage

Please see Ajcody-Server-Topics

Server Move

Please see:

http://wiki.zimbra.com/index.php?title=Ajcody-Notes-ServerMove

Backup Plans And Cheap HA/DR Secondary Server

Please see:

http://wiki.zimbra.com/index.php?title=Ajcody-Notes-BackupPlans

Hostname resolution issues and testing commands

Allot of failed installs are because administrators are skipping the steps to make sure resolution is working.

Check your /etc/hosts file. You should have the localhost similar to the one below. Your ZCS server name should be replaced with the zimbra.test.com and zimbra fields. The IP address should be the IP address bound to the network interface [ip addr or ifconfig -a to confirm].

127.0.0.1	localhost.localdomain	localhost
192.168.0.1	zimbra.test.com         zimbra

If your ZCS server is behind a firewall or is being NAT'd. Make sure that resolution for the hostname on the ZCS server returns the internal IP information rather than the external IP. To test:

hostname -f
zimbra.test.com

or

domainname -f
zimbra.test.com

As long as one of the above returns the full hostname, use the command for the following. The MX lookups depend on how you are doing your mail domain and server hostname.

host `hostname -f`
 zimbra.test.com has address 192.168.0.1
host -t MX `domainname -f`
 zimbra.test.com mail is handled by 10 zimbra.test.com.
host -t MX `domainname -d`
   test.com mail is handled by 10 zimbra.test.com.

You should also have reverse records (PTR) as well. Replace 192.168.0.1 using your internal ip address, it should return something like:

host 192.168.0.1
1.0.168.192.in-addr.arpa domain name pointer zimbra.test.com

Please review wiki page on split-DNS :

http://wiki.zimbra.com/index.php?title=Split_DNS

and also the variable use of lmtp_host_lookup .

http://bugzilla.zimbra.com/show_bug.cgi?id=27988

What's my time and timezone?

I wrote up the steps to make sure the ZCS server is using the correct time and timezone here:

http://wiki.zimbra.com/index.php?title=Time_Zones_in_ZCS#The_server_OS

Disable LDAP Replica

This is a work in progress, please don't use for production servers at this time.

References:

http://wiki.zimbra.com/index.php?title=Promoting_Replica_to_LDAP_Master

http://www.zimbra.com/docs/os/latest/multi_server_install/LDAP%20Replication.6.1.html

For each server that you want to change:

Stop the Zimbra services on the server, zmcontrol stop.

Check the existing ldap_url value.

zmlocalconfig | grep ldap_url

Update the ldap_url value. Remove the replica LDAP server URL, below assumes you only had one replica.

zmlocalconfig -e ldap_url=”ldap:// ”

If other replica's exist, then the list typed is like:

zmlocalconfig -e ldap_url=”ldap:// ldap:// ldap:// ” 

The hosts are tried in the order listed. The master URL must always be included and is listed last.

Additional Steps for MTA hosts.

After updating the ldap_url, rerun /opt/zimbra/libexe/zmmtainit. This rewrites the Postfix configuration with the updated ldap_url.

To stop the ldap service running on the ldap replica server.

su - zimbra
ldap stop

To now disable ldap from running on the old ldap replica. The - in front of zimbraServiceEnabled is [off], rather than a + for [on].

zmprov ms  -- -zimbraServiceEnabled ldap

Public Service Hostname

variable is zimbraPublicServiceHostname , as referenced in /opt/zimbra/openldap/etc/openldap/schema/zimbra.schema

Bugs about the url's for shares that use spaces (%20), the redirection drops the %20

http://bugzilla.zimbra.com/show_bug.cgi?id=27788

http://bugzilla.zimbra.com/show_bug.cgi?id=27477

Domain Rename Issues

We have a command to rename a "domain" and the resources within it - unfortunately it doesn't handle ALL of the different components for various reasons. Specifically Documents and Calendars [more down below].

Review this RFE that was done when they implemented the renameDomain command.

http://bugzilla.zimbra.com/show_bug.cgi?id=7655

Note comment 21 & 22 (confirming what QA'd). The syntax is

zmprov -l rd testold.com testnew.com

Problems that arise with Documents are explained here, comment #1 has workaround.

http://bugzilla.zimbra.com/show_bug.cgi?id=25873

Now, the very unfortunate part about calendars with a domainname change/move.

"Ability to change a username globally on all appointments"

http://bugzilla.zimbra.com/show_bug.cgi?id=26736

"Cant edit calendar entries after renameDomain"

http://bugzilla.zimbra.com/show_bug.cgi?id=27707

The root issue here's seems to be more about the calendar standards and practices with the use of the "Organizer" field and notifications. You'll see 3 "work arounds" in comment 2 of bug 26736.

Export/Importing of the calendar data is shown here:

http://wiki.zimbra.com/index.php?title=User_Migration

This part specifically :

http://wiki.zimbra.com/index.php?title=User_Migration#Copy_Calendar_From_One_Zimbra_User_to_Another

My Gal & LDAP Settings For A Domain

To see your setting, do the following - replacing with domainname with the domain in question.

zmprov gd domainname | egrep -i 'ldap|gal'

Zimlets

Location Of Zimlets

Zimlets should be already located on the zimbra server in one of these directories:

/opt/zimbra/zimlets/
/opt/zimbra/zimlets-admin-extra/
/opt/zimbra/zimlets-extra/
/opt/zimbra/zimlets-experimental/
/opt/zimbra/zimlets-network/

How To Deploy Zimlets

To deploy a zimlet, simply cd to the directory where the zimlet is located and issue this command:

/opt/zimbra/bin/zmzimletctl deploy <zimlet_name>

Archive & Discovery

References for A&D

The two main A&D references are :

http://wiki.zimbra.com/index.php?title=ZAD

http://www.zimbra.com/pdf/Zimbra%20Archiving%20and%20Discovery%20Release%20Notes.pdf

Multi-Server & New Mailstore A&D setup

See below for a very rough draft document I made for multi-server / new mailstore A&D setups.

http://wiki.zimbra.com/index.php?title=Ajcody-Notes-AD-mailstore

I also created a RFE for documentation on this.

http://bugzilla.zimbra.com/show_bug.cgi?id=25135

Finding The NO_SUCH_BLOB Errors

It would be good to review this wiki page before you start any "changes":

http://wiki.zimbra.com/index.php?title=Account_mailbox_database_structure

And great forum posting:

http://www.zimbra.com/forums/administrators/19811-solved-missing-blob-errors-zimbra-4-5-10-a.html

Check Permissions First

If you recently did a server move or similar type of operation, make sure zimbra:zimbra permissions are applied correctly to store directory. You could run the following to double check permissions (as root):

/opt/zimbra/libexec/zmfixperms --verbose --extended

Perl To Find MailboxID's That Have Problem

And down and dirty way to see what mailboxes have this error:

grep -B2 NO_SUCH_BLOB /opt/zimbra/log/mailbox.lo* |grep mailbox= |sed -r 's/.*mailbox=([0-9]*).*$/\1/' |sort -u

Mapping MailboxId to User Account

Remember, mailboxId is unique to the mailstore - it's not a global user variable that's unique.

Stolen from forum post:

http://www.zimbra.com/forums/administrators/16885-mailbox_id-email-address.html

su - zimbra
mysql
use zimbra;
select comment from mailbox where id=257;
And you'll get back:
+----------------------+
| comment |
+----------------------+
| user@domain.com |
+----------------------+
1 row in set (0.00 sec)

If you wanted to see all the info for that account:
select * from mailbox where id=257;
And you'll get back:
+-----+----------+--------------------------------------+-----------------+--------------------+---------------+-----------------+-------------------+---------------+---------------+----------------+----------------------+------------------+--------------+--------------------+
| id | group_id | account_id | index_volume_id | item_id_checkpoint | contact_count | size_checkpoint | change_checkpoint | tracking_sync | tracking_imap | last_backup_at | comment | last_soap_access | new_messages | idx_deferred_count |
+-----+----------+--------------------------------------+-----------------+--------------------+---------------+-----------------+-------------------+---------------+---------------+----------------+----------------------+------------------+--------------+--------------------+
| 257 | 71 | xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxxxxxxx | 2 | 257 | 0 | 1832 | 100 | 0 | 0 | NULL | user@domain.com | 1209652908 | 0 | 0 |
+-----+----------+--------------------------------------+-----------------+--------------------+---------------+-----------------+-------------------+---------------+---------------+----------------+----------------------+------------------+--------------+--------------------+
1 row in set (0.00 sec)

Zmblobchk for 5.0.6+ Systems

On 5.0.6+ there's a script in /opt/zimbra/bin called zmblobchk . This will list out information about the NO_SUCH_BLOB errors.

On pre-5.0.6 systems, ask support for zmblobchk.jar file. To run it:

su - zimbra
ls /opt/zimbra/zmblobchk.jar
  zmblobchk.jar
java -jar zmblobchk.jar
Retrieving volume information
Retrieving mboxgroup list
Spooling item list to /tmp/mailitems53650.lst
Retrieving items from mboxgroup1
Retrieving items from mboxgroup2
Retrieving items from mboxgroup3
Retrieving items from mboxgroup4
Retrieving items from mboxgroup5
Retrieving items from mboxgroup6
Retrieving items from mboxgroup7
Retrieving items from mboxgroup8
Retrieving items from mboxgroup9
Retrieving items from mboxgroup10
/
Processing BLOB store
-
/tmp/mailitems53650.lst: size 28141
Processed 329 items in 1223ms
No inconsistencies found

If the "Processing BLOB store" part errors out with Java out of memory, you could try adjusting the variable for it. The default is set to 30%. This would require a mailstore/jetty restart though [zmmailboxdctl restart]. If the mailstore doesn't come back up, check /opt/zimbra/log/zmmailbox.log - you most likely exceeded the threshold for your box. Remember, this is a percentage and a 32bit machine with more than 4GB can cause issues with this setting since it could try to allocate more than it actually can to the thread.

# ZCS 5.0 and later
$ zmlocalconfig -e mailboxd_java_heap_memory_percent=40

And zmmailboxdctl restart when it's appropriate.

BLOB Issue Script

I've commented out the two delete lines and also the "$msghash &= $fmask;" one.

If you're using HSM, this script isn't smart enough to detect it. It will flag and remove entries that are in the HSM store becuase it doesn't see them in the primary store. 'You can modify the output though to use the actual path to the HSM store. I had one case where this was the case and copying the missing blobs to the HSM path worked.'

Comment from other support staff member on a case that used this.

Once you've gotten a list of the 'NOT OK' files generated by this tool, you'll want to check the store directories to see if the message blobs exist with an incorrect change number. That's the second number in the filename. If the first number matches the ID it's looking for but the second does not, you can rename the file to the new change number and get the message blob back without losing data. If not, then the blob is completely missing. If that's the case, we should try to figure out what happened to it by going through the mailbox.log files. You should be able to grep for the missing message ID and the user's account ID to find a log entry showing something happening to it. It's possible it was deleted and the deletion was never written to mysql for some reason.

#!/usr/bin/perl

# This script compatible with Zimbra version 4.5.x only.  Do not use with any other version.
# OK, there's 2 MAILBOX_*_BITS values in the VOLUME table.
# Take the mailbox ID, right-shift it by MAILBOX_BITS, and take the lowest MAILBOX_GROUP_BITS of the result.
# That's your mailbox hash.
# Take the message ID, right-shift it by FILE_BITS, and take the lowest FILE_GROUP_BITS of the result.  That's your msgid hash.
# I think.
# <mbx-hash>/<mbx-id>/msg/<msgid-hash>/<msgid>-<mod_content>.msg

my ($fbits, $fgbits, $mbits, $mgbits, $basepath) = split (' ',`echo "select file_bits,file_group_bits,mailbox_bits,mailbox_group_bits,path from volume where type='1'" | mysql -N zimbra`);
my $ARGV = shift @ARGV;

chomp $basepath;

my $mbmask = sprintf "1" x $mgbits;
my $fmask = sprintf "1" x $fgbits;

foreach (`echo "select id, group_id, account_id, comment from mailbox where comment = '$ARGV'" | mysql -N zimbra`) {
	chomp;
	my $path = "$basepath/";
	my ($id, $grid, $aid, $nm) = (split);
	my $mbhash = $id >> $mbits;
	$mbhash &= $mbmask;
	$path .= $mbhash."/".$id."/msg/";
	foreach my $msgstuff (`echo "select id, mod_content, type from mail_item where blob_digest is not null and mailbox_id=${id};" | mysql -N mboxgroup${grid}`) {
		chomp $msgstuff;
		if ($msgstuff eq "") {next;}
		my ($msgid, $modContent, $type) = split (' ',$msgstuff);
		my $msghash = $msgid >> $fbits;
#		$msghash &= $fmask;
		my $nm = $msgid;
		if ($modContent) {$nm .= "-$modContent";}
		my $npath = $path.$msghash."/".$nm.".msg";
		if (-e $npath) {print $npath."\t\tOK\n";} 
		else {
		   print $npath."\t\tNOT OK\n";
		   # not ok, remove the entry from the database so it is not a nuisance
		   print "Delete from mail_item where MSGID is ${msgid} and MAILBOXID is ${id} \n";
		   print "Uncomment line below me in script to have me delete. \n";
#		   `echo "delete from mail_item where id=${msgid} and mailbox_id=${id}" | mysql -N mboxgroup${grid}`;
		   if ($type eq "11") {
	       print "Delete from appointment where MSGID is ${msgid} and MAILBOX_ID is ${id} \n";
	       print "Uncomment line below me in script to have me delete. \n";
#              `echo "delete from appointment where item_id=${msgid} and mailbox_id=${id}" | mysql -N mboxgroup${grid}`;
		   }
		}
#		print $npath."\n";
	}
}

If this doesn't run, make sure perl is installed and in the correct path.

You run this as the zimbra user.

What I did was the following.

Copied the script to /opt/zimbra/bin/ as root and called it blob-check.pl . chmod +x /opt/zimbra/bin/blob-check.pl

su - zimbra
mkdir /tmp/BLOB-CHECKS
for i in `zmprov gaa`; do blob-check.pl $i | grep "NOT OK" >> /tmp/BLOB-CHECKS/$i.txt; done
ls -la /tmp/BLOB-CHECKS

The ls will help identify the more troubled accounts. You'll end up with something like this:

[zimbra@mail3 ~]$ ls -la /tmp/BLOB-CHECKS/
total 16
drwxr-x--- 2 zimbra zimbra 4096 Jul 10 23:40 .
drwxrwxrwt 8 root   root   4096 Jul 10 23:46 ..
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 admin@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 ajcody2@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra   49 Jul 10 23:40 ajcody@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 forward@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 ham.bidiob2mm@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 secondary@secondary.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 spam.rormmtcyy@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 wiki@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 wiki@secondary.internal.homeunix.com.txt

And the output of one that shows some size:

$ cat /tmp/BLOB-CHECKS/ajcody@mail3.internal.homeunix.com.txt
/opt/zimbra/store/0/3/msg/0/360-6070.msg		NOT OK

Remember this from above, "you'll want to check the store directories to see if the message blobs exist with an incorrect change number. That's the second number in the filename. If the first number matches the ID it's looking for but the second does not, you can rename the file to the new change number and get the message blob back without losing data."

One can uncomment the 2 lines in the script with delete and run the script again. This will remove the reference to the blob.

To re-index the users mailbox [this can take start, cancel, status]:

zmprov rim user@domainname start

Running the script again shouldn't show "NOT OK" lines.

FYI - I still need to gather more information about this situation and how to guide one to make a decision to delete, re-index, and so forth.

BLOB Script To Copy From Restore

You might need to use these variables with the restore command. I need to find a way for someone to figure out what restore to goto for a particular blob:

-restoreToTime <arg> - Replay the redo logs until the time specified.

-restoreToIncreLabel <arg> - Replay redo logs up to and including this incremental backup.

-restoreToRedoSeq <arg> - Replay up to and including this redo log sequence

-br - Replays the redo logs in backup only, which excludes archived and current redo logs of the system.

-rf - Restores to the full backup only, does not include any incremental backups since that backup

This is a walk through to test the proceedure.

$ zmrestore -ca -a ajcody@mail3.internal.homeunix.com -pre restored-
$ zmprov gmi restored-ajcody@mail3.internal.homeunix.com
mailboxId: 12
quotaUsed: 251513
$ zmprov gmi ajcody@mail3.internal.homeunix.com
mailboxId: 3
quotaUsed: 251512
$ diff /opt/zimbra/store/0/12/msg/0/257-25.msg /opt/zimbra/store/0/3/msg/0/257-25.msg
$ rm /opt/zimbra/store/0/3/msg/0/257-25.msg
rm: remove regular file `/opt/zimbra/store/0/3/msg/0/257-25.msg'? y
$ cp /opt/zimbra/store/0/12/msg/0/257-25.msg /opt/zimbra/store/0/3/msg/0/257-25.msg
$ grep Subject /opt/zimbra/store/0/3/msg/0/257-25.msg
Log into main account and confirm I can see/use that mail.

The script to look at the blob-check.pl output and do the copies after a redirected restore is here:

#!/bin/bash

#START OF VARIABLES#
#CHANGEME to existing user account
USER=ajcody@mail3.internal.homeunix.com

#CHANGEME to the restored account name
# zmrestore -ca -a ajcody@mail3.internal.homeunix.com -pre restored-
RESTOREDUSER=restored-ajcody@mail3.internal.homeunix.com

# To get FILE, blob-check.pl ajcody@mail3.internal.homeunix.com > /tmp/BLOB-CHECKS/ajcody.txt
# blob-check.pl located at http://wiki.zimbra.com/index.php?title=Ajcody-Notes#BLOB_Issue_Script
FILE=/tmp/BLOB-CHECKS/ajcody.txt #CHANGEME

#CHANGEME, if needed, to the mailstore. /opt/zimbra/store is default
STOREDIR=/opt/zimbra/store

#END VARIABLES#

USERUID=`zmprov gmi $USER | grep mailboxId | cut -f2 -d: | cut -c 2-10`
USERGID=`perl -e 'print $USERUID >> 12 ; print "\n"'`
USERPATH=`echo $USERGID/\$USERUID`

RESTOREDUID=`zmprov gmi $RESTOREDUSER | grep mailboxId | cut -f2 -d: | cut -c 2-10`
RESTOREDGID=`perl -e 'print $RESTOREDUID >> 12 ; print "\n"'`
RESTOREDPATH=`echo $RESTOREDGID/\$RESTOREDUID`

echo USERUID $USERUID
echo USERGID $USERGID
echo USERPATH $USERPATH
echo RESTOREDUID $RESTOREDUID
echo RESTOREDGID $RESTOREDGID
echo RESTOREDPATH $RESTOREDPATH

cd $STOREDIR
echo "Run ECHO first to confirm copies look right and then uncomment COPY in script"
for ROOTBLOBPATH in `cat $FILE | grep "NOT OK" | cut -f7-9 -d/ | cut -f1`
do
# Comment out echo and uncomment copy after dry run
echo $RESTOREDPATH/$ROOTBLOBPATH $USERPATH/$ROOTBLOBPATH
#cp -uv $RESTOREDPATH/$ROOTBLOBPATH $USERPATH/$ROOTBLOBPATH
done

Migration Issues

IMAPSYNC with admin login

Reference - http://wiki.zimbra.com/index.php?title=User_Migration

imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates \
--host1 server.gtds.lan --user1 yourAccount --password1 yourPassword \
--user2 yourZimbraAccount --authuser2 admin --password2 adminZimbraPassword --authmech2 LOGIN
I found this description in one of the imapsync files:

"You may authenticate as one user (typically an admin user), but be authorized as someone else, which means you don't need to know every user's personal password. Specify --authuser1 "adminuser" to enable this on host1. In this case, --authmech1 PLAIN will be used, but otherwise, --authmech1 CRAM-MD5 is the default. Same behavior with the --authuser2 option."

Mailing Lists And Mailman

If you are planning to use a distro list of over 60K members, you may be inclined to use a dedicated machine for this purpose. According to bug 19153, you have 2 options: "set up mailing list manager or change the value of virtual_alias_expansion_limit as a customization. We have never tested with > 1000 so this should be done carefully, and will pound LDAP for any messages with lots of recipients." Mailman integration is an option. There is a forum thread on how to integrate zcs and mailman. This would be the preferred fix to your issue, also noted in the bug report. We probably won't see Mailman integration in ZCS at least for another year or so, not until 6.0

Restricting Who Can Send To Mailing List

http://wiki.zimbra.com/index.php?title=RestrictPostfixRecipients

Problems Resolving Virtual Aliases For Members Of Large Distribution Lists

http://bugzilla.zimbra.com/show_bug.cgi?id=19153

Mailman Configuration

http://wiki.zimbra.com/index.php?title=ZCS_Mailman_Integration

http://bugzilla.zimbra.com/show_bug.cgi?id=8968

http://www.zimbra.com/forums/administrators/1380-solved-zimbra-mailman-howto.html

Jump to: navigation, search