Ajcody-Notes: Difference between revisions

No edit summary
Line 272: Line 272:


=Apple/Mac Issues=
=Apple/Mac Issues=
Please see dedicated page for these issues:
http://wiki.zimbra.com/index.php?title=Ajcody-Apple-Mac-Issues


==AddressBook==
==AddressBook==

Revision as of 03:40, 16 July 2008

General Notes

Dos/Unix New Lines (License issues)

vi file and then

:set list

Zimbra Variable

Ways to check on different zimbra variables.

su - zimbra
zmlocalconfig -i
zmlocalconfig -d
cd /opt/zimbra/conf/attrs
more zimbra-attrs.xml
cd /opt/zimbra/openldap/etc/openldap/schema/
more zimbra.schema

Zimbra Contact Fields For CSV Import Mapping

Please see /opt/zimbra/conf/zimbra-contact-fields.xml

What's My Server Like

To dump out your server details, these commands are useful.

zmprov gs yourdomainname
zmprov gacf
zmlocalconfig
zmdumpenv

Another good document to review for "odd performance" issues is the large server wiki.

http://wiki.zimbra.com/index.php?title=Performance_Tuning_Guidelines_for_Large_Deployments

Debug via Browser

See http://wiki.zimbra.com/index.php?title=Web_Client_URL_Tricks&redirect=no

What's up with all the logs?

Logging page resources:

http://www.zimbra.com/docs/ne/latest/administration_guide/9_Monitoring.14.1.html

http://wiki.zimbra.com/index.php?title=Server_Monitoring

http://wiki.zimbra.com/index.php?title=Monitoring_Zimbra_Servers

http://wiki.zimbra.com/index.php?title=Log_Files

http://wiki.zimbra.com/index.php?title=Unresponsive_Server_Troubleshooting

The Bread And Butter Logs

/opt/zimbra/log/mailbox.log - This log is a mailboxd log4j server log containing the logs from the mailbox server. This includes the mailbox store, LMTP server, IMAP and POP servers, and Index server.

/opt/zimbra/log/zmmailbox.out - Mailstore not coming up and nothing is being logged in mailbox.log, check here.

/var/log/zimbra.log - The Zimbra syslog details the activities of the Zimbra MTA (Postfix, amavisd, antispam, antivirus), Logger, Authentication (cyrus-sasl), and Directory (OpenLDAP). By default LDAP activity is logged to Zimbra.log.

/opt/zimbra/db/data/YOURHOSTNAME.err - This is the message store database error log.

/var/log/messages

Logging Increase

Depending on the ZCS version the below will clear all the per account loggers on zimbra reboot. IMAP is used for these examples.

zmprov aal user@domain.com zimbra.imap debug
zmprov ral user@domain.com zimbra.imap

If you want to do it globally edit /opt/zimbra/conf/log4j.properties.in (add a line at the end that's similar but like log4j.logger.zimbra.imap=DEBUG). No restart of any service is needed.

Log4J Variables

"zimbra.misc" logger. For all events that don't have a specific-catagory.

"zimbra.index" logger. For indexing-related events.

"zimbra.index.lucene" logger. For logging of low-level lucene operations (debug-level only)

"zimbra.searchstat" logger. For logging statistics about what kinds of searches are run

"zimbra.redolog" logger. For redolog-releated events.

"zimbra.lmtp" logger. For LMTP-related events.

"zimbra.smtp" logger. For SMTP-related events.

"zimbra.nio" logger. For NIO-related events.

"zimbra.imap" logger. For IMAP-related events.

"zimbra.imap" logger. For POP-related events.

"zimbra.mailbox" logger. For mailbox-related events.

"zimbra.calendar" logger. For calendar-related events.

"zimbra.im" logger. For instant messaging-related events.

"zimbra.account" logger. For account-related events.

"zimbra.gal" logger. For account-related events.

"zimbra.ldap" logger. For ldap-related events.

"zimbra.security" logger. For security-related events

"zimbra.soap" logger. For soap-related events

"zimbra.test" logger. For testing-related events

"zimbra.sqltrace" logger. For tracing SQL statements sent to the database

"zimbra.dbconn" logger. For tracing database connections

"zimbra.perf" logger. For logging performance statistics

"zimbra.cache" logger. For tracing object cache activity

"zimbra.filter" logger. For filter-related logs.

"zimbra.session" logger. For session- and notification-related logs.

"zimbra.backup" logger. For backup/restore-related logs.

"zimbra.system" logger. For startup/shutdown and other related logs.

"zimbra.sync" logger. For sync client interface logs.

"zimbra.synctrace" logger. For sync client interface logs.

"zimbra.syncstate" logger. For sync client interface logs.

"zimbra.wbxml" logger. For wbxml client interface logs.

"zimbra.extensions" logger. For logging extension loading related info.

"zimbra.zimlet" logger. For logging zimlet related info.

"zimbra.wiki" logger. For wiki and document sharing.

"zimbra.op" logger. Logs server operations

"zimbra.dav" logger. Logs dav operations

"zimbra.io" logger. Logs file IO operations.

"zimbra.datasource" logger. Logs data source operations.

remote management.

public static final Log rmgmt = LogFactory.getLog("zimbra.rmgmt");

"zimbra.webclient" logger. Logs ZimbraWebClient servlet and jsp operations.

"zimbra.scheduler" logger. Logs scheduled task operations.

"zimbra.store" logger. Logs filesystem storage operations.

"zimbra.fb" logger. Logs free/busy operations.

"zimbra.purge" logger. Logs mailbox purge operations.

"zimbra.mailop" logger. Logs changes to items in the mailbox.

User , Mailbox ID's, And Who Is What

ZimbraID [UserID] is system wide.

MailboxID is per server store.

To get the ZimbraID:

$ zmprov ga user@domain.com | grep -i zimbraid
zimbraId: aeca260b-6faf-4cfe-b407-7673748aabf4
zimbraIdentityMaxNumEntries: 20

To get the MailboxID, get on the appropriate mailserver and:

zmprov gmi user@domain.com
mailboxId: 3
quotaUsed: 251512

or globally:

/opt/zimbra/bin/mysql -e "use zimbra; select id from mailbox where account_id = 'UserID HERE including the leading 0'"

Other details can be found here:

http://wiki.zimbra.com/index.php?title=Account_mailbox_database_structure

Restarting Jetty - ZCS 5+

As zimbra

zmmailboxdctl restart

Get Version From Ajax Client

Put the below in the search field of the Ajax client:

$set:get

What's Compiled With Postfix

Do the following (return is from my test box):

postconf -m
btree
cidr
environ
hash
ldap
mysql
pcre
proxy
regexp
static
unix

Mailing Lists And Mailman

If you are planning to use a distro list of over 60K members, you may be inclined to use a dedicated machine for this purpose. According to bug 19153, you have 2 options: "set up mailing list manager or change the value of virtual_alias_expansion_limit as a customization. We have never tested with > 1000 so this should be done carefully, and will pound LDAP for any messages with lots of recipients." Mailman integration is an option. There is a forum thread on how to integrate zcs and mailman. This would be the preferred fix to your issue, also noted in the bug report. We probably won't see Mailman integration in ZCS at least for another year or so, not until 6.0

Restricting Who Can Send To Mailing List

http://wiki.zimbra.com/index.php?title=RestrictPostfixRecipients

Problems Resolving Virtual Aliases For Members Of Large Distribution Lists

http://bugzilla.zimbra.com/show_bug.cgi?id=19153

Mailman Configuration

http://wiki.zimbra.com/index.php?title=ZCS_Mailman_Integration

http://bugzilla.zimbra.com/show_bug.cgi?id=8968

http://www.zimbra.com/forums/administrators/1380-solved-zimbra-mailman-howto.html

Server Move

Please see:

http://wiki.zimbra.com/index.php?title=Ajcody-Notes-ServerMove

Backup Plans And Cheap HA/DR Secondary Server

Please see:

http://wiki.zimbra.com/index.php?title=Ajcody-Notes-BackupPlans

General CALDAV Issues

Lightning & Thunderbird

Lightning does not support Caldav + Free/Busy

Free/Busy support is only available for the Sun Cal Server.

Maybe we could "fake" this out within our Ldap configuration. Like RFE for Apple OD:

http://bugzilla.zimbra.com/show_bug.cgi?id=26619

Address/username lookup doesn't occur within a new appointment

Unfortunately, I've not figured a way around this at this time.

Zimbra CalDav doesn't allow user to dismiss alarms from lightning

Please see bug from Mozilla:

https://bugzilla.mozilla.org/show_bug.cgi?id=432540

Apple/Mac Issues

Please see dedicated page for these issues:

http://wiki.zimbra.com/index.php?title=Ajcody-Apple-Mac-Issues

AddressBook

Logging-Debug for Addressbook

First shutdown Addressbook app.

Launch terminal app.

defaults write com.apple.addressbook ABShowDebugMenu -bool YES

Turn it off by just reversing it:

defaults write com.apple.addressbook ABShowDebugMenu -bool NO

iCal Issues

Logging-Debug for CalDAV+iCAL

If you are using CalDAV account with ZCS, please help us investigating this problem. On your Mac please do the following and gather some data.

First shutdown iCal app.

Launch terminal app.

You can turn on the debug options in iCal by:

defaults write com.apple.iCal IncludeDebugMenu 1

Turn them off by just reversing it:

defaults write com.apple.iCal IncludeDebugMenu 0

Furthermore, you can log all of the HTTP transaction to the console via:

defaults write com.apple.iCal LogHTTPActivity yes

Turn them off by just reversing it:

defaults write com.apple.iCal LogHTTPActivity no

Logging will show up in /var/log/system.log

Launch iCal app.

Select the shared folder, right click, refresh.

Not seeing Calendar's

'''DO NOT USE SPACES IN THE CALENDAR NAME'''

This is resolved for the 5.0.7 release. (Need bug # for this)

Can't see items in shared Calendar

You might be seeing this:

http://bugzilla.zimbra.com/show_bug.cgi?id=23671

Comment #30 give some instructions on a case they can reproduce.

zmprov gd <your domain name> zimbraPublicServiceHostname
zmprov gs <your server name> zimbraServiceHostname

If zimbraPublicServiceHostname is not set, or set to a different name (such as an alias of the machine or a load balancer), you are hitting this issue.

iCal & Entourage Use for Calendars - Leopard/10.5 users

Update The 5.0.6+ Connectors will no longer work with Entourage. Your only option for Calendars will be using CalDAV with iCal.App. You still will see an option about Entourage in the Connector but it will be grayed out.

Because of some core issues with Apple's SyncServices, we recommend that Entourage and iCal users use iCALv3 (Mac 10.5) configured for CALDAV. Please use the Zimbra Connector instructions to setup iCal for CALDAV.

For Entourage, this recommendation stems from the fact that Microsoft decided to use WebDAV rather than CALDAV. If MS decides later to include CALDAV support for Entourage, we'll have another option.

Please see bug for details:

http://bugzilla.zimbra.com/show_bug.cgi?id=27380

RFE to support WebDAV for Entourage:

http://bugzilla.zimbra.com/show_bug.cgi?id=24502

Can't see Calendar's after configuring Z-iSync for CALDAV - 10.5 users

You didn't follow the directions most likely. Goto iCal Preferences Accounts.

There should be NO configured accounts prior to configuring Z-iSync for CALDAV.

Remove the current Account listed in iCal and redo configuration for CALDAV with Z-iSync.

Calendar events are displaying the wrong time

Apple expects the year of 1971 for the DTSTART variable within an ICS file. There is no standard that dictates this.

Zimbra (prior to version 5.0.5) was using 1601, this is in /opt/zimbra/conf/timezones.ics .

Please see bug for more details (resolved 5.0.5):

http://bugzilla.zimbra.com/show_bug.cgi?id=22808

Alarm & Calendar Notification Issues with iCal

These issues are getting worked out for the 5.0.7 release.

Please review:

http://bugzilla.zimbra.com/show_bug.cgi?id=28057

Some more details:

http://bugzilla.zimbra.com/show_bug.cgi?id=28883

http://bugzilla.mozilla.org/show_bug.cgi?id=432540

Trick to work around address lookup for the Attendee field in iCal Events

Configure the Addressbook using one of the other recommendations on this page.

You can now drag entries from your Addressbook into the Attendee field of iCal.

Attendee Lookups for iCALv3/MacOS10.5

iCal3 [webdav] needs to use the Apple Directory Access utility for attendees (It's on 10.5). It will only work against Apple OD/CAL servers.

RFE filed to include necessary LDAP attributes to provide ical oattendee lookups

See bug [scheduled for 5.0.7]:

http://bugzilla.zimbra.com/show_bug.cgi?id=26619

Free/Busy Lookup not working as expected

The Free/Busy lookup requires auto-attendee lookup to be working.

This requires bug 26619 to be resolved for iCalv3/Mac 10.5.

Attendee Lookup for iCal/Mac 10.4

iCal.app on Mac 10.4 only uses local entries in Apples Addressbook for address/username lookup for new appointments.

Addressbook+LDAP configuration requires a copy of an "all/*" search in ldap into the local addressbook. Sync doesn't seem to work.

Please see this bug comment and the 10.4 section for more details:

http://bugzilla.zimbra.com/show_bug.cgi?id=26619#c3

iCal calendar color changing modifies calendar name and/or doesn't retain color

Fixed in 5.0.5 . See bugs for details:

http://bugzilla.zimbra.com/show_bug.cgi?id=26627

http://bugzilla.zimbra.com/show_bug.cgi?id=26625

New Calendars don't sync unless you select "all". Mac 10.4/Z-isync

Please see bug for details:

http://bugzilla.zimbra.com/show_bug.cgi?id=26653

Directory Access configuration on Macs

File that configures [system wide] for Address Book resolution

/Library/Preferences/DirectoryService/DSLDAPv3PlugInConfig.plist

Mac clients cause spamming of invitation events when they adjust calendar events

This, at it's root, is caused by other bugs listed here. Usually this will get resolved with the proper upgrades that the bugs require (ZCS 5.0.5+)

There is a RFE/Bug though that will also resolve this, by allowing the "action" of an appointment to be set.

Please see bug for details [scheduled for 5.0.7]:

http://bugzilla.zimbra.com/show_bug.cgi?id=10536

Calendar Invites can't be imported into iCal

If you look at the ics file, you might see an extra return character at the end.

You'll see this if an Outlook client send an invite to a user using a Mac with a thick client [Mail.app/Entourage].

This was resolved in the 5.0.5 release. Please see bug for more details:

http://bugzilla.zimbra.com/show_bug.cgi?id=26487

iTunes & iPhone configuration for CALDAV

Please see main wiki page on iPhone http://wiki.zimbra.com/index.php?title=IPhone

If you follow the instructions about configuring the Zimbra iSync Connector for CALDAV use, there should be no issues about sync'ing with your iPhone.

At this time, new events created on a iPhone/CALDAV setup will write the event to a local calendar in iCal.app . There's no way around this yet. It's a limitation at this time with Apple's software. You can change the events calendar assignment though later via iCal.app.

Please see Screenshot of iTunes & iPhone

Clearing iCal cache

Make and run script:

#!/bin/sh
rm -rf ~/Library/Application\ Support/SyncServices/Local/clientdata/com.apple.iCal
rm -rf ~/Library/Calendars
rm -rf ~/Library/Preferences/com.apple.iCal.plist
rm -rf ~/Library/Preferences/IcalExternalSync.plist
rm -rf ~/Library/Preferences/ByHost/com.apple.iCal.helper.*
rm -rf ~/Library/Caches/com.apple.iCal
rm -rf ~/Library/Caches/Metadata/iCal

Clearing ALL of iCal

Careful, This Will Remove All Calendar Data. Have all iSync related programs shutdown [addressbook, ical, iMail]

You will also need to redo your CalDAV account setup for the Zimbra iSync Connector. It might also be necessary to uninstall and reinstall the Zimbra Connector. Make and run script:

#!/bin/sh
rm -rf ~/Library/Application\ Support/SyncServices/Local
rm -rf ~/Library/Application\ Support/SyncLocalCopy
rm -rf ~/Library/Application\ Support/iSync/SyncLocalCopy
rm -rf ~/Library/Application\ Support/iCal
rm -rf ~/Library/Calendars
rm -rf ~/Library/Preferences/com.apple.iCal.*
rm -rf ~/Library/Preferences/IcalExternalSync.plist
rm -rf ~/Library/Preferences/ByHost/com.apple.iCal.helper.*
rm -rf ~/Library/Caches/com.apple.iCal
rm -rf ~/Library/Caches/Metadata/iCal

Use LDAP directory from ZCS for Mac Addressbook / Mail.App / iCal for Mac 10.4

This is generally for those that aren't using Z-iSync.

First, test on shell that you have ldap connectivity.

ldapsearch -x -H ldap://ZCSserver-name:389 "uid=ajcody"

That returned successfully.

Then Addressbook Preferences

LDAP  + (for new)
Name: anything really
Server: ip/hostname that was used successfully in the ldapsearch string
Search Base:  [leave blank]
Port: 389
Scope: Subtree
No auth needed
* Save

I have no idea what the "Auto Update LDAP Cards" does.

Then setup Mail.App for IMAP to the Zimbra server. Check preferences to use LDAP - it will use the Address Book configuration that you did prior and it should auto-resolve to a username when you do a new message.

iCal will only use address that are in the LOCAL addressbook when you use the attendee field for a new event.

One way around this is to do a wildcard search in the Addressbook and drag those items to the local addressbook.

Goto to Address Book, selected Directories my ldap server and put in the below for the search field:

*.

This search result can then be dragged to the "All" folder in the Group column. It creates a "Last Import" item in the Group column now - it might warn about updating entries that all ready match.

Sync To Do/Tasks Items in Leopard

RFE filed, please see http://bugzilla.zimbra.com/show_bug.cgi?id=12917

No target date, please vote for this RFE.

What's my time and timezone?

I wrote up the steps to make sure the ZCS server is using the correct time and timezone here:

http://wiki.zimbra.com/index.php?title=Time_Zones_in_ZCS#The_server_OS

Hostname resolution issues and testing commands

Allot of failed installs are because administrators are skipping the steps to make sure resolution is working.

Check your /etc/hosts file. You should have the localhost similar to the one below. Your ZCS server name should be replaced with the zimbra.test.com and zimbra fields. The IP address should be the IP address bound to the network interface [ip addr or ifconfig -a to confirm].

127.0.0.1	localhost.localdomain	localhost
192.168.0.1	zimbra.test.com         zimbra

If your ZCS server is behind a firewall or is being NAT'd. Make sure that resolution for the hostname on the ZCS server returns the internal IP information rather than the external IP. To test:

hostname -f
zimbra.test.com

or

domainname -f
zimbra.test.com

As long as one of the above returns the full hostname, use the command for the following. The MX lookups depend on how you are doing your mail domain and server hostname.

host `hostname -f`
 zimbra.test.com has address 192.168.0.1
host -t MX `domainname -f`
 zimbra.test.com mail is handled by 10 zimbra.test.com.
host -t MX `domainname -d`
   test.com mail is handled by 10 zimbra.test.com.

You should also have reverse records (PTR) as well. Replace 192.168.0.1 using your internal ip address, it should return something like:

host 192.168.0.1
1.0.168.192.in-addr.arpa domain name pointer zimbra.test.com

Please review wiki page on split-DNS :

http://wiki.zimbra.com/index.php?title=Split_DNS

and also the variable use of lmtp_host_lookup .

http://bugzilla.zimbra.com/show_bug.cgi?id=27988

Disable LDAP Replica

This is a work in progress, please don't use for production servers at this time.

References:

http://wiki.zimbra.com/index.php?title=Promoting_Replica_to_LDAP_Master

http://www.zimbra.com/docs/os/latest/multi_server_install/LDAP%20Replication.6.1.html

For each server that you want to change:

Stop the Zimbra services on the server, zmcontrol stop.

Check the existing ldap_url value.

zmlocalconfig | grep ldap_url

Update the ldap_url value. Remove the replica LDAP server URL, below assumes you only had one replica.

zmlocalconfig -e ldap_url=”ldap:// ”

If other replica's exist, then the list typed is like:

zmlocalconfig -e ldap_url=”ldap:// ldap:// ldap:// ” 

The hosts are tried in the order listed. The master URL must always be included and is listed last.

Additional Steps for MTA hosts.

After updating the ldap_url, rerun /opt/zimbra/libexe/zmmtainit. This rewrites the Postfix configuration with the updated ldap_url.

To stop the ldap service running on the ldap replica server.

su - zimbra
ldap stop

To now disable ldap from running on the old ldap replica. The - in front of zimbraServiceEnabled is [off], rather than a + for [on].

zmprov ms  -- -zimbraServiceEnabled ldap

IMAPSYNC with admin login

Reference - http://wiki.zimbra.com/index.php?title=User_Migration

imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates \
--host1 server.gtds.lan --user1 yourAccount --password1 yourPassword \
--user2 yourZimbraAccount --authuser2 admin --password2 adminZimbraPassword --authmech2 LOGIN
I found this description in one of the imapsync files:

"You may authenticate as one user (typically an admin user), but be authorized as someone else, which means you don't need to know every user's personal password. Specify --authuser1 "adminuser" to enable this on host1. In this case, --authmech1 PLAIN will be used, but otherwise, --authmech1 CRAM-MD5 is the default. Same behavior with the --authuser2 option."

ZCO Zimbra Outlook Connector

The complete troubling-shooting guide (dev info requests)

  1. Reproduce
  2. ZCO install logs
    • msiexec /i zco-installer.msi /lv 00022083-zco-install.log
  3. Get zco logs
  4. Check for core dumps - mandatory for all crashes
  5. Winhttp trace (issues where they can&amp;amp;amp;amp;amp;#39;t connect - check zco logs first). Run this on the machine with Outlook.
  6. Are there any local failures/server failures?
  7. Winmsd (dump of the local environment configuration)
    • Start Run winmsd
      • This might take awhile to bring up the application
      • From app [System Information titled] , File Export
    • What about msinfo32.exe ?
  8. External factors (add-ins, other mapi clients)
  9. Network topology (firewall, proxy, etc)
  10. Server topology (single/multi-node, clustering, nginx, 3rd party lb)
  11. Search bugzilla
  12. Search old support cases
  13. Can we get remote access?
  14. Can we get an account on the server?
  15. Can we access the account in question?
  16. Get the id of the item
    • get the original of the item from the server (mime representation)
    • get the .msg of the item from outlook
  17. Quick analysis of zco logs
    • sync - change record creation - CHANGE RECORD
    • follow soap requests, look for soap faults
    • look for the word - exception
    • follow soap traces
    • ignore MAPI_E_NOT_FOUND

Performance Issues

Sources:

http://msexchangeteam.com/archive/2007/12/17/447750.aspx

http://blogs.msdn.com/outlook/

http://support.microsoft.com/kb/940226/

http://technet.microsoft.com/en-us/library/bb738147.aspx

The skinny is this, [per Microsoft]

  • ...recommended max OST size (2GB)
  • ...we strongly recommend storing no more than 5,000 items in core folders, such as the Inbox and Sent Items folders. Creating more top-level folders, or subfolders underneath the Inbox and Sent Items folders, greatly reduces the costs associated with this index creation, so long as the number of items in any one folder does not exceed 5,000.

Public Service Hostname

variable is zimbraPublicServiceHostname , as referenced in /opt/zimbra/openldap/etc/openldap/schema/zimbra.schema

Bugs about the url's for shares that use spaces (%20), the redirection drops the %20

http://bugzilla.zimbra.com/show_bug.cgi?id=27788

http://bugzilla.zimbra.com/show_bug.cgi?id=27477

Domain Rename Issues

We have a command to rename a "domain" and the resources within it - unfortunately it doesn't handle ALL of the different components for various reasons. Specifically Documents and Calendars [more down below].

Review this RFE that was done when they implemented the renameDomain command.

http://bugzilla.zimbra.com/show_bug.cgi?id=7655

Note comment 21 & 22 (confirming what QA'd). The syntax is

zmprov -l rd testold.com testnew.com

Problems that arise with Documents are explained here, comment #1 has workaround.

http://bugzilla.zimbra.com/show_bug.cgi?id=25873

Now, the very unfortunate part about calendars with a domainname change/move.

"Ability to change a username globally on all appointments"

http://bugzilla.zimbra.com/show_bug.cgi?id=26736

"Cant edit calendar entries after renameDomain"

http://bugzilla.zimbra.com/show_bug.cgi?id=27707

The root issue here's seems to be more about the calendar standards and practices with the use of the "Organizer" field and notifications. You'll see 3 "work arounds" in comment 2 of bug 26736.

Export/Importing of the calendar data is shown here:

http://wiki.zimbra.com/index.php?title=User_Migration

This part specifically :

http://wiki.zimbra.com/index.php?title=User_Migration#Copy_Calendar_From_One_Zimbra_User_to_Another

Finding The NO_SUCH_BLOB Errors

It would be good to review this wiki page before you start any "changes":

http://wiki.zimbra.com/index.php?title=Account_mailbox_database_structure

And great forum posting:

http://www.zimbra.com/forums/administrators/19811-solved-missing-blob-errors-zimbra-4-5-10-a.html

If you recently did a server move or similar type of operation, make sure zimbra:zimbra permissions are applied correctly to store directory. You could run the following to double check permissions (as root):

/opt/zimbra/libexec/zmfixperms --verbose --extended

And down and dirty way to see what mailboxes have this error:

grep -B2 NO_SUCH_BLOB /opt/zimbra/log/mailbox.lo* |grep mailbox= |sed -r 's/.*mailbox=([0-9]*).*$/\1/' |sort -u

On 5.0.6+ there's a script in /opt/zimbra/bin called zmblobchk . This will list out information about the NO_SUCH_BLOB errors.

On pre-5.0.6 systems, ask support for zmblobchk.jar file. To run it:

su - zimbra
ls /opt/zimbra/zmblobchk.jar
  zmblobchk.jar
java -jar zmblobchk.jar
Retrieving volume information
Retrieving mboxgroup list
Spooling item list to /tmp/mailitems53650.lst
Retrieving items from mboxgroup1
Retrieving items from mboxgroup2
Retrieving items from mboxgroup3
Retrieving items from mboxgroup4
Retrieving items from mboxgroup5
Retrieving items from mboxgroup6
Retrieving items from mboxgroup7
Retrieving items from mboxgroup8
Retrieving items from mboxgroup9
Retrieving items from mboxgroup10
/
Processing BLOB store
-
/tmp/mailitems53650.lst: size 28141
Processed 329 items in 1223ms
No inconsistencies found

If the "Processing BLOB store" part errors out with Java out of memory, you could try adjusting the variable for it. The default is set to 30%. This would require a mailstore/jetty restart though [zmmailboxdctl restart]. If the mailstore doesn't come back up, check /opt/zimbra/log/zmmailbox.log - you most likely exceeded the threshold for your box. Remember, this is a percentage and a 32bit machine with more than 4GB can cause issues with this setting since it could try to allocate more than it actually can to the thread.

# ZCS 5.0 and later
$ zmlocalconfig -e mailboxd_java_heap_memory_percent=40

And zmmailboxdctl restart when it's appropriate.

BLOB Issue Script

I've commented out the two delete lines and also the "$msghash &= $fmask;" one.

If you're using HSM, this script isn't smart enough to detect it. It will flag and remove entries that are in the HSM store becuase it doesn't see them in the primary store.

Comment from other support staff member on a case that used this.

Once you've gotten a list of the 'NOT OK' files generated by this tool, you'll want to check the store directories to see if the message blobs exist with an incorrect change number. That's the second number in the filename. If the first number matches the ID it's looking for but the second does not, you can rename the file to the new change number and get the message blob back without losing data. If not, then the blob is completely missing. If that's the case, we should try to figure out what happened to it by going through the mailbox.log files. You should be able to grep for the missing message ID and the user's account ID to find a log entry showing something happening to it. It's possible it was deleted and the deletion was never written to mysql for some reason.


#!/usr/bin/perl

# This script compatible with Zimbra version 4.5.x only.  Do not use with any other version.
# OK, there's 2 MAILBOX_*_BITS values in the VOLUME table.
# Take the mailbox ID, right-shift it by MAILBOX_BITS, and take the lowest MAILBOX_GROUP_BITS of the result.
# That's your mailbox hash.
# Take the message ID, right-shift it by FILE_BITS, and take the lowest FILE_GROUP_BITS of the result.  That's your msgid hash.
# I think.
# <mbx-hash>/<mbx-id>/msg/<msgid-hash>/<msgid>-<mod_content>.msg

my ($fbits, $fgbits, $mbits, $mgbits, $basepath) = split (' ',`echo "select file_bits,file_group_bits,mailbox_bits,mailbox_group_bits,path from volume where type='1'" | mysql -N zimbra`);
my $ARGV = shift @ARGV;

chomp $basepath;

my $mbmask = sprintf "1" x $mgbits;
my $fmask = sprintf "1" x $fgbits;

foreach (`echo "select id, group_id, account_id, comment from mailbox where comment = '$ARGV'" | mysql -N zimbra`) {
	chomp;
	my $path = "$basepath/";
	my ($id, $grid, $aid, $nm) = (split);
	my $mbhash = $id >> $mbits;
	$mbhash &= $mbmask;
	$path .= $mbhash."/".$id."/msg/";
	foreach my $msgstuff (`echo "select id, mod_content, type from mail_item where blob_digest is not null and mailbox_id=${id};" | mysql -N mboxgroup${grid}`) {
		chomp $msgstuff;
		if ($msgstuff eq "") {next;}
		my ($msgid, $modContent, $type) = split (' ',$msgstuff);
		my $msghash = $msgid >> $fbits;
#		$msghash &= $fmask;
		my $nm = $msgid;
		if ($modContent) {$nm .= "-$modContent";}
		my $npath = $path.$msghash."/".$nm.".msg";
		if (-e $npath) {print $npath."\t\tOK\n";} 
		else {
		   print $npath."\t\tNOT OK\n";
		   # not ok, remove the entry from the database so it is not a nuisance
		   print "Delete from mail_item where MSGID is ${msgid} and MAILBOXID is ${id} \n";
		   print "Uncomment line below me in script to have me delete. \n";
#		   `echo "delete from mail_item where id=${msgid} and mailbox_id=${id}" | mysql -N mboxgroup${grid}`;
		   if ($type eq "11") {
	       print "Delete from appointment where MSGID is ${msgid} and MAILBOX_ID is ${id} \n";
	       print "Uncomment line below me in script to have me delete. \n";
#              `echo "delete from appointment where item_id=${msgid} and mailbox_id=${id}" | mysql -N mboxgroup${grid}`;
		   }
		}
#		print $npath."\n";
	}
}

If this doesn't run, make sure perl is installed and in the correct path.

You run this as the zimbra user.

What I did was the following.

Copied the script to /opt/zimbra/bin/ as root and called it blob-check.pl . chmod +x /opt/zimbra/bin/blob-check.pl

su - zimbra
mkdir /tmp/BLOB-CHECKS
for i in `zmprov gaa`; do blob-check.pl $i | grep "NOT OK" >> /tmp/BLOB-CHECKS/$i.txt; done
ls -la /tmp/BLOB-CHECKS

The ls will help identify the more troubled accounts. You'll end up with something like this:

[zimbra@mail3 ~]$ ls -la /tmp/BLOB-CHECKS/
total 16
drwxr-x--- 2 zimbra zimbra 4096 Jul 10 23:40 .
drwxrwxrwt 8 root   root   4096 Jul 10 23:46 ..
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 admin@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 ajcody2@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra   49 Jul 10 23:40 ajcody@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 forward@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 ham.bidiob2mm@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 secondary@secondary.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 spam.rormmtcyy@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 wiki@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 wiki@secondary.internal.homeunix.com.txt

And the output of one that shows some size:

$ cat /tmp/BLOB-CHECKS/ajcody@mail3.internal.homeunix.com.txt
/opt/zimbra/store/0/3/msg/0/360-6070.msg		NOT OK

Remember this from above, "you'll want to check the store directories to see if the message blobs exist with an incorrect change number. That's the second number in the filename. If the first number matches the ID it's looking for but the second does not, you can rename the file to the new change number and get the message blob back without losing data."

One can uncomment the 2 lines in the script with delete and run the script again. This will remove the reference to the blob.

To re-index the users mailbox [this can take start, cancel, status]:

zmprov rim user@domainname start

Running the script again shouldn't show "NOT OK" lines.

FYI - I still need to gather more information about this situation and how to guide one to make a decision to delete, re-index, and so forth.

BLOB Script To Copy From Restore

You might need to use these variables with the restore command. I need to find a way for someone to figure out what restore to goto for a particular blob:

-restoreToTime <arg> - Replay the redo logs until the time specified.

-restoreToIncreLabel <arg> - Replay redo logs up to and including this incremental backup.

-restoreToRedoSeq <arg> - Replay up to and including this redo log sequence

-br - Replays the redo logs in backup only, which excludes archived and current redo logs of the system.

-rf - Restores to the full backup only, does not include any incremental backups since that backup

This is a walk through to test the proceedure.

$ zmrestore -ca -a ajcody@mail3.internal.homeunix.com -pre restored-
$ zmprov gmi restored-ajcody@mail3.internal.homeunix.com
mailboxId: 12
quotaUsed: 251513
$ zmprov gmi ajcody@mail3.internal.homeunix.com
mailboxId: 3
quotaUsed: 251512
$ diff /opt/zimbra/store/0/12/msg/0/257-25.msg /opt/zimbra/store/0/3/msg/0/257-25.msg
$ rm /opt/zimbra/store/0/3/msg/0/257-25.msg
rm: remove regular file `/opt/zimbra/store/0/3/msg/0/257-25.msg'? y
$ cp /opt/zimbra/store/0/12/msg/0/257-25.msg /opt/zimbra/store/0/3/msg/0/257-25.msg
$ grep Subject /opt/zimbra/store/0/3/msg/0/257-25.msg
Log into main account and confirm I can see/use that mail.

The script to look at the blob-check.pl output and do the copies after a redirected restore is here:

#!/bin/bash

#START OF VARIABLES#
#CHANGEME to existing user account
USER=ajcody@mail3.internal.homeunix.com

#CHANGEME to the restored account name
# zmrestore -ca -a ajcody@mail3.internal.homeunix.com -pre restored-
RESTOREDUSER=restored-ajcody@mail3.internal.homeunix.com

# To get FILE, blob-check.pl ajcody@mail3.internal.homeunix.com > /tmp/BLOB-CHECKS/ajcody.txt
# blob-check.pl located at http://wiki.zimbra.com/index.php?title=Ajcody-Notes#BLOB_Issue_Script
FILE=/tmp/BLOB-CHECKS/ajcody.txt #CHANGEME

#CHANGEME, if needed, to the mailstore. /opt/zimbra/store is default
STOREDIR=/opt/zimbra/store

#END VARIABLES#

USERUID=`zmprov gmi $USER | grep mailboxId | cut -f2 -d: | cut -c 2-10`
USERGID=`perl -e 'print $USERUID >> 12 ; print "\n"'`
USERPATH=`echo $USERGID/\$USERUID`

RESTOREDUID=`zmprov gmi $RESTOREDUSER | grep mailboxId | cut -f2 -d: | cut -c 2-10`
RESTOREDGID=`perl -e 'print $RESTOREDUID >> 12 ; print "\n"'`
RESTOREDPATH=`echo $RESTOREDGID/\$RESTOREDUID`

echo USERUID $USERUID
echo USERGID $USERGID
echo USERPATH $USERPATH
echo RESTOREDUID $RESTOREDUID
echo RESTOREDGID $RESTOREDGID
echo RESTOREDPATH $RESTOREDPATH

cd $STOREDIR
echo "Run ECHO first to confirm copies look right and then uncomment COPY in script"
for ROOTBLOBPATH in `cat $FILE | grep "NOT OK" | cut -f7-9 -d/ | cut -f1`
do
# Comment out echo and uncomment copy after dry run
echo $RESTOREDPATH/$ROOTBLOBPATH $USERPATH/$ROOTBLOBPATH
#cp -uv $RESTOREDPATH/$ROOTBLOBPATH $USERPATH/$ROOTBLOBPATH
done

Account & Domain Summary

Run zmaccts

Here's what it would return:

su - zimbra
[zimbra@mail3 ~]$ zmaccts
           account                          status             created       last logon
------------------------------------   -----------     ---------------  ---------------
admin@mail3.internal.homeunix.com           active      05/06/08 18:46   07/08/08 09:56
ajcody@mail3.internal.homeunix.com          active      05/06/08 20:43   06/23/08 15:48
ajcody2@mail3.internal.homeunix.com         active      05/28/08 11:48   06/30/08 17:44
forward@mail3.internal.homeunix.com         active      05/06/08 21:06   05/29/08 17:24
ham.bidiob2mm@mail3.internal.homeuni        active      05/06/08 18:47            never
spam.rormmtcyy@mail3.internal.homeun        active      05/06/08 18:47            never
wiki@mail3.internal.homeunix.com            active      05/06/08 18:46            never
           account                          status             created       last logon
------------------------------------   -----------     ---------------  ---------------
secondary@secondary.internal.homeuni        active      06/23/08 15:26   06/23/08 15:27
wiki@secondary.internal.homeunix.com        active      06/23/08 15:25            never
-
                                domain summary
-
    domain                  active    closed    locked    maintenance     total
-----------------------   --------  --------  --------  -------------  --------
mail3.internal.homeunix          7         0         0              0         7
secondary.internal.home          2         0         0              0         2


My Gal & LDAP Settings For A Domain

To see your setting, do the following - replacing with domainname with the domain in question.

zmprov gd domainname | egrep -i 'ldap|gal'

Zimlets

Zimlets should be already located on the zimbra server in one of these directories:

/opt/zimbra/zimlets/
/opt/zimbra/zimlets-admin-extra/
/opt/zimbra/zimlets-extra/
/opt/zimbra/zimlets-experimental/
/opt/zimbra/zimlets-network/

To deploy a zimlet, simply cd to the directory where the zimlet is located and issue this command:

/opt/zimbra/bin/zmzimletctl deploy <zimlet_name>

Archive & Discovery

The two main A&D references are :

http://wiki.zimbra.com/index.php?title=ZAD

http://www.zimbra.com/pdf/Zimbra%20Archiving%20and%20Discovery%20Release%20Notes.pdf

See below for a very rough draft document I made for multi-server / new mailstore A&D setups.

http://wiki.zimbra.com/index.php?title=Ajcody-Notes-AD-mailstore

I also created a RFE for documentation on this.

http://bugzilla.zimbra.com/show_bug.cgi?id=25135

Zmmailbox Stuff

Remove All Shares

Here's a script I wrote. Remove the echo statements to actually run the commands.

#!/bin/bash
USER="ajcody@mail3.internal.homeunix.com"
SHARE="/Shared"
GETPERM="zmmailbox -z -m $USER gfg $SHARE"
MODPERM="zmmailbox -z -m $USER mfg $SHARE"
DUMBPASS="34lkoso"
NEWPERM=none

$GETPERM | egrep -i 'all|guest|public|accoun|domain|group' | gawk '{print $2 " " $3}' | while read SHAREPERM
do
TYPE=`echo $SHAREPERM|awk '{print $1}'`
DISPLAY=`echo $SHAREPERM|awk '{print $2}'`

case $TYPE in
        accoun) echo $MODPERM account $DISPLAY $NEWPERM
        ;;
        guest) echo $MODPERM $TYPE $DISPLAY $DUMBPASS $NEWPERM
        ;;
        all) echo $MODPERM $TYPE $NEWPERM
        ;;
        *) echo $MODPERM $SHAREPERM $NEWPERM
        ;;
        esac
done

Ouput of an example:

[zimbra@mail3 ~]$  zmmailbox -z -m ajcody@mail3.internal.homeunix.com gfg /Shared
Permissions    Type  Display
-----------  ------  -------
          r     all
          r   guest  ajcody@digitalhandshakes.com
          r  accoun  admin@mail3.internal.homeunix.com
          r   group  mydl@mail3.internal.homeunix.com
          r  domain  mail3.internal.homeunix.com
[zimbra@mail3 ~]$ /tmp/remove-share.sh
zmmailbox -z -m ajcody@mail3.internal.homeunix.com mfg /Shared all none
zmmailbox -z -m ajcody@mail3.internal.homeunix.com mfg /Shared guest ajcody@digitalhandshakes.com 34lkoso none
zmmailbox -z -m ajcody@mail3.internal.homeunix.com mfg /Shared account admin@mail3.internal.homeunix.com none
zmmailbox -z -m ajcody@mail3.internal.homeunix.com mfg /Shared group mydl@mail3.internal.homeunix.com none
zmmailbox -z -m ajcody@mail3.internal.homeunix.com mfg /Shared domain mail3.internal.homeunix.com none

I then removed the echo statements:

[zimbra@mail3 ~]$ vi /tmp/remove-share.sh
[zimbra@mail3 ~]$ /tmp/remove-share.sh
[zimbra@mail3 ~]$  zmmailbox -z -m ajcody@mail3.internal.homeunix.com gfg /Shared
Permissions    Type  Display
-----------  ------  -------
[zimbra@mail3 ~]$

Mobility

Help Me Understand the Mobility Options

Let's break it down to three forms of access for mobile devices.

Mobile Web Client - http://www.zimbra.com/products/mobile_web_browser.html

A. Setups the web access to the Zimbra server to be viewable from a mobile device. This is accessing mail over the "web browser" on the mobile phone.

A1. http://servername.com/zimbra/m/

B. Built in - no license or additional software needed.

Zimbra Mobile - http://www.zimbra.com/products/mobile_smartphone.html

A. This allows two-way, over-the-air synchronization between the mobile device and Zimbra server.

A1. Think of this as "fat client" mail sync'ing on a mobile device

B. This page will describe that process in better detail - http://wiki.zimbra.com/index.php?title=Moble_Device_Setup

C. License required

Blackberry - http://www.zimbra.com/products/mobile_blackberry.html

A. Blackberry "two-way, over-the-air synchronization" requires the use of a Blackberry server. We use a "connector" on the Blackberry server that allows BES device to use their "fat client" for mail sync'ing.

B. This option requires a separate server, a Windows box that will run the Blackberry server software.

C. Zimbra Mobile license required. Windows Blackberry server require with appropriate Blackberry licenses. Installation of Zimbra BlackberryConnector on the Blackberry server.

D. This is still in BETA

Zimbra Mobile/Blackberry requires the license file to enable it. This cost can be discussed with your sales contact. It's only the Blackberry configuration that requires a "software" installation - so to speak. Enabling "Zimbra Mobility/option 2 above" is a license requirement - not a software installation one - and then configuration for the user. "You enable Zimbra Mobile in the ZCS COS or for individual Accounts."

Jump to: navigation, search