Ajcody-Notes: Difference between revisions
Line 25: | Line 25: | ||
{{:Ajcody-User-Management-Topics}} | {{:Ajcody-User-Management-Topics}} | ||
{{:Ajcody-Server-Topics}} | |||
{{:Ajcody-Client-Topics}} | |||
=Mobility= | =Mobility= |
Revision as of 05:39, 16 July 2008
This page holds a Table of Contents of all my pages but each main Subject is an actual link to another page.
I'm hoping this gives the best of both worlds. One long page with all topics as well as pages that contain details to a specific subject.
Done by way of putting something like this on this page:
{{:Ajcody-Apple-Mac-Issues}}
And on the target page:
===Actual Apple/Mac Issues Homepage=== Please see [[Ajcody-Apple-Mac-Issues]]
Enjoy, Adam
General Notes
Actual General Notes Homepage
Please see Ajcody-General-Notes
Zimbra Variables
Ways to check on different zimbra variables.
su - zimbra zmlocalconfig -i zmlocalconfig -d cd /opt/zimbra/conf/attrs more zimbra-attrs.xml cd /opt/zimbra/openldap/etc/openldap/schema/ more zimbra.schema
Made the following concerning zmlocalconfig's behavior:
- "RFE: zmlocalconfig sanity check and clean up of -i output"
Also, with newer versions of ZCS, you can get it via zmprov - for example:
#zmprov desc -a zimbraFreeBusyExchangeAuthScheme zimbraFreebusyExchangeAuthScheme auth scheme to use type : enum value : basic,form callback : immutable : false cardinality : single requiredIn : optionalIn : cos,domain,globalConfig,account flags : accountInherited,domainInherited defaults : min : max : id : 611 requiresRestart : since : 5.0.3 deprecatedSince :
Addition note, example, for those checking source : src/6.0.7/com/zimbra/cs/account/ProvUtil.java.html
Zimbra Contact Fields For CSV Import Mapping
Please see /opt/zimbra/conf/zimbra-contact-fields.xml
This will show you what "fields" Zimbra is looking for and mapping to in regards to importing from another applications contact csv files.
One easy way to do this is the following:
- From zimbra ajax client as a test user (or any old user) do an export Addressbook.
- This is under the preference tab > Address Book > Import / Export > Export - select Addressbook and click Export button
- Then open file file something like Excel or a Text Editor
- Your only interested in the Top line - that gives the default fields for a Zimbra Addressbook.
- You'll get something like this:
"company","companyPhone","email2","fileAs","firstName","homepageURL","imAddress3","lastName","otherIMLabel","workCity","workCountry","workEmail2","workEmail3","workEmail4","workEmail5","workPostalCode","workState","workStreet"
- Note, ordering [left to right] isn't important. What is important, is the data lines up under the right column header.
- With your other Mail system, export the addressbook to a csv file.
- Open this file in another Excel window.
- Copy the column data from the one Excel sheet and paste it into the appropriate column within the Zimbra Excel sheet.
- Move as much as you can and then save.
- For example:
"company","companyPhone","email2","fileAs","firstName","homepageURL","imAddress3","lastName","otherIMLabel","workCity","workCountry","workEmail2","workEmail3","workEmail4","workEmail5","workPostalCode","workState","workStreet" "Apple Computer Inc.","1-800-MY-APPLE","","3","","http://www.apple.com","","","","Cupertino","United States","","","","","95014","CA","1 Infinite Loop"
- Now you can try importing this csv file back into Zimbra.
- Either with the webclient or you can use some on the CLI commands
- See User_Migration for variations of CLI commands that could be used.
- Here's the documented example:
curl -u schemers:password --data-binary @/tmp/new.csv http://server/service/home/schemers/contacts?fmt=csv
Example Header For Horde Contacts
I have not used this myself, I found it on the forums.
"assistantPhone","birthday","callbackPhone","carPhone","company","companyPhone","email","email2","email3","firstName","fullName","homeCity","homeCountry","homeFax","homePhone","homePhone2","homePostalCode","homeState","homeStreet","homeURL","imAddress1","imAddress2","imAddress3","jobTitle","lastName","middleName","mobilePhone","notes","otherCity","otherCountry","otherFax","otherPhone","otherPostalCode","otherState","otherStreet","otherURL","pager","workCity","work"
I'm hoping the forum user simply renamed the top header column to match description fields that Zimbra knows and that it's in order of the data (left to right) that Horde normally exports in. You'll need to review it and make sure it fits with your exported data columns from Horde.
Admin With Curl And Wget
You will most likely need to include :7071 in the url string to get around the pop-up issue that you would get if you used the same string in a web browser, without :7071 you'll see a permission error when trying to see/get data from a users account via the admin account.
curl -u admin:pass https://hostname:7071/home/user/Contacts wget https://admin:pass@hostname:7071/home/user/Contacts
What's My Server Like
To dump out your server details, these commands are useful. Login to your zimbra server and as the zimbra user do:
zmprov gs `zmhostname` zmprov gacf zmlocalconfig zmdumpenv
Another good document to review for "odd performance" issues is the large server wiki.
http://wiki.zimbra.com/index.php?title=Performance_Tuning_Guidelines_for_Large_Deployments
Restarting Jetty - ZCS 5+
As zimbra
zmmailboxdctl restart
How Do I Find Out What Version Of Zimbra I'm Using
See:
Get Version From Ajax Client
In newer versions of ZWC, the Help > About will also state the ZCS server version being used.
For older version, put the below in the search field of the Ajax client and hit enter:
$set:get version
Also:
https://SERVERNAME/js/zimbraMail/share/model/ZmSettings.js
Search for the CLIENT_VERSION line.
Other Ways To Get Version Of Server
See this forum post:
What's Compiled With Postfix
Moved to What's_Complies_With_Zimbra's_Postfix
REST Information
Resources to review
- Rest Overview
- For queries:
- Working Examples
Format to get around permission denied on multi-domain hosting servers
If you getting permission denied rather than resource not found...try this format example:
http://MAILSTOREofUSER.DOMAIN.com/zimbra/user/FIRST.LAST@USERS_DOMAIN.com/inbox.zip
or
http://MAILSTOREofUSER.DOMAIN.com/zimbra/user/USERNAME@USERS_DOMAIN.com/inbox.zip
I but both in case one issue is where the username has special characters, like the period.
Server Source Doc's
Please see:
http://svn.sourceforge.net/viewvc/zimbra/trunk/ZimbraServer/docs/
Logging
Actual Logging Homepage
Please see Ajcody-Logging
Server
RFE's Related To Better Logging And Historical Data Of Systems
See : Ajcody-Testing-Debugging#RFE.27s_Related_To_Better_Logging_And_Historical_Data_Of_Systems
Debugging
See : Ajcody-Testing-Debugging for more complete debugging information.
When Was A ZCS Service Enabled Or Disabled
See : Ajcody-Notes-Archive-Discovery#When_Was_A_ZCS_Service_Enabled_Or_Disabled
Syslog Items
Single Server Setup
/etc/syslog.conf should have lines similiar to:
[towards bottom of conf file] local0.* -/var/log/zimbra.log auth.* -/var/log/zimbra.log mail.* -/var/log/zimbra.log
Make sure syslog allows messages from log4j to be written, log4j doesn't do unix pipes. Specifically, it uses internet domain sockets (514/upd) instead of unix domain sockets (/dev/log). On a single server setup, this means log4j talking to localhost via 514/udp. Again, /dev/log is never used by log4j (AFAIK, this could change in the future). Adding the "-r" applies for the centralized syslog server as usual, but it also applies for the case where you want log4j data to be logged via syslog locally [single server setup]:
[root@zimbra sysconfig]# diff -u /etc/sysconfig/syslog.ORIG /etc/sysconfig/syslog --- /etc/sysconfig/syslog.ORIG 2008-03-25 09:14:28.000000000 -0400 +++ /etc/sysconfig/syslog 2009-06-09 16:01:13.000000000 -0400 @@ -3,7 +3,7 @@ # -r enables logging from remote machines # -x disables DNS lookups on messages recieved with -r # See syslogd(8) for more details -SYSLOGD_OPTIONS="-m 0" +SYSLOGD_OPTIONS="-r -m 0" # Options to klogd # -2 prints all kernel oops messages twice; once for klogd to decode, and # once for processing with 'ksymoops'
Restart syslog:
/etc/init.d/syslog restart
Setup ZCS to be aware of syslog [we'll restart Zimbra later once all the changes are done]:
zmprov mcf zimbraLogToSyslog TRUE
This will set the appropriate ldap values (after a zimbra restart) to make log4j.properties be written correctly:
/opt/zimbra/conf/log4j.properties will now show some modifications, for example:
[snips - DON'T cut cut/paste this info below for your server] #log4j.rootLogger=INFO,LOGFILE log4j.rootLogger=INFO,LOGFILE,SYSLOG ... # Syslog appender log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender log4j.appender.SYSLOG.SyslogHost=localhost log4j.appender.SYSLOG.Facility=LOCAL0 log4j.appender.SYSLOG.layout=com.zimbra.common.util.ZimbraPatternLayout log4j.appender.SYSLOG.layout.ConversionPattern=mailboxd: %-5p [%t] [%z] %c{1} - %m
This will actually cause very little to goto syslog though, it's a very minimalist setup. Now adjust /opt/zimbra/conf/log4j.properties.in to log more items to syslog.
Note: You may or may not want all these things going to syslog and you may be able to send more/other things to syslog as well. I've not spend lots of time with log4j and I'm not familiar enough with all the code to know which pieces are using which logger/settings.
WARNING: the AUDIT logs may have sensitive data so be careful to protect the logs so that sensitive data is not leaked accidentally.
[zimbra@zimbra conf]$ diff -u log4j.properties.in.ORIG log4j.properties.in --- log4j.properties.in.ORIG 2009-06-05 15:31:20.000000000 -0400 +++ log4j.properties.in 2009-06-09 15:54:07.000000000 -0400 @@ -35,7 +35,7 @@ # Save zimbra.security to AUDIT appender log4j.additivity.zimbra.security=false -log4j.logger.zimbra.security=INFO,AUDIT +log4j.logger.zimbra.security=INFO,AUDIT,SYSLOG # Syslog appender log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender @@ -88,7 +88,7 @@ log4j.additivity.zimbra.wbxml=false log4j.logger.zimbra.wbxml=DEBUG,WBXML -log4j.logger.zimbra=INFO +log4j.logger.zimbra=INFO,SYSLOG log4j.logger.zimbra.op=WARN -log4j.logger.com.zimbra=INFO +log4j.logger.com.zimbra=INFO,SYSLOG
At this point, you can restart zimbra for all the changes to be in effect:
zmcontrol stop ; zmcontrol start
Centralized Syslog Server
Please see:
This will show you how to setup a syslog server and then to configure the "clients" to log to the local filesystem as well as the syslog server.
You'll need to then take into account the information in Ajcody-Logging#Single_Server_Setup to handle the "Zimbra" part as well as dealing with log4j.
Centralized Log Server Project Ideas
Problems:
- Logs are to large to manage on each server.
- Want to build a central logging server for all logging events.
- The syslog events are very minimal with Zimbra (/var/log/zimbra.log). Most of the logging is control by Log4j (tomcat/jetty). Output from log4j generally goes into the /opt/zimbra/log/ directory. There's some conf files in /opt/zimbra/conf/log4j.properties* for log4j. You can adjust this file to also output into the syslog environment but syslog has very limited choices for "facility". See Ajcody-Logging#Single_Server_Setup and Log4j & Syslog facility for details. You'll end up getting huge files and god knows how many log events per second if you adjusted log4j to dump all zimbra logging into syslog control.
General Thought On This:
Use syslog-ng with mysql for the central log server. Add on php-syslog-ng for comfort.
Leave the log4j stuff alone and configure syslog-ng to monitor the actual log files as they are generated by log4j in /opt/zimbra/log/* .
Then decrease the log sizes and archiving done on the local servers, i.e. zimbraLogRawLifetime & zimbraLogSummaryLifetime .
References:
- Syslog-ng Homepage
- "O'Reilly - Building Secure Servers With Linux - Chapter On Syslog-ng"
- http://eduunix.ccut.edu.cn/index2/html/linux/O'Reilly%20-%20Building.Secure%20Servers%20with%20Linux/bssrvrlnx-CHP-10-SECT-2.html
- Most notably, the section on "sources"
- Php-syslog-ng
- http://code.google.com/p/php-syslog-ng/
- How-To Setup and Install Syslog-ng, mysql, and php-syslog-ng
Log Rotation and Removal
Logs In /opt/zimbra/log Directory
Please look at the output of your crontab - AS ZIMBRA, look for the "Log pruning" line.
crontab -l
And also the config files for logrotate:
/etc/logrotate.conf
/etc/logrotate.d/zimbra
Example From 5.0.11 - Complete ZCS Install Single Server
crontab for zimbra:
# Log pruning # 30 2 * * * find /opt/zimbra/log/ -type f -name \*.log\* -mtime +8 -exec rm {} \; > /dev/null 2>&1 35 2 * * * find /opt/zimbra/log/ -type f -name \*.out.???????????? -mtime +8 -exec rm {} \; > /dev/null 2>&1 # Log pruning # 30 2 * * * find /opt/zimbra/mailboxd/logs/ -type f -name \*log\* -mtime +8 -exec rm {} \; > /dev/null 2>&1
And the logrotate files:
$ cat /etc/logrotate.conf # see "man logrotate" for details # rotate log files weekly weekly # keep 4 weeks worth of backlogs rotate 4 # create new (empty) log files after rotating old ones create # uncomment this if you want your log files compressed #compress # RPM packages drop log rotation information into this directory include /etc/logrotate.d # no packages own wtmp -- we'll rotate them here /var/log/wtmp { monthly minsize 1M create 0664 root utmp rotate 1 } # system-specific logs may be also be configured here.
And also:
$ cat /etc/logrotate.d/zimbra /var/log/zimbra.log { daily missingok notifempty create 0644 zimbra zimbra postrotate killall -HUP syslogd 2> /dev/null || true kill -HUP `cat /opt/zimbra/log/swatch.pid 2> /dev/null` 2> /dev/null || true kill -HUP `cat /opt/zimbra/log/logswatch.pid 2> /dev/null` 2> /dev/null || true endscript compress } /opt/zimbra/log/myslow.log { daily missingok copytruncate rotate 30 notifempty create 0644 zimbra zimbra compress } /opt/zimbra/log/logger_myslow.log { daily missingok copytruncate notifempty create 0660 zimbra zimbra compress size 5000k rotate 7 } /opt/zimbra/log/clamd.log { daily missingok copytruncate notifempty create 0660 zimbra zimbra postrotate kill -HUP `cat /opt/zimbra/log/clamd.pid 2> /dev/null` 2> /dev/null || true endscript compress size 5000k rotate 7 } /opt/zimbra/log/zmlogswatch.out { daily missingok copytruncate notifempty create 0740 zimbra zimbra postrotate su - zimbra -c "/opt/zimbra/bin/zmlogswatchctl stop" su - zimbra -c "/opt/zimbra/bin/zmlogswatchctl start" endscript rotate 5 compress } /opt/zimbra/log/zmswatch.out { daily missingok copytruncate notifempty create 0740 zimbra zimbra postrotate su - zimbra -c "/opt/zimbra/bin/zmswatchctl stop" su - zimbra -c "/opt/zimbra/bin/zmswatchctl start" endscript rotate 5 compress } /opt/zimbra/log/zmmtaconfig.log { daily missingok copytruncate notifempty create 0740 zimbra zimbra postrotate su - zimbra -c "/opt/zimbra/bin/zmmtaconfigctl restart" endscript rotate 5 compress } /opt/zimbra/log/nginx.log { daily missingok notifempty create 0644 zimbra zimbra postrotate kill -USR1 `cat /opt/zimbra/log/nginx.pid 2> /dev/null` 2> /dev/null || true endscript rotate 7 compress } /opt/zimbra/log/zmconvertd.log { daily missingok copytruncate notifempty create 0644 zimbra zimbra rotate 7 } /opt/zimbra/zmstat/zmstat.out { daily missingok copytruncate rotate 7 notifempty compress }
Openldap Logs - In /opt/zimbra/openldap-data/
Ldap logs files in /opt/zimbra/openldap-data/ and in /opt/zimbra/data/ldap/* subdirectories [ZCS 6+] that are like log.0000000001 , log.0000000002 , etc.
This logpurge is controlled by:
$ cd /opt/zimbra/openldap/etc/openldap/ [zimbra@mail3 openldap]$ grep logpurge * master-accesslog-overlay.conf:logpurge 07+00:00 01+00:00
Logpurge Directive Description
From Accesslog Chapter
logpurge Directive
logpurge age interval
Defines both the maximum age for log entries to be retained in the database and how often to scan the database for old entries. Bothage and interval are specified as a time span in days, hours, minutes, and seconds. The time format is [ddd+]hh:mm[:ss], for example, the days and seconds components are optional but hours and minutes are required. Except for days, which can be up to 5 digits, each numeric field must be exactly two digits. Example:
- the log database will be scanned every day
- entries older than two days will be deleted.
logpurge 2+00:00 1+00:00
When using a log database that supports ordered indexing on generalizedTime attributes, specifying an eq index on the reqStart attribute will increase the performance of purge operations.
What's up with all the logs?
Other Logging Page Resources
http://www.zimbra.com/docs/ne/latest/administration_guide/9_Monitoring.14.1.html
http://wiki.zimbra.com/index.php?title=Server_Monitoring
http://wiki.zimbra.com/index.php?title=Log_Files
http://wiki.zimbra.com/index.php?title=Unresponsive_Server_Troubleshooting
The Bread And Butter Logs
/opt/zimbra/log/mailbox.log - where most of your mailbox store activity is logged
- This log is the mailboxd log4j server log containing the logs from the mailbox server. This includes activity from the mailbox store, LMTP server, IMAP and POP servers, and Index server.
- Location: /opt/zimbra/log/mailbox.log
- This log is the mailboxd log4j server log containing the logs from the mailbox server. This includes activity from the mailbox store, LMTP server, IMAP and POP servers, and Index server.
/opt/zimbra/log/zmmailboxd.out - mailboxd/jvm output log
- Mailstore not coming up and nothing is being logged in mailbox.log, check here for errors.
- Location: /opt/zimbra/log/zmmailboxd.out
- Mailstore not coming up and nothing is being logged in mailbox.log, check here for errors.
/opt/zimbra/log/stacktrace.<pid> - stacktrace logs
- stacktraces related to mailboxd
- Location: /opt/zimbra/log/stacktrace.<pid>
- stacktraces related to mailboxd
/opt/zimbra/db/data/YOURHOSTNAME.err - errors for MySQL - ZCS 4.x & ZCS 5.x
- This is the message store database error log.
- Location: /opt/zimbra/db/data/YOURHOSTNAME.err
- This is the message store database error log.
/var/log/zimbra.log - mta and system status log, postfix, amavisd
- The Zimbra syslog details the activities of the Zimbra MTA (Postfix, amavisd, antispam, antivirus), Logger, Authentication (cyrus-sasl), and Directory (OpenLDAP). By default LDAP activity is logged to Zimbra.log.
- Location: /var/log/zimbra.log
- The Zimbra syslog details the activities of the Zimbra MTA (Postfix, amavisd, antispam, antivirus), Logger, Authentication (cyrus-sasl), and Directory (OpenLDAP). By default LDAP activity is logged to Zimbra.log.
- Probably has mta logging events [which also show in /var/log/zimbra.log] and also log events related to your OS
- Location: /var/log/messages
- Probably has mta logging events [which also show in /var/log/zimbra.log] and also log events related to your OS
/opt/zimbra/log/mysql_error.log - problems with MySQL
- If there is data corruption or another problem causing direct mysql errors, events will be logged here.
- /opt/zimbra/log/mysql_error.log
- If there is data corruption or another problem causing direct mysql errors, events will be logged here.
/opt/zimbra/log/myslow.log - slow db/MySQL queries
- If certain search requests are taking longer to complete than others, they will be logged here.
- /opt/zimbra/log/myslow.log
- If certain search requests are taking longer to complete than others, they will be logged here.
Other Logs
/opt/zimbra/log/audit.log - authentication events
- A log of all admin actions taken as well as logins to the server.
- /opt/zimbra/log/audit.log
- A log of all admin actions taken as well as logins to the server.
- Want to see all uses of the "View Mail" button from the admin console? Will have the IP address, User Agent of the browser, user that the admin is logged in as, and the account that they are viewing.
grep DelegateAuth /opt/zimbra/log/audit.log
/opt/zimbra/log/clamd.log - antivrius db
- Status and checks for clamav
- /opt/zimbra/log/clamd.log
- Status and checks for clamav
- If your clamav db is out of date, you'll see a log event here like this:
- LibClamAV Warning: **************************************************
- LibClamAV Warning: *** The virus database is older than 7 days! ***
- LibClamAV Warning: *** Please update it as soon as possible. ***
- LibClamAV Warning: **************************************************
/opt/zimbra/log/convertd.log - attachment conversion
- If your having problems with your attachments and the ability to view them in ZWC, check here for errors.
- /opt/zimbra/log/convertd.log
- If your having problems with your attachments and the ability to view them in ZWC, check here for errors.
/opt/zimbra/log/freshclam.log - clam antivirus updates
- Shows the actual attempt to update the clamav updates.
- /opt/zimbra/log/freshclam.log
- Shows the actual attempt to update the clamav updates.
/opt/zimbra/log/spamtrain.log - spam/ham training details
- Spam/Ham training details are logged here. Errors are also logged here related to this service.
- From a default single ZCS configurations crontab [zimbra]:
- # Spam training
- #:
- 0 :23 * * * /opt/zimbra/bin/zmtrainsa >> /opt/zimbra/log/spamtrain.log 2>&1
- #
- # Spam training cleanup
- #
- 45 23 * * * /opt/zimbra/bin/zmtrainsa --cleanup >> /opt/zimbra/log/spamtrain.log 2>&1
/opt/zimbra/log/sync.log - zimbra mobile and activesync
- Your mobile devices will basically log here.
- /opt/zimbra/log/sync.log
- Your mobile devices will basically log here.
/opt/zimbra/log/synctrace.log - zimbra mobile and activesync
- Still looking for description that describes difference with sync.log
- /opt/zimbra/log/synctrace.log
- Still looking for description that describes difference with sync.log
/opt/zimbra/log/syncstate.log - no description found
- No description found.
- /opt/zimbra/log/syncstate.log
- No description found.
/opt/zimbra/log/wbxml.log - no description found
- No description found.
- /opt/zimbra/log/wbxml.log
- No description found.
/opt/zimbra/log/zmlogswatch.out -
/opt/zimbra/log/zmmtaconfig.log -
/opt/zimbra/log/zmmyinit.log -
/opt/zimbra/log/zmsetup.`date`-'pid'.log -
/tmp/install.log -
After install, gets moved to : /opt/zimbra/.install/
/tmp/zmsetup.log -
After install, gets moved to : /opt/zimbra/log/zmsetup.log.[some numbers]
/opt/zimbra/httpd_access.log -
/opt/zimbra/httpd_error.log -
New Data Directory In ZCS6+
In ZCS 6+ , there's a directory to organize 'data' from our various components.
[root@mail3 data]# pwd /opt/zimbra/data [root@mail3 data]# ls -F altermime/ amavisd/ clamav/ dspam/ ldap/ postfix/ tmp/
This information could prove useful as well with trouble-shooting and debugging problems.
Increase Logging
Increase Logging Per User
Depending on the ZCS version the below will clear all the per account loggers on zimbra reboot. IMAP is used for these examples.
- addAccoutLogger
zmprov aal user@domain.com zimbra.imap debug
- removeAccountLogger
zmprov ral user@domain.com zimbra.imap
Example usuage:
zmprov aal user@domain.com zimbra.soap debug
And tail the log file, while attempting to login to the AJAX client.:
tail -f /opt/zimbra/log/mailbox.log
See Ajcody-Logging#Log4J_Variables for variables that can be used.
To See What Accounts Have Extended Logging Enabled
There's the gaal option for zmprov . It list the accounts on a per mailstore basis :
getAllAccountLoggers(gaal) [-s/--server hostname]
[zimbra@zcs806 ~]$ zmprov gaal `zmhostname` # name test01@zcs806.domain.com zimbra.imap=debug
Bugs And RFE's
I filed the following RFE:
- "Allow "secondary" logging file for zmprov addAccountLogger"
Increase Logging Per Server
If you want to do it globally edit this file:
vi /opt/zimbra/conf/log4j.properties
Add a line at the end that's similar to something like this: log4j.logger.zimbra.imap=DEBUG.
No restart of any service is needed, BUT if something happens that causes a regeneration of the log4j.properties file then your changes will be overwritten. A regeneration of this file takes the contents of /opt/zimbra/conf/log4.properties.in .
To permanently make a change, modify the /opt/zimbra/conf/log4.properties.in file, then a restart would be necessary.
$ zmcontrol stop $ zmcontrol start
or you could run this on the server:
$ zmmtaconfig mailboxd
Wait for a minute for the server to pick up your change.
See Ajcody-Logging#Log4J_Variables for variables that can be used.
Log4J Variables
An Official page has been made for this topic now per my RFE request, bug 41894
Please see:
OpenLDAP Logging Levels
Please see King0770-Notes-Change-LDAP-Log-Levels
Want To See What's In Redolog Files
If you suspect there's too much redolog activity during a time window or have another need to inspect the contents of the redolog, dump it and examine it:
$ zmjava com.zimbra.cs.redolog.util.RedoLogVerify /opt/zimbra/redolog/redo.log > out.file
Pick the right redolog file, either redo.log or one of the files under archive/, based on timestamp.
See Ajcody-Backup-Restore-Issues#Redolog_Files for more details.
Internal Zimbra Charting - zmstat-chart
RFE/Bugs You Might Be Interested In
- "support for real time counters (snmp, jmx, etc)"
- "zmstat - expand documentation"
- "zmstat-chart - graphing besides just one day"
- "zmstat - integrate webpage setup and allow configuration of chart run"
- "zmstat - should have some trending data on ALL client connections (web/soap)"
zmstat IS NOT Logger (graphs in admin console)
Please note, zmstat is different than the processes and graphs involved with Logger. Logger is behind the graphs that show up in the admin web console.
Please see Logger for details and trouble-shooting steps with Logger.
Rick has some other steps for King0770-Notes#Reinitialize_the_Logger_DB
Main References For zmstat-chart Command
Please see the following:
zmstat-chart CLI Administration Guide
zmstat-chart-config CLI Administration Guide
zmstatctl CLI Administration Guide
Monitoring Administration Guide
JFreeChart is the project/binary behind the charting tool. You'll find the jar files here [ZCS 5.0.9 example]:
- /opt/zimbra/lib/jars/jfreechart-1.0.1.jar
- /opt/zimbra/jetty-6.1.5/common/lib/jfreechart-1.0.1.jar
Issues Being Investigated With zmstat And Other "Charting" Items
I currently have an internal thread going with the dev's about some issues with zmstat and also with attempts of getting the active sessions data from the admin web console to be seen in trends.
I'll just be dumping my notes here and then cleaning them up once I'm done with the internal conversation.
- Question: (mailboxd: active connections by client protocol) What exactly is this tracking? I'm on a test box with no connections and it has the different threads in the 100's/1000's. Customer was hoping (as was I) that this would be giving data much like is shown in the admin console about user connections. Also, there doesn't seem to be a comparable "active connections" for the web client (soap)?
- Answer: For the connections issue, it's the number of clients that are currently connected. There's no analog for SOAP because SOAP doesn't have persistent connections. We may be able to log stats for number of SOAP sessions, but that would include noise for cases where the user closes the browser window and the session hasn't timed out yet. If you think this is useful, please file an RFE.
- Question: (graph time plot) If I run zmstat-chart with the --aggregate-start-at option with something like "08/07/2008 01:00:00" I notice that the graphs don't actually reference "days" at the bottom but still use what looks to be "hours". Is this a bug where the bottom reference of the graph doesn't get adjusted or is the syntax to run zmstat-chart to do this more tricky that I think.
- Answer: zmstat-chart doesn't support multi-day ranges, it really only supports a single directory with a single day of charts, the aggregate options are for specifying ranges at which to calculate the max(col)/min(col)/avg(col) values. Our usual practice is to generate one set of charts for each day we want to look at; I think it gives us a better look at when peaks occur and how they compare day-to-day (it's much easier to look at them in an overlay fashion).
- Answer: Seems like a reasonable request to be able to generate charts for multiple days. Please file an RFE if you think this would be useful.
- Question: How reliable are these numbers are in admin console....
- Answer: I imagine they should be quite reliable; but HTTP (Web) is stateless, so you can't really determine whether a session is really active or not. I believe multiple sessions appearing for a single user would be the result of reloading or closing and then reopening the browser to log in again.
- Perhaps it is best to ask the users in question what their usage is like? Is there any concern over the high number of sessions? I don't think it should affect server performance.
- Answer: I imagine they should be quite reliable; but HTTP (Web) is stateless, so you can't really determine whether a session is really active or not. I believe multiple sessions appearing for a single user would be the result of reloading or closing and then reopening the browser to log in again.
- Question: Could you please explain the following:
- Mailboxd Mailbox Add Rate (Delivery Rate)
- Answer: The rate at which messages are being added to the mailbox server, e.g. 1 message being delivered per second is... 1 m/s
- Mailboxd Mailbox Add Latency (Delivery Speed)
- Answer: How long it takes on average to add a message to a mailbox (averaged over the period of a minute)
- Mailboxd Active Connections by Client Protocol- This one does NOT give us counts for web client. Host zcs2.mail.XXXX.xxx: Mailboxd: Active Connections by Client Protocol avg(IMAP) = 169.86 avg(IMAP SSL) = 0.00 avg(POP) = 0.21 avg(POP SSL) = 0.00
- Answer: It doesn't give counts for web client because the web client does not have persistent connections (being HTTP-based). SOAP requests (used by the Web client) use transient connections that often close after they are used; i.e. if you took the number of "SOAP" connections and compared it to the number of "active" sessions, it'd likely be something like 0-1% of the latter figure.
- Mailboxd Mailbox Get Count
- Answer: The number of mailboxes that got opened in the last minute (I don't recall if this counting cache hits or not?)
- Mailboxd Mailbox Get Latency
- Answer: How long it takes to open a mailbox, on average (over a minute)
- Also, we don't get stats for any of the soap parameters:
- SOAP Invocation Count Summary (Top 10 max)
- SOAP Average Call Duration AuthRequest
- Answer: I don't know why this could be; perhaps zmstat-chart-config.xml needs to be regenerated, is /opt/zimbra/zmstat/soap.csv available and does it contain data?
- Mailboxd Mailbox Add Rate (Delivery Rate)
Running zmstat-chart
Confirm it's running:
zmprov gs [mailserver hostname] | grep -i stats zimbraServiceEnabled: stats zimbraServiceinstalled: stats
Default location of stat files is:
/opt/zimbra/zmstat/
Example command to run:
mkdir /tmp/stats zmstat-chart -s /opt/zimbra/zmstat -d /tmp/stats/
Like To Have zmstat-chart Data Integrated With Zimbra
I've made an RFE for this integration:
- "zmstat - integrate webpage setup and allow configuration of chart run"
With Zimbra Jetty
- Note:
- The default Jetty behavior will NOT display directory listing. You'll need a valid html type file in the directory target.
- I would recommend the apache solution over this one as you can avoid hitting performance issues this might cause.
Dump some graphing directories in there:
zmstat-chart -s /opt/zimbra/zmstat -d /opt/zimbra/jetty/webapps/zimbra/downloads/zmstat-chart/`date +%F-%H-%M`
You'll see there's the directory and when you go into it - there's your charts.
You can now place this command in your crontab to run on a schedule basis.
Things you'll want to check before deploying:
- Place charting directory in a location where space is available and will not create a risk to the mail services if it's full.
- Look at the zmstat-chart options to see what other parameters you want to use - especially if it's going in your crontab.
- Make sure your following and security guidelines for your environment.
You'll be able to view the data with the following url:
http://hostname/zimbra/downloads/zmstat-chart/
With Zimbra Apache Or Non-Zimbra Server With Apache
Before You Proceed
Things to check or confirm first before deploying:
- Place your directory root [/opt/zimbra/support in this example] in a location where space is available and will not create a risk to the mail services if it's full.
- If you'll be using NFS to have a centralized storage point:
- Your NFS mount point on the Zimbra servers would be /opt/zimbra/support in this example.
- You'll be exporting /opt/zimbra/support in this example from the NFS server.
- If this is a non-zimbra server, copy the zimbra entry from a zimbra server's /etc/passwd and /etc/group into the non-zimbra servers passwd and group file. This way, if NFS is also used the UID/GID matches.
- Apache Directory variable will be /opt/zimbra/support/data in this example.
- Apache .htpasswd location will be /opt/zimbra/support/.htpasswd in this example, putting it above of the apache Directory variable.
- Make sure your following the security guidelines for your environment.
- Look at the zmstat-chart options to see what other parameters you want to use - especially if it's going in your crontab.
Directory Layout
As root:
mdir -p /opt/zimbra/support/data cd /opt/zimbra/support/data for i in `/opt/zimbra/bin/zmprov gas` do mkdir $i done
If your not on a ZCS server, you'll be manually creating directories named after the output of zmhostname from each of your ZCS servers.
Now, create subdirectory paths under each zmhsotname directory.
cd /opt/zimbra/support/data/ for i in `ls` do mkdir $i/debug $i/logs $i/zmstat $i/tops done
Set permissions on the directories we made:
chown -R zimbra:zimbra /opt/zimbra/support chmod -R 755 /opt/zimbra/support
With this structure and with NFS setup, you'll have a common repository that will have a standard path to use on a per server basis. Example :
script-command -PathOption /opt/zimbra/support/data/`zmhostname`/TYPE_OF_DATA/`date +%F-%H-%M`
Apache Configuration
Let's configure apache to show this directory.
- For ZCS Apache Setup:
vi /opt/zimbra/conf/httpd.conf ** Towards the end, add the following** # Include zmstat-chart directory Include /opt/zimbra/httpd/conf/extra/zimbra-support.conf
- Non-ZCS Server With Apache. Check that there's an existing Include directive that will see our zimbra-support.conf:
vi /etc/httpd/conf/httpd.conf ** Look for something that will allow the following** ** /etc/httpd/conf.d/zimbra-support.conf to be used** ** Your distro might differ on apache paths as well** ** as Include statement for other conf files** Include conf.d/*.conf
Let's now make the apache conf file for zimbra-support.conf
- For ZCS Apache Setup:
vi /opt/zimbra/httpd/conf/extra/zimbra-support.conf Alias /support "/opt/zimbra/support/data" <Directory "/opt/zimbra/support/data"> AuthName "Secure Area For Zimbra Support" AuthType Basic AuthUserFile /opt/zimbra/support/.htpasswd require valid-user Options Indexes IndexOptions FancyIndexing VersionSort AllowOverride None Order allow,deny Allow from all </Directory>
- Non-ZCS Server With Apache:
vi /etc/httpd/conf.d/zimbra-support.conf Alias /support "/opt/zimbra/support/data" <Directory "/opt/zimbra/support/data"> AuthName "Secure Area For Zimbra Support" AuthType Basic AuthUserFile /opt/zimbra/support/.htpasswd require valid-user Options Indexes IndexOptions FancyIndexing VersionSort AllowOverride None Order allow,deny Allow from all </Directory>
Make sure permissions are set correctly on zimbra-support.conf so apache will use it.
Now let's setup a username and password to use with the .htpasswd file.
cd /opt/zimbra/support/ htpasswd -c .htpasswd zimbrasupport chmod 644 .htpasswd
Restart apache so the changes take effect:
- For ZCS Apache Setup:
zmapachectl stop zmapachectl start
- Non-ZCS Server With Apache:
httpd restart
Now, let's dump some graphing directories in there:
zmstat-chart -s /opt/zimbra/zmstat -d /opt/zimbra/support/data/`zmhostname`/zmstat/`date +%F-%H-%M`
To now view the charts. Adjust, SERVERHOSTNAME, for your servername.
- For ZCS Apache Setup:
http://servername:7780/support/data/SERVERHOSTNAME/zmstat
- Non-ZCS Server With Apache:
http://servername/support/data/SERVERHOSTNAME/zmstat
You'll see there's the directory and when you go into it - there's your charts.
You can now place this command in your crontab to run on a schedule basis.
Tweaking zmstat-chart
zmstat-chart also has the option to use a configuration file. The default configuration file is:
/opt/zimbra/conf/zmstat-chart.xml
Copy this to another location and edit away. Then add the -c /location/to/config/file to use your new configuration.
Default Items Charted
Here's a list of the default items that will be charted with zmstat-chart:
- Postfix Queue Size
- convertd CPU time used
- convertd Resident Memory
- convertd Processes and Threads
- Total CPU
- Process CPU
- Process Total Memory
- Process Resident Memory
- Virtual Memory
- Context Switches
- Run/Blocked Process Queue Size
- Disk Partition Throughput
- Disk Utilization
- Disk Throughput
- Disk IOPs
- Swap Activity
- Total file descriptors open
- SOAP Invocation Count Summary (Top 10 max)
- SOAP Average Call Duration Summary (Top 10 avg)
- SOAP Invocation Count GetServerRequest
- SOAP Invocation Count BackupRequest
- SOAP Invocation Count GetAllConfigRequest
- SOAP Invocation Count GetDomainRequest
- SOAP Invocation Count GetAllDomainsRequest
- SOAP Invocation Count AuthRequest
- SOAP Average Call Duration GetServerRequest
- SOAP Average Call Duration BackupRequest
- SOAP Average Call Duration GetAllConfigRequest
- SOAP Average Call Duration GetDomainRequest
- SOAP Average Call Duration GetAllDomainsRequest
- SOAP Average Call Duration AuthRequest
- MySQL Database Connections
- MySQL InnoDB Buffer Pool Pages
- MySQL InnoDB Buffer Pool Hit Rate
- MySQL Tables Open/Opened
- MySQL Total Slow Queries Count
- Mailboxd Connection Pool Get Latency
- Mailboxd Dirty Lucene Index Writers
- Mailboxd Lucene IndexWriterCache Hitrate
- Mailboxd Lucene IO
- Mailboxd LMTP Delivery Throughput
- Mailboxd LMTP Delivery Rate
- Mailboxd Mailbox Add Rate (Delivery Rate)
- Mailboxd Mailbox Add Latency (Delivery Speed)
- Mailboxd Request Rate by Client Protocol
- Mailboxd Response Time by Client Protocol
- Mailboxd Active Connections by Client Protocol
- Mailboxd Mailbox Get Count
- Mailboxd Mailbox Get Latency
- Mailboxd Mailbox Cache Hit Rate
- Mailboxd Mailbox Item/Blob Cache Hit Rate
- Mailboxd Garbage Collection Time
- Mailboxd Garbage Collection Count
- Mailboxd JVM Heap Used
- Mailboxd JVM Heap Free
- Mailboxd JVM Permanent Generation and Code Cache
Moved to Getting_All_Users_Quota_Data
Moved to Getting_All_Users_Quota_Data
Third Party Tools (Monitoring & Logging)
FYI - Support For Real Time Counters (snmp,etc.)
This is for those that need to go beyond what zmstat-chart is doing or need to intergrate within a third party monitoring systems.
Please see RFE:
- "Support for real time counters (snmp, jmx, etc)"
Currently, this RFE has only resolved support for JMX, not SNMP. Customers who need SNMP can use a product like jManage to do the translation from JMX to SNMP.
SNMP And Zimbra
SNMP Related Bugs And RFEs
Please read the following below as they'll have little bits of information that you might need for your customizations / external setup for snmp use against Zimbra. :
- "support for real time counters (snmp, jmx, etc)"
- "zimbra snmp is broken"
- "snmp disk checks report incorrect usage."
- "replace zmdisklog / zimbra-snmp integration"
SNMP Setup On Zimbra To Notify A Remote Host
Our SNMP support is pretty basic; currently, we only send traps when a service (mta, mailbox, ldap) changes state (stop/start).
First, you'll need net-snmp package install on the zimbra host for the notifications to be sent to a remote host.
- When you install the net-snmp package, it creates /etc/snmp/snmpd.conf. This file needs to be edited with the correct community string to allow for snmp mibs to be read from remote machines The default value is "public" with the net-snmp package.
- /opt/zimbra/conf/swatchrc.in is set to use [ perlcode 0 my $snmpargs="-v 2c -c zimbra localhost "; ] for the community string. Adjust this if needed for your corporate snmp environment.
- Your security policy for your company might require you to review the other options there as well - i.e. limit to read-only.
- Also, if there is a firewall between the zimbra server and the snmp server host you'll need to open up port 161/UDP.
- When you install the net-snmp package, it creates /etc/snmp/snmpd.conf. This file needs to be edited with the correct community string to allow for snmp mibs to be read from remote machines The default value is "public" with the net-snmp package.
Then your zimbra host with the zimbra snmp service installed, do the following for a basic default setup:
zmlocalconfig -e snmp_notify=1 zmlocalconfig -e snmp_trap_host=your.host.name
/opt/zimbra/libexec/zmsnmpinit
zmswatchctl stop zmswatchctl start
We watch for something matching /err: Service status change/ and send the trap with:
/opt/zimbra/snmp/bin/snmptrap
See other topics below for customizations that might be needed for your snmp environment.
Other reference for Zimbra and SNMP:
Files To Review For SNMP
Look at the contents of the following files:
- /opt/zimbra/libexec/zmsnmpinit
- /opt/zimbra/conf/swatchrc.in
- zmsnmpinit reads swatchrc.in and writes out the file swatchrc for the running configuration
- /opt/zimbra/conf/swatchrc
- /opt/zimbra/snmp/share/snmp/snmpd.conf.in which is the SOURCE file to
- /opt/zimbra/conf/snmpd.conf [see /opt/zimbra/libexec/zmsnmpinit ]
- The two above are used by zmsnmpinit to generate the /opt/zimbra/conf/swatchrc
- /opt/zimbra/conf/snmpd.conf [see /opt/zimbra/libexec/zmsnmpinit ]
- /opt/zimbra/conf/swatchrc.in
- /opt/zimbra/net-snmp/share/snmp/mibs/zimbra.mib
- /opt/zimbra/net-snmp/share/snmp/mibs/zimbra_traps.mib
- /opt/zimbra/log/zmswatch.out
- Monitor this to see the services when they go up and down successfully send out the email notification for your snmp configuration
Zimbra MIBS
You'll find zimbra.mib and zimbra_traps.mib in the following directory. This listing is mine under 5.0.19 :
[root@mail3 ~]# cd /opt/zimbra/net-snmp/share/snmp/mibs/ [root@mail3 mibs]# ls AGENTX-MIB.txt IPV6-TC.txt SNMP-USER-BASED-SM-MIB.txt DISMAN-EVENT-MIB.txt IPV6-UDP-MIB.txt SNMP-USM-AES-MIB.txt DISMAN-SCHEDULE-MIB.txt NET-SNMP-AGENT-MIB.txt SNMP-USM-DH-OBJECTS-MIB.txt DISMAN-SCRIPT-MIB.txt NET-SNMP-EXAMPLES-MIB.txt SNMPv2-CONF.txt EtherLike-MIB.txt NET-SNMP-EXTEND-MIB.txt SNMPv2-MIB.txt HCNUM-TC.txt NET-SNMP-MIB.txt SNMPv2-SMI.txt HOST-RESOURCES-MIB.txt NET-SNMP-TC.txt SNMPv2-TC.txt HOST-RESOURCES-TYPES.txt NET-SNMP-VACM-MIB.txt SNMPv2-TM.txt IANA-ADDRESS-FAMILY-NUMBERS-MIB.txt NOTIFICATION-LOG-MIB.txt SNMP-VIEW-BASED-ACM-MIB.txt IANAifType-MIB.txt RFC1155-SMI.txt TCP-MIB.txt IANA-LANGUAGE-MIB.txt RFC1213-MIB.txt TRANSPORT-ADDRESS-MIB.txt IANA-RTPROTO-MIB.txt RFC-1215.txt UCD-DEMO-MIB.txt IF-INVERTED-STACK-MIB.txt RMON-MIB.txt UCD-DISKIO-MIB.txt IF-MIB.txt SMUX-MIB.txt UCD-DLMOD-MIB.txt INET-ADDRESS-MIB.txt SNMP-COMMUNITY-MIB.txt UCD-IPFWACC-MIB.txt IP-FORWARD-MIB.txt SNMP-FRAMEWORK-MIB.txt UCD-SNMP-MIB.txt IP-MIB.txt SNMP-MPD-MIB.txt UDP-MIB.txt IPV6-ICMP-MIB.txt SNMP-NOTIFICATION-MIB.txt zimbra.mib IPV6-MIB.txt SNMP-PROXY-MIB.txt zimbra_traps.mib IPV6-TCP-MIB.txt SNMP-TARGET-MIB.txt
What Is Looked For
Take a look at your /opt/zimbra/conf/swatchrc - this is mine under 5.0.19
perlcode 0 my %notifications=(); perlcode 0 $notifications{smtp}="yes"; perlcode 0 $notifications{snmp}="yes"; perlcode 0 my $fr='admin@mail3.zimbra.REMOVED.com'; perlcode 0 my $pwc='admin@mail3.zimbra.REMOVED.com'; perlcode 0 my $snmpargs="-v 2c -c zimbra localhost ''"; perlcode 0 my $snmptrap="/opt/zimbra/snmp/bin/snmptrap $snmpargs"; perlcode 0 my $snmpsvctrap="ZIMBRA-TRAP-MIB::zmServiceStatusTrap"; perlcode 0 my $snmpsvcname="ZIMBRA-MIB::zmServiceName"; perlcode 0 my $snmpsvcstatus="ZIMBRA-MIB::zmServiceStatus"; perlcode 0 my %statuses=('started'=>1,'stopped'=>0); perlcode 0 my $hostname="mail3.zimbra.homeunix.com"; perlcode 0 sub donotify { my %args = (@_); if ($args{HOST} eq "localhost") {$args{HOST}=$hostname;}; if ($notifications{smtp}) { dosmtp(%args) if $args{SERVICE}; dodisksmtp(%args) if $args{DISK};}; if ($notifications{snmp}) {dosnmp(%args);}; } perlcode 0 sub dosmtp { my %args = (@_); print "SMTP notification: $args{MESSAGE}\n"; open (FOO, "|/opt/zimbra/postfix/sbin/sendmail -Am -t"); print FOO "To: $pwc\nFrom: $fr\nSubject: Service $args{SERVICE} $args{STATUS} on $args{HOST}\n\n$args{MESSAGE}\n"; close FOO; } perlcode 0 sub dodisksmtp { my %args = (@_); print "SMTP notification: $args{MESSAGE}\n"; open (FOO, "|/opt/zimbra/postfix/sbin/sendmail -Am -t"); print FOO "To: $pwc\nFrom: $fr\nSubject: Disk $args{DISK} at $args{UTIL}\% on $args{HOST}\n\n$args{MESSAGE}\n"; close FOO; } perlcode 0 sub dosnmp { my %args = (@_); print "SNMP notification: $args{MESSAGE}\n"; `$snmptrap $snmpsvctrap $snmpsvcname s $args{SERVICE} $snmpsvcstatus i $statuses{$args{STATUS}}`; } ignore /DEBUG/ watchfor /err: Service status change: (\S+) (.*) changed from stopped to running/ donotify SERVICE=$2,STATUS=started,HOST=$1 watchfor /err: Service status change: (\S+) (.*) changed from running to stopped/ donotify SERVICE=$2,STATUS=stopped,HOST=$1 watchfor /err: Disk warning: (\S+) (\S+) at (\d+)/ donotify DISK=$2,UTIL=$3,HOST=$1 watchfor /crit: Disk warning: (\S+) (\S+) at (\d+)/ donotify DISK=$2,UTIL=$3,HOST=$1
Enhanced MIB Files For HP OpenView
I've created an RFE for this:
- "enchanced MIB and OID information to work with HP Openview"
Zimbra does not provide "enhanced" mib files" at this time. Thresholds can be set by the customer within their individual monitoring system. Zimbra is alerting on is service up or service down, see the other information above in Ajcody-Logging#SNMP_And_Zimbra .
Some Choices
Charting & Graphing The Data
Montoring Software
- Nagios
- Centreon - based upon Nagios
- Splunk
- You could also look at using Splunk and Nagios together.
- BigSister - take off of BigBrother
- Zenoss
- Cacti
- OpenNMS
- Munin
Nagios On Zimbra
This is a really rough draft for ideas I have in background. Shouldn't be used by anyone.
Configure Nagios to run on single server Zimbra box - Centos 5.x
Configure yum with repo and install nagios
vi /etc/yum.repos.d/Dag.repo yum update yum install nagios nagios-plugins nagios-devel nagios-plugins-nrpe
Move nagios.conf http file into main zimbra directory.
cp /etc/httpd/conf.d/nagios.conf /opt/zimbra/httpd/conf/extra/
Setup nagios to run as zimbra
vi /etc/nagios/nagios.cfg **Change nagios user to zimbra** nagios_user=zimbra nagios_group=zimbra
Change ownership of directories from nagios to zimbra.
chown -R 500:500 /var/log/nagios/ chown -R 500:500 /etc/nagios/ chown -R 500:500 /usr/share/nagios/
Configure authentication within Nagios
vi /etc/nagios/nagios.cfg # AUTHENTICATION USAGE use_authentication=1 # SYSTEM/PROCESS INFORMATION ACCESS authorized_for_system_information=nagiosadmin # CONFIGURATION INFORMATION ACCESS authorized_for_configuration_information=nagiosadmin # SYSTEM/PROCESS COMMAND ACCESS authorized_for_system_commands=nagiosadmin # GLOBAL HOST/SERVICE VIEW ACCESS authorized_for_all_services=nagiosadmin authorized_for_all_hosts=nagiosadmin # GLOBAL HOST/SERVICE COMMAND ACCESS authorized_for_all_service_commands=nagiosadmin authorized_for_all_host_commands=nagiosadmin
Set up httpasswd's for the accounts for Nagios
htpasswd -c /etc/nagios/htpasswd.users nagiosadmin htpasswd /etc/nagios/htpasswd.users guest
Configure Zimbra's http/apache to use nagios http config file
vi /opt/zimbra/conf/httpd.conf **Add the following towards bottom** # Include Nagios Include /opt/zimbra/httpd/conf/extra/nagios.conf
Starting nagios is done as root
/etc/init.d/nagios start
Restarting apache for nagios issues would be done with (as zimbra)
zmapachectl stop zmapachectl start
The webpage address to view Nagios will be like this:
http://IP_OF_SERVER:7780/nagios/
Use the rest of this how-to to configure it now: http://wiki.centos.org/HowTos/Nagios
MRTG - SNMP On Zimbra
This is a really rough draft for ideas I have in background. Shouldn't be used by anyone.
Configure yum with repo and install mrtg, net-snmp, net-snmp-utils
vi /etc/yum.repos.d/Dag.repo yum update yum instal mrtg net-snmp net-snmp-utils
Follow some how-to on setting up the basics.
Create a http config:
vi /opt/zimbra/httpd/conf/extra/mrtg.conf Alias /mrtg "/opt/zimbra/mrtg" <Directory "/opt/zimbra/mrtg"> # SSLRequireSSL Options None AllowOverride None Order allow,deny Allow from all # Order deny,allow # Deny from all # Allow from 127.0.0.1 </Directory>
Add mrtg to http configuration within zimbra:
vi /opt/zimbra/conf/httpd.conf # Include Mrtg Include /opt/zimbra/httpd/conf/extra/mrtg.conf
Restart apache:
zmapachectl stop zmapachectl start
Create directory to hold mrtg data:
mkdir /opt/zimbra/mrtg
Address will be something like:
http://IP_OF_SERVER:7780/mrtg/index.html
Mailq Pointing To Right Binary
# ls -la /usr/bin/mailq lrwxrwxrwx 1 root root 27 Sep 3 17:00 /usr/bin/mailq -> /etc/alternatives/mta-mailq # ls -la /etc/alternatives/mta-mailq lrwxrwxrwx 1 root root 23 Apr 1 10:17 /etc/alternatives/mta-mailq -> /usr/bin/mailq.sendmail # rm /usr/bin/mailq # ln -s /opt/zimbra/postfix/sbin/mailq /usr/bin/mailq # mailq Mail queue is empty
Web Client Logging
Active Sessions
Please see Zmsoap#Active_Server_Sessions_With_DumpSessionsRequest
Debug (SOAP) via Browser
See http://wiki.zimbra.com/index.php?title=Web_Client_URL_Tricks&redirect=no
Admins To View Client Issues
Within the admin console, you can view users mail.
- Goto accounts and highlight the user having the problem.
- Click on the View Mail button above that frame.
- Then goto the url field of that new window and modify it to look like this [replace mailserver with yours]:
- Hit your return key to cause the browser to reload.
- If you get warning about pop-up, accept it.
- If the debug window doesn't show, just mouse in the url field and hit the return key again. It should now pop up.
User Management Topics
Actual User Management Topics Homepage
Please see Ajcody-User-Management-Topics
Resetting A User's Account From CLI
Resetting A Password
Standard Method
SetPassword [sp] from zmprov:
zmprov sp joe@domain.com test321
Resetting Users Auth Session - Force Disconnect
Please See First - In Case Of Compromised Accounts / Spammers
Note - Restarting the mta services will be important once you reset the password/s or lock the account. It's required to ensure the active connections will be closed and any existing auth tokens no longer are valid. See:
- Force currently active SMTP authenticated sessions to be renegotiated when locking an account
Changing the Users Password To Expire Session
See Resetting A Password Via CLI or change it via the admin console.
- "Auth token should be invalidated if a user resets their password"
Invalidate sessions by removing zimbraAuthTokens
Changing zimbraAuthTokenValidityValue from the command line in Zimbra is not a reliable way to end sessions, if you use SSDB you can use flushdb as described in https://wiki.zimbra.com/wiki/Ssdb#Invalidate_all_user_sessions
To clear or reset all auth token values we need to enter token data in a particular format like "1689192272|1548369012160|8.8.15_GA_3890".
These are the steps to clear the auth tokens from an account. 1). First check few stored token for the account.
zmprov ga USERNAME@DOMAIN.COM zimbraAuthTokens | head
2). Now pick anyone token value and set it with below command, with this step only one token will be set and others will be removed.
zmprov -l ma USERNAME@DOMAIN.COM zimbraAuthTokens '1689192272|1548369012160|8.8.15_GA_3890'
3). Flush the account cache at the end.
zmprov fc account USERNAME@DOMAIN.COM
All sessions of USERNAME@DOMAIN.COM are now ended.
6.0.5+ You Have Admin Console Option
In the admin console, under the Manage Accounts window you can right click on the user name and choose "Expire Sessions".
- "Manually Invalidate Auth Token(s)"
User , Mailbox ID's, And Who Is What
ZimbraID [UserID] is system wide.
MailboxID is per server store.
To get the ZimbraID:
$ zmprov ga user@domain.com | grep -i zimbraid zimbraId: aeca260b-6faf-4cfe-b407-7673748aabf4 zimbraIdentityMaxNumEntries: 20
To get the MailboxID, get on the appropriate mailserver and:
zmprov gmi user@domain.com mailboxId: 3 quotaUsed: 251512
or globally:
/opt/zimbra/bin/mysql -e "use zimbra; select id from mailbox where account_id = 'UserID HERE including the leading 0'"
Other details can be found here:
http://wiki.zimbra.com/index.php?title=Account_mailbox_database_structure
Account & Domain Summary
Run zmaccts
Here's what it would return:
su - zimbra [zimbra@mail3 ~]$ zmaccts account status created last logon ------------------------------------ ----------- --------------- --------------- admin@mail3.internal.domain.com active 05/06/08 18:46 07/08/08 09:56 ajcody@mail3.internal.domain.com active 05/06/08 20:43 06/23/08 15:48 ajcody2@mail3.internal.domain.com active 05/28/08 11:48 06/30/08 17:44 forward@mail3.internal.domain.com active 05/06/08 21:06 05/29/08 17:24 ham.bidiob2mm@mail3.internal.domain.com active 05/06/08 18:47 never spam.rormmtcyy@mail3.internal.domain.com active 05/06/08 18:47 never wiki@mail3.internal.domain.com active 05/06/08 18:46 never account status created last logon ------------------------------------ ----------- --------------- --------------- secondary@secondary.internal.domain.com active 06/23/08 15:26 06/23/08 15:27 wiki@secondary.internal.domain.com active 06/23/08 15:25 never - domain summary - domain active closed locked maintenance total ----------------------- -------- -------- -------- ------------- -------- mail3.internal.domain 7 0 0 0 7 secondary.internal.domain 2 0 0 0 2
Last Logon comes from the variable zimbraLastLogonTimestamp . This is used to update the "Last Login Time" column in the admin web console as well. It also shows up with [ zmprov ga user@domain ]. Login's based upon session type would only be found in either audit.log or the mailbox.log files. It should have a reference to the user id and the session type for the login [ pop, imap, etc. ].
RFE To Expand zmaccts Output And Options
Please see the following RFE I made:
- "zmaccts to include more options"
Zmmailbox Stuff
RFE's And Bugs To Review
Please see these RFE's first:
- "Admin (zimbra) account to be able to setup resources for accounts (auto-acceptance)"
- http://bugzilla.zimbra.com/show_bug.cgi?id=25740
- Was marked a dup of the work being done for bug7473
- "Share management and discovery"
- "New share property that grants the user the ability to work on email but unable to delete or empty folders"
- http://bugzilla.zimbra.com/show_bug.cgi?id=31466
- In comment 4, I made an extensive suggestion on expanding the permission variables one could use.
- http://bugzilla.zimbra.com/show_bug.cgi?id=31466
Some other's to look at:
- "Expand permission share model"
- "Allow/disallow sharing to all authenticated users via user interface"
- "Calendar Share permission refinement - ability to accept/decline but NOT edit/remove"
- "Revoked view permissions not removed until after logout"
- "RFE - Ability for anyone (not just people with a share) to view some meeting details of resources"
- "After revoking calendar permissions, the web UI still shows the share still exists."
- "Cannot remove sharing permissions for mail folders"
- "Allow non-user "public" folders"
- "share ownership to disti-group -- not just end-user -- delegation (folder, calendar, doc/wiki, task, project)"
- "Global Admin control for Zimbra shared resources (and subscription) -- folders, calendar, address book, task, project, documents"
- "share roles - custom (editor / author) levels"
- "Notification of shared resources for distribution list members"
To See All Folders For A User
Do the following for the user:
[zimbra@mail3 ~]$ zmmailbox -z -m ajcody@DOMAIN gaf Id View Unread Msg Count Path ---------- ---- ---------- ---------- ---------- 1 conv 0 0 / 16 docu 0 2 /Briefcase 10 appo 0 1 /Calendar 14 mess 0 0 /Chats 7 cont 0 0 /Contacts 6 mess 0 0 /Drafts 13 cont 0 9 /Emailed Contacts 2 mess 0 11 /Inbox 4 mess 0 0 /Junk 344 mess 0 0 /Junk E-mail 12 wiki 0 0 /Notebook 302 appo 0 0 /Restored 5 mess 0 15 /Sent 420 mess 0 0 /Share 421 mess 0 0 /Share/Share1 422 mess 0 0 /Share/Share1/Share1-1 423 mess 0 0 /Share/Share2 424 mess 0 0 /Share/Share2/Share2-1 15 task 0 2 /Tasks 3 conv 0 0 /Trash
Do the following for the user [ I'm cutting some of the output to keep it short ]:
[zimbra@mail3 ~]$ zmmailbox -z -m ajcody@DOMAIN gaf -v { "id": "1", "name": "USER_ROOT", "path": "/", "parentId": "11", "flags": null, "color": "defaultColor", "unreadCount": 0, "messageCount": 0, "view": "conversation", "url": null, "effectivePermissions": null, "children": [ { #### CUT HERE #### { "id": "5", "name": "Sent", "path": "/Sent", "parentId": "1", "flags": null, "color": "defaultColor", "unreadCount": 0, "messageCount": 15, "view": "message", "url": null, "effectivePermissions": null }, { "id": "420", "name": "Share", "path": "/Share", "parentId": "1", "flags": "i", "color": "defaultColor", "unreadCount": 0, "messageCount": 0, "view": "message", "url": null, "effectivePermissions": null, "grants": [ { "type": "usr", "name": "ajcody2@mail3.internal.domain.com", "id": "88fd808e-a526-419d-9eda-ad50100d23b6", "permissions": "rwidx", "args": null }, { "type": "all", "name": null, "id": null, "permissions": "rwx", "args": null } ], "children": [ { "id": "421", "name": "Share1", "path": "/Share/Share1", "parentId": "420", "flags": "i", "color": "defaultColor", "unreadCount": 0, "messageCount": 0, "view": "message", "url": null, "effectivePermissions": null, "grants": [ { "type": "usr", "name": "ajcody2@mail3.internal.domain.com", "id": "88fd808e-a526-419d-9eda-ad50100d23b6", "permissions": "rwidx", "args": null }, { "type": "usr", "name": "admin@mail3.internal.domain.com", "id": "5ab13330-2e9b-4a45-9b30-de2c70858265", "permissions": "rwidx", "args": null } ], "children": [ { "id": "422", "name": "Share1-1", "path": "/Share/Share1/Share1-1", "parentId": "421", "flags": null, "color": "defaultColor", "unreadCount": 0, "messageCount": 0, "view": "message", "url": null, "effectivePermissions": null } ] }, { "id": "423", "name": "Share2", "path": "/Share/Share2", "parentId": "420", "flags": null, "color": "defaultColor", "unreadCount": 0, "messageCount": 0, "view": "message", "url": null, "effectivePermissions": null, "children": [ { "id": "424", "name": "Share2-1", "path": "/Share/Share2/Share2-1", "parentId": "423", "flags": null, "color": "defaultColor", "unreadCount": 0, "messageCount": 0, "view": "message", "url": null, "effectivePermissions": null } ### CUT HERE ### ] }
RFE I filed for zmmailbox to have options for this and "recursive".
- "zmmailbox folder should have option to remove ALL shares & recursive option"
Here's a script I wrote. Remove the echo statements to actually run the commands.
#!/bin/bash USER="ajcody@mail3.internal.domain.com" SHARE="/Shared" GETPERM="zmmailbox -z -m $USER gfg $SHARE" MODPERM="zmmailbox -z -m $USER mfg $SHARE" DUMBPASS="34lkoso" NEWPERM=none $GETPERM | egrep -i 'all|guest|public|accoun|domain|group' | gawk '{print $2 " " $3}' | while read SHAREPERM do TYPE=`echo $SHAREPERM|awk '{print $1}'` DISPLAY=`echo $SHAREPERM|awk '{print $2}'` case $TYPE in accoun) echo $MODPERM account $DISPLAY $NEWPERM ;; guest) echo $MODPERM $TYPE $DISPLAY $DUMBPASS $NEWPERM ;; all) echo $MODPERM $TYPE $NEWPERM ;; *) echo $MODPERM $SHAREPERM $NEWPERM ;; esac done
Ouput of an example:
[zimbra@mail3 ~]$ zmmailbox -z -m ajcody@mail3.internal.domain.com gfg /Shared Permissions Type Display ----------- ------ ------- r all r guest ajcody@domain.com r accoun admin@mail3.internal.domain.com r group mydl@mail3.internal.domain.com r domain mail3.internal.domain.com [zimbra@mail3 ~]$ /tmp/remove-share.sh zmmailbox -z -m ajcody@mail3.internal.domain.com mfg /Shared all none zmmailbox -z -m ajcody@mail3.internal.domain.com mfg /Shared guest ajcody@domain.com none zmmailbox -z -m ajcody@mail3.internal.domain.com mfg /Shared account admin@mail3.internal.domain.com none zmmailbox -z -m ajcody@mail3.internal.domain.com mfg /Shared group mydl@mail3.internal.domain.com none zmmailbox -z -m ajcody@mail3.internal.domain.com mfg /Shared domain mail3.internal.domain.com none
I then removed the echo statements:
[zimbra@mail3 ~]$ vi /tmp/remove-share.sh [zimbra@mail3 ~]$ /tmp/remove-share.sh [zimbra@mail3 ~]$ zmmailbox -z -m ajcody@mail3.internal.domain.com gfg /Shared Permissions Type Display ----------- ------ ------- [zimbra@mail3 ~]$
Please TEST this on a test box or a test account before running against a production situation. ZCS version change and commands might act different. Also note, this is a user contributed script and not one from Zimbra developers or the support staff. Also - the wiki formatting might throw of the script and could require you to fix before it runs correctly.
Script is called - zmshares - and should be named such.
#!/usr/bin/env perl # # # This program was written by Pablo Garaitonandia on Nov. 26 2012. # This program is for viewing and deleting all the shares that a user # may have in the event that removing the many shares a user has is # time consuming. # This was written on a system running Zimbra 7.2.0, RHEL 5.8, with perl v5.8.8 use strict; use warnings; use Getopt::Long; my $id=getpwuid($<); my $help=0; my $user_id=0; my $option=0; my @shares; sub view_share; sub del_share; chomp $id; if ($id ne "zimbra") { print STDERR "Error: must be run as zimbra user\n"; exit (1); } GetOptions( 'h|help' => \$help, 'u|uid=s' => \$user_id, 'o|option=s'=> \$option, ) or die "Incorrect usage!\n"; # Check for usage, definition, and correct argument types if ((defined ($user_id) && ($user_id =~ /([a-z0-9]+@[a-z.]+\.[a-z.]+)/gi)) && (defined ($option) && (($option eq "delete") || ($option eq "view")))) { print "\n$option: shares for $user_id \n\n"; } elsif ($help) { usage(); } else { usage(1); } if ($option eq "view"){ view_share();} if ($option eq "delete"){ del_share();} sub view_share { open(VIEW, "/opt/zimbra/bin/zmprov getShareInfo $user_id |"); print <VIEW>; } sub del_share { open(SHARES, "zmprov getShareInfo $user_id |awk '{print substr(\$0,70,6) ,substr(\$0,131,36), substr(\$0,168,15)}' | awk 'NR>2' |"); @shares = <SHARES>; if (!(@shares)){ print "EXITING: User has no shares to delete. \n\n"; exit (1); } foreach my $share (@shares){ my @line = split(/\s+/, $share); if (defined ($line[2])){ print "zmmailbox -z -m $user_id mfg $line[0] account $line[1] none\n"; system("/opt/zimbra/bin/zmmailbox -z -m $user_id mfg $line[0] account $line[1] none") == 0 or die "Command Failed"; } else { print "zmmailbox -z -m $user_id mfg $line[0] account \"\" none\n"; system("zmmailbox -z -m $user_id mfg $line[0] account \"\" none") == 0 or die "Command Failed"; } } } sub usage { my ($msg) = (@_); $msg && print STDERR "\nINCORRECT USAGE: $msg\n"; print STDERR <<USAGE; zmshares -u username\@domain -o (delete|view) Where: -u: (user\@domain) The full user id with domain for user. -o: (delete|view) Delete or view ALL shares for the user USAGE exit (1); } __END__
- I've yet to test these against all items (resources) listed in bug 25740 and work as expected.
To see current perms
zmmailbox -z -m faxfinder@example.com gfg /Inbox
To modify perms:
- r = read
- w = write
- i = insert
- d = delete
- x = accept/decline invites
- a = administer
zmmailbox -z -m faxfinder@example.com mfg /Inbox account user@example.com rwidx
To confirm perms are set:
zmmailbox -z -m faxfinder@example.com gfg /Inbox
To mount "folder" into a user account that was given permission:
zmmailbox -z -m user@example.com cm --view message "/Incoming_Faxes" faxfinder@example.com /Inbox
To confirm folder is mounted:
zmmailbox -z -m user@example.com gaf
Additions notes/options see:
zmmailbox help folder
For mfg it shows it can take the below as a target:
- account {name}
- group {name} *This could be a DL?*
- domain {name}
- all
- public
- guest
Scripting note to do this with multiple users:
- zmmailbox cm could use the zmprov gaa to provide a list of all accounts, this would include system & archive (if exist) accounts though.
How To Turn Off Sharing
You can enable / disable sharing from admin console:
- Admin console --> class of service --> select the CoS (eg default) --> features --> general features --> check/uncheck 'Sharing' option
Alternatively, this can be achieved by having the following CoS attribute either 'TRUE' or 'FALSE', from command line: zimbraFeatureSharingEnabled
Searches With zmmailbox
Special Note If Your Search String Needs Spaces
Here is an example using the correct format to include required spaces to have your search do what you want. For instance, many shared folders will end up using, by default, spaces in the folder name.
$ zmmailbox -z -m ajcody@`zmhostname` gaf | grep appo 10 appo 0 0 /Calendar 263 appo 0 2 /Large Share's Calendar (large-share@mail71.DOMAIN.com:10) $ zmmailbox -z -m ajcody@`zmhostname` s -t appo in:"\"Large Share's Calendar"\" num: 2, more: false Id Type From Subject Date ---------------------------------------- ---- ---------- --------------------- ------ 1. 799efb72-2e6b-400a-8881-c5f9d7c282b1:265 appo <na> Test On Thu 10/28/10 00:02 2. 799efb72-2e6b-400a-8881-c5f9d7c282b1:263 appo <na> test for friday 10/28/10 00:02
Note, the "\"Text1 Text2"\" is for a [s option] search string query, when querying for the folder name with other zmmailbox options - normal quoting works. For example:
$ zmmailbox -z -m ajcody@`zmhostname` gfg "Large Share's Calendar" Permissions Type Display ----------- -------- ------- rwidxa account ajcody@mail71.DOMAIN.com
Search For Messages And Then Delete Them
Here's some examples to grab the message id's from a search and then put them in a variable to use for the delete command.
Other reference: King0770-Notes#Removing_Messages_with_Zmmailbox_based_on_the_Subject
Note - Crossmailbox Search And Delete Is Currently An RFE
See the following :
- RFE: Bulk deletion of a mail - crossmailbox
First - Default Search Returns Only 25 Results
From zmmailbox [help search] & zmmailboxsearch
--limit (optional) -l Sets the limit for the number of results returned. The default is 25.
Example Search With A From And To Date - Multiple Variable Search
This allows you to restrict your search in-between a date range.
zmmailbox -z -m user@domain.com s -t message -l 999 "before:6/15/2011 and after:6/9/2011"
Note - If your trying to do this for a tgz export, please see the following :
Example Search With To Field
[zimbra@mail3 ~]$ zmmailbox -z -m ajcody@mail3.internal.DOMAIN.com s -t message "To: Adam" num: 4, more: false Id Type From Subject Date ---- ---- -------------------- -------------------------------------------------- -------------- 1. 269 mess Adam Re: 8-7-08 11:37 AM to both outside accounts 08/07/08 11:57 2. 268 mess Adam Re: 8-7-08 11:37 AM to both outside accounts 08/07/08 11:39 3. 266 mess Adam Re: 8-7-08 11:37 AM to both outside accounts 08/07/08 11:38 4. 263 mess Adam Re: test on 8-7-08 to zimbra account 08/07/08 11:37 [zimbra@mail3 ~]$ zmmailbox -z -m ajcody@mail3.internal.DOMAIN.com s -t message "To: Adam" |awk '{ if (NR!=1) {print}}'| grep mess | awk '{ print $2 "," }' | tr -d '\n' 269,268,266,263, [zimbra@mail3 ~]$ message=`zmmailbox -z -m ajcody@mail3.internal.DOMAIN.com s -t message "To: Adam" |awk '{ if (NR!=1) {print}}'| grep mess | awk '{ print $2 "," }' | tr -d '\n'` [zimbra@mail3 ~]$ zmmailbox -z -m ajcody@mail3.internal.DOMAIN.com dm `echo $message` [zimbra@mail3 ~]$ zmmailbox -z -m ajcody@mail3.internal.DOMAIN.com s -t message "To: Adam" num: 0, more: false
Example Search With From Field
[zimbra@mail3 ~]$ zmmailbox -z -m ajcody@mail3.internal.DOMAIN.com s -t message "From: Adam" num: 8, more: false Id Type From Subject Date ---- ---- -------------------- -------------------------------------------------- -------------- 1. 464 mess Adam test 3 10/02/08 11:43 2. 463 mess Adam test 2 10/02/08 11:43 3. 462 mess Adam test 1 10/02/08 11:43 4. 461 mess Adam test 09/29/08 16:18 5. 460 mess Adam test for mailbox log 09/29/08 16:17 6. 265 mess Adam 8-7-08 11:37 AM to both outside accounts 08/07/08 11:38 7. 261 mess Adam test on 8-7-08 to zimbra account 08/07/08 11:36 8. 257 mess Adam test from zimbra on 8-7-08 08/07/08 11:27 [zimbra@mail3 ~]$ zmmailbox -z -m ajcody@mail3.internal.DOMAIN.com s -t message "From: Adam" |awk '{ if (NR!=1) {print}}'| grep mess | awk '{ print $2 "," }' | tr -d '\n' 464,463,462,461,460,265,261,257, [zimbra@mail3 ~]$ message=`zmmailbox -z -m ajcody@mail3.internal.DOMAIN.com s -t message "From: Adam" |awk '{ if (NR!=1) {print}}'| grep mess | awk '{ print $2 "," }' | tr -d '\n'` [zimbra@mail3 ~]$ echo $message 464,463,462,461,460,265,261,257, [zimbra@mail3 ~]$ zmmailbox -z -m ajcody@mail3.internal.DOMAIN.com dm `echo $message` [zimbra@mail3 ~]$ zmmailbox -z -m ajcody@mail3.internal.DOMAIN.com s -t message "From: Adam" num: 0, more: false [zimbra@mail3 ~]$
More Search Possibilities
Please see [Search Tips]
Export & Import Of Users Data In TGZ Format
Please see Ajcody-Migration-Notes#ZCS_User_to_Another_ZCS_Server_-_With_Rest_.26_TGZ
Seeing What & Where Of A Message ID
If your need to figure out what the actual email/message is from a logging event.
For example, log shows:
2009-03-03 22:04:58,969 INFO [btpool0-5532] [name=USER@DOMAIN.com;mid=8;ip=10.0.0.1;ua=ZimbraWebClient - IE6 (Win)/5.0.11_GA_2695.UBUNTU8_64;] mailop - moving Message (id=10955) to Folder Trash (id=3)
To see the details of the message, do the following:
zmmailbox -z -m USER@DOMAIN gm 10955 Id: 10955 Conversation-Id: 11155 Folder: /Trash Subject: FW: How are you doing? From: User External <USER@DOMAIN.com> To: <USER@DOMAIN.com> ...etc...
Message Count Mismatches
Message Count Via zmprov
To see a listing of message count in folders, replace USER@DOMAIN w/ user:
zmmailbox -z -m USER@DOMAIN gaf
You can also do something like this:
zmmailbox -z -m USER@DOMAIN s -t mess in:"FOLDER_IN_QUESTION"
If the folder has spaces, use the following format : "\"Large Share's Calendar"\"
zmprov rmc RecalculateMailboxCounts
From the zmprov help for rmc:
RecalculateMailboxCounts rmc {name@domain|id} When unread message count and quota usage are out of sync with the data in the mailbox, use this command to immediately recalculate the mailbox quota usage and unread messages count. Important: Recalculating mailbox quota usage and message count should be schedule to run in off peak hours and used on one mailbox at a time. Example: $zmprov rmc user@domain
Users should log into a new ZWC session after this was done.
If User Is Using IMAP Client
We have some bugs/rfe's in regards to how various IMAP clients operate with their delete/purge and it's impact on the our message counting.
Here's a recent one:
- "Item count should account for \Deleted IMAP messages"
- http://bugzilla.zimbra.com/show_bug.cgi?id=20620
- Resolved with 6.0.8
- http://bugzilla.zimbra.com/show_bug.cgi?id=20620
One work around was by configuring the IMAP client to move messages to a Trash/Deleted Items folder [if available] and to delete/purge messages immediately or upon sign off.
Check The Message Blobs On The File System
This is more of a sanity check, confirming the user does have what you would estimate for message blobs on the file system under their message store path.
See: Ajcody-Mysql-Topics#How_To_Locate_Users_Mailstore_and_Message_Store_Directory
You might also see "No Such Blob" messages in the ZWC client and the mailbox.log file.
See: Ajcody-Notes-No-Such-Blob
Make Sure Your Not Auto-purging Messages
These are set at the global or server level.
zmprov gacf | egrep "zimbraMailPurgeSleepInterval|zimbraMailTrashLifetime|\ zimbraMailSpamLifetime|zimbraMailMessageLifetime" zmprov gs server.domain.com | egrep "zimbraMailPurgeSleepInterval|\ zimbraMailTrashLifetime|zimbraMailSpamLifetime|zimbraMailMessageLifetime"
These at the user level:
zmprov ga user@domain | egrep "zimbraPrefInboxReadLifetime|zimbraPrefInboxUnreadLifetime|\ zimbraPrefSentLifetime|zimbraPrefJunkLifetime|zimbraPrefTrashLifetime"
Reference:
Managing Legal Requests for Information
Description:
- The ZCS legal intercept feature is used to obtain copies of email messages that are sent, received, or saved as drafts from targeted accounts and send these message to a designated “shadow” email address. Legal Intercept can be configured to send the complete content of the message or to send only the header information. When a targeted account sends, receives, or saves a draft message, an intercept message is automatically created to forward copies of the messages as attachments to the specified email address.
Please see:
- http://www.zimbra.com/docs/ne/latest/administration_guide/managing_other_zcs_features.8.1.html
- http://wiki.zimbra.com/index.php?title=Legal_Intercept
- http://bugzilla.zimbra.com/show_bug.cgi?id=17539
Persona, Identities, Send As, Send On Behalf Of Issues
For ZCS 8 And Above You Must Grant ACL Rights For sendAs and sendAsDistList for internal users
This section below only applies to versions ZCS 6 and ZCS 7. For ZCS 8 and above, the zimbraAllowFromAddress variable only is valid for external accounts and can NOT be used for internal users or ZCS distribution lists [DL's]. Please see the following for ZCS 8+
CLI Commands To Manage Persona, Identities, External Account
The following should provide you with the necessary commands to manage these user configurations:
zmprov help command| grep -i data createDataSource(cds) {name@domain} {ds-type} {ds-name} zimbraDataSourceEnabled {TRUE|FALSE} zimbraDataSourceFolderId {folder-id} [attr1 value1 [attr2 value2...]] deleteDataSource(dds) {name@domain|id} {ds-name|ds-id} getDataSources(gds) {name@domain|id} [arg1 [arg2...]] modifyDataSource(mds) {name@domain|id} {ds-name|ds-id} [attr1 value1 [attr2 value2...]]
zmprov help command| grep -i identit createIdentity(cid) {name@domain} {identity-name} [attr1 value1 [attr2 value2...]] deleteIdentity(did) {name@domain|id} {identity-name} getIdentities(gid) {name@domain|id} [arg1 [arg...]] modifyIdentity(mid) {name@domain|id} {identity-name} [attr1 value1 [attr2 value2...]]
Bugs And RFE's To Look At
Send As Issues
- "support sendAs right on server (as opposed to on-behalf-of)"
- http://bugzilla.zimbra.com/show_bug.cgi?id=22819
- "Composer should allow user to send message as self if replying on-behalf-of"
- "Implement "sendAs" rights for user accounts"
- http://bugzilla.zimbra.com/show_bug.cgi?id=22819
- "save copy of send-as message to sent-as user's Sent folder"
On Behalf Of Issues
- "send on behalf of for delegate access for ZWC"
- "reply to message in shared subfolder doesn't follow typical on behalf of behavior"
- "Need "Send on behalf of" pref"
- "Make the checkbox configurable for "Sent on behalf of""
Persona Setup With Send As [zimbraAllowFromAddress] Rights Rather Than On Behalf Of
This section below only applies to versions ZCS 6 and ZCS 7. For ZCS 8 and above, the zimbraAllowFromAddress variable only is valid for external accounts and can NOT be used for internal users or ZCS distribution lists [DL's]. Please see the following for ZCS 8+
- This was tested against ZCS 6.0.8p1 .
Attribute descriptions - 608 :
zimbra-attrs.xml:<attr id="427" name="zimbraAllowAnyFromAddress" type="boolean" cardinality="single" optionalIn="account,cos" flags="accountInfo,accountInherited"> zimbra-attrs.xml:<attr id="428" name="zimbraAllowFromAddress" type="email" max="256" cardinality="multi" optionalIn="account" flags="accountInfo,domainAdminModifiable">
- First, created a test user account:
- ajcody@rr608.zimbra.DOMAIN.com
- In the admin web console, under the users preferences tab :
- Sending Mail > checked : "Allow sending email from any address"
zmprov ma ajcody@rr608.zimbra.DOMAIN.com zimbraAllowAnyFromAddress TRUE
- Note, this could be setup in a COS as well and then assign the users you want to that COS
- If this is to permissive, because it does allow the user to send as anybody, then you'll want to do this instead:
zmprov ma ajcody@rr608.zimbra.DOMAIN.com zimbraAllowAnyFromAddress FALSE
zmprov ma ajcody@rr608.zimbra.DOMAIN.com zimbraAllowFromAddress personal-dl@rr608.zimbra.DOMAIN.com
- Note - bug alert.
- Testing shown that you could still have a persona setup for a particular address that wasn't set for the zimbraAllowFromAddress variable and zimbraAllowAnyFromAddress is set as FALSE. You'll be able to select it when composing a message and the message is sent with no error. But, what happens is the email is delivered to the recipient with your primary account details rather than the persona's.
- Note - bug alert.
- Sending Mail > checked : "Allow sending email from any address"
- In the admin web console, under the users preferences tab :
- ajcody@rr608.zimbra.DOMAIN.com
- If your only using a DL for the mail traffic, you would:
- Create a new DL :
- persona-dl@rr608.zimbra.DOMAIN.com
- checked "Can receive email"
- Added a user/s to the DL:
- ajcody@rr608.zimbra.DOMAIN.com
- persona-dl@rr608.zimbra.DOMAIN.com
- Create a new DL :
- Now, once that is done we can setup the persona for our "test user" - ajcody. Login as testuser
- Create a Folder called "Persona DL" and then a filter rule to move all emails with persona-dl@rr608.zimbra.DOMAIN.com to the "Persona DL" folder.
- Under the users perferences, Mail > Accounts > Add Persona button:
- Persona Name : Persona DL
- From : Persona DL # personal-dl@rr608.zimbra.DOMAIN.com
- Reply-To : Persona DL # personal-dl@rr608.zimbra.DOMAIN.com
- Use this persona:
- check "when replying or forwarding messages sent to: Persona DL # personal-dl@rr608.zimbra.DOMAIN.com
- check "when replying or forwarding messages in folder(s) : Personal DL
- Persona Name : Persona DL
- Under the users perferences, Mail > Accounts > Add Persona button:
- Create a Folder called "Persona DL" and then a filter rule to move all emails with persona-dl@rr608.zimbra.DOMAIN.com to the "Persona DL" folder.
- Things to note when using persona
- A new message in the "From" section will give a drop down for your persona choice.
- It's best to create a mail filter rule to put related messages for the persona account into a folder and then in the persona choose the option about using the persona as the default when replying to messages in that folder.
This section below only applies to versions ZCS 6 and ZCS 7. For ZCS 8 and above, the zimbraAllowFromAddress variable only is valid for external accounts and can NOT be used for internal users or ZCS distribution lists [DL's]. Please see the following for ZCS 8+
- The below how-to was tested against ZCS 6.0.8p1 .
- First, created a test user account:
- ajcody@rr608.zimbra.DOMAIN.com
- In the admin web console, under the users preferences tab :
- Sending Mail > checked : "Allow sending email from any address"
zmprov ma ajcody@rr608.zimbra.DOMAIN.com zimbraAllowAnyFromAddress TRUE
- Note, this could be setup in a COS as well and then assign the users you want to that COS
- If this is to permissive, because it does allow the user to send as anybody, then you'll want to do this instead:
zmprov ma ajcody@rr608.zimbra.DOMAIN.com zimbraAllowAnyFromAddress FALSE
zmprov ma ajcody@rr608.zimbra.DOMAIN.com zimbraAllowFromAddress personal-source@rr608.zimbra.DOMAIN.com
- Note - bug alert.
- Testing shown that you could still have a persona setup for a particular address that wasn't set for the zimbraAllowFromAddress variable and zimbraAllowAnyFromAddress is set as FALSE. You'll be able to select it when composing a message and the message is sent with no error. But, what happens is the email is delivered to the recipient with your primary account details rather than the persona's.
- Note - bug alert.
- Sending Mail > checked : "Allow sending email from any address"
- In the admin web console, under the users preferences tab :
- ajcody@rr608.zimbra.DOMAIN.com
- If I was only using a "shared mailbox" for the mail traffic, I would:
- First create a DL that will have the user accounts you want to share this 'new' mailbox [Inbox]:
- Create a new DL:
- persona-share@rr608.zimbra.DOMAIN.com
- checked "Can receive email"
- Added a user to the DL:
- ajcody@rr608.zimbra.DOMAIN.com
- persona-share@rr608.zimbra.DOMAIN.com
- Create a new DL:
- First create a DL that will have the user accounts you want to share this 'new' mailbox [Inbox]:
- The create a new account/mailbox that others will share:
- persona-source@rr608.zimbra.DOMAIN.com
- From the 'admin console', do "View Mail" on the new account
- Share the Inbox to the DL : persona-share@rr608.zimbra.DOMAIN.com w/ Manager or Admin Rights
- From the 'admin console', do "View Mail" on the new account
- persona-source@rr608.zimbra.DOMAIN.com
- Log back into the 'test user' acocunt - ajcody@rr608.zimbra.DOMAIN.com
- Accept the share and confirm you see the "Inbox" from the "persona-source" account.
- Then, under the users perferences, Mail > Accounts > Add Persona button::
- Persona Name : Persona Source
- From : Persona Source # persona-source@rr608.zimbra.DOMAIN.com
- Reply-To : Persona Source # persona-source@rr608.zimbra.DOMAIN.com
- Use this persona:
- check "when replying or forwarding messages sent to: persona-source@rr608.zimbra.DOMAIN.com
- check "when replying or forwarding messages in folder(s) : Persona Source's Inbox
- Persona Name : Persona Source
- Then, under the users perferences, Mail > Accounts > Add Persona button::
- Accept the share and confirm you see the "Inbox" from the "persona-source" account.
- Things to note when using persona
- A new message in the "From" section will give a drop down for your persona choice.
- It's best to create a mail filter rule to put related messages for the persona account into a folder and then in the persona choose the option about using the persona as the default when replying to messages in that folder.
- Need An RFE/BUG Report? - When you have a shared mailbox folder, the 'normal' operation when replying to messages from that folder is to send them "on behalf of". You don't want this option, since your wanting to use the persona rules. You might need to "uncheck" the box under the new message that says:
- uncheck box for "Send this message on behalf of: persona-source@rr608.zimbra.DOMAIN.com"
- I couldn't find a way to have this "unchecked" as the default.
Sieve Rules
Administrating Rules For Users - CLI
Please see King0770-Notes-Sieve_Rules_By_Proxy
Ajcody Server Topics
I moved the following pages out of this section to Ajcody-Server-Plan-Move-Migration-Upgrade-DR. It was getting to large to edit and load in web browser.
- Ajcody-Notes-ServerPlanning , was referenced as Backup Plans And Cheap HA-DR Secondary Server
- Ajcody-Migration-Notes
- Ajcody-Notes-Server-Move , also referenced as Moving To A New Server
- Ajcody-Notes-Multi-Server-Restore-DR
- Ajcody-Disaster-Recovery-Specific-Notes
- Ajcody-Notes-Upgrade-Options
Server Topics
Actual Server Topics Homepage
Please see Ajcody-Server-Topics
Issues Being Investigated
Actual Server Issues Being Investigated Homepage
Please see Ajcody-Server-Issues-Being-Investigated
License Issues
Getting License Details Via Soap - Consumed Licenses
First, flush the cache on all servers for the license information. Then do the query via soap.
zmprov fc -a license zmsoap -z GetLicenseRequest
Two lines that are generally of interest are, for example :
<attr name="TotalAccounts">3</attr> <attr name="ArchivingAccounts">3</attr>
What Should Count Against License
Real accounts, as listed in admin console under Addresses > Accounts. The admin account will count against this but the ham, spam, and wiki ones will not.
Accounts or entries listed under : Aliases , Distribution Lists, Resources should not. See below for bug about Resources though.
Deleted Accounts Still Show In Use
Please see:
- "deleted accounts don't update license count till server restart"
Another suggestion if zimbra restarts don't work, as zimbra:
zmprov fc license
When you have multi ZCS servers:
zmprov fc -a license
Resources Counting Against License
Please see:
- "resources should not count against the licensed user count"
Workaround is :
zmcontrol stop
zmcontrol start
Tested on 5.0.9
- license count 5
- created two resources
- license count 7
zmcontrol stop
zmcontrol start
- license count 5
Performance Issues When Using Mini-Cal And You Have zimbraMailCanonicalAddress Set To Domains You Don't Have
Background Bugs:
Do you have any user's with the variable zimbraMailCanonicalAddress set using a domain that is not within your Zimbra infrastructure? There was case that had that set for a particular user to a domain they didn't have within Zimbra and the symptom showed as a performance issue within the mini-calendar & calendar. The root cause was actually the ldap lookups occurring in the background (those against the zimbraMailCanonicalAddress domain).
- One work around was setting:
zmlocalconfig -e ldap_starttls_supported=0
ldap stop
ldap start
zmlocalconfig -e zimbra_require_interprocess_security=0
- To update the postfix configuration files.
/opt/zimbra/libexec/zmmtainit
- To update amavis config files.
/opt/zimbra/libexec/zmmtaconfig amavis
- Then restart the system. Still need to double check this will be necessary.
- See release notes about ldap_starttls_supported
- And bug: http://bugzilla.zimbra.com/show_bug.cgi?id=24762
- Also see the below bug, comments 75 onward:
- The other workaround was to remove the zimbraMailCanonicalAddress variable.
5.0.7+ Performance & Hanging Issues
Administrators might or might not catch this events being tied to calendars or ics data. Here's what I've gather from other cases so far about the issue, there's about 5 of them I've seen. None are resolved at this time (July 23, 08), so use with caution.
1. bug: http://bugzilla.zimbra.com/show_bug.cgi?id=29596 The resolution for this bug would involve an upgrade to 5.0.8 .
- One customer has reported the upgrade to 5.0.8 has resolved their issue so far. They also confirm that the ics files were being processed with much faster times as logged in mailbox.log
- Second customer has confirmed upgrade to 5.0.8 has resolved their issue.
2. Check a the thread dump if the message is getting stuck during an invite email delivery to a conference room. You can guess the calendar object based on the emails in the conference room's Inbox.
3. Also check their recurrence expansion configuration in LDAP with:
"zmprov gacf | grep zimbraCalendarRecurrence".
On a clean install you should see:
zimbraCalendarRecurrenceDailyMaxDays: 730 zimbraCalendarRecurrenceMaxInstances: 0 zimbraCalendarRecurrenceMonthlyMaxMonths: 360 zimbraCalendarRecurrenceOtherFrequencyMaxYears: 1 zimbraCalendarRecurrenceWeeklyMaxWeeks: 520 zimbraCalendarRecurrenceYearlyMaxYears: 100
If these are set to 0, the sysadmin enabled near-infinite expansion on purpose. If these are missing, it's an upgrade problem. The code will default the values to 0 and thus infinite loop. Set them to the above values to avoid long expansions.If these are set to 0, please set to the above values to avoid long exp.
- One customer has reported that the variables weren't set and they set them to the defaults. Restarted zimbra and issues appear to be resolved. They are holding off on 5.0.8 upgrade at this point.
4. Also you are might hitting bug ( http://bugzilla.zimbra.com/show_bug.cgi?id=28397 - this is a private bug) or something similar like this caused by an offending appointment. You can also find out the mailbox (conference room/user) and put it into maintenance mode to keep the mails flowing. Then try to flush the queue.
Upgrade Issues
Please check the Support Portal page for the most recent issues related to newly released ZCS versions. That is were "issues" are generally posted when we discover "new" situations arising from newly released versions.
Very Long Upgrade Times
zmfixperms Causing Long Upgrade Times - HSM Configurations Effected Usually
Please see:
- "zmfixperms : pass in directories not to touch OR don't include other dirs unless -extended specified"
Upper Case Hostname Causes Problems With Install/Upgrade
I believe this is new for version 5.0.8+. Until you adjust the case, the installer script will not continue. It's usually picking up the upper case hostname from the server's /etc/hosts entry. Please don't do this, use upper case in your hosts file ... Unix is not Windows.
- Make the install case insensitive on hostname
LDAP Topics
Actual LDAP Topics Homepage
Please see Ajcody-LDAP-Topics
LDAP Error Codes
A nice reference on ldap errors codes:
Working With ldap Commands - The Easy Way
Tired of working with long ldap strings on the CLI with ldapmodify, ldapsearch, ldapdelete.
zimbra$ source ~/bin/zmshutil zimbra$ zmsetvars
You can now use this string syntax to make things a little easier.
ldapCOMMAND -x -H $ldap_master_url -D $zimbra_ldap_userdn -w $zimbra_ldap_password ACTION
Anonymous Binds
For new installations of ZCS 6.0, anonymous searches of the directory are disabled. (Bug 15378) When you upgrade to 6.0, anonymous searches of the directory are enabled, matching previous release behavior.
- To disable anonymous search after upgrading, on each LDAP server, as zimbra run
/opt/zimbra/libexec/zmldapanon -d
- To enable anonymous access at any point after it is disabled, on each LDAP server run
/opt/zimbra/libexec/zmldapanon -e
Important: Enabling anonymous binds is not recommended as it may expose data not intended for anonymous access
Ref:
- "Obviate the need for and disallow LDAP anonymous binds"
Searching For Account In LDAP
To do ldapsearch for the account, and see if there are any related entries in ldap.
su - zimbra source ~/bin/zmshutil zmsetvars ldapsearch -x -H $ldap_master_url -D $zimbra_ldap_userdn -w $zimbra_ldap_password "mail=USER@DOMAIN.com"
To just dump the whole thing [to grep and so forth]:
su - zimbra source ~/bin/zmshutil zmsetvars ldapsearch -x -H $ldap_master_url -D $zimbra_ldap_userdn -w $zimbra_ldap_password
Searching For Accounts On A Particular Mailstore
To do ldapsearch for the account, and see if there are any related entries in ldap.
su - zimbra source ~/bin/zmshutil zmsetvars ldapsearch -x -H $ldap_master_url -D $zimbra_ldap_userdn -w $zimbra_ldap_password "zimbraMailHost=YOURMAILSTORE.COM"
Other LDAP Search Example
See http://wiki.zimbra.com/wiki/ShanxT-LDAP-CheatSheet
ldapsearch over 636 from non-ZCS linux client
From your ZCS ldap server, you'll want to get the following information:
zmhostname zmlocalconfig -s | egrep 'ldap_master_url|ldap_url|ldap_starttls_supported|ldap_port|zimbra_ldap_password'
Replace the details below as followed: ZMHOSTNAME with zmhostname's output above, PASSWORD with zimbra_ldap_password from above. Also adjust the -b 'ou=people,dc=DOMAIN,dc=com' to reflect your domain string. The other variables should state port 636 if your setup for ldaps [ref: How_to_enable_ldaps Now try the following now on the ldap server below to confirm ldapsearch works :
ldapsearch -x -H ldaps://ZMHOSTNAME:636 -D uid=zimbra,cn=admins,cn=zimbra -w PASSWORD -LLL -b 'ou=people,dc=DOMAIN,dc=com' -Z
Now, example uses a centos/rhe machine and therefor I use yum vs apt-get, let's setup the non-ZCS linux machine to test the query:
yum install openldap-clients vi /etc/openldap/ldap.conf * add TLS_REQCERT allow
And now the search test from the non-zcs linux host [I provide a couple lines of the output in the example below]. Remember to replace the ZMHOSTNAME and PASSWORD values below with your variables and the -b 'ou=people,dc=DOMAIN,dc=com' to reflect your domain string. :
ldapsearch -x -H ldaps://ZMHOSTNAME:636 -D uid=zimbra,cn=admins,cn=zimbra -w PASSWORD -LLL -b 'ou=people,dc=DOMAIN,dc=com' -Z ldap_start_tls: Operations error (1) additional info: TLS already started dn: ou=people,dc=DOMAIN,dc=com objectClass: organizationalRole ou: people cn: people
Deleting An Account In LDAP
Via zmprov
- Caution - this should normally only be done under guidance or request by support.
This will delete the account from ldap and not from the db (mysql). It also retains the mail store and index data:
zmprov -l da <USER@DOMAIN.com>
Via ldapdelete
- Caution - this should normally only be done under guidance or request by support.
One way:
su - zimbra source ~/bin/zmshutil zmsetvars ldapsearch -LLL -x -H $ldap_master_url -D $zimbra_ldap_userdn -w $zimbra_ldap_password "mail=ajcody@zimbra.DOMAIN.com" dn dn: uid=ajcody,ou=people,dc=zimbra,dc=DOMAIN,dc=com ldapdelete -r -x -H $ldap_master_url -D $zimbra_ldap_userdn -w $zimbra_ldap_password uid=ajcody,ou=people,dc=zimbra,dc=DOMAIN,dc=com
Once done, you should be able to add or remove the account using 'zmprov ca' command.
db_recover For OpenLdap DB In /opt/zimbra/openldap-data
- Caution - this should normally only be done under guidance or request by support.
For ZCS 5.x
This will cause an impact to your user base, since we'll be stopping the ldap service.
su - zimbra cd /opt/zimbra/openldap-data ls -la **Notice the __db.00# files** ldap stop /opt/zimbra/sleepycat/bin/db_recover ls -la **Notice the __db.00# files are gone** ldap start
Attempt To Cover Versions Higher Than ZCS5 - I've yet to confirm the below
ZCS 6.X.X
# su - zimbra $ ldap stop $ cd /opt/zimbra/openldap-data $ /opt/zimbra/sleepycat/bin/db_recover
and also, if this is a ldap master:
$ cd /opt/zimbra/openldap-data/accesslog/db $ /opt/zimbra/sleepycat/bin/db_recover
ZCS 7.0.x
unconfirmed
7.1.x
# su - zimbra $ ldap stop $ cd /opt/zimbra/data/ldap/hdb/db $ /opt/zimbra/bdb/bin/db_recover
7.1.x Ldap Replica & Mailstore - Case Notes
Error customer had in /var/log/zimbra.log after /opt/zimbra partition hit 100% usage.
Aug 25 15:51:12 SERVER postfix/smtpd[20470]: NOQUEUE: reject: MAIL from localhost.localdomain[127.0.0.1]: 452 4.3.1 Insufficient system storage; proto=ESMTP helo=<localhost> Aug 25 15:51:12 SERVER amavis[16494]: (16494-03) smtp resp to MAIL (pip): 452 4.3.1 Insufficient system storage and later.... Aug 25 17:34:27 SERVER slapd[5627]: @(#) $OpenLDAP: slapd 2.4.26 (Sep 7 2011 12:24:16) $#012#011build@zre-rhel6-64.eng.vmware.com:/home/build/p4/HELIX/ThirdParty/openldap/openldap-2.4.26.5z/servers/slapd Aug 25 17:34:27 SERVER slapd[5628]: hdb_db_open: database "": unclean shutdown detected; attempting recovery. Aug 25 17:34:27 SERVER slapd[5628]: bdb(): file unknown has LSN 146/8895192, past end of log at 146/7885578 Aug 25 17:34:27 SERVER slapd[5628]: bdb(): Commonly caused by moving a database from one database environment Aug 25 17:34:27 SERVER slapd[5628]: bdb(): to another without clearing the database LSNs, or by removing all of Aug 25 17:34:27 SERVER slapd[5628]: bdb(): the log files from a database environment Aug 25 17:34:28 SERVER slapd[5628]: bdb(): file id2entry.bdb has LSN 146/8895192, past end of log at 146/7887443 Aug 25 17:34:28 SERVER slapd[5628]: bdb(): Commonly caused by moving a database from one database environment Aug 25 17:34:28 SERVER slapd[5628]: bdb(): to another without clearing the database LSNs, or by removing all of Aug 25 17:34:28 SERVER slapd[5628]: bdb(): the log files from a database environment Aug 25 17:34:28 SERVER slapd[5628]: bdb(): /opt/zimbra/data/ldap/hdb/db/id2entry.bdb: unexpected file type or format Aug 25 17:34:28 SERVER slapd[5628]: hdb_db_open: database "": db_open(/opt/zimbra/data/ldap/hdb/db/id2entry.bdb) failed: Invalid argument (22). Aug 25 17:34:28 SERVER slapd[5628]: backend_startup_one (type=hdb, suffix=""): bi_db_open failed! (22) Aug 25 17:34:28 SERVER slapd[5628]: bdb_db_close: database "": alock_close failed Aug 25 17:34:28 SERVER slapd[5628]: slapd stopped. Aug 25 17:34:32 SERVER slapd[5631]: @(#) $OpenLDAP: slapd 2.4.26 (Sep 7 2011 12:24:16) $#012#011build@zre-rhel6-64.eng.vmware.com:/home/build/p4/HELIX/ThirdParty/openldap/openldap-2.4.26.5z/servers/slapd Aug 25 17:34:32 SERVER slapd[5632]: bdb(): file id2entry.bdb has LSN 146/8895192, past end of log at 146/7887499 Aug 25 17:34:32 SERVER slapd[5632]: bdb(): Commonly caused by moving a database from one database environment Aug 25 17:34:32 SERVER slapd[5632]: bdb(): to another without clearing the database LSNs, or by removing all of Aug 25 17:34:32 SERVER slapd[5632]: bdb(): the log files from a database environment Aug 25 17:34:32 SERVER slapd[5632]: bdb(): /opt/zimbra/data/ldap/hdb/db/id2entry.bdb: unexpected file type or format Aug 25 17:34:32 SERVER slapd[5632]: hdb_db_open: database "": db_open(/opt/zimbra/data/ldap/hdb/db/id2entry.bdb) failed: Invalid argument (22). Aug 25 17:34:32 SERVER slapd[5632]: backend_startup_one (type=hdb, suffix=""): bi_db_open failed! (22) Aug 25 17:34:32 SERVER slapd[5632]: bdb_db_close: database "": alock_close failed Aug 25 17:34:32 SERVER slapd[5632]: slapd stopped.
Error from the command line when attempting zmcontrol start or ldap start:
Starting ldap...Done. Failed. Failed to start slapd. Attempting debug start to determine error. hdb_db_open: database "": db_open(/opt/zimbra/data/ldap/hdb/db/id2entry.bdb) failed: Invalid argument (22). backend_startup_one (type=hdb, suffix=""): bi_db_open failed! (22) bdb_db_close: database "": alock_close failed
Attempted ldap db recovery.
su - zimbra zmcontrol stop cd /opt/zimbra/data/ldap/hdb/db /opt/zimbra/bdb/bin/db_recover
That didn't work.
Removal of the alock [/opt/zimbra/data/ldap/hdb/db/alock] file also didn't work, ldap start failed with same error and created a new alock file.
Proceeded with some of the steps taken from this reference: http://wiki.zimbra.com/wiki/LDAP_data_import_export
cd /opt/zimbra/data/ldap/ mv hdb hdb.old
Create the new directory structure :
mkdir -p hdb/db mkdir -p hdb/logs mv hdb hdb.old ldap start
Confirmed ldap started ok and also checked replica status.
/opt/zimbra/libexec/zmreplchk Code: 0 Status: In Sync
zimbra.log looked good.
zmcontrol start
And the rest of the zimbra services came up cleanly.
Ldap Restore
To find the LDAP session labels type -lbs.
zmrestoreldap -lbs
Restore the complete LDAP directory server [example]
zmrestoreldap -lb full20061130135236
ldapmodify Examples
Removing An Alias That Is Also Primary Account
Bug Reference:
- "renameAccount allows primary to be renamed as existing AccountAlias - breaking account"
External Reference:
Description of problem:
When I try to remove the alias that matches the primary account with zmprov or to delete the Alias in the web console it fails. And attempt to restore the account to another name [ -ca -pre old_] fails as well [Error occurred: Read timed out].
An attempt to rename the primary account doesn't avoid the issue with the matching alias name.
[zimbra@mail root]$ zmprov ra USER@DOMAIN.com USER_OLD@DOMAIN.com [zimbra@mail root]$ zmprov raa USER_OLD@DOMAIN.com USER@DOMAIN.com ERROR: account.NO_SUCH_ALIAS (no such alias: USER@DOMAIN.com) [zimbra@mail root]$ zmprov ra USER_OLD@DOMAIN.com USER@DOMAIN.com
Also, whenever I rename the account, the alias gets renamed with it. Even when using the zimbraID to rename the account.
The following will confirm the issue as it will show a zimbraMailAlias matching a mail variable.
su - zimbra source ~/bin/zmshutil zmsetvars ldapsearch -x -H $ldap_master_url -D $zimbra_ldap_userdn -w $zimbra_ldap_password "mail=USER@DOMAIN.com"
##shows us something like this##
[cut] # USER, people, DOMAIN.com dn: uid=USER,ou=people,dc=DOMAIN,dc=com [cut] mail: USER@DOMAIN.com mail: USER_Alias1@DOMAIN.com mail: USER_Alias2@DOMAIN.com mail: USER_Alias3@DOMAIN.com zimbraMailAlias: USER@DOMAIN.com zimbraMailAlias: USER_Alias1@DOMAIN.com zimbraMailAlias: USER_Alias2@DOMAIN.com zimbraMailAlias: USER_Alias3@DOMAIN.com [cut]
To use ldapmodify to replace the aliases, you would construct a statement like this:
[if you haven't already done this in the current shell] su - zimbra source ~/bin/zmshutil zmsetvars [end]
ldapmodify -x -H $ldap_master_url -D $zimbra_ldap_userdn -w $zimbra_ldap_password dn: uid=USER,ou=people,dc=DOMAIN,dc=com changetype: modify replace: zimbraMailAlias zimbraMailAlias: USER_Alias1@DOMAIN.com zimbraMailAlias: USER_Alias2@DOMAIN.com zimbraMailAlias: USER_Alias3@DOMAIN.com [ctrl-D executes this change and drops you to prompt]
zmprov flushCache account USER@DOMAIN.com
Notice the absence of the zimbraMailAlias: USER@DOMAIN.com variable.
To use ldapmodify to delete ALL aliases, you would construct a statement like this:
[if you haven't already done this in the current shell] su - zimbra source ~/bin/zmshutil zmsetvars [end]
ldapmodify -x -H $ldap_master_url -D $zimbra_ldap_userdn -w $zimbra_ldap_password dn: uid=USER,ou=people,dc=DOMAIN,dc=com changetype: modify delete: zimbraMailAlias [ctrl-D executes this change and drops you to prompt]
zmprov flushCache account USER@DOMAIN.com
You can also place the edit's in a text file and run it like:
[if you haven't already done this in the current shell] su - zimbra source ~/bin/zmshutil zmsetvars [end]
ldapmodify -x -H $ldap_master_url -D $zimbra_ldap_userdn -w $zimbra_ldap_password -f fix-account.ldif
zmprov flushCache account USER@DOMAIN.com
If you were to do multiple operations, you would need to include the - marker. For example:
userprompt> ldapmodify -D bindDN -w password -h server1 dn: cn=Niels Nelissen,ou=people,dc=example,dc=com changetype: modify delete: telephonenumber - add: manager manager: cn=Peter Petersen,ou=people,dc=example,dc=com ^D userprompt>
db_recover For OpenLdap Accesslog DB In /opt/zimbra/openldap-data/accesslog/db
- Caution - this should normally only be done under guidance or request by support.
For ZCS 5.x Only
This will cause an impact to your user base, since we'll be stopping the ldap service.
For replication issues.
- Steps Removed At This Time.
Re-indexing Openldap
- Caution - this should normally only be done under guidance or request by support.
This will cause an impact to your user base, since we'll be stopping the ldap service.
su - zimbra cd /opt/zimbra/openldap-data ldap stop /opt/zimbra/openldap/sbin/slapindex -f /opt/zimbra/conf/slapd.conf
Purging Logs From /opt/zimbra/openldap-data/logs
- Caution - this should normally only be done under guidance or request by support.
DB_CONFIG sets the log variable, it should be /opt/zimbra/openldap-data/logs .
ldap stop db_checkpoint -1 -h /opt/zimbra/openldap-data db_archive -h /opt/zimbra/openldap-data rm <logs listed from db_archive> ldap start
What Does - connection_read : no connection! - In zimbra.log Indicate
Developer responses I've had on this question:
- One ldap server is seeing it and others aren't, then confirm the logging levels are set the same
- This informative message indicates that a client disconnected without sending an unbind request first. It is quite common and certainly something to ignore.
Note, customer also gave me feedback that they ended up seeing this as a result of they way their load-balancers were operating.
Change LDAP Port On Zimbra
It is recommend that there is no other LDAP service running on the ZCS besides Zimbra's. But if you find the need to change it, here's the steps.
You can modify the default [389] port ldap uses to port 390 by doing the following [change LDAPHOSTNAME.com to your server name]:
zmlocalconfig -e ldap_port=390 zmlocalconfig -e ldap_master_url=ldap://LDAPHOSTNAME.com:390 zmlocalconfig -e ldap_url=ldap://LDAPHOSTNAME.com:390
Note you need to do all three of these or things will break. Once you've run those commands, you will need to restart ZCS:
zmcontrol stop zmcontrol start
Once this is completed, Zimbra will run with LDAP on port 390 rather than port 389 (the default). If you'd like to use a port other than 390, substitute that port for 390 and use the same process.
Can't Setup/Install LDAP Replica
Please see bug against documentation:
- "Update ch5 multi-server replication instructions"
- http://bugzilla.zimbra.com/show_bug.cgi?id=26117
- Updates from the bug seem to be mostly going into the Multi-Server documentation.
- http://bugzilla.zimbra.com/show_bug.cgi?id=26117
Checking /tmp/zmsetup.log will expose errors with installation configuration, for example below, others are listed in my notes farther down:
: Thu Aug 21 16:00:12 2008 *** Running as zimbra user: /opt/zimbra/bin/zmlicense -c Error: exception occurred: system failure: ZimbraLdapContext when running 'zmcontrol start': [zimbra@mail1b ~]$ zmcontrol start Host mail1b.DOMAINNAME Unable to determine enabled services from ldap. Unable to determine enabled services. Cache is out of date or doesn't exist.
Make sure you are using the right passwords. If the ldap replica can't authenticate to the master LDAP server it will not be able to pull down the ldap configurations it needs - one of which is the license data.
My Notes From The Case That I Made For Bug 26117
- Verified Fixed for 5.0.10+ in regards to bug 26117
Summary
Action Items:
- Update doc's to reflect ldap repli password setting during install (all ref docs miss this step)
- If this is done during install, no other action is needed except logger and sshkeys (zmupdateauthkeys)
- Update doc's to reflect recovery steps on an improper install for ldap replication server
- Make comments consistent. See LDAP wiki page
- Update installer to require ldap repli password rather default to random set
password.
References:
- "Update ch5 multi-server replication instructions"
- http://bugzilla.zimbra.com/show_bug.cgi?id=26117
- Verified fix for 5.0.10+
- http://bugzilla.zimbra.com/show_bug.cgi?id=26117
- Main LDAP Wiki Page
- http://www.zimbra.com/docs/os/latest/multi_server_install/LDAP%20Replication.6.1.html
- http://www.zimbra.com/elqNow/elqRedir.htm?ref=http://files.zimbra.com/website/docs/Zimbra%20NE%20Multi-Server%20Install.pdf
LDAP replication has some comments that aren't in other places
- Set the master LDAP password to the correct value (run zmlocalconfig -s ldap_root_password on the master to determine this value)
- Note: In order to install an LDAP replica server with no MBS (Mailbox Server), set zimbra_zmprov_default_to_ldap to true, using the following command:
zmlocalconfig -e zimbra_zmprov_default_to_ldap=true.
- If you later add an MBS to your LDAP replica server, set zimbra_zmprov_default_to_ldap to false.
- After the installation is complete, run /opt/zimbra/libexec/zmldapenablereplica on the replica server. This will enable replication in the LDAP server and will force an LDAP restart to begin pulling data from the master. zmldapenablereplica must be run on both the master and replica for LDAP replication to work.
The Work
Duplicated on ZCS 5.0.2 - OS Redhat.
Current Servers For Test:
- Setup LDAP MAIN server - zldap1 (ldap only configured server)
- setting passwords to "PASSWORD"
- Mailstore server - zmail2 (mailstore only config)
- MTA server - zmta1 (mta only config)
- Post install
- enable logger
- enable ssh keys
- run zmupdateauthkeys on servers
- created two test accounts and confirmed email delivery between the two.
- On LDAP MAIN (zldap1)- zmldapenablereplica
- Setup LDAP MAIN server - zldap1 (ldap only configured server)
Install LDAP repli - zldap3
- Doc show only to set
- ldap master host
- Master ldap server password
- Create Domain to no
- It fails to include
- Replication Password (part of "Ldap configuration")
- If this is not done, install will finish but you get these errors.
- Replication Password (part of "Ldap configuration")
- [zimbra@zldap3 ~]$ zmcontrol status
- Cannot determine services - exiting
- zldap3.DOMAIN.com...ERROR: service.FAILURE (system failure: getDirectContext)
- (cause: javax.naming.AuthenticationException [LDAP: error code 49 - Invalid Credentials])
- Doc show only to set
To Correct Without Reinstalling
If this has happened and you want to correct without a reinstall. Do the following, please adjust for HOSTNAMES for LDAP MAIN and LDAP REPLICA.
- confirm ssh keys are updated
- run zmupdateauthkeys on all servers
- confirm current settings on LDAP MAIN
- zmlocalconfig -s | grep replication
- grep credentials /opt/zimbra/conf/slapd.conf
- confirm current settings on LDAP REPLI
- grep credentials /opt/zimbra/conf/slapd.conf
- zmlocalconfig -s | grep replication
- Now reconfigure on LDAP REPLI
zmldappasswd -l [passwd from ldap main slapd.conf]
- This does not dynamically update the slapd.conf variable, it puts it in the zmconfig file /opt/zimbra/libexec/zmldapenablereplica
- You'll see that it updates the slapd.conf
- There might be one auth error, until the slapd.conf is reconfigured.
- zmprov gacf will fail with auth errors
- Confirm again settings on LDAP REPLI
- grep credentials /opt/zimbra/conf/slapd.conf
- zmlocalconfig -s | grep replication
- zmlocalconfig | grep ldap_url
- Should have repli first and ldap main second
- Now to confirm it worked
- zmcontrol stop
- zmcontrol start
- zmcontrol status
- Create a test account on LDAP MAIN (example uses test3)
- On LDAP REPLI - replace hostname and test account below
ldapsearch -x -H ldap://LDAPREPLICA.DOMAIN.com:389 | grep test3
Disable LDAP Replica
References:
http://wiki.zimbra.com/index.php?title=Promoting_Replica_to_LDAP_Master
http://www.zimbra.com/docs/os/latest/multi_server_install/LDAP%20Replication.6.1.html
Remove LDAP Replica From All Active Servers
On each member server, including the replica itself, verify the ldap_url value.
zmlocalconfig ldap_url
Modify the ldap_url to only include enabled ZCS LDAP servers. The master LDAP server should always be at the end of the ldap_url string value.
zmlocalconfig -e ldap_url="ldap://<replica-server-host> ldap://<master-server-host>"
Disable LDAP On The Replica
Stop Zimbra services.
zmcontrol stop
The - in front of zimbraServiceEnabled is [off], rather than a + for [on].
zmprov -l ms `zmhostname` -zimbraServiceEnabled ldap
To enable the ldap service, prefix the zimbraServiceEnabled attribute with a "+".
zmprov -l ms `zmhostname` +zimbraServiceEnabled ldap
If other services are enabled on this host, start them.
zmcontrol start
Additional Steps for MTA hosts
After updating the ldap_url with zmlocalconfig, rerun /opt/zimbra/libexec/zmmtainit. This rewrites the Postfix configuration with the updated ldap_url.
Disabling Replication On The Master
ZCS 5.x
Use only if disabling replication entirely across all nodes.
Edit /opt/zimbra/conf/slapd.conf.in adding the following comments. Please use the precise number of hash marks (#) shown.
change
include /opt/zimbra/conf/master-accesslog.conf
to
###include /opt/zimbra/conf/master-accesslog.conf
change
overlay syncprov syncprov-checkpoint 20 10 syncprov-sessionlog 500 include /opt/zimbra/conf/master-accesslog-overlay.conf
to
#overlay syncprov #syncprov-checkpoint 20 10 #syncprov-sessionlog 500 ###include /opt/zimbra/conf/master-accesslog-overlay.conf
Restart the master LDAP server
ldap stop; ldap start
Promoting Replica To LDAP Master
Please see:
- Promoting_Replica_to_LDAP_Master
- [1]
- Be sure to reference, especially area around "zmenableldapreplica": ZCS Multi-Server Installation Guide, LDAP Replication
Importing LDAP data from master to replica 6.0
Please see:
BDB - Underlying Database Used For LDAP Data
BDB is the underlying high-performance transactional database used to store the LDAP data.
See:
- Performance_Tuning_Guidelines_for_Large_Deployments
- Performance_Tuning_Guidelines_for_Large_Deployments#Configuring_the_BDB_subsystem_to_increase_LDAP_server_performance
db_stat
The db_stat utility displays statistics for Berkeley DB environments.
Example output:
[zimbra@mail3 openldap-data]$ db_stat -c -h /opt/zimbra/openldap-data 462 Last allocated locker ID. 2147M Current maximum unused locker ID. 9 Number of lock modes. 3000 Maximum number of locks possible. 1500 Maximum number of lockers possible. 1500 Maximum number of lock objects possible. 19 Number of current locks. 73 Maximum number of locks at any one time. 62 Number of current lockers. 68 Maximum number of lockers at any one time. 19 Number of current lock objects. 50 Maximum number of lock objects at any one time. 1784391 Total number of locks requested. 1784372 Total number of locks released. 0 Total number of lock requests failing because DB_LOCK_NOWAIT was set. 8 Total number of locks not immediately available due to conflicts. 0 Number of deadlocks. 0 Lock timeout value. 0 Number of locks that have timed out. 0 Transaction timeout value. 0 Number of transactions that have timed out. 968KB The size of the lock region.. 6 The number of region locks granted after waiting. 3543865 The number of region locks granted without waiting.
db_archive
The db_archive utility writes the pathnames of log files that are no longer in use (for example, no longer involved in active transactions), to the standard output, one pathname per line. These log files should be written to backup media to provide for recovery in the case of catastrophic failure (which also requires a snapshot of the database files), but they may then be deleted from the system to reclaim disk space.
db_checkpoint
The db_checkpoint utility is a daemon process that monitors the database log, and periodically calls DB_ENV->txn_checkpoint to checkpoint it.
db_deadlock
db_dump
db_load
db_recover
Ldap Replica Styles And Timeframes
Please see this external reference:
- OpenLDAP syncrepl RefreshAndPersist
- http://www.zytrax.com/books/ldap/ch7/#ol-syncrepl-rap
- Full Chapter Topic
- http://www.zytrax.com/books/ldap/ch7/
LDAP And OverLays
We don't [officially] support running additional overlays with OpenLDAP.
SLAPO-RWM OVERLAY RWM
And slapo-rwm is known to be buggy in OpenLDAP 2.3.43 and continues to be buggy to this day in OpenLDAP 2.4. It certainly won't work with ZCS 5.0.16.
We would advise customers to avoid using it until it stabilizes, though they need to understand it's still going to be unsupported by us.
Where one places "overlay rwm" in the slapd.conf file has been known to cause issues as well. RWM has problems in the order in which it is loaded. There are at least 2 open bugs currently in the OpenLDAP ITS tracker.
GAL (Server) Topics
Actual GAL (Server) Topics Notes Homepage
Please see Ajcody-Notes-ServerGAL
GAL Behavior Bugs/RFE's
GAL And Alises
Please see the following:
- "hide aliases in GAL on a per-alias basis"
- Some background bugs/rfe's on gal - alias issues:
- "hide account alias in gal search for ZWC"
- "All aliases appear when doing GAL auto-complete when composing new message"
Canonical Addresses - Beyond GAL Issues Really
Please see the following:
- "deprecate canonical address"
- "Please add subaddressing support to email addresses"
External GAL Source
Please see:
Setting/Checking GAL Varaibles
CLI To Sync Gal On Server
Truthfully, I'm not sure of the reasoning behind the command but this might be useful in trouble shooting connector issues with GAL sync issues.
zmprov syg DOMAINNAME.com
4.5.x versions will need quotes at the end.
zmprov syg DOMAINAME.com ""
My Gal & LDAP Settings For A Domain
To see your setting, do the following - replacing with domainname with the domain in question.
su - zimbra zmprov gd [domainname] zimbraGalLdapSearchBase zmprov gd [domainname] zimbraGalSyncLdapSearchBase
You'll see more GAL/LDAP related variables with:
zmprov gd domainname | egrep -i 'ldap|gal'
They are set using:
zmprov md [domainname] zimbraGalLdapSearchBase variable zmprov md [domainname] zimbraGalSyncLdapSearchBase variable
Want My GAL To See All Domains
The default of a domain GAL [zimbraGalInternalSearchBase] is to see only it's domain. To have the GAL for a domain to see all domains on the server you need to set the variable to ROOT.
To see the existing setting:
zmprov gd [domainname] zimbraGalLdapSearchBase
To change the variable for the domain:
zmprov md [domainname] zimbraGalInternalSearchBase ROOT
All global change would be done with:
zmprov mcf zimbraGalInternalSearchBase ROOT
Some Other Ideas - Especially With Multiple Domains And ZCS Servers
This is just some notes I made for a specific case where there were multiple ZCS installs across sister companies. They didn't have a multi-server install and weren't sharing LDAP data therefore.
Here's some ideas I'm considering for this situation:
- Dump GAL data from each server and combine date to either:
- an external LDAP server you can then configure the ZCS servers to use as an external GAL
- admin console > domain > GAL > Configure GAL > Both or External
- reformat GAL data so it can then be imported as an address book via the CLI into a sharead adddressbook on each server. Setup command to run from cron.
- an external LDAP server you can then configure the ZCS servers to use as an external GAL
- Just setup an external Openldap server (which would have replica's at each site) that can be used as an external GAL - make this authoritative in your company. Then setup the ZCS to use it as an external GAL.
- To actually move your infrastructure of ZCS servers into one primary ZCS configuration. Instead of having each site/domain having it's own primary LDAP/ZCS server they would become LDAP replica servers with their domain mailstore's being local to each site. Configure each site/domain to use it's own MTA - if you want/need. And then internally to Zimbra you could set the GAL to be ROOT rather than DOMAIN.
Some bugs to review that touch on this:
- "Add support for multiple GALs per domain"
- "groupings for GAL contacts for 'Select Addresses' dialog and or GAL"
- "GAL via contacts folders with sorting, browsing and real sync"
I'm sure there's some other possibilities.
GAL Related Attributes Usage
This is from earlier 5.x versions, newer version might include more variables.
Attribute scope For GAL op - zimbra - autocomplete - external - serarch - both - sync - all ====================================================================================================================================================================== zimbraHideInGal account,DL,CR zimbra all zimbraFeatureGalEnabled account,cos both search,sync zimbraFeatureGalAutoCompleteEnabled account,cos both autocomplete zimbraPrefGalAutoCompleteEnabled account,cos both autocomplete zimbraGalMode domain both all zimbraGalLdapFilterDef globalConfig both all zimbraGalLdapAttrMap domain,globalConfig both all zimbraGalMaxResults domain,globalConfig both autocomplete, search (for sync, system does not specify a max when searching LDAP) zimbraGalTokenizeAutoCompleteKey domain both autocomplete zimbraGalTokenizeSearchKey domain both search zimbraGalLdapPageSize domain,globalConfig both autocomplete, search, (sync if zimbraGalSyncLdapPageSize is not set) zimbraGalSyncLdapPageSize domain,globalConfig both sync ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- zimbraGalInternalSearchBase domain,globalConfig zimbra autocomplete, search, (sync if zimbraGalSyncInternalSearchBase is not set) zimbraGalSyncInternalSearchBase domain,globalConfig zimbra sync ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- zimbraGalLdapURL domain external autocomplete, search, (sync if zimbraGalSyncLdapURL is not set) zimbraGalSyncLdapURL domain external sync zimbraGalLdapSearchBase domain external autocomplete, search, (sync if zimbraGalSyncLdapSearchBase is not set) zimbraGalSyncLdapSearchBase domain external sync zimbraGalLdapAuthMech domain external autocomplete, search, (sync if zimbraGalSyncLdapAuthMech is not set) zimbraGalSyncLdapAuthMech domain external sync zimbraGalLdapBindDn domain external autocomplete, search, (sync if zimbraGalSyncLdapBindDn is not set) zimbraGalSyncLdapBindDn domain external sync zimbraGalLdapBindPassword domain external autocomplete, search, (sync if zimbraGalSyncLdapBindPassword is not set) zimbraGalSyncLdapBindPassword domain external sync zimbraGalLdapKerberos5Principal domain external autocomplete, search, (sync if zimbraGalSyncLdapKerberos5Principal is not set) zimbraGalSyncLdapKerberos5Principal domain external sync zimbraGalLdapKerberos5Keytab domain external autocomplete, search, (sync if zimbraGalSyncLdapKerberos5Keytab is not set) zimbraGalSyncLdapKerberos5Keytab domain external sync zimbraGalAutoCompleteLdapFilter domain,globalConfig external autocomplete zimbraGalLdapFilter domain external search, (sync if zimbraGalLdapSyncFilter is not set) zimbraGalSyncLdapFilter domain external sync
Ajcody-External-Authentication
Ajcody MTA Postfix Topics
Postfix - MTA
Actual MTA & Postfix Topics Homepage
Please see Ajcody-MTA-Postfix-Topics
Missing main.cf Error
Moved to Missing_main.cf_Error_-_MTA
MTA Mail Flow - Birds-eye Overview
Moved to Postfix-Amavisd_Mail_Flow_-_Birds-eye_Overview_-_MTA
Understanding /var/log/zimbra.log And Postfix Log Events
Moved to: Understanding_zimbra.log_And_Postfix_Log_Events_-_MTA
Postfix Queue ID vs. message-id
Moved to: Postfix_Queue_ID_vs._message-id_-_MTA
Authentication Log Events
Moved to Understanding_And_Troubleshooting_Authentication_Log_Events
IMAP And Authenticated SMTP [SSL] Example
IMAP Test Via Telnet And Logging Events Of It - Proxy Included
IMAP Login Via Openssl - LOGIN TLS - Proxy Included
Network Tracing Between A Remote Host And A ZCS MTA
Moved to Network_Tracing_Between_A_Remote_Host_And_A_ZCS_MTA
Finding Messages - zmmsgtrace
See the following for details [for 7.1.1+]:
- zmmsgtrace replacement
Documentation at CLI_zmmsgtrace . Note, if you get command not found as the zimbra user, try /opt/zimbra/libexec/zmmsgtrace instead.
How To Increase SMTP Debug Logging - MTA
Moved to How_To_Increase_SMTP_Debug_Logging_-_MTA
Simple Troubleshooting For SMTP Via Telnet, Openssl
Moved to Simple_Troubleshooting_For_SMTP_Via_Telnet_And_Openssl
First - Understanding Your Authentication Requirements In ZCS
Second - Encoding Username And Passwords For AUTH Sequence
For ESMTP Auth is LOGIN - Example
Moved to Simple_Troubleshooting_For_SMTP_Via_Telnet_And_Openssl#For_ESMTP_Auth_is_LOGIN_-_Example
For ESMTP Auth is Plain - Example
Moved to Simple_Troubleshooting_For_SMTP_Via_Telnet_And_Openssl#For_ESMTP_Auth_is_Plain_-_Example
For TLS/SSL - Example
Moved to Simple_Troubleshooting_For_SMTP_Via_Telnet_And_Openssl#For_TLS.2FSSL_-_Example
Testing Against Port 465
Moved to Simple_Troubleshooting_For_SMTP_Via_Telnet_And_Openssl#Testing_Against_Port_465
To Confirm An Auth User Can't Send With Another FROM Address
Adding A New MTA Server
Basic instructions can be found here:
- http://www.zimbra.com/docs/ne/latest/multi_server_install/toc.html
- See "Installing Zimbra MTA on a Server"
Additional instructions needed beyond the above will follow as I hear about them.
Load Balancing For SMTP - Out Bound Mail
Currently, 5.x code, you have the following options:
- Configure zimbraMtaRelayHost and zimbraSmtpHostname zimbraSmtpHostname Details to:
- An external load balancing device that will then split the traffic behind it
- Setup a round-robin A record situation in your DNS for the external mta's you'll be using.
In, GNR/6.x, you are able to add multiple targets to the variables and we'll have some degree of "balancing" between them.
- "allow list for zimbraSmtpHostname"
- "make zimbraSmtpHostname fault tolerant"
User Alias Mapping And Mail Transport with Postfix & LDAP
See User_Alias_Mapping_and_Mail_Transport_with_Postfix_&_LDAP
Multiple LDAP Servers?
Completed RFE:
- "mta should be able to take a list of LDAP servers to take advantage of replicas."
- http://bugzilla.zimbra.com/show_bug.cgi?id=9353
- zmmtainit to allow for multiple command line options that will set the URL. Grab the contents of the ldap_url localconfig variable.
- http://bugzilla.zimbra.com/show_bug.cgi?id=9353
From :
server_host (default: localhost) The name of the host running the LDAP server, e.g. server_host = ldap.example.com Depending on the LDAP client library you're using, it should be possible to specify multiple servers here, with the library trying them in order should the first one fail. It should also be possible to give each server in the list a different port (overriding server_port below), by naming them like server_host = ldap.example.com:1444 With OpenLDAP, a (list of) LDAP URLs can be used to specify both the hostname(s) and the port(s): server_host = ldap://ldap.example.com:1444 ldap://ldap2.example.com:1444 All LDAP URLs accepted by the OpenLDAP library are supported, including connections over UNIX domain sockets, and LDAP SSL (the last one provided that OpenLDAP was compiled with support for SSL): server_host = ldapi://%2Fsome%2Fpath ldaps://ldap.example.com:636 **my note** This thread - http://archives.neohapsis.com/archives/postfix/2004-09/1763.html give me the impression they made a mistake in modifying the help file on this and they dropped the use/need of the command: server_host = ldap://ldap.example.com:1444, ldap://ldap2.example.com:1444
Just a small note on where var shows up:
[root@mail3 conf]# pwd /opt/zimbra/conf [root@mail3 conf]# grep server_host * amavisd.conf.in:$myhostname = '@@zimbra_server_hostname@@'; # must be a fully-qualified domain name! ldap-scm.cf:server_host = ldap://mail3.zimbra.DOMAIN.com:389 ldap-transport.cf:server_host = ldap://mail3.zimbra.DOMAIN.com:389 ldap-vad.cf:server_host = ldap://mail3.zimbra.DOMAIN.com:389 ldap-vam.cf:server_host = ldap://mail3.zimbra.DOMAIN.com:389 ldap-vmd.cf:server_host = ldap://mail3.zimbra.DOMAIN.com:389 ldap-vmm.cf:server_host = ldap://mail3.zimbra.DOMAIN.com:389 localconfig.xml: <key name="zimbra_server_hostname"> zmmta.cf: LOCAL zimbra_server_hostname zmmta.cf: POSTCONF myhostname LOCAL zimbra_server_hostname
References:
Traditional Aliases Use - /etc/aliases Type Lookups
Moved to Traditional_Aliases_Use_-_/etc/aliases_Type_Lookups
Allowing Accounts To Change The From Address
Please see:
Related BUG/RFE's
- Identities: Auto verify user settable from address [marked as dup of 29974]
- persona/external account from field address verification
- ZCO Support for zimbraAllowAnyFromAddress
Creating A Domain Alias
Please see ManagingDomains#Creating_a_Domain_Alias
Relay Domain Forwarding
Please see ManagingDomains#Relaying.2FDomain_Forwarding
Domain Catchall
Please see ManagingDomains#Domain_Catchall
Rewriting From Address For Outbound Email
Please see ManagingDomains#Domain_Masquerading
Rewrite Recipient Address For Incoming Email
There is a way to rewrite the incoming mail, but it's not a standard Zimbra feature. You can implement it as a configuration change in Postfix. Here's what you do:
- Create a file in /opt/zimbra/conf named 'postfix_recipientmap'.
- The format is a single line that reads something like: @alias.domain.com @domain.com
- Run 'postmap postfix_recipientmap' in the conf directory.
- Run "postconf -e recipient_canonical_maps=hash:/opt/zimbra/conf/postfix_recipientmap".
- Run 'postfix reload'.
This will cause postfix to map any incoming mail with a recipient of '@alias.domain.com' to '@domain.com'. You will need to re-apply this postconf change after upgrades, though the postfix_recipientmap file should survive.
Automatic BCC
Option 1 - Via Postfix Customization
From the postfix website:
- always_bcc = address
- Deliver a copy of all mail to the specified address. In Postfix versions before 2.1, this feature is implemented by smtpd(8), qmqpd(8), or pickup(8).
- sender_bcc_maps = type:table
- Search the specified "type:table" lookup table with the envelope sender address for an automatic BCC address. This feature is available in Postfix 2.1 and later.
- recipient_bcc_maps = type:table
- Search the specified "type:table" lookup table with the envelope recipient address for an automatic BCC address. This feature is available in Postfix 2.1 and later.
- Note: automatic BCC recipients are produced only for new mail. To avoid mailer loops, automatic BCC recipients are not generated for mail that Postfix forwards internally, nor for mail that Postfix generates itself.
Please see the following:
- A very nice forum post on the subject from our very own mmorse
- Postfix workaround
- "User defined auto bcc"
- "Next rev of (mail) identities preferences management (server side)"
Option 2 - Via ZCS Legal Intercept
Generally used for Managing Legal Requests for Information
Description:
- The ZCS legal intercept feature is used to obtain copies of email messages that are sent, received, or saved as drafts from targeted accounts and send these message to a designated “shadow” email address. Legal Intercept can be configured to send the complete content of the message or to send only the header information. When a targeted account sends, receives, or saves a draft message, an intercept message is automatically created to forward copies of the messages as attachments to the specified email address.
Please see:
- http://www.zimbra.com/docs/ne/latest/administration_guide/managing_other_zcs_features.8.1.html
- http://wiki.zimbra.com/index.php?title=Legal_Intercept
- http://bugzilla.zimbra.com/show_bug.cgi?id=17539
Option 3 - Zimbra's Archiving And Discovery
See Ajcody-Notes-Archive-Discovery concerning A&D setup and options.
Limiting Or Increasing Number Of Recipents / Messages
Mailing Lists - Distribution Lists
Policy Daemon
If you want to restrict messages per hour, you can look into Policy Daemon:
- http://wiki.zimbra.com/wiki/Postfix_Policyd
- http://wiki.zimbra.com/wiki/How-to_for_cbpolicyd
- http://www.policyd.org/features.html
Beta release in ZCS 7 , see:
- "make support for postfix-policyd easier"
Postfix
Also, there are some default postfix parameters set to control sending a message to x amount recipients. The parameters you will need to look at are smtpd_recipient_limit & smtpd_recipient_overshoot_limit, these have a default value of 1000.
Postfix defines these parameters as:
- smtpd_recipient_limit: The maximum number of recipients that the Postfix SMTP server accepts per message delivery request.
- smtpd_recipient_overshoot_limit: The number of recipients that a remote SMTP client can send in excess of the limit specified with $smtpd_recipient_limit, before the Postfix SMTP server increments the per-session error count for each excess recipient.
From the command line you can change the default values.
su - zimbra postconf -e smtpd_recipient_limit=<new value> postconf -e smtpd_recipient_overshoot_limit=<new value> postfix reload
Bugs RFE's For Customers To Get Behind
I'm wondering if policyd gives one the control everyone is looking for? I've not used it myself.
Policyd References:
- http://www.policyd.org/tiki-index.php?page=Documentation
- http://www.policyd.org/tiki-index.php?page=Quotas&structure=Documentation
- http://www.policyd.org/tiki-index.php?page=Accounting&structure=Documentation
- http://www.policyd.org/tiki-index.php?page=Policies%20%26%20Groups&structure=Documentation
- http://wiki.zimbra.com/index.php?title=Postfix_Policyd
There's other additions [add-on's] one can get for policyd.
We have this RFE in regards to policyd support:
- "make support for postfix-policyd easier"
- http://bugzilla.zimbra.com/show_bug.cgi?id=8791
- Target Milestone currently for Helix release [ http://pm.zimbra.com ]
- http://bugzilla.zimbra.com/show_bug.cgi?id=8791
Other related rfe/bugs, specially to push variables into admin web console:
- "rate limit amount of mail sent via web client"
- http://bugzilla.zimbra.com/show_bug.cgi?id=22300
- *Target Milestone currently for Helix release
- http://bugzilla.zimbra.com/show_bug.cgi?id=22300
- "mta "advanced" tab"
- http://bugzilla.zimbra.com/show_bug.cgi?id=14645
- Target Milestone currently for Helix release
- http://bugzilla.zimbra.com/show_bug.cgi?id=14645
- "Option to IP Blocking through UI"
- http://bugzilla.zimbra.com/show_bug.cgi?id=19240
- Target Milestone currently for Helix release
- http://bugzilla.zimbra.com/show_bug.cgi?id=19240
- "Mail policies and access control for sending to distribution lists"
- http://bugzilla.zimbra.com/show_bug.cgi?id=9620
- Target Milestone currently for GunsNRoses
- http://bugzilla.zimbra.com/show_bug.cgi?id=9620
Controlling SMTPD Client Connections
Mmorse did a good write up on these variables in the forum:
Postfix Resources At Their Site (All Clients/Connections):
- Measures against clients that make too many connections
- anvil - Postfix session count and request rate control
- anvil_rate_time_unit - The time unit over which client connection rates and other rates are calculated.
- smtpd_client_connection_count_limit - How many simultaneous connections any client is allowed to make to this service.
- smtpd_client_message_rate_limit - The maximal number of message delivery requests that any client is allowed to make to this service per time unit, regardless of whether or not Postfix actually accepts those messages.
- smtpd_client_recipient_rate_limit - The maximal number of recipient addresses that any client is allowed to send to this service per time unit, regardless of whether or not Postfix actually accepts those recipients.
- smtpd_client_connection_rate_limit - The maximal number of connection attempts any client is allowed to make to this service per time unit.
Postfix Resources At Their Site (Exceptions To Clients/Connections Or Single Source):
Restrictions
Besides using external mailing list software, Mailman or Sympa, here's some other topical items in regards to restrictions.
Some user contributed articles:
- RestrictPostfixRecipients
- Restrict_sending_to_certain_domains
- Restrict_users_to_certain_domain
- Forum Post: GUIDE: Postifx: HOW TO: Multiple access lists for protected Distribution-lists
Some Postfix references:
- http://www.postfix.org/RESTRICTION_CLASS_README.html#internal
- http://www.postfix.org/SMTPD_POLICY_README.html
Some RFE's related to mta based restrictions [targets are based upon today - July 21, 2010]:
- "Dynamic distribution lists - Internal Directory"
- "per-domain send restriction" - Not Committed
- http://bugzilla.zimbra.com/show_bug.cgi?id=5595
- These are marked as dup's of the above:
- "disable outbound e-mail for one user"
- "Add an facility to detemine internal relay users in admin"
- These are marked as dup's of the above:
- http://bugzilla.zimbra.com/show_bug.cgi?id=5595
- "policy for who can send to a distribution lists" - Helix
- http://bugzilla.zimbra.com/show_bug.cgi?id=9620
- RFE 9620 is also a blocker for the following RFE:
- "milter to check if sender can send to a distribution list"
- These are marked as dup's of the above:
- "Ability to Specify Mail Policy"
- "domain level filters rules"
- "Distribution List Restrictions"
- "Feature request - Mail Policies"
- "limit "send from" to certain domains"
- "'Internal email only' options in admin control panel"
- "Access control for free busy and resources (ie permission to invite)"
- "RFE: Admin GUI: Restrict the use of Distribution List among users."
- RFE 9620 is also a blocker for the following RFE:
- http://bugzilla.zimbra.com/show_bug.cgi?id=9620
- "Implement smtpd_sender_restrictions"
- "How to restrict a user to only send via zwc"
- "enable configuration of "smtpd_sender_restriction""
Spam Control And Related Issues
High Over View Steps Of What To Do
- Step 1: Confirm your not an open relay and double check your postfix $mynetworks variable.
- Step 2: Stop or put on-hold mail queue.
- Put all messages into HOLD queue:
- Get a report of your current mailq [can be useful if you clean out the queue later but need to identify what external mail hosts are now denying you and who you'll need to contact about getting removed from their denial list]
- Example: /opt/zimbra/postfix/sbin/mailq > /tmp/zimbra_mailq_report.txt
- /opt/zimbra/postfix/sbin/postsuper -h ALL
- Get a report of your current mailq [can be useful if you clean out the queue later but need to identify what external mail hosts are now denying you and who you'll need to contact about getting removed from their denial list]
- Or put all messages match compromised account into HOLD queue:
- /opt/zimbra/postfix/sbin/mailq | grep user_compromised@domain | awk '{ print $1 }' | tr -d '!*' | /opt/zimbra/postfix/sbin/postsuper -h -
- Note, this is an example - you might with the grep grab more than the compromised account with the match.
- /opt/zimbra/postfix/sbin/mailq | grep user_compromised@domain | awk '{ print $1 }' | tr -d '!*' | /opt/zimbra/postfix/sbin/postsuper -h -
- See whole section - Managing The Postfix Queue
- Put all messages into HOLD queue:
- Step 3: Check your mail log [On ZCS servers running MTA services] - /var/log/zimbra.log
- Understanding the zimbra.log file and Postfix log events. , see subsection about queue ID and message ID also.
- Who's My Spammer?
- Step 4: Identify compromised account authenticating SMTP AUTH connection or block ip address where emails are coming from at firewall.
- Who's My Spammer?
- Continue to monitor compromised account and block ip addresses:
- tail -f /var/log/zimbra.log | grep username | grep sasl
- Jun 8 18:14:10 mail postfix/smtpd[15794]: 004358EEB16: client=unknown[XXXX.236.197.216], sasl_method=LOGIN, sasl_username=username@domain
- tail -f /var/log/zimbra.log | grep username | grep sasl
- Step 5: Disable the exploited email account, expire auth session, etc.
- Resetting Or Expiring User Auth
- Note - Restarting the mta services will be important once you reset the password/s or lock the account. It's required to ensure the active connections will be closed and any existing auth tokens no longer are valid. See:
- Force currently active SMTP authenticated sessions to be renegotiated when locking an account
- Step 6: Move the mail queue or delete the spam email
- See whole section - Managing The Postfix Queue
- Step 7: Release Mail queue
Who's My Spammer?
Getting Some Initial Summary Data
zmdialyreport
First, some notable bug/RFE's in regards to the zmdailyreport:
- RFE - add explanations to Daily mail report / pflogsumm.pl output
- Daily mail report shows incorrect output because pflogsumm.pl doubles the result
- Descriptions of mta_counts numbers vs daily reports and other msg stats
You can first get some summary data by doing the following:
[zimbra@zcs806 ~]$ /opt/zimbra/libexec/zmdailyreport Grand Totals ------------ messages 7 received 11 delivered 0 forwarded 0 deferred 0 bounced 3 rejected (21%) 0 reject warnings 0 held 0 discarded (0%) 2780 bytes received 10914 bytes delivered 2 senders 1 sending hosts/domains 1 recipients 1 recipient hosts/domains Per-Hour Traffic Summary time received delivered deferred bounced rejected -------------------------------------------------------------------- 0000-0100 0 0 0 0 0 0100-0200 1 3 0 0 0 0200-0300 0 0 0 0 0 0300-0400 0 0 0 0 0 0400-0500 0 0 0 0 0 0500-0600 0 0 0 0 0 0600-0700 0 0 0 0 0 0700-0800 1 0 0 0 2 0800-0900 1 0 0 0 0 0900-1000 0 0 0 0 1 1000-1100 0 0 0 0 0 1100-1200 0 0 0 0 0 1200-1300 4 8 0 0 0 1300-1400 0 0 0 0 0 1400-1500 0 0 0 0 0 1500-1600 0 0 0 0 0 1600-1700 0 0 0 0 0 1700-1800 0 0 0 0 0 1800-1900 0 0 0 0 0 1900-2000 0 0 0 0 0 2000-2100 0 0 0 0 0 2100-2200 0 0 0 0 0 2200-2300 0 0 0 0 0 2300-2400 0 0 0 0 0 Host/Domain Summary: Message Delivery (top 50) sent cnt bytes defers avg dly max dly host/domain -------- ------- ------- ------- ------- ----------- 11 10914 0 7.4 s 24.0 s zcs806.DOMAIN.com Host/Domain Summary: Messages Received (top 50) msg cnt bytes host/domain -------- ------- ----------- 5 2780 zcs806.DOMAIN.com top 50 Senders by message count ------------------------------- 4 zimbra@zcs806.DOMAIN.com 1 admin@zcs806.DOMAIN.com top 50 Recipients by message count ---------------------------------- 11 admin@zcs806.DOMAIN.com top 50 Senders by message size ------------------------------ 1974 zimbra@zcs806.DOMAIN.com 806 admin@zcs806.DOMAIN.com top 50 Recipients by message size --------------------------------- 10914 admin@zcs806.DOMAIN.com message deferral detail: none message bounce detail (by relay): none message reject detail --------------------- MAIL 5.3.4 Message size exceeds fixed limit (total: 3) 3 domain-ext.com message reject warning detail: none message hold detail: none message discard detail: none smtp delivery failures: none Warnings -------- sendmail (total: 3) 1 or the command is run from a set-uid root process 1 the Postfix sendmail command has set-uid root file permissions 1 the Postfix sendmail command must be installed without set-uid ... smtpd (total: 1) 1 7A735345A: queue file size limit exceeded Fatal Errors: none Panics: none Master daemon messages: none
client_usage_report.py
This will give some stats on your mail activity. Note, there are some issue with this script double reporting mail counts etc, but it's useful to identify the top 50 for activity.
[zimbra@zcs806 ~]$ /opt/zimbra/libexec/client_usage_report.py Reading /opt/zimbra/log/access_log.2014-04-17 .. Reading /opt/zimbra/log/access_log.2014-04-18 .. Reading /opt/zimbra/log/access_log.2014-04-19 .. Reading /opt/zimbra/log/access_log.2014-04-20 .. Reading /opt/zimbra/log/access_log.2014-04-21 .. Reading /opt/zimbra/log/access_log.2014-04-22 .. Reading /opt/zimbra/log/access_log.2014-04-23 .. Writing /opt/zimbra/zmstat/client_usage_report_2014-04-24.csv ..
Then review the file it will create , it will give : "user_agent","client_IP","req_count"
[zimbra@zcs806 ~]$ cat /opt/zimbra/zmstat/client_usage_report_2014-04-24.csv "user_agent","client_IP","req_count" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:28.0) Gecko/20100101 Firefox/28.0","192.168.1.166","14" "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0","192.168.1.166","93" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0","192.168.1.174","6" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0","192.168.1.166","71"
qshape
You can also look at the results of [qshape] - the default is the active queue. For more on qshape, see [Postfix Qshape Readme] .
qshape deferred T 5 10 20 40 80 160 320 640 1280 1280+ TOTAL 12 0 0 0 0 0 0 0 0 0 12 gmail.com 9 0 0 0 0 0 0 0 0 0 9 yahoo.com 3 0 0 0 0 0 0 0 0 0 3
3rd Party Log Reports - postfix-logwatch and amavis-logwatch
Created RFE for us to include these in ZCS:
- Include postfix-logwatch_and_amavis-logwatch
You can download them from http://logreporters.sourceforge.net/ . It's a fairly simply install, download and then extract - cd into extracted directory and as root type :
make install-standalone
They will install to /usr/local/bin/amavis-logwatch & postfix-logwatch . The config files are in /usr/local/etc/amavis-logwatch.conf & postfix-logwatch.conf . Here's an example of the output.
/usr/local/bin/amavis-logwatch output example:
[root@zcs806 amavis-logwatch-1.51.02]# /usr/local/bin/amavis-logwatch /var/log/zimbra.log ****** Summary ************************************************************************************* 4 Total messages scanned ------------------ 100.00% 1.926K Total bytes scanned 1,972 ======== ================================================== 4 Passed ---------------------------------- 100.00% 4 Clean passed 100.00% ======== ================================================== 4 Ham ------------------------------------- 100.00% 4 Clean passed 100.00% ======== ================================================== ================================================================================== Spam Score Percentiles 0% 50% 90% 95% 98% 100% ---------------------------------------------------------------------------------- Score Ham (4) -1.900 -1.900 -1.900 -1.900 -1.900 -1.900 ================================================================================== ====================================================================================================== Spam Score Frequency <= -10 <= -5 <= 0 <= 5 <= 10 <= 20 <= 30 > 30 ------------------------------------------------------------------------------------------------------ Hits (4) 0 0 4 0 0 0 0 0 Percent of Hits 0.00% 0.00% 100.00% 0.00% 0.00% 0.00% 0.00% 0.00% ======================================================================================================
/usr/local/bin/postfix-logwatch output example:
[root@zcs806 amavis-logwatch-1.51.02]# /usr/local/bin/postfix-logwatch /var/log/zimbra.log ****** Summary ************************************************************************************* 1 *Warning: Queue file size limit exceeded 6.512K Bytes accepted 6,668 1.928K Bytes sent via SMTP 1,974 4.584K Bytes sent via LMTP 4,694 ======== ================================================== 10 Accepted 76.92% 3 Rejected 23.08% -------- -------------------------------------------------- 13 Total 100.00% ======== ================================================== 3 5xx Reject message size 100.00% -------- -------------------------------------------------- 3 Total 5xx Rejects 100.00% ======== ================================================== 10 Connections 10 Disconnections 8 Removed from queue 4 Sent via SMTP 4 Sent via LMTP 4 Filtered ****** Detail (10) ********************************************************************************* 3 5xx Reject message size ----------------------------------------------------------------- 3 192.168.1.166 remote.domain.com 3 *unavailable 3 *unavailable 4 Sent via SMTP --------------------------------------------------------------------------- 4 zcs806.DOMAIN.com 4 Sent via LMTP --------------------------------------------------------------------------- 4 zcs806.DOMAIN.com 4 Filtered -------------------------------------------------------------------------------- 2 smtp-amavis:[127.0.0.1]:10024 2 Sender address 1 admin@zcs806.DOMAIN.com 1 admin@zcs806.DOMAIN.com 1 192.168.1.166 remote.domain.com 1 user@DOMAIN.com 1 admin@zcs806.DOMAIN.com 1 192.168.1.184 remote2.domain.com 2 smtp-amavis:[127.0.0.1]:10026 2 Sender address 1 admin@zcs806.DOMAIN.com 1 admin@zcs806.DOMAIN.com 1 192.168.1.166 remote.domain.com 1 user@DOMAIN.com 1 admin@zcs806.DOMAIN.com 1 192.168.1.184 remote2.domain.com === Delivery Delays Percentiles ============================================================ 0% 25% 50% 75% 90% 95% 98% 100% -------------------------------------------------------------------------------------------- Before qmgr 0.04 0.09 0.11 0.11 0.23 0.35 0.43 0.48 In qmgr 0.00 0.00 0.01 0.01 0.04 0.07 0.08 0.09 Conn setup 0.00 0.01 0.29 1.30 2.05 2.23 2.33 2.40 Transmission 0.10 2.81 4.85 9.60 21.00 21.00 21.00 21.00 Total 0.20 2.91 5.20 11.00 23.30 23.65 23.86 24.00 ============================================================================================
Note - First, look at the options each command has using the -h output. You might want to use --full when doing an investigation and also include a wildcard - /var/log/zimbra.lo* to take in all the log data. For example:
[zimbra@zcs806 ~]$ /usr/local/bin/postfix-logwatch --full /var/log/zimbra.lo* ****** Summary ************************************************************************************* 9 *Fatal: General fatal 1 *Warning: Queue file size limit exceeded 21 Miscellaneous warnings 710.888K Bytes accepted 727,949 193.036K Bytes sent via SMTP 197,669 520.114K Bytes sent via LMTP 532,597 ======== ================================================== 1041 Accepted 99.71% 3 Rejected 0.29% -------- -------------------------------------------------- 1044 Total 100.00% ======== ================================================== 3 5xx Reject message size 100.00% -------- -------------------------------------------------- 3 Total 5xx Rejects 100.00% ======== ================================================== 65 Connections 65 Disconnections 1041 Removed from queue 523 Sent via SMTP 517 Sent via LMTP 2 Bounced (local) 1 Bounced (remote) 9 Filtered 2 Notifications sent 4 Timeouts (inbound) 1 PIX workaround enabled ****** Detail (10) ********************************************************************************* 9 *Fatal: General fatal ----------------------------------------------------------------- 3 Queue report unavailable - mail system is down 3 Usage: sendmail [options] 2 The Postfix mail system is not running 1 Usage: send-mail [options] 21 Miscellaneous warnings ------------------------------------------------------------------ 7 or the command is run from a set-uid root process 7 the Postfix sendmail command has set-uid root file permissions 7 the Postfix sendmail command must be installed without set-uid root file permissions 3 5xx Reject message size ----------------------------------------------------------------- 3 10.X.X.166 fence.DOMAIN.com 3 *unavailable 3 *unavailable 523 Sent via SMTP --------------------------------------------------------------------------- 507 86.lab 15 zcs806.DOMAIN.com 1 domaina.com 517 Sent via LMTP --------------------------------------------------------------------------- 507 86.lab 10 zcs806.DOMAIN.com 2 Bounced (local) ------------------------------------------------------------------------- 2 5.0.0: Permanent failure: Other/Undefined status: Other undefined status 2 zcs806.DOMAIN.com 2 Zcs806.DOMAIN.com 1 subject:test 1 zimbra 1 Bounced (remote) ------------------------------------------------------------------------ 1 5.0.0: Permanent failure: Other/Undefined status: Other undefined status 1 domain.com 1 user 1 64.X.X.28 sentry.DOMAIN.com 1 505 5.0.0 Unknown recipient: RCPT TO 9 Filtered -------------------------------------------------------------------------------- 7 smtp-amavis:[127.0.0.1]:10026 7 Sender address 3 admin@zcs806.DOMAIN.com 3 admin@zcs806.DOMAIN.com 2 10.X.X.36 zcs806.DOMAIN.com 1 10.X.X.166 gatewayXX.DOMAIN.com 2 zimbra@zcs806.DOMAIN.com 2 admin@zcs806.DOMAIN.com 2 10.X.X.36 zcs806.DOMAIN.com 1 ajcody@DOMAIN.com 1 admin@zcs806.DOMAIN.com 1 10.X.X.184 edgeXX.DOMAIN.com 1 san5@zcs806.DOMAIN.com 1 b@zcs806.DOMAIN.com 1 10.X.X.36 zcs806.DOMAIN.com 2 smtp-amavis:[127.0.0.1]:10024 2 Sender address 1 admin@zcs806.DOMAIN.com 1 admin@zcs806.DOMAIN.com 1 10.X.X.166 gatewayXX.DOMAIN.com 1 ajcody@DOMAIN.com 1 admin@zcs806.DOMAIN.com 1 10.X.X.184 edgeXX.DOMAIN.com 2 Notifications sent ---------------------------------------------------------------------- 2 Non-delivery 2 sender 4 Timeouts (inbound) ---------------------------------------------------------------------- 4 After END-OF-MESSAGE 1 PIX workaround enabled ------------------------------------------------------------------ 1 disable_esmtp delay_dotcrlf 1 64.X.X.28 sentry.DOMAIN.com === Delivery Delays Percentiles ============================================================ 0% 25% 50% 75% 90% 95% 98% 100% -------------------------------------------------------------------------------------------- Before qmgr 0.01 0.03 0.06 0.14 0.27 0.34 0.48 2.60 In qmgr 0.00 0.00 0.08 117.50 193.00 216.00 231.48 246.00 Conn setup 0.00 0.00 0.00 0.00 0.02 0.04 0.22 20.00 Transmission 0.05 0.09 3.60 9.80 10.00 10.00 11.00 160.00 Total 0.07 0.13 3.80 129.00 203.00 226.00 241.64 259.00 ============================================================================================
zmaccts
One way to note accounts that are actively logging in vs. those that aren't, can help shrink the number of accounts you might want to investigate or monitor. [example below, I cut out a lot of the accounts]
account status created last logon ------------------------------------ ----------- --------------- --------------- zcstest001@zcs806.DOMAIN.com active 01/20/14 18:47 03/02/14 21:11 zcstest002@zcs806.DOMAIN.com active 01/30/14 01:48 02/19/14 00:07 admin-20140415@zcs806.DOMAIN.com active 04/15/14 14:42 never archtest-prod-20140402@zcs806.DOMAIN active 04/02/14 07:42 never account status created last logon ------------------------------------ ----------- --------------- --------------- bruce@test1.lab active 02/22/14 09:32 never test.cal@test1.lab active 04/06/14 05:35 04/06/14 05:35 test200@test1.lab active 04/12/14 00:50 never domain summary domain active closed locked maintenance total ----------------------- -------- -------- -------- ------------- -------- test1.lab 3 0 0 0 3 test2.com 2 0 0 0 2 angad.com 2 0 0 0 2 test.test 3 0 0 0 3 test.DOMAIN.com 6 0 0 0 6 zcs806.DOMAIN.com 58 0 0 0 58 zcs806.DOMAIN.com 2 0 0 0 2
By Authentication Attempts
A fast way to see who is doing a lot of authentications, which normally happens when a spammer has compromised an account with a weak password, is to do:
# cat /var/log/zimbra.log | sed -n 's/.*sasl_username=//p' | sort | uniq -c | sort -n 1 Auser@domain.com 3 Buser@domain.com 4 Cuser@domain.com 5 Duser@domain.com 36 SPAMMER@domain.com
Note - This might take a long time, if so - try pruning it down
Example:
# cat /var/log/zimbra.log | grep sasl_username > /tmp/zimbra_sasl_username.txt # cat /tmp/zimbra_sasl_username.txt | sed -n 's/.*sasl_username=//p' | sort | uniq -c | sort -n 1 Auser@domain.com 3 Buser@domain.com 4 Cuser@domain.com 5 Duser@domain.com 36 SPAMMER@domain.com
The full log event will look like this:
zimbra1 postfix/smtpd[29431]: B28914D5978: client=xxxxx.server.com[w.x.y.z], sasl_method=LOGIN, sasl_username=user zimbra1 postfix/cleanup[5522]: B28914D5978: message-id=<20090420154255.B28914D5978@zimbraserver.com> zimbra1 postfix/qmgr[20690]: B28914D5978: from=<spam@spam.com>, size=6026, nrcpt=10 (queue active) zimbra1 postfix/cleanup[3983]: 2BA56465D28: message-id=<20090420154255.B28914D5978@zimbraserver.com>
Against your older logs, you could:
# zcat /var/log/zimbra.log* | sed -n 's/.*sasl_username=//p' | sort | uniq -c | sort -n
And you can look at the specific information for the user in question with:
# grep -C2 "sasl_username=SPAMMER@domain.com" /var/log/zimbra.log
Or if searching against the older logs:
# zgrep -C2 "sasl_username=SPAMMER@domain.com" /var/log/zimbra.log*
If you want to check on a specific message ID, do:
grep 9DF7520804A /var/log/zimbra.log*
For older message logs:
zgrep 9DF7520804A /var/log/zimbra.log*
To read/view the message in the queue:
/opt/zimbra/postfix/sbin/postcat -q 9DF7520804A
One would then normally lock/change password on the one account showing the most activity. Grep'ing the /var/log/zimbra.log with the username in question will also show the ip address being used, this can be blocked with your firewall.
To be alerted of a compromised account and have it lock automatically see below. Slightly modified from this reference : http://www.zimbra.com/forums/administrators/62613-identify-compromised-accounts.html#post278732 :
#!/bin/bash # checks log file and gets a count of authentications sent per minute, per user # and if the count exceeds the maxmails value the user's account is locked. logfile="/var/log/zimbra.log" maxmails="10" mydomain="example.com" support="<postmaster-userid>@$mydomain" accounts="/tmp/active_accounts" su zimbra -c "/opt/zimbra/bin/zmaccts" | grep "@" | grep active | awk '{print $1}' > $accounts zgrep -i "auth ok" $logfile | sed 's/ / /g' | awk -F"[ :]" '{print $3":"$4,$11;}' | uniq -c | sort -n | \ while read line do count=`echo ${line} | cut -d' ' -f 1` userid=`echo ${line} | cut -d' ' -f 3` timestamp=`echo ${line} | cut -d' ' -f 2` active=`grep "$userid@$mydomain" $accounts` if [ "$count" -gt "$maxmails" ] && [ "$active" == "$userid@$mydomain" ]; then echo "Maximum email rate exceeded, $userid@$mydomain will be locked" su zimbra -c "/opt/zimbra/bin/zmprov ma $userid@$mydomain zimbraAccountStatus locked" subject="$userid account locked due to excessive connections" # Email text/message message="/tmp/emailmessage.txt" echo "$userid account has been locked as there were $count connections made at"> $message echo "$timestamp. Please have the user change their password, and check for phishing" >>$message echo "emails if possible." >>$message # send an email using /bin/mail /usr/bin/mail -s "$subject" "$support" < $message rm -f $message #update list of active accounts su zimbra -c "/opt/zimbra/bin/zmaccts" | grep "@" | grep active | awk '{print $1}' > $accounts fi done rm -f $accounts
Then run it as a cron job. The frequency will depend on the number of accounts you're managing.
* * * * * /opt/zimbra/find_spammer.sh
By Connecting IP - Useful For Blocking IP At Firewall
See also the following:
To find the originating IP address of where the emails are coming from:
grep 'connect from' /var/log/zimbra.log | sed 's/.*connect from.*\[\(.[^]]*\)\]/\1/g' | sort | uniq -c | sort -nr | head
To check your older logs [example output below]:
zgrep 'connect from' /var/log/zimbra.log* | sed 's/.*connect from.*\[\(.[^]]*\)\]/\1/g' | sort | uniq -c | sort -nr | head 36 10.137.xx.34 34 127.0.0.1
Open Relay Check
You should also confirm you aren't an open relay.
$ host -t mx DOMAIN.com DOMAIN.com mail is handled by 10 mail.DOMAIN.com. $ telnet mail.DOMAIN.com 25 Trying 184.###.##.## ... Connected to mail.DOMAIN.com. Escape character is '^]'. 220 mail.DOMAIN.com ESMTP Postfix helo support.test 250 mail.DOMAIN.com mail from:<SPAMMER@domain.com> 250 2.1.0 Ok rcpt to:<TEST@DOMAIN.COM> 554 5.7.1 <TEST@DOMAIN.COM>: Relay access denied rcpt to:<SPAMMER@domain.com> 554 5.7.1 Service unavailable; Client host [71.XXX.XX.XX] blocked using zen.spamhaus.org; http://www.spamhaus.org/query/bl?ip=71.202.XX.XX quit 221 2.0.0 Bye Connection closed by foreign host.
Telnet Test To Confirm/Show Authentication Required For SMTP/Port 25
This is an example:
esx2:~ ajcody$ telnet zcs723.EXAMPLE.com 25 Trying 10.137.27.32... Connected to zcs723.EXAMPLE.com. Escape character is '^]'. 220 zcs723.EXAMPLE.com ESMTP Postfix helo zcs723.EXAMPLE.com << I typed 250 zcs723.EXAMPLE.com mail from:ajcody@zcs723.EXAMPLE.com << I typed 250 2.1.0 Ok rcpt to:ajcody2@zcs723.EXAMPLE.com << I typed 553 5.7.1 <ajcody@zcs723.EXAMPLE.com>: Sender address rejected: not logged in
But note - if you do this from the ZCS server or a server that is within the ip range or has it's specific ip listed in the mynetworks, you will not get this authentication requirement.
[root@zcs723 ~]# telnet localhost 25 Trying ::1... telnet: connect to address ::1: Connection refused Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. 220 zcs723.EXAMPLE.com ESMTP Postfix helo myworkstation 250 zcs723.EXAMPLE.com ehlo myworkstation 250-zcs723.EXAMPLE.com 250-PIPELINING 250-SIZE 10240000 250-VRFY 250-ETRN 250-STARTTLS 250-AUTH LOGIN PLAIN 250-AUTH=LOGIN PLAIN 250-ENHANCEDSTATUSCODES 250-8BITMIME 250 DSN mail from:<ajcody@zcs723.EXAMPLE.com> 250 2.1.0 Ok rcpt to: <ajcody2@zcs723.EXAMPLE.com> 250 2.1.5 Ok data 354 End data with <CR><LF>.<CR><LF> From: Adam <ajcody@zcs723.EXAMPLE.com> To: Adam 2 <ajcody2@zcs723.EXAMPLE.com> Subject: From Localhost - NOT Auth test . 250 2.0.0 Ok: queued as 8B19E1E78D1 quit 221 2.0.0 Bye Connection closed by foreign host.
Resources
A list of resources you'll find useful:
- Zimbra_MTA#Anti-Spam_Training_Filters
- CLI_zmtrainsa
- Improving_Anti-spam_system
- Postfix_Policyd
- IP_Address_whitelisting
- Spam_training
- Restrict by user
- Restrict by ip addresses and sender and other items:
- Postfix - Restriction Class Readme
- Note, from the readme:
- "What follows is based on the SMTP client IP address, and therefore is subject to IP spoofing."
- "What follows is based on the sender SMTP envelope address, and therefore is subject to SMTP sender spoofing."
- Note, from the readme:
- Postfix - Restriction Class Readme
Wiki articles that have been assigned to the anit-spam category:
Down to the end-user:
External Relay Test Pages
- http://www.checkor.com/
- Note - this test is in regards to the From spoofing spammers sometimes do for DL's.
- Also, for the test - make an account/DL on your system for test1@[your domain] . Otherwise you'll just error about account not existing.
RSET 250 2.0.0 Ok MAIL FROM: spam@mail59.DOMAIN.com 250 2.1.0 Ok RCPT TO: test1@mail59.DOMAIN.com Test Failed, 250 2.1.5 Ok
Blocking MAIL FROM - smtpd_sender_restrictions - Default Is Empty
External References
- External Sources
Zimbra References And Bugs & RFE's
- Zimbra Related Soures
- "policy for who can send to a distribution lists"
- https://bugzilla.zimbra.com/show_bug.cgi?id=9620
- Note - This will not stop spammers from mailing into your DL's by way of forged Mail From and guessing your DL address in the To
- https://bugzilla.zimbra.com/show_bug.cgi?id=9620
- "support smtpd_sender_login_maps for smtp auth"
- "Implement smtpd_sender_restrictions"
- "milter to check if sender can send to a distribution list"
- https://bugzilla.zimbra.com/show_bug.cgi?id=46311
- Dependent upon bug 9620
- https://bugzilla.zimbra.com/show_bug.cgi?id=46311
- Zimbra Forum Post on using smtpd_sender_restrictions options
- [http://www.zimbra.com/forums/administrators/28770-how-enforce-sasl_username-address.html How to enforce sasl_username=FROM ADDRESS"
- Another Zimbra Forum Post on using smtpd_sender_restrictions options
- "policy for who can send to a distribution lists"
Protecting DL's From Spammers - Forging Mail From
First Recommendation - As Given By Dev's From Critical Meeting Notes
- Enabled SASL/SMTP Authentication
- Implement how-to as described in :
- Permitted Senders: RestrictPostfixRecipients
- Note: You'll see on the above page a reference to the spoof hole.
- "This method can be spoofed by forging the MAIL FROM: header (so mail appears to originate from within the domain), so it isn't foolproof, but it works for basic needs."
- Note: You'll also modify the instructions as above with addition details provided below.
- Note: You'll see on the above page a reference to the spoof hole.
- Permitted Senders: RestrictPostfixRecipients
- Force authentication for local-domain senders:
- modify the main.cf to have the following:
- smtpd_sender_restrictions = check_sender_access hash:/path/to/file
- Then for the /path/to/file that you used in the mail.cf for smtpd_sender_restrictions, you'll have a line like:
- example.com permit_sasl_authenticated, reject
- modify the main.cf to have the following:
Second Recommendation - Unpredictable DL name or Non-routing Domain
Do not use predictable DL names. Instead of using everyone@company.com , use something like everyone-[random-string]@company.com .
Another option is to use a non-routing domain - company.local - and setup your DL's there. You'll want to configure your main domain to be able to query the GAL of this domain.
To see the existing setting:
zmprov gd [domainname] zimbraGalLdapSearchBase
To change the variable for the domain:
zmprov md [domainname] zimbraGalInternalSearchBase ROOT
Third Recommendation - Using smtpd_sender_restrictions
- Work In Progress. I'm testing this now. Please don't attempt until this line is removed.
Update: See the following:
- "Enforcing a match between the FROM Address and sasl_username in Zimbra Collaboration Server (2011281)"
This should work if your "clients" are using ZWC, ZCO, or ActiveSync mobile devices. If you are using IMAP/POP + STMP thick clients, you'll most likely have to enable smtp authentication [sasl] and use the reject_authenticated_sender_login_mismatch variable instead.
postconf -e smtpd_sender_restrictions=reject_sender_login_mismatch postfix reload
This option is described as:
- reject_sender_login_mismatch
- Reject the request when $smtpd_sender_login_maps specifies an owner for the MAIL FROM address, but the client is not (SASL) logged in as that MAIL FROM address owner; or when the client is (SASL) logged in, but the client login name doesn't own the MAIL FROM address according to $smtpd_sender_login_maps. Man page
Some Other SMTP Sending Restrictions
Blocking Incoming From Domain And By User
See the following:
check_client_access
The smtpd_client_restrictions parameter restricts what clients this system accepts SMTP connections from. The default behavior is to allow SMTP connections from any client. This is discussed under Spam Controls on the Postfix site.
Example:
check_client_access regexp:/etc/postfix/access_sender_client_server,
Example:
check_sender_access regexp:/etc/postfix/access_sender_toplevel
smtpd_reject_unlisted_sender
Details can be found on the mail.cf Postfix page.
Example:
smtpd_reject_unlisted_sender = yes
Possible Bug:
http://bugzilla.zimbra.com/show_bug.cgi?id=24889
What's Your SPF Records Say, When Getting "does not designate 74.x.x.x as permitted sender Errors"
This is most likely related to the SPF records for your domain and what the header content of the sending email states as it's Mail From. For example, this is from the header of an email that was "received":
Received: from mail.XYZ-FAKE.com (mailhost.XYZ-FAKE.com [74.X.X.244]) by mta01.ABC-FAKE.com with ESMTP id Cft0mO3fjlFGQjTA for <support@ABC-FAKE.com>; Tue, 21 Apr 2009 05:14:13 -0700 (PDT) X-Barracuda-Envelope-From: testuser@XYZ-FAKE.com Received-SPF: pass (mta01.ABC-FAKE.com: domain of testuser@XYZ-FAKE.com designates 74.X.X.244 as permitted sender) receiver=mta01.ABC-FAKE.com; client_ip=74.X.X.244; envelope-from=testuser@XYZ-FAKE.com;
To see what this check was done against, do the following below. I'll trim the output and adjust the information used to protect the innocent. Also, notice how a DNS "alias" might cause an issue here? :
$ host 74.X.X.244 244.X.X.74.in-addr.arpa domain name pointer mailhost.XYZ-FAKE.com. $ host mailhost.XYZ-FAKE.com mailhost.XYZ-FAKE.com has address 74.X.X.244 $ host mail.XYZ-FAKE.com mail.XYZ-FAKE.com is an alias for mailhost.XYZ-FAKE.com. mailhost.XYZ-FAKE.com has address 74.X.X.244 $ dig XYZ-FAKE.com MX ;; QUESTION SECTION: ;XYZ-FAKE.com. IN MX ;; ANSWER SECTION: XYZ-FAKE.com. 3600 IN MX 22 serverA.DNS-FAKE.com. XYZ-FAKE.com. 3600 IN MX 11 serverB.DNS-FAKE.com. $ dig XYZ-FAKE.com TXT ;; QUESTION SECTION: ;XYZ-FAKE.com. IN TXT ;; ANSWER SECTION: XYZ-FAKE.com. 3600 IN TXT "v=spf1 a:mail.XYZ-FAKE.com ~all"
See the following for more information:
- http://www.openspf.org/
- http://www.schlitt.net/spf/spf_classic/draft-schlitt-spf-classic-02.html#publishing
- All of the 3.x section.
Using Different SMTP Server For Webclient (ZWC), Mobiles, And ZCO
Note Of Caution About Using External MTAs
Using non-zimbra MTA's can cause some options in zimbra to not function anymore - since it no longer has zimbra's mta services available.
Zimbra Mail Forwarding Possibly Will Not Work - Turn Off User Option To Set MailForwarding
Mail forwarding might no longer work depending on the configuration you setup regarding the use of your external mta's. When this happens you'll most likely want to disable the option for users to set a mail forwarding address in their preferences. This can be done via their COS or USER configuration.
- In the admin console, goto the COS configuration the user/s are using and the "Features" tab. Uncheck the option "Allow the user to specify a forwarding address" under Mail Features. It is in the same location under a USERs configuration panel in the admin console.
In the CLI, you will see these set as the defaults for the default COS. The admin gui option above only adjusts the zimbraFeatureMailForwardingEnabled variable :
$ zmprov gc default | grep zimbraFeatureMailForwarding zimbraFeatureMailForwardingEnabled: TRUE zimbraFeatureMailForwardingInFiltersEnabled: TRUE
$ zmprov ga ajcody@`zmhostname` | grep zimbraFeatureMailForwarding zimbraFeatureMailForwardingEnabled: TRUE zimbraFeatureMailForwardingInFiltersEnabled: TRUE
More details about them are in the /opt/zimbra/conf/attrs/zimbra-attrs.xml file.
<attr id="342" name="zimbraFeatureMailForwardingEnabled" type="boolean" cardinality="single" optionalIn="account,cos" flags="accountInfo,accountInherited,domainAdminModifiable"> <defaultCOSValue>TRUE</defaultCOSValue> <desc>enable end-user mail forwarding features</desc> </attr> <attr id="704" name="zimbraFeatureMailForwardingInFiltersEnabled" type="boolean" cardinality="single" optionalIn="account,cos" flags="accountInfo,accountInherited, domainAdminModifiable" since="5.0.10"> <defaultCOSValue>TRUE</defaultCOSValue> <desc>enable end-user mail forwarding defined in mail filters features</desc> </attr>
Configure External MTA To Use LDAP Virtual Alias Maps
Here's the basic info in regards to how Zimbra's mta [postfix/etc] uses Zimbra's LDAP to get the forwarding information:
$ grep Forward conf/ldap-* conf/ldap-vam.cf:result_attribute = zimbraMailDeliveryAddress,zimbraMailForwardingAddress, zimbraPrefMailForwardingAddress,zimbraMailCatchAllForwardingAddress
$ postconf |grep vam virtual_alias_maps = proxy:ldap:/opt/zimbra/conf/ldap-vam.cf
See http://www.postfix.org/postconf.5.html#virtual_alias_maps for more information.
Confirming And Setting zimbraMtaRelayHost And zimbraMtaDnsLookupsEnabled
First we'll set zimbraMtaRelayHost and zimbraMtaDnsLookupsEnabled variables. These options are also shown in the admin console and can be configured there. These variable alone will not redirect ALL traffic to an external MTA first though. There's alao a variable called zimbraSmtpHostname that is in the global (zmprov gacf) and server (zmprov gs `hostname`) configs - addressed in the section below. It's defaulted value is 'localhost' - at least on a single ZCS configuration.
In situations where you need/want all mail to be processed by, for example, an external non-Zimbra spam filter box you could set this variable to the spam filter servers hostname.
Normally, when zimbraMtaRelayHost is set to a non-zimbra external MTA would disable DNS lookups. If you disable DNS Lookups (under the MTA tab of the admin console, or with zmprov), Zimbra will end up using (according to the postconf man page) the "gethostbyname() system library routine which normally also looks in /etc/hosts" (based on the entries on the "hosts" line in /etc/nsswitch.conf). If you do this but don't also specify an SMTP relay host (typically your ISP's SMTP server), which will take care of checking DNS, you will reverse your ability to send mail: suddenly you can send mail to other users on the Zimbra server, but you can't send to the internet (though you can still receive mail from the internet either way).
Query Global - `zmhostname` would use the value returned, you can manual type out the servername as well :
zmprov gacf zimbraMtaRelayHost zmprov gacf zimbraMtaDnsLookupsEnabled
Query Per Server - `zmhostname` would use the value returned, you can manual type out the servername as well :
zmprov gs `zmhostname` zimbraMtaRelayHost zmprov gs `zmhostname` zimbraMtaDnsLookupsEnabled
Note - if you get errors about doing the query on your non-mailstores like, "ERROR: zclient.IO_ERROR (invoke Connection refused, server: localhost) (cause: java.net.ConnectException Connection refused)" then you might need to adjust this variable. First query it, it's most likely set to localhost
zmlocalconfig zimbra_zmprov_default_soap_server
If you had the error and it was set to localhost, modify it to be one of your mailstores.
zmlocalconfig -e zimbra_zmprov_default_soap_server=mailstore.example.com
No restart of anything is needed, the zmprov query should now work.
Modify Global - `zmhostname` would use the value returned, you can manual type out the servername as well :
zmprov mcf zimbraMtaRelayHost hostname-of-ext-server:PORT ** ex : zmprov mcf zimbraMtaRelayHost primary.YYY.state.XX.us:25 ** At the end, 25 is the port number for smtp on the targeted system. ** Adjust this number if you changed the smtp port. zmprov mcf zimbraMtaDnsLookupsEnabled FALSE
Modify Per Server - `zmhostname` would use the value returned, you can manual type out the servername as well :
zmprov ms `zmhostname` zimbraMtaRelayHost hostname-of-ext-server:PORT ** ex : zmprov ms `zmhostname` zimbraMtaRelayHost primary.YYY.state.XX.us:25 ** At the end, 25 is the port number for smtp on the targeted system. ** Adjust this number if you changed the smtp port. zmprov ms `zmhostname` zimbraMtaDnsLookupsEnabled FALSE
Confirming And Setting zimbraSmtpHostname
I'm assuming you already set, zimbraMtaRelayHost and zimbraMtaDnsLookupsEnabled for your needs - see above section. The variable called zimbraSmtpHostname is in the global (zmprov gacf) and server (zmprov gs `hostname`) configs. It's default value is set to 'localhost' - at least on a single ZCS configuration.
In cases where you need/want all mail to be processed by, for example, an external non-Zimbra spam filter box you could set this variable to the spam filter boxes hostname.
Query Global - `zmhostname` would use the value returned, you can manual type out the servername as well :
zmprov gacf zimbraSmtpHostname
Query Per Server - `zmhostname` would use the value returned, you can manual type out the servername as well :
zmprov gs `zmhostname` zimbraSmtpHostname
Note - if you get errors about doing the query on your non-mailstores like, "ERROR: zclient.IO_ERROR (invoke Connection refused, server: localhost) (cause: java.net.ConnectException Connection refused)" then you might need to adjust this variable. First query it, it's most likely set to localhost
zmlocalconfig zimbra_zmprov_default_soap_server
If you had the error and it was set to localhost, modify it to be one of your mailstores.
zmlocalconfig -e zimbra_zmprov_default_soap_server=mailstore.example.com
No restart of anything is needed, the zmprov query should now work.
Modify Global - `zmhostname` would use the value returned, you can manual type out the servername as well :
zmprov mcf zimbraSmtpHostname hostname-of-ext-server ** ex : zmprov mcf zimbraSmtpHostname primary.YYY.state.XX.us
Modify Per Server - `zmhostname` would use the value returned, you can manual type out the servername as well :
zmprov ms `zmhostname` zimbraSmtpHostname hostname-of-ext-server ** ex : zmprov ms `zmhostname` zimbraSmtpHostname primary.YYY.state.XX.us
External Email Clients Setting A SMTP Server
The above variable, zimbraSmtpHostname, will not alter your third party email clients that are setting the smtp server to your ZCS mta's. Here are your options if you also need to have that traffic to go through another device [mta, spam filter, etc.] prior to local delivery [lmtp] to an internal address.
- Set your clients to use the another devices ip address or hostname that you set for zimbraSmtpHostname.
If you can't do the above, for whatever reason -- maybe security constraints or issues that might arise being a hosting provider, then see below.
- You could investigate the alternation of postfix's content_filter option to place the external device/host [a barracuda for example] within that process. postfix.org has information on this - Postfix After-Queue Content Filter. This would be unsupported by Zimbra.
- Contact Zimbra's Professional Services [PS] team for help.
- Setup another server using a mta of your choice [postfix , sendmail] that the clients can use for the smtp server variable. This "new" mta would then simply relay to the device - a barracuda box for example. The barracuda would then do what it needs to and then forwards the messages to the appropriate servers for delivery. Your ZCS mta's in the case of local delivery that would of normally occurred over lmtp - userA@domainC.com sending to userB@domainC.com .
Global Or System Wide Filters
There is no "supported" solution for this - depending on the exact circumstances. It's a complex issue because each request for "global filters" tends to be very specific on either what's to be filter, what actions are to be taken, and so forth.
RFE's Related To Global Filters
- "Define a default filter rule for spam that users can apply before custom filters"
- "admin assignable mail filters"
Use The Legal Intercept Method
Depending on the details of your request, using the Legal Intercept options might be useful. You could take the results that goto the LI mailbox and then perform your admin global searches against your mailbox accounts to take the appropriate action.
See Legal_Intercept
Sieve Filter Set For Every Account
One could setup a forloop for all of your users and setup a sieve filter. Downside here is the rules are editable by the users and you would have to manage the rules for new accounts going forward.
See User_Migration#Migrating_Sieve_Filter_Rules for details.
Double Check The Current Anti-Spam Options
Make sure your request can't be solved by the current solutions described in articles.
Postfix , Amavis Customizations
I've seen some posts on the forums that customers found their own workarounds by customizing postfix and amavis. This will most likely result in an unsupported situation. Unfortunately, those forum posts don't also include details that I can share here. Hopefully I can find them going forward and I'll post them here.
Global Disclaimer Options
Here's the url to review for a "current" possibility:
And in the notes section there's a comment about multi-servers:
Please note though, "This article is a community contribution and may include unsupported customizations." Meaning, it's an unsupported customization, so please take the necessary precautions.
In regards to an official and supported way to do this, please review this RFE:
Quota Issues
Where To Adjust Message User Gets When They Are Over Quota
Moved to How_To_Adjust_The_Over_Quota_Message_The_User_Receives
See Current User Qoutas
Moved to Getting_All_Users_Quota_Data
Controlling Behavior For Messages Sent To Over Quota Mailbox - LMTP
Moved to Controlling_Behavior_For_Messages_Sent_To_Over_Quota_Mailbox_-_LMTP
Controlling Behavior For Messages Sent To Over Quota Mailbox - SMTP
Moved to Controlling_Behavior_For_Messages_Sent_To_Over_Quota_Mailbox_-_SMTP
Message Senders Receive About Mailbox Over Quota
Moved to Message_Senders_Receive_About_Mailbox_Over_Quota
Quota Not Showing In Admin Console - After ZCS Upgrade
Moved to Quota_Not_Showing_In_Admin_Console_-_After_ZCS_Upgrade
Managing Postfix Queue
Moved to Managing_The_Postfix_Queues#Managing_The_Postfix_Queues
Postfix, Amavis, Clamav Spool Directory Paths And Names
Moved to Managing_The_Postfix_Queues#Postfix.2C_Amavis.2C_Clamav_Spool_Directory_Paths_And_Names
Stop And Starting Postfix And Mta
Moved to Managing_The_Postfix_Queues#Stop_And_Starting_Postfix_And_Mta
To See Postfix Queue
Moved to Managing_The_Postfix_Queues#To_See_The_Postfix_Queues
Qshape - Print Postfix queue domain and age distribution
Moved to Managing_The_Postfix_Queues#Qshape_-_Print_Postfix_queue_domain_and_age_distribution
To View A Message In The Queue
Moved to Managing_The_Postfix_Queues#To_View_A_Message_In_The_Queue
To Flush Postfix Queue
Moved to Managing_The_Postfix_Queues#To_Flush_Postfix_Queue
To Requeue Messages In Postfix
Moved to Managing_The_Postfix_Queues#To_Requeue_Messages_In_Postfix
To Put Messages On Hold
Moved to Managing_The_Postfix_Queues#To_Put_Messages_On_Hold
To Delete Messages From Queue
Moved to Managing_The_Postfix_Queues#To_Delete_Messages_From_Queue
Cautionary Note
Moved to Managing_The_Postfix_Queues#Cautionary_Note
Relevant Sections Of Postsuper Man Page
Moved to Managing_The_Postfix_Queues#Relevant_Sections_Of_Postsuper_Man_Page
To Delete Single Message From Queue
Moved to Managing_The_Postfix_Queues#To_Delete_Single_Message_From_Queue
To Delete ALL Messages From Queue
Moved to Managing_The_Postfix_Queues#To_Delete_ALL_Messages_From_Queue
To Delete ALL Messages From The Deferred Queue
Moved to Managing_The_Postfix_Queues#To_Delete_ALL_Messages_From_The_Deferred_Queue
To Delete ALL Messages From The Hold Queue
Moved to Managing_The_Postfix_Queues#To_Delete_ALL_Messages_From_The_Hold_Queue
To Delete Many Messages From Queue
Moved to Managing_The_Postfix_Queues#To_Delete_Many_Messages_From_Queue
Delete From Queue By Email Address
Moved to Managing_The_Postfix_Queues#Delete_From_Queue_By_Email_Address
From CLI
Moved to Managing_The_Postfix_Queues#From_CLI
Script To Delete From Queue By Email Address
Moved to Managing_The_Postfix_Queues#Script_To_Delete_From_Queue_By_Email_Address
Script To Delete From Queue By Various Variable Targets
Moved to Managing_The_Postfix_Queues#Script_To_Delete_From_Queue_By_Various_Variable_Targets
Ajcody MySQL Topics
Mysql
Actual Mysql Topics Homepage
Please see Ajcody-Mysql-Topics
Mysql Issues And Memory Usuage
My Ajcody-General-Notes#Trouble_Shooting_Memory_Issues has some stuff on memory troubleshooting - generalities.
Our performance guide for Mysql:
http://wiki.zimbra.com/index.php?title=Performance_Tuning_Guidelines_for_Large_Deployments#MySQL
Variable Within MySQL & Zimbra
These percentages below are used against physical RAM - disregard swap in the percentage division.
From the above url, we get some details.
"By default, we try to reserve 30% of system memory for use by this process, and 40% for use by MySQL."
# ZCS 5.0 and later $ zmlocalconfig -e mailboxd_java_heap_memory_percent=40
Java - mailboxd_java_heap_memory_percent
The command to set the java percentage is:
zmlocalconfig -e mailboxd_java_heap_memory_percent=30
Mysql - mysql_memory_percent & my.cnf w/ innodb_buffer_pool_size
Looks like there's an issue with the command/variable (zmlocalconfig -e mysql_memory_percent=##) in regards to MySQL though.
"There is a local config variable [mysql_memory_percent] for mysql memory percent, but today my.cnf doesn't get rewritten after install, so you have to edit my.cnf for this setting if you want to change it."
In summary, this variable [ mysql_memory_percent ] for MySQL is basically not working.
Do this to get memory in bytes for division numbers:
free -b
Figure out what your % in bytes using the physical memory (Mem:) output. Now edit the /opt/zimbra/conf/my.cnf file to change the following line with your memory percentage in bytes:
vi /opt/zimbra/conf/my.cnf #Modify this var to adjust memory usage innodb_buffer_pool_size = xxx
This will require a restart.
A GREAT forum post on this is here:
- http://www.zimbra.com/forums/administrators/21519-memory-usage-slowness-backup-error.html#post107080
So other background info:
- http://www.mysqlperformanceblog.com/2007/11/01/innodb-performance-optimization-basics/
- http://www.mysqlperformanceblog.com/2007/11/03/choosing-innodb_buffer_pool_size/
Mysql And Swap
See this article first before exploring the options below.
Should you have your swap file enabled while running MySQL ?
Set swappiness Kernel Var To 0
Source of this summary is here: [http://www.mysqlperformanceblog.com/2007/09/28/heikki-tuuri-to-answer-your-in-depth-innodb-questions/#comment-174442 Swappiness To 0"
Set it on running box:
echo 0 > /proc/sys/vm/swappiness
Set it to persist after reboot - vi /etc/sysctl.conf :
vm.swappiness = 0
The value is a percentage. By setting it to 100, Linux will always swap programs out and never shrink the buffer cache.
Swap Ram Disk Instead Of Disk
Source of this summary is here:
MySQL and the Linux swap problem
Make a (or some) swap disks:
mkdir /mnt/ram0 dd bs=1024 count=14634 if=/dev/zero of=/mnt/ram0/swapfile mkswap /mnt/ram0/swapfile swapon /mnt/ram0/swapfile
Will look like this:
[root@mail3 ~]# mkdir /mnt/ram0 [root@mail3 ~]# dd bs=1024 count=14634 if=/dev/zero of=/mnt/ram0/swapfile 14634+0 records in 14634+0 records out 14985216 bytes (15 MB) copied, 0.246329 seconds, 60.8 MB/s [root@mail3 ~]# mkswap /mnt/ram0/swapfile Setting up swapspace version 1, size = 14979 kB [root@mail3 ~]# free total used free shared buffers cached Mem: 1555472 1461200 94272 0 159400 344328 -/+ buffers/cache: 957472 598000 Swap: 2031608 104 2031504 [root@mail3 ~]# swapon /mnt/ram0/swapfile [root@mail3 ~]# free total used free shared buffers cached Mem: 1555472 1476252 79220 0 159404 344376 -/+ buffers/cache: 972472 583000 Swap: 2046232 104 2046128 [root@mail3 ~]# swapon -s Filename Type Size Used Priority /dev/mapper/VolGroup00-LogVol01 partition 2031608 104 -1 /mnt/ram0/swapfile file 14624 0 -2 [root@mail3 ~]# cat /proc/swaps Filename Type Size Used Priority /dev/mapper/VolGroup00-LogVol01 partition 2031608 104 -1 /mnt/ram0/swapfile file 14624 0 -2
Just add those lines to your relevant startup file, like /etc/rc.d/rc.local, and it’ll persist after reboots.
O_DIRECT On Linux And INNODB To Fix Swapping
Source reference for this part is here:
Using O_DIRECT on Linux and INNODB to Fix Swap Insanity
See also: Ajcody-Mysql-Topics#IO_DIRECT_And_Ext3_On_Linux_As_Possible_Reason_For_Corruption
IO_DIRECT And Ext3 On Linux As Possible Reason For Corruption
This is being investigated. Below are my notes on the issue.
- We recommend this setting that might be causing issue:
- IO_DIRECT For MySQL Can Be Detrimental
- Mysql Bug - "innodb_flush_method = O_DIRECT may degrade performance of InnoDB on SAN":
- Comment about O_Direct from Linus:
- EXT3 not RAID-5 may be the cause of performance issues
- Numerous Items On This Blog Page:
- http://mysqldba.blogspot.com/2008_01_01_archive.html
- Most relevant are these sections:
- http://mysqldba.blogspot.com/2008_01_01_archive.html
Mysql Backup And Restore
Please see the MySQL_Backup_and_Restore page.
Mysql Database Corruption/Repair/Checks
Please see the Mysql_Crash_Recovery page rather than the normal search results that give mysql-logger details.
Mysql Table Checks
Mysql Table Types - InnoDB or MyISAM
Tables are either InnoDB or MyISAM. Putting this information here because if a table is in need of repair or something, the steps to fix it might be different depending on the tables ENGINE type.
Example for MyISAM:
$ mysql -e "show create table zimbra.jiveRoster" [cut] ) ENGINE=MyISAM DEFAULT CHARSET=utf8 | [cut]
Example for InnoDB:
$ mysql -e "show create table zimbra.mailbox" [cut] ) ENGINE=InnoDB DEFAULT CHARSET=utf8 | [cut]
check zimbra.mailbox database tables
$ mysql -e "check table zimbra.mailbox" +----------------+-------+----------+----------+ | Table | Op | Msg_type | Msg_text | +----------------+-------+----------+----------+ | zimbra.mailbox | check | status | OK | +----------------+-------+----------+----------+
check mboxgroup[#].mail_item table
$ mysql -e "check table mboxgroup3.mail_item" +----------------------+-------+----------+----------+ | Table | Op | Msg_type | Msg_text | +----------------------+-------+----------+----------+ | mboxgroup3.mail_item | check | status | OK | +----------------------+-------+----------+----------+
jive.Roster or zimbra.jiveUserProp Errors From Daily Report With mysqlcheck Errors
- Please note, these tables are MyISAM. Don't use the below steps for other type of tables as each table might be treated differently or with extra caution. [InnoDB vs MyISAM]
Your report from /opt/zimbra/libexec/zmdbintegrityreport might give the following for example:
Database errors found. /opt/zimbra/mysql/bin/mysqlcheck --defaults-file=/opt/zimbra/conf/my.cnf -S /opt/zimbra/db/mysql.sock -A -C -s -u root --password=[mysql password] zimbra.jiveID warning : 1 client is using or hasn't closed the table properly zimbra.jiveRoster warning : 1 client is using or hasn't closed the table properly zimbra.jiveUserProp warning : 2 clients are using or haven't closed the table properly
Attempt the following:
mysql -e "check table zimbra.jiveRoster" +-------------------+-------+----------+----------+ | Table | Op | Msg_type | Msg_text | +-------------------+-------+----------+----------+ | zimbra.jiveRoster | check | status | OK | +-------------------+-------+----------+----------+
To see if the data "looks" fine or works correctly against a select, do:
mysql -e "select * from zimbra.jiveRoster"
Note, if the above command might dump out a lot of data, try:
mysql -e "select * from zimbra.jiveRoster" > /tmp/zimbra.jiveRoster.out
Yours might show something different for Msg_text. If you need to repair, you would run:
mysql -e "repair table zimbra.jiveRoster"
You can also do the same above but replacing zimbra.jiveRoster with zimbra.jiveUserProp .
Manual Change Of Mysql Passwords
Please see:
http://wiki.zimbra.com/index.php?title=Issues_with_mysql_and_logmysql_passwords
Working Within Mysql For Accounts
- Note, the mailboxId is specific to the mailstore the user resides on. The zimbraId is the users unique id system wide - zmprov ga user@domain.com zimbraId .
Other References:
Getting User Variables We Need To Query MySQL
Getting The Users mailboxId # - zmprov gmi user@domain
For Mysql, this # returned below will become apart of mailbox_id=### . Example used is mailbox_id=3 .
- Note, the mailboxId is specific to the mailstore the user resides on. The zimbraId is the users unique id system wide - zmprov ga user@domain.com zimbraId .
[root@mail39 ~]# su - zimbra [zimbra@mail39 ~]$ zmprov gmi ajcody@mail39.zimbra.DOMAIN.com mailboxId: 3 quotaUsed: 169831
Getting The Users Mailstore That Their DB Resides On - zmprov ga user@domain zimbraMailHost
The mailboxId found above for the user is unique to their actual mailstore and the db that resides there. Again:
- Note, the mailboxId is specific to the mailstore the user resides on. The zimbraId is the users unique id system wide - zmprov ga user@domain.com zimbraId .
To get their particular mailstore their database resides on:
$ zmprov ga ajcody@mail39.zimbra.DOMAIN.com zimbraMailHost # name ajcody@mail39.zimbra.DOMAIN.com zimbraMailHost: mail39.zimbra.DOMAIN.com
So you would now log onto that mailstore to do your mysql operations (searches).
Confirm zimbra_mailbox_groups Variable
Need to confirm zimbra_mailbox_groups variable to get the users mboxgroup# since we'll then need to run the expression below to determine it.
[zimbra@mail39 ~]$ zmlocalconfig zimbra_mailbox_groups zimbra_mailbox_groups = 100
The Mysql group_id Value
The group_id is controlled by a local configuration value. It defaults to 100.
$ zmlocalconfig zimbra_mailbox_groups zimbra_mailbox_groups = 100
You'll see group_id in the following:
$ mysql -e 'SELECT * FROM zimbra.mailbox WHERE id=3\G' *************************** 1. row *************************** id: 3 group_id: 3 *** CUT REST OF OUTPUT ***
Get Users mboxgroup #
Finally confirming what the mboxgroup is for the user by running the following below. The mboxgroup is comprised of multiple users that are unique by way of their mailbox_id number. This is to minimize the number of db's that are needed on the mailstore, rather than doing one db per user.
- 3 ='s the users mailboxId
- 100 ='s the variable from zimbra_mailbox_groups
For Mysql, this # returned below will become apart of mboxgroup###.mail_item = mboxgroup3.mail_item
[zimbra@mail39 ~]$ expr 3 % 100 3
Queries Against Mysql For User Information In The zimbra DB And mailbox Table - zimbra.mailbox
By mailboxId
You'll need to know the mailboxId for the user for this. [ zmprov gmi USER@DOMAIN.com ]
Format:
$ mysql -e 'SELECT * FROM zimbra.mailbox WHERE id=<mailboxID of USER>\G'
Working example:
$ mysql -e 'SELECT * FROM zimbra.mailbox WHERE id=3\G' *************************** 1. row *************************** id: 3 group_id: 3 account_id: 3be48432-926c-4a54-bd66-3b16185a37a4 index_volume_id: 2 item_id_checkpoint: 339 contact_count: 5 size_checkpoint: 35912 change_checkpoint: 7200 tracking_sync: 0 tracking_imap: 0 last_backup_at: 1231567229 comment: ajcody@zimbra.DOMAIN.com last_soap_access: 1229584283 new_messages: 1 idx_deferred_count: 0
By zimbraId
You'll need to know the zimbraId for the user for this.
Via zmprov:
$ zmprov ga ajcody@zimbra.DOMAIN.com zimbraId # name ajcody@zimbra.DOMAIN.com zimbraId: 3be48432-926c-4a54-bd66-3b16185a37a4
Or ldap:
$ su - zimbra $ source ~/bin/zmshutil $ zmsetvars $ ldapsearch -LLL -x -H $ldap_master_url -D $zimbra_ldap_userdn -w $zimbra_ldap_password "mail=ajcody@zimbra.DOMAIN.com" zimbraId dn: uid=ajcody,ou=people,dc=zimbra,dc=DOMAIN,dc=com zimbraId: 3be48432-926c-4a54-bd66-3b16185a37a4
Format:
$ mysql -e 'SELECT * FROM zimbra.mailbox WHERE account_id=<zimbraID of USER>\G'
Working example:
$ mysql -e 'SELECT * FROM zimbra.mailbox WHERE account_id="3be48432-926c-4a54-bd66-3b16185a37a4"\G' *************************** 1. row *************************** id: 3 group_id: 3 account_id: 3be48432-926c-4a54-bd66-3b16185a37a4 index_volume_id: 2 item_id_checkpoint: 339 contact_count: 5 size_checkpoint: 35912 change_checkpoint: 7200 tracking_sync: 0 tracking_imap: 0 last_backup_at: 1231567229 comment: ajcody@zimbra.DOMAIN.com last_soap_access: 1229584283 new_messages: 1 idx_deferred_count: 0
By Comment Field - I.E. Username
This way searches against the "comment field". The comment field is mostly correct, but it is deemed unreliable since there are some corner cases where it can be duplicated. Use it in a pinch.
Format:
$ mysql -e 'SELECT * FROM zimbra.mailbox WHERE comment LIKE "<USER>%"\G'
Working example:
$ mysql -e 'SELECT * FROM zimbra.mailbox WHERE comment LIKE "ajcody%"\G' *************************** 1. row *************************** id: 3 group_id: 3 account_id: 3be48432-926c-4a54-bd66-3b16185a37a4 index_volume_id: 2 item_id_checkpoint: 339 contact_count: 5 size_checkpoint: 35912 change_checkpoint: 7200 tracking_sync: 0 tracking_imap: 0 last_backup_at: 1231567229 comment: ajcody@zimbra.DOMAIN.com last_soap_access: 1229584283 new_messages: 1 idx_deferred_count: 0
Show all accounts on mailstore
Show all accounts on mailstore.
$ mysql -e 'SELECT * FROM zimbra.mailbox WHERE comment LIKE "%"\G' | grep comment comment: admin@zimbra.DOMAIN.com comment: wiki@zimbra.DOMAIN.com comment: spam.1c4mokie@zimbra.DOMAIN.com comment: ham.iudu0ic_mz@zimbra.DOMAIN.com comment: shared-account@zimbra.DOMAIN.com comment: testuser@zimbra.DOMAIN.com comment: large-share@zimbra.DOMAIN.com comment: cos-user@zimbra.DOMAIN.com comment: arch-prod@zimbra.DOMAIN.com
Query All Users In A mboxgroup##
This is very useful, for example, if you experience corruption in one of your mboxgroup## db files when you want to see who is effected. Remember, these are unique to mailstore (group_id = mboxgroup)(id = userid). The account_id is a global variable, the comment field should reflect the account_id's email address - another global variable.
Note - I'm on a small test server, hence why only one account is showing. zimbra_mailbox_groups = 100 [default] determines the 'round-robin' behavior of assigning users to mboxgroup's
$ mysql -e 'select id, comment, account_id, group_id from zimbra.mailbox where group_id=5' +----+-------------------------------------------+--------------------------------------+----------+ | id | comment | account_id | group_id | +----+-------------------------------------------+--------------------------------------+----------+ | 5 | ham.1msrt1ugis@mail37.DOMAIN.com | 9ded4fa5-62fe-4ae4-ac14-7a8928386ea8 | 5 | +----+-------------------------------------------+--------------------------------------+----------+
You can also, double check, information the account_id and comment with zmprov then:
$ zmprov gmi 9ded4fa5-62fe-4ae4-ac14-7a8928386ea8 mailboxId: 5 quotaUsed: 0 $ zmprov gmi ham.1msrt1ugis@mail37.DOMAIN.com mailboxId: 5 quotaUsed: 0
Just to be complete, below I'll list the location of the various db's in question based upon the filesystem.
$ pwd /opt/zimbra/db/data/zimbra $ ls mailbox* mailbox.frm mailbox.ibd mailbox_metadata.frm mailbox_metadata.ibd $ cd ../mboxgroup5/ $ pwd /opt/zimbra/db/data/mboxgroup5 $ ls appointment.frm data_source_item.ibd imap_folder.ibd mail_item.frm open_conversation.ibd revision.frm tombstone.ibd appointment.ibd db.opt imap_message.frm mail_item.ibd pop3_message.frm revision.ibd data_source_item.frm imap_folder.frm imap_message.ibd open_conversation.frm pop3_message.ibd tombstone.frm
Deleting An Account In Mysql
This comes up in very rare circumstances, usually when an account rename was attempted and for some reason it fails -- leaving the account in an odd state, where ldap and mysql data no longer match up. Use with extreme caution and under guidance of support staff.
The Steps To Delete User In Mysql
Caution - This Should Almost Never Need To Be Done. Usually Only Needed When Ldap Doesn't Have Account Anymore And For Some Reason Account Still Exists In Mysql
Use the above section, Ajcody-Mysql-Topics#Getting_User_Variables_We_Need_To_Query_MySQL , to get mailboxId or other needed data. The below example will most likely fail with the zmprov command because of missing data in ldap. You still might need to do the expr example below when you do your mysql delete commands.
# Fetch the user's mailbox Id # zmprov will most likely fail in the situation where you need to do this process # Use Ajcody-Mysql-Topics#Getting_User_Variables_We_Need_To_Query_MySQL # alternative methods to get this variable - mailboxId [root@]# su - zimbra [zimbra@]$ zmprov getMailboxInfo user1@example.com mailboxId: 11 quotaUsed: 30620 # Fetch the user's mboxgroup Id # Default is 100 for zimbra_mailbox_groups
$ zmlocalconfig zimbra_mailbox_groups zimbra_mailbox_groups = 100
# expr [mailboxId] % [zimbra_mailbox_groups]
$ expr 11 % 100 11
Each user is referenced by a unique mailboxId within Mysql. Each mailstore has it's own Mysql database and therefore the user mailboxId's are only locally unique - i.e., the id isn't a unique global variable. Replace id=#####, mboxgroup## and mailbox_id=##### with relevant data. An example might have mboxgroup77 , id=18577 and mailbox_id=18577.
[root@]# su - zimbra [zimbra@]# mysql mysql> SET foreign_key_checks = 0; mysql> use mboxgroup##; mysql> delete from appointment where mailbox_id=#####; mysql> delete from data_source_item where mailbox_id=#####; mysql> delete from imap_folder where mailbox_id=#####; mysql> delete from imap_message where mailbox_id=#####; mysql> delete from mail_item where mailbox_id=#####; mysql> delete from open_conversation where mailbox_id=#####; mysql> delete from pop3_message where mailbox_id=#####; mysql> delete from revision where mailbox_id=#####; mysql> delete from tombstone where mailbox_id=#####; mysql> use zimbra; mysql> delete from mailbox where id=#####; mysql> delete from mailbox_metadata where mailbox_id=#####; mysql> SET foreign_key_checks = 1 mysql> quit
[zimbra@]# zmprov fc account
Useful Mysql Query Examples
First - Get User Variables we need to query MySQL
Please see Ajcody-Mysql-Topics#Getting_User_Variables_We_Need_To_Query_MySQL
Mysql Query For A Users Specific Blob-Data ID
Query of Mysql against users mboxgroup [mboxgroup#.mail_item] for a SPECIFIC message [id=#]
- Note, in this case, the users mailbox_id isn't necessary
[zimbra@mail39 ~]$ mysql -e 'SELECT * FROM mboxgroup3.mail_item WHERE id=321\G' *************************** 1. row *************************** mailbox_id: 3 id: 321 type: 5 parent_id: NULL folder_id: 5 index_id: 321 imap_id: 321 date: 1281984004 size: 448 volume_id: 1 blob_digest: 0pe,M6lHRy4KBXbIyEeR7AJzfkA= unread: 0 flags: 8193 tags: 0 sender: Adam Cody subject: test w mobile sending name: NULL metadata: d1:f0:1:s45:Adam Cody <ajcody@mail39.zimbra.DOMAIN.com>1:t32:admin@mail38.zimbra.DOMAIN.com1:vi10ee mod_metadata: 21394 change_date: 1281984004 mod_content: 21394
Content Of Message Example Above As On Filesystem
How To Locate Users Mailstore and Message Store Directory
The basics:
- $ zmprov ga USER@DOMAIN zimbraMailHost
- $ zmprov gmi USER@DOMAIN
- Note the mailboxId for the user.
- Then ssh to the zimbraMailHost the user is on.
- $ zmvolume -l
- Confirm the path for the primaryMessage volume, default is /opt/zimbra/store .
- cd to the primaryMessage volume path, example uses the default path.
- $ cd /opt/zimbra/store
- find . -maxdepth 2 -name [replace with the mailboxId of the user] -print
- or do perl -e 'print mailboxId# >> 12 ; print "\n"'
- You can now cd to the users msg directory.
Working example:
$ zmprov ga userA@DOMAIN zimbraMailHost # name userA@mail71.DOMAIN.com zimbraMailHost: mail71.DOMAIN.com [I'm already on the zimbraMailHost for this user, no need to ssh to it] $ zmprov gmi userA@DOMAIN mailboxId: 17 quotaUsed: 2032 $ zmvolume -l Volume id: 1 name: message1 type: primaryMessage path: /opt/zimbra/store compressed: false current: true Volume id: 2 name: index1 type: index path: /opt/zimbra/index compressed: false current: true $ find . -maxdepth 2 -name 17 -print ./0/17 or to find the top directory the user directory is in: $ perl -e 'print 17 >> 12 ; print "\n"' 0 $ cd 0/17/msg/ $ find . -name \*.msg -print ./0/268-751.msg ./0/269-756.msg ./0/306-2119.msg
The Message On The Filesystem - id index_id imap_id
Change directory to the users root directory under your mailstores 'store' path. Default is /opt/zimbra/store , this example user is under /opt/zimbra/store/0/3 . Everything under that is "theirs".
[zimbra@mail39 0]$ pwd /opt/zimbra/store/0/3/msg/0 [zimbra@mail39 0]$ ls 267-9778.msg 280-13700.msg 287-13743.msg 291-13777.msg 295-13763.msg 300-13771.msg 305-13784.msg 268-9780.msg 283-13715.msg 288-13747.msg 292-13758.msg 296-13764.msg 301-13774.msg 307-13791.msg 269-9782.msg 284-13740.msg 289-13752.msg 293-13759.msg 297-13765.msg 302-13775.msg 320-21392.msg 270-9854.msg 285-13741.msg 290-13755.msg 294-13760.msg 299-13769.msg 304-13782.msg 321-21394.msg
In our example above, we used "WHERE id=321". Remember, "id=321" is correlated to the 321 in the filename, 321-21394.msg. Also, you'll notice there's actually 3 matches :
- id: 321
- index_id: 321
- imap_id: 321
mod_metadata And mod_content
From the output above, you see two matches for 21394 - remember, filename is 321-21394.msg :
- mod_metadata: 21394
- mod_metadata might not as there are changes exclusively in the db.
- For example - marking read/unread, tagging, flagging, moving to a new folder, etc..
- mod_metadata might not as there are changes exclusively in the db.
- mod_content: 21394
- mod_content should always be reflected in the filename on the system
- mod_content is updated, for example, when saving a draft, editing a briefcase document, etc.
- Note : "Edit As New" on a message actually creates a new file on the filesystem and uses a different id .
- In ZCS6+ , using the "Remove Attachments" option on an email with attachments would most likely alter the mod_content number as well.
- mod_content is updated, for example, when saving a draft, editing a briefcase document, etc.
- mod_content should always be reflected in the filename on the system
Notice that the main difference between the two is where changes are only exclusively made in the db vs. a change that would necessary alter the "file".
If I then "tag" the message in ZWC for this message, you'll notice the mod_metadata is altered to 23826 but mod_content and the filename on the fs still uses 21394 . Notice as well, "tags: 0" was now changed to "tags: 1" .
[zimbra@mail39 data]$ mysql -e 'SELECT * FROM mboxgroup3.mail_item WHERE id=321\G' *************************** 1. row *************************** mailbox_id: 3 id: 321 type: 5 parent_id: NULL folder_id: 5 index_id: 321 imap_id: 321 date: 1281984004 size: 448 volume_id: 1 blob_digest: 0pe,M6lHRy4KBXbIyEeR7AJzfkA= unread: 0 flags: 1 tags: 1 sender: Adam Cody subject: test w mobile sending name: NULL metadata: d1:f0:1:s45:Adam Cody <ajcody@mail39.zimbra.DOMAIN.com>1:t32:admin@mail38.zimbra.DOMAIN.com1:vi10ee mod_metadata: 23826 change_date: 1282864623 mod_content: 21394 $ cd /opt/zimbra/store/0/3/msg/0 $ ls -la 321* -rw-r----- 1 zimbra zimbra 448 Aug 16 14:40 321-21394.msg
Mysql Query For date And change_date For ALL id's For A Specific User
Another item to point out in this example is the change_date and date fields. Notice that change_date has changed compared to the first time we ran the query. The change_date use to match the date variable number - date: 1281984004 . This field, change_date, is updated when the item has 'changed', not necessarily a mod_metadata type change -- that might be reflected in the time stamp of the file on the filesystem.
- date: 1281984004
- change_date: 1281984004
- vs. after we applies a 'tag' to message
- date: 1281984004
- change_date: 1282864623
To convert that into a 'normal' readable time format, using perl:
# perl -e 'print localtime(1281984004). "\n"' Mon Aug 16 14:40:04 2010 *** Which matches the ls -la time above. ***
vs.
# perl -e 'print localtime(1282864623). "\n"' Thu Aug 26 19:17:03 2010 *** Which is the time I tagged the message in ZWC. ***
To translate Thursday Aug 26 2010 19:17:03 to epoch seconds and back again, for example [Replace '-d' with '-ud' for GMT/UTC time]:
# date +%s -d "08/26/2010 19:17:03" 1282864623
or
# date +%s -ud "Thu Aug 26 19:17:03 EDT 2010" 1282864623
And to prove the conversion goes back again with the date command:
# date -d @1282864623 Thu Aug 26 19:17:03 EDT 2010
Another reference is, the from_unixtime() and unix_timestamp() mysql functions - see :
- http://dev.mysql.com/doc/refman/5.1/en/date-and-time-functions.html
- Please see the note as well under the following, note 100% - so you might want to fudge the time/numbers you use when searching 'dates/times'
Examples within mysql:
$ mysql -e 'SELECT UNIX_TIMESTAMP("2010-08-26 19:17:03")' +---------------------------------------+ | UNIX_TIMESTAMP("2010-08-26 19:17:03") | +---------------------------------------+ | 1282864623 | +---------------------------------------+
And back again:
$ mysql -e 'SELECT FROM_UNIXTIME(1282864623)' +---------------------------+ | FROM_UNIXTIME(1282864623) | +---------------------------+ | 2010-08-26 19:17:03 | +---------------------------+
An example mysql search query using this format:
$ mysql -e 'SELECT * FROM mboxgroup3.mail_item WHERE mailbox_id=3 AND change_date=(SELECT UNIX_TIMESTAMP("2010-08-26 19:17:03"))\G'
This might be useful if you find yourself wanting to query against the change_date for a particular user and see what messages were altered at that time or time range. Example might be, where the user is reporting their "problem" occurring at a certain time or during a time range.
A search against the change_date field and mailbox_id we've been discussing here, shows:
$ mysql -e 'SELECT * FROM mboxgroup3.mail_item WHERE mailbox_id=3 AND change_date=1282864623\G' *************************** 1. row *************************** mailbox_id: 3 id: 64 type: 3 parent_id: NULL folder_id: 8 index_id: NULL imap_id: NULL date: 1282864623 size: 0 volume_id: NULL blob_digest: NULL unread: NULL flags: 0 tags: 0 sender: NULL subject: Test-Tag name: Test-Tag metadata: d1:vi10ee mod_metadata: 23825 change_date: 1282864623 mod_content: 23825 *************************** 2. row *************************** mailbox_id: 3 id: 321 type: 5 parent_id: NULL folder_id: 5 index_id: 321 imap_id: 321 date: 1281984004 size: 448 volume_id: 1 blob_digest: 0pe,M6lHRy4KBXbIyEeR7AJzfkA= unread: 0 flags: 1 tags: 1 sender: Adam Cody subject: test w mobile sending name: NULL metadata: d1:f0:1:s45:Adam Cody <ajcody@mail39.zimbra.DOMAIN.com>1:t32:admin@mail38.zimbra.DOMAIN.com1:vi10ee mod_metadata: 23826 change_date: 1282864623 mod_content: 21394
See how we also identified the 'tag' entry, id: 64 type: 3 , that was made as the id=321 type=5 was altered.
To search by a range of the change_date variable. Notice the syntax of :
- change_date>=1282864600 ; meaning greater than or equal to
- we could also use this instead, change_date>=(SELECT UNIX_TIMESTAMP("2010-08-26 19:17:03")
- change_date<=1282865366 ; meaning less than or equal to
- we could also use this instead, change_date<=(SELECT UNIX_TIMESTAMP("2010-08-26 19:29:26")
$ mysql -e 'SELECT * FROM mboxgroup3.mail_item WHERE mailbox_id=3 AND change_date>=1282864600 AND change_date<=1282865366\G' *************************** 1. row *************************** mailbox_id: 3 id: 64 type: 3 parent_id: NULL folder_id: 8 index_id: NULL imap_id: NULL date: 1282864623 size: 0 volume_id: NULL blob_digest: NULL unread: NULL flags: 0 tags: 0 sender: NULL subject: Test-Tag name: Test-Tag metadata: d1:vi10ee mod_metadata: 23825 change_date: 1282864623 mod_content: 23825 *************************** 2. row *************************** mailbox_id: 3 id: 321 type: 5 parent_id: NULL folder_id: 5 index_id: 321 imap_id: 321 date: 1281984004 size: 448 volume_id: 1 blob_digest: 0pe,M6lHRy4KBXbIyEeR7AJzfkA= unread: 0 flags: 1 tags: 1 sender: Adam Cody subject: test w mobile sending name: NULL metadata: d1:f0:1:s45:Adam Cody <ajcody@mail39.zimbra.DOMAIN.com>1:t32:admin@mail38.zimbra.DOMAIN.com1:vi10ee mod_metadata: 23826 change_date: 1282864623 mod_content: 21394 *************************** 3. row *************************** mailbox_id: 3 id: 323 type: 5 parent_id: NULL folder_id: 6 index_id: 323 imap_id: 323 date: 1282865366 size: 444 volume_id: 1 blob_digest: l5E3l5rbE5XUzHcp+hRNkN1E7os= unread: 0 flags: 65 tags: 0 sender: Adam Cody subject: test w mobile sending - edit as new name: NULL metadata: d1:dd2:do3:3214:idnt36:db166bd3-2405-49d2-aa38-91159a3c9302e1:f4:test1:s45:Adam Cody <ajcody@mail39.zimbra.DOMAIN.com>1:t32:admin@mail38.zimbra.DOMAIN.com1:vi10ee mod_metadata: 23829 change_date: 1282865366 mod_content: 23829
Confirming subject Matches Subject Line In Message
You'll see that the message 321-21394.msg on the filesystem also matches the output we had with our mysql query - compare the Subject line.
[zimbra@mail39 0]$ head 321-21394.msg Date: Mon, 16 Aug 2010 14:40:04 -0400 (EDT) From: Adam Cody <ajcody@mail39.zimbra.DOMAIN.com> To: admin@mail38.zimbra.DOMAIN.com Message-ID: <27950055.531281984004882.JavaMail.root@mail39.zimbra.DOMAIN.com> Subject: test w mobile sending MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [192.168.0.13] X-Mailer: Zimbra 5.0.23_GA_3242.RHEL5 (zclient/5.0.23_GA_3242.RHEL5)
This is a good field to query against also. See Mysql Query For Subject Line In Message Of A Specific User.
Mysql Query For Subject Line In Message Of A Specific User
This is a good field to query against, depending on your circumstances. User's have very little information from their end, ZWC, that correlates into the mysql fields we've been discussing [generally speaking]. They can't see the message id as needed for id= in mboxgroup#.mail_item. Let's say you only have the mailbox_id of the user, therefor the mboxgroup#, and the Subject line of the message in question. You would do something like the following:
$ mysql -e 'SELECT * FROM mboxgroup3.mail_item WHERE mailbox_id=3 AND subject="test w mobile sending"\G' *************************** 1. row *************************** mailbox_id: 3 id: 321 type: 5 parent_id: NULL folder_id: 5 index_id: 321 imap_id: 321 date: 1281984004 size: 448 volume_id: 1 blob_digest: 0pe,M6lHRy4KBXbIyEeR7AJzfkA= unread: 0 flags: 1 tags: 1 sender: Adam Cody subject: test w mobile sending name: NULL metadata: d1:f0:1:s45:Adam Cody <ajcody@mail39.zimbra.DOMAIN.com>1:t32:admin@mail38.zimbra.DOMAIN.com1:vi10ee mod_metadata: 23826 change_date: 1282864623 mod_content: 21394
Mysql Query For ALL Message ID's Of A User
As you'll see in the above query, it has type=5 . Description of mboxgroup[#].mail_item Type shows that type 5 is 'messages'.
[zimbra@mail39 ~]$ mysql -e 'SELECT * FROM mboxgroup3.mail_item WHERE mailbox_id=3 AND type=5\G' *************************** 1. row *************************** mailbox_id: 3 id: 261 type: 5 parent_id: 263 folder_id: 5 index_id: 261 imap_id: 261 date: 1273609779 size: 458 volume_id: 3 blob_digest: baQatWGZSt5THO6tAGImpEOqVQY= unread: 0 flags: 1 tags: 0 sender: Adam Cody subject: test test name: NULL metadata: d1:f9:test test1:s45:Adam Cody <ajcody@mail39.zimbra.DOMAIN.com>1:t45:Adam Cody <ajcody@mail39.zimbra.DOMAIN.com>1:vi10ee mod_metadata: 6216 change_date: 1273609779 mod_content: 6215 *************************** 2. row *************************** mailbox_id: 3 id: 262 type: 5 parent_id: 263 folder_id: 2 index_id: 262 imap_id: 262 date: 1273609779 size: 1599 volume_id: 3 blob_digest: s2BgZHas3+fgRhBYpIHv7uAEuCY= unread: 0 flags: 0 tags: 0 sender: Adam Cody subject: test test name: NULL metadata: d1:f9:test test1:s45:Adam Cody <ajcody@mail39.zimbra.DOMAIN.com>1:vi10ee mod_metadata: 6217 change_date: 1273609784 mod_content: 6216 *************************** 3. row *************************** mailbox_id: 3 id: 264 type: 5 parent_id: NULL folder_id: 2 index_id: 264 imap_id: 264 date: 1273610021 size: 3467 volume_id: 3 blob_digest: n,VKvXmfL,jiA6kC8zE2g3TkWuM= unread: 0 flags: 0 tags: 0 sender: Test User subject: test html name: NULL metadata: d1:f14:BOLD Underline1:s43:Test User <test@mail39.zimbra.DOMAIN.com>1:vi10ee mod_metadata: 6220 change_date: 1273610084 mod_content: 6219 *************************** 4. row ***************************
- cut -- it then continues for all the messages that exist for this account
Mysql Query For ALL Message ID's In A Particular Folder Of A User
As you'll see in the above query, it has type=5 . Description of mboxgroup[#].mail_item Type shows that type 5 is 'messages'.
Get folder_id Number And Description
One way to get the folder_id is to use the zmmailbox against the user to find the id# that we want.
[zimbra@mail39 ~]$ zmmailbox -z -m ajcody@mail39.zimbra.DOMAIN.com gaf Id View Unread Msg Count Path ---------- ---- ---------- ---------- ---------- 1 conv 0 0 / () 16 docu 0 1 /Briefcase () 10 appo 0 1 /Calendar () 14 mess 0 6 /Chats () 7 cont 0 1 /Contacts () 6 mess 0 1 /Drafts () 13 cont 0 2 /Emailed Contacts () 2 mess 10 19 /Inbox () 4 mess 0 0 /Junk () 12 wiki 0 0 /Notebook () 5 mess 0 6 /Sent () 15 task 0 0 /Tasks () 3 conv 0 0 /Trash ()
Or we could use a mysql query to get the output of the descriptions as well. folder_type=1 if for "Folders" and instead of selecting * , we'll use "id,name".
[zimbra@mail39 ~]$ mysql -e 'SELECT id,name FROM mboxgroup3.mail_item WHERE mailbox_id=3 AND type=1\G' *************************** 1. row *************************** id: 1 name: USER_ROOT *************************** 2. row *************************** id: 2 name: Inbox *************************** 3. row *************************** id: 3 name: Trash *************************** 4. row *************************** id: 4 name: Junk *************************** 5. row *************************** id: 5 name: Sent *************************** 6. row *************************** id: 6 name: Drafts *************************** 7. row *************************** id: 7 name: Contacts *************************** 8. row *************************** id: 8 name: Tags *************************** 9. row *************************** id: 9 name: Conversations *************************** 10. row *************************** id: 10 name: Calendar *************************** 11. row *************************** id: 11 name: ROOT *************************** 12. row *************************** id: 12 name: Notebook *************************** 13. row *************************** id: 13 name: Emailed Contacts *************************** 14. row *************************** id: 14 name: Chats *************************** 15. row *************************** id: 15 name: Tasks *************************** 16. row *************************** id: 16 name: Briefcase
The Query For ALL Message ID's In A Particular Folder Of A User
Now we'll setup the query to also use the folder_id. In this example folder_id=5 for the users "Sent" folder.
[zimbra@mail39 ~]$ mysql -e 'SELECT * FROM mboxgroup3.mail_item WHERE mailbox_id=3 AND type=5 AND folder_id=5\G' *************************** 1. row *************************** mailbox_id: 3 id: 261 type: 5 parent_id: 263 folder_id: 5 index_id: 261 imap_id: 261 date: 1273609779 size: 458 volume_id: 3 blob_digest: baQatWGZSt5THO6tAGImpEOqVQY= unread: 0 flags: 1 tags: 0 sender: Adam Cody subject: test test name: NULL metadata: d1:f9:test test1:s45:Adam Cody <ajcody@mail39.zimbra.DOMAIN.com>1:t45:Adam Cody <ajcody@mail39.zimbra.DOMAIN.com>1:vi10ee mod_metadata: 6216 change_date: 1273609779 mod_content: 6215 *************************** 2. row *************************** mailbox_id: 3 id: 281 type: 5 parent_id: 286 folder_id: 5 index_id: 281 imap_id: 281 date: 1277122331 size: 463 volume_id: 3 blob_digest: LifXXOMYCgjFsIj,+bsxi0cboOQ= unread: 0 flags: 1 tags: 0 sender: Adam Cody subject: test mail39 name: NULL metadata: d1:f0:1:s45:Adam Cody <ajcody@mail39.zimbra.DOMAIN.com>1:t33:ajcody@mail59.zimbra.DOMAIN.com1:vi10ee mod_metadata: 13741 change_date: 1278082563 mod_content: 13701 *************************** 3. row *************************** mailbox_id: 3 id: 297 type: 5 parent_id: 286 folder_id: 5 index_id: 297 imap_id: 297 date: 1278084134 size: 14896 volume_id: 1 blob_digest: 66QUCgivALxwGk+p3hnQ4viVZXU= unread: 0 flags: 8195 tags: 0 sender: Adam Cody subject: test mail39 name: NULL metadata: d1:f22:-- Original Message --1:p4:Re: 1:s45:Adam Cody <ajcody@mail39.zimbra.DOMAIN.com>1:t45:Adam Cody <ajcody@mail59.zimbra.DOMAIN.com>1:vi10ee mod_metadata: 13765 change_date: 1278084134 mod_content: 13765 *************************** 4. row ***************************
- cut -- it then continues for all the messages that exist for this account
The Query For ALL Appointments For A Particular User
$ mysql -e 'SELECT * FROM mboxgroup3.appointment WHERE mailbox_id=3\G' *************************** 1. row *************************** mailbox_id: 3 uid: 3252c10b-b889-41ab-bbea-e07eafba0d76 item_id: 325 start_time: 2010-08-27 18:55:00 end_time: 2010-08-27 20:00:00 *************************** 2. row *************************** mailbox_id: 3 uid: 87fa4fc8-70c6-4459-9d3d-e139c0f6e64a item_id: 266 start_time: 2010-06-01 17:40:00 end_time: 2010-06-01 18:00:00
The only relationship I find so far at this point with the above data to 'other' areas within ZCS in regards to mysql or the filesystem is the UID [ 3252c10b-b889-41ab-bbea-e07eafba0d76 ] above will be the UID as well in the ics format that will be in the message that is sent. And that messages will be on the filesystem under the user's store directory path. The message filename was 326-24060.msg - which doesn't match the item_id [325] .
$ pwd /opt/zimbra/store/0/3/msg/0 $ grep 3252c10b-b889-41ab-bbea-e07eafba0d76 * 326-24060.msg:UID:3252c10b-b889-41ab-bbea-e07eafba0d76 $ ls -la 326-24060.msg -rw-r----- 1 zimbra zimbra 4619 Aug 27 18:04 326-24060.msg $ cat 326-24060.msg ## CUT ### BEGIN:VEVENT UID:3252c10b-b889-41ab-bbea-e07eafba0d76 SUMMARY:Test for Mysql Query LOCATION:My office ATTENDEE;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE:mailto:admin@m ail39.zimbra.DOMAIN.com ORGANIZER;CN=Adam Cody:mailto:ajcody@mail39.zimbra.DOMAIN.com DTSTART;TZID="(GMT-06.00) Central Time (US & Canada)":20100827T180000 DTEND;TZID="(GMT-06.00) Central Time (US & Canada)":20100827T190000 ### ###
Mail Items Received Per Day
Actually saw this query from a customer in a case, thought it might be useful for others. Example uses mboxgroup3 and mailbox_id3 , you would adjust that for your own purposes.
$ mysql mysql> connect mboxgroup3; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Connection id: 18414 Current database: mboxgroup3 mysql> select FROM_UNIXTIME( date, '%d %m %Y'), count(*) from mail_item where mailbox_id=3 group by FROM_UNIXTIME( date, '%d %m %Y') order by date; +----------------------------------+----------+ | FROM_UNIXTIME( date, '%d %m %Y') | count(*) | +----------------------------------+----------+ | 13 09 2010 | 17 | | 14 09 2010 | 5 | | 15 09 2010 | 4 | | 16 09 2010 | 4 | | 17 09 2010 | 4 | +----------------------------------+----------+ 5 rows in set (0.00 sec) mysql>
All Blobs Associated With A Particular Volume ID
Assumptions to the command below. There's 100 mboxgroups and the volume we are searching for is volume id 3 [zmvolume -l] . Also, this was against ZCS 8.6 - which uses locator instead of volume_id field. Older versions of ZCS will need to swap locator below to read volume_id .
for i in `seq 100`; do mysql --skip-column-names -e "SELECT CONCAT_WS(',',mailbox_id,id,locator) \ FROM mboxgroup$i.mail_item WHERE locator=1" ; done >> /tmp/locator.txt
Mysql Tables
zimbra database
SHOW zimbra Database Tables
This output shows the tables of the zimbra database. From ZCS 8.6 :
mysql -e 'SHOW tables FROM zimbra' +-------------------+ | Tables_in_zimbra | +-------------------+ | config | | current_sessions | | current_volumes | | deleted_account | | mailbox | | mailbox_metadata | | mobile_devices | | out_of_office | | pending_acl_push | | scheduled_task | | service_status | | table_maintenance | | volume | | volume_blobs | +-------------------+
DESCRIBE zimbra.mailbox Database Table
Description of the zimbra.mailbox table in the zimbra database. From ZCS 8.6 :
mysql -e 'DESCRIBE zimbra.mailbox' +----------------------+---------------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +----------------------+---------------------+------+-----+---------+-------+ | id | int(10) unsigned | NO | PRI | NULL | | | group_id | int(10) unsigned | NO | | NULL | | | account_id | varchar(127) | NO | UNI | NULL | | | index_volume_id | tinyint(3) unsigned | NO | MUL | NULL | | | item_id_checkpoint | int(10) unsigned | NO | | 0 | | | contact_count | int(10) unsigned | YES | | 0 | | | size_checkpoint | bigint(20) unsigned | NO | | 0 | | | change_checkpoint | int(10) unsigned | NO | | 0 | | | tracking_sync | int(10) unsigned | NO | | 0 | | | tracking_imap | tinyint(1) | NO | | 0 | | | last_backup_at | int(10) unsigned | YES | MUL | NULL | | | comment | varchar(255) | YES | | NULL | | | last_soap_access | int(10) unsigned | NO | | 0 | | | new_messages | int(10) unsigned | NO | | 0 | | | idx_deferred_count | int(11) | NO | | 0 | | | highest_indexed | varchar(21) | YES | | NULL | | | version | varchar(16) | YES | | NULL | | | last_purge_at | int(10) unsigned | NO | | 0 | | | itemcache_checkpoint | int(10) unsigned | NO | | 0 | | +----------------------+---------------------+------+-----+---------+-------+
Example Of User Information Of zimbra.mailbox
I use both the user's :
- {ldap} mailboxId equals id {mysql}
- remember, this is for the zimbra.mailbox and not the mboxgroup# db, for mboxgroup# you would use mailbox_id.
- {ldap} zimbraId equals account_id {mysql}
below in this example.
Example is from ZCS 8.6
$ zmprov gmi user1@`zmhostname` mailboxId: 7 quotaUsed: 17000 $ mysql -e 'SELECT * FROM zimbra.mailbox WHERE id=7\G' *************************** 1. row *************************** id: 7 group_id: 7 account_id: 2aa65168-7b18-4b13-b0e5-a91e4cd87bf5 index_volume_id: 2 item_id_checkpoint: 266 contact_count: 0 size_checkpoint: 17000 change_checkpoint: 800 tracking_sync: 0 tracking_imap: 0 last_backup_at: 1423893615 comment: user1@ldap2.zimbra.DOMAIN.com last_soap_access: 0 new_messages: 10 idx_deferred_count: 0 highest_indexed: NULL version: 2.7 last_purge_at: 1424184635 itemcache_checkpoint: 0 $ zmprov ga user1@`zmhostname` zimbraId # name user1@ldap2.zimbra.homeunix.com zimbraId: 2aa65168-7b18-4b13-b0e5-a91e4cd87bf5 $ mysql -e 'SELECT * FROM zimbra.mailbox WHERE account_id="2aa65168-7b18-4b13-b0e5-a91e4cd87bf5"\G' *************************** 1. row *************************** id: 7 group_id: 7 account_id: 2aa65168-7b18-4b13-b0e5-a91e4cd87bf5 index_volume_id: 2 item_id_checkpoint: 266 contact_count: 0 size_checkpoint: 17000 change_checkpoint: 800 tracking_sync: 0 tracking_imap: 0 last_backup_at: 1423893615 comment: user1@ldap2.zimbra.DOMAIN.com last_soap_access: 0 new_messages: 10 idx_deferred_count: 0 highest_indexed: NULL version: 2.7 last_purge_at: 1424184635 itemcache_checkpoint: 0
mboxgroup[#] database
Users are associated with a mboxgroup database. Each mboxgroup database will be comprised of multiple users, the users and their data are unique by way for the mailbox_id of the user.
SHOW mboxgroup[#] Database Tables
This output shows the tables of the mboxgroup# database. From ZCS 8.6 :
$ mysql -e 'SHOW tables FROM mboxgroup7' +----------------------+ | Tables_in_mboxgroup7 | +----------------------+ | appointment | | appointment_dumpster | | data_source_item | | imap_folder | | imap_message | | mail_item | | mail_item_dumpster | | open_conversation | | pop3_message | | revision | | revision_dumpster | | tag | | tagged_item | | tombstone | +----------------------+
DESCRIBE mboxgroup[#].mail_item Database Table
Description of the mail_item table in the mboxgroup# database. From ZCS 8.6 :
$ mysql -e 'DESCRIBE mboxgroup7.mail_item' +--------------+---------------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+---------------------+------+-----+---------+-------+ | mailbox_id | int(10) unsigned | NO | PRI | NULL | | | id | int(10) unsigned | NO | PRI | NULL | | | type | tinyint(4) | NO | | NULL | | | parent_id | int(10) unsigned | YES | | NULL | | | folder_id | int(10) unsigned | YES | | NULL | | | prev_folders | text | YES | | NULL | | | index_id | int(10) unsigned | YES | | NULL | | | imap_id | int(10) unsigned | YES | | NULL | | | date | int(10) unsigned | NO | | NULL | | | size | bigint(20) unsigned | NO | | NULL | | | locator | varchar(1024) | YES | | NULL | | | blob_digest | varchar(44) | YES | | NULL | | | unread | int(10) unsigned | YES | | NULL | | | flags | int(11) | NO | | 0 | | | tags | bigint(20) | NO | | 0 | | | tag_names | text | YES | | NULL | | | sender | varchar(128) | YES | | NULL | | | recipients | varchar(128) | YES | | NULL | | | subject | text | YES | | NULL | | | name | varchar(255) | YES | | NULL | | | metadata | mediumtext | YES | | NULL | | | mod_metadata | int(10) unsigned | NO | | NULL | | | change_date | int(10) unsigned | YES | | NULL | | | mod_content | int(10) unsigned | NO | | NULL | | | uuid | varchar(127) | YES | | NULL | | +--------------+---------------------+------+-----+---------+-------+
Description of mboxgroup[#].mail_item Type
Ref: /source/xref/zimbra-zcs-JUDASPRIEST-860/ZimbraServer/src/java/com/zimbra/cs/mailbox/MailItem.java
75 public abstract class MailItem implements Comparable<MailItem>, ScheduledTaskResult { 76 77 public enum Type { 78 UNKNOWN(-1), 79 /** Item is a standard {@link Folder}. */ 80 FOLDER(1), 81 /** Item is a saved search {@link SearchFolder}. */ 82 SEARCHFOLDER(2), 83 /** Item is a user-created {@link Tag}. */ 84 TAG(3), 85 /** Item is a real, persisted {@link Conversation}. */ 86 CONVERSATION(4), 87 /** Item is a mail {@link Message}. */ 88 MESSAGE(5), 89 /** Item is a {@link Contact}. */ 90 CONTACT(6), 91 /** Item is a {@link InviteMessage} with a {@code text/calendar} MIME part. */ 92 @Deprecated INVITE(7), 93 /** Item is a bare {@link Document}. */ 94 DOCUMENT(8), 95 /** Item is a {@link Note}. */ 96 NOTE(9), 97 /** Item is a memory-only system {@link Flag}. */ 98 FLAG(10), 99 /** Item is a calendar {@link Appointment}. */ 100 APPOINTMENT(11), 101 /** Item is a memory-only, 1-message {@link VirtualConversation}. */ 102 VIRTUAL_CONVERSATION(12), 103 /** Item is a {@link Mountpoint} pointing to a {@link Folder}, possibly in another user's {@link Mailbox}. */ 104 MOUNTPOINT(13), 105 /** Item is a {@link WikiItem} */ 106 @Deprecated WIKI(14), 107 /** Item is a {@link Task} */ 108 TASK(15), 109 /** Item is a {@link Chat} */ 110 CHAT(16), 111 /** Item is a {@link Comment} */ 112 COMMENT(17), 113 /** Item is a {@link Link} pointing to a {@link Document} */ 114 LINK(18); 115
DESCRIBE mboxgroup[#].appointment Database Table
Description of the appointment table in the mboxgroup# database.
$ mysql -e 'DESCRIBE mboxgroup7.appointment' +------------+------------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +------------+------------------+------+-----+---------+-------+ | mailbox_id | int(10) unsigned | NO | PRI | NULL | | | uid | varchar(255) | NO | PRI | NULL | | | item_id | int(10) unsigned | NO | | NULL | | | start_time | datetime | NO | | NULL | | | end_time | datetime | YES | | NULL | | +------------+------------------+------+-----+---------+-------+
Mysql Database Location On Filesystem
mboxgroup# Database Default Example for ZCS5
Using my examples above of the mboxgroup3 .
[zimbra@mail39 data]$ pwd /opt/zimbra/db/data [zimbra@mail39 data]$ ls -F backup/ ib_logfile0 ib_logfile1 ibdata1 mboxgroup1/ mboxgroup2/ mboxgroup3/ mboxgroup4/ mboxgroup5/ mboxgroup6/ mysql/ test/ zimbra/ [zimbra@mail39 data]$ ls mboxgroup3/ appointment.frm data_source_item.ibd imap_folder.ibd mail_item.frm open_conversation.ibd revision.frm tombstone.ibd appointment.ibd db.opt imap_message.frm mail_item.ibd pop3_message.frm revision.ibd data_source_item.frm imap_folder.frm imap_message.ibd open_conversation.frm pop3_message.ibd tombstone.frm
You'll notice there's matching files on the system for the various tables reported with:
$ mysql -e 'SHOW tables FROM mboxgroup3'
zimbra Database Default Example for ZCS5
[zimbra@mail39 data]$ pwd /opt/zimbra/db/data [zimbra@mail39 data]$ ls -F backup/ ib_logfile0 ib_logfile1 ibdata1 mboxgroup1/ mboxgroup2/ mboxgroup3/ mboxgroup4/ mboxgroup5/ mboxgroup6/ mysql/ test/ zimbra/ [zimbra@mail39 data]$ ls zimbra config.frm jiveGroupUser.MYI jiveProperty.MYD jiveSASLAuthorized.frm mucAffiliation.MYD mucRoomProp.frm config.ibd jiveGroupUser.frm jiveProperty.MYI jiveUserProp.MYD mucAffiliation.MYI out_of_office.frm current_volumes.frm jiveID.MYD jiveProperty.frm jiveUserProp.MYI mucAffiliation.frm out_of_office.ibd current_volumes.ibd jiveID.MYI jiveRemoteServerConf.MYD jiveUserProp.frm mucConversationLog.MYD scheduled_task.frm db.opt jiveID.frm jiveRemoteServerConf.MYI jiveVCard.MYD mucConversationLog.MYI scheduled_task.ibd deleted_account.frm jiveOffline.MYD jiveRemoteServerConf.frm jiveVCard.MYI mucConversationLog.frm service_status.MYD deleted_account.ibd jiveOffline.MYI jiveRoster.MYD jiveVCard.frm mucMember.MYD service_status.MYI jiveExtComponentConf.MYD jiveOffline.frm jiveRoster.MYI jiveVersion.MYD mucMember.MYI service_status.frm jiveExtComponentConf.MYI jivePrivacyList.MYD jiveRoster.frm jiveVersion.MYI mucMember.frm table_maintenance.frm jiveExtComponentConf.frm jivePrivacyList.MYI jiveRosterGroups.MYD jiveVersion.frm mucRoom.MYD table_maintenance.ibd jiveGroupProp.MYD jivePrivacyList.frm jiveRosterGroups.MYI mailbox.frm mucRoom.MYI volume.frm jiveGroupProp.MYI jivePrivate.MYD jiveRosterGroups.frm mailbox.ibd mucRoom.frm volume.ibd jiveGroupProp.frm jivePrivate.MYI jiveSASLAuthorized.MYD mailbox_metadata.frm mucRoomProp.MYD jiveGroupUser.MYD jivePrivate.frm jiveSASLAuthorized.MYI mailbox_metadata.ibd mucRoomProp.MYI
You'll notice there's matching files on the system for the various tables reported with :
$ mysql -e 'SHOW tables FROM zimbra'
Getting The Size Of The Mbox Tables In MBs
Here is what you would run.
$ su - zimbra ## Note : I added \'s in the command below so it would format ok in wiki ## $ mysql -e "SELECT table_schema AS 'Mbox table', \ Round( Sum( data_length + index_length ) / 1024 / 1024, 3)\ AS 'Mbox Size (MB)' FROM information_schema.tables \ GROUP BY table_schema ;" +--------------------+----------------+ | Mbox table | Mbox Size (MB) | +--------------------+----------------+ | information_schema | 0.004 | | mboxgroup1 | 0.453 | | mboxgroup2 | 0.453 | | mboxgroup3 | 0.516 | | mboxgroup4 | 0.453 | | mboxgroup5 | 0.453 | | mboxgroup6 | 0.453 | | mysql | 0.520 | | zimbra | 0.319 | +--------------------+----------------+
This was a response from the developers on a question a customer had concerning the sizes of their mboxgroup*ibd files and why they were of varying sizes and if the larger ones could be "shrunk".
- The innodb tablespace can physically grow, but won't shrink. We set innodb_file_per_table to store an idb file per table instead of one large idb file for all tables. If a table has a lot of activity, the idb file will grow. The rows can be deleted but the idb file will still be at the high-water mark. That doesn't mean there isn't a bunch of free space that can be used, just that it will remain large on the filesystem. You can find out the actual size of the space being used by querying the information_schema dictionary view.
- Note - see syntax above that I did. Adam
- SELECT table_schema AS 'Mbox table',
- Round( Sum( data_length + index_length ) / 1024 / 1024, 3) AS Mbox Size (MB)',
- FROM information_schema.tables
- GROUP BY table_schema ;
- Note - see syntax above that I did. Adam
- As you can see, you can include both the data and index lengths, or you could exclude one and run separately to see exactly how large each is. That will get you the actual size which you can compare to the physical size of the idb on disk.If you want to reclaim the disk space by data, I think you are going to have to export the data, drop the table (I don't think a truncate will do it), then import the data again. As far as checking for the percentage of fragmentation, you can't really determine if the difference between physical size and data size is due to many records being deleted or due to fragmentation waste. You can STILL take the same administrative action of rebuilding the tables to help avoid the problem though.
Concerning the dropping of data/tables, see the Mysql_Crash_Recovery on the basic concepts for that. Zimbra Support uses the steps on the Mysql_Crash_Recovery for extreme or DR sitautions, falling back to a full DR recovery process using the zmrestore if the Mysql Crash Recovery steps fail. One should not pursue the Mysql_Crash_Recovery steps unless your willingly to accept the possible need of doing a full DR restore if things don't work out.
To see or understand more about the data in the mboxgroup* mail_item tables, see:
- Ajcody-Mysql-Topics#Mysql_Query_For_ALL_Message_ID.27s_Of_A_User
- Ajcody-Mysql-Topics#Query_All_Users_In_A_mboxgroup.23.23
Also, for future references since 608 is at 5.0.90 - this query might prove useful in regards to the fragmentation question:
Misc Issues Related To Mysql
Error - inconsistent state: unread
You might see a error like the following:
"com.zimbra.common.service.ServiceException: system failure: inconsistent state: unread < 0 for item X" (X is the id of the folder in question.)
Please see the following on the issue and the work around and/or later fix.
- "Add RecalculateMailboxCounts admin command and CLI"
Third Party Tools And References
Recovery Issues
- "InnoDB Recovery Tool" - is a tool to recover corrupted or deleted InnoDB tables
Ajcody Logger Issues
You might want to cross review these other sections as well:
Actual Logger Issues Homepage
Please see Ajcody-Logger-Issues
Logger and ZCS 6.x+
Please see Logger(GnR) , as logger has changed dramatically in 6.x+ as compared to 5.x.
Missing Stats
Are You Using rsyslog, syslog-ng, or syslog
See this forum thread: upgrade to 6.0.2 stats and status no longer working
See the bugs listed below as well.
Confirm All hosts Are Still Present
Follow the instructions at Ajcody-Logger-Issues#What_Tables_And_To_View_Them and confirm all the hosts displayed are still present in your environment. You'll probably also see an error about this in the output from the instructions given at Logger(GnR)#Statistics_show_no_data_available
Logger Not Running
You might want to check /opt/zimbra/log/zmlogswatch.out and see if it mentions a missing file or directory. For example:
tail -f log/zmlogswatch.out Error opening /var/log/zimbra-stats.log: No such file or directory at /opt/zimbra/data/tmp/.swatch_script.21119 line 92
Here's the ownership as that file should be:
ls -la /var/log/zimbra-stats.log -rw-r--r-- 1 zimbra zimbra 2500220 Feb 17 05:37 /var/log/zimbra-stats.log
Logger 6.x Bugs To Review
Some bugs to review:
- "Server statistics fail after logrotate with rsyslogd"
- "Services status: all servers reported as not running/error in web console"
- ""no data available" message after upgrading from 5.0.18 to 6.0.2"
- "48 hour server specific no stats - loading data for chart..."
- "Stats service failing due to folder permissions getting set wrong"
- "zmloggerinit [in zcs5+] for zcs6+"
- "rework logger"
- http://bugzilla.zimbra.com/show_bug.cgi?id=45705
- Dev comments on this bug:
- "Lots of issues have been filed against logger problems. For affected deployments, the current recommendation is to disable the logger until the problems are solved."
- Dev comments on this bug:
- http://bugzilla.zimbra.com/show_bug.cgi?id=45705
sqlite3 and Logger
External Reference: Command Line Shell For SQLite
Login To DB and Quit
How to log into the logger db and to quit:
[zimbra@mail3 data]$ pwd /opt/zimbra/logger/db/data [zimbra@mail3 data]$ sqlite3 logger.sqlitedb SQLite version 3.3.6 Enter ".help" for instructions sqlite> .quit [zimbra@mail3 data]$
What Tables And To View Them
$ pwd /opt/zimbra/logger/db/data $ sqlite3 logger.sqlitedb SQLite version 3.3.6 Enter ".help" for instructions sqlite> .mode list sqlite> .tables config hosts rrd_column_type rrds sqlite> select * from hosts; 1|mail3|mail3.zimbra.REMOVED.com 2|mail3.zimbra.REMOVED.com|mail3.zimbra.REMOVED.com sqlite>.quit $
Delete A Host
This is the command to delete a host, id is the number shown on the left of the line where the hostname is display from "select * from hosts;"
sqlite> DELETE from hosts WHERE id=1;
That would delete the first hostname [id 1] .
Query Of ZCS Service Status History Via zmrrdfetch
Example were done on ZCS 8.0.6 and a single ZCS server setup. First, if you need a converter for unix time see : http://www.epochconverter.com/ . Secondy, note the columns are described below. Some of my examples will not include this first row. :
timestamp,opendkim,zmconfigd,mailbox,proxy,spell,logger,antivirus,archiving,snmp,ldap,memcached,convertd,stats,antispam,mta
The below example checks to see when the archiving service was reported as up. Archiving is the 9th column.
/opt/zimbra/libexec/zmrrdfetch -f zmstatuslog -h zcs806.us.DOMAIN.com -s 1388534400 -e 1405618603 | awk -F, '$9 == 1' [cut] 1405454400,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 1405458000,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 1405461600,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 1405465200,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 1405468800,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 1405472400,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 1405476000,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 1405479600,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 [cut]
The below example checks to see when the archiving service was reported as unavailable 100%. Archiving is the 9th column. [zimbra@zcs806 data]$ /opt/zimbra/libexec/zmrrdfetch -f zmstatuslog -h zcs806.us.DOMAIN.com -s 1388534400 -e 1405618603 | awk -F, '$9 == ""'
1405069200,1,1,1,1,1,1,1,,1,1,1,1,1,1,1 1405072800,1,1,1,1,1,1,1,,1,1,1,1,1,1,1 1405076400,1,0.964354991388889,1,1,1,1,1,,1,1,1,1,1,1,1 1405080000,1,0.946143796944445,1,1,1,1,1,,1,1,1,1,1,1,1 1405083600,1,1,1,1,1,1,1,,1,1,1,1,1,1,1 1405087200,1,1,1,1,1,1,1,,1,1,1,1,1,1,1 1405090800,1,1,1,1,1,1,1,,1,1,1,1,1,1,1 1405094400,1,1,1,1,1,1,1,,1,1,1,1,1,1,1 1405098000,1,1,1,1,1,1,1,,1,1,1,1,1,1,1 1405101600,1,1,1,1,1,1,1,,1,1,1,1,1,1,1 1405105200,1,0.987032061666667,1,1,1,1,1,,1,1,1,1,1,1,1 1405108800,1,1,1,1,1,1,1,,1,1,1,1,1,1,1 1405112400,1,1,1,1,1,1,1,,1,1,1,1,1,1,1 1405116000,1,0.982143184166667,1,1,1,1,1,,1,1,1,1,1,1,1 1405119600,1,1,1,1,1,1,1,,1,1,1,1,1,1,1 1405123200,1,1,1,1,1,1,1,,1,1,1,1,1,1,1 1405126800,1,1,1,1,1,1,1,,1,1,1,1,1,1,1 1405130400,1,0.967078515555556,1,1,1,1,1,,1,1,1,1,1,1,1 1405134000,1,1,1,1,1,1,1,,1,1,1,1,1,1,1 1405137600,1,1,1,1,1,1,1,,1,1,1,1,1,1,1 1405141200,1,0.920520648333333,1,1,1,1,1,,1,1,1,1,1,1,1 1405144800,1,0.9539186725,1,1,1,1,1,,1,1,1,1,1,1,1 1405148400,1,0.981826185277778,1,1,1,1,1,,1,1,1,1,1,1,1 1405152000,1,1,1,1,1,1,1,,1,1,1,1,1,1,1 1405155600,1,1,1,1,1,1,1,,1,1,1,1,1,1,1 1405159200,1,1,1,1,1,1,1,,1,1,1,1,1,1,1 1405162800,1,0.996330343333333,1,1,1,1,1,,1,1,1,1,1,1,1 1405166400,1,1,1,1,1,1,1,,1,1,1,1,1,1,1 1405170000,1,1,1,1,1,1,1,,1,1,1,1,1,1,1 1405173600,1,1,1,1,1,1,1,,1,1,1,1,1,1,1 1405177200,1,1,1,1,1,1,1,,1,1,1,1,1,1,1 [cut]
The below example checks to see when the mailbox service was reported as NOT available 100% or the time but was NOT 100% unavailable either. Mailbox is the 3rd column.
[zimbra@zcs806 data]$ /opt/zimbra/libexec/zmrrdfetch -f zmstatuslog -h zcs806.us.DOMAIN.com -s 1388534400 -e 1405618603 | awk -F, '$3 ~ 0' [cut] 1405076400,1,0.964354991388889,1,1,1,1,1,,1,1,1,1,1,1,1 1405080000,1,0.946143796944445,1,1,1,1,1,,1,1,1,1,1,1,1 1405105200,1,0.987032061666667,1,1,1,1,1,,1,1,1,1,1,1,1 1405116000,1,0.982143184166667,1,1,1,1,1,,1,1,1,1,1,1,1 1405130400,1,0.967078515555556,1,1,1,1,1,,1,1,1,1,1,1,1 1405141200,1,0.920520648333333,1,1,1,1,1,,1,1,1,1,1,1,1 1405144800,1,0.9539186725,1,1,1,1,1,,1,1,1,1,1,1,1 1405148400,1,0.981826185277778,1,1,1,1,1,,1,1,1,1,1,1,1 1405162800,1,0.996330343333333,1,1,1,1,1,,1,1,1,1,1,1,1 [cut]
An example of a zmcontrol restart and what zmrrdfetch will show. Note I adjusted the ending flag to be : -e `date +%s --date="1 minute ago" : as compared to the above examples :
[zimbra@zcs806 data]$ /opt/zimbra/libexec/zmrrdfetch -f zmstatuslog -h zcs806.us.DOMAIN.com -s `date +%s --date="15 minute ago"` -e `date +%s --date="1 minute ago"` timestamp,opendkim,zmconfigd,mailbox,proxy,spell,logger,antivirus,archiving,snmp,ldap,memcached,convertd,stats,antispam,mta 1405621920,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 1405621950,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 1405621980,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 1405622010,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 1405622040,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 1405622070,0.105026416666667,0.1050323,1,1,0.105029991666667,1,0.105016,0.105017466666667,0.105028733333333,1,1,1,0.10503115,0.105014483333333,0.105025008333333 1405622100,0.105026416666667,0.1050323,1,1,0.105029991666667,1,0.105016,0.105017466666667,0.105028733333333,1,1,1,0.10503115,0.105014483333333,0.105025008333333 1405622130,0.105026416666667,0.1050323,1,1,0.105029991666667,1,0.105016,0.105017466666667,0.105028733333333,1,1,1,0.10503115,0.105014483333333,0.105025008333333 1405622160,0.105026416666667,0.1050323,1,1,0.105029991666667,1,0.105016,0.105017466666667,0.105028733333333,1,1,1,0.10503115,0.105014483333333,0.105025008333333 1405622190,,0,,,,,,,,,,,,, 1405622220,,0,,,,,,,,,,,,, 1405622250,,0,,,,,,,,,,,,, 1405622280,,0,,,,,,,,,,,,, 1405622310,,0,,,,,,,,,,,,, 1405622340,,0.893621444444444,,,,,,,,,,,,, 1405622370,,0.893621444444444,,,,,,,,,,,,, 1405622400,,0.893621444444444,,,,,,,,,,,,, 1405622430,,1,,,,,,,,,,,,, 1405622460,,1,,,,,,,,,,,,, 1405622490,,1,,,,,,,,,,,,, 1405622520,,1,,,,,,,,,,,,, 1405622550,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 1405622580,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
Logger Related Bugs
5.0.11 - Fixed in 5.0.12 But See Below Before Upgrading
UPDATE: From release notes: kill all zmlogger processes before upgrading to 5.0.12 from 5.0.10+
- "zmlogger fails to terminate at zmcontrol stop"
- Which generally is causing a whole host of problems. Shows up in 5.0.11 release
- http://bugzilla.zimbra.com/show_bug.cgi?id=33730 dup of below
- Please get the script (attachment) within this bug that updates the control script.
- http://bugzilla.zimbra.com/show_bug.cgi?id=33604
Steps To Resolve
- Edit /opt/zimbra/conf/my.logger.cnf, and modify this line for 20 connections.
- max_connections = 20
- Review the following bug and download the zmlogswatchctl script listed in the attachments section.
- Shutdown logger.
zmloggerctl stop
- find any orphaned logger or swatch processes, and kill them
ps -ef | grep logg
ps -ef | grep watch
- Replace the logswatch control file
mv /opt/zimbra/bin/zmlogswatchctl /opt/zimbra/bin/original.zmlogswatchctl
- Add the new zmlogswatchctl file from the above bug to /opt/zimbra/bin
- as root
chown root:root /opt/zimbra/bin/zmlogswatchctl
- as root
chmod 755 /opt/zimbra/bin/zmlogswatchctl
- as root
- Start up the logger process.
zmloggerctl start
You can be checking the output in the following as well to see if any errors are still being reported:
- /tmp/logprocess.out
- /tmp/gengraphs.out
- /opt/zimbra/logger/db/data/<hostname>.err
Logger Table Check And Repair
Other items to check:
Also, check the logger tables to see if any of the tables need to be repaired
logmysql -e "check table zimbra_logger.amavis" logmysql -e "check table zimbra_logger.amavis_aggregate" logmysql -e "check table zimbra_logger.config" logmysql -e "check table zimbra_logger.disk_aggregate" logmysql -e "check table zimbra_logger.disk_status" logmysql -e "check table zimbra_logger.mta" logmysql -e "check table zimbra_logger.mta_aggregate" logmysql -e "check table zimbra_logger.processing_history" logmysql -e "check table zimbra_logger.raw_logs" logmysql -e "check table zimbra_logger.service_status"
If any of the tables need to be repaired, you replace 'check' with 'repair'. Example:
logmysql -e "repair table zimbra_logger.amavis"
/tmp/logprocess.out Shows MySQL Can't Connect
Please follow these instructions:
Logger Not Working , No Stats, Services Show Not Running In Admin Console
ZCS 6 And Above
On The Logger Host Server
On the Logger monitor host [zmprov gacf |grep zimbraLogHostname] start with the following. Note: run below steps in non-peak hours.
su - zimbra zmcontrol stop ; exit
Run as root user:
/opt/zimbra/libexec/zmfixperms -e -v
On the logger monitor host, you must enable syslog [if your running rsyslog, see below] to log statistics from remote machines. Run the following as root also :
- Edit the /etc/sysconfig/syslog file, add -r to the SYSLOGD_OPTIONS setting, SYSLOGD_options=”-r -m 0”
- Run : /opt/zimbra/libexec/zmsyslogsetup
- Stop the syslog daemon. Type : /etc/init.d/syslogd restart
- Or by: service syslog restart
- Use stop / start if restart isn't available.
If you are using rsyslog instead of syslog, then you will need the following changes in rsyslog config. First, enable the module for remote logging on the logger host. Edit the rsyslog config file "/etc/rsyslog.conf" and enable "imudp" module by uncommenting the same:
$ModLoad imudp $UDPServerRun 514
After the above changes to the following if you are running rsyslog: restart rsyslog service.
- Run : /opt/zimbra/libexec/zmsyslogsetup
- Restart rsyslog with : /etc/init.d/rsyslog restart
- Or by: service rsyslog restart
- Use stop / start if restart isn't available.
Your last steps are to then run:
su - zimbra zmcontrol start /opt/zimbra/libexec/zmloggerinit /opt/zimbra/bin/zmsshkeygen /opt/zimbra/bin/zmupdateauthkeys
On The Other Servers
On all other servers (if you have multiple zimbra servers in your setup). Verify the LogHostname (it should be set to name of Logger monitor Host on all servers) if its different, change the same to Logger monitor Host using below command.
su - zimbra zmprov gacf |grep zimbraLogHostname
If it is set wrong, you can correct it by doing:
zmprov mcf zimbraLogHostname <Logger monitor Hostname>
Then run the following after the zimbraLogHostname variable is confirmed or set correctly:
/opt/zimbra/bin/zmsshkeygen /opt/zimbra/bin/zmupdateauthkeys ; exit /opt/zimbra/libexec/zmsyslogsetup (run as root user) Restart either syslogd or rsyslog, depending on what your server is using service syslog restart or service rsyslog restart su - zimbra zmcontrol restart
Reference Link:
- http://www.zimbra.com/docs/ne/latest/administration_guide/wwhelp/wwhimpl/js/html/wwhelp.htm (Pleaserefer Chapter 15: Monitoring ZCS Servers)
This Section Written For ZCS5
Check the following log files first to gather some information:
- Make sure the /etc/hosts details make the actual hostname and the zmhostname output
- If someone recently changed the hostname variables for this box, logger can be effected if done improperly and will not log the obvious issue.
- /tmp/logprocess.out
- /tmp/gengraphs.out
- /opt/zimbra/logger/db/data/<hostname>.err
- /var/log/zimbra.log (logswatch monitors this file)
- grep'ing for things such as:
egrep 'postfix|amavis|STATUS|DISK|QUEUE' /var/log/zimbra.log
- grep'ing for things such as:
- You might also want to look at the following directory:
- /opt/zimbra/logger/db/work/
- You should see data in here, gif and rrd files.
Then proceed with the following wiki pages, as ordered:
- Logger
- If you find that tables are in need of repair, also consult the following wiki:
- If all else fails, you can clear the old logger database and reinitalize it. Please see:
Large Logger Database Killing Performance Of ZCS
Review the following to see what might be best option for you:
- King0770-Notes#Logger_Issues_-_Improve_Disk_Utilization_for_the_logger_service
- Adjusting the [ zimbraLogRawLifetime: 31d ] & [ zimbraLogSummaryLifetime: 730d ] variables.
- Ajcody-Logger-Issues#Reinitializing_Logger_Database_From_Scratch
Reinitializing Logger Database From Scratch
This will effectively blow away your old logger database, allowing you to start over.
zmloggerctl stop cd /opt/zimbra/logger/db
Remove the old database:
mv db db-old
or data , if db doesn't exist
mv data data-old
You can remove (rm -rf db or data) the directory if your not worried about old directory being perserved
Reinitialize a new logger database:
/opt/zimbra/libexec/zmloggerinit
Logger is probably running after this, but to make sure
zmloggerctl start
You'll need to manually start this one though
zmlogswatchctl start
You'll have to allow some time to pass for there to be enough data for the graphs to be built in the web interface.
Turning Off Logger
To not have logger startup [ use `hostname` or type in hostname] :
zmprov ms `hostname` -zimbraServiceEnabled logger
To manually shutdown logger
zmloggerctl stop
Customization Or Modification Of Swatch Config
The swatch config file is :
/opt/zimbra/conf/swatchrc.in
If you make a customization, you'll need to do the following for it to take effect:
zmsnmpinit zmlogswatchctl stop zmlogswatchctl start zmswatchctl stop zmswatchctl start
Lucene
Actual Lucene Topics Homepage
Please see Ajcody-Lucene-Topics
Other References to Lucene Index
Please see:
Some General Questions On The Lucene Indexing
Index Directory Numbering
We know the directory under the index volume path is like the following:
- /opt/zimbra/index/ "X" / "Y" / index / "Z" /
We believe that "X" is the number which is determined by bitshifting the mailbox_id to the right by 12 bits. That the "Y" is the maibox_id of the user. However, how do you get "Z"?
- Answer: It's always '0'.
When Is User Message Store Directory Created
When is the directory for the message data for an account (/opt/zimbra/store/0/...) created? When a message is stored for the first time? Which would also mean, that it would not be created if message data does not exist, correct?
- Answer: Yes.
When Is User Index Directory And Index Files Created
Concerning the index directory. We know the index directory already exists even when mail data does not exist [see question above]. When will the index directory be created? With the account's first login?
- Answer: The directory is created with user creation. The index files that will exist in the user's directory are created with the first indexing or search event.
Example Walk Through
On my 5.0.24 test box.
Create a test account:
[zimbra@mail37 ~]$ zmprov gmi index-test@`zmhostname` mailboxId: 6 quotaUsed: 0
Notice that the 'store' directory ISN'T automatically created for the user upon user creation :
[zimbra@mail37 ~]$ ls -latr /opt/zimbra/store/0 total 20 drwxr-x--- 3 zimbra zimbra 4096 Sep 13 14:57 1 drwxr-xr-x 4 zimbra zimbra 4096 Sep 13 14:57 .. drwxr-x--- 3 zimbra zimbra 4096 Sep 13 14:57 2 drwxr-x--- 3 zimbra zimbra 4096 Sep 13 15:05 3 drwxr-x--- 5 zimbra zimbra 4096 Sep 13 15:05 .
Notice that the 'index' directory IS automatically created for the user upon user creation but the actual indexing files are not :
[zimbra@mail37 ~]$ ls -latr /opt/zimbra/index/0/6/index/0/ total 8 drwxr-x--- 3 zimbra zimbra 4096 Oct 21 12:11 .. drwxr-x--- 2 zimbra zimbra 4096 Oct 21 12:11 .
Let's see what changes when the user is sent and email BUT they still have not logged in yet:
[zimbra@mail37 ~]$ mail index-test@`zmhostname` tSubject:test from localhost test . Cc:
Checking the relevant user directory for changes. Notice we now have a 6 directory, matching the users mailboxId. We don't see any index files though under their index directory:
[zimbra@mail37 ~]$ ls -latr /opt/zimbra/store/0/ mailboxId: 6 quotaUsed: 1563 total 24 drwxr-x--- 3 zimbra zimbra 4096 Sep 13 14:57 1 drwxr-xr-x 4 zimbra zimbra 4096 Sep 13 14:57 .. drwxr-x--- 3 zimbra zimbra 4096 Sep 13 14:57 2 drwxr-x--- 3 zimbra zimbra 4096 Sep 13 15:05 3 drwxr-x--- 3 zimbra zimbra 4096 Oct 21 12:13 6 drwxr-x--- 6 zimbra zimbra 4096 Oct 21 12:13 . [zimbra@mail37 ~]$ ls -latr /opt/zimbra/index/0/6/index/0/ total 8 drwxr-x--- 3 zimbra zimbra 4096 Oct 21 12:11 .. drwxr-x--- 2 zimbra zimbra 4096 Oct 21 12:11 .
Let's see if logging into the webclient as the user changes anything. Log into the webclient and then check the user directories again. Still no change, no index files created.
[zimbra@mail37 ~]$ ls -latr /opt/zimbra/store/0/ mailboxId: 6 quotaUsed: 1563 total 24 drwxr-x--- 3 zimbra zimbra 4096 Sep 13 14:57 1 drwxr-xr-x 4 zimbra zimbra 4096 Sep 13 14:57 .. drwxr-x--- 3 zimbra zimbra 4096 Sep 13 14:57 2 drwxr-x--- 3 zimbra zimbra 4096 Sep 13 15:05 3 drwxr-x--- 3 zimbra zimbra 4096 Oct 21 12:13 6 drwxr-x--- 6 zimbra zimbra 4096 Oct 21 12:13 . [zimbra@mail37 ~]$ ls -latr /opt/zimbra/index/0/6/index/0/ total 8 drwxr-x--- 3 zimbra zimbra 4096 Oct 21 12:11 .. drwxr-x--- 2 zimbra zimbra 4096 Oct 21 12:11 .
Let's do a manual index of the user account and confirm index files are made.
[zimbra@mail37 ~]$ zmprov rim index-test@`zmhostname` start status: started [zimbra@mail37 ~]$ ls -latr /opt/zimbra/index/0/6/index/0/ total 20 drwxr-x--- 3 zimbra zimbra 4096 Oct 21 12:11 .. -rw-r----- 1 zimbra zimbra 45 Oct 21 12:15 segments_2 -rw-r----- 1 zimbra zimbra 20 Oct 21 12:15 segments.gen -rw-r----- 1 zimbra zimbra 2455 Oct 21 12:15 _0.cfs drwxr-x--- 2 zimbra zimbra 4096 Oct 21 12:15 .
So far, we've confirm user creation doesn't create the store directory until a message or something similar is processed. That the users index directory path will be created with user creation but the index files will not be. That the index files aren't created when the user first logs in but are created with a manual index [zmprov rim user@domain].
Let's confirm if a 'search' creates the index files. First, I'll remove the existing index files that were made. Then log into the webclient as the user and do an email search. Confirming after words that the index files were made from that search - which it does.
[zimbra@mail37 ~]$ cd /opt/zimbra/index/0/6/index/0/ [zimbra@mail37 0]$ ls _0.cfs segments.gen segments_2 [zimbra@mail37 0]$ rm -rf * [zimbra@mail37 0]$ ls
Perform webclient search and check index directory again.
[zimbra@mail37 0]$ ls segments.gen segments_1
Delete Flag
Does lucene create delete flag when index is deleted?
- Answer: Yes.
Delete Flag Operational Details
If so, we believe that it creates only delete flag, and files having an actual index (such as segment file) will be deleted (reuse of disk space) only when segment is merged, or any function for optimization is called, is this correct?
- Answer: Yes.
- Update
- See also this bug/rfe:
- "Index data needs to reclaim disk space after deletes"
- http://bugzilla.zimbra.com/show_bug.cgi?id=54969
- See also this bug/rfe:
Is It The Same For zmmailboxmove With purgeOld
The above behavior is the same when executing zmmailboxmove with purgeOld?
- Answer: No, it physically deletes the entire files.
Cleaning Up Or Shrinking Index For Users
From the ZCS 8 Release Notes:
- "large mail volume DOS's lucene"
- http://bugzilla.zimbra.com/show_bug.cgi?id=76414
- Index data for mailboxes is never deleted so a mailbox index can become very large over time and might be consuming excess disk space because of the large index data. In 8.0, a new zmprov CLI, compactIndexMailbox (cim) was created to compact index data. This command can be used to reclaim disk space when the index volume starts to become full. To compact a mailbox’s index, type:
- zmprov cim <name@domain|id> start
- Note - Depending on the size of the mailbox and the number of deletes this might take awhile. This might require additional free space on the index directory.
- You can run this command concurrently. It is recommended to run this command during off peak hours. You cannot cancel the command once it is started.
- To see the status of index compaction on a mailbox, type:
- zmprov getIndexStats <name@domain|id>
- http://bugzilla.zimbra.com/show_bug.cgi?id=76414
- "Sorting by recipient does not appear to work correctly"
- http://bugzilla.zimbra.com/show_bug.cgi?id=74521
- Customers currently on ZCS 7.x upgrading to latest version of ZCS will require full re-indexing mailboxes for sort-by recipients feature to work properly. Without full re-indexing the mailbox, sorting by "To" field in the "Sent" folder message view will skip all the mess ages from the sorted results added before the upgrade. Note: re-indexing mailbox is an expensive operation and if this feature is NOT so required then, its NOT recommended to do mailbox re-indexing.
- http://bugzilla.zimbra.com/show_bug.cgi?id=74521
Manually Deleting Lucene Index Directories
Please see King0770-Notes#Manually_Delete_Index_Directories
Performance Tuning
Please see Performance_Tuning_Guidelines_for_Large_Deployments#Lucene_Index
Some smaller notes:
- Upgrade to 6.0.8:
- See Bug 47890
- http://bugzilla.zimbra.com/show_bug.cgi?id=47890
- Search in soap SendMsg dosen't scale
- FIXED: 6.0.8 or 6.0.9
- See Bug 47890
- These last 2 will decrease Indexing overhead, but obviously with a loss of functionality
- set zimbraPrefAutoAddAddressEnabled to FALSE
- set zimbraAttachmentsIndexingEnabled to FALSE
Ajcody Notes Archive Discovery
Actual Archive & Discovery Notes Homepage
Ajcody-Notes-Archive-Discovery
References for A&D
Updated A&D Documentation In Admin Manual
Please see:
Zimbra ZSC Version 8.x
The version 7.x manual is more comprehensive. Version 8 Documentation is missing a lot of content available in Version 7 documentation.
Zimbra ZSC Version 7.x
Older Notes About ZAD
Important Note:
I think we need to clean up the documentation that refers to this "download" as no longer being necessary as it's bundled in the zcs-network edition tar ball now. I found an old email to the development team where I inquired about this - I didn't get a response from on it at the time.
If this assumption is true [Like I see with the 5.x setups I've helped with] then I think we need to adjust the documents for 4.5.x stuff to state:
- You'll need to download your version of 4.5.x zcs network edition tarball, untar it, and then run the install script . Which effectively will allow you to now select the "archive" package which will also include the search zimlet [zmbxsearch].
The two main A&D references are :
http://wiki.zimbra.com/index.php?title=ZAD
http://www.zimbra.com/pdf/Zimbra%20Archiving%20and%20Discovery%20Release%20Notes.pdf
But I Just Want Some BCC's To Happen - Not All This A&D Stuff
Please see Ajcody-MTA-Postfix-Topics#Automatic_BCC instead then.
But We Have A Non-Zimbra MTA - zimbraSmtpHostname & zimbraMtaRelayHost Pointed To External Device
Please see this RFE I made:
- "RFE: Ability to handle A&D when another MTA systems is being specified"
- http://bugzilla.zimbra.com/show_bug.cgi?id=45198
- Private note on this rfe explains the 'problem':
- "This feature requires the use of our MTA because amavis needs to process the message."
- Private note on this rfe explains the 'problem':
- http://bugzilla.zimbra.com/show_bug.cgi?id=45198
Notable Bugs Or RFE's
Update Jan 22, 2015 . I filed these while testing against ZCS 8.6 .
- "amavis needs to be a service if archiving is enabled"
- "Archiving To Get It's Own Specific Reporting And Monitoring Section - Also, Confirmation On Upgrades"
- "zmarchivesearch should accept -m "*" or multiple mailbox targets like zmmboxsearch does"
- "admin console mailbox search doesn't autocomplete/suggest for target mailbox account for search results"
- "crossmailbox / mailbox search in admin console limits still broken and documentation descriptions are poor"
- "zmmboxsearch zmarchivesearch -p switch does accept the * [unlimited] variable. Document and qa for each release."
- "zmmboxsearch zmarchivesearch -l does have a hard limit still - it's 1000"
- "zmmboxsearch and zmarchivesearch -p and -l options makes no sense, they need to be rewritten"
Older Bug/RFE's I've noted:
- "Need ability to restrict zmmboxsearch to specific domains"
- "Populate archive accounts from parent account"
- "automatic archive submail box creation by month"
- "Other collaboration data (IM, Docs, Calendars) need to be archived"
- "Archiving/Compliance: Detailed audit trail for all access/update to user archives"
- "Archiving/Compliance: COS-based user selection for archiving"
Archive Stop Working After Upgrade Or Disabling Anti-Spam - Anti-Virus
Prior to JP/8.5 , the archiving feature required the anti-spam or anti-virus service to be enabled. If you disabled both of them, then amavis as a whole was disabled. This is required for archiving to work. The following bug resolves this for JP/8.5 - it makes amavis its own service:
- "It should be possible to remove Amavis as a service"
Also note, my testing on ZCS 8.0.7 did confirm that disabling av/as [therefore, amavis] will cause archiving to not work. This is very confusing since zmcontrol status AND the admin console will show that the service for "archiving" is running - though archiving is not actually working since amavis is disabled.
Update Jan 22, 2015. New bug filed to deal with other upgrade issues we discovered.
- "amavis needs to be a service if archiving is enabled"
First - Short & Sweet How-To On CLI With Single ZCS 8.6 Server
In this example, I'll turn off the antivirus and antispam service in 8.6 and then enable amavis and archiving. Create an archive cos and then archive user. This was a single ZCS 8.6 server with all services installed during the installation - just using default values. This was NOT an upgrade to ZCS 8.6 .
[zimbra@ldap2 ~]$ zmcontrol -v Release 8.6.0_GA_1153.RHEL6_64_20141215151258 RHEL6_64 NETWORK edition. [zimbra@ldap2 ~]$ zmprov gs `zmhostname` | grep Service | egrep 'amavis|antivirus|antispam|archiving' zimbraServiceEnabled: amavis zimbraServiceEnabled: antivirus zimbraServiceEnabled: antispam zimbraServiceEnabled: archiving zimbraServiceInstalled: amavis zimbraServiceInstalled: antivirus zimbraServiceInstalled: antispam zimbraServiceInstalled: archiving [zimbra@ldap2 ~]$ zmprov ms `zmhostname` +zimbraServiceInstalled archiving +zimbraServiceEnabled archiving [zimbra@ldap2 ~]$ zmprov ms `zmhostname` -zimbraServiceEnabled antispam -zimbraServiceEnabled antivirus [zimbra@ldap2 ~]$ zmprov gs `zmhostname` | grep Service | egrep 'amavis|antivirus|antispam|archiving' zimbraServiceEnabled: amavis zimbraServiceEnabled: archiving zimbraServiceInstalled: amavis zimbraServiceInstalled: antivirus zimbraServiceInstalled: antispam zimbraServiceInstalled: archiving !! Note - zmmtactl restart didn't cause the amavisd.conf file to change, hence why I then went with zmcontrol restart. [zimbra@ldap2 ~]$ zmcontrol restart [zimbra@ldap2 ~]$ grep archive_quar /opt/zimbra/conf/amavisd.conf archive_quarantine_method => undef, # Don't run archiving a second time $archive_quarantine_method = 'smtp:[127.0.0.1]:10025'; $archive_quarantine_to = undef; [zimbra@ldap2 ~]$ zmprov cc archive 71f9d7f4-54cc-4bf3-a6e1-94a1fc38a129 [zimbra@ldap2 ~]$ zmarchiveconfig enable user1@`zmhostname` archive-cos archive [zimbra@ldap2 ~]$ vi /tmp/email.txt [zimbra@ldap2 attrs]$ cat /tmp/email.txt To: user1@ldap2.zimbra.DOMAIN.com Subject: Test For Archive - CLI From: admin@ldap2.zimbra.DOMAIN.com test [zimbra@ldap2 ~]$ /opt/zimbra/postfix/sbin/sendmail -t < /tmp/email.txt [zimbra@ldap2 ~]$ grep archive /var/log/zimbra.log Jan 15 16:09:43 ldap2 amavis[29283]: (29283-01) h9nw0WdCl1Tw(h9nw0WdCl1Tw) SEND from <> -> <user1-20150115@ldap2.zimbra.DOMAIN.com.archive>, ENVID=AM.h9nw0WdCl1Tw.20150115T210943Z@ldap2.zimbra.DOMAIN.com BODY=7BIT 250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 2DE506445C Jan 15 16:09:43 ldap2 amavis[29283]: (29283-01) Passed CLEAN {RelayedInbound,Archived}, <zimbra@ldap2.zimbra.DOMAIN.com> -> <user1@ldap2.zimbra.DOMAIN.com>, quarantine: user1-20150115@ldap2.zimbra.DOMAIN.com.archive, Message-ID: <20150115210943.087106438E@ldap2.zimbra.DOMAIN.com>, mail_id: h9nw0WdCl1Tw, Hits: -, size: 351, queued_as: 3049C64466, 138 ms Jan 15 16:09:45 ldap2 postfix/lmtp[3478]: 2DE506445C: to=<user1-20150115@ldap2.zimbra.DOMAIN.com.archive>, relay=ldap2.zimbra.homeunix.com[192.168.1.172]:7025, delay=2.2, delays=0/0.02/0.17/2, dsn=2.1.5, status=sent (250 2.1.5 Delivery OK)
When Was A ZCS Service Enabled Or Disabled
Note - single ZCS server deployment example. Have to double check on multi-server deployment if the command needs to be tweaked.
Check Current Status Of Services Enabled-Disable and Installed-NotInstalled
Example is done on the host in question:
zmprov -l gs `zmhostname` | egrep 'zimbraServiceEnabled|zimbraServiceInstall'
The -l is used in case mailboxd is the service in question, -l makes the call to ldap to check the configuration of the server.
Check Backups And What LDAP Has Stored
Do the following, adjust the backup path if your not using the default path:
for i in `find /opt/zimbra/backup/sessions/ -name ldap.bak.gz -print`; \ do echo $i; zgrep zimbraServiceEnabled $i; done
Will show you something like :
zimbraServiceEnabled: antivirus zimbraServiceEnabled: antispam zimbraServiceEnabled: opendkim zimbraServiceEnabled: logger zimbraServiceEnabled: mailbox zimbraServiceEnabled: mta zimbraServiceEnabled: convertd zimbraServiceEnabled: stats zimbraServiceEnabled: snmp zimbraServiceEnabled: ldap zimbraServiceEnabled: spell zimbraServiceEnabled: archiving
Check The History Of The Service Status In zimbra-stats.log
In /var/log/ you'll see zimbra-stats.log and archives of that file, for example - zimbra-stats.log-20140714 . Here's an example of a zgrep for the archive service:
[zimbra@zcs807 log]$ zgrep archiv zimbra-stats* | tail zimbra-stats.log:Jul 22 08:24:08 zcs807 zimbramon[27645]: 27645:info: 2014-07-22 08:24:01, STATUS: zcs807.us.zimbralab.com: archiving: Running zimbra-stats.log:Jul 22 08:26:09 zcs807 zimbramon[28689]: 28689:info: 2014-07-22 08:26:01, STATUS: zcs807.us.zimbralab.com: archiving: Running zimbra-stats.log-20140714:Jul 15 14:40:19 zcs807 zimbramon[21519]: 21519:info: 2014-07-15 14:40:01, STATUS: zcs807.us.zimbralab.com: archiving: Stopped zimbra-stats.log-20140714:Jul 15 14:42:06 zcs807 zimbramon[24998]: 24998:info: 2014-07-15 14:42:01, STATUS: zcs807.us.zimbralab.com: archiving: Stopped zimbra-stats.log-20140714:Jul 15 14:44:09 zcs807 zimbramon[29250]: 29250:info: 2014-07-15 14:44:01, STATUS: zcs807.us.zimbralab.com: archiving: Running zimbra-stats.log-20140714:Jul 15 14:46:08 zcs807 zimbramon[32188]: 32188:info: 2014-07-15 14:46:01, STATUS: zcs807.us.zimbralab.com: archiving: Running zimbra-stats.log-20140714:Jul 15 14:48:26 zcs807 zimbramon[3174]: 3174:info: 2014-07-15 14:48:07, STATUS: zcs807.us.zimbralab.com: archiving: Running zimbra-stats.log-20140714:Jul 15 14:50:15 zcs807 zimbramon[6466]: 6466:info: 2014-07-15 14:50:01, STATUS: zcs807.us.zimbralab.com: archiving: Running zimbra-stats.log-20140714:Jul 15 14:52:10 zcs807 zimbramon[10260]: 10260:info: 2014-07-15 14:52:02, STATUS: zcs807.us.zimbralab.com: archiving: Running zimbra-stats.log-20140714:Jul 15 14:54:10 zcs807 zimbramon[19004]: 19004:info: 2014-07-15 14:54:01, STATUS: zcs807.us.zimbralab.com: archiving: Running
Check The Logger/RRD Data And Service History Uptime
Let's pull the stats from the logger database going back to Jan 1st of this year. Here's an example on how to do that and what to expect. Please send me a copy of the output also. Notice in the example below the lines that have ",," for column 9 - which is for archiving. This is how I can tell when the service was first enabled and running. Ref for timestamp: http://www.epochconverter.com/
Note: The columns are identified by:
timestamp,opendkim,zmconfigd,mailbox,proxy,spell,logger,antivirus,archiving,snmp,ldap,memcached, convertd,stats,antispam,mta
Please adjust the commands below to suit your needs. I use `zmhostname` below in the command rather than typing out the localhosts hostname. Also, see https://www.gnu.org/software/coreutils/manual/html_node/Examples-of-date.html for examples on the --date string.
For this example, I'm looking to see when the "archiving" service was running 100%. Archiving is the 9th column - '$9 == 1' and the 1 means it was available 100% for the time period. I searched a year's worth of data by using this part below in the examples : "-s `date +%s --date="12 month ago"` -e `date +%s --date="1 minute ago"`"
/opt/zimbra/libexec/zmrrdfetch -f zmstatuslog -h `zmhostname` -s `date +%s \ --date="12 month ago"` -e `date +%s --date="1 minute ago"` | awk -F, '$9 == 1' timestamp,opendkim,zmconfigd,mailbox,proxy,spell,logger,antivirus,archiving,snmp,ldap,memcached, convertd,stats,antispam,mta [cut] 1405454400,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 1405458000,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 1405461600,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 1405465200,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 1405468800,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 1405472400,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 1405476000,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 1405479600,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 [cut]
For this example, I'm looking to see when the "archiving" service was NOT enabled or running at all. Archiving is the 9th column - '$9 == ""' and the absence of a number value means it was UNAVAILABLE 100% for the time period.
/opt/zimbra/libexec/zmrrdfetch -f zmstatuslog -h `zmhostname` -s `date +%s \ --date="12 month ago"` -e `date +%s --date="1 minute ago"` | awk -F, '$9 == ""' | head timestamp,opendkim,zmconfigd,mailbox,proxy,spell,logger,antivirus,archiving,snmp,ldap,memcached, convertd,stats,antispam,mta [cut] 1388538000,,,,,,,,,,,,,,, 1388541600,,,,,,,,,,,,,,, 1388545200,,,,,,,,,,,,,,, 1388548800,,,,,,,,,,,,,,, 1388552400,,,,,,,,,,,,,,, 1388556000,,,,,,,,,,,,,,, 1388559600,,,,,,,,,,,,,,, 1388563200,,,,,,,,,,,,,,, 1388566800,,,,,,,,,,,,,,, 1388570400,,,,,,,,,,,,,,, [cut]
For this example, I'm looking to see when the "zmconfigd" service was NOT 100% available during the time period BUT was greater than 0% of the time. Zmconfigd is the 3rd column - '$3 ~ 0' and by using ~ [NOT] 0 , I can see when it was greater than 0 [100% UNAVAILABLE] but not 1 [100% available].
/opt/zimbra/libexec/zmrrdfetch -f zmstatuslog -h `zmhostname` -s `date +%s \ --date="12 month ago"` -e `date +%s --date="1 minute ago"` | awk -F, '$3 ~ 0' timestamp,opendkim,zmconfigd,mailbox,proxy,spell,logger,antivirus,archiving,snmp,ldap,memcached, convertd,stats,antispam,mta 1400670000,0.926591142901235,0.882009858611111,0.962549540740741,0.962548086419753,0.926603197839506, 1,,,0.964051042283951,1,0.962549647839506,0.962515237345679,0.292491091358025,,0.926590777777778 1400756400,1,0.968276889722222,1,1,1,1,,,1,1,1,1,1,,1 1401678000,0.9323037375,0.937399301388889,1,0.964109132777778,0.932308961111111,1,,,0.932305973888889, 1,0.964109800277778,1,0.932309575,,0.932302677222222 1401692400,0.999979394166667,0.9999799825,1,0.9999993925,0.999979395277778,1,,,0.999979393333333,1, 0.9999993975,1,0.999979395555556,,0.999979393333333 1401966000,1,0.964610447777778,1,1,1,1,,,1,1,1,1,1,,1 1402052400,0.999603703008394,0.999722196864111,0.999615163422937,0.999602498779275,0.999597924831049, 0.999619443248889,,,0.99959995491499,0.999626231525528,0.99961180945684,0.999632334525742, 0.999591523017581,,0.999607517027221 1402765200,1,0.966623691358025,1,1,0.966623606481481,1,1,,0.966623565432099,1,1,1,0.966623647530864, 1,0.966623437345679 1402768800,0.999471892261905,0.999587652037617,0.999491736309524,0.999464918452381,0.999464358928571, 0.999495648214286,0.999508635119048,,0.999464445238095,0.999496001785714,0.999483839285714, 0.999503813095238,0.999461091071429,0.99951724702381,0.999478128571429 1402776000,0.9999910775,0.999972398888889,0.999993851388889,0.999991085555555,0.999972394722222, 0.999991074166667,0.999991234166667,,0.999972394722222,0.999995651111111,0.999991075833333, 0.999991235277778,0.999972394722222,0.999991233333333,0.999972392777778 1403175600,0.999841834656085,0.999841587301587,0.999841991402116,0.999841788690476,0.999841692791005, 0.999842046957672,0.999842196097884,,0.999841743386243,0.999842097222222,0.999841937830688, 0.99984214484127,0.999841638227513,0.999842260251323,0.999841883267196 1405076400,1,0.964354991388889,1,1,1,1,1,,1,1,1,1,1,1,1 1405080000,1,0.946143796944445,1,1,1,1,1,,1,1,1,1,1,1,1 1405105200,1,0.987032061666667,1,1,1,1,1,,1,1,1,1,1,1,1 1405116000,1,0.982143184166667,1,1,1,1,1,,1,1,1,1,1,1,1 1405130400,1,0.967078515555556,1,1,1,1,1,,1,1,1,1,1,1,1 1405141200,1,0.920520648333333,1,1,1,1,1,,1,1,1,1,1,1,1 1405144800,1,0.9539186725,1,1,1,1,1,,1,1,1,1,1,1,1 1405148400,1,0.981826185277778,1,1,1,1,1,,1,1,1,1,1,1,1 1405162800,1,0.996330343333333,1,1,1,1,1,,1,1,1,1,1,1,1
An example of a zmcontrol restart
/opt/zimbra/libexec/zmrrdfetch -f zmstatuslog -h `zmhostname` -s `date +%s \ --date="15 minute ago"` -e `date +%s --date="1 minute ago"` timestamp,opendkim,zmconfigd,mailbox,proxy,spell,logger,antivirus,archiving,snmp,ldap,memcached, convertd,stats,antispam,mta 1405621920,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 1405621950,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 1405621980,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 1405622010,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 1405622040,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 1405622070,0.105026416666667,0.1050323,1,1,0.105029991666667,1,0.105016,0.105017466666667,0.105028733333333, 1,1,1,0.10503115,0.105014483333333,0.105025008333333 1405622100,0.105026416666667,0.1050323,1,1,0.105029991666667,1,0.105016,0.105017466666667,0.105028733333333, 1,1,1,0.10503115,0.105014483333333,0.105025008333333 1405622130,0.105026416666667,0.1050323,1,1,0.105029991666667,1,0.105016,0.105017466666667,0.105028733333333, 1,1,1,0.10503115,0.105014483333333,0.105025008333333 1405622160,0.105026416666667,0.1050323,1,1,0.105029991666667,1,0.105016,0.105017466666667,0.105028733333333, 1,1,1,0.10503115,0.105014483333333,0.105025008333333 1405622190,,0,,,,,,,,,,,,, 1405622220,,0,,,,,,,,,,,,, 1405622250,,0,,,,,,,,,,,,, 1405622280,,0,,,,,,,,,,,,, 1405622310,,0,,,,,,,,,,,,, 1405622340,,0.893621444444444,,,,,,,,,,,,, 1405622370,,0.893621444444444,,,,,,,,,,,,, 1405622400,,0.893621444444444,,,,,,,,,,,,, 1405622430,,1,,,,,,,,,,,,, 1405622460,,1,,,,,,,,,,,,, 1405622490,,1,,,,,,,,,,,,, 1405622520,,1,,,,,,,,,,,,, 1405622550,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 1405622580,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
Check The audit.log File For Service Enable Changes
There is the /opt/zimbra/log/audit.log file that will note a change in a service - for example [ use: zgrep zimbraService /opt/zimbra/log/audit* ]:
audit.log:2014-07-21 12:15:44,745 INFO [qtp1097575009-2632:https://127.0.0.1:7071/service/admin/soap/ModifyServerRequest] [name=zimbra;ip=127.0.0.1;ua=zmprov/8.0.7_GA_6029;] security - cmd=ModifyServer; name=zcs807.us.DOMAIN.com; -zimbraServiceEnabled=archiving; audit.log:2014-07-21 12:16:51,609 INFO [qtp1097575009-2640:https://127.0.0.1:7071/service/admin/soap/ModifyServerRequest] [name=zimbra;ip=127.0.0.1;ua=zmprov/8.0.7_GA_6029;] security - cmd=ModifyServer; name=zcs807.us.DOMAIN.com; +zimbraServiceEnabled=archiving;
Check The zmsetup logs For Changes During Installation/Re-installs
In regards to the /opt/zimbra/log/zmsetup.* log files, here's an example from a test machine :
[Someone rerunning the installer/zmsetup, archiving was enabled - greping' for archiv below]
zmsetup.07152014-150104.txt:Tue Jul 15 14:51:43 2014 enabled packages zimbra-logger zimbra-store zimbra-mta zimbra-convertd zimbra-core zimbra-apache zimbra-archiving zimbra-proxy zimbra-snmp zimbra-ldap zimbra-spell zmsetup.07152014-150104.txt:Tue Jul 15 14:52:02 2014 archiving is enabled zmsetup.07152014-150104.txt:Tue Jul 15 14:52:29 2014 checking isEnabled zimbra-archiving zmsetup.07152014-150104.txt:Tue Jul 15 14:52:29 2014 zimbra-archiving is enabled zmsetup.07152014-150104.txt:Tue Jul 15 14:54:43 2014 Updating cached config attribute for Server zcs807.us.DOMAIN.com: zimbraServiceInstalled=archiving zmsetup.07152014-150104.txt:Tue Jul 15 14:54:43 2014 *** Running as zimbra user: /opt/zimbra/bin/zmprov -r -m -l ms zcs807.us.DOMAIN.com zimbraServiceInstalled 'antivirus' zimbraServiceInstalled 'antispam' zimbraServiceInstalled 'opendkim' zimbraServiceInstalled 'archiving' zimbraServiceInstalled 'logger' zimbraServiceInstalled 'mailbox' zimbraServiceInstalled 'mta' zimbraServiceInstalled 'convertd' zimbraServiceInstalled 'stats' zimbraServiceInstalled 'proxy' zimbraServiceInstalled 'snmp' zimbraServiceInstalled 'ldap' zimbraServiceInstalled 'spell' zmsetup.07152014-150104.txt:Tue Jul 15 14:54:46 2014 Updating cached config attribute for Server zcs807.us.DOMAIN.com: zimbraServiceEnabled=archiving zmsetup.07152014-150104.txt:Tue Jul 15 14:54:46 2014 *** Running as zimbra user: /opt/zimbra/bin/zmprov -r -m -l ms zcs807.us.DOMAIN.com zimbraServiceEnabled 'antivirus' zimbraServiceEnabled 'archiving' zimbraServiceEnabled 'antispam' zimbraServiceEnabled 'opendkim' zimbraServiceEnabled 'logger' zimbraServiceEnabled 'mailbox' zimbraServiceEnabled 'mta' zimbraServiceEnabled 'convertd' zimbraServiceEnabled 'stats' zimbraServiceEnabled 'proxy' zimbraServiceEnabled 'snmp' zimbraServiceEnabled 'ldap' zimbraServiceEnabled 'spell' zmsetup.07152014-150104.txt:Tue Jul 15 14:55:30 2014 *** Running as zimbra user: /opt/zimbra/bin/zmprov -r -m -l mcf +zimbraComponentAvailable archiving zmsetup.07152014-150104.txt: Stopping archiving...Done. zmsetup.07152014-150104.txt: Starting archiving...Done. zmsetup.07152014-150104.txt:Tue Jul 15 15:00:06 2014 com_zimbra_archive... zmsetup.07152014-150104.txt:Tue Jul 15 15:00:06 2014 *** Running as zimbra user: /opt/zimbra/bin/zmzimletctl -l deploy zimlets-network/com_zimbra_archive.zip zmsetup.07152014-150104.txt:[] INFO: Deploying Zimlet com_zimbra_archive in LDAP. zmsetup.07152014-150104.txt:[] INFO: Installing Zimlet com_zimbra_archive on this host. zmsetup.07152014-150104.txt:[] INFO: Upgrading Zimlet com_zimbra_archive to 7.1.0 zmsetup.07152014-150104.txt:[] INFO: Enabling Zimlet com_zimbra_archive
Showing the ldap backup data in regards to the archiving service being disabled and then enabled after the 07/15 setup :
$ zgrep "zimbraServiceEnabled:" full-20140716.070025.112/ldap/* incr-20140714.070016.382/ldap/* :from the day after the re-install: full-20140716.070025.112/ldap/ldap.bak.gz:zimbraServiceEnabled: opendkim full-20140716.070025.112/ldap/ldap.bak.gz:zimbraServiceEnabled: archiving << full-20140716.070025.112/ldap/ldap.bak.gz:zimbraServiceEnabled: logger full-20140716.070025.112/ldap/ldap.bak.gz:zimbraServiceEnabled: mailbox full-20140716.070025.112/ldap/ldap.bak.gz:zimbraServiceEnabled: mta full-20140716.070025.112/ldap/ldap.bak.gz:zimbraServiceEnabled: convertd full-20140716.070025.112/ldap/ldap.bak.gz:zimbraServiceEnabled: stats full-20140716.070025.112/ldap/ldap.bak.gz:zimbraServiceEnabled: proxy full-20140716.070025.112/ldap/ldap.bak.gz:zimbraServiceEnabled: snmp full-20140716.070025.112/ldap/ldap.bak.gz:zimbraServiceEnabled: ldap full-20140716.070025.112/ldap/ldap.bak.gz:zimbraServiceEnabled: spell :from the day before the re-install: incr-20140714.070016.382/ldap/ldap.bak.gz:zimbraServiceEnabled: logger incr-20140714.070016.382/ldap/ldap.bak.gz:zimbraServiceEnabled: mailbox incr-20140714.070016.382/ldap/ldap.bak.gz:zimbraServiceEnabled: mta incr-20140714.070016.382/ldap/ldap.bak.gz:zimbraServiceEnabled: convertd incr-20140714.070016.382/ldap/ldap.bak.gz:zimbraServiceEnabled: stats incr-20140714.070016.382/ldap/ldap.bak.gz:zimbraServiceEnabled: snmp incr-20140714.070016.382/ldap/ldap.bak.gz:zimbraServiceEnabled: ldap incr-20140714.070016.382/ldap/ldap.bak.gz:zimbraServiceEnabled: spell incr-20140714.070016.382/ldap/ldap.bak.gz:zimbraServiceEnabled: proxy incr-20140714.070016.382/ldap/ldap.bak.gz:zimbraServiceEnabled: opendkim
Check The Zimbra Users BASH History File and .zmprov_history
You could check your .bash_history file to see if the command is still referenced there - if it goes back long enough for your purposes. I think the default lines the history file will remember is 500.
[zimbra@zcs807 ~]$ zmprov ms `zmhostname` -zimbraServiceEnabled archiving [zimbra@zcs807 ~]$ zmcontrol status | grep -i arch [zimbra@zcs807 ~]$ zmprov gs `zmhostname` | grep Enabled | grep archiving [zimbra@zcs807 ~]$ zmprov ms `zmhostname` +zimbraServiceEnabled archiving [zimbra@zcs807 ~]$ zmcontrol status | grep -i arch archiving Running [zimbra@zcs807 ~]$ zmprov gs `zmhostname` | grep Enabled | grep archiving zimbraServiceEnabled: archiving [zimbra@zcs807 ~]$ grep archiving .bash_history zmprov ms `zmhostname` +zimbraServiceInstalled archiving +zimbraServiceEnabled archiving zmprov ms `zmhostname` -zimbraServiceEnabled archiving zmprov gs `zmhostname` | grep Enabled | grep archiving zmprov ms `zmhostname` +zimbraServiceEnabled archiving zmprov gs `zmhostname` | grep Enabled | grep archiving grep archiving .bash_history
Note, there is also a /opt/zimbra/.zmprov_history you can also review
Getting Duplicate Emails In The Archive Account
Please see the following:
- Duplicate Email Forwarded to External account when sending to a DL and user
- https://bugzilla.zimbra.com/show_bug.cgi?id=81996
- fixed in 807
- See also an RFE request to remove duplicates from archive accounts:
- Case 00124484: Incoming Call - NEW CASE WITH PREMIUM SUPPORT
- https://bugzilla.zimbra.com/show_bug.cgi?id=88184
- See also an RFE request to remove duplicates from archive accounts:
- fixed in 807
- https://bugzilla.zimbra.com/show_bug.cgi?id=81996
Archiving Issues When Using 3rd Party MTA's
Please see the following:
- RFE: Ability to handle A&D when another MTA systems is being specified
Pre-Deployment Type Questions
How To Add OLD Data Or Retroactively Add Data To A&D Account
You'll want to have A&D all setup and then you can use imapsync to 'sync' the old data from the one account into the A&D account.
Recommendations:
- Confirm imap is enabled on the mailstores
- Setup a sub-folder in A&D account to hold data that existed prior to the A&D account being in use.
- Test the various imapsync flags you'll want to use first before running against all your accounts.
See also:
Can we set the archive settings for only sent mail?
Question: Can we set the archive settings for only sent mail?
Answer: No, archiving is done for any message that passes through the MTA for an account that has archiving enabled. You can setup separate MTA's for in/out bound delivery and enable archiving for only the inbound servers to achieve this goal.
Can we set the archive settings for only form port 25?
Question: Can we set the archive settings for only form port 25?
Answer: No, there are no per port level configurations. You can configure which accounts have archiving enabled and you can enable/disable the service on a per server level.
If Zimbra implemented an archive feature, is something in the following MTA files changes?
Question: If Zimbra implemented an archive feature, is something in the following MTA files changed? [ zmmta.cf , main.cf , main.cf.default , master.cf.in , master.cf ]
Answer: Nothing is specifically changed in the files you listed but settings in zmmta.cf determine how the postfix content filter is configured when archiving is enabled. Archiving functionality is handled by the amavis process on the Zimbra MTA.
Using One Archive Account For Multiple Accounts
You could also include other variable's here if they are needed, ex. - archive-cos COS_NAME
zmarchiveconfig enable user1@domainname archive-address user-archives@domainname.archive zmarchiveconfig enable user2@domainname archive-address user-archives@domainname.archive archive-create FALSE zmarchiveconfig enable user3@domainname archive-address user-archives@domainname.archive archive-create FALSE
You'll see the mapping in the primary email account settings that state what archive account is being used:
zmprov ga user1@example.com zimbraArchiveAccount zimbraArchiveAccount: user-archives@domainname.archive
zmprov ga user2@example.com zimbraArchiveAccount zimbraArchiveAccount: user-archives@domainname.archive
zmprov ga user3@example.com zimbraArchiveAccount zimbraArchiveAccount: user-archives@domainname.archive
External Host For Archive Accounts Setup
I installed 5.0.11 on a test box, selecting the archiving package during the installation.
The main domain was "zimbra.INTERNAL.com" and the servername is mail3.zimbra.INTERNAL.com .
'Note, I did have an external MTA relay server setup for zimbra. I was doing this test from home.
Once it was done, I then created a test account - ajcody@zimbra.INTERNAL.com
I then ran this from the CLI [on a multi-server environment, this is ran on the mta server]:
zmprov ms mail3.zimbra.INTERNAL.com +zimbraServiceInstalled archiving +zimbraServiceEnabled archiving
I confirm that I can send emails to the external account that I will be using with the zimbra admin account:
- Logged into zimbra admin web console , accounts > admin > View Mail
- Composed new message for ajcody@EXTERNAL-DOMAIN.com
I confirmed the external account received the email and did a reply back. Again, confirming now that the admin account gets the email as well from the external account.
Created the archive account to be used:
zmarchiveconfig enable ajcody@zimbra.INTERNAL.com archive-address ajcody@EXTERNAL-DOMAIN.com archive-create false
I then stopped/started zimbra.
zmcontrol stop zmcontrol start
Confirmed that amavis (antispam & antivirus) and archive show up as running process:
[zimbra@mail3 ~]$ zmcontrol status Host mail3.zimbra.INTERNAL.com antispam Running antivirus Running archiving Running ldap Running logger Running mailbox Running mta Running snmp Running spell Running stats Running
Checked my zimbra account to confirm the archive variables:
[zimbra@mail3 ~]$ zmprov ga ajcody@`hostname -d` | egrep -i archive amavisArchiveQuarantineTo: ajcody@EXTERNAL-DOMAIN.com zimbraArchiveAccount: ajcody@EXTERNAL-DOMAIN.com zimbraArchiveAccountDateTemplate: yyyyMMdd zimbraArchiveAccountNameTemplate: ${USER}-${DATE}@${DOMAIN}.archive
Tests I then performed:
- I sent an email from the zimbra admin account to the ajcody@zimbra.INTERNAL.com account. Confirmed that the external account [ajcody@EXTERNAL-DOMAIN.com] received a copy.
- I sent an email to ajcody@zimbra.INTERNAL.com from my ajcody@BUSINESS.com account and confirmed a copy went to ajcody@EXTERNAL-DOMAIN.com .
- I sent an email from ajcody@zimbra.INTERNAL.com to the zimbra admin account, again confirming a copy went to ajcody@EXTERNAL-DOMAIN.com .
What didn't work as expected:
- Messages to/from the two accounts don't create copies. If ajcody@zimbra.INTERNAL.com sends to ajcody@EXTERNAL-DOMAIN.com a "copy" will not be generated and vis-versa .
- The subject lines weren't altered as stated in the official A&D document.
- p3 , "When a message is received to a mailbox with archiving enabled, a copy of the message is sent to the archive mailbox with the text “to be archived” added to the subject line."
I've sent off a request about these two items to the developers to get their feedback on them.
Multi-Server & New Mailstore A&D Setup
I also created a RFE for documentation on this.
http://bugzilla.zimbra.com/show_bug.cgi?id=25135
The following is a very rough draft document I made for multi-server / new mailstore A&D setups.
Ajcody Multi-Server & New Mailstore A&D Setup Homepage
Actual Multi-Server & New Mailstore A&D Setup Homepage
Please see Ajcody-Notes-Archive-Discovery-Mailstore-Setup
Issues That Have Caused Confusion
What Gets Installed Where?
RFE To Clear Up The Confusion
- "Clear Up "archiving" service/package confusion"
zimbra-archive package/rpm - Mailstores
zimbra-archive (the package/rpm you see from the installer) should be installed on all mailstores which you want to use for cross mailbox search. This also sets the zimbraComponentAvailable archiving config attribute which allows the mta(s) to turn on archiving. zimbra-archive is not installed directly on the mta, it's just enabled.
Note, you install zimbra-archive on a mailbox server but the service runs on the mta node.
MTA's - Require Configuration
If you add zimbra-archiving to an existing install you need to :
- Install zimbra-archiving on one or more of your mailbox servers
- Then set zimbraServiceInstalled archiving and zimbraServiceEnabled archiving on all the mta servers
- Restart the mta services
For example:
zmprov ms mta.example.com +zimbraServiceInstalled archiving +zimbraServiceEnabled archiving
On the mta server:
zmmtactl restart
To confirm the /opt/zimbra/conf/amavisd.conf was modified correctly, you should see on the mta:
#$archive_quarantine_method = 'smtp:[127.0.0.1]:10025';
Was uncommented out:
$archive_quarantine_method = 'smtp:[127.0.0.1]:10025';
You'll be able to then notice in the /var/log/zimbra.log file if the redirect to the A&D account is happening [once A&D accounts are setup that is]. Example uses example.com.archive as the archive domain I setup for the A&D accounts :
grep "example.com.archive" /var/log/zimbra.log Dec 11 13:38:52 mta-server amavis[1978]: (01978-19) SEND via SMTP: <> -> <user-20081211@example.com.archive>,ENVID=AM.8ISxcrQG8uAj.20081211T193852Z@mailstore.example.com BODY=7BIT 250 2.6.0 Ok, id=01978-19, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 5ADF8F120C4 Dec 11 13:38:52 mta-server postfix/lmtp[21864]: 5ADF8F120C4: to=<user-20081211@example.com.archive>, relay=archive.example.com[X.X.X.93]:7025, delay=0.07, delays=0/0/0/0.06, dsn=2.1.5, status=sent (250 2.1.5 OK)
Enabling Amavis And Archiving With 8.5+ While Antispam And AntiVirus Are Disabled
With 808 and 8.5 , archiving should be able to run without as/av being enabled.
- It should be possible to remove Amavis as a service
For example, disabling antispam and antivirus but enabling amavis [required] and archiving on a mta server [note - this server had the full mta package already installed on it and had antivirus, antispam, [amavis], and postfix running on it] :
[zimbra@850-mta1 ~]$ zmcontrol status | egrep 'amavis|antispam|antivirus|archiving' amavis Running antispam Running antivirus Running ## NOTE , the below command will disable antispam & anitvirus for this "test". ## [zimbra@850-mta1 ~]$ zmprov ms `zmhostname` +zimbraServiceEnabled archiving \ +zimbraServiceEnabled amavis -zimbraServiceEnabled antispam -zimbraServiceEnabled antivirus [zimbra@850-mta1 ~]$ zmcontrol restart Host 850-ldap1.zimbra.homeunix.com Stopping vmware-ha...Done. [cut] Stopping ldap...Done. Host 850-ldap1.zimbra.homeunix.com Starting ldap...Done. Starting zmconfigd...Done. Starting dnscache...Done. Starting logger...Done. Starting convertd...Done. Starting mailbox...Done. Starting memcached...Done. Starting proxy...Done. Starting amavis...Done. Starting opendkim...Done. Starting archiving...Done. Starting snmp...Done. Starting spell...Done. Starting mta...Done. Starting stats...Done. Starting service webapp...Done. Starting zimbra webapp...Done. Starting zimbraAdmin webapp...Done. Starting zimlet webapp...Done. [zimbra@850-ldap1 ~]$ zmcontrol status | egrep 'amavis|antispam|antivirus|archiving' amavis Running archiving Running
zimbra_xmbxsearch zimlet
For 5.x installs, this zimbra_xmbsearch zimlet will get configured on each mailstore that you install the zimbra-archive package on. The documentation in various places might cause confusion on this matter, because for the 4.x releases it was a separate step.
You should find the zimlet already located at /opt/zimbra/zimlets-network/zimbra_xmbxsearch.zip
After the installation, you should see when you go to the admin web console on the mailstore you install the zimbra-archive package on that the cross-mailbox search zimlet is there. It shows up in two locations:
- Left Pane : Configuration > Admin Extensions > zimbra_xmbxsearch
- Left Pane : Tools > Search Mail
If you wanted this zimlet to also be available on a server that didn't have the zimbra-archiving packaged installed you could then deploy it on that server.
cd /opt/zimbra/zimlets-network/ zmzimletctl deploy zimbra_xmbxsearch.zip ## ls the directory and confirm the full name - you might need this: zmzimletctl deploy com_zimbra_xmbxsearch.zip
The How-To
Reference Documents
http://www.zimbra.com/docs/ne/latest/multi_server_install/multi-server_install.5.1.html
http://www.zimbra.com/docs/ne/latest/administration_guide/Archiving.16.1.html
Assumptions
This install how-to assumes you have an existing LDAP/Mailstore/MTA server(s) for your normal production environment, the Zimbra license and logger are installed on the primary ZCS server(s), and that you are NOT running the proxy module.
Example archive mailstore hostname is : archive.example.com
Example primary ZCS hostname is : mail.example.com
Preliminary Items
Things to do or check before install:
- DNS entry for new mailstore and primary ZCS server(s) can resolve to it.
- DNS configured properly on mailstore server.
- Master Root LDAP Server mail.example.com
- Master Root LDAP Password
- On LDAP server do : su – zimbra ; zmlocalconfig –s | grep ldap_root_password
- Master LDAP port – default is 389
- SMTP Server
Installation Of New Mailstore That Will Have A&D
Install Modules
- Type y to install the zimbra-store, zimbra-archiving and zimbra-spell (optional) packages.
- Do Not Install MTA! These Instructions Do Not Take That Into Account.
- When zimbra-spell is installed the zimbra-apache package is also installed.
- Installing: zimbra-core zimbra-store zimbra-apache zimbra-spell
Modify Configuration
Press Enter to modify the system. The selected packages are installed on the server.
At this point the Main menu displays the default entries for the Zimbra component you are installing.
To expand the menu to see the configuration values type x and press Enter.
To navigate the Main menu, select the menu item to change. You can modify any of the defaults.
- Common Configuration
- LDAP
- Ldap master host: [set this to the FQDN of your LDAP server]
- Ldap port: 389 [set this if your LDAP server isn’t using default]
- Ldap Admin password: [this is your LDAP servers Root LDAP password]
- On LDAP server do : su – zimbra ; zmlocalconfig –s ldap_root_password
- TimeZone: [set this]
- LDAP
- For zimbra-store
- Set the Admin Password
- +License filename: UNSET [if you see this, then something is wrong with your
- LDAP configuration. It should of pulled the license info from the LDAP server.
- Set the SMTP host
- Set the Admin Password
Type r to return to the Main menu, if you aren’t there already.
When the mailbox server is configured, return to the Main menu and type a to apply the configuration changes.
Press Enter to save the configuration data.
When Save Configuration data to a file appears, press Enter.
The next request is where to save the files. To accept the default, press Enter.
To save the files to another directory, enter the directory and then press Enter.
When “The system will be modified - continue?” appears type y and press Enter.
The server is modified.
Installing all the components and configuring the server can take a few minutes.
When Installation complete - press return to exit displays, press Enter.
The installation of the mailbox server is complete.
After Install
Confirm server status
su – zimbra ; zmcontrol status
Populate the ssh keys, on each server in your environment
su - zimbra ; zmupdateauthkeys
The key is updated on /opt/zimbra/.ssh/authorized_keys.
Upgrading A Zimbra Server For An Archive & Discovery Mailstore
Adding Package For A&D
This will retain your current settings for the system. Your server will experience downtime during the upgrade.
untar zcs*.tar that matches your existing system
cd zcs-version-directory ./install choose upgrade select zimbra-archiving
The upgrade of the mailbox server is complete.
After Upgrade
Confirm server status
su – zimbra ; zmcontrol status
Note, zimbra-archiving only runs as a service on a MTA server.
Populate the ssh keys, on each server in your environment
su-zimbra ; zmupdateauthkeys
The key is updated on /opt/zimbra/.ssh/authorized_keys.
Configure Zimbra For Use Of The New Mailstore and A&D
Example A&D mailstore hostname is : archive.example.com
- Go to your primary admin console url. [https://[example.com]:7071/zimbraAdmin]
- Confirm you see the new mailstore under Configuration > Servers
- Under Configuration > Servers > [MTA servername(s)] > Services
- [each MTA server needs this]
- You’ll see a box for Archiving and Discovery
- Check this to enable the MTA server(s) for Archiving and Discovery. If this is grayed out, run the command below (modified for your server) on your one of your mailstores.
- This effectively does:
zmprov ms mta.example.com +zimbraServiceInstalled archiving +zimbraServiceEnabled archiving
- Remember, zmprov uses the variable below. A mta only server can't be set for localhost, change it to point to a mailstore.
[root@mta ~]# zmlocalconfig | grep zmprov
zimbra_zmprov_default_soap_server = localhost
[root@mta ~]# zmlocalconfig -e zimbra_zmprov_default_soap_server=mailstore.example.com
- Check this to enable the MTA server(s) for Archiving and Discovery. If this is grayed out, run the command below (modified for your server) on your one of your mailstores.
- Under Configuration > Servers > [MTA servername(s)] > Services
Configuring Your COS's - Normal COS's and Archiving COS
It's recommended that the archive accounts be created on a dedicated mailstore. You can limit what mailstores are used for new account creation by restricting what mailstores are used under "Server Pool" within a COS. Your normal COS's should excluded your archive mailstores and your archive COS should only have archive mailstores selected.
You'll also want to make sure your archive COS isn't using any 'features' that aren't necessary for archive accounts. No point in consuming certain license features when they aren't needed.
- Go to Configuration > Class of Service > default [or your primary domain] > Server Pool
- You’ll want to make sure it’s limited to the correct server pools
- Your new mailstore for A&D should be unchecked.
- Click on New for a new Class of Server (COS)
- Call it archive or something similar
- Under Server Pool > Limit > have only the new mailstore checked
- Call it archive or something similar
- Make sure your not using licensed features [EWS for example] that aren't necessary for your archive accounts.
- You’ll want to make sure it’s limited to the correct server pools
Setup Initial A&D With First Account - Creation Of The Archive Domain
Revisit To COS - Naming Scheme Of Archive Accounts
When archive accounts are created they use the zimbraArchiveAccountNameTemplate variable from the COS. The default is:
$ zmprov gc default | grep -i archive zimbraArchiveAccountDateTemplate: yyyyMMdd zimbraArchiveAccountNameTemplate: ${USER}-${DATE}@${DOMAIN}.archive
I, personally, don't like the use of the $DATE variable in this. I change my ARCHIVE COS to use the normal username but retain the .archive for the domain.
zmprov mc archive zimbraArchiveAccountNameTemplate '${USER}@${DOMAIN}.archive'
Bug to be aware of:
- "zimbraArchiveAccountNameTemplate is case sensitive - PDF doc is wrong"
The Creation
On server with zmarchiveconfig (most likely mailstore you installed A&D on) and as zimbra (su – zimbra) do the following to setup your first A&D account.
format : zmarchiveconfig –s servername enable user@example.com archive-cos <cos>
example :
zmarchiveconfig –s archive.example.com enable account@example.com archive-cos archive
NOTE
- If the above command doesn't seem to create the archive account/domain. Drop the use of [ -s servername ]. Basically, just run this on the A&D mailstore:
zmarchiveconfig enable account@example.com archive-cos archive
The above command will create the mail domain for the archive accounts using the template defaults, user@example.com to make example.com.archive
On your main ldap server or where ever you usually access the zimbra admin web console, login to the admin web console.
- Confirm the archive domain was setup.
- Configuration > Domains > [domainname].archive > General
- Confirm or adjust the archive domain to use the right COS
- Configuration > Domains > [domainname].archive > General Information
- Change “Default Class of Service” to your COS [archive], if needed for your configuration.
- Configuration > Domains > [domainname].archive > General Information
- Now check for the new archive account you made
- Address > Accounts
- Click on account and hit the edit button
- In the top summary section you'll be able to confirm the COS and Mail Server being used for the account.
- NOTE, if it's showing the account is on the primary mailstore and NOT the A&D mailstore.
- Remove the A&D account
zmprov ra [user]@[domainname].archive
- Add the account back again using the zmarchiveconfig command
zmarchiveconfig enable account@example.com archive-cos archive
- Now confirm, as above, that the account is using the A&D mailstore.
- This might be a bug related to the archive domain being created for the first time.
- Remove the A&D account
- NOTE, if it's showing the account is on the primary mailstore and NOT the A&D mailstore.
Send the primary account a test email and then shortly afterwards do a "View Mail" within the admin console for the archive account. You should see the archive message in the archive account.
Error: unknown document: EnableArchiveRequest
If you get this error when trying to create the archive account "Error: unknown document: EnableArchiveRequest" you most likely needed to install a new license for A&D and have not restart the mailboxd services . Updating the license is not enough, you'll need to restart ZCS on the mailstores also.
See the following bug:
- zimbra-archive extension fails to load when init() fails due to LDAP server outage
RFE's On Archive Accounts
- RFE: COS option to create archiving account automatically by default
Testing Of Archive Mail Flow
Send the primary account a test email and then shortly afterwards do a "View Mail" within the admin console for the archive account. You should see the archive message in the archive account.
You should confirm mail flow copies occur with the following:
- Inbound
- External Account (email) to the primary zimbra account setup for archive.
- A zimbra account that ISN'T the archive account in question to the primary account setup for archive.
- Outbound
- With primary account setup for archive, send an email to an external email address.
- With primary account setup for archive, send an email to another internal zimbra email address.
Archive Account Isn't Getting Email
Let's double check everything was done correctly up above.
Assumption on syntax of account creation:
zmarchiveconfig enable user@example.com archive-cos archive
Let's check what actually was done:
zmprov ga user@example.com | grep -i archive amavisArchiveQuarantineTo: user-20081211@example.com.archive zimbraArchiveAccount: user-20081211@example.com.archive zimbraArchiveAccountNameTemplate: ${USER}-${DATE}@${DOMAIN}.archive
It should reference an account that's like, if you are using the archive templates:
user-[date]@example.com.archive
that account should exist and reference lmtp, rather than smtp, for the transport:
zmprov ga user-20081211@example.com.archive | grep -i trans zimbraMailTransport: lmtp:archive.example.com:7025
Checking Logs For Archive Operations
On the mta-server, you should find a reference to the archive account in /var/log/zimbra.log
grep archive /var/log/zimbra.log Dec 11 13:38:52 mta-server amavis[1978]: (01978-19) SEND via SMTP: <> -> <user-20081211@example.com.archive>,ENVID=AM.8ISxcrQG8uAj.20081211T193852Z@mailstore.example.com BODY=7BIT 250 2.6.0 Ok, id=01978-19, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 5ADF8F120C4 Dec 11 13:38:52 mta-server postfix/lmtp[21864]: 5ADF8F120C4: to=<user-20081211@example.com.archive>, relay=archive.example.com[X.X.X.93]:7025, delay=0.07, delays=0/0/0/0.06, dsn=2.1.5, status=sent (250 2.1.5 OK)
On the archive-server, you should find reference to the delivery in /opt/zimbra/log/mailbox.log
grep archive /opt/zimbra/log/mailbox.log 2008-12-11 14:45:32,923 INFO [LmtpServer-9] [name=user-20081211@example.com.archive;mid=7;] mailop - Adding Message: id=257, Message-ID=<1350363939.41021229024728317.JavaMail.root@EXTERNAL-MTA.DOMAIN.com>, parentId=-1, folderId=2, folderName=Inbox.
Mass Accounts Configuration
- Update 5/28/15, I filed the following RFE:
- "Redesign archive account creation process [autocreate, mass creation/enabling, etc]"
https://bugzilla.zimbra.com/show_bug.cgi?id=99710
- Update, our 6.0 release will have a zmarchiveconfig -f command for batch processing from a file input.
CHECK YOUR AVAILABLE LICENSES BEFORE YOU PROCEED!!
One could put all the accounts in a txt file and then use a for-loop to process the account@example.com variable.
zmprov -l gaa > /tmp/accounts.txt
Remove any accounts you've already done and those not necessary for archiving (ex. admin, ham, spam, etc.)
You can give gaa other options, look at zmprov help account. For example, you could also narrow this down to a dump of accounts in a domain:
zmprov -l gaa [DOMAIN] > /tmp/accounts.txt
Note, the below uses the above setup for A&D - You'll need to modify for your environment.
for i in `cat /tmp/accounts.txt` do zmarchiveconfig –s archive.example.com enable $i archive-cos archive sleep 3 done
You can be tailing /opt/zimbra/log/mailbox.log on the archive server to watch the progress.
Searches After Configuration Is Done
Please see Ajcody-Server-Misc-Topics#Cross_Mailbox_Searches_and_Tracing
Searches Limited To 500 or 1000 Maximum Results
See Ajcody-Server-Misc-Topics#Searches_Limited_To_500_or_1000_Maximum_Results
Special Circumstance Case Scenarios
Need Primary Account Going To Two "Archive" Targets
Currently it isn't possible, I have submitted an RFE for this though:
- "Allow current archive account target to be multi-value for redirection, like zimbraArchiveAccount allows"
I also submitted an RFE to clarify the difference/intention between amavisArchiveQuarantineTo and zimbraArchiveAccount
- "describe amavisArchiveQuarantineTo and strong clarify difference/relationship to zimbraArchiveAccount"
Using Legal Intercept For Secondary Target
Note - since we are setting the legal intercept against the ARCHIVE account and not the production account, we avoid what might be an issue with legal intercept's limitations. See :
Other references:
- http://wiki.zimbra.com/wiki/Legal_Intercept
- Has information on customizing the modifications done to the legal intercepted message.
Primary Archive Account Setup [zimbra@zcs807 ~]$ zmarchiveconfig create archive-test@zcs807.us.DOMAIN.com archive-address archive-test-manual@zcs807.us.DOMAIN.com.archive Setup For The Legal Intercept One We'll Use - You could also use the 'normal' account creation process, but those defaults also will show up in the GAL. [zimbra@zcs807 ~]$ zmarchiveconfig create archive-test2@zcs807.us.DOMAIN.com archive-address archive-test-manual@zcs807.us.DOMAIN.com.archive [zimbra@zcs807 ~]$ zmprov ga archive-test@zcs807.us.DOMAIN.com | egrep -i 'archive|amavisArchiveQuarantine' [cut - notes] confirm that amavisArchiveQuarantineTo & zimbraArchiveAccount are using only ONE of the 'archive' accounts, the primary archive account. Not the one we'll setup later to be the legal intercept target. [zimbra@zcs807 ~]$ zmprov ma archive-test@zcs807.us.DOMAIN.com amavisArchiveQuarantineTo archive-test-manual@zcs807.us.DOMAIN.com.archive [zimbra@zcs807 ~]$ zmprov ma archive-test@zcs807.us.DOMAIN.com zimbraArchiveAccount archive-test-manual@zcs807.us.DOMAIN.com.archive [note] by using just the variable name and not including a + or - in front of it, we'll set it to only be one variable if there were multiple ones set. [zimbra@zcs807 ~]$ zmprov ga archive-test@zcs807.us.DOMAIN.com | egrep -i 'archive|amavisArchiveQuarantine' # name archive-test@zcs807.us.DOMAIN.com amavisArchiveQuarantineTo: archive-test-manual@zcs807.us.DOMAIN.com.archive cn: archive-test mail: archive-test@zcs807.us.DOMAIN.com sn: archive-test uid: archive-test zimbraArchiveAccount: archive-test-manual@zcs807.us.DOMAIN.com.archive zimbraArchiveAccount: archive-test-manual2@zcs807.us.DOMAIN.com.archive zimbraArchiveAccountDateTemplate: yyyyMMdd zimbraArchiveAccountNameTemplate: ${USER}-${DATE}@${DOMAIN}.archive zimbraArchiveEnabled: TRUE zimbraMailDeliveryAddress: archive-test@zcs807.us.DOMAIN.com Set Up your Legal Intercept Variable Now To The Production Account [zimbra@zcs807 ~]$ zmprov ma archive-test@zcs807.us.DOMAIN.com zimbraInterceptAddress archive-test-manual2@zcs807.us.DOMAIN.com.archive For Testing - I'm purging the Inbox on all three test accounts. [zimbra@zcs807 ~]$ zmmailbox -z -m archive-test-manual2@zcs807.us.DOMAIN.com.archive ef /Inbox [zimbra@zcs807 ~]$ zmmailbox -z -m archive-test-manual@zcs807.us.DOMAIN.com.archive ef /Inbox [zimbra@zcs807 ~]$ zmmailbox -z -m archive-test@zcs807.us.DOMAIN.com ef /Inbox Create An Email That We Can Send From The CLI [zimbra@zcs807 ~]$ cat /tmp/email-archive.txt To: archive-test@zcs807.us.DOMAIN.com From: admin@zcs807.us.DOMAIN.com Subject: Test message test [zimbra@zcs807 ~]$ /opt/zimbra/postfix/sbin/sendmail -t < /tmp/email-archive.txt ; tail -f /var/log/zimbra.log | grep archive-test Jul 18 11:10:27 zcs807 amavis[31986]: (31986-02) ESMTP:[127.0.0.1]:10024 /opt/zimbra/data/amavisd/tmp/amavis-20140718T105435-31986-BmFt0jbd: <zimbra@zcs807.us.DOMAIN.com> -> <archive-test@zcs807.us.DOMAIN.com> SIZE=341 Received: from zcs807.us.DOMAIN.com ([127.0.0.1]) by localhost (zcs807.us.DOMAIN.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP for <archive-test@zcs807.us.DOMAIN.com>; Fri, 18 Jul 2014 11:10:27 -0600 (MDT) Jul 18 11:10:27 zcs807 amavis[31986]: (31986-02) Checking: qeJtNl1F-X9w <zimbra@zcs807.us.DOMAIN.com> -> <archive-test@zcs807.us.DOMAIN.com> Jul 18 11:10:27 zcs807 amavis[31986]: (31986-02) SEND from <> -> <archive-test-manual@zcs807.us.DOMAIN.com.archive>,ENVID=AM.qeJtNl1F-X9w.20140718T171027Z@zcs807.us.DOMAIN.com BODY=7BIT 250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as E7236E05EA Jul 18 11:10:27 zcs807 amavis[31986]: (31986-02) FWD from <zimbra@zcs807.us.DOMAIN.com> -> <archive-test@zcs807.us.DOMAIN.com>,BODY=7BIT 250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as EDAEBE0B7C Jul 18 11:10:27 zcs807 amavis[31986]: (31986-02) Passed CLEAN {RelayedInbound,Archived}, <zimbra@zcs807.us.DOMAIN.com> -> <archive-test@zcs807.us.DOMAIN.com>, quarantine: archive-test-manual@zcs807.us.DOMAIN.com.archive, Message-ID: <20140718171027.C8881E0B7B@zcs807.us.DOMAIN.com>, mail_id: qeJtNl1F-X9w, Hits: -, size: 341, queued_as: EDAEBE0B7C, 122 ms Jul 18 11:10:27 zcs807 postfix/smtp[12085]: C8881E0B7B: to=<archive-test@zcs807.us.DOMAIN.com>, relay=127.0.0.1[127.0.0.1]:10024, delay=0.22, delays=0.09/0/0/0.12, dsn=2.0.0, status=sent (250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as EDAEBE0B7C) Jul 18 11:10:28 zcs807 postfix/lmtp[12088]: E7236E05EA: to=<archive-test-manual@zcs807.us.DOMAIN.com.archive>, relay=zcs807.us.DOMAIN.com[10.137.27.38]:7025, delay=0.15, delays=0.02/0.01/0/0.12, dsn=2.1.5, status=sent (250 2.1.5 Delivery OK) Jul 18 11:10:28 zcs807 postfix/lmtp[12090]: EDAEBE0B7C: to=<archive-test@zcs807.us.DOMAIN.com>, relay=zcs807.us.DOMAIN.com[10.137.27.38]:7025, delay=0.21, delays=0.01/0.01/0/0.19, dsn=2.1.5, status=sent (250 2.1.5 Delivery OK) Jul 18 11:10:28 zcs807 postfix/smtp[12093]: 1C4ADE05EA: to=<archive-test-manual2@zcs807.us.DOMAIN.com.archive>, relay=127.0.0.1[127.0.0.1]:10030, delay=0.13, delays=0.05/0.01/0.01/0.06, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 2B480E0B9F) Jul 18 11:10:28 zcs807 postfix/smtp[12096]: 2B480E0B9F: to=<archive-test-manual2@zcs807.us.DOMAIN.com.archive>, relay=127.0.0.1[127.0.0.1]:10025, delay=0.09, delays=0.06/0.01/0/0.02, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 3B6C6E05EA) Jul 18 11:10:28 zcs807 postfix/lmtp[12088]: 3B6C6E05EA: to=<archive-test-manual2@zcs807.us.DOMAIN.com.archive>, relay=zcs807.us.DOMAIN.com[10.137.27.38]:7025, delay=0.15, delays=0.02/0/0/0.13, dsn=2.1.5, status=sent (250 2.1.5 Delivery OK) Let's Confirm That All Three Accounts Show A New Message [zimbra@zcs807 ~]$ zmmailbox -z -m archive-test-manual@zcs807.us.DOMAIN.com.archive gaf | grep Inbox 2 mess 1 1 /Inbox [zimbra@zcs807 ~]$ zmmailbox -z -m archive-test-manual2@zcs807.us.DOMAIN.com.archive gaf | grep Inbox 2 mess 1 1 /Inbox [zimbra@zcs807 ~]$ zmmailbox -z -m archive-test@zcs807.us.DOMAIN.com gaf | grep Inbox 2 mess 1 1 /Inbox [zimbra@zcs807 ~]$ zmmboxsearch -m archive-test@zcs807.us.DOMAIN.com,archive-test-manual@zcs807.us.DOMAIN.com.archive,archive-test-manual2@zcs807.us.DOMAIN.com.archive -q "Test Message" 1) ID: 12d3b4cf-61b8-4ba7-805f-be1d9ab6c0c2:283 Subject: Test message Address: [f] <admin@zcs807.us.DOMAIN.com> Date: 1405703428000 Size: 1041 Conv: -283 Flags: u Folder: 2 Fragment: test 2) ID: 0b283721-857e-4bdc-967a-c5f9e7ec7db2:257 Subject: Test message Address: [f] <admin@zcs807.us.DOMAIN.com> Date: 1405703428000 Size: 1269 Conv: -257 Flags: u Folder: 2 Fragment: test 3) ID: 3cb67670-1ad9-48c0-8f9b-372adaa2fe3f:258 Subject: Intercepted message for archive-test@zcs807.us.DOMAIN.com: Test message Address: [f] Postmaster <postmaster@zcs807.us.DOMAIN.com> Date: 1405703428000 Size: 2769 Conv: -258 Flags: au Folder: 2 Fragment: Intercepted message for archive-test@zcs807.us.DOMAIN.com. Operation=add message, folder=Inbox, folder ID=2.
Using Another Archive Account Against the Primary Archive Account
To Do
Domain Admin Searches For ZCS6
For setting up domain admin rights, please see the following : User:Vladimir-DomainAdminSetup-v6
And you'll see about half way down a screen shot of the "Rights" panel view -- the screen shot shows the ACL right called :
- adminConsoleCrossMailboxSearchRights --- Admin Console Cross Mailbox Search Rights
You'll also need to make sure domain admins have the following ACL right :
- AdminLoginAs
service.UNKNOWN_DOCUMENT Error When Trying Search
The service.UNKNOWN_DOCUMENT error generally means that there is no handler for specified document. Try redeploying the zimlet, restarting the mailboxd service and let us know how it goes. The xmbxsearch zimlet is located in /opt/zimbra/zimlets-network directory.
su - zimbra cd /opt/zimbra/zimlets-network zmzimletctl undeploy zimbra_xmbxsearch.zip zmzimletctl deploy zimbra_xmbxsearch.zip zmmailboxdctl restart
How To Turn Off / Disable ALL A&D Processing From Admin Web Console
From a "server" level, you would uncheck the "Archiving and Discovery" box in the admin web console under:
Configurations > Servers > [MTA's] > right pane > Services tab
This would stop the mta's from splitting the emails for archive.
How To Turn Off / Disable A&D For An Account
Disabling A&D at a user level, requires it to be done via the CLI:
zmarchiveconfig disable user@domainname
What Accounts Are Set For Archive And Checking Against License Usuage
Note - in regards to closing the parent account but leaving the "archive" accounts or multiple archive accounts enabled, the actual archive account doesn't count against the license - either the normal or archive limit. It's consumed based upon the variables and status of the parent email address. See :
- "Request to exclude closed accounts within license count"
- "3rd party Archiving consumes Archiving Account license"
This example below is done on a ZCS 8.6 server. The license usage is based upon the 'parent' account having amavisArchiveQuarantineTo set to something. :
$ zmprov -l gaa admin@ldap2.zimbra.DOMAIN.com spam.gktt7hym36@ldap2.zimbra.DOMAIN.com ham.0zrc899s@ldap2.zimbra.DOMAIN.com virus-quarantine.o944yi8wwl@ldap2.zimbra.DOMAIN.com galsync.cs_37n5wp@ldap2.zimbra.DOMAIN.com user1@ldap2.zimbra.DOMAIN.com user1-archive1@ldap2.zimbra.DOMAIN.com.archive user1-archive2@ldap2.zimbra.DOMAIN.com.archive galsync@test2.zimbra.DOMAIN.com testuser2@test2.zimbra.DOMAIN.com $ zmprov -l sa "amavisArchiveQuarantineTo=*" user1@ldap2.zimbra.DOMAIN.com $ zmprov ga user1@`zmhostname` | egrep -i "zimbraArchiveEnabled|amavisArchiveQuarantineTo|\ zimbraArchiveAccount:|zimbraIsSystemResource" amavisArchiveQuarantineTo: user1-archive1@ldap2.zimbra.DOMAIN.com.archive, user1-archive2@ldap2.zimbra.DOMAIN.com.archive zimbraArchiveAccount: user1-archive1@ldap2.zimbra.DOMAIN.com.archive zimbraArchiveAccount: user1-archive2@ldap2.zimbra.DOMAIN.com.archive zimbraArchiveEnabled: TRUE zimbraIsSystemResource: FALSE $ zmprov -l sa "zimbraIsSystemResource=TRUE" galsync.cs_37n5wp@ldap2.zimbra.DOMAIN.com galsync@test2.zimbra.DOMAIN.com ham.0zrc899s@ldap2.zimbra.DOMAIN.com spam.gktt7hym36@ldap2.zimbra.DOMAIN.com user1-archive1@ldap2.zimbra.DOMAIN.com.archive user1-archive2@ldap2.zimbra.DOMAIN.com.archive virus-quarantine.o944yi8wwl@ldap2.zimbra.DOMAIN.com $ zmprov -l sa "zimbraIsSystemResource=FALSE" user1@ldap2.zimbra.DOMAIN.com $ zmprov fc -a all ; zmsoap -z GetLicenseRequest | egrep 'TotalAccounts|ArchivingAccounts' <attr name="ArchivingAccountsLimit">10000</attr> <attr name="TotalAccounts">3</attr> <attr name="ArchivingAccounts">1</attr>
Need To Find A&D Accounts On Wrong MailStore To Move Them
Searching For Accounts On A Particular Mailstore
To do ldapsearch for the account, and see if there are any related entries in ldap.
su - zimbra source ~/bin/zmshutil zmsetvars ldapsearch -x -H $ldap_master_url -D $zimbra_ldap_userdn -w $zimbra_ldap_password "zimbraMailHost=YOURMAILSTORE.COM" | grep "archive.domain.com"
Renaming Production Domain And Want To Also Rename Archive Domain
- This was tested against ZCS 608p1.
There is no 'automatic' process to rename the associated targeted archive email addresses of the user or the archive domain when changing/renaming the domain.
- Get all the account details from the production users in regards to the archive acct they are using
- zimbraMailDeliveryAddress & amavisArchiveQuarantineTo
- Example via CLI. My initial production domain was primary.DOMAIN.com and the archive domain was primary.DOMAIN.com.archive :
- $ zmprov ga arch-prod@primary.DOMAIN.com zimbraArchiveAccount
- [Returns] zimbraArchiveAccount: arch-prod@primary.DOMAIN.com.archive
- $ zmprov ga arch-prod@primary.DOMAIN.com amavisArchiveQuarantineTo
- [Returns] amavisArchiveQuarantineTo : arch-prod@primary.DOMAIN.archive
- $ zmprov ga arch-prod@primary.DOMAIN.com zimbraArchiveAccount
- Prepare a script that will redo those variables, you'll run it after the domain renames.
- Example via CLI. My initial archive domain was primary.zimbra.homeunix.com.archive :
- $ zmprov ma arch-prod@primary71.DOMAIN.com zimbraArchiveAccount arch-prod@primary71.DOMAIN.com.archive
- $ zmprov ma arch-prod@primary71.DOMAIN.com amavisArchiveQuarantineTo arch-prod@primary71.DOMAIN.archive
- Since this is ran after the domain renames, both references in the command about a domain will use the 'new' domainnames.
- Ref: Bulk_Provisioning
- Example via CLI. My initial archive domain was primary.zimbra.homeunix.com.archive :
- Put domains into maintenance mode during a schedule downtime window
- Example:
- zmprov md primary.DOMAIN.com zimbraDomainStatus maintenance
- zmprov md primary.DOMAIN.com.archive zimbraDomainStatus maintenance
- Ref: Ajcody-Server-Misc-Topics#Domain_And_User_.22Status.22_Changes
- Example:
- Rename/forward the production domain
- Rename/forward the archive domain
- Run the script you had prepared earlier that will reset the user variable to their new archive domain.
- Put domains into active mode:
- Example [notice I have the 'new' domainname here] :
- $ zmprov md primary71.DOMAIN.com zimbraDomainStatus active
- $ zmprov md primary71.DOMAIN.com.archive zimbraDomainStatus active
- Example [notice I have the 'new' domainname here] :
- Send a test message to a prod address and you can see/confirm the transaction in the /var/log/zimbra.log on the mta .
Confirmation of things working [examples]:
### From /var/log/zimbra.log on the MTA ### Nov 17 14:42:27 mail71 amavis[7210]: (07210-01) SEND via SMTP: <> -> <arch-prod@primary71.DOMAIN.com.archive>, ENVID=AM.BYPyb4z8Cl0N.20101117T204227Z@mail71.DOMAIN.com BODY=7BIT 250 2.0.0 Ok, id=07210-01, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as C54F623E0E Nov 17 14:42:27 mail71 postfix/smtpd[7446]: DDA5323E10: client=localhost[127.0.0.1] Nov 17 14:42:27 mail71 postfix/cleanup[7262]: DDA5323E10: message-id=<201011172042.oAHKgKG1024723@mail3.zimbra.homeunix.com> Nov 17 14:42:27 mail71 postfix/qmgr[7530]: DDA5323E10: from=<root@mail3.DOMAIN.com>, size=1644, nrcpt=1 (queue active) Nov 17 14:42:27 mail71 amavis[7210]: (07210-01) FWD via SMTP: <root@mail3.DOMAIN.com> -> <arch-prod@primary71.DOMAIN.com>,BODY=7BIT 250 2.0.0 Ok, id=07210-01, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as DDA5323E10
You can also use the CLI to confirm delivery into the archives & production accounts Inbox. For example:
$ zmmailbox -z -m arch-prod@primary71.DOMAIN.com gaf | grep Inbox 2 mess 3 3 /Inbox $ zmmailbox -z -m arch-prod@primary71.DOMAIN.com.archive gaf | grep Inbox 2 mess 3 3 /Inbox
On Why You Have To Change The User Variables
I confirmed in my test that you do have to reset the user variables rather than just relying on the archive domain having a global forwarding. This is what shows up in /var/log/zimbra.log if you just use the domain rename/forwarding and don't reconfigure the user variable.
Nov 17 14:27:05 mail71 postfix/error[26043]: 8E5DD23E0E: to=<arch-prod@primary.DOMAIN.com.archive>, relay=none, delay=0.04, delays=0/0/0/0.03, dsn=5.0.0, status=bounced (User unknown in virtual alias table)
sending directly to archive account, works:
Nov 17 14:37:28 mail71 amavis[7209]: (07209-01) FWD via SMTP: <root@mail3.DOMAIN.com> -> <arch-prod@primary71.DOMAIN.com.archive>,BODY=7BIT 250 2.0.0 Ok, id=07209-01, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 1FCA523E0E
Ajcody HSM Or Hierarchical Storage Management
Actual HSM Or Hierarchical Storage Management Homepage
Please see Ajcody-HSM-Notes
General Q&A
What's It Look Like - Big Picture
HSM requires a new "volume" on the mailstore(s), i.e. the mailbox server. The Zimbra mailbox server(s) starts with dedicated volumes for the [Default paths listed]:
Reference is Zimbra Mailbox Server
- Message Store [/opt/zimbra/store]
- All email messages reside, including the message body and any file attachments.
- Messages are stored in MIME format.
- Each mailbox has a dedicated directory named after its internal Zimbra mailbox ID.
- Note: Mailbox IDs are unique per server, not system-wide.
- HSM Message Store (optional) [there is no default path, you use whatever partition you make for it]
- Hierarchical Storage Management (HSM) allows you to configure storage volumes for older messages.
- To manage your email storage resources, you can implement a different HSM policy for each message server.
- Messages and attachments are moved from a primary volume to the current secondary volume based on the age of the message.
- The messages are still accessible.
- Single-Copy Message Storage
- Single copy storage allows messages with multiple recipients to be stored only once in the file system.
- On UNIX systems, the mailbox directory for each user contains a hard link to the actual file.
- Note - this is limited by the variable in postfix called, default_destination_recipient_limit , which defaults to 50. See the following for more details - Ajcody-Hardlinks-And-Postfix-default_destination_recipient_limit
- Single copy storage allows messages with multiple recipients to be stored only once in the file system.
- All email messages reside, including the message body and any file attachments.
- Index Store [/opt/zimbra/index]
- Data (MySQL) Store [/opt/zimbra/db]
- Backup [/opt/zimbra/backup]
- Log files [/opt/zimbra/log].
Backup And HSM
Please see Bugs/RFE:
- "RFE: HSM and backup should not run at the same time if initated."
- "Separate Restore Mechanism for Primary/Secondary Volumes" [Helix]
- http://bugzilla.zimbra.com/show_bug.cgi?id=18566
- "add backup/restore for HSM only" [marked as duplicate of above]
- http://bugzilla.zimbra.com/show_bug.cgi?id=18566
HSM Impact With Backups
Zimbra mailbox servers cannot see, read, or write to another Zimbra server. The HSM store data is integrated into the mailbox servers normal backup process - they are NOT separate processes. If you find the HSM disks are having an impact on your backup times and you need to get the backup times shorter please look at the following:
- Improving backup performance issues
- See Ajcody-Notes-ServerPlanning#What_About_Backups.3F_I_Need_A_Plan
- Specifically look at Auto-group Backups
- User submitted bug below. I don't think this is going to be possible but activity on the bug might push the dev's & PM's to pursue other options. I believe the current answer to the issue is above, Auto-group backups.
- Add backup/restore for HSM only
- http://bugzilla.zimbra.com/show_bug.cgi?id=28200
- Which was marked as a duplicate of the following RFE
- "Separate Restore Mechanism for Primary/Secondary Volumes"
- Which was marked as a duplicate of the following RFE
- http://bugzilla.zimbra.com/show_bug.cgi?id=28200
- Add backup/restore for HSM only
HSM Impact To Server Performance
HSM currently iterates all mailboxes without pausing. This can result in a big disk/CPU hit. Please see below for more details.
- HSM should run as a background thread
HSM Running During ZCS Restarts
I created this RFE:
- "RFE: batch HSM option to avoid large redolog operations if interrupted"
Customer reported that mailboxd was manually restart during their hsm operation and it resulted in an extended downtime during the zimbra start processes because the hsm activity was in the redologs and had to be processed that way before mailboxd was fully running.
How Does HSM Determine When To Move Message?
Taken from an internal thread within bugzilla.
Question is:
- What date/timestamp does HSM use to determine when to move the message? The time a message is injected or the date/time in the Date: header?
Answer is:
- HSM uses the date that's stored in the database. This is either the time that the message was added or the value specified in the X-Zimbra-Received header.
HSM And Attachments - Any Options?
If you would like to remove attachments (to another type of storage), please see the RFE/bug below. Comment #9 says:
- Today most customers use our built-in HSM to allow for very large mailboxes but
- use cheaper storage. We don't have a way to strip attachments but are looking
- at some options of providing a way to move large attachments optionally to
- either online or offline storage.
Please see the RFE/bug below. Vote on this if you like it.
- Ability to remove attachment from received message
Aging Policy Options For HSM Data
This is needed otherwise your HSM volume would grow indefinitely.
Please see the following RFE and vote on it.
- Zimbra Message Store & HSM Aging Policies
What Doesn't Get HSM'd?
It's basically messages that get HSM'd. Here's some RFE/bug's I've found for missing items.
Wiki Items
RFE filed, please see:
- Zmhsm does not move wiki blobs
Document & Wiki Version Items
Please see:
- HSM should be able to handle Document revisions.
Briefcase Items
RFE filed, please see:
- RFE: moving briefcase items to secondary storage during HSM process
Does The Mailbox Go Into Maintenance Mode?
This was fixed in 5.0.3. Please see the following bug.
- HSM should not put mailbox in maintenance mode
HSM Logging
I found this RFE, it might prove useful.
- HSM logging improvements
More Than One HSM Volume (Secondary Message Store)
There is a RFE for this, please see the following:
- Add support for more than one current secondary storage volume in HSM
HSM/Secondary Volume for Spam & Junk
I'm not sure exactly what the details and dependencies are with this bug. I added a comment for clarity.
Please see the follow:
- Junk mail storage (secondary/HSM volume for Spam messages)
Consistency Checking Tool For HSM
This is available with zmblobchk, which checks messages in general.
Please see the RFE that it was built for for more information:
- Tool to do consistency checks and repair for missing blob for ID x
What If HSM Volume Becomes Full?
Q: What would happen if the HSM volume filled up while the HSM process was moving messages from the primary store to HSM store? Would it detect the full volume and abort the transaction(s) or would it keep trying? Is it possible that any mail would be lost if the HSM store filled up during an HSM run?
- A. It's transactional, so it will fail gracefully. More specifically, if anything goes wrong during the file copying process, we delete any copied files and abort. The volume id of messages processed before the failure remains the same. HSM runs one mailbox at a time, so the rollback only happens for the last mailbox.
Restoring HSM Volumes - RFE
Please see:
- "Separate Restore Mechanism for Primary/Secondary Volumes"
- http://bugzilla.zimbra.com/show_bug.cgi?id=18566
- Work around in the meantime is in bug notes [private]. Support can help you with this.
- http://bugzilla.zimbra.com/show_bug.cgi?id=18566
A How-To Example - CLI
Introduction
This is a testing example. You should adjust these steps to use a REAL PARTITION and NOT A DIRECTORY in the steps listed below.
Create The HSM Volume
Normally you would have the HSM volume as a separate partition on the server - the lower performance disks when compared to the partition the mailstore is using.
To do a "dry" test, I did the following.
As root.
[root@mail3 ~]mkdir /opt/zimbra/hsm [root@mail3 ~]chown zimbra:zimbra /opt/zimbra/hsm
Then switch to zimbra.
[zimbra@mail3 ~]su - zimbra [zimbra@mail3 ~]zmvolume -a -n hsm-volume -t secondaryMessage -p /opt/zimbra/hsm [zimbra@mail3 ~]zmvolume -l Volume id: 2 name: index1 type: index path: /opt/zimbra/index compressed: false current: true Volume id: 1 name: message1 type: primaryMessage path: /opt/zimbra/store compressed: false current: true Volume id: 3 name: hsm-volume type: secondaryMessage path: /opt/zimbra/hsm compressed: false current: false
Set HSM Volume To Current
Now let's set the hsm-volume to "current". Otherwise, if you try to run zmhsm you'll get a error of "invalid request: None of the secondary message volumes are marked as current."
[zimbra@mail3 ~]$zmvolume -sc -id 3 [zimbra@mail3 ~]$zmvolume -l Volume id: 2 name: index1 type: index path: /opt/zimbra/index compressed: false current: true Volume id: 1 name: message1 type: primaryMessage path: /opt/zimbra/store compressed: false current: true Volume id: 3 name: hsm-volume type: secondaryMessage path: /opt/zimbra/hsm compressed: false current: true
Starting HSM For First Time
This is example is on a test server of mine. I don't have any messages older than 30 days. The default global configuration for HSM Age is 30 days.
[zimbra@mail3 ~]$ ls /opt/zimbra/hsm/ [zimbra@mail3 ~]$ crontab -l | grep -i hsm [zimbra@mail3 ~]$ zmhsm -t HSM process started. [zimbra@mail3 ~]$ zmhsm -u Last HSM Session Stats ---------------------- Start time: Fri Aug 15 15:48:58 EDT 2008 End time: Fri Aug 15 15:48:58 EDT 2008 Not currently running. Moved 0 blobs dated earlier than Wed Jul 16 15:48:58 EDT 2008 to volume 3. Mailboxes processed: 7 out of 7. [zimbra@mail3 ~]$ ls /opt/zimbra/hsm/ [zimbra@mail3 ~]$ zmprov gacf | grep zimbraHsmAge zimbraHsmAge: 30d
So, it worked but didn't have anything to Age.
Adjusting the zimbraHsmPolicy variable - ZCS 6+
Default policy [ZCS 8.6] :
$ zmprov gacf zimbraHsmPolicy zimbraHsmPolicy: message,document:before:-30days
$ zmprov desc -a zimbraHsmPolicy zimbraHsmPolicy The policy that determines which mail items get moved to secondary storage during HSM. Each value specifies a comma-separated list of item types and the search query used to select items to move. See the spec for <SearchRequest> for the complete list of item types and query.txt for the search query spec. type : string value : callback : immutable : false cardinality : multi requiredIn : optionalIn : globalConfig,server flags : serverInherited defaults : message,document:before:-30days min : max : id : 1024 requiresRestart : since : 6.0.0_BETA2 deprecatedSince :
Adjusting the zimbraHsmAge variable - Deprecated since: 6.0.0_BETA2.
$ zmprov desc -a zimbraHsmAge zimbraHsmAge Deprecated since: 6.0.0_BETA2. deprecated in favor for zimbraHsmPolicy. Orig desc: Minimum age of mail items whose filesystem data will be moved to secondary storage.. Must be in valid duration format: {digits}{time-unit}. digits: 0-9, time-unit: [hmsd]|ms. h - hours, m - minutes, s - seconds, d - days, ms - milliseconds. If time unit is not specified, the default is s(seconds). type : duration value : callback : immutable : false cardinality : single requiredIn : optionalIn : globalConfig,server flags : serverInherited defaults : 30d min : 0 max : id : 8 requiresRestart : since : deprecatedSince : 6.0.0_BETA2
zimbraHsmAge must be a valid duration of: nnn[hsmd]
- "zimbraHsmAge variables unclear from output"
[zimbra@mail3 ~]$ zmprov mcf zimbraHsmAge 30d ### example initially had to reproduce issue for my RFE - please don't use 1d ### ### [zimbra@mail3 ~]$ zmprov mcf zimbraHsmAge 1d ### [zimbra@mail3 ~]$ zmprov gacf | grep zimbraHsmAge zimbraHsmAge: 1 [zimbra@mail3 ~]$ zmhsm -t HSM process started. [zimbra@mail3 ~]$ zmhsm -u Last HSM Session Stats ---------------------- Start time: Fri Aug 15 15:56:39 EDT 2008 End time: Fri Aug 15 15:56:40 EDT 2008 Not currently running. Moved 63 blobs dated earlier than Fri Aug 15 15:56:38 EDT 2008 to volume 3. Mailboxes processed: 7 out of 7. [zimbra@mail3 ~]$ ls /opt/zimbra/hsm/ 0 [zimbra@mail3 ~]$ ls /opt/zimbra/hsm/0/ 1 14 15 3
The above commands modified the zimbraHsmAge as a global setting. You can also modify it on a server basis.
[zimbra@mail3 ~]$ zmprov gs mail3.internal.homeunix.com | grep zimbraHsmAge zimbraHsmAge: 1d [zimbra@mail3 ~]$ zmprov gacf | grep zimbraHsmAge zimbraHsmAge: 1d [zimbra@mail3 ~]$ zmprov ms mail3.internal.homeunix.com zimbraHsmAge 30d [zimbra@mail3 ~]$ zmprov gs mail3.internal.homeunix.com | grep zimbraHsmAge zimbraHsmAge: 30d [zimbra@mail3 ~]$ zmprov gacf | grep zimbraHsmAge zimbraHsmAge: 1d
Moving zimbraHsmAge Back To A Lower Number
Continuing after the above steps, let's see what happens if we now run zmhsm.
[zimbra@mail3 ~]$ zmhsm -t HSM process started. [zimbra@mail3 ~]$ zmhsm -u Last HSM Session Stats ---------------------- Start time: Fri Aug 15 16:09:05 EDT 2008 End time: Fri Aug 15 16:09:05 EDT 2008 Not currently running. Moved 0 blobs dated earlier than Fri Aug 15 16:08:35 EDT 2008 to volume 3. Mailboxes processed: 7 out of 7.
Did the message files get moved back?
[zimbra@mail3 ~]$ find /opt/zimbra/store/0/15 -name *.msg -print [zimbra@mail3 ~]$ find /opt/zimbra/hsm/0/15 -name *.msg -print /opt/zimbra/hsm/0/15/msg/0/269-21.msg /opt/zimbra/hsm/0/15/msg/0/263-10.msg /opt/zimbra/hsm/0/15/msg/0/268-14.msg /opt/zimbra/hsm/0/15/msg/0/261-8.msg /opt/zimbra/hsm/0/15/msg/0/266-13.msg /opt/zimbra/hsm/0/15/msg/0/259-4.msg /opt/zimbra/hsm/0/15/msg/0/265-12.msg /opt/zimbra/hsm/0/15/msg/0/257-2.msg
Nope. I'm not sure what else to add here. Don't know if there's actually a way to move them back.
RFE - To Move Msg's Back To Primary Store
Please see:
- "HSM - should move messages back to main store if date to HSM was changed" [marked as WONTFIX]
It is possible to move blobs between volumes using the zmsoap command for ZCS 6 and above. Please read the source documents on this on your ZCS server in /opt/zimbra/docs - the two files are soap-admin.txt and soap.txt . The reference your looking for is MoveBlobsRequest, in the soap-admin.txt guide. To build the proper query, you'll also want to consult the soap.txt guide. As of ZCS 8, it reads as:
<MoveBlobsRequest types="{types}" sourceVolumeIds="{volume-ids}" destVolumeId="{volume-id}" [maxBytes="{n}"]> [<query>{search-query}</query>] </MoveBlobsRequest> <MoveBlobsResponse numBlobsMoved="{n}" numBytesMoved="{n}" totalMailboxes="{n}"/> Moves blobs between volumes. Unlike HsmRequest, this request is synchronous, and reads parameters from the request attributes instead of zimbraHsmPolicy. types: a comma-separated list of item types, or "all" for all types. See the spec for <SearchRequest> for details. volume-ids: a comma-separated list of volume ids. query: if specified, only items that match this query will be moved. maxBytes: Limit for the total number of bytes of data to move. Blob move will abort if this threshold is exceeded.
Some examples are listed below, please check your volume id's with zmvolume -l - you will NOT want to run these examples without updating the volume id's. The HSM policy and MoveBlobsRequest are search based, so you can add a query like this:
zmsoap -z MoveBlobsRequest @types=all @sourceVolumeIds=3 @destVolumeId=4 query=is:anywhere
An "is:anywhere" query will look in all folders, including /Trash for messages to move.
A query just in junk would be like the following below. This will move every blob for every mailbox located in volume 1 to volume 3 that is under the junk folder.
zmsoap -z MoveBlobsRequest @sourceVolumeIds=1 @destVolumeId=3 query=in:junk
Or you can specify a date:
zmsoap -z MoveBlobsRequest @sourceVolumeIds=1 @destVolumeId=3 query=before:1/1/2012
MoveBlobsRequest will move the blob and update mysql mail_item table accordingly.
Old Note Below
The desperate, could review the following. This would be an unsupported procedure. I'll try to engage developers on this and get some feedback though:
Now What? Place In Cron
You will have to manually put in the zimbra crontab file a line to run the zmhsm command.
Something like the following at the end of the crontab [ su - zimbra ; crontab -e ] :
# HSM 0 6 * * * /opt/zimbra/bin/zmhsm -t
This will run every morning at 0600. The question of each administrator is what is the right time to run this. I'm initial thought is to try and kick it off after your backups. One might want to have two entries for the zmhsm command, one after your daily incremental and another time frame for you full backup days.
Checking For The HSM Zimlet
In the Admin web console, you'll see the reference to the HSM zimlets in this path:
Configuration > Admin Extensions > com_zimbra_hsm
Deploying The HSM Zimlet
To deploy the zimlet for HSM
su - zimbra zmzimletctl deploy /opt/zimbra/zimlets-network/com_zimbra_hsm.zip
Log into the admin web console - fresh session.
What Can I Do For HSM In The Admin Web Console?
Confirm the HSM zimlet is installed first.
Setup The HSM Volume
Configuration > Servers > [Server For HSM Volume]
Then select the Volumes tab on the right-hand section.
Clicking on "Add", you'll be given a drop down chooser for "Volume Type". For HSM, you would select "Secondary Message".
The "Assign Current Volumes" section will show you what volume is in use for what function.
You will also see "HSM" on this page to set the zimbraHsmAge variable for the SERVER - rather than globally.
Adjust The zimbraHsmAge Date
Per Server
Configuration > Servers > [Server For HSM Volume]
Then select the Volumes tab on the right-hand section.
You will see "HSM" on this page to set the zimbraHsmAge variable for the SERVER - rather than globally.
Global
Configuration > Global Settings
The select the HSM tab on the right-hand section.
This will set the global (default) message age for HSM.
Starting & Stopping HSM (zmhsm)
Configurations > Servers > [Server With HSM Volume]
The on the right-hand section, above the details area for the server you'll see a button/tab HSM. Click on this and you'll be given the option to Start HSM Session. It will output progress details.
References From Official Documentation
zmhsm - command for HSM
Please see:
CLI_zmhsm_Network_Edition_only
zmvolume - command for volumes
Please see:
http://wiki.zimbra.com/index.php?title=CLI_zmvolume
Global HSM Session Setting
Reference: http://www.zimbra.com/docs/ne/latest/administration_guide/Managing_ZCS.10.1.html#1111022
Global Settings HSM (Hierarchical Storage Management) sets the default message age threshold to 30 days. The HSM global setting is the default unless you change the schedule in the Server configuration. See “Scheduling HSM Sessions” .
Scheduling HSM Sessions
Reference: http://www.zimbra.com/docs/ne/latest/administration_guide/Managing_ZCS.10.1.html#1111045
HSM can be configured for secondary storage volumes for older messages. Messages and attachments are moved from a primary volume to the current secondary volume based on the age of the message. Users are not aware of any change and do not see any noticeable difference when opening an older message that has been moved.
To manage your email storage resources, you can implement a different HSM policy for each mailbox server. The message age threshold for HSM is set globally on the HSM tab or for individual servers from the Server Volume tab. The default is 30 days. The thresholds configured on individual servers override the threshold configured as the global setting. Sessions to move messages to the secondary volume are scheduled in your cron table. From the administration console, when you select a server, you can manually start a session, monitor sessions, and abort sessions that are in progress from the Volumes tab.
When you abort a session and then restart the process, the HSM session looks for entries in the primary store that meet the HSM age criteria. Any entries that were moved in the previous run would be excluded, as they would no longer exist in the primary store.
- This article is NOT official Zimbra documentation. It is a user contribution and may include unsupported customizations, references, suggestions, or information. |
A Proxy On You!
Actual Proxy Notes Homepage
Please see Ajcody-Proxy-Notes
Very Rough Draft For Proxy Trouble Shooting Script - More of steam of consciousness really
Do the following on one of your ZCS servers:
$ su - zimbra $ touch /tmp/proxy_support.sh $ chmod +x /tmp/proxy_support.sh
Now just vi /tmp/proxy_support.sh again and cut and paste the contents below and save the file. Make sure your not getting odd line wrapping. If you do, try clicking on the view source option at the top of the wiki page and see if that allows your paste into your shell/vi to work better. Note - I removed the leading spaces one would normally use to make the script an easier read since it causes a paste to be a horrible format.
To Do Items: Have a table formatted output, like "Variable Name | Global | ServerA | ServerB | Server C | Server D | Server E | Description of Var | Recommendations" and to include a * next to a variable if we know it should be set to something else when compared to other vars in the environment.
#!/bin/bash printf "\nProxy Trouble Shooting Script For ZCS 8+ Servers\n\n" printf "\n\n Reference: https://wiki.zimbra.com/wiki/Ajcody-Proxy-Notes#Very_Rough_Draft_For_Proxy_Trouble_Shooting_Script_-_More_of_steam_of_consciousness_really.\n\n" ### Global Configurations printf "\n\n## Getting Some Global Variables ##\n\n" printf "## GARPU [getAllReverseProxyURLs] - Used to publish into nginx.conf what servers should be used for reverse proxy lookup.\n" printf "## Output from zmprov garpu ##\n\n `zmprov garpu` \n\n" printf "## GARPB [getAllReverseProxyBackends] - no description at this time.\n" printf "## Output from zmprov garpb ##\n\n `zmprov garpb` \n\n" printf "## GARPD [getAllReverseProxyDomains] - no description at this time.\n" printf "Output from zmprov -l garpd ##\n\n `zmprov -l garpd` \n\n" printf "## GAMCS [getAllMemcachedServers] - Used to list memcached servers (for nginix use).\n" printf "Ouput from zmprov gamcs ##\n\n `zmprov gamcs` \n\n" ; printf "## Some Global Configs from the following command: ##\n" printf "## [zimbra@]$ zmprov gcf | egrep 'zimbraWebClientLogoutURL|zimbraWebClientLoginURL|zimbraMailURL|zimbraMailReferMode|zimbraReverseProxySSLToUpstreamEnabled|zimbraMailProxyReconnectTimeout|zimbraMailProxyMaxFails' \n\n" zmprov gacf | egrep 'zimbraWebClientLogoutURL|zimbraWebClientLoginURL|zimbraMailURL|zimbraMailReferMode|zimbraReverseProxySSLToUpstreamEnabled|zimbraMailProxyReconnectTimeout|zimbraMailProxyMaxFails' printf "## [For ZCS8+] If zimbraReverseProxySSLToUpstreamEnabled is set to TRUE then zimbraMailMode on the server should NOT be set to http. It should be either https or both. \n\n" printf "## For details on zimbraMailProxyReconnectTimeout & zimbraMailProxyMaxFails , see https://bugzilla.zimbra.com/show_bug.cgi?id=80135 . Recommendation [ZCS8+] is to set to 0 , when encountering 502/504 timeout errors. \n\n" printf "\n\n Continuing To The Specific Server Variables...\n\n" zmlicense -p ### Server Configuration for server_hostname in `zmprov gas` do printf "## Need IP Information For Server $server_hostname . Using /etc/resolv.conf on local server to resolve.##\n\n" printf "`host $server_hostname` \n\n" printf "## Checking Enabled Services For Server $server_hostname ##\n\n" zmprov gs $server_hostname | grep ServiceEnabled printf "\n## Checking Running Services For $server_hostname ##\n\n" zmcontrol -H $server_hostname status printf "\n\n## Checking for Some Proxy Variables On $server_hostname ##\n\n" zmprov gs $server_hostname | egrep 'zimbraReverseProxyHttpEnabled|zimbraReverseProxyMailEnabled|zimbraMailReferMode|zimbraMailMode|zimbraReverseProxySSLToUpstreamEnabled|zimbraMailProxyReconnectTimeout|zimbraMailProxyMaxFails' printf "## [For ZCS8+] If zimbraReverseProxySSLToUpstreamEnabled is set to TRUE then zimbraMailMode on the server should NOT be set to http. It should be either https or both. \n\n" printf "## For details on zimbraMailProxyReconnectTimeout & zimbraMailProxyMaxFails , see https://bugzilla.zimbra.com/show_bug.cgi?id=80135 . Recommendation [ZCS8+] is to set to 0 , when encountering 502/504 timeout errors. \n\n" printf "\n\n## Checking for Some Proxy Port Variables On $server_hostname ##\n\n" zmprov gs $server_hostname | grep Port printf "\n\n## Checking for zimbraReverseProxyLookupTarget ; if TRUE then you should be able to telnet to port 7072 on that server. ##\n" printf "## telnet $server_hostname:7072 \n" printf "## If $server_hostname shouldn't be a part of the lookup pool. You can remove it by doing:\n" printf "## [zimbra@]$ zmprov ms $server_hostname zimbraReverseProxyLookupTarget FALSE \n" printf "## Then check the output of zmprov garpu:\n" printf "## [zimbra@]$ zmprov garpu \n" printf "## Then rebuild the configs on all the PROXIES and restart (one at a time is fine):\n" printf "## [zimbra@]$ zmnginxctl restart \n" printf "## After doing this, check that $server_hostname is no longer listed in any proxy configs on all proxies:\n" printf "## [zimbra@]$ grep $server_hostname /opt/zimbra/conf/nginx/includes/*\n" printf "## $server_hostname Currently has zimbraReverseProxyLookupTarget set to:\n\n" zmprov gs $server_hostname | grep zimbraReverseProxyLookupTarget printf "\n\n Finished Getting Variables for $server_hostname.\n\n" done ### Domain Configuration for domain in `zmprov gad` do printf "## Getting Some Domain Information.##\n\n" printf "Getting Information for $domain \n\n" printf "## Getting zimbraVirtualHostName And zimbraVirtualIPAddress Information for $domain ##\n\n" zmprov gd $domain | grep zimbraVirtual printf "## Getting zimbraPublicServiceHostname, zimbraPublicServicePort And zimbraPublicServiceProtocol for $domain ##\n\n" zmprov gd $domain | grep zimbraPublicService printf "\n\n Finished Getting Variables for $domain.\n\n" done printf "\n\n Script Complete.\n\n"
Not Finished Part
Not Finished - Need to now adjust how to handle non-zcs commands to be executed on remote hosts . I would like to be able to add to this script but can't with a default multi-server ZCS environment.
Notes on limitations:
- The below isn't effective for what we need because of the limitation noted after it.
- ssh -i .ssh/zimbra_identity -o strictHostKeyChecking=no zimbra@server.domain.com -p 22
- cat /opt/zimbra/.ssh/authorized_keys will give you something like:
- command="/opt/zimbra/libexec/zmrcd" ssh-dss AAAAB3NzaC1kc3MAA [cut] A== zcs806.us.zimbralab.com
- zmrcd restricts what commands can be ran, cat /opt/zimbra/libexec/zmrcd to see.
## cut - ifconfig output instead of using host $server_hostname for server_hostname in `zmprov gas` do printf "## Need IP Information For Server $server_hostname ##\n\n" printf "`ifconfig -a` \n\n" ## cut printf "\n\n## Checking for Some Proxy Port Variables On $server_hostname ##\n\n" zmprov gs $server_hostname | grep Port printf "\n\n## Confirmig Ports Are Available ## \n\n" ZCSPXYPORTS=\'`zmprov gs $server_hostname | grep Port | cut -d':' -f2 | tr '\n' '|' | tr -d ' ' | xargs echo`\' netstat -plnt | egrep `echo $ZCSPXYPORTS` ## cut
UPDATE PLEASE SEE THESE FIRST
No Route To Host
[working draft]
If your getting "No route to host" errors in /opt/zimbra/log/nginx.log files on the proxy servers, you should check:
- The resolution [DNS] to the host from the proxy servers is correct and working. This might happen more when you've deployed a new ZCS server in your environment.
- Following step one, you also have to confirm there is port level access to between proxy servers and the server it's trying to reach. memcache port is 11211 .
- The server/s are too busy to serve the request.
- You have a server in the list that shouldn't be in the lookup pool or you should remove the trouble server from the pool to avoid any more customer issues to deescalate the situation.
50X Errors
- 500 Internal Server Error
- A generic error message, given when an unexpected condition was encountered and no more specific message is suitable.
- 502 Bad Gateway
- The server was acting as a gateway or proxy and received an invalid response from the upstream server.
- 503 Service Unavailable
- The server is currently unavailable (because it is overloaded or down for maintenance). Generally, this is a temporary state. Sometimes, this can be permanent as well on test servers.
- 504 Gateway Timeout
- The server was acting as a gateway or proxy and did not receive a timely response from the upstream server.
These following configuration changes for an upstream mailbox server can improve proxy behavior related to timeouts:
- This will configure proxy to immediately reconnect to all mailbox server on any failure
$ zmprov mcf zimbraMailProxyReconnectTimeout 0
- If necessary, for each mailbox server
$ zmprov ms <mailbox server hostname> zimbraMailProxyReconnectTimeout 0
- This will configure proxy to ignore failures in regards to disconnects to all mailbox servers
$ zmprov mcf zimbraMailProxyMaxFails 0
- If necessary, for each mailbox server
$ zmprov ms <mailbox server hostname> zimbraMailProxyMaxFails 0
- Then, restart all the proxies to regenerate the nginx config files with the updated values for 'fail_timeout' & 'max_fails':
$ zmproxyctl restart
See the following bug for more details on this recommendation:
- "Improved proxy timeout defaults"
Admin Guide Update
Our Administration Guide now has everything you should need to setup your proxy server.
Please see:
How To Enable Proxy
And Also Another More Current How-To Enable Proxy here:
My Own Proxy How-To For Setup
Note: We are working on improving the official documentation. I haven't seen the document yet for the 5.0.11 release, but I've been told it will have many improvements.
- See - http://bugzilla.zimbra.com/show_bug.cgi?id=32551
- The final results of this RFE are here now:
Here's my attempt at writing something better in the meantime.
High Level Overview
One Package
First, there's only ONE package or rpm in the installer for "proxy".
Where Does It Get Installed?
It can be installed with any of your Zimbra servers (a complete ZCS install, ZCS MTA, ZCS LDAP, ZCS Mailstore) or on its own server. ( Reference )
Other proxy documentation might give the impression it needs to go onto a zimbra server that is running some other function - mta, ldap, or mailstore.
Three Components To The Package
This package, can be broken down into THREE components.
Nginx
Nginx is a high performance IMAP/POP3 proxy server which handles all incoming POP/IMAP requests. In addition to this, it is also able to reverse proxy HTTP requests to the right backend (mailstore) server. This helps to hide the hostnames of the backend mailbox servers from end users. The steps involved in this lookup are described in more detail under the Flow_Process_For_Proxy_Target_Lookups below.
Memcached
Memcached is a high performance, distributed memory object caching system (an open source distributed in-memory hashtable). Route information is cached for further use in order to increase performance.
Memcache does not have authentication and security features so the servers should have a firewall set up appropriately. The default port is 11211 and is controlled by zimbraMemcacheBindPort conf setting in zimbraserver.
- Note: Memcached will be split into its own service in the future. Meaning, for example, that you'll be able to configure a server to solely perform this function of providing the cache results. This will give more flexibility in managing performance issues, as related to the proxy environment. Rather than the same server having to share cpu/memory resources with nginx. It will also allow you to configure your different servers running nginx to quiry against a "shared" cached table of memcached. Currently, each nginx will have unique results in their memcached tables.
What is being cached, are the prior results of pop and imap lookups of users and their corresponding mailbox server they should be routed to. The process is described below in Flow_Process_For_Proxy_Target_Lookups.
Zimbra Proxy Route Lookup Handler
Zimbra Proxy Route Lookup Handler is a servlet located on the ZCS mailbox servers. This servlet handles queries for the user account route information (the server and port number where the user account resides). This is more about a difference in functionality when the proxy package is being used rather than a "new" software package being installed. The functionality difference is coming from the "software" related to nginx and memcached. The process is described in more detail in Flow_Process_For_Proxy_Target_Lookups.
- NOTE - Again, don't let the reference to "mailbox server" confuse you. The mailbox servers don't need the proxy package installed on them directly to see this effect. The proxy server (whatever one you install it on, will make configuration changes global to handle this function. The referenced servlet is already on a base mailbox server install. This is referring to how the proxy package components will or can interact with that pre-existing servlet on the mailbox server.
Flow Process For Proxy Target Lookups
Here's the flow of the process for pop and imap lookups:
- Nginx will query the LOCAL memcached service first to see if the lookup result has a cached result. [FYI - Memcache runs over port 11211]
- Note: when memcached is allowed to run on another server besides the one where nginx is at, you'll need to account for this port 11211 access between the servers. For now, it's all occurring locally to the server. If you can't access this memcache process, confirm you don't have any firewall/port level blocking occur for the local machine communicating over port 11211.
- If the initial lookups by nginx against the LOCAL memcached service have no results, nginx will then proceed to query against the "Zimbra Proxy Route Lookup Handler" services.
- By default, all mailbox servers participate in this lookup process via the "Zimbra Proxy Route Lookup Handler".
- This "where" to lookup is determined by the zimbraReverseProxyLookupTarget server attribute on servers running the mailbox service.
- The lookups are performed in a round-robin fashion across configured mailbox servers.
- Now that there is a successful lookup, the results are then cached in LOCAL memcached service and future lookups will bypass the "Zimbra Proxy Route Lookup Handler" steps for that specific lookup.
- Meaning, the lookups will solely operate between nginx and LOCAL memcached service and no longer involve the "Zimbra Proxy Route Lookup Handler"/mailbox stores.
- By default, all mailbox servers participate in this lookup process via the "Zimbra Proxy Route Lookup Handler".
Here's the flow of the process for reverse proxy http lookups:
- Nginx will query against the "Zimbra Proxy Route Lookup Handler" services.
- By default, all mailbox servers participate in this lookup process via the "Zimbra Proxy Route Lookup Handler".
- This "where" to lookup is determined by the zimbraReverseProxyLookupTarget server attribute on servers running the mailbox service.
- The lookups are performed in a round-robin fashion across configured mailbox servers.
- Ngninx will perform this lookup for every http request, it does not involve the memcached server for cache results.
- By default, all mailbox servers participate in this lookup process via the "Zimbra Proxy Route Lookup Handler".
Three Functions That Can Be Done With Proxy Package
There are THREE functions or options within Zimbra that relate to "proxy".
IMAP - POP Proxy
IMAP/POP Proxy is about the use of an IMAP/POP proxy server routes users of these services to the Zimbra mailbox server on which their mailbox resides.
For example, proxying allows users to enter imap.example.com as their IMAP server. The proxy running on imap.example.com inspects their IMAP traffic, does a lookup to determine which backend mailbox server a user’s mailbox lives on (mbs1.example.com, for example), and transparently proxies the connection from user’s IMAP client to the correct mailbox server.
Flow Of The Login Process: When an IMAP or POP3 client logs in through the proxy, the following takes place:
- The proxy analyzes the login sequence
- Extracts the user name of the user trying to login
- Does a HTTP lookup on a mailbox server to find out which server the mailbox of the user attempting to login lives on
- This lookup service runs on mailbox servers on port 7072, and this port on mailbox servers should be available from all proxy servers.
- Which mailbox servers participate in this lookup is determined by the zimbraReverseProxyLookupTarget server attribute on servers running the mailbox service.
- By default all mailbox servers participate in this lookup.
- Lookup is performed round-robin across configured mailbox servers.
- The result of the login name to mailbox server lookup are cached in memcached (an open source distributed in-memory hashtable).
- The memcached process is run alongside all IMAP/POP proxy services (nginx). Meaning wherever the proxy package was install and is configured to run.
- Which mailbox servers participate in this lookup is determined by the zimbraReverseProxyLookupTarget server attribute on servers running the mailbox service.
- This lookup service runs on mailbox servers on port 7072, and this port on mailbox servers should be available from all proxy servers.
- Also see Flow_Process_For_Proxy_Target_Lookups, if you have not.
Port Changes For IMAP/POP Proxying: When the proxy server is configured, the service ports on backend Zimbra mailbox server are changed to alternate ports. The proxy now services the standard ports for these protocols. This change is applied even if the proxy services are run on their own independent hosts, in order to distinguish and avoid confusion between the services.
- Here's the changes that will be made on the mailbox servers and what ports will now be handled by the proxy server(s).
- Note: If you have any other services running on these ports, turn them off.
- Standard Ports now server or handled by Proxy server(s)
- IMAP proxy port 143
- IMAP SSL proxy port 993
- POP proxy port 110
- POP proxy port 995
- Mailbox Servers will now work on these alternate ports
- IMAP port 7072
- IMAP SSL port 7143
- POP port 7110
- POP SSL port 7995
- Route Lookup Handler (new port) 7072
- This was described above in regards to the "Flow Of The Login Process".
- Standard Ports now server or handled by Proxy server(s)
- Note: If you have any other services running on these ports, turn them off.
Reverse HTTP Proxy
- Reverse HTTP Proxy
- Using an nginx-based reverse proxy for HTTP helps to hide names of backend mailbox servers from end users. For example, users can always use their web browser to visit the proxy server at http://mail.example.com. The connection from users whose mailbox lives on mbs1. example.com will be proxied to mbs1.example.com by the proxy running on the mail.example.com server. In addition to the ZCS web interface, clients such as REST and CalDAV clients, Zimbra Connector for Outlook and Zimbra Mobile Sync devices are also supported by the proxy.
- HTTP reverse proxy routes requests as follows:
- If the request has an auth token cookie (ZM_AUTH_TOKEN), the request is routed to the backend mailbox server of the authenticated user.
- If the requesting URL can be examined to determine the user name, then the request is routed to the backend mailbox server of the user in the URL. REST, Ca lDAV, and Zimbra Mobile Sync are supported through this mechanism.
- If the above methods do not work, the IP hash method is used to load balance the requests across the backend mailbox servers which are able to handle the request or do any necessary internal proxying.
- HTTP reverse proxy routes requests as follows:
- Using an nginx-based reverse proxy for HTTP helps to hide names of backend mailbox servers from end users. For example, users can always use their web browser to visit the proxy server at http://mail.example.com. The connection from users whose mailbox lives on mbs1. example.com will be proxied to mbs1.example.com by the proxy running on the mail.example.com server. In addition to the ZCS web interface, clients such as REST and CalDAV clients, Zimbra Connector for Outlook and Zimbra Mobile Sync devices are also supported by the proxy.
Virtual Hosting
Please see Ajcody-Server-Misc-Topics#Proxy_.5BNGINX.5D_Virtual_Hosting
Proxy References
NGINX Open-Source Project Homepage
Proxy documentation BUG/RFE's:
- "nginx: update docs"
- "add nginx section to the performance tunning guide"
- "need wiki page on setting up public service hostname and reverse proxy"
From Install Guide':
- Multiple-Server Installation - Installing the Zimbra-proxy package
- Configuring IMAP and POP Proxy Server
- Configuring ZCS HTTP Proxy (Beta 5.0.6)
- Configuring for Virtual Hosting
- Working With Zimbra Proxy (IMAP/POP)
- Working With Zimbra Proxy (HTTP) "...able to reverse proxy HTTP requests to the right backend server"
- Please note, that ZCS HTTP Proxy (http reverse proxy) is Beta, first released in 5.0.6.
- http://www.zimbra.com/docs/os/latest/administration_guide/ZimbraProxy.7.1.html#1099560
From Wiki:
- NGINX Configuration Tips
- Zimbra Proxy Guide (5.x)
- Configuring Perdition (4.5.x)
- Multiple SSL Virtual Hosts 5.0
zmproxyinit Help Output [Oct 16, 2008 - From 5.0.9 I believe]:
[zimbra@mail2 libexec]$ ./zmproxyinit Usage: ./zmproxyinit [-h] [-o] [-m] [-w] [-d [-r] [-s] [-a w1:w2:w3:w4] [-i p1:p2:p3:p4] [-p p1:p2:p3:p4] [-x mailmode]] [-e [-a w1:w2:w3:w4] [-i p1:p2:p3:p4] [-p p1:p2:p3:p4] [-x mailmode]] [-f] -H hostname -h: display this help message -H: Hostname of server on which enable/disable proxy functionality. -a: Colon separated list of Web ports to use. Format: HTTP-STORE:HTTP-PROXY:HTTPS-STORE:HTTPS-PROXY (Ex: 8080:80: 8443:443) -d: disable proxy -e: enable proxy -f: Full reset on memcached port and search queries and POP/IMAP throttling. -i: Colon separated list of IMAP ports to use. Format: IMAP-STORE:IMAP-PROXY:IMAPS-STORE:IMAPS-PROXY (Ex: 7143:14 3:7993:993) -m: Toggle mail proxy portions -o: Override enabled checks -p: Colon separated list of POP ports to use. Format: POP-STORE:POP-PROXY:POPS-STORE:POPS-PROXY (Ex: 7110:110:799 5:995) -r: Disable reverse proxy lookup target for store server. Only valid with -d. Be sure that you intend for all p roxy function for the server to be disabled -s: Set cleartext to FALSE (secure mode) on disable -w: Toggle Web proxy portions -x: zimbraMailMode to use on disable (Default: http) hostname is the value of the zimbra_server_hostname LC key for the server being modified. Required options are -f by itself, or -f with -d or -e. Note that -d or -e require one or both of -m and -w. Note that -i or -p require -m. Note that -a requires -w. Note that -x requires -w and -d for store. Note that -x requires -w for proxy. The following are the defaults for -a, -i, -p, and -x if they are not supplied as options. -a default on enable: 8080:80:8443:443 -a default on disable: 80:0:443:0 -i default on enable: 7143:143:7993:993 -i default on disable: 143:7143:993:7993 -p default on enable: 7110:110:7995:995 -p default on disable: 110:7110:995:7995 -x default on store disable: http -x default on proxy enable/disable: http
What's Wrong?
Did you check in /opt/zimbra/log/nginx.log ?
Notable Proxy Bugs
Unresolved At This Time (Sept 25, 2008)
- "RFE: Provide way to have different proxy backends and lookup urls on different Nginx hosts"
- "Proxy: unable to add a buddy"
- "In reverse-proxied mode, when redirecting with a 302, web client should set Location to proxy provided Host header"
- "nginx fails to redirect the request to mail server if i request using the alias of proxy"
- "Nginx: httpproxy infinite loop scenario"
- "multiple SSL certificates on one server"
- See : http://bugzilla.zimbra.com/show_bug.cgi?id=8128
- Allot of private comment details unfortunately.
- See : http://bugzilla.zimbra.com/show_bug.cgi?id=8128
Unsorted
- "ZCS Mailbox Server Proxy"
- See: http://bugzilla.zimbra.com/show_bug.cgi?id=9469
- From release notes, "HTTP proxy is supported (Beta). This includes REST sharing, Caldav, and Zimbra mobile. For Zimbra Mobile, ZCS redirect/forwards to the correct host. (Bug 9469)"
- Please note, that ZCS HTTP Proxy (http reverse proxy) is Beta, first released in 5.0.6.
- Though, as of 5.0.8,""For activesync connection to go thru the proxy, they'll have to configure reversed proxy. However, there're already a few known bugs about sharing broken with reversed proxy enabled. They're better off waiting for 5.0.9 than to try to configure reversed proxy now."
- "need to support multi-server (proxy) for mobile clients"
- See: http://bugzilla.zimbra.com/show_bug.cgi?id=8174
- Is listed as a dup of http://bugzilla.zimbra.com/show_bug.cgi?id=9469
Resolved In 5.0.9
Those That Are Sort Of Client Side Impacting
- "Enabling http proxy breaks sharing"
- "Inconsistent default mail/proxy mode"
- "WRONG_HOST error while viewing a public shared doc using http proxy"
- "mailbox does not load with zimbraPublicServiceHostname and reverse proxy enabled"
- "Modify SOAP harness to handle session ID to work with proxy"
- "mod_zimbra_proxy infinite redirect"
Configuration / Initial Setup Issues
- "Remove requirement that zimbraPublicServiceHostname have a corresponding zimbraServer object"
- "Remove hostname argument from zmproxyinit"
- "Zmmtaconfig does not monitor several imap proxy variables"
- "Zmproxyconfgen should not generate from templates if critical values are missing"
- "Zmproxyconfgen does not set $? correctly on error"
- "zmproxyconfgen -P has no effect"
- "Webproxy https in mixed mode does not stay in https"
- "Upgrade: starttls mode changed in nginx.conf"
- "better GSSAPI support in nginx"
- "nginx mail proxy does not translate kerberos principal name to zimbra user-name"
Resolved in 5.0.7
- "Proxy section of admin guide needs to be updated"
- "Zmproxyinit -e -w -x is not allowed"
- "Zmproxyinit does not set zimbraReverseProxyLookupTarget for store only server"
- "Zmproxyinit -e -w port conflict on store only install"
- "Zmproxyinit -e -w failure on proxy only install"
- "Zmproxyinit: can not run zmproxyinit on mbs without proxy installed"
- "Need attribute to track proxy mailmode"
- "nginx: zmproxygen needs to get attributes from server if an attribute is server/global"
- "Zmproxyinit -e -w failure on single node install"
- "zimbra imap/pop clear text not enabled for mail proxy install"
- "nginx: http proxy needs to have the rewrite module built in"
- "nginx: HTTP proxy configuration scripts need to be updated to reflect various mail modes"
- "nginx: http proxy should support all mail modes"
- "nginx: improve http proxy Location header fudging capabilities"
- "nginx uses client-ip instead of proxy-ip when caching login alias"
- "nginx http proxy should route by zauthtoken query-string argument"
- "Improvements to zmproxyinit"
- "zmproxyinit sets the wrong mail mode"
- "nginx doesn't proxy caldav"
- Appears to be about webdav issues and others as well.
- http://bugzilla.zimbra.com/show_bug.cgi?id=27042
- "Zmproxyinit hard code port settings"
- "Isync (REST) Needs To Work With Nginx Http Proxy"
- Note, many issues were still unresolved with this "resolved" bug. You'll see them above.
- Also, see notes under "ZCS Mailbox Server Proxy" under the Ajcody-Server-Topics#Unsorted section below.
- See: http://bugzilla.zimbra.com/show_bug.cgi?id=25981
Resolved in 5.0.6
Public Service Hostname
- The variable is zimbraPublicServiceHostname , as referenced in /opt/zimbra/openldap/etc/openldap/schema/zimbra.schema . Bugs about the url's for shares that use spaces (%20), the redirection drops the %20
Resolved in 5.0.5
- Nginx: http proxy does not work with REST
- Note, this isn't "completely" resolved, as you'll notice by other targeted bugs with 5.0.5+ releases.
- See: Nginx: http proxy does not work with REST
Single ZCS Server
Need To Disable Pop/Imap Proxy And Use POP/IMAP Normally
Sometimes, people install/setup proxy services on their single ZCS server and they don't need them. Here's how you would disable the proxy stuff and get imap/pop working over the default ports.
do a zmprov -l gs `zmhostname` | grep -i port get the ports, then set variables to port 0: zmprov ms `zmhostname` zimbraImapProxyBindPort 0 zmprov ms `zmhostname` zimbraImapSSLProxyBindPort 0 zmprov ms `zmhostname` zimbraPop3ProxyBindPort 0 zmprov ms `zmhostname` zimbraPop3SSLProxyBindPort 0 then, set the non "Proxy" ports to the desired standard ports zmprov ms `zmhostname` zimbraImapBindPort 143 zmprov ms `zmhostname` zimbraImapSSLBindPort 993 zmprov ms `zmhostname` zimbraPop3BindPort 110 zmprov ms `zmhostname` zimbraPop3SSLBindPort 995 once complete: zmprov ms `zmhostname` -zimbraServiceEnabled memcached zmprov ms `zmhostname` -zimbraServiceEnabled imapproxy zmproxyctl stop zmmemcachedctl stop zmmailboxdctl stop zmmailboxdctl start
Proxy Guide Rewrite Scratch Pad
Intro
Moved to Ajcody-Proxy-Guide-Rewrite-Project
Clustering Topics
Actual Clustering Topics Homepage
Please see Ajcody-Clustering
My Other Clustering Pages
- Ajcody-Notes-Upgrade-Options#Upgrade_Steps_for_Multi-Servers_with_Clustered_Mailstores
- Ajcody-Notes-Of-Customer-Cluster-Upgrade
RFE I made based upon the experience above:
- "QA & Corrections For Cluster Upgrade Document"
Clustering For ZCS 8 And Above
ZCS 8 Specifics
From the ZCS 8.0.5 Release Notes:
Red Hat Cluster Suite is not available with ZCS 8.0 To streamline support efforts, we will only test and certify in house availability solutions. Today VMware offers clustering failover and automated recovery through VMware HA. Zimbra integrates with the VMware HA cluster infrastructure to heartbeat Zimbra application services and provide automated recover in the event of a service failure. (Bugs 72215/72216)
- Private bugs about the EOL of RHCS
- Deprecate RHCS in 7.2
- EOL RHCS
If you use Zimbra Clustering
Zimbra Clustering is no longer available for ZCS 8.0. VMware provides integrated high availability between Zimbra and VMware HA for automated recovery of critical Zimbra application services and server components in the event of an application or infrastructure failure. Third party solutions such as network load balancers, storage mirroring, or OS clustering solutions like Red Hat Cluster Suite may be used in your deployment, but are not specifically tested or certified by VMware. (Bug75821)
- Clustering not in ZCS 8.0
Statement On Zimbra Support And "Clustering" For ZCS 8+ [In My Own Words - Ajcody]
I've sent this off to our PM team to review and to provide formal comments and documentation updates on it. - Ajcody
Official and Full support are NOT terms Zimbra Support applies to Redhat RHCS or Vmware's Clustering options since our team does not support either of those two options directly. We do not support customers in configuring or trouble-shooting the clustering components for Redhat's RHCS or Vmware's Clustering.
Will and can RHCS and Vmware's Clustering work with ZCS, yes. Does our team offer support in configuring, trouble-shooting, or diagnosing the clustering component of the setup - no. You would need to contact Redhat if you went with RHEL and RHCS or Vmware and your Linux OS distribution support channel if you went with Vmware Clustering.
Note - The zimbra clustering zimlets are completely removed with ZCS 8.6 . The Vmware clustering monitoring scripts are a separate issues that falls outside of general clustering support.
Future Releases After ZCS 8
Please see the following blog about our Always ON, Carrier Grade Architecture HA features we are hoping to have in future releases:
- http://blog.zimbra.com/blog/archives/2013/09/project-always-on.html
- http://blog.zimbra.com/blog/archives/2013/04/zimbra-judaspriest-release-update-1.html
Vmware Virtualization and Clustering
Overview
From the ZCS 8.0.5 Release Notes:
Red Hat Cluster Suite is not available with ZCS 8.0 To streamline support efforts, we will only test and certify in house availability solutions. Today VMware offers clustering failover and automated recovery through VMware HA. Zimbra integrates with the VMware HA cluster infrastructure to heartbeat Zimbra application services and provide automated recover in the event of a service failure. (Bugs 72215/72216)
- Private bugs about the EOL of RHCS
- Deprecate RHCS in 7.2
- EOL RHCS
VMware Heartbeat Service
ZCS utilizes the VMware application programming interface that allows software providers to deploy application monitoring components inside a VMware guest OS and inform VMware HA when problems arise. The VMware-heartbeat service provides information to the VMware HA components on the health and availability of the ZCS. If you have VMware HA components installed, the VMware HA service is shown as enabled on the Server>Services page.
General Description of VMware HA
VMware HA provides a simple, reliable way to increase the availability of virtual machines hosting critical applications. VMware HA is a visualization- based distributed infrastructure service of VMware vSphere 4.1+, which monitors the health of virtual machines and the VMware ESX® hosts upon which they reside. If a fault is detected, the virtual machine is automatically restarted on another ESX host with adequate capacity to host it. VMware HA is included in all vSphere editions and can be enabled on a VMware cluster with a single check box. As VMware HA utilizes the storage and network connectivity already in place to support vMotion, enabling high availability is as simple as ensuring you have adequate server capacity to handle failure of one or more ESX hosts.
ZCS Scripts For Vmware-HA
The three scripts for vmware-ha are:
- /opt/zimbra/bin/zmhactl
- /opt/zimbra/libexec/vmware-heartbeat
- /opt/zimbra/libexec/vmware-appmonitor
We have an open documentation RFE to better explain these :
- "Documentation for VMware HA scripts in Zimbra"
Some older bugs describing a little more about the vmware-heartbeat check:
- "VMware HA does not work with ZCS8 or ZCA8"
- "Service level control of VMware HA Clustering heartbeat"
Enable Vmware-HA Service
To confirm the service is available to enable:
[As the zimbra user]
zmprov gs `zmhostname` zimbraServiceInstalled | grep vmware-ha zimbraServiceInstalled: vmware-ha
To enable vmware-ha on a server:
[As the zimbra user]
su - zimbra zmprov ms `zmhostname` +zimbraServiceEnabled vmware-ha zmhactl start
Vmware Performance Recommendations For Vmware Clustering
See
VMware Host Based Replication (HBR)
Questions should be addressed to Vmware Support. This feature happens outside of Zimbra and is not QA tested by us.
Private Bug [Referenced here in case it's later made public]:
- Qualification of ZCS with VMware Host Based Replication (HBR)
Other References
See also:
- "Zimbra on NFS Storage through VMware ESX"
- http://bugzilla.zimbra.com/show_bug.cgi?id=50635
- This is NFS for the virtualized storage within ESX, not within the OS.
- Current NFS policy, when the nfs mount is done within the OS.
- "No support for RHEL Clustering on top of VMware"
Webinar On Virtualizing Zimbra with VMware vSphere and NetApp NFS
Learn how the combination of VMware vSphere and NetApp file-based storage provides full support for virtualization and all of vSphere's advanced features (vMotion, Storage vMotion, DRS), simplifies high availability through the use of VMware HA and server consolidation.
View Webinar Now (Duration: 45 minutes)
Critical Bugs/RFE's - False Restarts And So Forth
Log Rotation Causes Cluster Failover
See:
- "Log rotation causes cluster failover"
Recommendations to work around bug until fix is released - 2 methods:
- First Method - disable log rotation
- Remove the zmmtaconfigctl restart from the /etc/logrotate.d/zimbra file so that it will not attempt to restart the service that is being detected as down. We can remove the line that reads:
su - zimbra -c "/opt/zimbra/bin/zmmtaconfigctl restart"
- That will stop these failures, but will affect the logging for zmmtaconfigctl, probably causing it to write to a nonexistent file. We haven't seen problems in zmmtaconfig for a long time, so this is a pretty low risk workaround.
- Second Method - disable software monitoring in general
Software Monitoring Causes Problems
- "Request Modification to zmcluctl to support hardware only failover with Redhat Cluster Manager"
- http://bugzilla.zimbra.com/show_bug.cgi?id=25456
- Note: Beginning in ZCS 6.0, hardware only failover with Redhat Cluster Manager is supported.
- http://bugzilla.zimbra.com/show_bug.cgi?id=25456
- To Disable software monitoring.
- This will prevent failover if zmcluctl finds a service down. It will not prevent failover if there is a hardware fault detected by the cluster software.
- $zmcluctl mode hardware
- or
- $zmlocalconfig -e zimbra_cluster_mode=hardware
- Note from bug/rfe - Added switch for software or hardware only based fail-over implemented as a lc zimbra_cluster_mode attribute. default mode is software meaning any non-zero exit status from zmcontrol status will trigger a failover. hardware mode means that zmcluctl status will always return a zero exit status and will not consult zmcontrol status for the state.
- To also increase the chance of getting more information in the log on what might be going on:
- In /opt/zimbra-cluster/bin/zmcluctl you should see a line like:
- my @output = `su - zimbra -c 'zmcontrol status'`;
- Change that to:
- my @output = `su - zimbra -c 'date >> /opt/zimbra/log/zmcluster-status.log ; zmcontrol status >> /opt/zimbra/log/zmcluster-status.log 2>> /opt/zimbra/log/zmcluster-status.log'`;
- That should give us more logging. I believe zmcluctl is read every time from disk when it does the check, so no restart of services should be needed.
- In /opt/zimbra-cluster/bin/zmcluctl you should see a line like:
- This will prevent failover if zmcluctl finds a service down. It will not prevent failover if there is a hardware fault detected by the cluster software.
RHEL 5 Clusters And Cisco Switches
Please see the following:
- "Openais appears to fail, causing cluster member to fence"
- https://bugzilla.redhat.com/show_bug.cgi?id=469874
- The last comment mentions a cisco issue being the cause [cisco switches are used internally for ibm blades].
- Comment 9 states most likely hardware configuration issue w/ switch or iptables.
- https://bugzilla.redhat.com/show_bug.cgi?id=469874
- "Cman kills first node in initial cluster setup"
Mysql Related Items Impacting Cluster
Please see:
- "Mysql crash recovery causes repeated software failover"
- http://bugzilla.zimbra.com/show_bug.cgi?id=36690
- "finding root cause to failed failover situations as secondary issue from bug 36690"
- http://bugzilla.zimbra.com/show_bug.cgi?id=36690
- "Flush dirty innodb pages in mysql prior to shutting down."
Other Misc Bug/RFEs
Other bugs/rfe's you might be interested in looking at:
- "zmcluctl status can return errors even when services are up"
Failover Occurring Prior To Mysql Shutting Down - RHCS
Here is the summary of the situation that was sent for a case on the matter.
- Regarding RHCS desired configuration -
- In particular, it looks like mysqld can at times take quite a long time to shutdown, especially in situations when write IO is limited on the mysql db partition, and mysqld must write cache data to disk safely before shutting down. We believe this should be reflected in the "rc=1" output from "zmcontrol stop" - if mysqld does not stop within the 60 seconds via /opt/zimbra/bin/zmcontrol (via /opt/zimbra/bin/mysql.server), and then what appears to be happening is that RHCS either tries to unmount the filesystem and/or just kill -KILL the running process, which in turns causes mysqld to die ungracefully and cause the mysqld startup on the other box to go through a long log replay. The net result is one of two scenarios:
- (a) mysqld takes longer to startup than it would have taken to shutdown (normal shutdown can take 0-10 minutes if mysqld has to flush a bunch of data to write out)
- (b) mysqld is still running when RHCS tries to unmount the filesystem, which then fails, and therefore the failover fails
- What we'd like to be able to recommend is a more robust failover logic, such as:
- 1. Run zmcontrol stop, check output
- 2. If failed, wait X seconds (e.g., 120 seconds)
- 3. Run zmcontrol stop again
- 4. If failed, wait X seconds
- 5. Run zmcontrol third time
- 6. If failed third time, try a kill -KILL and force unmount
- If RHCS can be configured to do this, it would be the best way to handle this situation. The problem with modifying zmcontrol to have this logic internally is that it should not be granted the decision to wait indefinitely, in the case that mysqld is having a critical problem and is not shutting down. Having an external process manage this logic allows for additional triangulation on the determination of the situation, and allows additional capabilities such as monitoring the files in /opt/zimbra/db/data to confirm that files are changing. For example, if mysqld is still running and no files have changed in /opt/zimbra/db/data for a significant amount of time, it may indicate that mysqld is locked up, and other steps may be required. zmcontrol would not be able to handle all of this logic internally due to the needs for additional external monitoring outside of the zmcontrol process, including return code from the script.
RFE that was made about Zimbra adjusting the our script and the wait time:
- "mysql.server should wait longer to stop mysqld"
Good Summary For RHEL Clustering
This is a good solid summary about RHEL clustering:
http://www.linuxjournal.com/article/9759
Active-Active Clustering
There is a bug(rfe) for active-active configuration. Please see:
http://bugzilla.zimbra.com/show_bug.cgi?id=19700
Non-San Based Fail Over HA/Cluster Type Configuration
This RFE covers issues when your wanting a "copy" of the data to reside on an independent server - LAN/WAN.
Please see:
- "Disaster recovery through server to server sync (beta)"
- "HA/DR through Log Shipping"
RFE's/Bug Related To Supporting Clustering Options
- "Add VCS cluster support for Suse ES 10"
- http://bugzilla.zimbra.com/show_bug.cgi?id=24303
- SuSE Clustering resources - These might be useful.
- "Cluster Configuration on SLES"
- "Clustering Your Novell Groupwise Servers - Xen & Heartbeat2 on SLES"
- "SuSE paper about HA and Virtual Servers - PDF"
- What SLES has for clustering - Linux Virtual Server
- "Add support for otther cluster software lifekeeper and mc/service guard"
- "Add clustering support for Mac OS X"
Other Clustering RFE's And Bugs
- "RFE: How-to for adding additional mounts on existing cluster deployment"
- "RFE: Expand documentation or installer for 1+1 to X+1"
HA-Linux (Heartbeat)
HA-Linux How-To For Testing And Educational Use
References:
- HA-Linux Project Homepage
- Howto: Highly available Zimbra cluster using Heartbeat and DRBD
- DRBD Homepage
- DRBD currently unsupported at this time.
- 1. "disaster recovery through server to server sync (beta)"
- 2. "add active-active support to zcs" , marked as a dup of the above.
- DRBD currently unsupported at this time.
Actual HA-Linux How-To For Testing And Educational Use Homepage
Please see Ajcody-Notes-HA-Linux-How-To
Motive Behind How-To
I hope this gives an easy way to setup through some clustering concepts for an administrator to gain some real-world experience when they currently have none. I plan on walking through each "function" that is behind clustering rather than jumping to an end setup (Linux-HA, Shared Storage, And Zimbra).
The structure will be:
- Setup two machines (physical or virtual)
- Emphasis physical hostname / ip vs. the hostname and ip address that will be for HA.
- Setup virtual hostname and ip address for HA.
- Explain and do ip failover between the two machines.
- Setup a disk mount, we'll use probably use a nfs export from a third machine.
- This will give us an example of expanding the HA conf files to move beyond the ip address failover.
- Adjust HA conf's to now export via nfs a local directory from each server. This will not be a shared physical disk of course.
- Setup a shared disk between the two servers and include it in the HA conf files.
- Can use drbd or maybe figure out a way to share a virtual disk between the two vm's.
- Setup a very simple application to include between the two machines. Something like apache or cups.
- Go back and now readjust all variables between monitoring type (automatic) failover and simple manually initiated.
Virtualization Issues
Actual Virutalization Issues Homepage
Please see Ajcody-Virtualization
Vmware - ESX - Performance & Support Resources
Internal Resources:
- Update - Great New Resource Available!!!! [May 17, 2010]
- Document under the Zimbra Appliance stuff, but still relevant for all Vmware configurations.
- When working with Zimbra for VMware related issues, you should review the following and provide the information about your vmware setup.
- "Collecting diagnostic information for VMware ESX Server 2.5, 3.0, 3.5 and 4.0"
External Resources:
- "Performance Tuning for VI3"
- "Isolating Performance Problems (PDF)"
- "Representing Physical Machines in the Virtual World (PDF)"
- "Using esxtop to Troubleshoot Performance Problems (PDF)"
- "ESX Performance Tips and Tricks (PDF)"
- "Performance Tuning Best Practices for ESX Server 3"
- "ESX Workload Analysis: Lessons Learned"
- "VM Performance Monitoring and Logging"
- "Understanding Performance in a Virtualized Environment'
- "Interpreting Performance Statistics in V13"
- "Top 12 ways to Improve Guest Server Performance under VMware ESX Server"
Vmware And Clustering
Please see Ajcody-Clustering#Vmware_Virtualization_and_Clustering
Introduction To Using VMWare ESX For ZCS Test Servers
References
References I ended up consulting to resolve and develope my notes below. There appears to be many alterations on how to do this depending on the version of the Vmware software you are using.
- Moving or copying a virtual machine within a VMware environment
- How work without vCenter Server
- Cannot open Virtual Machine Console
Introduction To Notes
I just did this and I plan on going through it and improving it once I have time.
- I believe the hard coded reference to the volumes could use the sym-link for the path:
- /vmfs/volumes/Storage vs. /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af
- Setup /etc/hosts file on first vmimage to have entries for other machines
- Configure first vmimage to use ESX server for DNS [/etc/resolv.conf] , NTP , and other network services
- Setup ESX for DNS, DHCP, NTP [below for more details]
- Configure vm configuration files, *.vmx , to use manual set MAC addresses and had last number matched number used in hostname.
- Configure ESX server to act as DNS & DHCP server using to auto-allocate ip address and hostnames and set each hostname to have proper A, PTR, and MX records using hostname of image equally domainname for mail.
- for example:
- I would create three groups of images, if possible:
- host.dev.DOMAIN.com
- DEV is for pure testing
- host.qa.DOMAIN.com
- QA is for testing of changes prior to roll out to production
- host.prod.DOMAIN.com
- PROD would be replica's of production and used to test against production issues
- host.dev.DOMAIN.com
- Images configured as:
- image hostname = centos5-30.dev.DOMAIN.com
- server centos5-31.dev.DOMAIN.com has ip address of 192.168.0.31 - set A and PTR record
- MX equals centos5-31.dev.DOMAIN.com / 192.168.0.31
- Use more expansive descriptions in hostnames for standards and predictability.
- Note, underscores ARE NOT vaild DNS hostnames per RFC 1034 - http://www.ietf.org/rfc/rfc1034.txt
- Add the DEV - QA - PROD if your using that as well.
- Primary Domain = example.com
- LDAP Master hostname = ldap-1.DOMAIN.com
- LDAP Replica hostnames = ldap-2.DOMAIN.com , ldap-3.DOMAIN.com , etc.
- First MTA hostname = mta-1.DOMAIN.com
- Other MTA hostnames = mta-2.DOMAIN.com , mta-3.DOMAIN.com , etc.
- First Mailstore hostname = mailstore-1.DOMAIN.com
- Other Mailstore hostnames = mailstore-2.DOMAIN.com , mailstore-3.DOMAIN.com , etc.
- On proxy server hostname =
- Proxy on MTA's: mta-1.DOMAIN.com , mta-2.DOMAIN.com , etc.
- Though the proxy package can be installed on other servers, usually it is installed on the mta's or on it's own.
- Proxy on it's own: proxy-1.DOMAIN.com , proxy-2.DOMAIN.com , etc.
- Proxy on MTA's: mta-1.DOMAIN.com , mta-2.DOMAIN.com , etc.
- On archive [A&D] hostname:
- First Archive hostname = archive-1.DOMAIN.com
- Other Archive hostnames = archive-2.DOMAIN.com , archive-3.DOMAIN.com , etc.
- Configure proxy/firewall to be route point for primary domain and then pass to appropriate server for subdomain mail.
- All external requess for *.zimbra.DOMAIN.com and zimbra.DOMAIN.com route to ESX server which in turns routes to appropriate vm server for subdomain.
- Ports open on firewalls to allow vsphere connection to esx server:
- http://wiki.zimbra.com/index.php?title=Ajcody-Server-Topics#Using_VMWare_ESX_For_ZCS_Test_Servers_-_How-To
- 443, 902, 903, 5989 -- all of those?
- http://wiki.zimbra.com/index.php?title=Ajcody-Server-Topics#Using_VMWare_ESX_For_ZCS_Test_Servers_-_How-To
- Add post-clone script to the "gold image" to reconfigure the cloned image for things like hostname & network information. [Thanks Tony for the idea]
How-To Setup ESX For ZCS Test Servers
Creating Initial x64 VM
Initial Setup Of Guest Image In vSphere
- Right Click on ESX server name in left column listing in vSphere Client.
- Select "New Virtual Machine"
- Select "Typical"
- Name - ex. "Centos5-x64-30"
- Format being - Distro & Distro Version - platform x32 or x64 - last octet of ip address
- Datastore - your local esx data storage you'll be using
- Guest OS > Linux > RHEL5 64bit - RHEL for Centos [using my example]
- Create a disk
- Defaults to 8GB, but this isn't enough for ZCS because ZCS requires at least 5GB's free. The OS will take about 4-6+GB's between the swap partition, /boot , and /. Using installer defaults that is.
- I make the root image for 12GB. This will be enough to get a basic ZCS install done. One can add more later if needed.
- For a 'test' environment, I would recommend leaving the other options UNCHECKED.
- Leave UNCHECKED - Allocate and commit space on demand
- Leave UNCHECKED - Support clustering features such as Fault Tolerance.
- For a 'test' environment, I would recommend leaving the other options UNCHECKED.
- Ready to Complete - Check the box that says:
- check - "Edit the virtual machine settings before completion"
- Memory > Adjust Memory to be 1024 MB or 1 GB
- CD/DVD Drive 1 >
- Check "Connected at power on"
- Check "Datastore ISO File" and select the iso image for your OS - ex. CentOS-5.4-x86_64-bin-DVD.iso.
- Centos default disk partitioning will do:
- Also, see - need wiki section - about how to prep your server to host iso images via the datastore.
- The rest can be left as the defaults for now.
- check - "Edit the virtual machine settings before completion"
- Select "New Virtual Machine"
Initial Setup Of Guest Image Operating System - CentOS/RHEL Example
- In the vSphere Client and right click on Guest image, ex. "Centos5-x64-30", and select power on.
- Prep our 'base image' prior to cloning it.
- The image should give you your Distro's installer screen. Go through the OS installation.
- Note about CentOS example
- First stage of installer
- The partitioning example if using the defaults will give you:
/dev/sda1 /boot ext3 101 MB
/dev/sdb2 VolGroup00 LVM PV 12182 MB
LogVol00 / ext3 10144 MB
LovVol01 swap 2016 MB
- The Network Devices
- Network Devices > click on edit
- Leave CHECKED - "Enabled IPv4 support"
- UNCHECKED "Enable IPv6 support"
- Network Devices > click on edit
- Package Selection
- Check the "Server" option and then click "Next"
- Installer will now finish the first stage of the installation and reboot.
- Check the "Server" option and then click "Next"
- The partitioning example if using the defaults will give you:
- Second state of installation.
- Firewall > Disable firewall.
- SELinux > Disable SELinux
- Date and Time
- Network Time Protocol - Enable NTP if you'll be able to reach the NTP servers.
- Check the "Synchronize system clock before starting service"
- Network Time Protocol - Enable NTP if you'll be able to reach the NTP servers.
- Create User > No need to create additional users for ZCS purposes.
- Finish installation.
- Post-Installation
- Note - Ctrl+Alt allows you to switch the focus of the mouse in and out of the VM guest image showing in vSphere's client.
- Note - The client 'screen' is under the "Console" tab.
- Install VMwareTools
- In the vSphere Client and right click on Guest image, ex. "Centos5-x64-30", and select Guest > Install\Upgrade VMware Tools.
- Double click on the VMwareTools rpm or install it via CLI.
- You can also run the vmware-tool configuration script
- vmware-config-tools.pl
- Allows you to adjust the screen resolution and cut-n-paste between workstation and the guest vm.
- Launch a terminal. Applications > Accessories > Terminal
- Disable sendmail from being used. Note - CentOS/RHEL uses sendmail as default, for another distro you might need to disable postfix from being used.
- In terminal, paste [if you installed vmware tools] or type the following command.
chkconfig sendmail off ; /etc/init.d/sendmail stop
- Install some prereq packages that you might not already have installed - CentOS example:
yum install compat-libstdc++-33 compat-libstdc++-296 sysstat
- Umount the vmware-tools mount - example
umount /media/VMware\ Tools/
- confirm with df -h
- Reboot server now - the guest OS image.
- Disable sendmail from being used. Note - CentOS/RHEL uses sendmail as default, for another distro you might need to disable postfix from being used.
- See also Ajcody-Virtualization#RHEL_or_CentOS for adding 'disk/partitions' to an image.
- First stage of installer
- Note about CentOS example
Manual Cloning of x32 Setup
My Version of Vmware ESX does not have a cloning option - I was actually given the ESX box rebuilt. So here's what I did to 'clone' an image the manual way. See Ajcody-Virtualization-Named-DNS about setting up BIND/DNS for this example.
# Vmware ESX 4.0.0 Build 208167 # vSphere Client 4.0.0. Build 208111 # # Copy some iso files to the server, here's is what I used. # Please note the vmx files below in my examples to my iso name for the centos dvd. # [ /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/iso/CentOS/CentOS-5.4-i386-bin-DVD.iso ] # Create your first VM - I called my Centos5-30 and that standard and naming convention is used throughout. # I left the hostname just to use the localhost and installed the vmware tools. # I then shutdown the image and did a snapshot of it. pwd /vmfs/volumes ls -la total 1024 drwxr-xr-x 1 root root 512 Apr 9 23:59 . drwxrwxrwt 1 root root 512 Apr 9 00:39 .. drwxr-xr-t 1 root root 4200 Apr 9 23:20 4bbaf57f-6230127f-d432-00101849e4af lrwxr-xr-x 1 root root 35 Apr 9 23:59 Storage1 -> 4bbaf57f-6230127f-d432-00101849e4af cd Storage1/ ls Centos5-30/ esxconsole-4bbb574e-eae0-2eb3-9d5b-00101849e4af/ iso/ # Download and copy the various OS iso's you'll want and copy them underneath your storage volume, # I made an iso directory for them all. # Also, I used Firefox and the add-on DownThemAll and went to the Zimbra download pages # http://www.zimbra.com/downloads/ne-downloads.html # http://www.zimbra.com/downloads/ne-downloads-previous.html # And downloaded all the versions of ZCS that I wanted to test against. I then used K3B under Linux # to make iso files of the tar balls. Copy the iso files to your ESX server under the iso directory # you made. If you want to mount them remotely to confirm they are ok to or to review what they have: # ex. mount -t iso9660 ./zcs-x32-installs.iso /mnt/cdrom -o ro,loop # Having the Zimbra iso files like that will allow your images to easily install Zimbra . ls -R iso/ CentOS/ openSUSE/ SLES/ Ubuntu/ ZCS/ ./CentOS: CentOS-5.4-i386-bin-DVD.iso CentOS-5.4-x86_64-bin-DVD.iso ./openSUSE: openSUSE-11.2-Addon-NonOss-BiArch-i586-x86_64.iso openSUSE-11.2-DVD-i586.iso openSUSE-11.2-DVD-x86_64.iso ./SLES: SLES-10-SP3-DVD-i386-GM-DVD1.iso SLES-10-SP3-DVD-x86_64-GM-DVD1.iso SLES-11-DVD-i586-GM-DVD1.iso SLES-10-SP3-DVD-i386-GM-DVD2.iso SLES-10-SP3-DVD-x86_64-GM-DVD2.iso SLES-11-DVD-i586-GM-DVD2.iso ./Ubuntu: ubuntu-8.04.4-desktop-amd64.iso ubuntu-8.04.4-desktop-i386.iso ubuntu-8.04.4-server-amd64.iso ubuntu-8.04.4-server-i386.iso ./ZCS: zcs-x32-606-02_5023-018.iso zcs-x64-5018-23.iso zcs-x64-603-06.iso vmware-cmd -l /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-30/Centos5-30.vmx pwd /vmfs/volumes/Storage1 # Make more directories for you to copy the initial image vitrual disks to mkdir Centos5-{31..49} # Now you'll do a copy operation of your first image to the new directories you made pwd /vmfs/volumes/Storage1 vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-31/Centos5-31.vmdk vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-32/Centos5-32.vmdk vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-33/Centos5-33.vmdk vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-34/Centos5-34.vmdk vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-35/Centos5-35.vmdk vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-36/Centos5-36.vmdk vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-37/Centos5-37.vmdk vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-39/Centos5-39.vmdk vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-39/Centos5-39.vmdk vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-40/Centos5-40.vmdk vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-41/Centos5-41.vmdk vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-42/Centos5-42.vmdk vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-43/Centos5-43.vmdk vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-44/Centos5-44.vmdk vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-45/Centos5-45.vmdk vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-46/Centos5-46.vmdk vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-47/Centos5-47.vmdk vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-48/Centos5-48.vmdk vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-49/Centos5-49.vmdk pwd /vmfs/volumes/Storage1 # Now you'll copy the inital images configuration file to the directory above the image directories # so we can modify it and make it a template file cp Centos5-30/Centos5-30.vmx ./Centos5-31.vmx vi Centos5-31.vmx change variables to be more generic cat Centos5-31.vmx #!/usr/bin/vmware .encoding = "UTF-8" config.version = "8" virtualHW.version = "7" pciBridge0.present = "TRUE" pciBridge4.present = "TRUE" pciBridge4.virtualDev = "pcieRootPort" pciBridge4.functions = "8" pciBridge5.present = "TRUE" pciBridge5.virtualDev = "pcieRootPort" pciBridge5.functions = "8" pciBridge6.present = "TRUE" pciBridge6.virtualDev = "pcieRootPort" pciBridge6.functions = "8" pciBridge7.present = "TRUE" pciBridge7.virtualDev = "pcieRootPort" pciBridge7.functions = "8" vmci0.present = "TRUE" nvram = "Centos5-31.nvram" deploymentPlatform = "windows" virtualHW.productCompatibility = "hosted" unity.customColor = "|23C0C0C0" tools.upgrade.policy = "useGlobal" powerType.powerOff = "soft" powerType.powerOn = "default" powerType.suspend = "hard" powerType.reset = "soft" displayName = "Centos5-31" extendedConfigFile = "Centos5-31.vmxf" floppy0.present = "TRUE" scsi0.present = "TRUE" scsi0.sharedBus = "none" scsi0.virtualDev = "lsilogic" memsize = "1024" scsi0:0.present = "TRUE" scsi0:0.fileName = "Centos5-31.vmdk" scsi0:0.deviceType = "scsi-hardDisk" ide1:0.present = "TRUE" ide1:0.clientDevice = "FALSE" ide1:0.deviceType = "cdrom-image" ide1:0.startConnected = "TRUE" floppy0.startConnected = "FALSE" floppy0.clientDevice = "TRUE" ethernet0.present = "TRUE" ethernet0.networkName = "VM Network" ethernet0.addressType = "generated" guestOSAltName = "Red Hat Enterprise Linux 5 (32-bit)" guestOS = "rhel5" uuid.location = "56 4d 41 ed 87 cc a2 03-77 97 48 8c 65 16 7e ed" uuid.bios = "56 4d 41 ed 87 cc a2 03-77 97 48 8c 65 16 7e ed" vc.uuid = "52 9e a1 5c 02 25 0f c2-20 03 f8 bb 21 93 74 c7" ethernet0.generatedAddress = "" tools.syncTime = "FALSE" cleanShutdown = "TRUE" replay.supported = "FALSE" sched.swap.derivedName = "/vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-31/Centos5-31-d6823091.vswp" scsi0:0.redo = "" vmotion.checkpointFBSize = "4194304" pciBridge0.pciSlotNumber = "17" pciBridge4.pciSlotNumber = "21" pciBridge5.pciSlotNumber = "22" pciBridge6.pciSlotNumber = "23" pciBridge7.pciSlotNumber = "24" scsi0.pciSlotNumber = "16" ethernet0.pciSlotNumber = "32" vmci0.pciSlotNumber = "33" ethernet0.generatedAddressOffset = "0" vmci0.id = "-876333085" hostCPUID.0 = "0000000b756e65476c65746e49656e69" hostCPUID.1 = "000106a500100800009ce3bdbfebfbff" hostCPUID.80000001 = "00000000000000000000000128100800" guestCPUID.0 = "0000000b756e65476c65746e49656e69" guestCPUID.1 = "000106a500010800809822010febfbff" guestCPUID.80000001 = "00000000000000000000000128100800" userCPUID.0 = "0000000b756e65476c65746e49656e69" userCPUID.1 = "000106a500100800009822010febfbff" userCPUID.80000001 = "00000000000000000000000128100800" evcCompatibilityMode = "FALSE" ide1:0.fileName = "/vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/iso/CentOS/CentOS-5.4-i386-bin-DVD.iso" floppy0.fileName = "/dev/fd0" debugStub.linuxOffsets = "0xa356145,0xffffffff,0xfc052084,0xffffffff,0x0,0x0,0xa356150,0x0,0xa35616f,0x0,0xfc052088,0xffffffff,0x0,0x0" tools.remindInstall = "TRUE" sched.cpu.affinity = "all" sched.swap.hostLocal = "disabled" # Now we'll copy this template to the other image directories cp Centos5-31.vmx Centos5-31/Centos5-31.vmx cp Centos5-31.vmx Centos5-32/Centos5-32.vmx cp Centos5-31.vmx Centos5-33/Centos5-33.vmx cp Centos5-31.vmx Centos5-34/Centos5-34.vmx cp Centos5-31.vmx Centos5-35/Centos5-35.vmx cp Centos5-31.vmx Centos5-36/Centos5-36.vmx cp Centos5-31.vmx Centos5-37/Centos5-37.vmx cp Centos5-31.vmx Centos5-38/Centos5-38.vmx cp Centos5-31.vmx Centos5-39/Centos5-39.vmx cp Centos5-31.vmx Centos5-40/Centos5-40.vmx cp Centos5-31.vmx Centos5-41/Centos5-41.vmx cp Centos5-31.vmx Centos5-42/Centos5-42.vmx cp Centos5-31.vmx Centos5-43/Centos5-43.vmx cp Centos5-31.vmx Centos5-44/Centos5-44.vmx cp Centos5-31.vmx Centos5-45/Centos5-45.vmx cp Centos5-31.vmx Centos5-46/Centos5-46.vmx cp Centos5-31.vmx Centos5-47/Centos5-47.vmx cp Centos5-31.vmx Centos5-48/Centos5-48.vmx cp Centos5-31.vmx Centos5-49/Centos5-49.vmx # You'll want to swap the references of the initial image name [ Centos5-30 ] to be the directories image name vi Centos5-31/Centos5-31.vmx vi Centos5-32/Centos5-32.vmx vi Centos5-33/Centos5-33.vmx vi Centos5-34/Centos5-34.vmx vi Centos5-35/Centos5-35.vmx vi Centos5-36/Centos5-36.vmx vi Centos5-37/Centos5-37.vmx vi Centos5-38/Centos5-38.vmx vi Centos5-39/Centos5-39.vmx vi Centos5-40/Centos5-40.vmx vi Centos5-41/Centos5-41.vmx vi Centos5-42/Centos5-42.vmx vi Centos5-43/Centos5-43.vmx vi Centos5-43/Centos5-43.vmx vi Centos5-44/Centos5-44.vmx vi Centos5-45/Centos5-45.vmx vi Centos5-46/Centos5-46.vmx vi Centos5-47/Centos5-47.vmx vi Centos5-48/Centos5-48.vmx vi Centos5-49/Centos5-49.vmx # Now we can register those new images and they'll show up in your vSphere Client vmware-cmd Centos5-31/Centos5-31.vmx register vmware-cmd Centos5-32/Centos5-32.vmx register vmware-cmd Centos5-33/Centos5-33.vmx register vmware-cmd Centos5-34/Centos5-34.vmx register vmware-cmd Centos5-35/Centos5-35.vmx register vmware-cmd Centos5-36/Centos5-36.vmx register vmware-cmd Centos5-37/Centos5-37.vmx register vmware-cmd Centos5-38/Centos5-38.vmx register vmware-cmd Centos5-39/Centos5-39.vmx register vmware-cmd Centos5-40/Centos5-40.vmx register vmware-cmd Centos5-41/Centos5-41.vmx register vmware-cmd Centos5-42/Centos5-42.vmx register vmware-cmd Centos5-43/Centos5-43.vmx register vmware-cmd Centos5-44/Centos5-44.vmx register vmware-cmd Centos5-45/Centos5-45.vmx register vmware-cmd Centos5-46/Centos5-46.vmx register vmware-cmd Centos5-47/Centos5-47.vmx register vmware-cmd Centos5-48/Centos5-48.vmx register vmware-cmd Centos5-49/Centos5-49.vmx register # Confirm they all are registered vmware-cmd -l | sort /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-30/Centos5-30.vmx /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-31/Centos5-31.vmx /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-32/Centos5-32.vmx /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-33/Centos5-33.vmx /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-34/Centos5-34.vmx /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-35/Centos5-35.vmx /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-36/Centos5-36.vmx /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-37/Centos5-37.vmx /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-38/Centos5-38.vmx /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-39/Centos5-39.vmx /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-40/Centos5-40.vmx /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-41/Centos5-41.vmx /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-42/Centos5-42.vmx /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-43/Centos5-43.vmx /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-44/Centos5-44.vmx /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-45/Centos5-45.vmx /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-46/Centos5-46.vmx /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-47/Centos5-47.vmx /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-48/Centos5-48.vmx /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-49/Centos5-49.vmx # You'll see your images now in your vSphere Client. # When you first start them, do a console view of it because you'll see in that view # you need to select an option before it will start up. # Choose : "I copied it" # Once image is running, login as root and within a shell confirm your network interface has # an unique IP address as well as a unique MAC address for the interface # You then might want to suspend the image at this point.
Manual Cloning of x64 Setup
If you can to x64 with your VMware, here's the additions steps for that [cut-n-paste]
Make a new image for x64 rhel5 called Centos5-x64-50 and once it's all done, shutdown and halt it.
Short and sweet update for ESXi 5.5
[ Working Dir is /vmfs/volumes/datastore1 ] # ls -la /vmfs/volumes/ | grep data lrwxr-xr-x 1 root root 35 Aug 24 21:35 datastore1 -> 53f864a6-1d3302c1-1b97-00101849e4af # mkdir linuxsrv # vmkfstools -i Centos\ 6.5\ Base/Centos\ 6.5\ Base.vmdk linuxsrv/linuxsrv.vmdk # cp Centos\ 6.5\ Base/Centos\ 6.5\ Base.vmx linuxsrv/linuxsrv.vmx # vi linuxsrv/linuxsrv.vmx ESC Key :%s/Centos 6.5 Base/linuxsrv/g ESC Key # make the following variables blank , ethernet0.generatedAddress = "00:0c:29:44:16:8d" ethernet0.generatedAddress = "" :wq! # Note - if you have snapshots against your source image, you might need to adjust this var # scsi0:0.fileName = "850-Wapp2-0000104.vmdk" , for example. The numbers after the hostname get removed. # # vim-cmd solo/registervm /vmfs/volumes/datastore1/linuxsrv/linuxsrv.vmx 10
For ESX 4
pwd /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af mkdir Centos5-x64-{51..69} vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-51/Centos5-x64-51.vmdk vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-52/Centos5-x64-52.vmdk vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-53/Centos5-x64-53.vmdk vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-54/Centos5-x64-54.vmdk vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-55/Centos5-x64-55.vmdk vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-56/Centos5-x64-56.vmdk vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-57/Centos5-x64-57.vmdk vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-58/Centos5-x64-58.vmdk vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-59/Centos5-x64-59.vmdk vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-60/Centos5-x64-60.vmdk vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-61/Centos5-x64-61.vmdk vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-62/Centos5-x64-62.vmdk vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-63/Centos5-x64-63.vmdk vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-64/Centos5-x64-64.vmdk vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-65/Centos5-x64-65.vmdk vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-66/Centos5-x64-66.vmdk vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-67/Centos5-x64-67.vmdk vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-68/Centos5-x64-68.vmdk vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-69/Centos5-x64-69.vmdk # Setup the vmx file like a template like you did for the 32bit ones and then copy to the other x64 directories. cp Centos5-x64-50/Centos5-x64-50.vmx ./ cp Centos5-x64-50/Centos5-x64-50.vmx ./Centos5-x64-60.vmx # make this like be empty quoted - ethernet0.generatedAddress = "" vi Centos5-x64-50.vmx # Edit this for the ethernet and change to 50 to 60 -- make for editing easier for the 60 dir's vi Centos5-x64-60.vmx # copy to other directories. cp Centos5-x64-50.vmx Centos5-x64-51/Centos5-x64-51.vmx cp Centos5-x64-50.vmx Centos5-x64-52/Centos5-x64-52.vmx cp Centos5-x64-50.vmx Centos5-x64-53/Centos5-x64-53.vmx cp Centos5-x64-50.vmx Centos5-x64-54/Centos5-x64-54.vmx cp Centos5-x64-50.vmx Centos5-x64-55/Centos5-x64-55.vmx cp Centos5-x64-50.vmx Centos5-x64-56/Centos5-x64-56.vmx cp Centos5-x64-50.vmx Centos5-x64-57/Centos5-x64-57.vmx cp Centos5-x64-50.vmx Centos5-x64-58/Centos5-x64-58.vmx cp Centos5-x64-50.vmx Centos5-x64-59/Centos5-x64-59.vmx cp Centos5-x64-60.vmx Centos5-x64-60/Centos5-x64-60.vmx cp Centos5-x64-60.vmx Centos5-x64-61/Centos5-x64-61.vmx cp Centos5-x64-60.vmx Centos5-x64-62/Centos5-x64-62.vmx cp Centos5-x64-60.vmx Centos5-x64-63/Centos5-x64-63.vmx cp Centos5-x64-60.vmx Centos5-x64-64/Centos5-x64-64.vmx cp Centos5-x64-60.vmx Centos5-x64-65/Centos5-x64-65.vmx cp Centos5-x64-60.vmx Centos5-x64-66/Centos5-x64-66.vmx cp Centos5-x64-60.vmx Centos5-x64-67/Centos5-x64-67.vmx cp Centos5-x64-60.vmx Centos5-x64-68/Centos5-x64-68.vmx cp Centos5-x64-60.vmx Centos5-x64-69/Centos5-x64-69.vmx # Now edit them, replacing the 50 in the hostname reference to match the ## of the directory. vi Centos5-x64-51/Centos5-x64-51.vmx vi Centos5-x64-52/Centos5-x64-52.vmx vi Centos5-x64-53/Centos5-x64-53.vmx vi Centos5-x64-54/Centos5-x64-54.vmx vi Centos5-x64-55/Centos5-x64-55.vmx vi Centos5-x64-56/Centos5-x64-56.vmx vi Centos5-x64-57/Centos5-x64-57.vmx vi Centos5-x64-58/Centos5-x64-58.vmx vi Centos5-x64-59/Centos5-x64-59.vmx # Now edit them, replacing the 60 in the hostname reference to match the ## of the directory. vi Centos5-x64-61/Centos5-x64-61.vmx vi Centos5-x64-62/Centos5-x64-62.vmx vi Centos5-x64-63/Centos5-x64-63.vmx vi Centos5-x64-64/Centos5-x64-64.vmx vi Centos5-x64-65/Centos5-x64-65.vmx vi Centos5-x64-66/Centos5-x64-66.vmx vi Centos5-x64-67/Centos5-x64-67.vmx vi Centos5-x64-68/Centos5-x64-68.vmx vi Centos5-x64-69/Centos5-x64-69.vmx # Register the x64 images vmware-cmd Centos5-x64-51/Centos5-x64-51.vmx register vmware-cmd Centos5-x64-52/Centos5-x64-52.vmx register vmware-cmd Centos5-x64-53/Centos5-x64-53.vmx register vmware-cmd Centos5-x64-54/Centos5-x64-54.vmx register vmware-cmd Centos5-x64-55/Centos5-x64-55.vmx register vmware-cmd Centos5-x64-56/Centos5-x64-56.vmx register vmware-cmd Centos5-x64-57/Centos5-x64-57.vmx register vmware-cmd Centos5-x64-58/Centos5-x64-58.vmx register vmware-cmd Centos5-x64-59/Centos5-x64-59.vmx register vmware-cmd Centos5-x64-60/Centos5-x64-60.vmx register vmware-cmd Centos5-x64-61/Centos5-x64-61.vmx register vmware-cmd Centos5-x64-62/Centos5-x64-62.vmx register vmware-cmd Centos5-x64-63/Centos5-x64-63.vmx register vmware-cmd Centos5-x64-64/Centos5-x64-64.vmx register vmware-cmd Centos5-x64-65/Centos5-x64-65.vmx register vmware-cmd Centos5-x64-66/Centos5-x64-66.vmx register vmware-cmd Centos5-x64-67/Centos5-x64-67.vmx register vmware-cmd Centos5-x64-68/Centos5-x64-68.vmx register vmware-cmd Centos5-x64-69/Centos5-x64-69.vmx register # You'll see your images now in your vSphere Client. # When you first start them, do a console view of it because you'll see in that view # you need to select an option before it will start up. # Choose : "I copied it" # Once image is running, login as root and within a shell confirm your network interface has # an unique IP address as well as a unique MAC address for the interface # You then might want to suspend the image at this point.
Other Useful Steps To Get Zimbra Installed And Running
RHEL or CentOS
chkconfig sendmail off /etc/init.d/sendmail stop
Some prereq packages that you might not already have installed:
yum install compat-libstdc++-33 compat-libstdc++-296 sysstat
Adding Additional Storage
Adding additional storage for Zimbra installation requirement
- Add new virtual disk - requires at least 5GB's of free space on available partition
- Power Off VM And then create a new disk giving it at least 5GB's of space
- Power On the VM
- You can then run this to see/confirm the new "disk"
fdisk -l
- In my example here, my new disk is /dev/sdb
- Partition the new disk
fdisk /dev/sdb
- Select "n" for new partition
- Select "p" for primary partition
- Select "1" for partition number
- Select default of "1" for first cylinder
- Select default, which should be the highest number given in range. This will change based upon the size of the virtual disk you made.
- You new disk is now partition.
- Hit "p" to print out the partition table to confirm.
- Hit "w" to write table to disk and exit fdisk.
- Create new filesystem for the new disk/partition
- This example uses ext3 and the example partition path of /dev/sdb1
mkfs.ext3 /dev/sdb1
- Setup /etc/fstab to mount the new partition for zimbra use
mkdir /opt/zimbra
vi /etc/fstab
- And now add a line like the following:
/dev/sdb1 /opt/zimbra ext3 defaults 1 1
- Partition the new disk
Moving Zimbra To New Partitions For zimbra and backups
Adding additional storage for Zimbra installation requirement
- Disable zimbra from starting via init.d - how this is done depends on your distro.
- Power Off VM And then create two new virtual disks. Spacing is based upon your current usage and what you expect you'll need going forward. Recommend the disk is allocated now vs allowing it to dynamically grow.
- Power On the VM
- Confirm zimbra isn't running, stop it if it is running:
su - zimbra ; zmcontrol stop
- You should now confirm the new "disks" exist
fdisk -l
- In my example here, my new disks are /dev/sdb and /dev/sdc
- Partition the new sdb disk
fdisk /dev/sdb
- Select "n" for new partition
- Select "p" for primary partition
- Select "1" for partition number
- Select default of "1" for first cylinder
- Select default, which should be the highest number given in range. This will change based upon the size of the virtual disk you made.
- You new disk is now partition.
- Hit "p" to print out the partition table to confirm.
- Hit "w" to write table to disk and exit fdisk.
- Partition the new sdc disk
fdisk /dev/sdc
- Select "n" for new partition
- Select "p" for primary partition
- Select "1" for partition number
- Select default of "1" for first cylinder
- Select default, which should be the highest number given in range. This will change based upon the size of the virtual disk you made.
- You new disk is now partition.
- Hit "p" to print out the partition table to confirm.
- Hit "w" to write table to disk and exit fdisk.
- Create new filesystem for the new disks/partitions
- This example uses ext3 and the example partition path of /dev/sdb1 and /dev/sdc1
mkfs.ext3 /dev/sdb1
mkfs.ext3 /dev/sdc1
- Move zimbra data to a temporary location
- as root
mv /opt/zimbra /opt/zimbra_old
- Setup /etc/fstab to mount the new partition for zimbra use
vi /etc/fstab
- And now add a line like the following, they must be below the entry for your / partition:
/dev/sdb1 /opt/zimbra ext3 defaults 1 1
/dev/sdc1 /opt/zimbra/backup ext3 defaults 1 1
- Save the file.
- We first need to get /opt/zimbra mounted before the backup partition.
mkdir /opt/zimbra
mount /opt/zimbra
- Now we can do the backup one.
mkdir /opt/zimbra/backup
chown zimbra:zimbra /opt/zimbra/backup
mount /opt/zimbra/backup
- Now we sync the data over onto our new partitions.
rsync -avzHS --progress /opt/zimbra_old/ /opt/zimbra
- Once you confirm your rsync is done and is correct, you could remove the old data.
rm -rf /opt/zimbra_old
- You should now be able to start zimbra.
su - zimbra ; zmcontrol start
- Remember to re-enable zimbra to start from init.d now also if you disabled it at the beginning.
- Partition the new sdb disk
Final Look
Comment About Hyper-threading And CPU Performance
TheE5520 has 4 cores, which do show up in vSphere as 4 available proc's but the cpuinfo just shows the one. Difference comes when Intel VT [vmx flag in cpuinfo] or AMD-V [svm flag in cpuinfo] is enabled while hyperthreading [ht flag in cpuinfo] is turned off. Turning on HT in the bios I think would show 8 proc in vm and report 4 cores in the cpuinfo output from base OS [don't want to reboot and reconfigure at the moment to double check]. Note, cpuinfo will show HT if the cpu's support it - not just if it's enabled. You still need to enable it and confirm APCI is enabled in the BIOS if your not seeing the 'virtual cpu's'. VT is about seeing the physical cores and individual cpu's. HT is about virtualizing each cpu or core into 2 virtual cpu's.
Spec
ESX Server
[root@vmware-server ~]# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 26 model name : Intel(R) Xeon(R) CPU E5520 @ 2.27GHz stepping : 5 cpu MHz : 2266.688 cache size : 8192 KB fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm bogomips : 4535.95 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: [8] [root@vmware-server ~]# cat /proc/meminfo MemTotal: 356684 kB MemFree: 30392 kB Buffers: 9068 kB Cached: 127804 kB SwapCached: 14180 kB Active: 264444 kB Inactive: 41320 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 356684 kB LowFree: 30392 kB SwapTotal: 730916 kB SwapFree: 671960 kB Dirty: 208 kB Writeback: 0 kB AnonPages: 168356 kB Mapped: 36188 kB Slab: 12448 kB PageTables: 3160 kB NFS_Unstable: 0 kB Bounce: 0 kB CommitLimit: 909256 kB Committed_AS: 533636 kB VmallocTotal: 34359738367 kB VmallocUsed: 25036 kB VmallocChunk: 34359705099 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 Hugepagesize: 2048 kB MachineMem: 12580415 kB [root@vmware-server ~]# uname -a Linux vmware-server.zimbra.homunix.com 2.6.18-128.ESX #1 Thu Oct 15 16:11:16 PDT 2009 x86_64 x86_64 x86_64 GNU/Linux [root@vmware-server ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 5.1 (Tikanga) [root@vmware-server ~]# fdisk -l Disk /dev/sda: 750.1 GB, 750156374016 bytes 255 heads, 63 sectors/track, 91201 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 * 1 140 1124518+ 83 Linux /dev/sda2 141 154 112455 fc VMware VMKCORE /dev/sda3 155 91201 731335027+ 5 Extended /dev/sda5 155 91201 731334996 fb VMware VMFS Disk /dev/sdb: 8095 MB, 8095006720 bytes 255 heads, 63 sectors/track, 984 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdb1 1 91 730926 82 Linux swap / Solaris /dev/sdb2 92 346 2048287+ 83 Linux /dev/sdb3 347 984 5124735 5 Extended /dev/sdb5 347 984 5124703+ 83 Linux
I've setup the vm's to have 1 cpu and 1024MB RAM. I have 9 running right now. Here's one of my 64bit image details:
[root@mail59 ~]# cat /proc/meminfo MemTotal: 1026932 kB MemFree: 85816 kB Buffers: 14884 kB Cached: 87132 kB SwapCached: 117444 kB Active: 705916 kB Inactive: 143664 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 1026932 kB LowFree: 85816 kB SwapTotal: 2064376 kB SwapFree: 1453736 kB Dirty: 644 kB Writeback: 0 kB AnonPages: 734740 kB Mapped: 34284 kB Slab: 38884 kB PageTables: 33304 kB NFS_Unstable: 0 kB Bounce: 0 kB CommitLimit: 2577840 kB Committed_AS: 3257840 kB VmallocTotal: 34359738367 kB VmallocUsed: 263932 kB VmallocChunk: 34359473927 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 Hugepagesize: 2048 kB [root@mail59 ~]# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 26 model name : Intel(R) Xeon(R) CPU E5520 @ 2.27GHz stepping : 5 cpu MHz : 2266.631 cache size : 8192 KB fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc up ida nonstop_tsc pni cx16 popcnt lahf_lm bogomips : 4533.26 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: [8] [root@mail59 ~]# uname -a Linux mail59.zimbra.homeunix.com 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux [root@mail59 ~]# cat /etc/redhat-release CentOS release 5.4 (Final) [root@mail59 ~]# fdisk -l Disk /dev/sda: 10.7 GB, 10737418240 bytes 255 heads, 63 sectors/track, 1305 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 * 1 13 104391 83 Linux /dev/sda2 14 1305 10377990 8e Linux LVM Disk /dev/sdb: 8589 MB, 8589934592 bytes 255 heads, 63 sectors/track, 1044 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdb1 1 1044 8385898+ 83 Linux [root@mail59 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 7.7G 3.5G 3.9G 48% / /dev/sda1 99M 13M 82M 14% /boot tmpfs 502M 0 502M 0% /dev/shm /dev/hdc 6.8G 6.8G 0 100% /media/zcs-x64-603-06 /dev/sdb1 7.9G 1.8G 5.8G 24% /opt/zimbra
Here's my esxtop output, which is ran on the base os of your ESX server. You'll see the running vm's I mentioned.
[root@vmware-server ~]# esxtop 9:20:07pm up 32 days 20:43, 139 worlds; CPU load average: 0.14, 0.15, 0.16 PCPU USED(%): 7.7 7.3 6.8 6.8 AVG: 7.1 PCPU UTIL(%): 10.6 10.4 10.0 9.9 AVG: 10.2 CCPU(%): 0 us, 2 sy, 98 id, 0 wa ; cs/sec: 229 ID GID NAME NWLD %USED %RUN %SYS %WAIT %RDY %IDLE %OVRLP %CSTP %MLMTD %SWPWT 1 1 idle 4 367.57 370.04 0.00 0.00 34.02 0.00 0.00 0.00 0.00 0.00 2 2 system 6 0.01 0.01 0.00 600.00 0.00 0.00 0.00 0.00 0.00 0.00 6 6 helper 58 0.01 0.01 0.00 5800.00 0.00 0.00 0.00 0.00 0.00 0.00 7 7 drivers 9 0.00 0.00 0.00 900.00 0.00 0.00 0.00 0.00 0.00 0.00 8 8 vmotion 4 0.00 0.00 0.00 400.00 0.00 0.00 0.00 0.00 0.00 0.00 10 10 console 2 1.93 1.89 0.01 200.00 0.08 99.04 0.03 0.00 0.00 0.00 15 15 vmkapimod 9 0.00 0.00 0.00 900.00 0.00 0.00 0.00 0.00 0.00 0.00 17 17 FT 1 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 0.00 0.00 18 18 vobd.4231 8 0.00 0.00 0.00 800.00 0.00 0.00 0.00 0.00 0.00 0.00 19 19 net-cdp.4239 1 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 0.00 0.00 20 20 vmware-vmkauthd 1 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 0.00 0.00 114 114 Centos5-x64-59 4 4.43 5.17 0.01 398.60 0.29 95.33 0.29 0.00 0.00 0.00 125 125 Centos5-x64-50 4 2.64 3.49 0.00 400.00 0.30 97.30 0.27 0.00 0.00 0.00 126 126 Centos5-x64-51 4 2.38 3.13 0.00 400.00 0.31 97.63 0.28 0.00 0.00 0.00 127 127 Centos5-x64-52 4 2.70 3.54 0.00 400.00 0.31 97.21 0.28 0.00 0.00 0.00 128 128 Centos5-x64-53 4 2.65 3.50 0.00 400.00 0.32 97.25 0.28 0.00 0.00 0.00 139 139 Centos5-x64-54 4 2.36 3.12 0.00 400.00 0.29 97.63 0.27 0.00 0.00 0.00 140 140 Centos5-x64-55 4 2.36 3.13 0.00 400.00 0.28 97.65 0.29 0.00 0.00 0.00 141 141 Centos5-x64-56 4 2.37 3.11 0.00 400.00 0.29 97.66 0.27 0.00 0.00 0.00 142 142 Centos5-x64-57 4 2.44 3.21 0.00 400.00 0.29 97.57 0.29 0.00 0.00 0.00
Screenshot
Optional Setups To Enhance Your Test Environment
Using RINETD To Redirect To Your Various VM For External HTTP Access
RINETD Setup
rinetd redirects TCP connections from one IP address and port to another. rinetd is a single-process server which handles any number of connections to the address/port pairs specified in the file /etc/rinetd.conf.
Here's an example /etc/rinetd.conf on an internal server [192.168.0.16] that will redirect to the various vm's to port 80. The 'firewall', a cheap one like I have, only allows me to do very basic port redirection to an ip address -- NOT an internal ip address AND a different port. So my cheap firewall is setup to redirect port 80## to the rinetd server - 192.168.0.16 and then that server redirects to the vm and the right port - 80.
If your using a DynDNS service this works out very well. As I can now use the DynDNS domain name and simply add a port number to get redirected to the various ZCS servers. For example:
- http://EXTERNAL-DOMAIN:8059 would get redirected to my vm server that is using 192.168.0.59 and to port 80 for the http ZWC login page.
Here's the /etc/rinetd.conf I have setup:
192.168.0.16 8030 192.168.0.30 80 192.168.0.16 8031 192.168.0.31 80 192.168.0.16 8032 192.168.0.32 80 192.168.0.16 8033 192.168.0.33 80 192.168.0.16 8034 192.168.0.34 80 192.168.0.16 8035 192.168.0.35 80 192.168.0.16 8036 192.168.0.36 80 192.168.0.16 8037 192.168.0.37 80 192.168.0.16 8038 192.168.0.38 80 192.168.0.16 8039 192.168.0.39 80 192.168.0.16 8040 192.168.0.40 80 192.168.0.16 8041 192.168.0.41 80 192.168.0.16 8042 192.168.0.42 80 192.168.0.16 8043 192.168.0.43 80 192.168.0.16 8044 192.168.0.44 80 192.168.0.16 8045 192.168.0.45 80 192.168.0.16 8046 192.168.0.46 80 192.168.0.16 8047 192.168.0.47 80 192.168.0.16 8048 192.168.0.48 80 192.168.0.16 8049 192.168.0.49 80 192.168.0.16 8050 192.168.0.50 80 192.168.0.16 8051 192.168.0.51 80 192.168.0.16 8052 192.168.0.52 80 192.168.0.16 8053 192.168.0.53 80 192.168.0.16 8054 192.168.0.54 80 192.168.0.16 8055 192.168.0.55 80 192.168.0.16 8056 192.168.0.56 80 192.168.0.16 8057 192.168.0.57 80 192.168.0.16 8058 192.168.0.58 80 192.168.0.16 8059 192.168.0.59 80 192.168.0.16 8060 192.168.0.60 80 192.168.0.16 8061 192.168.0.61 80 192.168.0.16 8062 192.168.0.62 80 192.168.0.16 8063 192.168.0.63 80 192.168.0.16 8064 192.168.0.64 80 192.168.0.16 8065 192.168.0.65 80 192.168.0.16 8066 192.168.0.66 80 192.168.0.16 8067 192.168.0.67 80 192.168.0.16 8068 192.168.0.68 80 192.168.0.16 8069 192.168.0.69 80 logcommon logfile /var/log/rinetd.log
Setting Up A Mailhub
Continuing with the example of using 192.168.0.16 with a hostname of mail3.[SUB].[DOMAIN].com as my mail hub which the firewall routes all smtp traffic to and also is running rinetd. This assumes you've setup DNS like I outlined in Ajcody-Virtualization-Named-DNS which I mentioned at the top of this wiki page.
Install postfix [centos5 example]
yum install postfix rpm -q postfix postfix-2.3.3-2.1.el5_2
Configure /etc/postfix/main.cf
/etc/init.d/postfix stop cd /etc/postfix/ mv main.cf main.cf-backup
Now paste in the contents:
## paste the below into file and adjust for your setup the following variables ## myhostname & mydomain with my example of mail3.[SUB].[DOMAIN].com ## might need to also adjust : mynetworks = 192.168.0.0/24 ## You might want to compare our default main.cf below and adjust for different paths, etc ## that your distro or postfix version might be setup to use ## and then save -- :wq! vi main.cf queue_directory = /var/spool/postfix command_directory = /usr/sbin daemon_directory = /usr/libexec/postfix mail_owner = postfix myhostname = mail3.[SUB].[DOMAIN].com mydomain = mail3.[SUB].[DOMAIN].com myorigin = $mydomain mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain mynetworks = 192.168.0.0/24, 127.0.0.0/8 relay_domains = $mydestination, $mynetworks, hash:/etc/postfix/relay-domains transport_maps = hash:/etc/postfix/transport local_transport = local alias_maps = hash:/etc/aliases alias_database = hash:/etc/aliases debug_peer_level = 2 debugger_command = PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin xxgdb $daemon_directory/$process_name $process_id & sleep 5 sendmail_path = /usr/sbin/sendmail.postfix newaliases_path = /usr/bin/newaliases.postfix mailq_path = /usr/bin/mailq.postfix setgid_group = postdrop html_directory = no manpage_directory = /usr/share/man sample_directory = /usr/share/doc/postfix-2.3.3/samples readme_directory = /usr/share/doc/postfix-2.3.3/README_FILES
You might need to make the aliases db. Path is referenced in the main.cf above. This command below will end up creating /etc/aliases.db
postmap /etc/aliases
Now we'll setup the relay-domains file. Adjust my example for your domain using substitution in vi [ ESC key and then :%s/SUB.DOMAIN/With Your Domain Details/g hit return key].
vi /etc/postfix/relay-domains mail30.SUB.DOMAIN.com OK mail31.SUB.DOMAIN.com OK mail32.SUB.DOMAIN.com OK mail33.SUB.DOMAIN.com OK mail34.SUB.DOMAIN.com OK mail35.SUB.DOMAIN.com OK mail36.SUB.DOMAIN.com OK mail37.SUB.DOMAIN.com OK mail38.SUB.DOMAIN.com OK mail39.SUB.DOMAIN.com OK mail40.SUB.DOMAIN.com OK mail41.SUB.DOMAIN.com OK mail42.SUB.DOMAIN.com OK mail43.SUB.DOMAIN.com OK mail44.SUB.DOMAIN.com OK mail45.SUB.DOMAIN.com OK mail46.SUB.DOMAIN.com OK mail47.SUB.DOMAIN.com OK mail48.SUB.DOMAIN.com OK mail49.SUB.DOMAIN.com OK mail50.SUB.DOMAIN.com OK mail51.SUB.DOMAIN.com OK mail52.SUB.DOMAIN.com OK mail53.SUB.DOMAIN.com OK mail54.SUB.DOMAIN.com OK mail55.SUB.DOMAIN.com OK mail56.SUB.DOMAIN.com OK mail57.SUB.DOMAIN.com OK mail58.SUB.DOMAIN.com OK mail59.SUB.DOMAIN.com OK mail61.SUB.DOMAIN.com OK mail62.SUB.DOMAIN.com OK mail63.SUB.DOMAIN.com OK mail64.SUB.DOMAIN.com OK mail65.SUB.DOMAIN.com OK mail66.SUB.DOMAIN.com OK mail67.SUB.DOMAIN.com OK mail68.SUB.DOMAIN.com OK mail69.SUB.DOMAIN.com OK
Now build the db, the command will end up creating /etc/postfix/relay-domains.db
postmap /etc/postfix/relay-domains
Now we'll create a transport map for each domain. Adjust my example for your domain using substitution in vi [ ESC key and then :%s/SUB.DOMAIN/With Your Domain Details/g hit return key].
vi /etc/postfix/transport mail30.SUB.DOMAIN.com smtp:mail30.SUB.DOMAIN.com mail31.SUB.DOMAIN.com smtp:mail31.SUB.DOMAIN.com mail32.SUB.DOMAIN.com smtp:mail32.SUB.DOMAIN.com mail33.SUB.DOMAIN.com smtp:mail33.SUB.DOMAIN.com mail34.SUB.DOMAIN.com smtp:mail34.SUB.DOMAIN.com mail35.SUB.DOMAIN.com smtp:mail35.SUB.DOMAIN.com mail36.SUB.DOMAIN.com smtp:mail36.SUB.DOMAIN.com mail37.SUB.DOMAIN.com smtp:mail37.SUB.DOMAIN.com mail38.SUB.DOMAIN.com smtp:mail38.SUB.DOMAIN.com mail39.SUB.DOMAIN.com smtp:mail39.SUB.DOMAIN.com mail40.SUB.DOMAIN.com smtp:mail40.SUB.DOMAIN.com mail41.SUB.DOMAIN.com smtp:mail41.SUB.DOMAIN.com mail42.SUB.DOMAIN.com smtp:mail42.SUB.DOMAIN.com mail43.SUB.DOMAIN.com smtp:mail43.SUB.DOMAIN.com mail44.SUB.DOMAIN.com smtp:mail44.SUB.DOMAIN.com mail45.SUB.DOMAIN.com smtp:mail45.SUB.DOMAIN.com mail46.SUB.DOMAIN.com smtp:mail46.SUB.DOMAIN.com mail47.SUB.DOMAIN.com smtp:mail47.SUB.DOMAIN.com mail48.SUB.DOMAIN.com smtp:mail48.SUB.DOMAIN.com mail49.SUB.DOMAIN.com smtp:mail49.SUB.DOMAIN.com mail50.SUB.DOMAIN.com smtp:mail50.SUB.DOMAIN.com mail51.SUB.DOMAIN.com smtp:mail51.SUB.DOMAIN.com mail52.SUB.DOMAIN.com smtp:mail52.SUB.DOMAIN.com mail53.SUB.DOMAIN.com smtp:mail53.SUB.DOMAIN.com mail54.SUB.DOMAIN.com smtp:mail54.SUB.DOMAIN.com mail55.SUB.DOMAIN.com smtp:mail55.SUB.DOMAIN.com mail56.SUB.DOMAIN.com smtp:mail56.SUB.DOMAIN.com mail57.SUB.DOMAIN.com smtp:mail57.SUB.DOMAIN.com mail58.SUB.DOMAIN.com smtp:mail58.SUB.DOMAIN.com mail59.SUB.DOMAIN.com smtp:mail59.SUB.DOMAIN.com mail60.SUB.DOMAIN.com smtp:mail60.SUB.DOMAIN.com mail61.SUB.DOMAIN.com smtp:mail61.SUB.DOMAIN.com mail62.SUB.DOMAIN.com smtp:mail62.SUB.DOMAIN.com mail63.SUB.DOMAIN.com smtp:mail63.SUB.DOMAIN.com mail64.SUB.DOMAIN.com smtp:mail64.SUB.DOMAIN.com mail65.SUB.DOMAIN.com smtp:mail65.SUB.DOMAIN.com mail66.SUB.DOMAIN.com smtp:mail66.SUB.DOMAIN.com mail67.SUB.DOMAIN.com smtp:mail67.SUB.DOMAIN.com mail68.SUB.DOMAIN.com smtp:mail68.SUB.DOMAIN.com mail69.SUB.DOMAIN.com smtp:mail69.SUB.DOMAIN.com
Now build the db, the command will end up creating /etc/postfix/transport.db
postmap /etc/postfix/transport
We should be all set now. Start postfix and monitor the log file as you do your testing.
/etc/init.d/postfix start tail -f /var/log/maillog
Configuring Your ZCS VM's To Use The Mailhub
Note, another 'trick' you might need to do on your vm's to get them working in this type of situation from the outside is to setup one server [I used my 192.168.0.16 machine] to redirect and send emails from the other zcs servers. This is assuming your using a DynDNS setup where you can wildcard for a base domain and also wildcard it for MX requests as well. This is also necessary when your firewall device can only route all port 25 requests to a single ip address internally to your server. This setup assumes that server is running a mta that can then do MX lookups internally for the domainname being used for routing information. The adjustments below setup the ZCS vm's to accept the mail relay from that internal server.
Adjust in the Configuration > Global Settings > MTA tab and under Configuration > Servers > Your server > MTA tab on the other ZCS servers to use the one server for:
- Web mail MTA Port 25
- Relay MTA for external delivery
- Inbound SMTP Host name
- zmcontrol stop and then zmcontrol start on the server/s
Ajcody Backup & Restore Issues
Actual Backup & Restore Issues Homepage
Please see Ajcody-Backup-Restore-Issues
Update - May, 2017
I'll be updating this page after the ZCS 8.8 release is done to only reflect what is true in that version and future versions. It will also include information about the Zimbra_Suite_Plus modules where appropriate. If you currently are using the backup module for ZSP, please see Zimbra_Suite_Plus/Zimbra_Backup_Plus for now.
Backup-Restore Training Material Rough Drafts I Wrote
- Ajcody-Troubleshooting-Recover_Missing_Data_-_Server
- Ajcody-Troubleshooting-Recover_Missing_Data_-_User
Bug/RFE's I Filed Against ZCS 8.6
- [story] Ability to search data within backup and do "item" restores or identify locations of search results
- admin console backup label view doesn't list accounts in the all accounts tab
- admin console restore - doesn't autocomplete / suggest account matches when filling out email address box
- document new restore functions / options with ZCS 8+ for admin console restore
- admin console restore - rename "Selected Servers" panel to "Restore Options"
- admin console restore - if only one mailstore in env. then state such in second panel of restore about "server for the restored accounts"
- admin console restore - expand restore To options - To full backup label, To incremental target
- admin console restore - "restore to the latest backup" incorrectly described / broken
- admin console restore - unable to restore individual accounts [sort of]
- admin console restore - reuse GAL/Contact Picker Window for "restore individual accounts"
Miscellaneous Bugs & RFE's
For those that address other specific issues described on this page:
- "Separate Restore Mechanism for Primary/Secondary Volumes" [Helix]
- http://bugzilla.zimbra.com/show_bug.cgi?id=18566
- "add backup/restore for HSM only" [marked as duplicate of above]
- http://bugzilla.zimbra.com/show_bug.cgi?id=18566
RFE For Live DR Restore Option
- zmrestore / zmrestoreoffline have options to --skipDeleteMailboxes - Provides Live DR option
Backups And Upgrades And Prior Versions
- Upgrades And Compatibility Of Older Backups
- "Backups must be compatible across patch releases"
- "Incorrect upgrade documentation regarding backups"
- "support for restore across major versions"
- "Add conversion tool to upgrade backup versions to allow restore on later zcs versions"
- From 5.0.7 - 5.0.10 You Might See minor version upgrades moving your backups into a subdirectory
- "upgrade incorrectly invalidates backups."
Information To Provide When Submitting A Support Case For Backup Issues
Basic Backup Information To Submit To Support
Disk Space Usage Issues
Trend Data
If there is concerns about disks/partitions getting full, this command would be helpful for trending data on your server. Send support the resulting df.tar file . Note - adjust the tail command if you want more than 20 day's worth of trending data, the -n 20 option.
[zimbra@zcs806 tmp]$ /tmp [zimbra@zcs806 tmp]$ tar cvf /tmp/df.tar `find /opt/zimbra/zmstat -name df.cs\* | sort | tail -n 20` tar: Removing leading `/' from member names /opt/zimbra/zmstat/2014-03-10/df.csv.gz /opt/zimbra/zmstat/2014-03-11/df.csv.gz [cut - Ajcody] /opt/zimbra/zmstat/2014-03-27/df.csv.gz /opt/zimbra/zmstat/2014-03-28/df.csv.gz /opt/zimbra/zmstat/df.csv [zimbra@zcs806 tmp]$ ls -lah /tmp/df.tar -rw-r----- 1 zimbra zimbra 80K Mar 29 06:44 /tmp/df.tar [zimbra@zcs806 tmp]$ tar tvf /tmp/df.tar -rw-r----- zimbra/zimbra 2566 2014-03-11 00:00 opt/zimbra/zmstat/2014-03-10/df.csv.gz -rw-r----- zimbra/zimbra 2553 2014-03-12 00:00 opt/zimbra/zmstat/2014-03-11/df.csv.gz [cut - Ajcody] -rw-r----- zimbra/zimbra 2513 2014-03-28 00:00 opt/zimbra/zmstat/2014-03-27/df.csv.gz -rw-r----- zimbra/zimbra 2531 2014-03-29 00:00 opt/zimbra/zmstat/2014-03-28/df.csv.gz -rw-r----- zimbra/zimbra 8013 2014-03-29 06:40 opt/zimbra/zmstat/df.csv
Directory Sizes In /opt/zimbra
Please see the following and provide the output to support. Note, even though this method is faster than doing a du it still can take awhile.
* Ajcody-Server-Misc-Topics#Faster_Way_To_Get_Directory_Size_On_Filesytem_-_find_vs_du
Adjusting The Disk Alert Threshold
Note - zmlocalconfig smtp_notify must return yes if you want to receive the notifications.
If you just need to adjust the disk alert threshold, then see the following:
See current values:
zmlocalconfig | grep zmdisklog
Example adjustment:
su - zimbra zmlocalconfig -e zmdisklog_critical_threshold=98 zmlocalconfig -e zmdisklog_warn_threshold=95 zmstatctl
To exclude a partition from the checks [example of two being excluded]:
su - zimbra zmlocalconfig -e zmstat_df_excludes="/mount/point:/mount/point2" zmstatctl
They might be a bug on this, where you'll keep getting email until a logrotate happens [zimbra.log?].
- Changing Zmstat-df values do not take affect until logrotate
Some things to do to confirm and share with support or in bug. As zimbra
su - zimbra ls -la /var/log/zimbra.log df -h /dev/mapper/vg_rhel664-lv_root 5.5G 3.5G 1.7G 68% / tmpfs 939M 0 939M 0% /dev/shm /dev/sda1 485M 79M 381M 18% /boot /dev/sdb1 30G 6.2G 23G 22% /opt date zmlocalconfig | grep zmdisklog zmdisklog_critical_threshold = 80 zmdisklog_warn_threshold = 85 zmlocalconfig -e zmdisklog_critical_threshold=95 zmlocalconfig -e zmdisklog_warn_threshold=90 zmlocalconfig | grep zmdisklog zmdisklog_critical_threshold = 95 zmdisklog_warn_threshold = 90 zmstatctl restart date ps -eaf | grep zmstat-df ls -la /var/log/zimbra.log date ; grep "Disk warning" /var/log/zimbra* ; zmmailbox -z -m admin@`zmhostname` s -l 100 -t message "Subject: Disk and after:yesterday" ##Note - Emails by default go out every 10 minutes - for example: [zimbra@zcs803 ~]$ date ; grep "Disk warning" /var/log/zimbra* ; zmmailbox -z -m admin@`zmhostname` s -l 100 -t message "Subject: Disk and after:yesterday" Thu May 22 09:40:08 PDT 2014 /var/log/zimbra.log:May 22 08:30:00 zcs803 zimbramon[18826]: 18826:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% /var/log/zimbra.log:May 22 08:40:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% /var/log/zimbra.log:May 22 08:50:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% /var/log/zimbra.log:May 22 09:00:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% ## Note - I had readjusted the variable to not warn during this time segment ## /var/log/zimbra.log:May 22 09:20:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% /var/log/zimbra.log:May 22 09:30:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% /var/log/zimbra.log:May 22 09:40:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82% num: 7, more: false Id Type From Subject Date ---- ---- -------------------- -------------------------------------------------- -------------- 1. 328 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 09:40 2. 327 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 09:30 3. 326 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 09:20 ## Note - I had readjusted the variable to not warn during this time segment ## 4. 325 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 09:00 5. 324 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 08:50 6. 323 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 08:40 7. 320 mess admin Disk / at 82% on zcs803.DOMAIN.com: 05/22/14 08:31
Continue to monitor your zmmailbox search results for an hour.
The Basic Information Support Needs
as root:
- cat /etc/fstab
- Shows us what is mounted upon boot
- cat /proc/mounts
- Shows us what is currently mounted and its status - you can see if a mount is read-only here.
- df -hT
- Lists current mounts using human-readable size information and also notes the filesystem type.
as zimbra:
- zmprov -l gs `zmhostname` | egrep 'Back|Redo'
- Will show us a number of variables related to backup and redologs. Also tell us if your using auto-group or the default method.
- du -sh /opt/zimbra/redolog
- Will might notice your redolog logs aren't rolling over, causing a possible issue.
- ls -latr /opt/zimbra/backup
- This is the default backup target, please adjust this path here and below if you are using a different zimbraBackupTarget value.
- zmprov gs `zmhostname` zimbraBackupTarget
- We'll be able to confirm permissions are right.
- ls -latr /opt/zimbra/backup/tmp
- This will show us if you have failed backup jobs and confirm tmp is being cleaned appropriately after the backup is done.
- ls -latr /opt/zimbra/backup/sessions
- This will show us what backup sessions are available and confirm permissions are correct.
- Adjust path if your zimbraBackupTarget value is not the default path.
- su - zimbra
- zmjava com.zimbra.cs.backup.util.GetVersion
- cd /opt/zimbra/backup/sessions/full-[YOUR MOST RECENT FULL]/
- head -n6 session.xml
- cd /opt/zimbra/backup/sessions/incr-[YOUR MOST RECENT INCREMENTAL]/
- head -n6 session.xml
- Some directory sizes in the backup directory:
- Default path first
- du -sh `find /opt/zimbra/backup -maxdepth 2 -type d`
- If your using a different backup target, check that directory also. Replace /opt/zimbra/backup above with your backup path.
- Default path first
- zmbackupquery
- This should match what's in the sessions directory and it will also tell us if status of each backup and how many accts were done.
- crontab -l | grep -i back
- This will show use when backups are support to run and with what options they are running with.
- zmlocalconfig | grep -i back
- This is useful to see a number of backup options not exposed in the crontab, things related to the zip options.
- zmvolume -l
- This is useful to see how many volumes are being used, if HSM is being used, and if compression is being done at the volume level.
Additional Log Files Support Might Need
And send the following logs:
- /var/log/messages
- Filesystem issues often times are noted here and also in syslog. This might explain an interruption in the backup process. Server restarts, filesystem going full, filesystem going read-only, etc.
- /var/log/syslog
- /opt/zimbra/log/mailbox.log
- The backup activity is logged here.
- And any other mailbox.log file that would cover the event
Additional Checks For Performance Specific Issues
If Your Using a SAN or NFS For Your Backup Target - Please Check Your IOWait
Ideally, you would compare iowait and performance data from the target backup host as well as the stats available on the ZCS servers. To get graphs and stats on this from ZCS, please see Ajcody-Testing-Debugging#zmstat_and_zmstat-chart . You should submit this data and iowait conclusions if you still need to submit a support case about backup performance issues.
Is HSM Running During Your Backup Window
- Are you running HSM? HSM should not be ran during your backup window.
- "RFE: HSM and backup should not run at the same time if initated."
Are You Using --zipStore
--zipStore zips the blobs vs. keeping the blobs as individual files. --zipStore does not use compression either. For most circumstances, this will give the best performance, especially with NFS. This should be the default behavior of the backups, the following RFE is when it became the default [ZCS6+] :
- "backup: default to the zip option"
- https://bugzilla.zimbra.com/show_bug.cgi?id=31836#c6
- Link to comment that explains options and default behavior.
- https://bugzilla.zimbra.com/show_bug.cgi?id=31836#c6
To see if zip's are being used for backups for example, in the backup/session directory you'll know if it is by seeing .zip files:
mail:~/backup/sessions/full-20080820.160003.770/accounts/115/988/11598896-a89b-4b9d-bedb-1ed1afcb6c87/blobs zimbra$ ls blobs-1.zip blobs-2.zip blobs-3.zip blobs-4.zip
To see if the zip file is using compression [-Z option for unzip will indicate whether or not the archive is actually compressed] :
unzip -Z blobs-4.zip 293 files, 5982984 bytes uncompressed, 5982984 bytes compressed: 0.0%
Also, if your zmvolume has compression enabled the blobs will remain compressed within the zip also upon backup. The point being, they are uncompressed to be then put into a zip file when the backup is using --zipStore.
Restore Compatibility Between ZCS Versions
Please see the following for details. In summary, user level restores should work against older ZCS backup data.
- "support for restore across major versions"
Other related bug/rfe's:
- support for restore across major versions
- Restore across multiple versions
- Restore across multiple versions
- Backups must be compatible across patch releases
- write ZCS version into backup
- Restore should deal with database schema changes that add columns with defined default value
- Incorrect upgrade documentation regarding backups
NFS Use For Backups
Please see this RFE:
- "Need clarity on supporting nfs mounted zimbra directories - report error/msg if nfs mount is present"
This is the proposed statement to be included in the release notes following the RFE:
- ZCS & NFS:
- Zimbra will support customers that store backups (e.g. /opt/zimbra/backup) on an NFS-mounted partition. Please note that this does not relieve the customer from the responsibility of providing a storage system with a performance level appropriate to their desired backup and restore times. In our experience, network-based storage access is more likely to encounter latency or disconnects than is equivalent storage attached by fiber channel or direct SCSI.
- Zimbra continues to view NFS storage of other parts of the system as unsupported. Our testing has shown poor read and write performance for small files over NFS implementations, and as such we view it unlikely that this policy will change for the index and database stores. We will continue to evaluate support for NFS for the message store as customer demand warrants.
- When working with Zimbra Support on related issues, the customer must please disclose that the backup storage used is NFS.
Things To Check
- Check the /var/log/messages on both the zimbra server and the nfs server for nfs related errors during the time frame of your backup.
- Check /opt/zimbra/log/mailbox.log for error messages about folders/files not being able to be written or missing directory errors.
- Is root_squash configured on the nfs server? If it's changed to no_root_squash , does the behavior of the backup change?
- Is the */backup directory owned by zimbra:zimbra with at least 750 or 755 permissions?
- This parent directory as given in:
- zmprov gs `zmhostname` zimbraBackupTarget
- This parent directory as given in:
- Does zimbraBackupTarget have at least the subdirectories of : sessions and tmp : and are owned by zimbra:zimbra with 750 or 755 permissions?
- If not, try manually creating them and then running a test backup.
- IF USING A NAS - MAKE SURE YOUR NOT USING EXTENDED ACLS OR THAT YOU HAVE THEM CONFIGURED PROPERLY
- If your backup session directory shows something like : drwxrwx---+ 2 zimbra zimbra 4096 Sep 14 00:00 TO_DELETE-full-XXXXXX , that + sign indicated extended acls are in use.
Debugging Example
Steps I wrote for one customer, where saving out the information as you walk through all the commands would give enough information [hopefully] to submit a good rfe/bug:
1. make a test partition on nfs server - /nfs-test 2. mount on zimbra server 2A. mkdir /nfs-test 2B. chmod 755 /nfs-test 2C. mount nfs-server:/nfs-test /nfs-test 2D. ls -la /nfs-test 2E. mkdir /nfs-test/backup 2F. chown zimbra:zimbra /nfs-test/backup 2G. chmod 755 /nfs-test/backup 2H. su - zimbra ; touch /nfs-test/backup/testfile 2I. ls -laR /nfs-test/ 2J. rm /nfs-test/backup/testfile 3. Set zimbraBackupTarget 3A. zmprov ms `zmhostname` zimbraBackupTarget /nfs-test/backup 4. Run a full backup against one account 4A. ex. zmbackup -f -a user@domain.com 5. ls -laR /nfs-test/ 6. If you again, run into the same problem. You could also repeat the backup after increasing the backup logging variable for the account your trying to backup. If you didn't run into the same problem, it might had to do with the initial setup of the nfs mount and permissions being used during the directory creation. 6A. zmprov aal user@domain.com zimbra.backup debug 6B. logging will show up in /opt/zimbra/log/mailbox.log 6C. Remove account logging when your done. zmprov ral user@domain.com zimbra.backup 8. Change zimbraBackupTarget back to your production path.
Setup A Fast Test NOT Using NFS
A way to "avoid" the NFS issues for testing purposes would be to setup a new zimbraBackupTarget to try doing a full backup of a couple of user accounts. I DON'T recommend this if your using auto-group for zimbraBackupMode [ zmprov gs `zmhostname` zimbraBackupMode ] , only if your using Standard mode.
[as root] ** adjust your new "backup" directory path as needed - mine is just an example** mkdir /mnt/usb1/backup-test chown zimbra:zimbra /mnt/usb1/backup-test chmod 750 /mnt/usb1/backup-test su - zimbra **confirm backup mode as standard** zmprov gs `zmhostname` zimbraBackupMode **if not stand, please stop** zmprov ms `zmhostname` zimbraBackupTarget /mnt/usb1/backup-test **Find a couple of test accounts to do a full for. Make sure you'll have space to do the backup regards to need free space.** zmbackup -f -a user1@domain.com user2@domain.com user3@domain.com **Watch and confirm status of backup you just started.** zmbackupquery **Confirm files were backed up in right location** ls /mnt/usb1/backup-test/sessions/ **Failed backups would most likely results in left over directory in tmp directory** ls /mnt/usb1/backup-test/tmp/
Restore For Disaster Recovery
For Full Single ZCS Server DR Restores
Please see Network_Edition_Disaster_Recovery
Some additional notes I have on it - Ajcody-Disaster-Recovery-Specific-Notes
For Multi-Server DR Restore Specifics
Along with the above references, please see Ajcody-Notes-Multi-Server-Restore-DR
To Restore Just The LDAP Date
Let's say your ldap data was 'lost/destoyed' but everything else was intact, you should look at the zmrestoreldap command.
This section should have more precaution and background information to handle this section.
The basics:
- To find the LDAP session labels type -lbs.
- zmrestoreldap -lbs
- Restore the complete LDAP directory server
- zmrestoreldap -lb full20061130135236
- Restore LDAP data for specific accounts
- zmrestoreldap -lb full20061130135236 -a tac@DOMAIN.com jane@DOMAIN.com
To Restore Just The Mysql DB
Option available from the following RFE work:
- "Allow backup of only primary message volume"
- https://bugzilla.zimbra.com/show_bug.cgi?id=35278
- Options to exclude types of data as described in the 6.0.6 Admin Guide:
- Search index : If you do not restore the search index data, the mailbox will have to be reindexed after the restore.
- zmrestore <all or account> --exclude-search-index
- Blobs : This is a useful option when all blobs for the mailbox being restored already exists.
- zmrestore <all or account>|--exclude-blobs
- HSM-blobs : This is useful when all HSM blobs for the mailbox being restored already exists.
- zmrestore <all or account> --exclude-hsm-blobs
- Search index : If you do not restore the search index data, the mailbox will have to be reindexed after the restore.
- Options to exclude types of data as described in the 6.0.6 Admin Guide:
- https://bugzilla.zimbra.com/show_bug.cgi?id=35278
Let's say your mysql data was 'lost/destoyed' but everything else was intact. This might be the solution for the situation:
The steps below are to REPRODUCE A DR situation to test.
- zmcontrol stop
- Caution - step to reproduce DR situation to test against
- mv ~/db/data ~/db/data.OLD
- Getting old mysql passwords
- zmlocalconfig -s mysql_root_password
- zmlocalconfig -s zimbra_mysql_password
- [as zimbra] /opt/zimbra/libexec/zmmyinit
- Most likely your mysql passwords will change.
- See the following article to set them back:
- ldap start
- zmconvertctl start
- mysql is already running per the zmmyinit - mysql.server status - to check.
- zmrestoreoffline -a all -br --systemData --excludeSearchIndex --excludeBlobs --excludeHsmBlobs
- Note - you most likely will want to use the -br or -rf option for this situation.
- -br,--backedupRedologsOnly : Replays the redo logs in backup only, which excludes archived and current redo logs of the system
- -rf,--restoreFullBackupOnly : Restores to last full backup only, which excludes incremental backups.
- -sys,--systemData : Restores global tables and local config.
- This restores the 'zimbra' mysql data - this stuff Ajcody-Mysql-Topics#zimbra_Database_Default_Example_for_ZCS5
- Important - the zimbra db holds information about the volumes. This needs to be restored/existing prior to user db restoration.
- Note the use of -a all , you could also do this for one account first to confirm operation is successful for your circumstances.
- Note - you most likely will want to use the -br or -rf option for this situation.
- If you used -rf or -br with your zmestoreoffline, you might also need to use zmplayredo to finish it up to get the most complete restore.
- Improve zmrestore & zmrestoreoffline performance
- Ajcody-Backup-Restore-Issues#zmplayredo_-_Replaying_Content_From_Any_Redolog_File and other 'redolog' sections on that page.
Performance Issues And Time To Complete
Please see Ajcody-Notes-BackupPlans#What_About_Backups.3F_I_Need_A_Plan
Also created an RFE for increase backup performance:
- "Improve zmrestore & zmrestoreoffline performance"
- "Improvement to backup performance"
- http://bugzilla.zimbra.com/show_bug.cgi?id=36220
- Above marked as a dupe of "Reduce duration of maintenance mode during backup"
Understanding Option Flags For zmbackup & zmrestore
First, they don't make sense if your just reading from the help output - I will not argue this point at all.
The biggest problem with the options I point out below is that you can often include them in the command and they do nothing or you include them for a particular situation and they don't apply. Why is this a problem? Because they are silent and give no output telling you that it's not necessary, it's redundant, or it will actually cause your intended results to fail simply because you included the option.
zmrestore Options
Problems mostly revolve around these options.
To Times In The Past (If -lb Isn't Used, Implies Your Using Times/Incr/RedoSeq AFTER Last Full)
- -restoreToIncrLabel
- <arg> Replay redo logs up to and including this incremental backup
- Requires: --label or -lb
- <arg> Replay redo logs up to and including this incremental backup
- -restoreToTime
- <arg> Replay rodo logs until this time
- Requires: --label or -lb
- <arg> Replay rodo logs until this time
- -restoreToRedoSeq
- <arg> Replay up to and including this redo log sequence
Redolog Variables
- --backedupRedologs or -br
- Replays the redo logs in backup only, which excludes archived and current redo logs of the system
- Only useful when restoring against latest full backup (NOT using the -lb option).
- Will restore using incremental backup data after the last full backup as well but NOT including any redolog activity.
- Replays the redo logs in backup only, which excludes archived and current redo logs of the system
- --restorefullBackup or -rf
- Restores to the full backup only, not any incremental backups since that backup.
- The default behavior of zmrestore in general is to always play from a "full" and "incrementals" that are associated with it UNLESS you state otherwise.
- If you do:
zmrestore -a user@domain.com -lb full6monthsago
- It will playback the incremental data associated with that full from 6 months ago.
- If you do:
zmrestore -a user@domain.com -lb full6monthsago -rf
- It will ONLY playback the data in the full from 6 months ago.
- If you do:
- Will not progress past the data that's in the last full backup, no incremental backups after it in other words.
- Implies NO redolog play, so there's no need to use -br.
- The default behavior of zmrestore in general is to always play from a "full" and "incrementals" that are associated with it UNLESS you state otherwise.
- Restores to the full backup only, not any incremental backups since that backup.
Targets And Labels
- --label or -lb
- <arg> The label of the full backup to restore. Restores to the latest full backup if this is omitted.
- --target or -t
- <arg> Specifies the backup target location. The default is <zimbra_home>/backup.
Impact Of AutoGroup Option Being Used
Place for notes about how autogroup backup option might impact or limit command options.
zmbackup Options
Problems mostly revolve around these options:
- --target or -t
- <arg> Specifies the target backup location. The default is <zimbra_home>/backup.
- --zip or -z
- Zips email blobs in backup - using compression
- --zipStore
- Zips email blobs in backup - does NOT use compression
Deleting Old Backups -del
Caution You want to delete from the oldest label to newest. The -del option will automatically purge all older sessions prior to the label you used. To find out the label names, use zmbackupquery.
Format example:
zmbackup -del <oldest_backup_label>
Impact Of AutoGroup Option Being Used
Place for notes about how autogroup backup option might impact or limit command options.
Changing Default Backup Target
To find out what the current backup target is, do:
zmprov gacf | grep zimbraBackupTarget zimbraBackupTarget: /opt/zimbra/backup
This is also configurable at the "server" level:
zmprov gs `zmhostname` | grep zimbraBackupTarget zimbraBackupTarget: /opt/zimbra/backup
For example:
zmprov ms `zmhostname` zimbraBackupTarget /san/mount/backup
Issues of changing the default path in regards to the admin web console. Please see:
- "Work backup path if it was changed in admin interface"
Another way to change the backup path is described at Changing_Backup_directory_and_General_Information. I recommend reviewing it as well.
See Change Location For Backup Or Restore Source Data for non-default adjustments to CLI commands in regards to a non-default path for backups.
RFE & Bugs Concerning Option Flags For Restore And Backup
Setting Account Status After Restore Is Done
I filed an RFE for this:
- "RFE: zmrestore & zmrestoreoffline have option to set account status"
Use DL COS Status To Generate User List Of Accounts
I filed an RFE for this:
- "RFE: backup/restore should allow -a option to use DL's COS Status"
An Overview Of Some Backup/Restore Items
I believe we are a little light on describing the shared blobs situation. Shared blobs can cause different corrections to a problem as compared to a normal message issue that isn't shared. I'll start some notes on this here.
From Backup and Restore
- "When backing up shared messages, the backup process looks to see whether a Binary Large Object file (BLOB) representing a message already exists in the backup. If it does, it simply flags this object as such and does not copy its content again."
- "Keeping the same backup target saves disk space, because shared binary large object files (BLOB) and other files do not have to be duplicated every time the backup process runs.
Bugs/RFE's
- "Use zip files for shared blobs of a full backup made with --zip option"
Remote copies of backup data for DR use
You will want to copy over /opt/zimbra/redolog/archive/* and /opt/zimbra/redolog/redo.log frequently in order to stay current. The redo.log file being open is not a problem since the crash recovery step can work with redo.log file in any state.
The redolog/archive/ contains logs that have not yet been backed up by an incremental (or by a full in auto-grouped mode)
The redo.log rolls over when it reaches zimbraRedoLogRolloverFileSizeKB (by default 100MB prior to ZCS 5.0.11 and 1GB after). When ZCS restarts after a crash, it seems to work through the current redo.log ok regardless of its state, if the current log really must be copied."
My Initial Thoughts On This
Start of process:
- Weekend full on prod
- rsync full-xxx on prod > remote sessions/
- rsync redolog/* > remote redolog/
- through non-full and incremental times every x about of minutes
- weekday nights 10pm incre-xxx on prod
- rsync incr-xxx on prod > remote sessions/
- rsync redlog/* > remote redolog/
Created three separate rsync cron rules.
- Full - once a week
- Confirms full is done and then looks for latest full-xxxx and rsyncs that specific directory
- Incre - once a night except for full schedule night
- Confirms incre is done and then looks for latest incr-xxx and rsyncs that specific directory
- Redolog/ - every x amount of minutes (outside of full and incr backups sessions)
- Soes full rsync of redolog/ - probably want delete/remove option?
- lsof will report /opt/zimbra/redolog/redo.log is open.
Somewhere we need to account for accounts.xml in this process. And also confirm what else might be missing. Also, steps on the actually restore process depending on when/where the DR event took place.
What's Needed For Later Restores
In regards to what is moved and what is needed for later restores, you must remember this "flow" of the backups:
- Full backup files that contains all the information needed to restore mailboxes (to that point in time)
- Incremental backup files that contains the LDAP directory server files and all the redo log transactions written since the last backup (to that point in time)
- Redo logs that contains current and archived transactions processed by the Zimbra server since the last incremental backup (to that point in time - more about this topic is above)
Variables to be aware of in regards to backup/restore [ viewed with - zmprov gs [server-name] | grep Backup ]:
- This is that gui path option - note, it doesn't change default to redo file path [see below]
zimbraBackupTarget zimbraBackupMode zimbraBackupAutoGroupedInterval zimbraBackupAutoGroupedNumGroups zimbraBackupAutoGroupedThrottled zimbraBackupReportEmailSubjectPrefix zimbraBackupReportEmailSender zimbraBackupReportEmailRecipients
- Variables to be aware of in regards to redo files [ viewed with - zmprov gs [server-name] | grep Redo ]:
zimbraRedoLogArchiveDir zimbraRedoLogDeleteOnRollover zimbraRedoLogEnabled zimbraRedoLogFsyncIntervalslMS zimbraRedoLogLogPath zimbraRedoLogRolloverFileSizeKB
Mysql Table That References Most Recent Backup Session Of Users (AutoGroup Backup Mode)
During the backup we update the zimbra.mailbox table for each mailbox to record the most recent backup time. This is in the "last_backup_at" column within Mysql.
This data is used by auto-grouped backup to figure out which mailboxes to backup.
Creating A List Of Last Backup Of Users
Remove the , | head , below to get a full listing of all your accounts. Note, this reports on the users that exist on the mailstore your running the command.
[zimbra@zcs806 tmp]$ mysql zimbra -NBe 'select from_unixtime(last_backup_at), comment from mailbox' | sort | head 2014-06-05 01:00:24 dluser558@zcs806.DOMAIN.com 2014-06-05 01:00:26 dluser557@zcs806.DOMAIN.com 2014-06-05 01:00:28 dluser556@zcs806.DOMAIN.com 2014-06-05 01:00:30 dluser555@zcs806.DOMAIN.com 2014-06-05 01:00:31 dluser554@zcs806.DOMAIN.com 2014-06-05 01:00:32 dluser551@zcs806.DOMAIN.com 2014-06-05 01:00:32 dluser552@zcs806.DOMAIN.com 2014-06-05 01:00:32 dluser553@zcs806.DOMAIN.com 2014-06-05 01:00:33 dluser550@zcs806.DOMAIN.com 2014-06-05 01:00:34 dluser549@zcs806.DOMAIN.com
I include also below the results from the zmbackupquery for the first user:
[zimbra@zcs806 tmp]$ date ; zmbackupquery -a dluser558@zcs806.DOMAIN.com Wed Jun 11 14:29:48 PDT 2014 Account: dluser558@zcs806.DOMAIN.com Label: full-20140605.080023.296 Type: full Started: Thu, 2014/06/05 01:00:23.296 PDT Ended: Thu, 2014/06/05 01:02:37.739 PDT Acct ID: 85d3c8f4-eea8-4cf2-8a84-8f1fcc87d555 Label: full-20140529.080016.546 Type: full Started: Thu, 2014/05/29 01:00:16.546 PDT Ended: Thu, 2014/05/29 01:02:51.556 PDT Acct ID: 85d3c8f4-eea8-4cf2-8a84-8f1fcc87d555 Label: full-20140522.080015.818 Type: full Started: Thu, 2014/05/22 01:00:15.818 PDT Ended: Thu, 2014/05/22 01:02:42.126 PDT Acct ID: 85d3c8f4-eea8-4cf2-8a84-8f1fcc87d555 Label: full-20140515.080016.160 Type: full Started: Thu, 2014/05/15 01:00:16.160 PDT Ended: Thu, 2014/05/15 01:02:34.204 PDT Acct ID: 85d3c8f4-eea8-4cf2-8a84-8f1fcc87d555
Possible Issue That A Failed Or Interrupted Backup Causes
An interrupted backup can cause an issue because the table currently gets updated right off the bat rather than waiting for backup to be successfully completed.
Possible RFE: To update zimbra.mailbox.last_backup_at column for successfully backed-up mailboxes to the very end of the backup process, to either just before or just after renaming the /opt/zimbra/backup/tmp/<backup label> directory to /opt/zimbra/backup/sessions/<backup label>.
Setting To Null To Cause A New Backup For User
To undo what was done by an interrupted backup for example, you need to clear this column (set it to null) for the affected mailboxes. By clearing the column, you're forcing the next AG backup to choose these mailboxes because they look like they have never been backed up. If you don't clear this column, you have to wait until the next cycle. (7 days)
Example syntax to view:
mysql zimbra -e "select last_backup_at from mailbox where id=27"
Example syntax to change data of the last_backup_at to NULL"
mysql zimbra -e "UPDATE mailbox SET last_backup_at = NULL WHERE id = 27"
Related Bugs RFE's
- "In auto-grouped backup, delay the update of mailboxes' last_backup_at timestamp to the very end of backup"
- "Partial backup should be able to finish successfully when backup volume runs out of space"
Restore Requires accounts.xml File
- "accounts.xml file dependency needed for zmrestore"
Also, see:
- "accounts.xml file improperly updated when a full backup is deleted"
Change "Location" For Backup Or Restore Source Data
Remember that zmbackup and zmrestore can take flags as well in regards to the location of items.
- zmrestore & zmbackup can both take : -t,--target (default <zimbra_home/backup)
You can't state a different location with redo logs though. There's a command called zmplayredo [for newer versions of ZCS] and it has a variable to point to the redologs to play from [ --logfiles ]. It will replay into the default redolog directory or redolog file. The mailbox has to be stop to run zmplayredo . This is a command to manual kick off a replay of a redo log. This is normally done with the zmrestore when options about to a specific time aren't included.
Manual Removal Of Older Backup Sessions
General Situation:
- Keep in mind every restore requires starting with data from a full backup.
- For each account on the server, there must be at least 1 full backup after the deletion is complete.
- You should also make sure all incremental backups made after the oldest of the remaining backups are retained.
- This basically gets reduced to deleting only those backups that are old enough, based on your full/incremental schedule.
More Specific Issues:
- Does the accounts.xml dependency cause issues with this?
- No. Just don't delete it.
- What about the contents of the backup/tmp/ data or shared blobs type references?
- Don't touch this directory either. It is used during backup and restore. You don't want to change its content while an operation is going on, so best to leave it alone.
- What if a zimbra server is running some type of restore of backup command while the manual removal is running on the nfs server?
- You shouldn't remove the backups that are being used in a restore currently underway. You are responsible for avoiding the race condition.
- Please understand you are responsible for avoiding the race condition. Make sure no backup or restore is happening at the moment, then rename the directories that will be deleted, preferably move them to another subdirectory, e.g. /opt/zimbra/backup/sessions_to_delete. Then delete.
- You shouldn't remove the backups that are being used in a restore currently underway. You are responsible for avoiding the race condition.
LDAP Backup Related Items
Backup Schedule On LDAP Only Server [non-Mailstore]
If you look at the code in /opt/zimbra/bin/zmschedulebackup :
if ($BACKUP_MODE eq 'Standard') { # default schedule: full backup 1am every sunday, incr backup 1am every weekday # deletes backups older than a month at 12am everyday if (isLdapOnly()) { @default = ("0 1 * * 6 /opt/zimbra/bin/zmbackup $target\n", "0 0 * * * /opt/zimbra/bin/zmbackup -del 1m\n"); } else { @default = ("0 1 * * 6 /opt/zimbra/bin/zmbackup -f $account $target $compress \n", "0 1 * * 0-5 /opt/zimbra/bin/zmbackup -i $compress\n", "0 0 * * * /opt/zimbra/bin/zmbackup -del 1m\n"); } %commandmap = ("f", "/opt/zimbra/bin/zmbackup -f $target $compress $account", "i", "/opt/zimbra/bin/zmbackup -i $compress", "d", "/opt/zimbra/bin/zmbackup -del"); } else { # Auto-Grouped mode # default schedule: full backup 1am everyday, no incr backup # deletes backups older than a month at 12am everyday @default = ("0 1 * * 0-6 /opt/zimbra/bin/zmbackup -f $target $compress\n", "0 0 * * * /opt/zimbra/bin/zmbackup -del 1m\n"); %commandmap = ("f", "/opt/zimbra/bin/zmbackup -f $target $compress", "i", "/opt/zimbra/bin/zmbackup -i $compress", "d", "/opt/zimbra/bin/zmbackup -del"); }
Notice the specific check and then format for ldapOnly:
if (isLdapOnly()) { @default = ("0 1 * * 6 /opt/zimbra/bin/zmbackup $target\n", "0 0 * * * /opt/zimbra/bin/zmbackup -del 1m\n");
Related Bugs To LDAP Backups
Some bugs to be aware of - most are resolved/fixed:
- "Zmbackup ldap data is not backed up if target server is not hosting it."
- "System LDAP is backuped only when -a all is specified"
- "LDAP backup directory format difference between full install and ldap only install"
- "backups in the admin console should back up LDAP data"
- "LDAP backup failure due to library version mismatch"
- "ldap backup not kept with rest of backup"
- "zmbackup ldap backup is not logged"
- "Backup default cronjob is wrong for ldap only install"
- "zmbackupldap fails to rename directory on NFS"
- "default ldap server only backup schedule is weekly"
- "zmbackupldap fails when using month intervals and succeeding month has more days"
- "zmbackup on LDAP-only hosts should have help/usage argument, or at least reject invalid arguments"
- "corp: ldap backups fail"
A Way To Verify Backup Integrity
I filed an RFE for this:
- "A way to verify backup integrity"
Negative Seek Offset Error & RFE
Explanation of negative seek offset error during a restore attempt and manual fixes are located here:
- "RFE: Monitor/test backup *.zip files for corruption & repair tools"
Auto-Group Backups Rather Than Default Method Topics
General Description And Official References
Having trouble completing that entire full backup during off-hours? Enter the hybrid auto-grouped mode, which combines the concept of full and incremental backup functions - you’re completely backing up a target number of accounts daily rather than running incrementals.
Auto-grouped mode automatically pulls in the redologs since the last run so you get incremental backups of the remaining accounts; although the incremental accounts captured via the redologs are not listed specifically in the backup account list. This still allows you to do a point in time restore for any account.
Administrative manual page:
http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.17.01.html
Compare the sections called "Standard Backup Method" & "Auto-Grouped Backup Method"
http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.17.02.html
Configuration details are here on that page:
http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.17.08.html
Good explanation:
http://www.zimbrablog.com/blog/archives/2008/08/recent-admin-tidbits-part-1.html
Simply divide your total accounts by the number of groups you choose (zimbraBackupAutoGroupedNumGroups is 7 by default) and that’s how many will get a full backup session each night. Newly provisioned accounts, and accounts whose last backup is a specified number of days older are picked first. (zimbraBackupAutoGroupedInterval is defaulted to 1d)
Think of auto-grouped mode as a full backup for the scheduled group as well as an incremental (via redologs) for the all other accounts at the same time.
Bugs - RFE's To Review For Auto-Group
Please see:
- "In auto-grouped backup, delay the update of mailboxes' last_backup_at timestamp to the very end of backup"
- "improve error reporting to ignore harmless missing sequences"
- "unbalanced auto-grouped backups"
Enabling Auto-Group For Backups - Schedule In Crontab
- You'll need to make the variable changes as listed in the administrative guide and then the following for set the crontab correctly.
- Run zmschedulebackup --help to see a list of options.
- Run zmschedulebackup -D , which will now set a new default schedule (crontab) that uses auto-group settings now that your variable are set for it.
- If you want the zip option to also be used do:
zmschedulebackup -D -z
- See this bug about zmschedulebackup not being able to pass -zipStore option.
- If you want the zip option to also be used do:
- Incremental backups aren't performed as they were.
- You'll see there's no longer the -i option in cron.
- Incrementals are performed, in a manner, but the redologs are just copied into the full backup session
Two bugs to look at as well:
- cannot backup using admin ui in autogrouped mode
- Expose autogroup backup configuration to admin UI
Standard Mode Default Cron Setup
- Here is a schedule without auto-grouped backups enabled:
- The Default Schedule For Normal Backups [ zmschedulebackup --help ]:
f 0 1 * * 6 i 0 1 * * 0-5 d 1m 0 0 * * *
- To set the backup to standard mode and use the default schedule one would:
zmprov mcf zimbraBackupMode Standard
zmschedulebackup -D
- The crontab would look like:
- To set the backup to standard mode and use the default schedule one would:
0 1 * * 6 /opt/zimbra/bin/zmbackup -f -a all 0 1 * * 0-5 /opt/zimbra/bin/zmbackup -i -a all 0 0 * * * /opt/zimbra/bin/zmbackup -del 1m
Auto-Group Mode Cron Setup
- Here is a schedule with auto-grouped enabled:
- The Default Schedule For Auto-Group Backups [ zmschedulebackup --help ]:
f 0 1 * * 0-6 d 1m 0 0 * * *
- To set the backup to auto-group mode and use the default schedule one would:
zmprov mcf zimbraBackupMode Auto-Grouped
zmschedulebackup -D
- The crontab would look like:
- To set the backup to auto-group mode and use the default schedule one would:
0 1 * * 0-6 /opt/zimbra/bin/zmbackup -f 0 0 * * * /opt/zimbra/bin/zmbackup -del 1m
Some Variables For Auto-Group
The below might not be complete or the defaults, I just wanted to save this before I forget them. Try to get more complete details on these later.
zmprov gacf | grep Backup zimbraBackupAutoGroupedInterval: 1d zimbraBackupAutoGroupedNumGroups: 7 zimbraBackupAutoGroupedThrottled: FALSE zimbraBackupMode: Auto-Grouped
Auto-group And Redologs
Please see Ajcody-Backup-Restore-Issues#Redologs_And_Auto-group_In_Regards_To_Backups
Problems Switching To Auto-Groups Because It Wants To Run A Full Against All Accounts
Please see the following bug/rfe made about problems switching over to Auto-group when the first backup run of it tries to backup ALL of he accounts. I have the full how-to within the bug. It basically manipulates the last_backup_at for each account.
- Ability to set last_backup_at to NULL or to the day before for All users, group of users, user
Backup And Deletion Schedule - zmschedulebackup
How To Adjust The Deletion Schedule
For my example below, I first set the backup to the "default" schedule. And I then adjust that "default" to have backups delete with a 14 day interval rather than 1 month.
[zimbra@mail3 ~]$ zmschedulebackup -D Default schedule set Current Schedule: f 0 1 * * 6 -a all i 0 1 * * 0-5 d 1m 0 0 * * * [zimbra@mail3 ~]$ zmschedulebackup -q Current Schedule: f 0 1 * * 6 -a all i 0 1 * * 0-5 d 1m 0 0 * * * [zimbra@mail3 ~]$ zmschedulebackup -R f "0 1 * * 6" i "0 1 * * 0-5" d 14d "0 0 * * *" Schedule replaced Current Schedule: f 0 1 * * 6 -a all i 0 1 * * 0-5 d 14d 0 0 * * * [zimbra@mail3 ~]$ zmschedulebackup -q Current Schedule: f 0 1 * * 6 -a all i 0 1 * * 0-5 d 14d 0 0 * * * [zimbra@mail3 ~]$ crontab -l | grep backup 0 1 * * 6 /opt/zimbra/bin/zmbackup -f -a all 0 1 * * 0-5 /opt/zimbra/bin/zmbackup -i 0 0 * * * /opt/zimbra/bin/zmbackup -del 14d
The Zip - Compression Option For Backups
Using the zip option will compress all those thousands of single files that exist under a user's backup, decreasing performance issues that arise from writing out thousands of small files as compared to large ones. This is often seen when one is :
- Using nfs for the backup directory
- Copying/rsyncing backups to a remote server
- Are using some third party backup software (to tape) to archive/backup the zimbra backup sessions.
Optional Tweaks To The Zip Options
Please see this comment and those underneath it within this RFE:
- "Use zip files for shared blobs of a full backup made with --zip option"
- http://bugzilla.zimbra.com/show_bug.cgi?id=26624#c6
- Use zmlocalconfig. To set:
- $ zmlocalconfig -e key=val
- To unset:
- $ zmlocalconfig -u key
- Once you set the key you will be able to view it.
- $ zmlocalconfig
- backup_zip_copier_private_blob_zips
- How many zip files to distribute a mailbox's private (unshared) blobs over; default 4 (blobs-1.zip through blobs-4.zip); range 1 to 10,000
- backup_zip_copier_copy_buffer_size
- File copy buffer size; default 16384 (16KB); range 4KB to 1MB
- backup_zip_copier_queue_capacity
- Each zip file gets a queue of files to add. This key sets the queue size. Default is 10. Range is 1 to 10,000.
- backup_zip_copier_deflate_level
- Compression level. Default is -1. (same as in java.util.zip.ZipOutputStream). -1 is same as level 6. This behavior comes from zlib library which the JVM uses to implement zip. Other than the special default value, the level can range from 0 to 9. 0 means no compression. 1 means fastest compression and 9 means best compression.
- backup_disable_shared_blobs
- This one isn't limited to zip backups. When this is set to true, all blobs are backed up as private backups. Default is false.
- backup_debug_use_old_zip_format
- If true, backup will behave like ZCS 5.0.4 and earlier. Shared blobs are never zipped, and private blobs are added to a single blobs.zip file in zip backup. Default is false.
- Use zmlocalconfig. To set:
- http://bugzilla.zimbra.com/show_bug.cgi?id=26624#c6
Need To Write Fewer Files - Add The Zip Option To Your Backup Commands
RFE to make zip option the default for backups:
- "backup: default to the zip option"
There is very little details in the official documentation on this option unfortunately. This does have a really good explanation though:
http://www.zimbrablog.com/blog/archives/2008/08/recent-admin-tidbits-part-1.html
From the administrative manual on the Backup section:
http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.15.1.html
It says,
- "-zip can be added to the command line to zip the message files during backup. Zipping these can save backup storage space."
It's implied that instead of having all the individual message files in the backup that it will bunch them all together into zip files. The body of a shared blob is added once to a shared-blobs zip file, then a small pointer-only entry is added to a mailbox's zip file. Same effect as in non-zipped case. This will be useful when the number of message files is causing disk i/o issues. Maybe your trying to rsync the backup session directories off to another server or your running a third party backup on it to save to tape. The default use of -zip will use compression, if this also causes overhead that you need to avoid you can use the -zipStore option.
Note about -zipStore:
- "when used with the -zip option, it allows the backup to write fewer files (-zip), but not incur the compression overhead as well"
The zip options effect backups that are in blob formats (full's). Incremental backups are bascially redologs, not the full message store of the user. In summary, the zip option will not impact the increment type backups. Auto-group backups are a mixture of both fulls and incrementals.
How To Use As A Default Option?
You'll add the options to the zimbra crontab file. This can be done with the zmschedulebackup command.
Run zmschedulebackup with help option:
zmschedulebackup --help
You'll see:
-z: compress - compress email blobs with zip
It appears that you'll need to manually add the options about -zipStore , if you want it, to the crontab file.
See bug :
http://bugzilla.zimbra.com/show_bug.cgi?id=30981
What Does It Look Like When I Use Zip?
Shared blobs are zipped and blobs (messages) are zipped per root store directory.
mail:~/backup/sessions/full-20080820.160003.770/accounts/115/988/11598896-a89b-4b9d-bedb-1ed1afcb6c87/blobs zimbra$ ls blobs-1.zip blobs-2.zip blobs-3.zip blobs-4.zip
General Backup & Restore Debugging
You'll be monitoring the /opt/zimbra/log/mailbox.log file
Include the -d / --debug option on the CLI for either zmrestore or zmbackup .
To increasing logging for backup/restore-related logs - /opt/zimbra/log/mailbox.log . Enable DEBUG log level for "zimbra.backup" logger in :
- /opt/zimbra/conf/log4j.properties for "temporary" change - until next restart. This could take a couple of minutes before jetty "sees" the changes.
- /opt/zimbra/conf/log4.properties.in for "permament" change that will stick after restart. A restart of jetty/mailbox would be required for this change - zmmailboxctl restart .
log4j.logger.zimbra.backup=DEBUG
For incremental backups, this should log each redolog being copied to the backup and also log which ones will be deleted out of archive directory. Those not deleted are kept because they are newer than 1 hour (default). The kept logs are deleted (but not copied again) during the next incremental backup.
Redolog Files
Redologs Copied To Backup Session And When Deleted
Archived logs that are less than an hour old at the time of incremental backup are copied to the backup but aren't deleted to support post-crash waitset reinitialization mechanism. The interval is set in localconfig key backup_archived_redolog_keep_time, which is in seconds, default=3600.
An Outline Of The Step
- /opt/zimbra/redolog/redo.log (starts and then grows to zimbraRedoLogRolloverFileSizeKB size - default 100MB)
- This flushes to /opt/zimbra/redolog/archive/[file] upon hitting the zimbraRedoLogRolloverFileSizeKB.
- 1 & 2 keep repeating when zimbraRedoLogRolloverFileSizeKB is hit.
- When a backup is done, the archive/* files are copied. The redo.log file is not moved.
- When the backup processes archive/* logs, it first figures out the last sequence copied to backup. All newer logs are copied to the current backup. Then, all logs are deleted except those that are too new, determined by localconfig parameter backup_archived_redolog_keep_time, which defaults to 1 hour. (This is part of the waitset feature.)
- In standard backup mode, only incremental backups move the redologs.
- In auto-grouped mode, every backup is a hybrid of full and incremental and thus redologs are moved.
- When the backup processes archive/* logs, it first figures out the last sequence copied to backup. All newer logs are copied to the current backup. Then, all logs are deleted except those that are too new, determined by localconfig parameter backup_archived_redolog_keep_time, which defaults to 1 hour. (This is part of the waitset feature.)
Redologs And Auto-group In Regards To Backups
http://www.zimbra.com/forums/administrators/21360-auto-grouped-backups.html
Think of auto-grouped mode as a full backup for the scheduled group as well as an incremental (via redologs) for the all other accounts at the same time. Auto-grouped mode automatically pulls in the redologs since the last run, so you get incremental backups of the remaining accounts. Although the incremental accounts captured via the redologs are not listed specifically in the backup account list. This still allows you to do a point in time restore for any account.
If You Have Older Redologs Not Being Deleted
According to the code, only archived logs newer than 1 hour old (default for backup_archived_redolog_keep_time) should remain after an incremental backup. It is a bug if you are seeing older logs sticking around. If so, look at mailbox.log and see if any error was logged. If you enable DEBUG logging for "zimbra.backup" logger in log4j.properties you will see log statements for each copy and deletion.
The zimbraRedoLogDeleteOnRollover variable
zimbraRedoLogDeleteOnRollover shouldn't have an effect on "If you have older redologs not being deleted". By default it's FALSE and affects whether or not stuff makes it into /opt/zimbra/redolog/archive at all. With it set to TRUE there's just /opt/zimbra/redolog/redo.log and it's deleted/not rolled over into archive. As discussed above old redologs are deleted after the incremental; thus if you don't take incremental backups you should set this value to TRUE or periodically script manual deletion of /opt/zimbra/redolog/archive. (And with zimbraRedoLogEnabled FALSE there's no redo.log at all.)
If You Don't Run Incremental Backups Or Don't Need Archive Redologs
You would set zimbraRedoLogDeleteOnRollover to TRUE.
(Auto-Grouped backups you can still leave this to the default of FALSE.)
Redolog Sequence And The Backup Session
Redologs will exist in the incremental backup sessions. The zmbackupquery command will reference the redologs associated with the backup. For example"
[zimbra@mail3 ~]$ ls /opt/zimbra/backup/sessions/incr-20080925.224528.230/redologs/ redo-20080925.213136.165-seq53.log redo-20080925.220726.521-seq55.log redo-20080925.224209.287-seq57.log redo-20080925.215516.450-seq54.log redo-20080925.221749.133-seq56.log [zimbra@mail3 ~]$ zmbackupquery -lb incr-20080925.224528.230 Label: incr-20080925.224528.230 Type: incremental Status: completed Started: Thu, 2008/09/25 18:45:28.230 EDT Ended: Thu, 2008/09/25 18:45:39.099 EDT Redo log sequence range: 53 .. 57 Number of accounts: 2
In the above example, we see the sequence range of "53 .. 57" is referring to the files in the backup session directory called redologs.
RedoLog Variables
Changing Redolog File Size And Location
The /opt/zimbra/redolog/redo.log rolls over when it reaches zimbraRedoLogRolloverFileSizeKB (by default 100mb).
The "roll overs" then goto /opt/zimbra/redolog/archive/
zmprov gacf | grep Redo zimbraRedoLogArchiveDir: redolog/archive zimbraRedoLogDeleteOnRollover: FALSE zimbraRedoLogEnabled: TRUE zimbraRedoLogFsyncIntervalMS: 10 zimbraRedoLogLogPath: redolog/redo.log zimbraRedoLogRolloverFileSizeKB: 102400
Need To Move Redologs Because Partition Getting Full
Let's say you have a partition getting full and you need to move the redolog to another partition or nfs mount temporary to deal with the potential crisis that will happen when the partition becomes full. You'll need to reallocate the complete redolog/ directory and the archive subdirectory to the same partition because the roll over from redo.log to the archive directory happens with a rename function within the java code. This will require downtime since you'll need to move the actual redo.log file and zimbra can't be running while you do this. You can use a symlink to your new partition path. For example:
su - zimbra zmcontrol stop su - ** as root mkdir /data/redolog chown zimbra:zimbra /data/redolog mount /dev/sdb1 /data/redolog mv /opt/zimbra/redolog/* /data/redolog/ ** or use rsync rmdir /opt/zimbra/redolog ln -s /opt/zimbra/redolog /data/redolog ls -laR /data/redolog ** confirm ownership is with zimbra and double check zimbra can write in this directory su - zimbra touch /data/redolog/testfile rm /data/redolog/testfile zmcontrol start
Automatic Deleting Of Redo Logs On Rollover
This variable (zimbraRedoLogDeleteOnRollover) is set TRUE or FALSE.
zmprov gacf | grep zimbraRedoLogDeleteOnRollover
To modify it
zmprov mcf zimbraRedoLogDeleteOnRollover TRUE
Want To See What's In Redolog Files
This is for older versions of ZCS - newer versions should use zmredodump if it's available.
If you suspect there's too much redolog activity during a time window or have another need to inspect the contents of the redolog, dump it and examine it:
$ zmjava com.zimbra.cs.redolog.util.RedoLogVerify /opt/zimbra/redolog/redo.log > out.file
Pick the right redolog file, either redo.log or one of the files under archive/, based on timestamp.
zmplayredo And zmredodump
zmplayredo - Replaying Content From Any Redolog File
zmplayredo is a newer command, first introduced in 5.0.5 I believe. The mailbox has to be stop to run zmplayredo.
The help output from 6.0.8:
$ zmplayredo --help usage: zmplayredo <options> --fromSeq <arg> Replay from this redolog sequence (inclusive) --fromTime <arg> Replay from this time (inclusive) -h,--help Show help (this output) --logfiles <arg> Replay these logfiles, in order --mailboxId <arg> Replay for this mailbox only --queueCapacity <arg> Queue capacity per player thread; default=100 --stopOnError Stop replay on any error --threads <arg> Number of parallel redo threads; default=50 --toSeq <arg> Replay to this redolog sequence (inclusive) --toTime <arg> Replay to this time (inclusive) Specify date/time in one of these formats: 2010/11/19 13:55:08 2010/11/19 13:55:08 802 2010/11/19 13:55:08.802 2010/11/19-13:55:08-802 2010/11/19-13:55:08 20101119.135508.802 20101119.135508 20101119135508802 20101119135508 Specify year, month, date, hour, minute, second, and optionally millisecond. Month/date/hour/minute/second are 0-padded to 2 digits, millisecond to 3 digits. Hour must be specified in 24-hour format, and time is in local time zone.
zmredodump - Replaying Content From Any Redolog File
zmredodump is a newer command and very useful. It does not require mailboxd to be stopped like zmplayredo does.
The help output from 6.0.8:
$ zmredodump --help usage: zmredodump [options] <redolog file/directory> [...] where [options] are: -h,--help show this output --m <arg> one or more mailbox ids separated by comma or white space. The entire list must be quoted if using space as separator. If this option is given, only redo ops for the specified mailboxes are dumped. Omit this option to dump redo ops for all mailboxes. --no-offset don't show file offsets and size for each redo op -q,--quiet quiet mode. Only print the log filename and any errors. This option can be used to verify the integrity of redologs with minimal output. --show-blob show blob content. Item's blob is printed, surrounded by <START OF BLOB> and <END OF BLOB> markers. The last newline before end marker is not part of the blob. Multiple log files/directories can be specified. For each directory, all redolog files directly under it are processed, sorted in ascending redolog sequence order.
Using zmredodump To Get Message Blobs To Inject With zmlmtpinject - RFE
Please see:
- "RFE: zmredodump blobs to single files for zmlmtpinject [for example]"
- See also:
- "RFE: zmplayredo option for --frommailboxId --tomailboxId"
Expand Ability To Parse Redologs - RFE
Please see:
- "expand ability to parse redologs for recovery purposes - zmredodump / zmplayredo"
Getting A Sequence or Time Variable For Restore Or Replay
You can see the changes within the redo logs with the command below. You can point it to any redolog.
zmjava com.zimbra.cs.redolog.util.RedoLogVerify /opt/zimbra/redolog/redo.log > out.file
You'll get output like this:
VERIFYING: redo.log HEADER ------ sequence: 59 open: 1 filesize: 512 serverId: d5c5d6a7-b82f-4c29-b0cd-91818057196b firstOpTstamp: 1222385426273 lastOpTstamp: 1222385426273 version: 1.22 ------ txn 1222383600.1 [PurgeOldMessages] ver=1.22, tstamp=1222385426273, change=20200, mailbox=1 txn 1222383600.1 [CommitTxn] ver=1.22, tstamp=1222385426329, mailbox=1, txnType=PurgeOldMessages txn 1222383600.2 [PurgeOldMessages] ver=1.22, tstamp=1222385486337, change=13500, mailbox=3 txn 1222383600.2 [CommitTxn] ver=1.22, tstamp=1222385486351, mailbox=3, txnType=PurgeOldMessages txn 1222383600.3 [PurgeOldMessages] ver=1.22, tstamp=1222385546357, change=20201, mailbox=1 txn 1222383600.3 [CommitTxn] ver=1.22, tstamp=1222385546383, mailbox=1, txnType=PurgeOldMessages txn 1222383600.4 [PurgeOldMessages] ver=1.22, tstamp=1222385606391, change=13501, mailbox=3 txn 1222383600.4 [CommitTxn] ver=1.22, tstamp=1222385606404, mailbox=3, txnType=PurgeOldMessages txn 1222383600.5 [PurgeOldMessages] ver=1.22, tstamp=1222385666416, change=20202, mailbox=1 txn 1222383600.5 [CommitTxn] ver=1.22, tstamp=1222385666428, mailbox=1, txnType=PurgeOldMessages txn 1222383600.6 [PurgeOldMessages] ver=1.22, tstamp=1222385726435, change=13502, mailbox=3 txn 1222383600.6 [CommitTxn] ver=1.22, tstamp=1222385726459, mailbox=3, txnType=PurgeOldMessages txn 1222383600.7 [PurgeOldMessages] ver=1.22, tstamp=1222385786476, change=20203, mailbox=1 txn 1222383600.7 [CommitTxn] ver=1.22, tstamp=1222385786486, mailbox=1, txnType=PurgeOldMessages txn 1222383600.8 [PurgeOldMessages] ver=1.22, tstamp=1222385846493, change=13503, mailbox=3 txn 1222383600.8 [CommitTxn] ver=1.22, tstamp=1222385846506, mailbox=3, txnType=PurgeOldMessages txn 1222383600.9 [PurgeOldMessages] ver=1.22, tstamp=1222385906739, change=20204, mailbox=1 txn 1222383600.9 [CommitTxn] ver=1.22, tstamp=1222385906775, mailbox=1, txnType=PurgeOldMessages txn 1222383600.10 [PurgeOldMessages] ver=1.22, tstamp=1222385966944, change=13504, mailbox=3 txn 1222383600.10 [CommitTxn] ver=1.22, tstamp=1222385966963, mailbox=3, txnType=PurgeOldMessages txn 1222383600.11 [PurgeOldMessages] ver=1.22, tstamp=1222386026972, change=20205, mailbox=1 txn 1222383600.11 [CommitTxn] ver=1.22, tstamp=1222386026990, mailbox=1, txnType=PurgeOldMessages ...
How Do I Figure Out Which Sequence or Time Variable To Use For Restore Or Replay
- In 5.0.10+ we'll have a CLI wrapper (zmredodump) with a slightly different command line syntax, but the below long syntax works in earlier versions.
To locate the correct restore-to time, you have to start with an approximate time the message was added/deleted. Look at the redolog files. The filename contains the GMT time when the file was rolled over, which is roughly the tstamp of the last operation in the file. If your time data is accurate you can find the specific file. Or you have a range of files to examine.
Use the redolog verify tool to dump the contents into text form, the -m / --message option to show message body data:
zmjava com.zimbra.cs.redolog.util.RedoLogVerify -m <filename or directory> ... > out.file
If the message was deleted and you don't know the id, you must go by some other clue such as the subject. Search the file to locate your message. You can cut/paste the message and lmtp-inject it to recover the message. No need to go through with a restore if this is all you needed.
Are You Messages Really Gone - Things To Check If zmplayredo Isn't Doing What You Expect
Here's something I found out testing zmplayredo for a customer case. Testing on a ZCS 6.0.8 single ZCS server.
Created a test account and sent it one message that is in the Inbox. I delete the msg in zwc but don't purge the Trash - msg is in Trash now.
Log events of above action:
2010-10-27 15:07:13,375 INFO [btpool0-3://192.168.0.71/service/soap/ConvActionRequest] [name=ajcody@mail71.DOMAIN.com;mid=17;ip=192.168.0.17;ua=ZimbraWebClient - FF3.0 (Linux)/6.0.8_GA_2661;] mailop - Moving VirtualConversation (id=-257) to Folder Trash (id=3). Affected message ids: 257.
Stop mailboxd so I can use zmplayredo and then start mailboxd back up after zmplayredo is finished :
zmmailboxdctl stop zmplayredo --mailboxId 17 --logfiles /opt/zimbra/redolog/archive/* --fromTime 20101027090000 --toTime 20101027140000 zmmailboxdctl start
Log event for above:
2010-10-27 15:07:50,383 INFO [main] [] mailop - Adding Message: id=257, Message-ID= <1604937630.920784.1288201208382.JavaMail.root@corp.zimbra.com>, parentId=-1, folderId=2, folderName=Inbox. 2010-10-27 15:07:50,404 INFO [main] [] RedoableOp - Message 257 is already in mailbox 17
Log into ZWC with the test account. The msg is not in the Inbox, but it's still in Trash folder. I purge it from Trash.
Log event for deletion.
2010-10-27 15:09:38,761 INFO [btpool0-2://192.168.0.71/service/soap/ConvActionRequest] [name=ajcody@mail71.DOMAIN.com;mid=17;ip=192.168.0.17;ua=ZimbraWebClient - FF3.0 (Linux)/6.0.8_GA_2661;] mailop - Deleting Message (id=257).
Then I redo the stop/start of mailboxd and zmplayredo again.
zmmailboxdctl stop zmplayredo --mailboxId 17 --logfiles /opt/zimbra/redolog/archive/* --fromTime 20101027090000 --toTime 20101027140000 zmmailboxdctl start
Log event for the above:
2010-10-27 15:10:29,192 INFO [main] [] mailop - Adding Message: id=257, Message-ID= <1604937630.920784.1288201208382.JavaMail.root@corp.zimbra.com>, parentId=-1, folderId=2, folderName=Inbox.
Log back into ZWC with test account and now I can confirm that the msg is not in Trash and it is now showing in Inbox.
Gap In Redo Log
The error message from either a backup or restore command:
- "Error occurred: Found gap in redo log sequence; missing 5965 through 6149;
- To avoid future restore problems, discard all existing backups and take a
- full backup of all accounts; If this error occurred during restore,
- try the --ignoreRedoErrors option"
The output is pretty accurate in how to handle the situation.
If you get the error during a backup, the recommendation is to move your old backups out. The directories in /opt/zimbra/backup/sessions/* . You'll want to keep them around just in case and then proceed to do a full backup.
If you get the error during a restore, you would add the flag --ignoreRedoErrors to your restore command.
Another possible related issue is if your /tmp or /opt/zimbra/redolog/ is filling up.
Error Executing redoOp
Errors with restores that involve the message 'error executing redoOp' will not show up in the admin console but will when you attempt the restore from CLI. This can also be the cause when you use the RestoreToTime option from the admin console and it doesn't seem to work correctly - the restore stopping prematurely from the specified date/time.
I created the following RFE in regards to the admin console issue:
- "Include --ignoreRedoErrors option and error feedback in Admin Console for restore"
- http://bugzilla.zimbra.com/show_bug.cgi?id=52358
- This could explain why your restore to time isn't working in the Admin Console but does from CLI when you see an error about redologs and then reattempt restore with the --ignoreRedoErrors and it works.
- http://bugzilla.zimbra.com/show_bug.cgi?id=52358
Another RFE that was made but marked as 'WONTFIX' that gives a background story to the issue is:
- "Need more robust redolog serialization format"
When you hit this error with your backup data during restore attempts, there's basically only a couple of options to handle this that are recommended by support:
- Try to get full backups of your accounts or the accounts in question and then test against them and the preceding incrementals after the full.
- Attempt restores via the CLI using the additional option of : --ignoreRedoErrors
- Similar to the various steps described here - Ajcody-Backup-Restore-Issues#Restore_To_Time_Problems - but with the --ignoreRedoErrors option included.
- Do your restore against the latest full backup of the account in question and then use the zmplayredo command against the redologs in the incrementals and/or the /opt/zimbra/redologs/* directory . This will give you more control to walk the restored account up to the point in time you want it at. One should really read through the whole section above, Ajcody-Backup-Restore-Issues#Redolog_Files , to understand the whole concept of redologs and then the use of zmplayredo.
Generally, "fixing" the redolog itself is not an option.
Why Do My Fulls Not Report All Accounts?
Are you sure it was a full backup that was ran or just a full session that was generated from your incremental backup job? When an incremental is ran, it will create a "full" session for any new accounts it discovers after the last actual full backup job.
For example, here's a full session that was created by an incremental backup job:
Label: full-20081010.060126.559 Type: full Status: completed Started: Fri, 2008/10/10 01:01:26.559 CDT Ended: Fri, 2008/10/10 01:01:28.988 CDT Redo log sequence range: 705 .. 705 Number of accounts: 1 Label: incr-20081010.060009.420 Type: incremental Status: completed Started: Fri, 2008/10/10 01:00:09.420 CDT Ended: Fri, 2008/10/10 01:01:26.413 CDT Redo log sequence range: 700 .. 704 Number of accounts: 392 [zimbra@servername ~]$ du -sh /opt/zimbra/backup/sessions/full-20081010.060126.559 1.2M /opt/zimbra/backup/sessions/full-20081010.060126.559 [zimbra@servername ~]$ du -sh /opt/zimbra/backup/sessions/incr-20081003.060010.622 452M /opt/zimbra/backup/sessions/incr-20081003.060010.622
Notice the Start and End times, this will show that the full is related to the incremental job.
You'll want to run zmbackupquery against your full labels to see your "main" full backup session - assuming you can't simply guess based upon the cron entry for it [ su - zimbra ; crontab -l | grep backup ]. For example, to see all your fulls from today's date back to October 01, 2008 and the accounts within each session - you would do:
zmbackupquery -v --from 20081001.000000 --type full
The -v flag outputs the accounts, the --from uses YearMonthDay.HourMinuteSecond , and the --type can be full or incremental. To just see one particular sessions date, you would use the lb [label] flag:
zmbackupquery -v -lb [your full label, ex. full-20081001.000000]
Issues After /opt/zimbra/backup Became Full
Bugs/RFE's on this issue:
- "backup should give better error when disk is full"
- Was marked dup'd of above, "Issues when backup partition becomes full"
See alsoAjcody-Backup-Restore-Issues#Possible_Issue_That_A_Failed_Or_Interrupted_Backup_Causes and the Bugs & RFE's under that section.
You can run into numerous issues if you allow your backup directory to become full.
Confirm your /opt/zimbra/backup/accounts.xml is being updated after a backup. You might see that the newer account.xml* file is accounts.xml.new . This is a sign of problems.
Confirm that the files in /opt/zimbra/backup/tmp/* don't have 0 byte lengths. There might be files like 1.xml and 3.xml in there. If they show 0 bytes, you need to remove them. The backup/restore commands if the file exist and they are empty. Your errors might look like this:
[zimbra@mailb ~]$ zmrestore -a USER@DOMAIN -ca -pre restore_ Error occurred: system failure: Unable to parse XML file /opt/zimbra/backup/tmp/restore/shared_blobs/1.xml
If you tried doing restores (redirected -ca -pre) before to clear up the above issues, you might find you can't do a successful restore AND you can't delete the account afterwards.
If you get errors like :
zmprov da restore3_tester3@XXXXXX.edu ERROR: service.FAILURE (system failure: writing new mailbox row for account 89e7d9f4-013e-4cf1-a352-7b2f0a00d5af) zmprov gmi restored_USER ERROR: service.FAILURE (system failure: writing new mailbox row for account 56a7f654-f85b-45cc-931a-81d9bb9076bf)
You'll need to delete the account via a ldapdelete command.
Query And Stopping A Backup or Restore In Progress
Please see the url below for Backup In Progress - topic name "Aborting Full Backup In Progress":
- A matter of using zmbackupquery & zmbackupabort .
Please see url below for Restore In Progress - topic name "Stopping a Restore Process" :
- A matter of using : zmbackupabort -r
- Latest ZCS Admin Guide Table of Contents, search for the topic name mentioned above.
- ZCS 4.5.10 Admin Guide with static links to back section, you should review info in latest admin guide though.
- Note - what is absent is an easy way to query restores in progress though. Please see:
- "Status for Zmrestore"
- http://bugzilla.zimbra.com/show_bug.cgi?id=47238
- Currently, status of a restore can only be monitored via the logging events in the mailbox.log file.
- "Status for Zmrestore"
A -del Delete In Progress
Please see the bug / rfe's I filed
- "need way to kill/recover from zmbackup -del when in progress"
- "zmbackup -del shouldn't prevent zmbackup -f/-i from running for ALL cases"
zmbackupabort syntax
From ZCS 6.0.8
$ zmbackupabort -h usage: zmbackupabort <options> -d,--debug Display diagnostics for debugging purposes. -h,--help Displays this help message. -lb,--label <arg> Label of full backup set to abort. -r,--restore Abort the restore in progress. -s,--server <arg> Mail server hostname. Default is localhost. -t,--target <arg> Backup target location (default <zimbra_home>/backup).
zmbackupquery syntax
From ZCS 6.0.8
$ zmbackupquery -h usage: zmbackupquery <options> -a,--account <arg> Account email addresses seperated by white space or "all" for all accounts. -d,--debug Display diagnostics for debugging purposes. --from <arg> List backups whose start date/time is at or after this date/time. -h,--help Displays this help message. -lb,--label <arg> The label of full backup to query. -s,--server <arg> Mail server hostname. Default is localhost. -t,--target <arg> Backup target location (default <zimbra_home>/backup). --to <arg> List backups whose start date/time is at or before this date/time. --type <arg> Backup set type to query. "full" or "incremental"; both if unspecified. -v,--verbose Show account list in each backup. Specify date/time in one of these formats: 2010/11/19 14:06:22 2010/11/19 14:06:22 923 2010/11/19 14:06:22.923 2010/11/19-14:06:22-923 2010/11/19-14:06:22 20101119.140622.923 20101119.140622 20101119140622923 20101119140622 Specify year, month, date, hour, minute, second, and optionally millisecond. Month/date/hour/minute/second are 0-padded to 2 digits, millisecond to 3 digits. Hour must be specified in 24-hour format, and time is in local time zone.
Restore To Time Problems
There's seems to be some syntax issues when using this variable. Please review the following to confirm your syntax.
http://wiki.zimbra.com/index.php?title=CLI_zmrestore_restoreToTime_Network_Edition_only
The gist, you MUST use the -lb full-200xxxxxx option when your trying to restore anything that ISN'T meant to include the latest information for the mailbox. The -lb argument should specify a full backup that took place prior to the time of the backup you wish to restore.
Find Out What Backup Session Labels You Need First
To find out what backups are associated with a particular account, you would do the following :
zmbackupquery -a user@domain
You'll want to note what is the first full that occurs before the point in time you want to restore. And then the incremental that follows right after your point in time.
Backup label (-lb) for fulls can be found by doing [include the -v option if you want to see a listing of the user accounts within the backups] :
zmbackupquery --type full
Backup labels (-restoreToIncrLabel) for incrementals can be found by:
zmbackupquery --type incremental
Command Syntax Example For Restores On The CLI
Example to full label and stop :
zmrestore -a USER@DOMAIN.com -lb full-20080726.050017.306 -rf -ca -pre restore_
Example to incremental label and stop :
zmrestore -a USER@DOMAIN.com -restoreToIncrLabel incr-20080731.060007.644 -lb full-20080726.050017.306 -br -ca -pre restore_
Example to specific time and stop :
zmrestore -a USER@DOMAIN.com -restoreToTime 20080801011800 -lb full-20080726.050017.306 -br -ca -pre restore_
Important Options You Might Want Or Need To Include
--ignoreRedoErrors : If you attempt a restore and you see an error about problems related to playing the redolog, you'll want to run the restore command again and include this option.
--skipDeletes : Please see http://bugzilla.zimbra.com/show_bug.cgi?id=31824#c5 for details on this.
-t /path/to/backup_dir : If you are restoring from another backup directory besides your current default path.
Variables that are asking for TIME rather than LABELS should follow this syntax (from zmrestore --help):
Specify date/time in one of these formats: 2008/08/06 09:55:50 2008/08/06 09:55:50 572 2008/08/06 09:55:50.572 2008/08/06-09:55:50-572 2008/08/06-09:55:50 20080806.095550.572 20080806.095550 20080806095550572 20080806095550 Specify year, month, date, hour, minute, second, and optionally millisecond. Month/date/hour/minute/second are 0-padded to 2 digits, millisecond to 3 digits. Hour must be specified in 24-hour format, and time is in local time zone.
Bugs And RFE's To Review
Update - Bug/RFE's I Filed Against ZCS 8.6
- [story] Ability to search data within backup and do "item" restores or identify locations of search results
- admin console backup label view doesn't list accounts in the all accounts tab
- admin console restore - doesn't autocomplete / suggest account matches when filling out email address box
- document new restore functions / options with ZCS 8+ for admin console restore
- admin console restore - rename "Selected Servers" panel to "Restore Options"
- admin console restore - if only one mailstore in env. then state such in second panel of restore about "server for the restored accounts"
- admin console restore - expand restore To options - To full backup label, To incremental target
- admin console restore - "restore to the latest backup" incorrectly described / broken
- admin console restore - unable to restore individual accounts [sort of]
- admin console restore - reuse GAL/Contact Picker Window for "restore individual accounts"
Older Bug/RFE's
- zmrestore using restoreToTime option restores data after the specified time
- See: http://bugzilla.zimbra.com/show_bug.cgi?id=15320
- Resulting in documentation changes : CLI_zmrestore_Network_Edition_only
- See: http://bugzilla.zimbra.com/show_bug.cgi?id=15320
- zmrestore -restoreToTime should fail if no backup label is passed
- See: http://bugzilla.zimbra.com/show_bug.cgi?id=28320
- "unable to restore to point in time/incremental" Admin GUI
- See: http://bugzilla.zimbra.com/show_bug.cgi?id=28320
- Admin console issues and other details I've posted are in this bug:
- http://bugzilla.zimbra.com/show_bug.cgi?id=27746
- Added for search terms, admin console can't restore to point in time
- Include --ignoreRedoErrors option and error feedback in Admin Console for restore
- http://bugzilla.zimbra.com/show_bug.cgi?id=52358
- This could explain why your restore to time isn't working in the Admin Console but does from CLI when you see an error about redologs and then reattempt restore with the --ignoreRedoErrors and it works.
- http://bugzilla.zimbra.com/show_bug.cgi?id=52358
Restore An Individual Message
The zmrestore command is at a mailbox level.
An RFE was filed already to expand this. It is currently targeted for the Helix release.
- "More Granular Restore: per folder & per-message"
- http://bugzilla.zimbra.com/show_bug.cgi?id=8849
- Note - the above was marked as a duplicate of our work on "dumpster" option. I disagree with this choice and created the following RFE below.
- http://bugzilla.zimbra.com/show_bug.cgi?id=8849
- "Ability to search data within backup and do "item" restores or identify locations of search results"
A way around the current limitations would be to use lmtpinject. Please see the following for details on that:
- http://www.zimbra.com/forums/installation/12617-recover-data-store-folders.html#post64962
- http://www.zimbra.com/forums/administrators/729-using-zmlmtpinject.html#post4017
The difficultly would be determining the message your trying to find within your backups that was "deleted" in prod.
User Deleted A Bunch Of Data And Notified You Hours Later Wanting It Restored
- To determine the time of the delete, use the zmredodump command.
- You'll use this "time" for the restore command.
- Example:
- $ zmprov gmi USER@DOMAIN.com ::: gives me mailboxid 17
- $ zmredodump --show-blob --m 17 /opt/zimbra/redolog/ | grep Delete ::: returns:
- [0000f311 - 0000f350: 64 bytes; tstamp: 2010/11/19 11:21:30.852 CST] txn 1290033504.2544 [DeleteItem] ver=1.28, tstamp=1290187290852, change=3913, mailbox=17, ids=[304, 308], type=5
- [0000f351 - 0000f382: 50 bytes; tstamp: 2010/11/19 11:21:30.869 CST] txn 1290033504.2544 [CommitTxn] ver=1.28, tstamp=1290187290869, mailbox=17, txnType=DeleteItem
- See Ajcody-Backup-Restore-Issues#Restore_To_Time_Problems about setting up the proper restore syntax.
- You'll want to include the -ca -pre restore_ options since we'll first get the old data into a temporary account.
- Remember to use zmbackupquery -a USER@DOMAIN.com --type full to find what full label will be needed for the restore.
- Also, if you need/want the incremental label, do a zmbackupquery -a USER@DOMAIN.com --type incremental .
- Example:
- $ zmbackupquery -a USER@DOMAIN.com --type full ::: shows the latest as being:
- Label: full-20101119.070018.607
- Type: full
- Started: Fri, 2010/11/19 01:00:18.607 CST
- Ended: Fri, 2010/11/19 01:00:29.751 CST
- Acct ID: f33d6daf-8875-4496-8bee-6df345f295e7
- $ zmbackupquery -a USER@DOMAIN.com --type full ::: shows the latest as being:
- Example:
- Do the restore now with the proper information for the variable flags.
- Note: You might need to include -br , -rf , or neither depending on the time frames involved.
- Example:
- $ zmrestore -a USER@DOMAIN.com -restoreToTime 20101119112000 -lb full-20101119.070018.607 -ca -pre restore_
- It would be best to log into the restore_USER@DOMAIN.com account to confirm the data is as you expect it.
- Use zmmailbox with the getRestUrl option against the "restore_USER" account now to export the data.
- Examples:
- Export ALL the 'old' data from the restored account
- $ /opt/zimbra/bin/zmmailbox -z -m restore_USER@DOMAIN.com getRestURL "//?fmt=tgz" > /var/tmp/user-export.tgz
- You can also export SOME of the 'old' data from the restore account using other options. One option is with the before and after variables. NOTE - We have to set the query string as a variable to get around some of the shell issues.
- For example:
- $ query='before:11/20/2010 after:11/1/2010'
- ZCS5 might require you to have a %20 rather than the actual space character.
- $ /opt/zimbra/bin/zmmailbox -z -m restore_USER@DOMAIN.com getRestURL "//?fmt=tgz&query=$query" > /var/tmp/user-export.tgz
- Note': the $query above will be the expansion of the variable you set prior in the shell, query='before:11/20/2010 after:11/1/2010 .
- $ query='before:11/20/2010 after:11/1/2010'
- Export ALL the 'old' data from the restored account
- Examples:
- Use zmmailbox with the postRestUrl option to IMPORT the RESTORE_USER account into the "USER" account now with appropriate options.
- Examples:
- $ /opt/zimbra/bin/zmmailbox -z -m USER@DOMAIN.com postRestURL “//?fmt=tgz&resolve=replace” /var/tmp/user-export.tgz
- Note : A critical option in the above command is the &resolve=replace one. There are various ways you can handle the importing of data. Please review the following to determine what is best for you needs.
- Examples:
Restore Deleted Items - skipDeletes Option - ZCS6+
- "skip delete operations flag to zmrestore"
From the RFE comment:
Added new option --skipDeletes to zmrestore. If specified, skip over delete operations during redo replay. Delete ops are: DeleteItem (hard delete) DeleteMailbox EmptyFolder MoveItem, if moving item to Trash folder PurgeImapDeleted PurgeOldMessages Skipping these deletes can lead to other problems later, such as conflicting paths, but it is assumed the priority is recovering as much data as possible when using this option.
Restore Account Not Yet In Backups
Please see:
- "add ability to restore accounts not yet backed up (but still in redologs)"
How To Restore To Events Within The Same Day
Work in progress and investigation.
Users Trash Items
User Ability To Recover Trash Purge
Please see RFE:
- "RFE: Recover Deleted Items ability for users"
Retention Policy About Purges
Please see Mailbox_Purge
Can't Restore Or Find An Account That Was Renamed
When an account is "renamed", the old account name will no longer be "found" is your "default" type restore or backup queries. This can cause some confusions when one needs to restore to a time frame of when the account was under it's older name.
Restoring From CLI
First, identify what sessions labels hold the differing account names:
zmbackupquery -a USER@DOMAIN
zmbackupquery -a USER-RENAMED@DOMAIN
You should know have what you need to do follow the steps in the Ajcody-Backup-Restore-Issues#Restore_To_Time_Problems section.
Restoring From Admin Console GUI
I'll give a detail explanation of the situation when working around restores of renamed accounts in the admin console [web GUI]:
- If you in the GUI goto the "Restore" button, it first asks for an account rather than giving an option for date/time/session. I think you already stated, that "renamed" accounts don't show up in this query window. Therefore, one wouldn't really progress to the next window that would allow you to change the backup session label.
- They way you get around this is, you actually double click on the full session listing that you see on the backup page in the admin UI. This will bring you to another page, that is specific to that session. In there, you should see the old account name prior to the rename. You can then highlight that account listing and click on the "Restore" button. This will bring up the restore dialog, which will now have the date/time/session label auto-filled out.
Quota Is Stopping A Redirected Restore
Update
- "accounts with quota set by COS fail to restore when over default quota"
Reasoning for need, maybe the msg files coming from the restore are no longer "shared message blobs" and therefore increase the mailbox to a size it wasn't in the past. Changes to HSM maybe?
I think I'll need to create a RFE about adding an option to the zmrestore command to also include an option to set the COS value on a created account. Until then... Create a new COS and set it up to NOT have any quota. Once you kick off the backup and you see the account is created you can then apply the COS to that account. Call the cos something like no-quota. Here's the steps below.
To copy your cos (assuming your quota cause is the default one, change default to match your production cos your using).
To see all cos's
zmprov gac
Copies cos called default to cos named no-quota
zmprov cpc default no-quota
To remove quota to the new cos no-quota
zmprov mc no-quota zimbraMailQuota 0
To confirm
zmprov gc no-quota | grep -i quota
Now you would start your redirected restore and once you see the account is created, run the below in a separate shell. (example)
zmrestore -a USER@DOMAIN.com -restoreToTime 20080801011800 -lb full-20080726.050017.306 -br -ca -pre restore-
Once restore has kicked off - To apply the no-quota cos to the restored account:
zmprov sac restore-USER@DOMAIN.com no-quota
Restore Of Non-Account Items - Example - COS DL Etc
Cos and DL's are ldap entries basically.
From Backup and Restore
- zmrestoreldap. This command restores the complete LDAP directory server, including accounts, domains, servers, COS and other data."
You'll see in a DR process, Network_Edition_Disaster_Recovery , that the zmrestoreldap is done before the zmrestoreoffline .
- zmrestoreldap doesn't have options that allow specific items to be restored (COS, DL's, etc.). It only has option for named accounts (-a). One could try a ldapadd with a ldif of the COS or DL details. One could also take the information on the COS or DL within the ldap file in the backup session to at least have all the variables to manually add it back (via the zmprov command). Your looking at the backups on the LDAP master if your in a multi-server configuration.
/opt/zimbra/backup/sessions/full-xxxxxx/ldap/ldap.bak
Start of ldap entry example to search for:
- Cos example
- dn: cn=default,cn=cos,cn=zimbra
- DL example
- dn: uid=dl-group,ou=people,dc=mail,dc=domain,dc=com
To compare a current DL with past details, just save out the ldap entry from the backup to a txt file. And then do:
zmprov gdl maillist@domain.com
Make the necessary changes after comparing the two.
Restoring A Calendar (ics)
There seems to be a bug or odd expectation on how this command is currently working. If the appointment exists in the Calendar and the time is different with the same appointment in the ics file your importing - the time of the appointment will not change to the imported ics one. If you delete the event first, then the imported appointment will reflect the correct time.
Here's what I did to reproduce this situation. It seems this has been true for sometime, customer was on 4.5.11 and I was on 5.0.8
Created test account and made two appointments on friday - 9am and 4pm. Did a full backup. Restored test account to restore_user Ran : zmmailbox -z -m user@domain.com gru /Calendar > /tmp/calendarA.ics zmmailbox -z -m restore_user@domain.com gru /Calendar > /tmp/calendarB.ics And then diff /tmp/calendarA.ics /tmp/calendarB.ics [no differences] Now some tests. As user, I deleted the two appointments and then: zmmailbox -z -m user@domain.com pru /Calendar /tmp/calendarB.ics Refreshed Calendar as User in webclient. 9am and 4pm appointment shows up. I then moved in the webclient the 9am appointment to 11am Did another restore: zmmailbox -z -m user@domain.com pru /Calendar /tmp/calendarB.ics Refreshed Calendar as User in webclient. 11am and 4pm appointment shows up. ** The restore did not move the 11am appointment back to the 9am slot as in /tmp/calendarB.ics ** Assumption, this process will not over-write an appointment if it's there - it does not look to the time. Let's do a diff of the state of the calendar zmmailbox -z -m user@domain.com gru /Calendar > /tmp/calendarC.ics diff /tmp/calendarB.ics /tmp/calendarC.ics The DTSTAMP and SEQUENCE shows the difference in the time. If I delete the 11am appointment and then do the calendarB.ics restore the appointment shows up again at 9am.
I see this same behavior if I also use the web interface to export/import the calendar between the restored account and user one. Even when I import it into a NEW calendar, it even changes the two appointments to reflect the new calendar rather than the default one.
One Fix
One fix, if the situation allows, is to purge the current Calendar and then import the full ics file. This would be done like this:
zmmailbox -z -v -m ajcody@mail3.internal.homeunix.com ef /Calendar * ef is for emptyFolder zmmailbox help folder *webclient shows all events gone but Calendars are still listed. zmmailbox -z -v -m ajcody@mail3.internal.homeunix.com pru /Calendar /tmp/calendarB.ics *webclient show all events with times as expected.
Second Fix
Haven't tried yet, but someone said you should be able to adjust the SEQUENCE number in the appointment and the import process will use that data (data/time) of the newer sequence number appointment.
Archiving User Accounts Out Of Production Use In Zimbra
Backup - Restore Method
To "archive" the user data with the ability of Zimbra later restoring if needed, one would rely on the backup/restore tools.
For example:
zmbackup -f -a user@domain
If you want to make this more "intelligent" later, one could do this:
mkdir /nfs-mount/archived-users/[user-name]
zmbackup -f -a userA@domain -t /nfs-mount/archived-users/userA-domain/
This would give some "intelligent" information for recovery later. Confirms it was an archived account, shows the "real" username, the date of the full backup, and so forth.
ls /nfs-mount/archived-users/userA-domain/ full-20081104.131643.006 ls /nfs-mount/archived-users/userA-domain/full-20081104.131643.006 accounts/ session.xml shared_blobs sys
If one would need to "restore" the account later, it would consume a license if the account was "deleted".
zmrestore -a userA@domain -ca -pre restored- -t /nfs-mount/archived-users/userA-domain/ -lb full-20081104.131643.006
Setup A Secondary Zimbra Box For Restores Of Archive Accounts
Your Zimbra license can be installed on multiple machines. One idea that might prove useful in handling these "archive" accounts for those situations when you need to investigate something is to setup a "archive" Zimbra box. You'll want to isolate this box from any "production" use. It will need to be configured to have the "domains" of the archive accounts. You can then use this box to restore the "archive" account and then use the administration tools to investigate the user data.
Use Of REST And Other Tools For Specific User Data
The following page, User_Migration , will shows numerous examples of how to export different types of data from a user account into a neutral file format that one could use for "archive" purposes.
Use Of The REST Command To Export ALL User Data - Version Dependant, 5.0.9+ [I believe]
Example of the commend syntax:
/opt/zimbra/bin/zmmailbox -z -m user@domain.com getRestURL “//?fmt=tgz” > /tmp/account.tgz
Backup And HSM
Please see Bugs/RFE:
- "RFE: HSM and backup should not run at the same time if initated."
Using zmprov -l gaa To Create User Listing for zmbackup & zmrestore -a option
For example, see below. Note the use of the egrep -v option below, this would remove any matches from the grep. Useful if you want to only backup certain domains for example.
[zimbra@ldap2 sessions]$ zmbackup -f -a `zmprov -l gaa | egrep -v "ham|spam|virus|admin|galsync"` full-20150130.210821.124 [zimbra@ldap2 sessions]$ zmbackup -f -a `zmprov -l gaa` full-20150130.210922.637 [zimbra@ldap2 sessions]$ zmbackupquery -lb full-20150130.210821.124 -v Label: full-20150130.210821.124 Type: full Status: completed Started: Fri, 2015/01/30 16:08:21.124 EST Ended: Fri, 2015/01/30 16:08:50.181 EST Redo log sequence range: 23 .. 23 Number of accounts: 3 out of 3 completed Accounts: archive-search-results@ldap2.zimbra.DOMAIN.com: completed user1-20150115@ldap2.zimbra.DOMAIN.com.archive: completed user1@ldap2.zimbra.DOMAIN.com: completed Total space: 14020MB Free space: 5977MB [zimbra@ldap2 sessions]$ zmbackupquery -lb full-20150130.210922.637 -v Label: full-20150130.210922.637 Type: full Status: completed Started: Fri, 2015/01/30 16:09:22.637 EST Ended: Fri, 2015/01/30 16:09:33.083 EST Redo log sequence range: 23 .. 23 Number of accounts: 9 out of 9 completed Accounts: admin-20150122@ldap2.zimbra.DOMAIN.com.archive: completed admin@ldap2.zimbra.DOMAIN.com: completed archive-search-results@ldap2.zimbra.DOMAIN.com: completed galsync.tfnapjb9@ldap2.zimbra.DOMAIN.com: completed ham.yeygcogdd2@ldap2.zimbra.DOMAIN.com: completed spam.dpxyqjnm6t@ldap2.zimbra.DOMAIN.com: completed user1-20150115@ldap2.zimbra.DOMAIN.com.archive: completed user1@ldap2.zimbra.DOMAIN.com: completed virus-quarantine.wfkd4vvm1g@ldap2.zimbra.DOMAIN.com: completed Total space: 14020MB Free space: 5977MB
Scripting Out Individual Backups Of Accounts
If you want to do individual backups of accounts using a for-loop, for example, you might want to include the -sync option from zmbackup. zmbackup without this will normal give an error as it passing the next zmbackup command stating that there's a current backup in progress.
Example command in some for-loop script is [without the -sync option]:
sudo -u zimbra /opt/zimbra/bin/zmbackup --fullBackup --account "$acct" --server "$host" --target "$dest"
zmbackup --help shows:
--sync -sync Runs full backup synchronously.
The admin guide mentions:
- "Full backup is usually run asynchronously. When you begin the full backup, the label of the ongoing backup process is immediately displayed. The backup continues in the background. You can use the zmbackupquery command to check the status of the running backup at any time."
I couldn't find any other indication beyond that to explain in more details the purpose of that flag. But from what is stated above, it does look like the -sync flag will resolve the issues of "backup in progress" when scripting out multiple zmbackup commands.
If not, you could query for "Status: in progress" from the zmbackupquery command.
You can give the zmbackupquery command flags for date/time, label, account, -t target, and so forth [Do a zmbackupquery --help to see the options format].
Finding Message Blob In Users Backup
Get the zimbraId of the user, this is what is used in the backup session directory layout:
$ zmprov ga admin@`zmhostname` zimbraId # name admin@zcs806.DOMAIN.com zimbraId: e46a828b-cdda-4635-98ab-31b8ac0129b6
Get the mailboxId, this is what is used in the zmvolume primary message volume directory layout:
$ zmprov gmi admin@`zmhostname` mailboxId: 1 quotaUsed: 77794874
Finding a message in the primary message volume:
$ cd /opt/zimbra/store/0/1/msg/9/ $ ls -la 40511-49376.msg -rw-r----- 1 zimbra zimbra 1372 Mar 16 02:01 40511-49376.msg
Finding the users directory in a full backup session directory layout:
$ find /opt/zimbra/backup/sessions/full-20140426.080019.153/ -name e46a828b-cdda-4635-98ab-31b8ac0129b6 -print /opt/zimbra/backup/sessions/full-20140426.080019.153/accounts/e46/a82/e46a828b-cdda-4635-98ab-31b8ac0129b6
Locating the message in the backup session - this example is where the backups are using the zip option:
$ cd /opt/zimbra/backup/sessions/full-20140426.080019.153/accounts/e46/a82/e46a828b-cdda-4635-98ab-31b8ac0129b6/blobs/ $ ls blobs-1.zip blobs-2.zip blobs-3.zip blobs-4.zip $ for i in `ls /opt/zimbra/backup/sessions/full-20140426.080019.153/accounts/e46/a82/e46a828b-cdda-4635-98ab-31b8ac0129b6/blobs/`; do echo $i ; unzip -l $i | grep 40511-49376.msg; done blobs-1.zip 1372 04-26-2014 01:00 1/9/sha256_bKH8aeWJSh9bhzv5zsXrweA7+jZ88NUllkfV+7m9cJo=_40511-49376.msg1
Backing Up Backups - 3rd Party Tools And Software - Dealing With Directories With Hard Links
Description Of Hard Links
Zimbra uses hard links and special attention needs to be given to this fact. See hard links if your not familiar with hard links and their difference to symbolic links. Not all 3rd party backup software will handle or respect hard links. Many unix commands will need special flags to maintain hard links. When hard links are respected and also "copied" to the new location you could find your data usage become a large multiplication of the original size.
An good thread I found on the topic of preserving hard links for copy/move/backup operations is here:
- "How to Copy a Filesystem and Preserve Hard Links in Linux - Some random bits scribbled by Jeremy Zawodny"
I'll be summarizing the comments and including additional information I find on the topic below, based upon the command or software being used.
Zimbra and Single Instance Storage - Hard Links
If hard links are possible, we use them. The message must be identical and on to users on the same mailstore and stored on zmvolumes where hard linking is possible. Hard links exist only on the same partition. Postfix has a veriable, default_destination_recipient_limit , which will cause large recipient emails to be delivered in a way where they aren't identical [ see Ajcody-Hardlinks-And-Postfix-default_destination_recipient_limit for more details ].
Mike gives a good description of it here:
- http://www.zimbra.com/forums/administrators/14435-single-instance-storage.html#post73746
- Here's the one wiki page Mike mentions in there: Account_mailbox_database_structure
Redoing Hard Links RFE
For 8.0.2+ , see this option - zmdedupe :
- RFE: Shipping a tool to convert duplicate blobs to hardlinks
Easy Way To SEE Hard Links In Use
I sent a test email with 5 accounts listed in the To field, that is shown below with the inode listing of 2133394. the first column of the ls [because of the i option] is listing the inode number of the file. I included the -type f because the . and .. will show directories using the same inode as the 'name' of the directory as listed.
# pwd /opt/zimbra/store2 # find . -links +1 -type f -exec ls -lai '{}' \; | sort 2133387 -rw-r----- 2 zimbra zimbra 1600 May 7 05:30 ./0/13/msg/0/281-4320.msg 2133387 -rw-r----- 2 zimbra zimbra 1600 May 7 05:30 ./0/27/msg/0/261-3021.msg 2133394 -rw-r----- 5 zimbra zimbra 1789 May 7 05:53 ./0/13/msg/0/282-4322.msg 2133394 -rw-r----- 5 zimbra zimbra 1789 May 7 05:53 ./0/14/msg/0/412-8476.msg 2133394 -rw-r----- 5 zimbra zimbra 1789 May 7 05:53 ./0/15/msg/0/284-9052.msg 2133394 -rw-r----- 5 zimbra zimbra 1789 May 7 05:53 ./0/25/msg/0/260-4021.msg 2133394 -rw-r----- 5 zimbra zimbra 1789 May 7 05:53 ./0/27/msg/0/268-3033.msg 2133404 -rw-r----- 2 zimbra zimbra 1518 May 7 05:22 ./0/13/msg/0/280-4318.msg 2133404 -rw-r----- 2 zimbra zimbra 1518 May 7 05:22 ./0/27/msg/0/260-3018.msg
Why Does A Message Blob With The Same Message-ID Have Multiple Inodes
Most likely, the message has a large recipient list and because of a variable in postfix, default_destination_recipient_limit , causes multiple deliveries of the message to have slight differences between them - for example, the times in the headers.
Please see the following:
Hard Links Used Within Zimbra Backup Directory - Sessions
Note, this describes an example when zimbraBackupMode equals Standard rather than Auto-Grouped
In regards to the backup sesions, hard links are only used for data that is in the shared_blobs directories from the various full backup sessions you have.
If you goto [below is an example used throughout]:
cd /opt/zimbra/backup/sessions/full-[some full dir]/shared_blobs/[some path]/t9RRjTIdwAZ3k,iJSWo0DxFKCbs=/
Then do a:
ls -li blob.dat 3869621 -rw-r----- 2 zimbra zimbra 5419 Jul 15 01:00 blob.dat
The first number is the inode of the file, which will be the same for all other hard links in use.
cd /opt/zimbra/backup/sessions find . -inum 3869621 -print /opt/zimbra/backup/sessions/full-[some full dir #1]/shared_blobs/[some path]/t9RRjTIdwAZ3k,iJSWo0DxFKCbs=/blob.dat /opt/zimbra/backup/sessions/full-[some full dir #2]/shared_blobs/[some path]/t9RRjTIdwAZ3k,iJSWo0DxFKCbs=/blob.dat
Notice also, that the directory path naming scheme is also used for the user backup paths. The directory name that the blob.dat is in, will also be used in the user paths.
For example:
/opt/zimbra/backup/sessions/full-[some full dir]/shared_blobs/[some path]/t9RRjTIdwAZ3k,iJSWo0DxFKCbs=/blob.dat
if I then goto and do:
cd /opt/zimbra/backup/sessions/full-[some full dir]/ find . -name t9RRjTIdwAZ3k* print
I'll see matches in various user backups, with:
full-[some full dir]/accounts/c2e/23b/[zimbra ID of user]/blobs/3/0/t9RRjTIdwAZ3k,iJSWo0DxFKCbs=260-2041.msg4
Note though, those "user" files are not links [hard or soft] and they are of zero length in size.
Reference To Bugs Tied To Backups And Links:
- This is an "old" one but is here for historical reasons because it does describe the expected behavior
- "Backup directory on dogfood has zero-length blob files"
Bacis Unix Commands
cp
cp [copy] Man page:
- http://linux.die.net/man/1/cp
- -a, --archive : same as -dpR
- -d : same as --no-dereference --preserve=link
- -p : same as --preserve=mode,ownership,timestamps
- -R, -r, --recursive : copy directories recursively
- -a, --archive : same as -dpR
- -l, --link : link files instead of copying
- -L, --dereference : always follow symbolic links
- -P, --no-dereference : never follow symbolic links
- -p : same as --preserve=mode,ownership,timestamps
- --preserve[=ATTR_LIST] : preserve the specified attributes (default: mode,ownership,timestamps) and security contexts, if possible additional attributes: links, all
- --no-preserve=ATTR_LIST : don't preserve the specified attributes
- -s, --symbolic-link : make symbolic links instead of copying
No explicit mention of "hard links".
Formats from blog url above [I'll test these later and confirm/deny]:
- cp -rpv
- cp -av --preserve=all . /mnt/new
- Simple `cp -a` using cp (GNU coreutils) 5.97 on my debian does the job quite nicely, I just checked. No need for the --preserve=all option, -a implies --preserve=link. It didn't seem to take too long either, but I would be surprised if it was very much better than rsync. Much easier to remember though.
mv
mv [move] Man page:
tar
tar [tape archive] Man page:
- http://linux.die.net/man/1/tar
- --check-links : warn if number of hard links to the file on the filesystem mismatch the number of links recorded in the archive
- -h, --dereference : don't dump symlinks; dump the files they point to
- No explicit reference to "hard links".
Please see the following on hard-links and GNU/Tar:
- http://www.gnu.org/software/tar/manual/html_node/hard-links.html
- "Normally, when tar archives a hard link, it writes a block to the archive naming the target of the link (a ‘1’ type block). In that way, the actual file contents is stored in file only once."
dd
dd [copy and covert - cc was reserved for the C complier] Man page:
- http://linux.die.net/man/1/dd
- nofollow : do not follow symlinks
No explicit mention of "hard links" in Man page.
rsync and nice
rsync and nice Man page:
- http://linux.die.net/man/1/rsync
- rsync has the following option: [ -H, --hard-links, preserve hard links ]
- http://linux.die.net/man/1/nice
- Nicenesses range from -20 (most favorable scheduling) to 19 (least favorable)
- Some distro's "nice" requires -n ## vs just the ##.
- Nicenesses range from -20 (most favorable scheduling) to 19 (least favorable)
A reasonable syntax to use for rsync is [ taken from Ajcody-Notes-Server-Move#The_Actual_Steps ]:
nice +19 rsync -avzH -e ssh --progress /opt/zimbra/ root@NEWHOSTIP:/opt/zimbra
or
nice -20 rsync -avzH -e ssh --delete --progress /opt/zimbra/ root@NEWHOSTIP:/opt/zimbra
- Notice the use of --delete in the last one.
find And cpio
find and cpio Man pages:
Possible syntax use to try [recursive copy of current working directory] :
mkdir /path/to/dest cd /path/to/DATA find . -print | cpio -Bpdumv /path/to/dest
Issue to note: "...cpio didn't properly preserve timestamps on directories."
dump And restore
dump and restore Man pages:
ditto - For Mac
ditto Man page:
xfs_copy
xfs_copy Man page:
LVM Tricks
LVM Snapshots
See Back Up (And Restore) LVM Partitions With LVM Snapshots
SAN Snapshot
Possible note of caution I've seen from someone, "When implementing snapshots for ZCS, you should do the snapshot across all ZCS LUNs for a single host at the same time using a consistency group (for netapp, I believe this means cg-start/cg-commit)."
Please see:
- 5.0.x_Network_Edition_Backup_and_Restore#Using_snapshots_to_backup_and_restore
- Individual_Mailbox_Restore_from_Snapshot
- Developer discussion on the issue
Cloud Backups
Amazon S3 , Amazon EC2 , SecoBackup And/Or Tar
I've not used Amazon S3 or SecoBackup. I have no idea about the pricing structure of Amazon S3 and how differing solutions might cause price differences. What I think would be a reasonable approach:
- Adjust zimbra cron to:
- run zmbackup as normally scheduled but then include:
- tar and gzip "new" backup that was made to a "staging" partition.
- Setup Secobackup [CLI method for cron] to then copy this tar'd/gzip'd file to the Amazon S3 cloud.
- Remove local tar'd/gzip'd file from staging partition.
- run zmbackup as normally scheduled but then include:
I purpose the tar'd/gzip'd step because I doubt there's a way to avoid the hard link issue with SecoBackup/Amazon S3. Why pay multiple times for the same data?
Some information a customer reported to me:
S3 does not work as a normal filesystem and you cannot mount it; hence it wouldn't normally work. However, there are various projects out there which let you use S3 as a local POSIX-compliant file system. Possible options: s3fs-fuse jungle disk subcloud persistentfs Amazon EBS To cut a long story short, PersistentFS came out on tops - it worked extremely well - however did not work with Zimbra at all once I set it up as the store (/opt/zimbra/store) The problem is that the filesystem while it is POSIX compliant does not have support for hard linking (Which is what Zimbra does with tmp incoming messages to the store). -- [bug 43019 below] So, overall it's not really possible to do it right now with S3. They should have hard link support soon.
References:
- Amazon Web Services Homepage - There's various "services" available there
- http://aws.amazon.com/
- Support, QA, testing for Amazon Web Services - AWS
- Amazon Simple Storage Service (Amazon S3)
- http://aws.amazon.com/s3/
- Zimbra RFE "Support, QA, testing for Amazon S3"
- Amazon Elastic Compute Cloud (Amazon EC2)
- http://aws.amazon.com/ec2/
- Zimbra RFE "Support, QA, testing for Amazon EC2"
- http://aws.amazon.com/
- Zimbra RFE "store to support a link yes/no setting"
- http://www.secobackup.com
- Ajcody-Backup-Restore-Issues#tar
- http://www.zimbra.com/forums/installation/25306-remote-backup-cloud-storage-amazon-s3.html
3rd Party Backup Tools And Software - Generally Not Apart Of The Basis OS
Amanda And Zmanda
Zmanda Home Page:
Amanda Home Page - Advanced Maryland Automatic Network Disk Archiver:
Found this, "Hard links. Maintains the integrity of hard links during backup."
Arkeia
Arkeia Home Page:
Found this, "The Arkeia solution accommodates full and incremental backups, scheduled or on demand, and preserves directory structure, registry, symbolic links and special attributes."
Also, a customer supplied what Arkeia's support sent them to the question. Here is that response:
- When Arkeia encounters a hard link (a regular file with more than one reference to it):
- if it is the first time we see this link, we backup the file and keep the inode number and the path to the file in a memory hash table.
-
- if we have already seen the file, we backup the fact that it is a hard link and the target of the link. The files data are not backed up again.
Backup Exec - Symantec / Veritas
Backup Exec Home Page:
- http://www.symantec.com/business/products/agents_options.jsp?pcid=pcat_storage&pvid=57_1
- Uses the "Backup Exec Remote Agent for Linux or UNIX Servers". The "server" is Windows based only.
Unable to find any reference about the Linux/Unix Agent and the Backup Exec server being able to handle or not symbolic and hard links.
Backup Exec "server" is only available on Windows. One might inquire with Symantec if you can "swap out" your current investment in "Backup Exec" and use their NetBackup product. This supports hard links and the "server" can run on Windows or other *nixes. See Ajcody-Backup-Restore-Issues#NetBackup_-_Veritas.2FSymantec
BackupPC
- Homepage for BackupPC
Bacula
Bacula Home Page:
Found this:
- hardlinks=yes|no
- When enabled (default), this directive will cause hard links to be backed up. However, the File daemon keeps track of hard linked files and will backup the data only once. The process of keeping track of the hard links can be quite expensive if you have lots of them (tens of thousands or more). This doesn't occur on normal Unix systems, but if you use a program like BackupPC, it can create hundreds of thousands, or even millions of hard links. Backups become very long and the File daemon will consume a lot of CPU power checking hard links. In such a case, set hardlinks=no and hard links will not be backed up. Note, using this option will most likely backup more data and on a restore the file system will not be restored identically to the original.
Source:
BRU - TOLIS Group
BRU Home Page:
Found this in one of their manuals, you should confirm with them based upon the product version you'll be using:
- Special Files - BRU will save and restore all types of filesystems and files with their proper ownership, access attributes, creation dates, and modification dates. BRU can be used to move an entire directory hierarchy from one system to another, with all files, including directories, block special files, character special files, fifos, hard links, and symbolic links reproduced with all attributes intact.
Lone-Tar - Lone Star Software Corp.
Lone-Tar Home Page:
Found the following references, though it's not explicit in stating "hard links" :
- BACKUP FEATURES - Backs up everything including device files, empty directories, links, symbolic links, Virtual Files and NFS mounted file systems.
- RESTORE FEATURES - Fast Seek File restore of files, sym-links, device files, and linked files.
Source:
NetBackup - Veritas/Symantec
NetBackup Home page:
Found the following:
- Hard links to directories :
- On most UNIX systems, only the root user can create a hard link to a directory. Some systems do not permit hard links and many vendors recommend that these links be avoided. NetBackup does not back up and restore hard-linked directories in the same manner as files:
- During a backup, if NetBackup encounters hard-linked directories, the directories are backed up once for each hard link.
- During a restore, NetBackup restores multiple copies of the hard-linked directory contents if the directories do not already exist on the disk. If the directories exist on disk, NetBackup restores the contents multiple times to the same disk location.
- On most UNIX systems, only the root user can create a hard link to a directory. Some systems do not permit hard links and many vendors recommend that these links be avoided. NetBackup does not back up and restore hard-linked directories in the same manner as files:
- Hard links to files :
- A hard link differs from a symbolic link in that a hard link is not a pointer to another file. A hard link is two directory entries that point to the same inode number.
- If the backup selection list includes hard-linked files, the data is backed up only once during a backup. NetBackup uses the first file name reference that is found in the directory structure. If a subsequent file name reference is found, it is backed up as a link to the name of the first file. Backup up only the link means that only one backup copy of the data is created, regardless of the number of hard links. Any hard link to the data works.
- For more information and examples, see “Hard links to files (NTFS volumes or UNIX)” on page 173.
Source:
- System Administrator's Guide for UNIX and Linux, Volume 1 ( http://seer.entsupport.symantec.com/docs/290201.htm )
NetVault - ORBiT
NetVault Home page:
Found this in their "NV Backup Administrators Guide" - pdf only:
- [screen shot of GUI check box] The Attempt to Restore Hard Links’ option as revealed in the Restore Options tab on a Linux/UNIX-based version of the File System Plugin.
- Attempt to Restore Hard Links (Linux/UNIX-based O/S, ONLY) - During a backup, when the first occurrence of a hard link is found, the complete data will be backed up. For all other occurrences, only the link is backed up. This data can only be restored when the first occurrence exists; trying to restore subsequent occurrences without the presence of the first causes the job to fail. Selecting this option will attempt to locate the full sequence so that all occurrences of the hard link will be restored.
rsnapshot
rsnapshot Home page:
Tivoli - IBM
Tivoli Home Page:
Found the following:
- Understanding how hard links are handled
- When you back up files that are hard-linked, Tivoli Storage Manager backs up each instance of the linked file. For example, if you back up two files that are hard-linked, Tivoli Storage Manager will back up the file data twice.
- When you restore hard-linked files, Tivoli Storage Manager attempts to reestablish the links. For example, if you had a hard-linked pair of files, and only one of the hard-linked files is on your workstation, when you restore both files, they will be hard-linked. The files will also be hard-linked even if neither of the files exists at the time of restore, if both of the files are restored together in a single command. The one exception to this procedure occurs if you back up two files that are hard-linked and then break the connection between them on your workstation. If you restore the two files from the server, Tivoli Storage Manager will respect the current file system and not re-establish the hard link.
- Attention: If you do not back up and restore all files that are hard-linked at the same time, problems will occur. To ensure that hard-linked files remain synchronized, back up all hard links at the same time and restore those same files together.
Source:
Other Related Items
freedup
freedup Man page:
freedup Homepage:
I've never used this tool, but from the description it seems it might come in handy for some circumstances.
- "Establishes hard or symbolic links between identical files. Search all given file system trees for identical files and link them to the most frequently referenced inode or if equally referenced to the inode of the first file tree. If the devices differ a symbolic link is used instead of a hard link. Symbolic links will not replace files, when at least one of the directory trees is not starting with a '/'."
Tape Devices
Many times, "drivers" aren't needed for tape devices for linux, but many administrators are unaware of this and never give it a test. Instead, they just assume the device doesn't work and the tape vendor isn't supporting it because they "didn't publish" a driver for it.
Quantum
Resource:
- Driver Software Compatibility Guide - Linux + LTO-4-HH
- "Installing a DLT, SDLT, VS, LTO, or DAT Tape Drive Into a Linux Operating System"
NO_SUCH_BLOB Errors
Actual No Such Blob Homepage
Please see Ajcody-Notes-No-Such-Blob
Other Resource Overviews
It would be good to review this wiki page before you start any "changes":
http://wiki.zimbra.com/index.php?title=Account_mailbox_database_structure
And great forum posting:
http://www.zimbra.com/forums/administrators/19811-solved-missing-blob-errors-zimbra-4-5-10-a.html
Check Permissions First
If you recently did a server move or similar type of operation, make sure zimbra:zimbra permissions are applied correctly to store directory. You could run the following to double check permissions (as root):
/opt/zimbra/libexec/zmfixperms --verbose --extended
Perl To Find MailboxID's That Have Problem
And down and dirty way to see what mailboxes have this error:
grep -B2 NO_SUCH_BLOB /opt/zimbra/log/mailbox.lo* |grep mailbox= |sed -r 's/.*mailbox=([0-9]*).*$/\1/' |sort -u
Mapping MailboxId to User Account
Remember, mailboxId is unique to the mailstore - it's not a global user variable that's unique.
Stolen from forum post:
http://www.zimbra.com/forums/administrators/16885-mailbox_id-email-address.html
su - zimbra mysql use zimbra; select comment from mailbox where id=257; And you'll get back: +----------------------+ | comment | +----------------------+ | user@domain.com | +----------------------+ 1 row in set (0.00 sec) If you wanted to see all the info for that account: select * from mailbox where id=257; And you'll get back: +-----+----------+--------------------------------------+-----------------+--------------------+---------------+-----------------+-------------------+---------------+---------------+----------------+----------------------+------------------+--------------+--------------------+ | id | group_id | account_id | index_volume_id | item_id_checkpoint | contact_count | size_checkpoint | change_checkpoint | tracking_sync | tracking_imap | last_backup_at | comment | last_soap_access | new_messages | idx_deferred_count | +-----+----------+--------------------------------------+-----------------+--------------------+---------------+-----------------+-------------------+---------------+---------------+----------------+----------------------+------------------+--------------+--------------------+ | 257 | 71 | xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxxxxxxx | 2 | 257 | 0 | 1832 | 100 | 0 | 0 | NULL | user@domain.com | 1209652908 | 0 | 0 | +-----+----------+--------------------------------------+-----------------+--------------------+---------------+-----------------+-------------------+---------------+---------------+----------------+----------------------+------------------+--------------+--------------------+ 1 row in set (0.00 sec)
Zmblobchk for 5.0.6+ Systems
On 5.0.6+ there's a script in /opt/zimbra/bin called zmblobchk . This will list out information about the NO_SUCH_BLOB errors and other inconsistencies.
zmblobchk - RFE's
zmblobchk currently just lists "issues" - for the most part. Please see the following RFE's, some will give details about resolutions based upon what zmblobchk reported.
- "blobchk: Add support for 'repair-mode' to the zmblobchk tool" Resolved for ZCS 6.0.6
- The initial RFE for the zmblobchk tool
- "tool to do consistency checks and repair for missing blob for ID x" Resolved for ZCS 5.0.6, "repair" part done in bug 27958.
Example Output On A System With No Issues
su - zimbra ls /opt/zimbra/zmblobchk.jar zmblobchk.jar java -jar zmblobchk.jar Retrieving volume information Retrieving mboxgroup list Spooling item list to /tmp/mailitems53650.lst Retrieving items from mboxgroup1 Retrieving items from mboxgroup2 Retrieving items from mboxgroup3 Retrieving items from mboxgroup4 Retrieving items from mboxgroup5 Retrieving items from mboxgroup6 Retrieving items from mboxgroup7 Retrieving items from mboxgroup8 Retrieving items from mboxgroup9 Retrieving items from mboxgroup10 / Processing BLOB store - /tmp/mailitems53650.lst: size 28141 Processed 329 items in 1223ms No inconsistencies found
Example Output On A System With Issues
Last login: Thu Oct 30 12:03:12 on ttys000 COLDMILE-LM-3:~ ajcody$ ssh root@mail3.internal.homeunix.com root@mail3.internal.homeunix.com's password: Last login: Sat Nov 1 14:25:50 2008 from 192.168.0.13 [root@mail3 ~]# su - zimbra [zimbra@mail3 ~]$ zmblobchk Retrieving volume information Retrieving mboxgroup list Spooling item list to /tmp/mailitems39690.lst Retrieving items from mboxgroup1 Retrieving items from mboxgroup2 Retrieving items from mboxgroup3 Retrieving items from mboxgroup4 Retrieving items from mboxgroup5 Retrieving items from mboxgroup6 Retrieving items from mboxgroup7 Retrieving items from mboxgroup8 Retrieving items from mboxgroup9 Retrieving items from mboxgroup10 Retrieving items from mboxgroup11 Retrieving items from mboxgroup12 Retrieving items from mboxgroup13 Retrieving items from mboxgroup14 Retrieving items from mboxgroup15 Retrieving items from mboxgroup16 Retrieving items from mboxgroup17 Retrieving items from mboxgroup18 Retrieving items from mboxgroup19 Retrieving items from mboxgroup20 - Processing BLOB store / /tmp/mailitems39690.lst: size 470853 Processed 5873 items in 18462ms 11844 inconsistencies found * MailboxGroup=1, mailbox=1, item=303: wrong volume, expected at /vol1/0/1/msg/0/303-2062.msg, found at /opt/zimbra/store/0/1/msg/0/303-2062.msg (move to correct volume) * MailboxGroup=1, mailbox=1, item=304: wrong volume, expected at /vol1/0/1/msg/0/304-2064.msg, found at /opt/zimbra/store/0/1/msg/0/304-2064.msg (move to correct volume) * MailboxGroup=1, mailbox=1, item=305: wrong volume, expected at /vol1/0/1/msg/0/305-2077.msg, found at /opt/zimbra/store/0/1/msg/0/305-2077.msg (move to correct volume) * MailboxGroup=1, mailbox=1, item=306: wrong volume, expected at /vol1/0/1/msg/0/306-2091.msg, found at /opt/zimbra/store/0/1/msg/0/306-2091.msg (move to correct volume) ## CUT OF 1000's OF SIMILAR LINES ## * MailboxGroup=1, mailbox=1, item=6480: file not found: /vol1/0/1/msg/1/6480-20100.msg (delete associated metadata) * MailboxGroup=3, mailbox=3, item=400: wrong volume, expected at /vol1/0/3/msg/0/400-11287.msg, found at /opt/zimbra/store/0/3/msg/0/400-11287.msg (move to correct volume) * MailboxGroup=3, mailbox=3, item=425: wrong volume, expected at /vol1/0/3/msg/0/425-12939.msg, found at /opt/zimbra/store/0/3/msg/0/425-12939.msg (move to correct volume) * MailboxGroup=3, mailbox=3, item=427: wrong volume, expected at /vol1/0/3/msg/0/427-12943.msg, found at /opt/zimbra/store/0/3/msg/0/427-12943.msg (move to correct volume) * MailboxGroup=3, mailbox=3, item=440: file not found: /vol1/0/3/msg/0/440-13400.msg (delete associated metadata) * /opt/zimbra/store/0/3/msg/0/363-6194.msg: no associated metadata (delete blob) * /opt/zimbra/store/0/3/msg/0/360-5000.msg: no associated metadata (delete blob) * /opt/zimbra/store/0/3/msg/0/350-3129.msg: no associated metadata (delete blob) * /opt/zimbra/store/0/3/msg/0/320-2272.msg: no associated metadata (delete blob) ## CUT OF 1000's OF SIMILAR LINES ## * /opt/zimbra/store/0/2/msg/0/304-6907.msg: no associated metadata (delete blob) * /opt/zimbra/store/0/2/msg/0/315-6918.msg: no associated metadata (delete blob) * /opt/zimbra/store/0/2/msg/0/312-6915.msg: no associated metadata (delete blob) * /opt/zimbra/store/0/2/msg/0/308-6911.msg: no associated metadata (delete blob) * /opt/zimbra/store/0/2/msg/0/303-6906.msg: no associated metadata (delete blob) * /opt/zimbra/store/0/2/msg/0/301-6904.msg: no associated metadata (delete blob) *** 11844 faults found Report saved to: /tmp/zmblobc39689.rpt [zimbra@mail3 ~]$
To see the reported output again:
zmblobchk -l /tmp/zmblobc39689.rpt
Error - Not Enough Memory To Run zmblobchk
If the "Processing BLOB store" part errors out with Java out of memory, you could try adjusting the variable for it. The default is set to 30%. This would require a mailstore/jetty restart though [zmmailboxdctl restart]. If the mailstore doesn't come back up, check /opt/zimbra/log/zmmailbox.log - you most likely exceeded the threshold for your box. Remember, this is a percentage and a 32bit machine with more than 4GB can cause issues with this setting since it could try to allocate more than it actually can to the thread.
# ZCS 5.0 and later $ zmlocalconfig -e mailboxd_java_heap_memory_percent=40
And zmmailboxdctl restart when it's appropriate.
BLOB Issue Script
I've commented out the two delete lines and also the "$msghash &= $fmask;" one.
If you're using HSM, this script isn't smart enough to detect it. It will flag and remove entries that are in the HSM store because it doesn't see them in the primary store. You can modify the output though to use the actual path to the HSM store. I had one case where this was the case and copying the missing blobs to the HSM path worked.
Comment from other support staff member on a case that used this.
Once you've gotten a list of the 'NOT OK' files generated by this tool, you'll want to check the store directories to see if the message blobs exist with an incorrect change number. That's the second number in the filename. If the first number matches the ID it's looking for but the second does not, you can rename the file to the new change number and get the message blob back without losing data. If not, then the blob is completely missing. If that's the case, we should try to figure out what happened to it by going through the mailbox.log files. You should be able to grep for the missing message ID and the user's account ID to find a log entry showing something happening to it. It's possible it was deleted and the deletion was never written to mysql for some reason.
#!/usr/bin/perl # On 5.0.6+ there's a script, /opt/zimbra/bin/zmblobchk , that you can use instead of this. # This script is compatible with Zimbra version 4.5.x and 5.x . # OK, there's 2 MAILBOX_*_BITS values in the VOLUME table. # Take the mailbox ID, right-shift it by MAILBOX_BITS, and take the lowest MAILBOX_GROUP_BITS of the result. # That's your mailbox hash. # Take the message ID, right-shift it by FILE_BITS, and take the lowest FILE_GROUP_BITS of the result. That's your msgid hash. # I think. # <mbx-hash>/<mbx-id>/msg/<msgid-hash>/<msgid>-<mod_content>.msg my ($fbits, $fgbits, $mbits, $mgbits, $basepath) = split (' ',`echo "select file_bits,file_group_bits,mailbox_bits,mailbox_group_bits,path from volume where type='1'" | mysql -N zimbra`); my $ARGV = shift; chomp $basepath; my $mbmask = "1" x $mgbits; my $fmask = "1" x $fgbits; foreach (`echo "select id, group_id, account_id, comment from mailbox where comment = '$ARGV'" | mysql -N zimbra`) { chomp; my $path = "$basepath/"; my ($id, $grid, $aid, $nm) = split; my $mbhash = $id >> $mbits; $mbhash &= $mbmask; $path .= $mbhash."/".$id."/msg/"; foreach my $msgstuff (`echo "select id, mod_content, type from mail_item where blob_digest is not null and mailbox_id=${id};" | mysql -N mboxgroup${grid}`) { chomp $msgstuff; next if $msgstuff eq ""; my ($msgid, $modContent, $type) = split (' ',$msgstuff); my $msghash = $msgid >> $fbits; # $msghash &= $fmask; my $nm = $msgid; if ($modContent) {$nm .= "-$modContent";} my $npath = $path.$msghash."/".$nm.".msg"; if (-e $npath) {print $npath."\t\tOK\n";} else { print $npath."\t\tNOT OK\n"; # not ok, remove the entry from the database so it is not a nuisance print "Delete from mail_item where MSGID is ${msgid} and MAILBOXID is ${id} \n"; print "Uncomment line below me in script to have me delete. \n"; # `echo "delete from mail_item where id=${msgid} and mailbox_id=${id}" | mysql -N mboxgroup${grid}`; if ($type eq "11") { print "Delete from appointment where MSGID is ${msgid} and MAILBOX_ID is ${id} \n"; print "Uncomment line below me in script to have me delete. \n"; # `echo "delete from appointment where item_id=${msgid} and mailbox_id=${id}" | mysql -N mboxgroup${grid}`; } } # print $npath."\n"; } }
If this doesn't run, make sure perl is installed and in the correct path.
You run this as the zimbra user.
What I did was the following.
Copied the script to /opt/zimbra/bin/ as root and called it blob-check.pl . chmod +x /opt/zimbra/bin/blob-check.pl
su - zimbra mkdir /tmp/BLOB-CHECKS for i in `zmprov gaa`; do blob-check.pl $i | grep "NOT OK" >> /tmp/BLOB-CHECKS/$i.txt; done ls -la /tmp/BLOB-CHECKS
The ls will help identify the more troubled accounts. You'll end up with something like this:
[zimbra@mail3 ~]$ ls -la /tmp/BLOB-CHECKS/ total 16 drwxr-x--- 2 zimbra zimbra 4096 Jul 10 23:40 . drwxrwxrwt 8 root root 4096 Jul 10 23:46 .. -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 admin@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 ajcody2@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 49 Jul 10 23:40 ajcody@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 forward@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 ham.bidiob2mm@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 secondary@secondary.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 spam.rormmtcyy@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 wiki@mail3.internal.homeunix.com.txt -rw-r----- 1 zimbra zimbra 0 Jul 10 23:40 wiki@secondary.internal.homeunix.com.txt
And the output of one that shows some size:
$ cat /tmp/BLOB-CHECKS/ajcody@mail3.internal.homeunix.com.txt /opt/zimbra/store/0/3/msg/0/360-6070.msg NOT OK
Remember this from above, "you'll want to check the store directories to see if the message blobs exist with an incorrect change number. That's the second number in the filename. If the first number matches the ID it's looking for but the second does not, you can rename the file to the new change number and get the message blob back without losing data."
One can uncomment the 2 lines in the script with delete and run the script again. This will remove the reference to the blob.
To re-index the users mailbox [this can take start, cancel, status]:
zmprov rim user@domainname start
Running the script again shouldn't show "NOT OK" lines.
FYI - I still need to gather more information about this situation and how to guide one to make a decision to delete, re-index, and so forth.
BLOB Script To Copy From Restore
You might need to use these variables with the restore command. I need to find a way for someone to figure out what restore to goto for a particular blob:
-restoreToTime <arg> - Replay the redo logs until the time specified.
-restoreToIncreLabel <arg> - Replay redo logs up to and including this incremental backup.
-restoreToRedoSeq <arg> - Replay up to and including this redo log sequence
-br - Replays the redo logs in backup only, which excludes archived and current redo logs of the system.
-rf - Restores to the full backup only, does not include any incremental backups since that backup
This is a walk through to test the proceedure.
$ zmrestore -ca -a ajcody@mail3.internal.homeunix.com -pre restored- $ zmprov gmi restored-ajcody@mail3.internal.homeunix.com mailboxId: 12 quotaUsed: 251513 $ zmprov gmi ajcody@mail3.internal.homeunix.com mailboxId: 3 quotaUsed: 251512 $ diff /opt/zimbra/store/0/12/msg/0/257-25.msg /opt/zimbra/store/0/3/msg/0/257-25.msg $ rm /opt/zimbra/store/0/3/msg/0/257-25.msg rm: remove regular file `/opt/zimbra/store/0/3/msg/0/257-25.msg'? y $ cp /opt/zimbra/store/0/12/msg/0/257-25.msg /opt/zimbra/store/0/3/msg/0/257-25.msg $ grep Subject /opt/zimbra/store/0/3/msg/0/257-25.msg Log into main account and confirm I can see/use that mail.
The script to look at the blob-check.pl output and do the copies after a redirected restore is here:
#!/bin/bash #START OF VARIABLES# #CHANGEME to existing user account USER=ajcody@mail3.internal.homeunix.com #CHANGEME to the restored account name # zmrestore -ca -a ajcody@mail3.internal.homeunix.com -pre restored- RESTOREDUSER=restored-ajcody@mail3.internal.homeunix.com # To get FILE, blob-check.pl ajcody@mail3.internal.homeunix.com > /tmp/BLOB-CHECKS/ajcody.txt # blob-check.pl located at http://wiki.zimbra.com/index.php?title=Ajcody-Notes#BLOB_Issue_Script FILE=/tmp/BLOB-CHECKS/ajcody.txt #CHANGEME #CHANGEME, if needed, to the mailstore. /opt/zimbra/store is default STOREDIR=/opt/zimbra/store #END VARIABLES# USERUID=`zmprov gmi $USER | grep mailboxId | cut -f2 -d: | cut -c 2-10` USERGID=`perl -e 'print $USERUID >> 12 ; print "\n"'` USERPATH=`echo $USERGID/\$USERUID` RESTOREDUID=`zmprov gmi $RESTOREDUSER | grep mailboxId | cut -f2 -d: | cut -c 2-10` RESTOREDGID=`perl -e 'print $RESTOREDUID >> 12 ; print "\n"'` RESTOREDPATH=`echo $RESTOREDGID/\$RESTOREDUID` echo USERUID $USERUID echo USERGID $USERGID echo USERPATH $USERPATH echo RESTOREDUID $RESTOREDUID echo RESTOREDGID $RESTOREDGID echo RESTOREDPATH $RESTOREDPATH cd $STOREDIR echo "Run ECHO first to confirm copies look right and then uncomment COPY in script" for ROOTBLOBPATH in `grep "NOT OK" $FILE | cut -f7-9 -d/ | cut -f1` do # Comment out echo and uncomment copy after dry run echo $RESTOREDPATH/$ROOTBLOBPATH $USERPATH/$ROOTBLOBPATH #cp -uv $RESTOREDPATH/$ROOTBLOBPATH $USERPATH/$ROOTBLOBPATH done
BLOB Script To Copy From Restore (HSM Issue)
One customer ran into an issue that the blob references were really expecting them in the HSM mailstore path.
Example:
Default mailstore for PROD : /opt/zimbra/mail/
HSM mailstore path : /opt/zimbra/mail0 which was another partition mount, compared to /opt/zimbra/mail
So, what this script did was used the output from the blob-check.pl script which has the missing blobs formatted for the PROD path rather than the HSM path. The blob-check.pl script doesn't handle HSM data. The script then finds the blobs in the full backup directory of the user and copies them into the HSM path that the WEBCLIENT is expecting. This whole situation with this customer might of been a fluke. So don't consider these steps as applying to your situation unless you do your research. This case took 9 days to resolve and figure out.
Notice, if you use this script below you really need to review it in details and adjust it for your situation. There's more to adjust beyond the base variables.
#!/bin/bash # Steps # Make sure directory path exists for the copy command # Get mailboxId - zmprov ga USER@DOMAIN | grep zimbraId: # mkdir /opt/zimbra/mail0/0/[mailboxId #]/msg/{1..15} or how what ever # you need. # Change USEREMAIL # Get path for BACKUPFILES and uncomment for use # Run find command with echo and confirm. Then rerun using the find with cp rather than echo # Put in user's email for USEREMAIL USEREMAIL="USER@DOMAIN" ZIMBRAID=`zmprov ga $USEREMAIL | grep zimbraId: | awk '{print $2}'` # See below # ROOTBACKUPPATH="/opt/zimbra/mail0/BACKUP/sessions/full-20080707.004336.789" #The line below takes to long, see instructions underneath #BACKUPFILES=`find $ROOTBACKUPPATH -name *$ZIMBRAID*` # To run manually, which will increase speed of script if you then paste full path in for BACKUPFILES var # zmprov ga USER@DOMAIN | grep zimbraId: | awk '{print $2}' # Then place the out in this find command, leave the wildcards on each side of it # find /opt/zimbra/mail0/BACKUP/sessions/full-20080707.004336.789 -name *OUTPUTHERE* # This should give you path like the exampe below, paste it between the quotes and uncomment the variable BACKUPFILES #EXAMPLE FOR BACKUPFILES="/opt/zimbra/mail0/BACKUP/sessions/full-20080707.004336.789/accounts/a07/1b5/a071b5ad-b341-4de4-b6ee-4463f322873b" # BACKUPFILES="PASTE_IN_PATH_HER" for BLOBPATH in `blob-check.pl $USEREMAIL | grep "NOT OK" | awk '{print $1}' | sed 's/store/mail0/'` do MSGID=`echo $BLOBPATH | awk -F/ '{print $NF}'` # Run echo first to make sure copy path seems correct, ctrl-c to cancel. And then uncomment the other find command with the cp. Comment the echo one. find $BACKUPFILES -name *$MSGID* -exec echo {} $BLOBPATH \; #find $BACKUPFILES -name *$MSGID* -exec cp -uv {} $BLOBPATH \; done
The Really Bad Situation, Blob References Gone & Mail Still Missing
For now, this is all I have to offer.
Method One - zmlmtpinject
See forum post:
http://www.zimbra.com/forums/installation/12617-recover-data-mbox-folders.html#post64962
Method Two - Use IMAPSYNC
I haven't tested this and would just be guessing, but I believe IMAPSYNC could be used.
From the IMAPSYNC notes, "imapsync is the adequate tool because it reduces the amount of data transferred by not transferring a given message if it is already on both sides. Same headers, same message size and the transfer is done only once. All flags are preserved, unread will stay unread, read will stay read, deleted will stay deleted."
This would involve having imap setup on ZCS and then doing a redirected user restore (which would consume a license).
See the following on imapsync details:
- Ajcody-Server-Plan-Move-Migration-Upgrade-DR#Non-Zimbra_IMAP_Accounts_To_Zimbra
- Mail_Migration#using_imapsync_.28Recommended_Method.29
Imapsync During A Certain Time Frame
You might want to check out the imapsync options of --maxage and --minage
Proposed Steps
- Confirm these steps are necessary
- Install imapsync
- Do a redirected restore of the account
- This is an example format of the redirected restore:
zmrestore -ca -a prod-user@servername.com -pre restored- -rf -lb full-'date of full'
- The "new" account would be restored-userPROD@servername.com . The -rf flags should look only at the backup data in regards to the restore and not catch the "production" references to missing blobs and what not.
- This is an example format of the redirected restore:
- Confirm in webclient that msg's are there
- Do a IMAPSYNC
- Two ways, unsure what is the best method
- From PROD to NEW
imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates --host1 servername.com --user1 prod-user --password1 test123 -user2 restored-prod-user --host2 servername.com --password2 test123 --authmech1 PLAIN --authmech2 PLAIN
- From NEW to PROD
imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates --host1 servername.com --user1 restored-prod-user --password1 test123 -user2 prod-user --host2 servername.com --password2 test123 --authmech1 PLAIN --authmech2 PLAIN
- From PROD to NEW
- Two ways, unsure what is the best method
- Reindex restored-prod-user
- Rename prod-user to prod-user-old
zmprov ra prod-user@servername.com prod-user-old@servername.com
- Rename restored-prod-user to prod-user
zmprov ra restored-prod-user@servername.com prod-user@servername.com
- Do a full backup of each account
zmbackup -f - <servername.com> -a <newaccountname@servername.com>
Notes On - No Associated Metadata
Tentative Research At This Point - Haven't Proven Out Yet
This is for the errors you see from the blobchk like this:
"* /opt/zimbra/store/0/1303/msg/0/1059-1726.msg: no associated metadata (delete blob) "
You could use the following command to bring the blob back in and regenerate the metadata -- zmmailbox am along -d . The am flag details are : addMessage(am) [opts] {dest-folder-path} {filename-or-dir} [{filename-or-dir} ...] . But the -d flag states : -d/--date <arg> received date (msecs since epoch) . To get the epoch time in seconds using the same receive data, for example :
ls -l --time-style=+%s /opt/zimbra/store/0/1303/msg/0/1059-1726.msg
And then remember the -d requires milliseconds [a thousandth (1/1,000) of a second] so you'll need to convert it. Or, you could just use something like this below. This with the cut would work well for a for loop script against a number of msg files.
ls -l --time-style=+%s000 --si /opt/zimbra/store/0/1303/msg/0/1059-1726.msg | cut -d" " -f6 1272642001000 ls -l --time-style=+%s000 --si /opt/zimbra/store/0/1303/msg/0/1059-1726.msg -rw-r--r-- 1 zimbra zimbra 0 1272642001000
Another suggestion for scripting was:
date=$(stat -c +%Y /opt/zimbra/store/0/1303/msg/0/1059-1726.msg); echo $(($date*1000));
Mysql Queries To See What Is Shown For Missing Message ID
References
See also :
Getting User Variables we need to query MySQL
Please see Ajcody-Mysql-Topics#Getting_User_Variables_We_Need_To_Query_MySQL
Mysql Query For A Users Specific Blob-Data ID
Please see Useful Mysql Query Examples
Mailing Lists, Distribution Lists, And Mailman
Actual Mailing Lists And Mailman Notes Homepage
Please see Ajcody-MailingLists-And-Mailman
Introduction And Choices - When Email Lists Have More Than 1000+ Recipients
If you are planning to use a distro list with 1000 to 3000 or more members, you should consider using a 3rd party mailing list software machine for this. This is the recommendation of the developers according the the bug/rfe's I reference below. According to bug 19153, you have 2 options:
- "set up mailing list manager or change the value of virtual_alias_expansion_limit as a customization. We have never tested with > 1000 so this should be done carefully, and will pound LDAP for any messages with lots of recipients."
- Mailman integration or another third party mailing list software package. There is a forum thread on how to integrate zcs and mailman. This would be the preferred fix to your issue, also noted in the bug report. We probably won't see Mailman integration in ZCS at least for another year or so, not until 6.0
Main Bug/RFE's to review:
- "Problems resolving virtual aliases for members of large distribution lists"
- https://bugzilla.zimbra.com/show_bug.cgi?id=19153
- Note - this is marked as a duplicate of bug 8968, "Mailman Integration".
- Comment 3 : "We have never tested with > 1000 so this should be done carefully, and will pound LDAP for any messages with lots of recipients."
- Note - In ZCS 8, "Release notes: If dynamic lists are used, the default is to allow lists of up to 10,000 members. This can be controlled via the localconfig key postfix_virtual_alias_expansion_limit"
- This does NOT resolve the issues for large email lists though, the recommendation is still to be using a dedicated mailing list software package.
- "too many recipients error when sending mail to dynamic lists with 1100+ members"
- https://bugzilla.zimbra.com/show_bug.cgi?id=64030
- This does NOT resolve the issues for large email lists though, the recommendation is still to be using a dedicated mailing list software package.
- Note - In ZCS 8, "Release notes: If dynamic lists are used, the default is to allow lists of up to 10,000 members. This can be controlled via the localconfig key postfix_virtual_alias_expansion_limit"
- Comment 5 : the answer for large lists is to use mailman. See bug 8968 and bug 21621
- https://bugzilla.zimbra.com/show_bug.cgi?id=19153
- "Mailman Integration"
- Note - this RFE is really about including mailing list functionality within ZCS, it isn't fully committed to Mailman being that solution at this time.
- https://bugzilla.zimbra.com/show_bug.cgi?id=8968
- "Test and document manual mailman integration"
- https://bugzilla.zimbra.com/show_bug.cgi?id=21621
- "This article [below] is a community contribution and may include unsupported customizations for network edition customers."
- https://bugzilla.zimbra.com/show_bug.cgi?id=21621
Other Bug/RFE's to review to see the full conversation on this from our developers - Note , it's important to note the bugs they are making duplicates of the main ones mentioned above.
- RFE - "integrate zimbra with sympa.org"
- RFE - "Mail policies and access control for sending to distribution lists"
- RFE: Mailing List Feature
- Reply-to mailing list option
- distribution list -- moderator
- Integration of Mailman or EZLM
- Auto Reply for Distribution Lists.
- auto-reply feature within distribution lists
- Autorespond for a MailList Account
- Distribution Lists Sequence Number
- Ability for external users to unsubscribe from a Zimbra Distribution List
References to Maining list software options:
- http://www.list.org/ - Mailman
- https://www.sympa.org/ - Sympa
- http://mlmmj.org/ - Mailing List Management Made Joyful
Restricting Who Can Send To Mailing List
These are unsupport customizations. Please be sure to make backup copies of modifications to prevent lost after upgrades/restores of Zimbra.
Please see:
- See the whole sub-section of this topic as well.
Also see the following RFE:
- "Mail policies and access control for sending to distribution lists"
Restrict Sending To Certain Domains
Please see:
Add Everyone To Distribution List - Admin Console
RFE was made for this:
- "Admin>DistributionList: AddAll to add members that are in only current-search-page"
Everyone@domain Without Manually Adding Users To A DL
Investigating if this can be done. Please don't attempt anything below, I'm simply keeping track of my "notes" here.
DON'T ATTEMPT ANY OF THIS! THIS WILL MOST LIKELY GET POSTED INTO BUGZILLA AS AN RFE
RFE Made For This:
- http://bugzilla.zimbra.com/show_bug.cgi?id=32341
- Was marked as a duplicate of the following:
- "Dynamic distribution lists"
- Was marked as a duplicate of the following:
One Possible Way Described In An External Resource:
main.cf: there is a line to resolve aliases: virtual_aias_maps = ldap: $ config_directory / ldap / ldap-aliases.cf ldap-aliases.cf that relates to filter: query_filter = (&(objectclass=mailgroup)(|(mail=%s)(mailalternateaddress=%s))) result_attribute = mail result_attribute = mail special_result_attribute = uniqueMember, memberUrl in LDAP-e for everyone is recording with such attributes: mail = everyone@domain.ru memberurl = ldap :///ou=People,o=organization??sub? (&(objectclass=person)(uid=*)) objectclass=mailgroup
- Had to use the following to get it to work:
receive_override_options = no_address_mappings Was a global setting.
Another Possible Way, again an external resource:
- Use Of mailAlternateAddress
- Confirmation that it can be done?
- New LDAP map transports:
Problems Resolving Virtual Aliases For Members Of Large Distribution Lists
See the following bug:
Other wiki page is here - Error_(MTA):_unreasonable_virtual_alias_maps_map_expansion_size
The default expansion that will be done is for 1000 accounts.
Solution
Note, the bug above recommends NOT going over 1000 due to ldap performance. They recommend looking into mailman as an alternative.
Increase the virtual_alias_expansion_limit Postfix parameter to the value of the highest number of distribution list members. All commands run as the zimbra user.
Check configuration
$ postconf virtual_alias_expansion_limit
Set configuration
$ postconf -e virtual_alias_expansion_limit=3000
Restart Postfix
$ postfix stop $ postfix start
Mailman - Mailing List Manager
Please see the following:
- ZCS_Mailman_Integration
- RFE - "Mailman Integration"
- http://www.zimbra.com/forums/administrators/1380-solved-zimbra-mailman-howto.html
Sympa - Mailing List Manager
This might be another option to Mailman that might even be better.
https://www.sympa.org/overview/features
Someone made this comment on the zimbra-heid-admins@sfu.ca list :
"We recently switched from mailman to sympa, which does the same thing. Sympa allows membership data sources to be defined from SQL queries, ldap queries, and more. I'm very pleased with it."
Some customers are advocating for it over Mailman on this RFE:
- RFE - "integrate zimbra with sympa.org"
- RFE - "Mail policies and access control for sending to distribution lists"
Duplicate Emails With Distribution Lists
Note - this has nothing to do with hard links on the filesystem.
Situation arises when a message goes out and the effected user is somehow targeted twice in the To/CC fields by either use of a DL list, multiple DL lists, and/or a direct inclusion of their email address as well.
There was a change between 4.5.x and 5.x . New variable is zimbraMessageIdDedupeCacheSize.
Setting the "zimbraMessageIdDedupeCacheSize" attribute to zero disables this feature.
su - zimbra zmprov gacf | grep zimbraMessageIdDedupeCacheSize
If you need to change the setting.
zmprov mcf zimbraMessageIdDedupeCacheSize 3000 zmmailboxdctl restart
Related Bugs:
- http://bugzilla.zimbra.com/show_bug.cgi?id=23076
- http://bugzilla.zimbra.com/show_bug.cgi?id=29371
- http://bugzilla.zimbra.com/show_bug.cgi?id=12072
In regards to DL messages sent by the sender only being in their sent box:
- http://bugzilla.zimbra.com/show_bug.cgi?id=402
- zimbraPrefDedupeMessagesSentToSelf vs. zimbraMessageIdDedupeCacheSize
Confirming Hard Links With Message Blobs To DL On Filesystem
Please see:
SSL Certificate Issues
Actual SSL Certificate Issues Homepage
Please see Ajcody-Notes-SSLCerts
Resources For SSL Certificates
- General
- Trouble Shooting
- Specific Cert Wiki Pages:
Bug & RFE's Related To SSL
Multiple SSL Certificates Aren't Supported On One Server
- "multiple SSL certificates on one server"
- http://bugzilla.zimbra.com/show_bug.cgi?id=8128
- Allot of private comment details unfortunately.
- http://bugzilla.zimbra.com/show_bug.cgi?id=8128
The -subjectAltNames option is the way to do, if your certificate provider doesn't support then you should inquire into another provider who does.
Recreating Self-Signed SSL Certificates
Other references:
- I believe this wiki page has the best instructions. Sorry for the confusion about the state of the wiki pages on ssl certs. We are attempting to get them cleaned up.
- Problem_with_Certificate_can_cause_MTA_Failure#For_Multi-Server:_Run_this_on_all_other_systems_in_the_multi-server_setup
- Zmcertinstall#Single-Node_Self-Signed_Certificate
- Has "Multi-Node Self-Signed Certificate" instructions as well.
- Recreating_a_Self-Signed_SSL_Certificate
- Zmcertinstall#Single-Node_Self-Signed_Certificate
- Problem_with_Certificate_can_cause_MTA_Failure#For_Multi-Server:_Run_this_on_all_other_systems_in_the_multi-server_setup
Steps I've used for a single ZCS 6 server that WAS NOT using commercial certificates. Also, this is for recreating self-signed certificates and not changing them. This documentation was done specifically for the issue when the self-signed certificates expired and caused upgrade issues.
** have zimbra running - ldap at least - need to double check this ** run as root ** Prep work mv /opt/zimbra/ssl/zimbra /opt/zimbra/ssl/zimbra_old ** I had used the below, but think the above step is *better* * mv /opt/zimbra/ssl /root/ssl_old * mkdir /opt/zimbra/ssl * chown zimbra:zimbra /opt/zimbra/ssl ** Creating new certs /opt/zimbra/bin/zmcertmgr createca -new /opt/zimbra/bin/zmcertmgr deployca -localonly ** others have just used: /opt/zimbra/bin/zmcertmgr deployca /opt/zimbra/bin/zmcertmgr createcrt self -new ** others have just used: /opt/zimbra/bin/zmcertmgr createcrt -new /opt/zimbra/bin/zmcertmgr verifycrt self /opt/zimbra/bin/zmcertmgr deploycrt self ** run as zimbra zmcontrol stop zmcontrol start
Keystore Password Errors - Server Move Stuff
Seen this with user attempting the 32 > 64 server move wiki page, mailboxd wouldn't start. Errors like:
keytool error: java.io.IOException: Keystore was tampered with, or password was incorrect * Zimbra server reserving server socket port=995 bindaddr=null ssl=true Fatal error: exception while binding to ports java.net.SocketException: Unbound server sockets not implemented * Exception in thread "main" java.io.IOException: Keystore was tampered with, or password was incorrect
And when I attempted to recreate the self-sign cert using the steps above, it ended with this error:
# /opt/zimbra/bin/zmcertmgr deploycrt self ** Saving server config key zimbraSSLCertificate...failed. ** Saving server config key zimbraSSLPrivateKey...failed. ** Installing mta certificate and key...done. ** Installing slapd certificate and key...done. ** Installing proxy certificate and key...done. ** Creating pkcs12 file /opt/zimbra/ssl/zimbra/jetty.pkcs12...done. ** Creating keystore file /opt/zimbra/mailboxd/etc/keystore...failed. Exception in thread "main" java.io.IOException: Keystore was tampered with, or password was incorrect at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:771) at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:38) at java.security.KeyStore.load(KeyStore.java:1185) at com.zimbra.cert.MyPKCS12Import.main(MyPKCS12Import.java:98) Caused by: java.security.UnrecoverableKeyException: Password verification failed at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:769) ... 3 more ** Installing CA to /opt/zimbra/conf/ca...done.
And the above recreating of self-cert fails because of issues about keystore password.
Fix was to :
[as root] mv /opt/zimbra/mailboxd/etc/keystore /root/keystore.old
And then run the recreating self-certs above.
Note: this also appears to happen on 8.6, with self-signed certs after rsync migration. Moving the old keystore and redeploying ca/certs corrects the issue.
Contrib from Baylink
(Here's an actual script you can copy to a file and run, by Baylink 16:08, 25 March 2010 (UTC))
echo "Backing up old certs..." mv /opt/zimbra/ssl/zimbra /opt/zimbra/ssl/zimbra.newcert-`timestamp` echo "Creating new CA..." /opt/zimbra/bin/zmcertmgr createca -new echo "Deploying new CA..." /opt/zimbra/bin/zmcertmgr deployca -localonly # added 12 Apr 2010 per tonyp@zimbra.com echo "Creating new CSR..." /opt/zimbra/bin/zmcertmgr createcsr self -new -subject "/C=US/ST=N\/A/L=N\/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration Suite/CN=server.company.com" -subjectAltNames "server.company.com,altname.company.com,othername.company.com" echo "Creating new cert..." /opt/zimbra/bin/zmcertmgr createcrt -new -subject "/C=US/ST=N\/A/L=N\/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration Suite/CN=server.company.com" -subjectAltNames "server.company.com,altname.company.com,othername.company.com" echo "Verifying new cert..." /opt/zimbra/bin/zmcertmgr verifycrt self echo "Deploying new cert..." /opt/zimbra/bin/zmcertmgr deploycrt self echo "Restarting Zimbra server" su - zimbra -c 'zmcontrol stop; zmcontrol start'
Note that if you're running Exchange ActiveSync with iPhones, the iPhones require that the "server.company.com" name (the primary name) must
- be the primary name on the SSL cert (I've tested this) and
- resolve to the same IP both inside and outside your firewall which
- requires a firewall that can do "hairpin" inbound-NAT
So, if your mailbox server has a "real" name, then that, and any "role" name you use for ZWC user access must be secondary names, and the Active Sync name (I chose "async") must be primary.
That script calls my 'timestamp' script, which is just:
date +%Y%m%d-%H%M%S
(end contrib)
Ldap And-Or MTA Doesn't Start After Cert Changes Or Upgrade
Brief summary of issues:
- The ca directory contained extra links to different certificates. This seemed to not bother 5.0.9 but under 5.0.10 postfix has a fit if there are more then 3 files in that directory.
Example of LDAP error:
[zimbra@server-01 ~]$ zmcontrol start Host server-01.DOMAIN.com Starting ldap...Done. FAILED Failed to start slapd. Attempting debug start to determine error. TLS: error:0906D06C:PEM routines:PEM_read_bio:no start line pem_lib.c:647 TLS: error:0906D06C:PEM routines:PEM_read_bio:no start line pem_lib.c:647 TLS: error:0906D06C:PEM routines:PEM_read_bio:no start line pem_lib.c:647 TLS: error:0906D06C:PEM routines:PEM_read_bio:no start line pem_lib.c:647 TLS: error:0906D06C:PEM routines:PEM_read_bio:no start line pem_lib.c:647 TLS: error:0906D06C:PEM routines:PEM_read_bio:no start line pem_lib.c:647 TLS: error:0906D06C:PEM routines:PEM_read_bio:no start line pem_lib.c:647 TLS: error:0906D06C:PEM routines:PEM_read_bio:no start line pem_lib.c:647 TLS: error:02001002:system library:fopen:No such file or directory bss_file.c:352 TLS: error:20074002:BIO routines:FILE_CTRL:system lib bss_file.c:354 main: TLS init def ctx failed: -1
The fix is described in:
Details as related to MTA/Postfix - Error_(MTA):_Unable_to_set_STARTTLS
More details as shared by a customer:
I moved the "ca" directory to "ca.BAK" to back it up, made a new ca directory, and then went through the steps detailed in that post. Once done, zimbra started up without a problem: 1) Clear all the contents of the /opt/zimbra/conf/ca directory by backing them up somewhere on disk. 2) Copy the /opt/zimbra/ssl/zimbra/commercial/commercial.key /opt/zimbra/conf/ca/ca.key 3) Copy /opt/zimbra/ssl/zimbra/commercial/commercial.crt /opt/zimbra/conf/ca/ca.pem 4) Create the hash value ln -f -s ca.pem /opt/zimbra/conf/ca/`openssl x509 -hash -noout -in /opt/zimbra/conf/ca/ca.pem`.0 5) Chmod 644 /opt/zimbra/conf/ca/* 6) Restart the zmcontrol A sample of what the /opt/zimbra/conf/ca directory looked like: lrwxrwxrwx 1 root root 20 Apr 9 20:29 0bb21872.0 -> commercial_ca_26.pem lrwxrwxrwx 1 root root 20 Apr 9 20:29 0c364b2d.0 -> commercial_ca_14.pem lrwxrwxrwx 1 root root 20 Apr 9 20:29 0e82f83a.0 -> commercial_ca_36.pem lrwxrwxrwx 1 root root 20 Apr 9 20:29 11f154d6.0 -> commercial_ca_49.pem lrwxrwxrwx 1 root root 19 Apr 9 20:29 128b9c8d.0 -> commercial_ca_9.pem lrwxrwxrwx 1 root root 19 Apr 9 20:29 1a147d5b.0 -> commercial_ca_5.pem lrwxrwxrwx 1 root root 20 Apr 9 20:29 1bb6c7e0.0 -> commercial_ca_24.pem lrwxrwxrwx 1 root root 20 Apr 9 20:29 1c647a6d.0 -> commercial_ca_21.pem lrwxrwxrwx 1 root root 20 Apr 9 20:29 256fd83b.0 -> commercial_ca_33.pem lrwxrwxrwx 1 root root 20 Apr 9 20:29 25f0cbee.0 -> commercial_ca_23.pem lrwxrwxrwx 1 root root 20 Apr 9 20:29 2d047263.0 -> commercial_ca_28.pem There we many more of the above entries, all of which pointed to .pem files that didn't exist. There was also: lrwxrwxrwx 1 root root 6 Apr 13 11:43 555ebb99.0 -> ca.pem lrwxrwxrwx 1 root root 17 Apr 9 20:29 8e6e2991.0 -> commercial_ca.pem lrwxrwxrwx 1 root root 17 Apr 13 11:43 c33a80d4.0 -> commercial_ca.pem lrwxrwxrwx 1 root root 20 Apr 9 20:29 c527e4ab.0 -> commercial_ca_57.pem -rw-r--r-- 1 root root 887 Apr 13 11:43 ca.key -rw-r--r-- 1 root root 989 Apr 13 11:43 ca.pem -rw-r--r-- 1 root root 1155 Apr 13 11:43 commercial_ca_1.pem -rw-r--r-- 1 root root 1156 Apr 13 11:43 commercial_ca.pem zmcertmgr output: ./zmcertmgr deploycrt comm /opt/zimbra/certs/server-01.DOMAIN.com.crt /opt/zimbra/certs/ca_chain-server-01.DOMAIN.com.crt ** Verifying /opt/zimbra/certs/server-01.DOMAIN.com.crt against /opt/zimbra/ssl/zimbra/commercial/commercial.key Certificate (/opt/zimbra/certs/server-01.DOMAIN.com.crt) and private key (/opt/zimbra/ssl/zimbra/commercial/commercial.key) match. Valid Certificate: /opt/zimbra/certs/server-01.DOMAIN.com.crt: OK ** Copying /opt/zimbra/certs/server-01.DOMAIN.com.crt to /opt/zimbra/ssl/zimbra/commercial/commercial.crt ** Appending ca chain /opt/zimbra/certs/ca_chain-server-01.DOMAIN.com.crt to /opt/zimbra/ssl/zimbra/commercial/commercial.crt ** Saving server config key zimbraSSLCertificate...done. ** Saving server config key zimbraSSLPrivateKey...done. ** Installing mta certificate and key...done. ** Installing slapd certificate and key...done. ** Installing proxy certificate and key...done. ** Creating pkcs12 file /opt/zimbra/ssl/zimbra/jetty.pkcs12...done. ** Creating keystore file /opt/zimbra/mailboxd/etc/keystore...done. ** Installing CA to /opt/zimbra/conf/ca...done.
Moving Your Certificates To New Or Another Server
Please see Transfer_SSL_certificates_between_servers
Commercial Cert Error - Subject Does Not Start With /
As reported by a customer to me:
When creating a commercial cert for a server the zmcertmgr will fail if you don't supply a subjectAltName ---- Took me awhile to figure this out since the error isn't correctly describing the problem. It says that "Subject does not start with '/'." Which is incorrect. Subject does start with "/" , it's the subjectAltName that was needed. After I supplied this name, it generated the csr. Here are my commands for your own reference.
- I modified the Some* entries below.
/opt/zimbra/bin/zmcertmgr createcsr comm -new -subject "/C=US/ST=SomeState/L=SomeCity/O=Some Community College/OU=ITS/CN=zimbra.somecommunitycollege.edu" -subjectAltNames zimbra.somecommunitycollege.edu
then I went to thawte and applied for a ssl cert.
How To Setup Certs With CACert.org - Free Certs
CaCert.Org References
Free Certs with http://www.cacert.org/
How-To (tested on 5.0.2)
Note, the following :
su - root ; cd /opt/zimbra/ssl/zimbra/commercial/
included in all steps in case someone is skipping through instructions.
Clean up and start fresh
su - root cd /opt/zimbra/ssl/zimbra/commercial/ tar -czvf /tmp/ssl.commercial.backup.tar.gz * rm -rf *
Generate new csr
[ su - root ; cd /opt/zimbra/ssl/zimbra/commercial/ ] /opt/zimbra/bin/zmcertmgr createcsr comm -new
This uses the defaults, note the items to change.
/opt/zimbra/bin/zmcertmgr createcsr comm -new "/C=US/ST=TX/L=Somewhere/O=Company, Inc./OU=ITDepartment/CN=mail.CHANGEME.com"
Confirm
[ su - root ; cd /opt/zimbra/ssl/zimbra/commercial/ ] ls -la
There should only be two files and time/date should match - commercial.csr & commercial.key
cat /opt/zimbra/ssl/zimbra/commercial/commercial.csr -----BEGIN CERTIFICATE REQUEST----- [delete]CCAWwCAQAwgZkxCzAJBgNVBAYTAlVTMQwwCgYDVQQIEwNOL0ExDDAKBgNV [delete]4vQTEjMCEGA1UEChMaWmltYnJhIENvbGxhYm9yYXRpb24gU3VpdGUxIzAh [delete]AsTGlppbWJyYSBDb2xsYWJvcmF0aW9uIFN1aXRlMSQwIgYDVQQDExttYWls [delete]nRlcm5hbC5ob21ldW5peC5jb20wgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJ [delete]AOri9/m6RtM1vASBROPgLvkUYybwf2WDI2xTdKUuAMI0rTpMH1IzjPRP/J+m [delete]RQTiJe1mRX3rJCy3qVooVzsLe2yJ1+rs3FzLSfQhazK6PqMD8GhpqHO0Y75 [delete]LEA/qdOCrTFjosO9C3j3WPCW8lutTxf/QsoKGkIVs5tjAgMBAAGgKTAnBgkq [delete]0BCQ4xGjAYMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgXgMA0GCSqGSIb3DQEB [delete]A4GBAKMLVFilRjI9xvU/vZmP69yReVZyxa5YVpF/cEvwFwbOU6E4USkdONGT [delete]DRj1XxfzYD+CDf8TVuTY4tapaLvKPRUtdd/mM1PidY5t126QAObyKjHBRzy [delete]RJFQeP+0ktxcYJ99+sfiescwR/qzPJM58i6daqmMamQBZi -----END CERTIFICATE REQUEST-----
Sign up for cacert.org use
Sign up - https://www.cacert.org/index.php?id=1
Verify the email the confirmation email.
Add a domain , it will send an email to some "admin" account to the domain. Make sure you can get it before you do this.
Get New Server Certificate
Now do a New Server Certificate from your administration page at http://www.cacert.org/
You'll copy in the /opt/zimbra/ssl/zimbra/commercial/commercial.csr contents in window [all of it].
It'll generate your cert. on the webpage.
Copy this onto the server [paste in cert details]:
vi /opt/zimbra/ssl/zimbra/commercial/commercial.crt -----BEGIN CERTIFICATE----- [delete]TCCAl2gAwIBAgIDBPRRMA0GCSqGSIb3DQEBBQUAMHkxEDAOBgNVBAoTB1Jv [delete]0ExHjAcBgNVBAsTFWh0dHA6Ly93d3cuY2FjZXJ0Lm9yZzEiMCAGA1UEAxMZ [delete]2VydCBTaWduaW5nIEF1dGhvcml0eTEhMB8GCSqGSIb3DQEJARYSc3VwcG9y [delete]NlcnQub3JnMB4XDTA4MDQxMTIwMjQzNloXDTA4MTAwODIwMjQzNlowJjEk [delete]1UEAxMbbWFpbDMuaW50ZXJuYWwuaG9tZXVuaXguY29tMIGfMA0GCSqGSIb3 [delete]AQUAA4GNADCBiQKBgQDq4vf5ukbTNbwEgUTj4C75FGMm8H9lgyNsU3SlLgDC [delete]B9SM4z0T/yfpoZc3yUUE4iXtZkV96yQst6laKFc7C3tsidfq7Nxcy0n0IWs [delete]BoaahztGO+ZgsfCxAP6nTgq0xY6LDvQt491jwlvJbrU8X/0LKChpCFbOb [delete]QABo4HcMIHZMAwGA1UdEwEB/wQCMAAwNAYDVR0lBC0wKwYIKwYBBQUHAwIG [delete]QUFBwMBBglghkgBhvhCBAEGCisGAQQBgjcKAwMwCwYDVR0PBAQDAgWgMDMG [delete]QUFBwEBBCcwJTAjBggrBgEFBQcwAYYXaHR0cDovL29jc3AuY2FjZXJ0Lm9y [delete]QYDVR0RBEowSIIbbWFpbDMuaW50ZXJuYWwuaG9tZXVuaXguY29toCkGCCsG [delete]wgFoB0MG21haWwzLmludGVybmFsLmhvbWV1bml4LmNvbTANBgkqhkiG9w0B [delete]AAOCAgEANzr/jRcEd5BF2QqF+X8deq4Xxp1tN9lFgji38C5ixNJ+Busq9Sk5 [delete]O7YYJQbSf5K14eZyC1jaNEOEwqgzFiM1HRWL1HCca3EM7TXUoH8sMXS1Ng [delete]M5oyfQcFWZYa22CtKQANQEX5l7EYNkr0yvD/YnP02l3hk1jZr+3pszCW6Iw [delete]vabHMYcAXus+iOGgws788QsMaqzoZwla1AaacZ98s0lFAR0xdRiuXCHUFz [delete]meS5sK+med95/z+Mb6ShJzC7KAi1nfZk9CoNHUHVxMis5Cr+GT7MoIvhQ [delete]8fkiANQQoEgam37lyHezPKyc6iLxW4ag2PWKrZa2+3pyTg/6aHKxZR325z [delete]kcdwKYo/eUGaN1tNmsY638N4hCz01FHHKr97W0m4u5wtwKBo4/5Gy9e5nG6 [delete]khOyjfOz6VYvZHNqDaqGJwsxitxSGGDc8bA+9d73RCOFuztwVrKYg5OJ [delete]Ei5C9gWzee7AmoGpgxOrYjgBrx4nuBw71EFzgKSOZqxUxSNiLuGAx+oVd2 [delete]Z4EAPsa90ZNb0mLGagAuTAdccekOqPVnyZrqiINelY7fpAAUvO9rgTSB9A [delete]RxUydTgY1jyBtoXjp59HMVbCkAtOtX43NqIhPYJNPeSoyw/5SU= -----END CERTIFICATE-----
Get Root CA's
Root CA certs are found here : https://www.cacert.org/index.php?id=3
Do the following on the server:
[ su - root ; cd /opt/zimbra/ssl/zimbra/commercial/ ] wget http://www.cacert.org/certs/root.crt mv root.crt commercial_ca.crt
Verify
Let's verify all is good.
[ su - root ; cd /opt/zimbra/ssl/zimbra/commercial/ ] /opt/zimbra/bin/zmcertmgr verifycrt comm commercial.key commercial.crt
Giving something like this:
** Verifying commercial.crt against commercial.key Certificate (commercial.crt) and private key (commercial.key) match. Valid Certificate: commercial.crt: OK
Errors - Double check time
- Date [is it wrong?]
- Install ntpd if it's not
- Stop ntpd :
/etc/init.d/ntpd stop
- Set time with :
ntpdate us.pool.ntp.org
- Confirm time change :
date
- Confirm hardware clock time :
hwclock
- Sync hardware clock time :
hwclock --systohc
- Confirm hardware clock time :
hwclock
- Start ntpd now :
/etc/init.d/ntpd start
Deploy CA
[ su - root ; cd /opt/zimbra/ssl/zimbra/commercial/ ]
/opt/zimbra/bin/zmcertmgr deploycrt comm commercial.crt commercial_ca.crt
Restart the webserver
su - zimbra zmmailboxdctl restart
- This article is NOT official Zimbra documentation. It is a user contribution and may include unsupported customizations, references, suggestions, or information. |
ZCA Appliance Topics
Actual ZCA Appliance Topics Homepage
Please see Ajcody-ZCA_Appliance
EOL Of ZCA
Please see the following:
Where Is The Installer Located On ZCA After Download
Here is an example of the ZCA 8.0.3 Release:
/opt/vmware-zca-installer/packages/zcs-NETWORK-8.0.3_GA_5664.UBUNTU10_64.20130305090216.tgz
Mailboxd Not Running After ZCS 803 Upgrade
Check /opt/zimbra/log/zmmailboxd.out for the following:
2013-03-22 09:44:50.496:INFO:oejpw.PlusConfiguration:No Transaction manager found - if your webapp requires one , please configure one. Total time for which application threads were stopped: 0.0002020 seconds 2013-03-22 09:44:50.788:INFO:oejsh.ContextHandler:started o.e.j.w.WebAppContext{/service,file:/opt/zimbra/jetty -distribution-7.6.2.z4/webapps/service/},/opt/zimbra/jetty-distribution-7.6.2.z4/webapps/service 2013-03-22 09:44:52.458:WARN:oejuc.AbstractLifeCycle:FAILED ZimbraQoSFilter: java.lang.NoSuchMethodError: com.g ooglecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Builder.maximumWeightedCapacity(J)Lcom/googlecode/con currentlinkedhashmap/ConcurrentLinkedHashMap$Builder; java.lang.NoSuchMethodError: com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Builder.maximumWeig htedCapacity(J)Lcom/googlecode/concurrentlinkedhashmap/ConcurrentLinkedHashMap$Builder;
This should be getting fixed ASAP, so hopefully no one else hits it.
- "mailboxd stops after upgrade to 8.0.3"
Hostname Keeps Being Set To Localhost
This is a 'bug' of sorts in the vmware scripts. It requires the hostname to have a valid PTR record, if it doesn't upon a DNS query it will fall back to localhost.
- "vami_set_hostname only valids DNS via TCP resulting in localhost if not available needs workaround"
- https://bugzilla.zimbra.com/show_bug.cgi?id=81262
- Was a clone of this bug:
- https://bugzilla.zimbra.com/show_bug.cgi?id=81262
Steps That Should Work, But Don't Because Of PTR Not Being Set
Edit /etc/hosts as root:
127.0.0.1 localhost.localdomain localhost XXX.XXX.X.X HOSTNAME.DOMAIN.COM HOSTNAME
Edit /etc/hostname as root to have your FQDN:
HOSTNAME.DOMAIN.COM
Push out changes, as root:
/etc/init.d/hostname start
The vmware tool sets the hostname also via this:
# /opt/vmware/share/vami/vami_config_net Main Menu 0) Show Current Configuration (scroll with Shift-PgUp/PgDown) 1) Exit this program 2) Default Gateway 3) Hostname 4) DNS 5) Proxy Server 6) IP Address Allocation for eth0 Enter a menu number [0]:
Time Is Not Set Right
Postfix might not start because of this.
The following RFE is to have this configured so upon start time is pulling from ntp.
- "ntpdate error in syslog after appliance single node install"
To manually update time to pull from a ntp server:
ntpdate ntp.ubuntu.com pool.ntp.org
Increasing Disk - Partition Space
General References On LVM And Commands For It
You should be comfortable with LVM and working with it. Zimbra Support does not support this directly, this is a normal Linux administrative tasks and trouble shooting and training issues should be directed to your Linux OS vendor.
References:
- https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Logical_Volume_Manager_Administration/VG_display.html
- https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Logical_Volume_Manager_Administration/custom_report.html
- A nice walk through on a LVM setup:
Increase - Adding Disk - : Add Storage First in vSphere
This wiki applies to ZCA 8.0.0 and newer.
Adding additional VMDKs for storage is done via the normal vSphere client process.
- For a single server installation, select the Zimbra Appliance; for a multi server installation, select the mailstore server to add storage.
- From Edit, click Add.
- Select Hard Disk and click Next.
- Select Create a new virtual disk. If you are using Raw Device Mapping (RDM), select Raw Device Mappings.
- Select the provision format that is the same as the Zimbra Appliance virtual machine. Recommended is Thick Provision Eager Zero format. If you are deploying on fiber channel storage, this provides the best performance for the appliance.
- Select Specify a datastore or datastore cluster.
- Click Browse to select a datastore to create the virtual disk on.
- Once you have added the appropriately sized VMDKs for your deployment, restart your Zimbra appliance virtual machine. The virtual appliance console indicates the volume group for the virtual appliance that is being increased.
- Login to the storage virtual appliance and as root enter:
- vgdisplay data_vg
- This displays the volume group properties for the Zimbra mailstore server. Verify that the volume group size reflects the size of the added storage plus 12GB.
If you do not see the increase disk space for the data_vg , proceed with the below section.
Manually Adding - Expanding the Disk - Partition In The OS
Note - Ideally the link below should work for you and be the best method to adding/increasing disk space on ZCA
Keyword - zca partition disk full.
All you should have to do is add new disks to be available to the ZCS virtual machine and then reboot the zimbra server. A script should detect it and auto-expand the partition to give you the newly available space. If this does not happen, you'll be able to confirm that the volume group [vg] is still the same size but you will see the new disks with the output of fdisk.
To see what the vg data_vg currently is:
root@localhost:~# vgdisplay data_vg --- Volume group --- VG Name data_vg System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 2 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 1 Max PV 0 Cur PV 1 Act PV 1 VG Size 12.00 GiB PE Size 4.00 MiB Total PE 3071 Alloc PE / Size 3071 / 12.00 GiB Free PE / Size 0 / 0 VG UUID BDfVzQ-zVEP-jiAm-DI1J-2sje-01cc-1rMJ9K
To see what disks are available to you - as root, fdisk -l . Example below :
root@localhost:~# fdisk -l Disk /dev/sda: 8589 MB, 8589934592 bytes 255 heads, 63 sectors/track, 1044 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000a46dc Device Boot Start End Blocks Id System /dev/sda1 * 1 16 123904 83 Linux Partition 1 does not end on cylinder boundary. /dev/sda2 16 1045 8261633 5 Extended /dev/sda5 16 32 123904 82 Linux swap / Solaris /dev/sda6 32 1045 8136704 83 Linux Disk /dev/sdb: 12.9 GB, 12884901888 bytes 255 heads, 63 sectors/track, 1566 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/sdb doesn't contain a valid partition table Disk /dev/sdc: 32.2 GB, 32212254720 bytes 255 heads, 63 sectors/track, 3916 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/sdc doesn't contain a valid partition table Disk /dev/sdd: 53.7 GB, 53687091200 bytes 255 heads, 63 sectors/track, 6527 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/sdd doesn't contain a valid partition table
Warning - What you're about to do is irreversible. You should confirm and double check that you have your data backed up and are knowledgeable about your DR options if things go wrong beyond this point.
To manually add the new disks to the lvm vg to expand the partition that's available to zimbra. Reusing the example from above. As root :
root@localhost:~# pvcreate /dev/sdc root@localhost:~# pvcreate /dev/sdd root@localhost:~# vgextend /dev/data_vg /dev/sdc root@localhost:~# vgextend /dev/data_vg /dev/sdd root@localhost:~# lvextend -L +79.99g /dev/data_vg/zimbra root@localhost:~# resize2fs /dev/data_vg/zimbra
Additional Information
Very useful commands that will show information about your PV's, LV's, and VG's :
- pvs -v
- lvs -v
- vgs -v
References about LVM:
- https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Logical_Volume_Manager_Administration/VG_display.html
- https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Logical_Volume_Manager_Administration/custom_report.html
- Nice walk through:
- http://www.idevelopment.info/data/Unix/Linux/LINUX_ManagingPhysicalLogicalVolumes.shtml
Zimlets
You are looking at legacy Zimlet documentation. For Zimbra Modern UI Zimlet development go to: https://wiki.zimbra.com/wiki/DevelopersGuide#Zimlet_Development_Guide. |
Actual Zimlet Notes Homepage
Please see Ajcody-Zimlet-Notes
Zimlets In ZCS 8+
List Of Supported Zimlets
$ find /opt/zimbra/zimlets -name *.zip -print /opt/zimbra/zimlets/com_zimbra_url.zip /opt/zimbra/zimlets/com_zimbra_tooltip.zip /opt/zimbra/zimlets/com_zimbra_clientuploader.zip /opt/zimbra/zimlets/com_zimbra_srchhighlighter.zip /opt/zimbra/zimlets/com_zimbra_viewmail.zip /opt/zimbra/zimlets/com_zimbra_ymemoticons.zip /opt/zimbra/zimlets/com_zimbra_date.zip /opt/zimbra/zimlets/com_zimbra_email.zip /opt/zimbra/zimlets/com_zimbra_adminversioncheck.zip /opt/zimbra/zimlets/com_zimbra_bulkprovision.zip /opt/zimbra/zimlets/com_zimbra_attachmail.zip /opt/zimbra/zimlets/com_zimbra_cert_manager.zip /opt/zimbra/zimlets/com_zimbra_webex.zip /opt/zimbra/zimlets/com_zimbra_phone.zip /opt/zimbra/zimlets/com_zimbra_attachcontacts.zip /opt/zimbra/zimlets/com_zimbra_proxy_config.zip
$ find /opt/zimbra/zimlets-network -name *.zip -print /opt/zimbra/zimlets-network/com_zimbra_backuprestore.zip /opt/zimbra/zimlets-network/com_zimbra_archive.zip /opt/zimbra/zimlets-network/com_zimbra_click2call_mitel.zip /opt/zimbra/zimlets-network/com_zimbra_voiceprefs.zip /opt/zimbra/zimlets-network/com_zimbra_click2call_cisco.zip /opt/zimbra/zimlets-network/com_zimbra_smime_cert_admin.zip /opt/zimbra/zimlets-network/com_zimbra_convertd.zip /opt/zimbra/zimlets-network/com_zimbra_smime.zip /opt/zimbra/zimlets-network/com_zimbra_license.zip /opt/zimbra/zimlets-network/com_zimbra_delegatedadmin.zip /opt/zimbra/zimlets-network/com_zimbra_mobilesync.zip /opt/zimbra/zimlets-network/com_zimbra_xmbxsearch.zip /opt/zimbra/zimlets-network/com_zimbra_cluster.zip /opt/zimbra/zimlets-network/com_zimbra_hsm.zip /opt/zimbra/zimlets-network/com_zimbra_ucconfig.zip
Undeploy All Zimlets And Redeploy Supported Zimlets
Let's flush the cache in regards to zimlets before we start:
zmprov fc zimlet
Create a text file of your currently deployed zimlets . Note - save this for future reference so you can recall what was removed.
/opt/zimbra/bin/zmzimletctl listZimlets
and
ls /opt/zimbra/zimlets-deployed
Add the zimlets to a file - This Is An Example Below - Be Sure Your Text File Lists YOUR Zimlets:
vi /tmp/list com_zimbra_adminversioncheck com_zimbra_apptsummary com_zimbra_attachcontacts com_zimbra_attachmail com_zimbra_backuprestore com_zimbra_bulkprovision com_zimbra_cert_manager com_zimbra_coloredemails com_zimbra_contactcleaner com_zimbra_contactorganizer com_zimbra_convertd com_zimbra_date com_zimbra_delegatedadmin com_zimbra_dnd com_zimbra_email com_zimbra_emailreminder com_zimbra_emailtemplates com_zimbra_hsm com_zimbra_license com_zimbra_linkedin com_zimbra_local com_zimbra_meebo com_zimbra_mobilesync com_zimbra_smime com_zimbra_social com_zimbra_srchhighlighter com_zimbra_stickynotes com_zimbra_tracking com_zimbra_url com_zimbra_webex com_zimbra_xmbxsearch com_zimbra_ycurrency com_zimbra_yfinance com_zimbra_ymaps com_zimbra_ymemoticons
Use the file to undeploy:
for i in `cat /tmp/list`; do ; zmzimletctl undeploy $i ; done
To deploy the basic supported zimlets:
vi /tmp/install-list /opt/zimbra/zimlets/com_zimbra_url.zip /opt/zimbra/zimlets/com_zimbra_tooltip.zip /opt/zimbra/zimlets/com_zimbra_clientuploader.zip /opt/zimbra/zimlets/com_zimbra_srchhighlighter.zip /opt/zimbra/zimlets/com_zimbra_viewmail.zip /opt/zimbra/zimlets/com_zimbra_ymemoticons.zip /opt/zimbra/zimlets/com_zimbra_date.zip /opt/zimbra/zimlets/com_zimbra_email.zip /opt/zimbra/zimlets/com_zimbra_adminversioncheck.zip /opt/zimbra/zimlets/com_zimbra_bulkprovision.zip /opt/zimbra/zimlets/com_zimbra_attachmail.zip /opt/zimbra/zimlets/com_zimbra_cert_manager.zip /opt/zimbra/zimlets/com_zimbra_webex.zip /opt/zimbra/zimlets/com_zimbra_phone.zip /opt/zimbra/zimlets/com_zimbra_attachcontacts.zip /opt/zimbra/zimlets/com_zimbra_proxy_config.zip /opt/zimbra/zimlets-network/com_zimbra_backuprestore.zip /opt/zimbra/zimlets-network/com_zimbra_smime_cert_admin.zip /opt/zimbra/zimlets-network/com_zimbra_convertd.zip /opt/zimbra/zimlets-network/com_zimbra_smime.zip /opt/zimbra/zimlets-network/com_zimbra_license.zip /opt/zimbra/zimlets-network/com_zimbra_delegatedadmin.zip /opt/zimbra/zimlets-network/com_zimbra_mobilesync.zip /opt/zimbra/zimlets-network/com_zimbra_xmbxsearch.zip /opt/zimbra/zimlets-network/com_zimbra_hsm.zip
Those that are absent from the above list from /opt/zimbra/zimlets & /opt/zimbra/zimlets-network are:
/opt/zimbra/zimlets-network/com_zimbra_archive.zip /opt/zimbra/zimlets-network/com_zimbra_click2call_mitel.zip /opt/zimbra/zimlets-network/com_zimbra_voiceprefs.zip /opt/zimbra/zimlets-network/com_zimbra_click2call_cisco.zip /opt/zimbra/zimlets-network/com_zimbra_cluster.zip /opt/zimbra/zimlets-network/com_zimbra_ucconfig.zip
To deploy the zimlets from /tmp/install-list :
for i in `cat /tmp/install-list`; do ; zmzimletctl deploy $i ; done
Or to deploy ALL supported zimlets:
cd /opt/zimbra/zimlets for i in `ls` ; do zmzimletctl deploy $i ; done
cd /opt/zimbra/zimlets-network for i in `ls` ; do zmzimletctl deploy $i ; done
Flush the cache again in regards to zimlets:
zmprov fc zimlet
Depending on issue or further troubles, you might want to restart the mailboxd service:
zmmailboxdctl restart
Do a current listing of your installed zimlets and confirm it's what you expect:
/opt/zimbra/bin/zmzimletctl listZimlets
Zimlet Changes In ZCS 6
New Directory Path For Deployed Zimlets
From ZCS 5, it was:
zmlocalconfig zimlet_directory zimlet_directory = ${mailboxd_directory}/webapps/service/zimlet **where mailboxd was /opt/zimbra/jetty/**
Under ZCS 6:
zimlet_directory = zimlet_directory = /opt/zimbra/zimlets-deployed
The related bug/rfe:
- "move zimlet repository out of service webapp"
Can't Deploy Zimlets - Admin Or Others
This variable & directory seems to have been dropped with 6.0.5+ .
If you can't deploy zimlets and mailbox.log is logging an error about unable to locate file, check the following exists.
drwxr-xr-x 17 zimbra zimbra 578 Nov 4 13:55 /opt/zimbra/zimlets-properties
zmlocalconfig zimlet_properties_directory zimlet_properties_directory = /opt/zimbra/zimlets-properties
Location Of Zimlets
Zimlets should be already located on the zimbra server in one of these directories:
/opt/zimbra/zimlets/ /opt/zimbra/zimlets-admin-extra/ /opt/zimbra/zimlets-extra/ /opt/zimbra/zimlets-experimental/ /opt/zimbra/zimlets-network/
How To List Currently Installed Zimlets
Do the following:
zmzimletctl listZimlets
You can also see them in the admin console.
Configuration > Zimlets
Configuration > Admin Extensions
How To Deploy Zimlets
To deploy a zimlet, simply cd to the directory where the zimlet is located and issue this command:
/opt/zimbra/bin/zmzimletctl deploy <zimlet_name>
Something like:
zmzimletctl deploy /opt/zimbra/zimlets-extra/com_zimbra_ycurrency.zip
You can also deploy them via the admin console.
Configuration > Zimlets
Configuration > Admin Extensions
How To Undeploy / Uninstall Zimlets
See how the zimlet is named:
zmzimletctl listZimlets
Now run the following with the naming convention used from the above output:
zmzimletctl undeploy com_zimbra_ycurrency
You can also undeploy them via the admin console.
Configuration > Zimlets
Configuration > Admin Extensions
Samba & Posix Zimlet - ZCS 6x
Main Samba & Posix How-To Reference
Please see:
Important Bugs-RFE's Related To Samba Posix Issues
Please see:
- Post Upgrade Issues
- Password sync between ZCS and Windows no longer working after upgrade to 608+
- "Support change password listeners in provisioning and support Samba change password in the samba admin extension"
- https://bugzilla.zimbra.com/show_bug.cgi?id=17321
- See comment 27
- https://bugzilla.zimbra.com/show_bug.cgi?id=17321
- "zimbraPasswordChangeListener resets to syncListener after zcs restart"
- "Support change password listeners in provisioning and support Samba change password in the samba admin extension"
- "Accounts disappear in Admin-Console after Update"
- Note with bug from devs
- "This is because slapd.conf.in is always reset on upgrade. This has been the case for all releases. With the move to cn=config in GnR, it will be possible for people to keep additional schema loaded across upgrades after their first initial move to GnR and adding them back in."
- http://bugzilla.zimbra.com/show_bug.cgi?id=33628
- The below bug will probably be marked a dup of 33628
- "Upgrade for zimbra_posixaccount and zimbra_samba"
- Note with bug from devs
- Password sync between ZCS and Windows no longer working after upgrade to 608+
Samba & Posix Zimlet - ZCS 5x
Main Samba & Posix How-To Reference
Please see:
Important Bugs-RFE's Related To Samba Posix Issues
Please see:
- Post Upgrade Issues
- Password sync between ZCS and Windows no longer working after upgrade to 608+
- "Support change password listeners in provisioning and support Samba change password in the samba admin extension"
- https://bugzilla.zimbra.com/show_bug.cgi?id=17321
- See comment 27
- https://bugzilla.zimbra.com/show_bug.cgi?id=17321
- "zimbraPasswordChangeListener resets to syncListener after zcs restart"
- "Support change password listeners in provisioning and support Samba change password in the samba admin extension"
- "Accounts disappear in Admin-Console after Update"
- Note with bug from devs
- "This is because slapd.conf.in is always reset on upgrade. This has been the case for all releases. With the move to cn=config in GnR, it will be possible for people to keep additional schema loaded across upgrades after their first initial move to GnR and adding them back in."
- http://bugzilla.zimbra.com/show_bug.cgi?id=33628
- The below bug will probably be marked a dup of 33628
- "Upgrade for zimbra_posixaccount and zimbra_samba"
- Note with bug from devs
- "BNR fails to restore accounts that were created and backed up prior to Zimbra-Samba integration"
- "zmrestore fails when posix & samba zimlets are active."
- "posix extension requires memberUid to contain a uidNumber"
- This is related to the restore issues as well.
- http://bugzilla.zimbra.com/show_bug.cgi?id=26423
- Password sync between ZCS and Windows no longer working after upgrade to 608+
- Other Issues
- "Have the Unix Windows LDAP Samba extensions installed and configured by default"
- Basically a request to have Samba/Posix items to more integrated into product and include more admin console UI controls.
- http://bugzilla.zimbra.com/show_bug.cgi?id=22509
- "Suggestions to improve Posix and Samba Zimlets"
- Items requested:
- a) Add an option to expire the Samba password to force them change the password.
- b) In the memberuid option under Posix Groups: Could you add an option to allow the users to select a single or multiple zimbra users to fill them up quickly?
- c) Add an additional button to display all users that belongs to this particular Posix Group.
- d) Add an additional button in the user profile screen that displays all the groups that he/she belongs to.
- http://bugzilla.zimbra.com/show_bug.cgi?id=18141
- Items requested:
- "Have the Unix Windows LDAP Samba extensions installed and configured by default"
Samba - LDAP - Overlays
We don't [officially] support running additional overlays with OpenLDAP.
SLAPO-RWM OVERLAY RWM
slapo-rwm is known to be buggy in OpenLDAP 2.3.43 and continues to be buggy to this day in OpenLDAP 2.4. It certainly won't work with ZCS 5.0.16.
We would advise customers to avoid using it until it stabilizes, though they need to understand it's still going to be unsupported by us.
Where one places "overlay rwm" in the slapd.conf file has been known to cause issues as well. RWM has problems in the order in which it is loaded. There are at least 2 open bugs currently in the OpenLDAP ITS tracker.
Can't Manage Users After Removing Samba & Posix Zimlet
This is after you have removed the samba & posix zimlets and now can't see or manage old accounts in the admin ui. You might need to remove the samba/posix references in each user account. You'll need the nis.schema and samba.schema configured for ldap for this to work.
Untested comment, 3 things needed for this.
1. deployed samba/posix zimlet
2. add/have the samba/nis schema
3. add/have the extra oc's
zmprov mcf +zimbraAccountExtraObjectClass posixAccount zmprov mcf +zimbraAccountExtraObjectClass sambaSamAccount
If you do this on one of the old accounts:
zmprov ga user@domain.com
And you see:
objectClass: posixAccount objectClass: sambaSamAccount
These steps might need to be done.
Create a file called mod.ldif . Modify the dn line - dn: uid=posix1,ou=people,dc=testdomain,dc=com - for your server and user.
# posix1, people, testdomain.com dn: uid=posix1,ou=people,dc=testdomain,dc=com changetype: modify delete: objectClass objectClass: posixAccount - delete: objectClass objectClass: sambaSamAccount - delete: uidNumber - delete: gidNumber - delete: loginShell - delete: sambaAcctFlags - delete: sambaSID - delete: homeDirectory - delete: sambaNTPassword # posix2, people, testdomain.com dn: uid=posix2,ou=people,dc=testdomain,dc=com changetype: modify delete: objectClass objectClass: posixAccount - delete: objectClass objectClass: sambaSamAccount - delete: uidNumber - delete: gidNumber - delete: loginShell - delete: sambaSID - delete: homeDirectory - delete: sambaNTPassword
Then run a command similar to this, modify it for your environment:
ldapmodify -D uid=zimbra,cn=admins,cn=zimbra -w PassWord -H ldap://ldapmaster.hostname.com:389 -x -f /tmp/mod.ldif
Problems With The Above Steps?
Please see :
http://wiki.zimbra.com/index.php?title=King0770-Notes#LDAP_-_Export_.26_Reimport
It should be possible to modify the dump and the re-import. This has not been tested yet, though.
Steps done in one test. Please note, you'll still need to visually review the ldap file to see what lines need to be remove, this can't be scripted out.
-as zimbra- zimbra$ libexec/zmslapcat /tmp/ldap zimbra$ egrep -iv 'sambaSamAccount|posixAccount|uidNumber|gidNumber|loginShell|sambaAcctFlags|sambaSID|homeDirectory|sambaNTPassword' ldap.bak > ldap.bak2 zimbra$ egrep -i 'samba|posix' ldap.bak2 zimbra$ vi ldap.bak2 zimbra$ egrep -i 'loginshell|HOMEDIRECTORY|MEMBERUID' ldap.bak2 zimbra$ vi ldap.bak2 zimbra$ zmcontrol stop zimbra$ ps -aux | grep slapd 6) su - root 7) mv /opt/zimbra/openldap-data /opt/zimbra/openldap-data.OLD 8) mkdir -p /opt/zimbra/openldap-data/logs 9) chown -R zimbra:zimbra /opt/zimbra/openldap-data/ 10) su - zimbra zimbra$ cd /opt/zimbra/openldap-data.OLD zimbra$ cp DB_CONFIG ../openldap-data/ zimbra$ cd zimbra$ /opt/zimbra/openldap/sbin/slapadd -f /opt/zimbra/conf/slapd.conf -l /tmp/ldap/ldap.bak2 The first database does not allow slapadd; using the first available one (2) is_entry_objectclass("cn=IT,ou=groups,dc=XXXXX,dc=com", "2.16.840.1.113730.3.2.6") no objectClass attribute slapadd: dn="cn=IT,ou=groups,dc=XXXXX,dc=com" (line=11179): no objectClass attribute zimbra$ zmcontrol start ### Output ### assigned-72-29-183-240:~ zimbra$ libexec/zmslapcat /tmp/ldap2/ UNKNOWN attributeDescription "LOGINSHELL" inserted. UNKNOWN attributeDescription "HOMEDIRECTORY" inserted. UNKNOWN attributeDescription "SAMBAACCTFLAGS" inserted. UNKNOWN attributeDescription "SAMBASID" inserted. UNKNOWN attributeDescription "SAMBADOMAINNAME" inserted. UNKNOWN attributeDescription "SAMBANTPASSWORD" inserted. UNKNOWN attributeDescription "SAMBAALGORITHMICRIDBASE" inserted. UNKNOWN attributeDescription "SAMBANEXTUSERRID" inserted. UNKNOWN attributeDescription "SAMBAMINPWDLENGTH" inserted. UNKNOWN attributeDescription "SAMBALOGONTOCHGPWD" inserted. UNKNOWN attributeDescription "SAMBAMAXPWDAGE" inserted. UNKNOWN attributeDescription "SAMBAMINPWDAGE" inserted. UNKNOWN attributeDescription "SAMBALOCKOUTDURATION" inserted. UNKNOWN attributeDescription "SAMBALOCKOUTOBSERVATIONWINDOW" inserted. UNKNOWN attributeDescription "SAMBALOCKOUTTHRESHOLD" inserted. UNKNOWN attributeDescription "SAMBAFORCELOGOFF" inserted. UNKNOWN attributeDescription "SAMBAREFUSEMACHINEPWDCHANGE" inserted. UNKNOWN attributeDescription "SAMBAPWDHISTORYLENGTH" inserted. UNKNOWN attributeDescription "SAMBAGROUPTYPE" inserted. UNKNOWN attributeDescription "MEMBERUID" inserted. UNKNOWN attributeDescription "SAMBAPASSWORDHISTORY" inserted. UNKNOWN attributeDescription "SAMBAPWDLASTSET" inserted.
I Lost My Users After An Upgrade - Samba Posix Zimlet
This is from a case I saw and how a customer fixed it.
- In order to fix this we did.
- 1. Replace slapd.conf.in
- 2. restarted zimbra (not sure if this is necessary, but its what we did.)
- 3. zmprov mcf +zimbraAccountExtraObjectClass posixAccount
- 4. zmprov mcf +zimbraAccountExtraObjectClass sambaSamAccount
- 5. ldap stop && ldap start && ldap stop && ldap start
- The odd thing is, I looked in the zmprov gcf originally and posixAccount and sambaSamAccount had already been added?
Upgrade Or Installation Of New Package On Zimbra Broken Samba - Another Situation
And customer was kind enough to write up a summary of our session when troubling shooting this. We believe it will be of use for others.
Brief overview of your configuration/setup
The Zimbra server is used as LDAP master server. It's easy to maintain and it's very easy to manage hybrid Zimbra/Posix/Samba user accounts. It provides LDAP service for Samba 3.0.24 and PAM on Debian Etch 4.0 servers and desktops. Additionally, I've got few LDAP replica servers which use syncrepl mechanism to get required Posix and Samba data from Zimbra server. As Posix and Samba objects are in use it's very important to keep the tweaked /opt/zimbra/conf/slapd.conf.in file the same after upgrade/ re-installation.
System spec:
- OS is Ubuntu 7.10 with all latest patches
- Zimbra 5.0.11
- LDAP 2.3.43.5z (/opt/zimbra/conf/slapd.conf configuration file)
- Zimbra zimlets-admin-extra: zimbra_posixaccount, zimbra_samba
- native packages Samba 3.0.24 on Debian 4.0 Etch with PAM and libnss-ldap, pam-ldap
- smbldap-tools 0.9.5 from tar file
The symptoms and what you did to confirm the issue
Operations that failed at end-user & admin
Because of the other issue with Zimbra server we had to install convertd on the box. To avoid any other unknown problems we were advised to re-run installation script from the zcs-NETWORK-5.0.11_GA_2695.UBUNTU6.20081117023813 folder on local file system.
During the process we confirmed installation of 'convertd'. So, from this point of time it's installed. The Zimbra installer restarted a few times slapd service. I'm not sure but I believe that during this process it dumps whole LDAP objects and clears it in directory. Then it loads them back to directory.
The odd thing is that THE INSTALLER RE-GENERATES the '/opt/zimbra/conf/slapd.conf.in' file to the standard one from the new package.
This is the real source of the problem for other objects than Zimbra's ones. i.e. Posix and Samba
After I restored the changes required for Posix and Samba in /opt/zimbra/conf/slapd.conf.in' file as described on Zimbra Wiki it seems that Samba and Posix attributes were unaccessible on all clients. ( UNIX_and_Windows_Accounts_in_Zimbra_LDAP_and_Zimbra_Admin_UI )
Symptoms
[ Documentation substituted 'my_corp.net' for real domain name ]
1. "Samba can't join any new box to domain MY_CORP" but this is just tip of the iceberg.
# grep machine /etc/samba/smb.conf ldap machine suffix = ou=machines add machine script = /usr/sbin/smbldap-useradd -t 0 -W "%u"
Firstly, I've tried to test existing object I know in LDAP directory via ldapsearch and it's fails giving zero results.
1A.zimbra# ldapsearch -x -D cn=config -W -h <name-of-zimbra-server> -b ou=machines,dc=my_corp,dc=net uid=my-pc$
I'm trying to query the object itself and then it works which means that all attributes are in directory.
1B.zimbra# ldapsearch -x -D cn=config -W -h <name-of-zimbra-server> -b uid=my-pc$, ou=machines,dc=my_corp,dc=net # extended LDIF # # LDAPv3 # base <uid=my-pc$,ou=machines,dc=my_corp,dc=net> with scope subtree # filter: (objectclass=*) # requesting: ALL # # my-pc$, machines, my_corp.net dn: uid=my-pc$,ou=machines,dc=my_corp,dc=net cn: my-pc$ uid: my-pc$ uidNumber: 1001 gidNumber: 515 loginShell: /bin/false description: Computer gecos: Computer objectClass: posixAccount objectClass: account objectClass: sambaSamAccount displayName: my-pc$ sambaDomainName: MY_CORP sambaSID: S-1-5-21-XXXXXXX-XXXXXXXXXX-XXXXX-3002
1C. One more test to be sure and no results as well:
zimbra# ldapsearch -x -D cn=config -W -h <name-of-zimbra-server> -b dc=my_corp,dc=net uid=my-pc$
1D. Then tests on Samba side.
pdc# pdbedit -Lv my-pc$ smbldap_search_domain_info: Searching for:[(&(objectClass=sambaDomain)(sambaDomainName=MY_CORP))] smbldap_open_connection: connection opened smbldap_search_domain_info: Searching for:[(&(objectClass=sambaDomain)(sambaDomainName=MY_CORP))] smbldap_open_connection: connection opened init_sam_from_ldap: Entry found for user: my-pc$ Unix username: my-pc$ NT username: my-pc$ Account Flags: [W ] User SID: S-1-5-21-XXXXXXX-XXXXXXXXXX-XXXXX-3002 pdb_get_group_sid: Failed to find Unix account for my-pc$ *Primary Group SID: (NULL SID) # THIS FAILS as well Full Name: my-pc$ Home Directory: \\pdc\my-pc_ HomeDir Drive: F: Logon Script: logon.cmd Profile Path: Domain: MY_CORP Account desc: Computer Workstations: Munged dial: Logon time: Logoff time: Tue, 19 Jan 2038 03:14:07 GMT Kickoff time: Tue, 19 Jan 2038 03:14:07 GMT Password last set: Mon, 05 Jan 2009 04:20:59 GMT Password can change: Password must change: Tue, 19 Jan 2038 03:14:07 GMT Last bad password : Bad password count : Logon hours : FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
So Samba finds the object but can't link it to the Posix account via PAM query.
1E. Let's check also Posix attribute on Samba server.
pdc# getent passwd my-pc$
Nothing.
After I switched the log level to 5 in Samba I'd see the communication with LDAP in log.my-pc file on Samba server.
pdc# grep "log level" /etc/samba/smb.conf log level = 5 pdc# grep -v "\[200" /var/log/samba/log.my-pc|less
It turns out that the Samba's search base for machines does not work (see 1A,1C tests above).
I checked also other objects in other branches and situation was similar for ou=people ,ou=machines, ou=groups. No sambaSID or uidNumber/gidNumber attributes were visible for Samba in 'dc=my_corp,dc=net' search base.
Steps To Fix Issue
To fix this problem it is necessary to 're-fresh' affected attributes.
In our case:
- ou=people branch - "sambaSID"
- ou=machines branch - "sambaSID,uidNumber,gidNumber"
- ou=groups branch - "sambaSID,gidNumber"
- ( basedn - root for above branches is dc=my_corp,dc=net)
For ou=people
For ou=people it seems very easy to fix as there is zmprov command you can use to do it.
The syntax should be like this:
zimbra# zmprov ma <full-name>@my_corp.net sambaSID <Samba-SID>
To avoid a manual method (mistype), it's possible to use ldapsearch to create a ready zmprov command list. It requires egrep, awk and sed - standard posix tools present in every Linux system.
As the 'zimbra' user on the Zimbra server I'm sending the output to /tmp/sambaSID-refresh.sh file:
zimbra# ldapsearch -x -h <zimbra-server> -D "cn=config" -W -b ou=people,dc=my-corp,dc=net sambaSID=* uid sambaSID|egrep -v "(^#|^dn)" |awk ' /uid:/ {print "zmprov ma "$2"@my_corp.net"}; /sambaSID/ {print "sambaSID " $2};'|sed '/com$/N;s/\n */ /' > /tmp/user-sambaSID-refresh.sh
It gives the possibility to edit the file before you apply it and remove some entries if not relevant.
To apply the changes simply run the file by bash.
zimbra# bash /tmp/user-sambaSID-refresh.sh
Depends on the amount of users it can take a while. For ~300 users it takes approximately 5-10 min. on busy server.
For ou-machines
For ou=machines it's not as easy and requires using ldapmodify tool and ldif file to be created and imported.
According to the ldapmodify manual we need to create file with multiple entries like the one below
dn: uid=my-pc$,ou=machines,dc=my_corp,dc=net changetype: modify replace: uidNumber uidNumber: 1001 - replace: sambaSID sambaSID: S-1-5-21-XXXXXXX-XXXXXXXXXX-XXXXX-3002 dn: (.....)
Make sure there is empty line before next 'dn:'.
So this task could be also automated by ldap-tools. The command below will create ldif output we can forward to the /tmp/machine-posix-smb-fix.ldif file.
zimbra# ldapsearch -x -h <zimbra-server-name> -D "cn=config" -W -b ou=machines,dc=my_corp,dc=net sambaSID=* uidNumber sambaSID|egrep -v "(^#)"|awk '/dn:/ {print "\n"$0"\nchangetype: modify"}; /uidNumber:/ {print "replace: uidNumber\nuidNumber: "$2"\n-"};/sambaSID:/ {print "replace: sambaSID\nsambaSID: "$2};' > /tmp/machine-posix-smb-fix.ldif
Please review the /tmp/machine-posix-smb-fix.ldif file as this example assumes the uidNumber attribute comes first then sambaSID one.
Then using ldapmodify we can replace the existing attributes from our file. Change command if necessary.
zimbra# ldapmodify -x -h <zimbra-server-name> -D cn=config -W -f /tmp/machine-posix-smb-fix.ldif
For ou=groups
For ou=groups it is possible to use Zimbra Admin web interface(RECOMMENDED).
If you don't have to many Posix groups you can easily go to Zimbra Admin web interface and click on "Posix Groups" in the menu. Then double-click on the required group and edit the 'gidNumber' and 'sambaSID' adding one extra digit and save. Then open again and return to the previous value and save again.
But as I mentioned before it's very easy to mistype/remove something important. We can use ldapsearch and create the appropriate ldif file as in ou=machines case above.
zimbra# ldapsearch -x -h <zimbra-server-name> -D "cn=config" -W -b ou=groups,dc=my_corp,dc=net sambaSID=* gidNumber sambaSID|egrep -v "(^#)"|awk '/dn:/ {print "\n"$0"\nchangetype: modify"}; /gidNumber:/ {print "replace: gidNumber\ngidNumber: "$2"\n-"};/sambaSID:/ {print "replace: sambaSID\nsambaSID: "$2"\n"};' > /tmp/groups-posix-smb-fix.ldif
Please review the /tmp/groups-posix-smb-fix.ldif file as this example assumes the gidNumber attribute comes first then sambaSID one. Change this command if necessary.
Admin/End-User test that were performed to confirm complete resolution
Last step is to test previously failed searches on Zimbra server. (Step Symptoms 1C from above)
zimbra# ldapsearch -x -h <name-of-zimbra-server> -b dc=my_corp,dc=net uid=my-pc$
This time it gives full list of attributes for my-pc$
Then on Samba server
pdc# pdbedit -Lv my-pc$ smbldap_search_domain_info: Searching for:[(&(objectClass=sambaDomain)(sambaDomainName=MY_CORP))] smbldap_open_connection: connection opened smbldap_search_domain_info: Searching for:[(&(objectClass=sambaDomain)(sambaDomainName=MY_CORP))] smbldap_open_connection: connection opened init_sam_from_ldap: Entry found for user: my-pc$ Unix username: my-pc$ NT username: my-pc$ Account Flags: [W ] User SID: S-1-5-21-XXXXXXX-XXXXXXXXXX-XXXXX-3002 init_group_from_ldap: Entry found for group: 515 init_group_from_ldap: Entry found for group: 515 Primary Group SID: S-1-5-21-XXXXXXX-XXXXXXXXXX-XXXXX-515 Full Name: my-pc$ Home Directory: \\pdc\my-pc_ HomeDir Drive: F: Logon Script: logon.cmd Profile Path: Domain: MY_CORP Account desc: Computer Workstations: Munged dial: Logon time: Logoff time: Tue, 19 Jan 2038 03:14:07 GMT Kickoff time: Tue, 19 Jan 2038 03:14:07 GMT Password last set: Mon, 05 Jan 2009 04:20:59 GMT Password can change: Password must change: Tue, 19 Jan 2038 03:14:07 GMT Last bad password : Bad password count : Logon hours : FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
Let's check also Posix attribute on Samba server. (Step Symptoms 1E from above)
pdc# getent passwd my-pc$ my-pc$:*:1001:515:Computer::/bin/false
Great it works.
Now Let's see if we can join new box to the Domain
pdc# /usr/sbin/smbldap-useradd -t 0 -W my-pc2 pdc# pdbedit -Lv my-pc2$ smbldap_search_domain_info: Searching for:[(&(objectClass=sambaDomain)(sambaDomainName=MY_CORP))] smbldap_open_connection: connection opened smbldap_search_domain_info: Searching for:[(&(objectClass=sambaDomain)(sambaDomainName=MY_CORP))] smbldap_open_connection: connection opened init_sam_from_ldap: Entry found for user: my-pc2$ Unix username: my-pc2$ NT username: my-pc2$ Account Flags: [W ] User SID: S-1-5-21-XXXXXXX-XXXXXXXXXX-XXXXX-3004 init_group_from_ldap: Entry found for group: 515 init_group_from_ldap: Entry found for group: 515 Primary Group SID: S-1-5-21-XXXXXXX-XXXXXXXXXX-XXXXX-515 Full Name: my-pc2$ Home Directory: \\pdc\my-pc2_ HomeDir Drive: F: Logon Script: logon.cmd Profile Path: Domain: MY_CORP Account desc: Computer Workstations: Munged dial: Logon time: Logoff time: Tue, 19 Jan 2038 03:14:07 GMT Kickoff time: Tue, 19 Jan 2038 03:14:07 GMT Password last set: Mon, 05 Jan 2009 04:20:59 GMT Password can change: Password must change: Tue, 19 Jan 2038 03:14:07 GMT Last bad password : Bad password count : Logon hours : FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF pdc# getent passwd my-pc2$ my-pc2$:*:1002:515:Computer::/bin/false
Yes, everything seems to be back up and running.
Conclusions (Of Customer)
In my personal opinion Zimbra installer should prevent changes to the /opt/zimbra/conf/slapd.conf.in file. Or if not to all file then just to the 'include /<path>/*.schema' directives.
So, "include /<path>/samba.schema" and "include /<path>/nis.schema" should be populated in the re-generated file. Any other changes seems not important as we can apply them after installation. This is required to ensure that server recognizes these attributes and won't break integrity of data.
Additionally, I'm not sure if there is any better solution. Maybe there is, but this one seem to be the quickest in terms of my knowledge about Zimbra installer behavior.
RFE To Address
Please see:
- "Accounts disappear in Admin-Console after Update"
Hostname And DNS Issues
Actual Hostname And DNS Issues Homepage
Please see Ajcody-Hostname-DNS
Hostname resolution issues and testing commands
Allot of failed installs are because administrators are skipping the steps to make sure resolution is working.
Check your /etc/hosts file. You should have the localhost similar to the one below. Your ZCS server name should be replaced with the zimbra.test.com and zimbra fields. The IP address should be the IP address bound to the network interface [ip addr or ifconfig -a to confirm].
127.0.0.1 localhost.localdomain localhost 192.168.0.1 zimbra.test.com zimbra
If your ZCS server is behind a firewall or is being NAT'd. Make sure that resolution for the hostname on the ZCS server returns the internal IP information rather than the external IP. To test:
hostname -f zimbra.test.com
or
domainname -f zimbra.test.com
As long as one of the above returns the full hostname, use the command for the following. The MX lookups depend on how you are doing your mail domain and server hostname.
host `hostname -f` zimbra.test.com has address 192.168.0.1 host -t MX `domainname -f` zimbra.test.com mail is handled by 10 zimbra.test.com. host -t MX `domainname -d` test.com mail is handled by 10 zimbra.test.com.
You should also have reverse records (PTR) as well. Replace 192.168.0.1 using your internal ip address, it should return something like:
host 192.168.0.1 1.0.168.192.in-addr.arpa domain name pointer zimbra.test.com
Please review wiki page on split-DNS. This also shows the basics of actually setting up DNS/Bind on Linux:
http://wiki.zimbra.com/index.php?title=Split_DNS
The following is a more complete how-to for setting up DNS/Bind:
http://www.zimbrablog.com/blog/archives/2007/06/making-zimbra-bind-work-together.html/
lmtp_host_lookup
Also review the variable lmtp_host_lookup to see if it applies to your situation:
http://bugzilla.zimbra.com/show_bug.cgi?id=27988
su - zimbra zmlocalconfig -e postfix_lmtp_host_lookup=native postfix reload
If the above value doesn't exist in your ZCS version, then you would do the following [Older ZCS version]:
su - zimbra postconf | grep host_lookup postconf -e lmtp_host_lookup=native postfix reload
smtp_host_lookup
Similar to above.
disable_dns_lookups
Similar to above.
Local DNS-BIND Configuration Example
The full blown example on setting up BIND is at Split_DNS article.
Detailed Example Zone Files
This example is from a CentOS5 test box I have behind a firewall. I actually setup a dynamic dns using a valid domainname, I've replaced my "real" domainname below with "example". I'm using DynDNS to do this.
Directory Structure
The default setup is a chroot configuration of BIND.
- The root of the configuration files is /var/named/chroot/ .
- The general bind configuration files are in /var/named/chroot/etc/
- Files you'll have there:
- db.cache
- localtime
- named.conf
- rndc.key
- Files you'll have there:
- The directory where your zone files will reside is /var/named/chroot/var/named/
- I have in this directory the following:
- data (a directory that's empty)
- slaves (a directory that's empty)
- internal.example.com.hosts (file for A, NS, MX, CNAME, etc. records)
- 0.168.192.rev (file for PTR records)
- I have in this directory the following:
/var/named/chroot/etc/named.conf example
Here's my named.conf file I'm using:
options { directory "/etc"; pid-file "/var/run/named/named.pid"; }; zone "." { type hint; file "/etc/db.cache"; }; zone "internal.example.com" { type master; file "/var/named/internal.example.com.hosts"; }; zone "0.168.192.in-addr.arpa" { type master; file "/var/named/0.168.192.rev"; };
/var/named/chroot/var/named/internal.example.com.hosts
The _xmpp-server._tcp entries are related to jabber/xmpp and shouldn't be necessary for a default zimbra install. I'll leave them here just for a reference in case they are useful for others depending on the circumstances.
I had a second test box "mail4" at one point but I shut it down.
The one MX record with the starting blank space :
IN MX 10 mail3
would "assume" for the internal.example.com domain.
My only physical box is "mail3". The other names and domains are done so I can configure multiple domains on this box. The blank space MX record (for internal.example.com) is also setup on mail3. So, in total my mail3 box has zimbra configured to handle mail for these five domains:
- internal.example.com
- mail3.internal.example.com
- secondary.internal.example.com
- move.internal.example.com
- alias.internal.example.com
$ttl 38400 internal.example.com. IN SOA internal.example.com. ajcody.mail3.internal.example.com. ( 1207910051 10800 3600 604800 38400 ) IN NS mail3 IN MX 10 mail3 mail3 IN MX 10 mail3 secondary IN MX 10 mail3 move IN MX 10 mail3 alias IN MX 10 mail3 mail4 IN MX 10 mail4 mail3 IN A 192.168.0.16 mail4 IN A 192.168.0.17 _xmpp-server._tcp SRV 5 0 5269 mail3 _xmpp-server._tcp.mail3 SRV 5 0 5269 mail3 _xmpp-server._tcp.secondary SRV 5 0 5269 mail3 _xmpp-client._tcp SRV 5 0 5269 mail3 _xmpp-client._tcp.mail3 SRV 5 0 5269 mail3 _xmpp-client._tcp.secondary SRV 5 0 5269 mail3
/var/named/chroot/var/named/0.168.192.rev
$ttl 38400 $ORIGIN 0.168.192.in-addr.arpa. @ IN SOA mail3.internal.example.com. ajcody.mail3.internal.example.com. ( 1207910558 10800 3600 604800 38400 ) IN NS mail3.internal.example.com. 16 IN PTR mail3.internal.example.com. 17 IN PTR mail4.internal.example.com.
Split-DNS - But I Already Have Bind Running Internally?
You should investigate using ACL's and the views options for BIND to effective setup the "purpose" of split-dns situation without actually running it on a separate server.
Please see:
Other Variables That Are Effected By Lookups
zimbraDNSCHeckHostname New In 5.0.11
If your MX records point to a spam-relay or any other external non-Zimbra server, enter the name of that server in the Inbound SMTP host name field. The "Check MX" button in the admin console compares the domain's MX records in DNS against the zimbraDNSCheckHostname setting, if set. If this attribute is not set, the domain MX setting is checked against zimbraSmtpHostname.
zimbraSmtpHostname
zmprov gs `zmhostname` zimbraSmtpHostname
zimbraMtaRelayHost
zmprov gs `zmhostname` zimbraMtaRelayHost
zimbraMtaDnsLookupEnabled
zmprov gs `zmhostname` zimbraMtaDnsLookupEnabled
zimbraAdminConsoleDNSCheckEnabled
zmprov gs `zmhostname` zimbraAdminConsoleDNSCheckEnabled
lmtp_host_lookup
postconf lmtp_host_lookup lmtp_host_lookup = dns
smtp_host_lookup
postconf smtp_host_lookup smtp_host_lookup = dns
IM - Instant Messaging
Actual Instant Messaging Topics Homepage
Please see Ajcody-Instant-Messaging-Topics
ZCS 8.6 And Above IM Topics
Pre ZCS 8.6 IM Topics
When Will IM Be Out Of Beta - Update
Update - Zimbra will not be release the IM BETA that was intended for ZCS 8.7. Instead, they have release Zimbra Talk.
Please track the following bug:
- "Remove IM server"
- Feature Notes states, "Remove embedded IM server from delivery and web client from Zimbra interface."
- Will be removed for IronMaiden - next main release after ZCS 7
- http://bugzilla.zimbra.com/show_bug.cgi?id=45932
- "RFE: include separate IM server"
- Not Committed
- http://bugzilla.zimbra.com/show_bug.cgi?id=47036
- IM GA (no ISP/HSP)
Performance Issues Caused By IM [BETA]
Please note, that this bug has had a significant enough impact on performance of production servers that we recommend not running IM [BETA] if you have no significant reason to do so.
- "IM service connection leak,memory leak"
To turn off IM:
zmprov -l -v mcf full zimbraXMPPEnabled FALSE
And then you'll need to start and start zimbra:
zmcontrol stop zmcontrol start
Enabling IM [BETA] - 5+ ZCS Server
Check to see if it's enabled:
zmprov gacf zimbraXMPPEnabled
If it returns FALSE, you'll need to set it as TRUE:
zmprov -l -v mcf zimbraXMPPEnabled TRUE
Now to enable via the COS's for the users. I'll assume you are using the COS called "default" for your domain below - change it if your using another COS for the variables below.
Check to see what current status is:
zmprov gc default zimbraFeatureIMEnabled zmprov gc default zimbraFeatureInstantNotify
If they return FALSE, set them to TRUE.
zmprov -v mc default zimbraFeatureIMEnabled TRUE zmprov -v mc default zimbraFeatureInstantNotify TRUE
And then you'll need to stop and start zimbra:
zmcontrol stop zmcontrol start
What Happen to AOL IM and MSN?
From the 5.x Release Notes:
- Note: Update about Instant Messaging feature - In order to ensure the scalability of our server software and provide a quality experience to our customers, we have removed IM interoperability with AOL and MSN that was present in previous 5.0.0 beta releases. This functionality may be restored in a future release.
3rd Party Client Setup Up Examples
Please see:
Additional information on those clients can also be found here:
Ajcody-Notes#IM_-_Instant_Messaging_Clients
Addresses things like group chat /conference rooms and so forth.
ZCS to ZCS IM Availability
Please review the following bug and note the DNS SRV entries I posted within it.
- "ZCS to ZCS IM issues."
Integration With External IM Servers
Please see:
- "Integration with external IM servers (XMPP proxy support)"
IM And Legal Intercept And Archiving
Please see:
- "Legal intercept support: IM"
- "Other collaboration data (IM, Docs, Calendars) need to be archived"
- "RFE: Saving IM history from external IM clients"
This might also be relevant:
- "Regulatory Compliance - code to them and achieve the ability to verify in our documentation"
IM - Conference Rooms
Please see the RFE I made:
- "IM : conference room admin settings"
Also the bug/RFE about Conference rooms in Multi-Server Configurations
- "IM Conference Rooms Don't Work in Multi-Server Setup"
Embedded IM Requests To Join Chat Session In Html/Email
Formats are (I think, still investigating this):
- Jabber/XMPP
- YahooIM
- ymsgr: or yim: (not sure if both work or not)
- AOL IM
- aim:
- MSN
- msnim: or msn: (not sure if both work or not)
You'll need to configure your OS to handle the protocol when called to, so that it launches the correct application.
Resource
- Great resource about this topic and how to adjust your OS to handle the different IM url's.
Related Bugs & RFE's
- "ZWC to handle differnt IM url's - option to set ZWC IM client or external client by OS"
- "Allow Calendar Events in ZWC to handle IM url calls"
Buddy Notification In ZWC
Please see:
- "Buddy notifications support for IM presense"
Other General Bugs & RFE's
Please see:
- "Renaming domain does not update zimbra.jiveRoster"
- "Unable to delete IM Buddy whos account has been renamed"
- "Ability to "restart" IM service without a full mailboxd restart required"
Multi-Mailbox Setup Can Not Message Between Each Other
If this is a new setup, make sure you have DNS setup - Ajcody-Instant-Messaging-Topics#IM_And_DNS_SRV_Records .
Check the /opt/zimbra/log/mailbox.log on the mailstores. If you see something like:
2013-03-04 07:59:40,891 INFO [pool-7-thread-1] [] im - IMPersona(user@domain.com ONLINE pri=0 st=null) - Presence.error: <presence to="user@domain.com/zcs" from="yahoo.domain.com" type="error"><error code="404" type="cancel"><remote-server-not-found xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"/></error></presence>
And you can find this type of error also there or when you run zimbra commands from the CLI:
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
You are experiencing a certificate issue and should submit a support case if you need help resolving it.
IM And DNS SRV Records
Show ports are in use:
[root@mail3 ~]# lsof -P -n -i :5222 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME java 5869 zimbra 143u IPv6 6530439 TCP *:5222 (LISTEN) [root@mail3 ~]# lsof -P -n -i :5269 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME java 5869 zimbra 144u IPv6 6530440 TCP *:5269 (LISTEN) [root@mail3 ~]# lsof -P -n -i :5269
An example of what I've added to my internal DNS/BIND zone record:
$ttl 38400 zimbra.DOMAIN.com. IN SOA zimbra.DOMAIN.com. ajcody.zimbra.DOMAIN.com. ( 1207910051 10800 3600 604800 38400 ) IN NS mail3.zimbra.DOMAIN.com. zimbra.DOMAIN.com. IN MX 10 mail3.zimbra.DOMAIN.com. _xmpp-client._tcp.zimbra.DOMAIN.com. SRV 5 0 5222 mail3.zimbra.DOMAIN.com. _xmpp-server._tcp.zimbra.DOMAIN.com SRV 5 0 5269 mail3.zimbra.DOMAIN.com. _jabber._tcp.zimbra.DOMAIN.com. SRV 5 0 5269 mail3.zimbra.DOMAIN.com. mail3 IN A 192.168.0.16
The _jabber._tcp if for backward compatibility for older servers and shouldn't be necessary.
Ajcody Server Misc Topics
Miscellaneous Server Topics
Actual Miscellaneous Server Topics Homepage
Please see Ajcody-Server-Misc-Topics
Mailbox Purge - Trash And Spam Lifetime And More
Other references to this topic:
- http://wiki.zimbra.com/wiki/Mailbox_Purge
- See the following about commands to manage the dumpster, pre 807 admin guide used bad examples.
- ZCS 8.0.2 Admin Guide - Email Retention Management
Variables Around Lifetime And Purge
Some values will only exist in some version of ZCS. You can see what attributes you have on your version by checking /opt/zimbra/conf/attrs/zimbra-attrs.xml .
Where To Modify In Admin Console or ZWC
- Admin Console
- To configure global retention or deletion policies, go to the Configure>Global Settings>Retention Policy page.
- To configure retention or deletion policies by COS, go to the Configure>Class of Service><COS>, Retention Policy page.
- Make sure Enable COS-level policies instead of inheriting from the policy defined in Global Settings is enabled.
- To configure retention or deletion policies by COS go to the Users properties section in the Admin console.
- User Edits In ZWC
- Right click on folder/object and click on Properties.
- View the Retention tab.
- Dumpster Specific
- Admin functions
- To enable this feature, go to the Configure>Class of service>[COSname], Features page, General Features section. Check Dumpster folder.
- To set Visibility lifetime in dumpster for end user, go to the COS’s, Advanced page, Timeout Policy section.
- To set Retention lifetime in dumpster before purging, go to the COS’s Advanced page, Email Retention Policy section.
- User Interaction
- Right click on Trash folder and click on Recover Deleted Items.
- Admin functions
How Purges Function Vs Time Variables - Admin Variables
- zimbraMailMessageLifetime
- Lifetime of a mail message regardless of location. Number of days a message can remain in a folder before it is purged. This includes data in RSS folders.
- Account / COS variable
- The minimum configuration for email message lifetime is 30 days.
- The default COS value is 0; email messages are not deleted.
- zimbraMailPurgeSleepInterval
- Purge thread activity is throttled by the zimbraMailPurgeSleepInterval server attribute. This value specifies the amount of time that the purge thread sleeps between subsequent purges.
- Global / Server variable
- If the interval is set to 0, purging is turned off.
- The default value is "1m" (one minute).
- zimbraMailDumpsterLifetime
- Retention period of messages in the dumpster.
- Account / COS variable
- 0 means that all messages will be retained.
- The default COS value is 30 days [30d].
- Important - "zimbraMailDumpsterLifetime", specifies how long the messages will be kept in the dumpster. However, the deletion of the messages in the dumpster depends on whether the "zimbraDumpsterPurgeEnabled" is set to TRUE or FALSE. Ref Bug 88170#c6
- Also, currently [8.0.6], there is no variable to control zimbraDumpsterPurge it uses change_date , from the RFE https://bugzilla.zimbra.com/show_bug.cgi?id=45284#c17 - "Dumpster data is purged with a configurable threshold, default 30 days. All items put into dumpster more than 30 days ago are purged. Note the threshold is for deletion time, not create or modified time." and "As mail_item rows are copied to dumpster, the mail_item_dumpster.change_date column is set to the deletion time. This is used by dumpster purge logic. It's okay to lose the last modified time info (which is the original meaning of change_date) because that information is not used once an items is put into dumpster."
- zimbraMailPurgeUseChangeDate[Trash|Spam]
- Note, you should confirm these variables on your server - the default for the variables below most likely are TRUE, meaning it purges by change_date vs. the date [creation date].
- zimbraMailPurgeUseChangeDateForTrash
- zimbraMailPurgeUseChangeDateForSpam
Basics On Editing Variables From CLI
From CLI as the zimbra user - [ su - zimbra ] .
- User account
- Get current variable
- zmprov ga user@domain variable
- Set to new value
- zmprov ma user@domain variable value
- Get current variable
- Cos
- Get current variable
- zmprov gc COSname variable
- Set to new value
- zmprov mc COSname variable value
- Get current variable
- Server
- Get current variable
- zmprov gs Servername variable
- Set to new value
- zmprov ms Servername variable value
- Get current variable
- Global
- Get current variable
- zmprov gacf variable
- Set to new value
- zmprov mcf variable value
- Get current variable
User Level Variables They Can Modify
The retention policy is specified by two sets of attributes on an account. The user retention policy set is set by the user and stored as duration attributes on the account:
- zimbraPrefInboxReadLifetime
- Retention period of read messages in the Inbox folder.
- Account / COS variable
- 0 means that all messages will be retained.
- Default COS value is 0.
- zimbraPrefInboxUnreadLifetime
- Retention period of unread messages in the Inbox folder.
- Account / COS variable
- 0 means that all messages will be retained.
- Default COS value is 0.
- zimbraPrefSentLifetime
- Retention period of messages in the Sent folder.
- Account / COS variable
- 0 means that all messages will be retained.
- Default COS value is 0.
- zimbraPrefJunkLifetime
- Retention period of messages in the Junk folder.
- This user-modifiable attribute works in conjunction with zimbraMailSpamLifetime, which is admin-modifiable. The shorter duration is used.
- Account / COS variable
- 0 means that all messages will be retained.
- Default COS value is 0.
- zimbraPrefTrashLifetime
- Retention period of messages in the Trash folder.
- This user-modifiable attribute works in conjunction with zimbraMailTrashLifetime, which is admin-modifiable. The shorter duration is used.
- Account / COS variable
- 0 means that all messages will be retained.
- Default COS value is 0.
Admin Level Variables That Users Can't Modify
The system retention policy is set by the administrator. These attributes are also at the account/COS level, but not modifiable by the end user:
- zimbraMailTrashLifetime
- Retention period of messages in the Trash folder. Number of days a message remains in the Trash folder before it is purged.
- Account / COS variable
- This admin-modifiable attribute works in conjunction with zimbraPrefTrashLifetime, which is user-modifiable. The shorter duration is used.
- 0 means that all messages will be retained.
- The default COS value is 30 days [30d].
- Note - This actually relates to ALL Trash folders and all Items in those trash folders. This is NOT restricted to just the Mail Trash Folder or to email messages in a trash folder.
- "Clarity on zimbraMailTrashLifetime"
- zimbraMailSpamLifetime
- Retention period of messages in the Junk folder. Number of days a message can remain in the Junk folder before it is purged.
- Account / COS variable
- This admin-modifiable attribute works in conjunction with zimbraPrefJunkLifetime, which is user-modifiable. The shorter duration is used.
- 0 means that all messages will be retained.
- The default COS value is 30 days [30d].
- zimbraShareLifetime
- Maximum allowed lifetime of shares to internal users or groups.
- Account / COS variable
- 0 indicates that there's no limit on an internal share's lifetime.
- The default COS value is 0.
- zimbraExternalShareLifetime
- Maximum allowed lifetime of shares to external users.
- Account / COS variable
- 0 indicates that there's no limit on an external share's lifetime.
- The default COS value is 0.
- zimbraPublicShareLifetime
- Maximum allowed lifetime of public shares.
- Account / COS variable
- 0 indicates that there's no limit on a public share's lifetime.
- The default COS value is 0.
- zimbraFileShareLifetime
- Maximum allowed lifetime of file shares to internal users or groups.
- Account / COS variable
- 0 indicates that there's no limit on an internal file share's lifetime.
- The default COS value is 0.
- zimbraFileExternalShareLifetime
- Maximum allowed lifetime of file shares to external users.
- Account / COS variable
- 0 indicates that there's no limit on an external file share's lifetime.
- The default COS value is 90 days [90d].
- zimbraFilePublicShareLifetime
- Maximum allowed lifetime of public file shares.
- Account / COS variable
- 0 indicates that there's no limit on a public file share's lifetime.
- The default COS value is 0.
- zimbraFileLifetime
- Period of inactivity after which a file gets deleted
- Account / COS variable
- zimbraFileVersionLifetime
- How long a file version is kept around
- Account / COS variable
Dumpster Specific
- How it Works
With Dumpster you can keep a copy of all the deleted files (Emails, Contacts, Calendar or apointments, Tasks, Files, etc.) from a period of time. This feature is really useful to recover specific files deleted by mistake, and makes the life of the System Administrator, and also of the users, much easier.
This is how the Database Looks like, please pay attention into the dumpser tables:
Dumspter keep a copy of all the deleted items. All the deleted items by default goes into the Trash, and dumpster keep a copy into the Database for the specified time.
During the retention time, users are able to recover any deleted item, even if they emptied it. The only thing that Users need to do is Right click on the Trash icon and Recover Deleted Items.
- zimbraDumpsterEnabled
- Enable or disable Dumpster.
- Account / COS variable
- The default COS value is FALSE.
- zimbraDumpsterPurgeEnabled
- Disables purging from dumpster when set to FALSE.
- Account / COS variable
- The default COS value is TRUE.
- Created the following RFE also:
- Better Clarity on zimbraDumpsterPurgeEnabled
- https://bugzilla.zimbra.com/show_bug.cgi?id=88170
- In regards to :
- a) zimbraMailDumpsterLifetime
- b) zimbraDumpsterPurgeEnabled
- The first one, "zimbraMailDumpsterLifetime", specifies how long the messages will be kept in the dumpster. However, the deletion of the messages in the dumpster depends on whether the "zimbraDumpsterPurgeEnabled" is set to TRUE or FALSE.
- The second one [zimbraDumpsterPurgeEnabled], basically enables the purge of messages until a period of time, set by the "zimbraMailDumpsterLifetime" parameter.
- In other words, the messages in the dumpster will be purged, only if the zimbraDumpsterPurgeEnabled
- Better Clarity on zimbraDumpsterPurgeEnabled
- zimbraMailDumpsterLifetime
- Retention period of messages in the dumpster.
- Account / COS variable
- 0 means that all messages will be retained.
- The default COS value is 30 days [30d].
- Important - "zimbraMailDumpsterLifetime", specifies how long the messages will be kept in the dumpster. However, the deletion of the messages in the dumpster depends on whether the "zimbraDumpsterPurgeEnabled" is set to TRUE or FALSE. Ref Bug 88170#c6
- Also, currently [8.0.6], there is no variable to control zimbraDumpsterPurge it uses change_date , from the RFE https://bugzilla.zimbra.com/show_bug.cgi?id=45284#c17 - "Dumpster data is purged with a configurable threshold, default 30 days. All items put into dumpster more than 30 days ago are purged. Note the threshold is for deletion time, not create or modified time." and "As mail_item rows are copied to dumpster, the mail_item_dumpster.change_date column is set to the deletion time. This is used by dumpster purge logic. It's okay to lose the last modified time info (which is the original meaning of change_date) because that information is not used once an items is put into dumpster."
- zimbraDumpsterUserVisibleAge
- Limits how much of a dumpster data is viewable by the end user, based on the age since being put in dumpster.
- Account / COS variable
- 0 means that all messages will be retained.
- The default COS value is 30 days [30d].
Dumpster Specific Commands
- See the following about commands to manage the dumpster, pre 807 admin guide used bad examples.
Searching Dumpster
- To search for an item in the dumpster folder, use the format below. The search field can be a date range: 'before:mm/dd/yyyy and after:mm/dd/yyyy' or emails from or to a particular person: 'from: Joe', etc. The -l # is the max. number of search returns, 25 is default.
Format: zmmailbox -z -m user@example.com s --dumpster -l <#> --types <message,contact,document> <search-field> Example - Larger than 1kb in size, display a max. of 100 search results. zmmailbox -z -m admin@`zmhostname` s --dumpster -l 100 --types message larger:1kb Example - Older than 30 days because we have zimbraMailDumpsterLifetime set to 30, we want to confirm if they are getting purged. Option to only display a max. of 100 search results. zmmailbox -z -m admin@`zmhostname` s --dumpster -l 100 --types message before:-30days Example - using a mysql query against the change_date vs date, 30 days from Apr. 14th 2014 $ date +%s -d "03/14/2014 00:00:00" 1394780400 $ date -d @1394780400 Fri Mar 14 00:00:00 PDT 2014 $ mysql -e 'SELECT * FROM mboxgroup1.mail_item WHERE mailbox_id=1 AND change_date<=1394780400\G' - note, drop the \G if you just want the rows per line.
- To see the dumpster contents via Mysql:
mysql -e "use mboxgroup5; select * from mail_item_dumpster where mailbox_id = 5\G;"
Delete Items From Dumpster
- To delete items in the dumpster folder, type
zmmailbox -z -m user@example.com -A dumpsterDeleteItem <item-ids>
Delete All Items From Dumpster - User
- Deletes all items in the dumpster for the user:
zmmailbox -z -m user@example.com -A emptyDumpster
Recover Items From Dumpster - CLI - Only Way For Non-Message Items
- To recovery an item from the dumpster folder [by item-id]. Please note this from the RFE - https://bugzilla.zimbra.com/show_bug.cgi?id=45284#c17 : "Only messages, contacts, documents, appointment, tasks and chats can be put in dumpster. Other item types like folders, tags and conversations are deleted immediately. Folder contents are put into dumpster when folder is deleted. UI support is limited to message type only at this time. zmmailbox CLI can be used to work with other item types." Can be ri or recoverItem in string below :
zmmailbox -z -m user@example.com ri {item-ids} {dest-folder-path}
Checking A Users Dumpster Total Size
$ su - zimbra # Confirm your on the right zimbraMailHost server for the user $ zmprov -l ga userA@domain zimbraMailHost zimbraMailHost: zcs806.us.DOMAIN.com $ zmprov gmi userA@domain mailboxId: 30019 quotaUsed: 169831 $ zmlocalconfig zimbra_mailbox_groups zimbra_mailbox_groups = 100 # Get the mboxgroup number for the user $ expr 3 % 100 19 $ mysql -N -e "use mboxgroup19; select sum(size) from mail_item_dumpster where mailbox_id = 30019;" +---------+ | 7405838 | +---------+ # To get the total of the 'dumpster' ONLY from zmvolume 1 [default is primary message volume]. Locator is a new db phrase, ZCS 5 bug 30550 - "Renamed "volume ID" to "locator" and made it a String" and you'll also have "mail_item.volume_id and mail_item_dumpster.volume_id" now. $ mysql -N -e "use mboxgroup19; select sum(size) from mail_item_dumpster where locator = 1 and mailbox_id = 30019;" +---------+ | 1523425 | +---------+
Dumpster Related Bug-RFEs
- "cannot modify immutable object error during reindex'
User UI Elements To Manage Message Retention And Message Disposal - If User Is Enabled For It
If they right click on their mail folders and do "Edit Properties" , they should see in the pop up window a tab labeled "Retention" . Within that tab, they'll see:
[ ] Enabled Message Retention : Messages in this folder which fall within the retention range will require explicit confirmation before being deleted. Retention range: [ Custom ] [ ] [ years] [ ] Enabled Message Disposal : Messages in this folder which are older than the disposal threshold will be subject to automated cleanup and deletion. Retention range: [ Custom ] [ ] [ years]
For user recovery, right click on the Trash folder in ZWC and select "Recover Deleted Items"
Setting A Default Domain For User Login
In the admin web console goto:
Configuration > Global Settings
On the General Information tab to the right you'll see a variable called "Default domain". This will be the assumed domain for user logins as well [as well as other functionality].
If you have multiple domains and want to allow users avoid having to type in their domainname with their login credentials you should look at Virtual Hosting.
For Multi-Domains, please see:
Virtual [vhost] Hosting
References
- Multiple_SSL_Virtual_Hosts_5.0
- "Configuring for Virtual Hosting"
- I think the below link is probably superseded now by the other options explored below. When we are talking about vhosting multiple domains on Zimbra and allowing users to login with only their username rather than user@domain.
Non-Proxy [NGINX] Virtual Hosting
- You can configure multiple virtual hostnames to host more than one domain name on a server. When you create a virtual host, users can log in without have to specify the domain name as part of their user name. Virtual hosts are configured from the administration console Domains>Virtual Hosts tab. The virtual host requires a valid DNS configuration with an A record. When users log in, they enter the virtual host name in the browser. For example, https://mail.example.com. When the Zimbra logon screen displays, users enter only their user name and password. The authentication request searches for a domain with that virtual host name. When the virtual host is found, the authentication is completed against that domain.
- Example, you have a single ZCS setup and your going to host multiple domains. [This is only true with a single ZCS mailbox setup. ] You want your users of those domains to use the "domainname" throughout all client configurations and interactions with the mailserver. So the pop/imap/http url's use the mail domain they are in, rather than the physical hostname+domainname of the ZCS server.
- You would configure in the /etc/hosts and/or DNS to resolve the domainname targets that the user will be using to point to the physical ip address of the ZCS server.
- You would then in the admin web console of ZCS, setup the domains under the "Domain Virtual Hosts" section. And confirm the "default domain login" is properly setup for the users/domain users.
- Example, you have a single ZCS setup and your going to host multiple domains. [This is only true with a single ZCS mailbox setup. ] You want your users of those domains to use the "domainname" throughout all client configurations and interactions with the mailserver. So the pop/imap/http url's use the mail domain they are in, rather than the physical hostname+domainname of the ZCS server.
Proxy [NGINX] Virtual Hosting
General Description:
- You can configure multiple virtual hostnames to host more than one domain name on a server. When you create a virtual host, users can log in without have to specify the domain name as part of their user name. Virtual hosts are configured from the administration console Domains>Virtual Hosts tab. The virtual host requires a valid DNS configuration with an A record. When users log in, they enter the virtual host name in the browser. For example, https://mail.example.com. When the Zimbra logon screen displays, users enter only their user name and password. The authentication request searches for a domain with that virtual host name. When the virtual host is found, the authentication is completed against that domain.
- Exception. It would be possible to do the functions of Virtual Hosting without installing the zimbra proxy package. Using jetty directly for example. Our recommendation is to use the zimbra proxy package to achieve this though.
- Example, you have a single ZCS setup and your going to host multiple domains. [This is only true with a single ZCS mailbox setup. ] You want your users of those domains to use the "domainname" throughout all client configurations and interactions with the mailserver. So the pop/imap/http url's use the mail domain they are in, rather than the physical hostname+domainname of the ZCS server.
- You would configure in the /etc/hosts and/or DNS to resolve the domainname targets that the user will be using to point to the physical ip address of the ZCS server.
- You would then in the admin web console of ZCS, setup the domains under the "Domain Virtual Hosts" section. And confirm the "default domain login" is properly setup for the users/domain users.
- Example, you have a single ZCS setup and your going to host multiple domains. [This is only true with a single ZCS mailbox setup. ] You want your users of those domains to use the "domainname" throughout all client configurations and interactions with the mailserver. So the pop/imap/http url's use the mail domain they are in, rather than the physical hostname+domainname of the ZCS server.
- Exception. It would be possible to do the functions of Virtual Hosting without installing the zimbra proxy package. Using jetty directly for example. Our recommendation is to use the zimbra proxy package to achieve this though.
References:
Domain And User "Status" Changes
If you need to manually modify the status of a domain, for example - migration.
Checking Status - Mode Of Domain
If the domain is in "active" mode, the CLI output will show this.
zmprov gd domainname zimbraDomainStatus zimbraDomainStatus: active
The variables are all lowercase by the way.
Setting A Domain To Maintenance Status - Modes
To set a domain to "maintenance" mode, you would do the following.
zmprov md domainname zimbraDomainStatus maintenance
Setting A Domain To Active Status - Modes
To set a domain to "maintenance" mode, you would do the following.
zmprov md domainname zimbraDomainStatus active
Descriptions Of Status - Modes
Here's the descriptions for each of the "status" or "modes":
- Active. Active is the normal status for a mailbox account. Mail is delivered and users can log into the client interface.
- Maintenance. When a mailbox status is set to maintenance, login is disabled, and mail addressed to the account is queued at the MTA. An account can be set to maintenance mode for backing up, importing or restoring the mailbox.
- Locked. When a mailbox status is locked, the user cannot log in, but mail is still delivered to the account. The locked status can be set, if you suspect that a mail account has been hacked or is being used in an unauthorized manner.
- Closed. When a mailbox status is closed, the login is disabled, and messages are bounced. This status is used to soft-delete an account before deleting the account from the server. A closed account does not change the account license.
- LockOut. Users who try to log in and do not enter their correct password are locked out of their account after a specified number of consecutive failed login attempts. An account’s status is automatically changed to Lockout. How long the account is locked out is set by COS or Account configuration, but you can change the lockout status at any time.
Cross Mailbox Searches and Tracing
The crossmailbox search tool from the Admin web console requires the Archiving & Discovery package to be installed. The zmmboxsearch (CLI) doesn't require this. For A&D specific searches, there's also zmarchivesearch .
Notable RFE/Bugs
- "Need ability to restrict zmmboxsearch to specific domains"
Searches And Timeout Variables - ZCS 6+
httpclient_soaphttptransport_so_timeout is set by default to 60 - in seconds. If you are using the -m "*" variable, to search for everyone, you might need to set this to 0 [no timeout] on the mailstore your doing the search from.
zmlocalconfig -e httpclient_soaphttptransport_so_timeout=0
Cross Mailbox Search
Events/errors will show in /opt/zimbra/log/mailbox.log
Search From the CLI
From the CLI, see zmmboxsearch help page
Here's an example use of the zmmboxsearch command:
su - zimbra mkdir /tmp/testing zmmboxsearch -m user1@mail3.internal.homeunix.com,user2@mail3.internal.homeunix.com -q "in:inbox" -d /tmp/testing/
Try using the formats I've written about here - Ajcody-Notes#Web_Client_Search - for the query string.
Search From the CLI - A&D Accounts - Even If Primary Account Was Deleted
Here's an example use of the zmmboxsearch command when you have all your archive accounts using the domain name of zimbra.homeunix.com.archive:
su - zimbra mkdir /tmp/testing zmmboxsearch -m `zmprov gaa | grep zimbra.homeunix.com.archive` -q "in:inbox" -d /tmp/testing/
Single A&D account search is this simple, for example. Primary account was user1@mail3.internal.homeunix.com with archive account being user1-20081211@mail3.internal.homeunix.com.archive. This will work even if the primary account, user1@mail3.internal.homeunix.com, was deleted.
su - zimbra mkdir /tmp/testing zmmboxsearch -m user1-20081211@mail3.internal.homeunix.com.archive -q "in:inbox" -d /tmp/testing/
If this isn't working, make sure the archive accounts still exist. Example uses my "archive" domain:
zmprov gaa | grep mail3.internal.homeunix.com.archive
Searches From Admin Web Console - Deleted Primary Account But A&D Account Still Exists
Please see the following:
- http://bugzilla.zimbra.com/show_bug.cgi?id=37921
- "archives should be searchable independent of their original mailboxes"
Message Tracing
Please see the command help page for zmmsgtrace . This was pulled for ZCS 6, there is an existing RFE for a replacement or inclusion again:
- "zmmsgtrace replacement"
Searches Limited To 500 or 1000 Maximum Results
Update Bugs filed against ZCS 8.6 . Ajc - Jan 22, 2015
- "zmarchivesearch should accept -m "*" or multiple mailbox targets like zmmboxsearch does"
- "admin console mailbox search doesn't autocomplete/suggest for target mailbox account for search results"
- "crossmailbox / mailbox search in admin console limits still broken and documentation descriptions are poor"
- "zmmboxsearch zmarchivesearch -p switch does accept the * [unlimited] variable. Document and qa for each release."
- "zmmboxsearch zmarchivesearch -l does have a hard limit still - it's 1000"
- "zmmboxsearch and zmarchivesearch -p and -l options makes no sense, they need to be rewritten"
Older Bugs:
- index OOME caused by cross- mailbox search
- Support large number of results in cross mailbox search
- zmmboxsearch does not show result more than 1000
There is a hard limit of returns being restricted to 500, per bug 43265 & 58901 [fixed for ZCS8]. There is also a limit in ZCS 8, bug 82757, targeted for the JudasPriest release. As a work around, download the zmmboxsearchx script located on bug 43265, it's listed as an attachment: https://bugzilla.zimbra.com/show_bug.cgi?id=43265 This requires the following installed via CPAN as root and accept the default options suggested (http://www.cpan.org/modules/INSTALL.html): Parallel::ForkManager SOAP::Lite e.g.: cpan Parallel::ForkManager cpan SOAP::Lite Usage of the zmmboxsearchx script is similar to zmmboxsearch, but with some differences. zimbra@zcs7-ga:/tmp/zmmboxsearchx-20100625/bin$ ./zmmboxsearchx zmmboxsearchx: --query is required zmmboxsearchx: use of --account, --searchdirectory or <userlistfile> is required zmmboxsearchx: try --help option for more information Usage: zmmboxsearchx [options] [<userlistfile> ...] Options: [*] == Required, [+] == Multiple vals OK, (val) == Default --query <query_string> [*]query string --dir <directory> directory to write messages to (no directory) --account <acct> [+]account to include, comma separated values ok --exclude <acct> [+]account to exclude, comma separated values ok --exclude-file <file> [+]file listing accounts to exclude --searchdirectory <filt> ldap filter to search for accounts --server <hostname> mailbox server (LC:zimbra_zmprov_default_soap_server) --limit <num> limit the total number of results returned (25) --offset <num> offset in hit list to start at (0) --proc <num> maximum number of child/worker processes to use (4) --url <url> URL of soap service (LC:...) --authuser <user> account for authentication (LC:zimbra_ldap_user) --password <password> password for authuser (LC:zimbra_ldap_password) --help display a brief help message --man display the entire man page --debug [<num>] verbose output useful for debugging --verbose increase verbosity (increments --debug) Note: LC:<key> means the default is read from localconfig if possible. Start by verifying its functionality with just a small subset of the accounts on the system. Once you confirm it's operation, try broadening the search to a large listing of accounts or the * variable. An example: ./zmmboxsearchx --proc 4 --query "whatever" --account "*" --d outputDir_path Should it still not work, please try enabling the --debug and --verbose, and send us back some attachments containing the output. :About getting the search results back into ZCS so someone could view them from ZWC: zmmboxsearchx does use a local directory on the system. You could then use the following to inject those msg's back into a ZCS account for your compliance officers to view. To create a folder into the account you want the results to be displayed see the below example to create a folder, using the cf option. zmmailbox -z -m destination_account@domain.com cf -V message /SearchResults_XXX You would then cd to the directory you had the search dump the messages in and then do something like : for i in * ; do zmmailbox -z -m destination_account@domain.com addMessage /SearchResults_XXX $i ; done You should double check against an actual search if the * will work, I don't recall offhand it uses subdirectories and if all of the search results will end up with *.msg's. destination_account@domain.com should be replaced with the email account the compliance officer will want to use to view the injected messages. It could be their account or a new one that you then share to them. /SearchResults_XXX would be replaced with the directory you want the messages to be stored in.
Deferred Emails - Error Of Connection Refused
Your having emails deferred and you check the /opt/log/zimbra.log file and see something like:
Sep 2 05:17:56 mail postfix/qmgr[12229]: 56A793151483: to=<USER@DOMAIN.net>, relay=none, delay=29404, delays=29404/0.08/0/0, dsn=4.4.1, status=deferred (delivery temporarily suspended: connect to 127.0.0.1[127.0.0.1]: Connection refused)
Things to do to help trouble shoot the issue:
- Please confirm that iptables and selinux aren't running or are configured properly.
service iptables status
sestatus
- Can you telnet to the different ports for example? port 25?
- What's your /etc/hosts look like and is the format correct - confirm localhost entry is right?
cat /etc/hosts
- Check for hostname issues:
hostname YOUR_FQDN
hostname -t MX YOUR_DOMAINNAME
cat /etc/resolv.conf
- Give the output of the following as well - replace YOUR_SERVER_NAME:
zmprov gs YOUR_SERVER_NAME | grep -i mta
- Have you tried to disable DNS lookups? [zimbraMtaDnsLookupsEnabled]
zmprov ms YOUR_SERVER_NAME zimbraMtaDnsLookupsEnabled FALSE
- What's the current status of zimbra services?
zmcontrol status
- If you can restart zimbra - while starting zimbra get logging events that might be related to issue or other errors/warnings:
zmcontrol stop
tail -f /opt/zimbra/log/mailbox.log & /opt/zimbra/log/zmmailboxd.out & /var/log/zimbra.log
zmcontrol start
- Have you tried to disable antivirus and antispam to see if queue flushes?
- Amavis
zmamavisdctl status
zmamavisdctl stop
zmamavisdctl start
- Clam
zmclamdctl stop
zmclamdctl start
- Amavis
- Give the following output - as root:
netstat -lntp
lsof | grep TCP
- Checking postfix for deferred and command to flush:
postqueue -p
postqueue -f
What's my time and timezone?
I wrote up the steps to make sure the ZCS server is using the correct time and timezone here:
http://wiki.zimbra.com/index.php?title=Time_Zones_in_ZCS#The_server_OS
Volumes & zmvolume
Basic Concepts
- From Mike's forum post. Modified a little.
Each Zimbra mailbox server is configured with one index & message volume (NE can add secondary HSM volumes).
A volume will state it's condition as "current" as either true or false.
When a new message is delivered or created, the message is saved in the current message volume. Additional message volumes can be created, but only one is configured as the current volume where new messages are stored. If the volume is in danger of becoming full, you can configure a new current message volume. The new current message volume would then receive all the new messages. New messages are never stored in the previous volume once it's current value is set to false. A current volume cannot be deleted - via zmvolume. If there are accounts with messages referencing the older volume it should not be deleted until you point the volume path to wherever you've now moved the items.
Each mailbox is assigned to a permanent index directory on the current index volume. When an account is created, the current index volume is automatically defined for the account. You cannot change which index volume the account is assigned. As volumes become full, you can create a new current index volume for new accounts. When a new index volume is added as current, the older index volume is no longer assigned new accounts. Index volumes not marked current are still actively in use as the index volumes for accounts assigned to them - unlike the message volumes. Any index volume that is referenced by a mailbox as it's index volume cannot be deleted.
The Network Edition has the HSM (Hierarchical Storage Management) feature which involves moving data to other storage locations automatically after x amount of time. Messages and attachments are moved from a primary volume to the current secondary volume based on the age of the message; completely transparent to the user.
(Also a handy tidbit of info: collapsing message volumes - first tip is good for HSM > back to store, Klug's also points out that you can essentially use the same concept of Adam's wiki page [see below]; making it a subdirectory of the new filesystem location keeps it simple.)
The admin console > server > volumes tab is straight forward, make sure you have permission to write to the target location - see also: CLI_zmvolume (-l and -dc arguments display your volumes)
If you're curious how to view that in the DB:
su - zimbra mysql select * from zimbra.volume;
Notable RFEs
- "multiple "current" primary, secondary, index zmvolumes"
- "Ability to segregate Briefcase storage from regular mail storage"
How To Move A User's Data To Another Volume
Notable RFE's to make moving user data to another volume easier:
- "zmprov ga and admin console should show volume path/s for user's blobs"
- "zmmboxmove option to move user volume data to different volume"
Using zmsoap Example
See the following:
How To Go About Changing Volume Paths
List your current volume details
zmvolume -l
Make your directories to the "new" volume path. For example:
mkdir /san/mount/index /san/mount/store chown zimbra:zimbra /san/mount/index /san/mount/store
Shutdown zimbra so we can move the data.
zmcontrol stop
Now move all the old data and then make symbolic links from old to new.
mv /opt/zimbra/index/* /san/mount/index/ mv /opt/zimbra/store/* /san/mount/store/ rmdir /opt/zimbra/store /opt/zimbra/index ln -s /san/mount/index /opt/zimbra/index ln -s /san/mount/store /opt/zimbra/store
Start zimbra back up.
zmcontrol start
- Goto the admin web console , Configuration > Servers > server-name > Volumes Tab.
- Select index volume
- Click edit
- Modify the path to the new path
- Click edit
- Select store volume
- Click edit
- Modify the path to the new path
- Click edit
- Select index volume
- Click on Save in the upper left hand section.
To Modify Volume From CLI After Data Move
See what the details are of your volumes:
zmvolume -l /opt/zimbra/index and zmvolume -l /opt/zimbra/store Volume id: 2 name: index1 type: index path: /opt/zimbra/index compressed: false current: true Volume id: 1 name: message1 type: primaryMessage path: /opt/zimbra/store compressed: false current: true
Let's say your sym links are the ones shown above:
/opt/zimbra/index /opt/zimbra/store
And your new directory paths are:
/mnt/nas/index /mnt/nas/store
You would run the following to modify the volumes to use the real directory paths rather than the sym links.
zmvolume -e -id 2 -p /mnt/nas/index zmvolume -e -id 1 -p /mnt/nas/store
Convertd
Version 1 Performance Issues - Upgrade To Version 2
From release notes:
An alternative implementation of the convertd daemon used for text extraction and conversion to HTML is now available as beta in 5.0.7. The new implementation is Apache multi-process based, as opposed to the original implementation which was multi-threaded. When libraries used to handle attachments fail or crash, the multi-process implementation allows for better availability. Since the new implementation is still in beta, the older multi- threaded implementation continues to be the default. If you are experiencing a high number of 400 (try again) error codes during LMTP delivery, or if you are seeing too many kvoop processes consuming memory, you can try the new Apache-based convertd.
- See Bug 2161 for more details.
Note - ALL mailstores must be running the same version of convertd. Multi-mailstore environments must plan upgrades/switch simultaneously.
- You must install the new zimbra-convertd package for it to be available.
- This is that reference during the installer about convertd. The basic convertd (v1) package is apart of zimbra-core.
- kvoop is when your running version 1 of convertd.
ps auxwww | grep -i kvoop
- Run the script you want. For 5.0.7+ , this script will now shutdown and restart necessary service components.
- Run /opt/zimbra/convertd/bin/upgrade_v2 to upgrade.
- Run /opt/zimbra/convertd/bin/downgrade_v1 to downgrade.
- Pre 5.0.7, you'll need to manually stop / start service components.
- Restart zimbra or one might be able to get away with just restarting convertd and mailstore.
zmcontrol stop
zmcontrol start
- Restart zimbra or one might be able to get away with just restarting convertd and mailstore.
High CPU Usage By Convertd - kvraster
Example situation might described as:
- We are running Zimbra with 3 mailstore servers, and all 3 mailstore servers installed with convertd. But i am seeing one of mailstore using most of the resource for convertd (Java). I found 2 of the process running and using 79% and 67% of CPU (2 CPU) on the server and overall CPU usage of the server is 93%. Do we can fix this, or do i need to add more processors for this server?
- Here the info i get from top command.
- 16265 zimbra 18 0 3355m 21m 8552 S 79 0.2 10176:49 java -Djava.awt.headless=true -classpath /opt/zimbra/keyview/ExportSDK/bin kvraster /opt/zimbra/convertd/convert/11/i
- 15995 zimbra 18 0 3355m 21m 8552 S 67 0.2 10171:12 java -Djava.awt.headless=true -classpath /opt/zimbra/keyview/ExportSDK/bin kvraster /opt/zimbra/convertd/convert/11/i
Developer response:
- Autonomy uses java for some image conversion which can be resource intensive. Convertd has a processing timeout but it currently just calls _exit() and does not signal any children.
- Steps that can help resolve this
- Just kill these runaway children.
- They are not doing anything useful and are stuck in some Sun code
- Upgrade to ZCS 608+
- It has a newer Autonomy SDK that does resolve some crashes and other problems
- Make sure they are running a current Java release that might fix some bugs in image conversions
- Set ConvertDebugLevel to 3 in convertd's conf file to save files that timeout.
- If reproducible, we can enter bugs with Autonomy
- If the problem keeps recurring, we can build a convertd that attempts to signal children after a processing timeout. I'll look at that for Helix
Domain Rename Issues
We have a command to rename a "domain" and the resources within it - unfortunately it doesn't handle ALL of the different components for various reasons. Specifically Documents and Calendars [more down below].
Review this RFE that was done when they implemented the renameDomain command.
http://bugzilla.zimbra.com/show_bug.cgi?id=7655
Note comment 21 & 22 (confirming what QA'd). The syntax is
zmprov -l rd testold.com testnew.com
Problem With Documents
Problems that arise with Documents are explained here, comment #1 has workaround.
http://bugzilla.zimbra.com/show_bug.cgi?id=25873
Problem With Calendar & Appointments
Now, the very unfortunate part about calendars with a domainname change/move.
- "Ability to change a username globally on all appointments"
- "Cant edit calendar entries after renameDomain"
- "Ability to assign ownership (ie change organizer) of meetings based on search criteria"
The root issue here's seems to be more about the calendar standards and practices with the use of the "Organizer" field and notifications. You'll see 3 "work arounds" in comment 2 of bug 26736.
Export/Importing of the calendar data is shown here:
http://wiki.zimbra.com/index.php?title=User_Migration
This part specifically :
http://wiki.zimbra.com/index.php?title=User_Migration#Copy_Calendar_From_One_Zimbra_User_to_Another
Instant Messaging
- "Renaming domain does not update zimbra.jiveRoster"
ZCS Service Can't Start Because Of Port In Use
Either CLI output or logs will note the port in question. The following commands can help identify what other service/application is causing the problem and conflict with the ZCS service.
This example shows port 995 causing a problem and uses nmap , netstat, and lsof to find details about what is 'running' on port 995. In this example, mailboxd wasn't able to start because of it.
[zimbra@zimbra1 log]$ nmap -P0 -p995 localhost Starting nmap 3.70 ( http://www.insecure.org/nmap/ ) at 2012-10-05 16:46 EDT Interesting ports on localhost.localdomain (127.0.0.1): PORT STATE SERVICE 995/tcp open pop3s [root@zimbra1 log]# netstat -tupl | grep pop tcp 0 0 *:pop3s *:* LISTEN 9293/rpc.statd [root@zimbra1 log]# lsof -i :995 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME rpc.statd 9293 rpcuser 8u IPv4 21045 TCP *:pop3s (LISTEN) ### search by port with netstat , this is the way it should look when rpc didn't steal the port ### [root@zimbra1 log]# netstat -plnt | grep 995 tcp 0 0 :::995 :::* LISTEN 16520/java
Changing Default Port 80 - No Proxy In Use And Single Server Example
Review the zmtlsctl wiki page to determine the http/https use.
* CLI_zmtlsctl_to_set_Web_Server_Mode
This example changes 80 to 60081 and 443 to 60443.
Confirm current port use/access and that no iptables/firewall are in use:
**as root** lsof -P -n -i :80 -i :443 iptables -L **Policy As Accepted & None Blocked** cat /etc/selinux/config **SELINUX=disable**
Make sure your not running the zimbra proxy service:
**as zimbra** zmprov gs `zmhostname` | grep ServiceEnabled
Resolve any issues above before continuing.
Now, make any changes to the Web Server Mode you need in regards to http and https.
For example, I'll change mine to "mixed"
**as zimbra** zmtlsctl mixed
Now let's change the ports in use [zimbraMtaAuthHost is a valid mailbox server hostname if your a multi-server ZCS environment]:
zmprov ms mail.yourdomain.com zimbraMailPort 60081 zmprov ms mail.yourdomain.com zimbraMailSSLPort 60443 zmprov ms mail.yourdomain.com zimbraMtaAuthHost mail.yourdomain.com zmcontrol stop zmcontrol start
And now confirm the changes:
**as root** lsof -P -n -i :60081 -i :60443
Themes, Branding, Logos, And Other UI Customizations
Main Wiki Page: Skins
Please see:
- For ZCS 5.x+
- For ZCS 4.5.x
Finding Skins Variable From CLI
What skins are available [zimbraInstalledSkin]:
[zimbra@mail3 ~]$ zmprov gacf | grep -i skin zimbraInstalledSkin: bones zimbraInstalledSkin: sky zimbraInstalledSkin: lemongrass zimbraInstalledSkin: lavender zimbraInstalledSkin: yahoo zimbraInstalledSkin: waves zimbraInstalledSkin: bare zimbraInstalledSkin: sand zimbraInstalledSkin: steel zimbraInstalledSkin: hotrod zimbraInstalledSkin: beach
Can Users Change Their Skin? This depends on their COS and the COS Value [zimbraFeatureSkinChangeEnabled]:
[[zimbra@mail3 ~]$ zmprov gac default [zimbra@mail3 ~]$ zmprov gc default | grep -i skin zimbraFeatureSkinChangeEnabled: TRUE [zimbra@mail3 ~]$ zmprov gac default [zimbra@mail3 ~]$ zmprov gc default | grep -i skinchange zimbraFeatureSkinChangeEnabled: TRUE
What is the default skin for the cos [zimbraPrefSkin]:
[zimbra@mail3 ~]$ zmprov gc default | grep -i prefskin zimbraPrefSkin: beach
To Modify the default skin for a cos:
zmprov mc [cos name] zimbraPrefSkin Name_Of_Skin
Sending Email From The CLI (Command line) On Zimbra Server
Make a txt file:
vi /tmp/email.txt
Put in something like:
To: test2@null.com Subject: Test message From: test@null.com
Body of message goes here
And then using the zimbra/postfix command for this, it would be:
/opt/zimbra/postfix/sbin/sendmail -t < /tmp/email.txt
or this, if that didn't work:
/opt/zimbra/postfix/sbin/sendmail -Am -t < /tmp/email.txt
I see the -Am referenced in the /opt/zimbra/conf/swatchrc.in file, but the man page for sendmail (zimbra/postfix) doesn't give clarity on this option.
Role Based Administration - Expansion Of Permission & Function Model For Admin Roles
Please see the following:
- "server side infrastructure for role based delegated administration"
- "role based delegate administration"
Spammers
Disconnect Web Session Spammer Hijacked
These might be related:
- "Manually Expire Web Sessions"
- http://bugzilla.zimbra.com/show_bug.cgi?id=34054
- probably related
- "Webclient timed logout setting appears to bind at login time"
- probably related
- http://bugzilla.zimbra.com/show_bug.cgi?id=34054
- Not sure about this one...
- "EndSessionRequest does not return"
Looks like they need some Votes/Comments
Mass Importing / Exporting Of Things (Tasks, Notes, Documents, Briefcase, etc.)
Notes
Please see, Ajcody-Client-Topics#Import_.26_Export_Notes_.26_Journal
Documents And Briefcase
Please see, Bulk Upload To Briefcase
Inter-Domain (Company To Company) Item Access [Sharing?] Via Rest
From /opt/zimbra/doc/rest.txt :
2. Inter-domain To access items across domains that span Zimbra installations, use the exact same naming convention. For example, if companyA installs Zimbra, and companyB installs Zimbra, someone at companyA should be able to reference an item from companyB using the same syntax: /home/john.smith@companyB.com/shared/contacts this can be accomplished with DNS SRV records. For example, companyB.com could publish a SRV record for _zimbra._tcp: _zimbra._tcp.companyB.com. SRV 10 5 80 zimbra.companyB.com The Zimbra proxy at companyA can then detect that companyB.com is not a local Zimbra domain, and do a DNS lookup for: _zimbra._tcp.companyB.com And get back "zimbra.companyB.com" as the name of the Zimbra sever to direct the request to. It would then do the equivalent of: http://zimbra.companyB.com/home/john.smith@companyB.com/shared/contacts
Rest Url's And Spaces , White Spaces, %20
From /opt/zimbra/doc/rest.txt :
ISSUES/NOTES 2. user-friendly item "ids"? It would be nice to have user-friendly names for items inside of a folder, such as (+ is url-encoded form of a space): /home/roland/calendar/Staff+Meeting /home/roland/contacts/Roland+Schemers /home/roland/inbox/Important+Meeting Exchange allows this type of naming, need to determine how/if we'd want to do this, and how to deal with collisions like it does. Might need to come up with a per-item query-param string, or use search: /home/roland/contacts/?query="Roland Schemers"
Faster Way To Get Directory Size On Filesytem - find vs du
Note - if your having issues with your disk/partitions getting full, you might also want to consult Ajcody-Backup-Restore-Issues#Basic_Backup_Information_To_Submit_To_Support .
Instead of using:
du -sh .
Try this instead:
find . -printf %k"\n" | awk '{ sum += $1 } END { print sum }'
For example [as root]:
for i in `find /opt/zimbra -maxdepth 1 -type d`; \ do export sum=`find $i -printf %k"\n" | awk '{ sum += $1 } END { print sum kb }'`; \ echo -e "$sum kb\t$i"; export sum=; done | sort -rn | head -n 20 [example output below] 6007764 kb /opt/zimbra 1966620 kb /opt/zimbra/db 837160 kb /opt/zimbra/backup 680932 kb /opt/zimbra/jetty-distribution-7.6.12.v20130726 387140 kb /opt/zimbra/data 286160 kb /opt/zimbra/jdk-1.7.0_45 211080 kb /opt/zimbra/store 207172 kb /opt/zimbra/zmstat 178628 kb /opt/zimbra/logger 162280 kb /opt/zimbra/mta 155700 kb /opt/zimbra/bdb-5.2.36 116520 kb /opt/zimbra/aspell-0.60.6.1 98820 kb /opt/zimbra/mysql-standard-5.5.32-pc-linux-gnu-i686-glibc23 79408 kb /opt/zimbra/zimbramon 72608 kb /opt/zimbra/lib 66940 kb /opt/zimbra/keyview-10.13.0.0 66488 kb /opt/zimbra/clamav-0.97.8 64676 kb /opt/zimbra/httpd-2.4.4 47408 kb /opt/zimbra/store2 47164 kb /opt/zimbra/index
Free Busy, FreeBusy, FB, F/B Topics
References
Current references for server topics for FB are:
- On your zimbra server:
- /opt/zimbra/doc/freebusy-interop.txt
- Free Busy Interop for Exchange 2003, 2007, and Non-Exchange Systems [similiar to above]:
- Setting Up Free Busy Interop with Exchange 2007:
- FB Interoperability setup & configuration - section "Zimbra Free/Busy Interoperability":
- Troubleshooting Exchange FB Interop:
Client Configuration topics for FB are:
- User sharing their FB with someone via email:
- Outlook and FB:
Free Busy Support Requirement Clarifications
Please see the following:
- "ZCS to ZCS free/busy interop"
- http://bugzilla.zimbra.com/show_bug.cgi?id=26398
- [marked as dup of above] "Free Busy documentation & configuration should include Zimbra to Zimbra"
- "Clarity on FB / Free Busy configuations we support"
- Similar, but not identical to above
- "Allow access to external users' internet free-busy, similar to Outlook"
- "ability to subscribe to externally published calendars"
- http://bugzilla.zimbra.com/show_bug.cgi?id=26398
Miscellaneous Bugs And Related RFEs
Please see:
- "Support checking free/busy from an external source for a zimbra account"
- "Interop: don't hard code cn=RECIPIENTS"
- "RFE: Extend Exchange Free/Busy interop to support multiple AD organizations"
- "RFE: Calendar preferences should be updated when folder level setting and takes precedence"
WedDAV Topics
Actual WebDAV Homepage
Please see Ajcody-WebDAV
WebDAV & Davfs2 - Briefcase - Documents - Notebook
At this time, I can not find anything in our documentation that states that Zimbra via WebDAV can replace a file server 100% and that it is a supported and promised feature & function of Zimbra. What is supported, seems to be the narrow action of "reading" objects from WebDAV. But the full functionality to grant complex permissions and shares, write files remotely, and expansive support for the various OS's and WebDAV clients is beyond official support at this time. If you believe differently, please share the source on this wiki page "Discussion" section.
Resources
- WebDAV
- Briefcase
- Lengthy forum post on WebDAV issues:
- In your zimbra installation in the documents directory - /opt/zimbra/docs
- dav.txt
- webdav-mountpoint.txt
Third Party Webdav Clients
Adam Recommends For File Transfers
Anyclient - Java GUI - Free
Anyclient seems to work against Zimbra reliably for file transfers between the workstation and the server. It is like a ftp client though, so it will not seamlessly integrate into your applications or your OS's file browser. That will most likely need to wait until we can resolve the bugs surrounding the native webdav client for the major OS's.
- Anyclient - The Free No-Install FTP, FTPS, SFTP and WebDAV Client (freeware)
- They even have a java applet from their webpage you could use.
- Host field for this would be formatted like this:
- http://HOSTNAME/dav/USERNAME or https://HOSTNAME/dav/USERNAME
- replace HOSTNAME with your zimbra servers hostname or ip address
- replace USERNAME with yours, can also use this format - USERNAME@DOMAIN
- http://HOSTNAME/dav/USERNAME or https://HOSTNAME/dav/USERNAME
Cadaver - CLI - Free
cadaver is a command-line WebDAV client for Unix. It supports file upload, download, on-screen display, namespace operations (move/copy), collection creation and deletion, and locking operations. The file locking operations will not work against Zimbra at this time [6-2010] and I still find I can't remove some of those ._ files that the Mac makes. They usually end up in the Trash on Zimbra and you can remove them via ZWC > Trash > Empty Trash .
References:
- http://www.webdav.org/cadaver/
- Linux clients should be able to get this through their normal packaging methods:
- For example [Ubuntu]: apt-get install cadaver
- Linux clients should be able to get this through their normal packaging methods:
- Windows port:
- Mac port:
Other Clients
Free
- AnyClient - The Free No-Install FTP, FTPS, SFTP and WebDAV Client (freeware)
- They even have a java applet from their webpage you could use.
- BitKinex 3.1.1 for Windows (freeware)
- Cyberduck - FTP, SFTP, WebDAV, Cloud Files & Amazon S3 Browser for Mac OS X. (OSS)
Commercial
- Webdrive - A Universal File Access Client that maps drive letters to FTP, WebDAV, SFTP and S3 Servers ($$)
- Netdrive ($$ but "Free" for home use)
- EngInSite DataFreeway - SSH (SSH1, SSH2, SFTP), FTP WebDAV client for Windows.
- (Listed as Freeware but will prompt about license upon first launch, trial version)
- Xythos Drive for Windows ($$)
General Bugs And RFE's
From Release Notes 6.0.3
- Briefcase Spreadsheet and Presentation functions are beta for 6.0.0
DAV Related
- "server side webdav repository for zimbra users"
- "NPE in DAV on DF"
- http://bugzilla.zimbra.com/show_bug.cgi?id=19984
- dup of above "WebDAV generates an error"
- http://bugzilla.zimbra.com/show_bug.cgi?id=19984
- "AJAX client does not handle notification for Documents in Briefcase"
- "files uploaded with webdav client and with spaces in the name become unusable"
- "davfs2 can't create/copy files via Zimbra WebDav"
- http://bugzilla.zimbra.com/show_bug.cgi?id=24178
- Release Notes description, "Linux users can now access documents in Briefcase as if they are accessing files on a network drive."
- http://bugzilla.zimbra.com/show_bug.cgi?id=24178
- "Webdav URL for a shared briefcase"
- "WebDAV rename turns ' ' into %20 in the filename"
Document Sorting In ZWC
- "Ability to sort items in the briefcase alphabetically, by date, type, size, owner, etc." [Helix target]
- http://bugzilla.zimbra.com/show_bug.cgi?id=25220
- Below will most likely become dup of above
- "Sort documents by date"
- Below will most likely become dup of above
- http://bugzilla.zimbra.com/show_bug.cgi?id=25220
- "Sorting should be available on ALL columns"
Unsorted
- "Change App name from Documents to Wiki or Notebook"
- "Invalid request while attempting to access sub folder of shared briefcase"
- "Briefcase for HTML lite client"
- "Documents for Html client"
- "Rename File in Briefcase"
- "Documents stored in browsing history creates document syncing issues"
- "Documents cannot be closed after closing history"
- "Table of contents not shown after clicking close on document history"
- "History/Edit links are not available half the time"
- "Briefcase - share expiration & share ability to file rather than just folder"
- "preview documents in briefcase "
- "Unable to import files into Documents Notebook"
- "Set a default file extension" [files are html files]
- "Convert (import/export) from popular document formats to edit via Zimbra AJAX editors"
- "Concurrent editing of Briefcase requires locking or notification"
- "Briefcase can show only 250 items - should do virtual paging"
- "Briefcase cleanup"
- "user/cos pref to set default ZWC view of Documents to 'browse' rather than 'page' - webdav issue"
Max File Upload Size
This impacts your webdav operations in regards to the maximum size the files can be.
Check the following variable:
zmprov gacf zimbraFileUploadMaxSize
To edit:
zmprov mcf zimbraFileUploadMaxSize 100000000
Also, with ZCS 6+ :
zmprov gacf zimbraMailContentMaxSize
To edit:
zmprov mcf zimbraMailContentMaxSize 100000000
Apple - Mac
Apple - Mac Bugs & RFE's
- Avoid Using Spaces In Filenames
- "WebDAV rename turns ' ' into %20 in the filename"
- " Hide/delete resource forks created by Mac OS X on WebDAV access to Briefcase "
- http://bugzilla.zimbra.com/show_bug.cgi?id=40924
- dot_clean does not remove these ._* files over a webdav mount.
- See also:
- "NPE when accessing folders via webdavfs (Mac OS X)"
- http://bugzilla.zimbra.com/show_bug.cgi?id=40924
- "Editing and Saving a file via WebDAV (Mac OS X) using Open Office"
- was called: "Editing and Saving a file via WebDAV (Mac OS X)"
- http://bugzilla.zimbra.com/show_bug.cgi?id=41199
- "Edit and Save As a new file name via WebDAV (Mac OS X)"
Webdav Connect Via Mac Finder
On a Mac, I've confirmed this process works for read. It appears it can write files [sometimes] but in the end it will not do it correctly. My test had left a file saved incorrectly and also left a dot [.*] file that could be seen in ZWC. I've added some of my testing notes to bug 40924 .
- Right Mouse Click or Ctrl+Mouse Button on Finder, this will bring up the option panel
- Click on "Connect To Server"
- Server Address Examples:
- http [80] or https [443] can be used
- For USERNAME, can be either username or username@domain.com
- http://SERVERNAME/dav/USERNAME
- This will show all folders
- For example: Briefcase/ Contacts/ Inbox/ Sent/ attachments/ Calendar/ Drafts/ Junk/ Tasks/ Chats/ Emailed Contacts/ Notebook/ Trash/
- This will show all folders
- http://SERVERNAME/dav/USERNAME/Briefcase
- http://SERVERNAME/dav/USERNAME/Contacts
- http://SERVERNAME/dav/USERNAME/Notebook
- http://SERVERNAME/dav/USERNAME/Calendar
- http://SERVERNAME/dav/USERNAME
- You'll be prompted to either do anonymous/guest or to login. username or usernamer@domain will work.
- Webdav mounts will show up on the CLI as well. Using http://SERVERNAME/dav/USERNAME as an example, you'll get:
- /Volumes/USERNAME
- Using the CLI, you can then use shell tools to test file modification.
I found that TextEdit wouldn't [most of the time] show the webdav path so I could save a new file there.Doh! Do a File > Save As and then to the right of the Save As field click on the down arrow. This will present the full filesystem for browsing. Webdav mounts would be listed under the shared section.- This attempt from the CLI gave open/create errors also:
- cd /Volumes/ajcody/Briefcase
- touch textedit-cli.rtf
- open -e textedit-cli.rtf
- /Applications/TextEdit.app/Contents/MacOS/TextEdit textedit-cli.rtf
- That failed for me on 10.6.3
- Cadaver for Mac, http://cadaver.darwinports.com/ , was a nice tool to use as well for testing.
- Webdav mounts will show up on the CLI as well. Using http://SERVERNAME/dav/USERNAME as an example, you'll get:
- Server Address Examples:
- Click on "Connect To Server"
Mounting WebDAV via CLI With mount_webdav
You can also access the webdav mount over the CLI. See man mount_webdav for details. In theory, you would just mount it and use it like any other mounted volume. This doesn't seem to bypass any existing issues though with webdav on mac against a zimbra server.
Dot And Temp Files Being Made On WebDAV Mount
Normally, one would use this method - Mac OS X v10.4 and later: How to prevent .DS_Store file creation over network connections - to prevent dot files from being made on remote/network mounts. This doesn't seem to work on OSX 10.6.3 when mounting with Finder.
Also, the command - dot_clean - does not remove these ._ files over a webdav mount. See also, Mac OS X: Apple Double Format Creates File Name With the Prefix '._'. This Confluence thread reports the same issue - Add a preference to hide "hidden" files in the attachments and they end up recommending to use another WebDAV client - Transmit [$] or AnyClient [Free].
TinkerTool
I don't have time to test this at this time, but I see that Tinkertool has an option under "Finder" for:
- Network Access
- Don't create hidden .DS_Store files over a network connection
- Note: Affected network folders will no longer store comments, labels, icon positions or similar Finder attributes.
- Don't create hidden .DS_Store files over a network connection
Seems to cause this change:
$ defaults read com.apple.desktopservices { DSDontWriteNetworkStores = true; } [ Now uncheck the option in Tinkertool and click on the Relaunch Finder option. ] $ defaults read com.apple.desktopservices 2010-12-21 06:17:24.200 defaults[18855:903] Domain com.apple.desktopservices does not exist
Reference:
Windows - General
Windows Bugs & RFE's
- Avoid Using Spaces In Filenames
- "WebDAV rename turns ' ' into %20 in the filename"
- "Windows WebDav Client Is Broken"
- Outlines Windows XP Issues & Configuration - Private Comments States steps in comment 13 works for Windows2000 & Windows 2003 as well.
- http://bugzilla.zimbra.com/show_bug.cgi?id=24278
- "WebDav access doesn't work with Vista"
- "WebDAV not compatible with Windows 7"
References
- Something wrong with Windows 7 and webdav. My testing came to the same results as this article states.
- Nice check list to go through for trouble-shooting Windows and Webdav .
- self signed certificate with IE8/windows 7 . This might come into play with the type of webdav connection your attempting [https] and if ZCS is configured with self-signed certificates rather than commercial ones.
Internet Explorer - IE
These various syntax tests either gives an error about access, page not found, or does output an xml type page listing details about the briefcase/notebook contents. It does not however, give it in a usable format - page indexing - where one could download the file so one could view it.
http/s ://$MAILSERVER/service/dav/$USERNAME http/s ://$MAILSERVER/service/dav/$USERNAME@$DOMAIN http/s ://$MAILSERVER/service/dav/$USERNAME%40$DOMAIN http/s ://$MAILSERVER/dav/$USERNAME/Briefcase/ http/s ://$MAILSERVER/dav/$USERNAME/Notebook/ webdav/s ://$MAILSERVER/dav/$USERNAME/Briefcase/ webdav/s ://$MAILSERVER/dav/$USERNAME/Notebook/
Internet Explorer - IE - WinXp Trick
Something I found in regards to Windows XP here and here:
Note: When connecting over non-SSL connections, append a “/#” to the URL here. Windows XP has two built-in methods for connecting via WebDAV. Appending /# is a workaround that forces the use of the Web Folder Client connection, which will authenticate successfully to the WebDav-enabled folder as configured here. [ example: http://ZIMBRA-SERVER/dav/USER@DOMAIN/Briefcase/# or http://ZIMBRA-SERVER/dav/USER@DOMAIN/Briefcase# ]
How-To For Windows 7, ZCS 6.0.6, And Mapping Webdav Drive
References for this section:
I'm leaving some notes here concerning other versions of Windows but your better off consulting WebDAV for non-windows 7 versions rather than here.
IE Performance Work Around
On Windows 7, all requests to WebDAV receive a 3 second delay in the Windows explorer. This makes WebDAV in Windows7 Explorer extremely slow. To fix this, you'll need to change IE's proxy settings:
Open IE : Tools : Internet Options : Connections : LAN settings : Un-check Automatically detect settings : Click Ok :Click Ok
Setup Authentication Type For Webdav
By default, the Mini-Redirector uses Digest authentication. You can change this to Basic authentication in Windows 7 registry.
To confirm webdav is accessible from the windows 7 client and if it requires https or can also use http, test the following links -- adjusting for your zimbra server name and the user. You'll be prompted for login information if you aren't logged into ZWC already when attempting these - login user the USER@DOMAIN format with the user Zimbra password. These should work both under IE8 and Firefox 3. These url's will just show a simple xml/text file with details about the resource.
http://ZIMBRA-SERVER/dav/USER@DOMAIN/Briefcase [this 'path' is the primary use for webdav] http://ZIMBRA-SERVER/dav/USER@DOMAIN/Notebook http://ZIMBRA-SERVER/dav/USER@DOMAIN/Calendar http://ZIMBRA-SERVER/dav/USER@DOMAIN/Tasks http://ZIMBRA-SERVER/dav/USER@DOMAIN/Contacts
To change the authentication style, edit the Windows registry [Windows 7, Vista, WinXP]:
Choose "Run" in the start menu and type: "regedit" Windows 7 & Vista Reg Path: [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WebClient\Parameters\BasicAuthLevel Windows XP Reg Path: [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WebClient\Parameters\UseBasicAuth -- default is set to 1 [good if your using commercial certificate and require https:// for authentication for zimbra] change to 2 to allow basic authentication [http:// for zimbra] The mapping for the value is described as: * 0 - Basic authentication disabled * 1 [dword:00000001] - Basic authentication enabled for SSL shares only [https] * 2 [dword:00000002] or greater - Basic authentication enabled for SSL [https] shares and for non-SSL [http] shares
Turn Off File Locking - Optional
Optional: To turn off locking [Windows 7 only?]:
Choose "Run" in the start menu and type: "regedit" [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WebClient\Parameters] "SupportLocking"=dword:00000000
The Mini-Redirector included in Windows 7 and Vista fully supports SSL and the port can be different from 80. Windows XP only supports port 80.
Confirm WebClient Service To Automatic
Make sure the "WebClient" service is started:
Use "Start->Run->services.msc" and confirm the service "WebClient" is set to automatic.
Restart Windows now.
You should of restarted the windows box after changing and registry values.
- Open Windows Explorer, Start > Run > explorer.exe
- And then do, Tools > Map Network Drive
- Drive Letter - pick a letter you want to use
- Folder Path example:
- Again - Briefcase is your primary webdav folder for Zimbra read/write use, others are just shown as examples for viewing.
- UNC examples
- \\ZIMBRA-SERVER@80\dav\USER@DOMAIN\Briefcase
- \\ZIMBRA-SERVER@80\dav\USER@DOMAIN\Notebook
- \\ZIMBRA-SERVER@80\dav\USER@DOMAIN\Tasks
- \\ZIMBRA-SERVER@80\dav\USER@DOMAIN\Calendar
- \\ZIMBRA-SERVER@80\dav\USER@DOMAIN\Contacts
- UNC examples
- Again - Briefcase is your primary webdav folder for Zimbra read/write use, others are just shown as examples for viewing.
- Reconnect at logon - your choice.
- "Connect using different credentials"
- If you check this box, you might you need to enter your username and password twice before you see the share.
- I was able to remount shares without checking it, but I believe Windows had already cached my authentication details at that point.
C:\Users\ajcody>net use p: http://ZIMBRA-SERVER/dav/USER/Calendar /user:USER /persistent:yes Enter the password for 'USER' to connect to 'ZIMBRA-SERVER': The command completed successfully. C:\Users\ajcody>dir p:\ Volume in drive P has no label. Volume Serial Number is 0000-0000 Directory of p:\ 04/28/2010 07:50 AM <DIR> . 04/28/2010 07:50 AM <DIR> .. The parameter is incorrect. ??,☼ 0 39bd19c5-e71d-4cd2-9777-badf118b34c9.ics 1 File(s) 0 bytes 2 Dir(s) 104,353,939,456 bytes free
How-To For Windows 7, ZCS 6.0.6, And Webdrive
Download evaluation version from url below and installed it. My version was 9.12:
I have the registry settings already set on my Windows 7 machine as described above prior to doing this test.
Setup a new site using:
- Enter the name for the site you will be connecting to:
- I put in mail59-briefcase since mail59 is my test server and I would be connecting to my briefcase
- Drive Letter
- I went with the default that was selected - W
- Next
- Server type
- WebDAV
- Left "connect securely" unchecked. If you require https to login to zimbra, you'll want to check this.
- Enter the URL for the Server
- I put in : http://mail59.DOMAIN.com/dav/ajcody/Briefcase
- left "use default port - 80" checked. 80 for http and 443 for https is your using the "connect securely" option in past window.
- Next
- For username, I put in : ajcody
- For password, put in my zimbra password
- left the "save password" checked.
- Clicked on the "Test Connection" and confirmed it worked.
- Next
- Checked the "connect to site now"
- Checked the "connect at login/startup"
- Finished
The default properties that are setup seem to cause some issues with renames and so forth. Left some ugly cache files as well. I adjusted the properties like below and things seem to work well with them like this. Open up the "Properties" section for the new site you just made.
- File
- check - Encode filenames in UTF-8
- check - Cache temporary MS Office files
- check - Ignore Desktop.ini files
- uncheck - Enable NTFS File Security [for applications requiring NT security]
- check - Cache small writes by applications
- check - Test for Write Access when files are created
- Ok
Disconnect the map drive if it is and reconnect it.
Linux
Linux Webdav Bug/RFE's
- "RFE: Official Support For Full Functionality Of Evolution Mail Client"
- http://bugzilla.zimbra.com/show_bug.cgi?id=47097
- I'll probably need to rename this as private comments on the RFE actually impact Evolution and KDE based apps in regards to webdav use.
- http://bugzilla.zimbra.com/show_bug.cgi?id=47097
- "RFE: Official Support For Full Functionality Of KDE Kontact Mail Client [webdav]"
- "davfs2 can't create/copy files via Zimbra WebDav"
Mounting WebDAV
mount.davfs - davfs2
Summary: Isn't working reliably. Appears to mount drive but shortly after I encounter problems. Doing a df -h will show the mount path but with the error of "Transport endpoint is not connected". Can umount if I use the -f flag as root.
Ubuntu example:
apt-get install davfs2 sudo dpkg-reconfigure davfs2 ** SUID bit set so non-root users can mount.
You can customize davfs's behavior in /etc/davfs2/davfs2.conf . These config files will also exist in the user's home directory if you want to enable them there - $HOME/.davfs2/ . If you setup davfs2 for suid, then you'll want to note this option in davfs2.conf below. You'll want to add the unix usernames that will be doing the webdav mounts in the group mentioned in that variable in /etc/group .
dav_group users
I've also read that davfs2 users object titles for filenames, meaning that if two 'objects' have the same title then only one will be displayed on the filesystem. To fix, edit /etc/davfs2/davfs2.conf and add or change:
use_displayname 0
Another option you might want to set in davfs2.conf is below. Some references to turning off locking states it as a mount option - -o nolocks , this no longer works or is available with mount with my testing. [Most likely you'll want to add this to the users $HOME/.davfs2/davfs2.conf file]
# use_locks 1 [default is 1, meaning it's on] use_locks 0
Setup mount point as user, ajcody in example:
mkdir -p /home/ajcody/davfs/mail59
Test mounting via the CLI, example:
mount -t davfs 'http://192.168.0.59/dav/ajcody' /home/ajcody/davfs/mail59
Example in /etc/fstab :
http://192.168.0.59/dav/ajcody /home/ajcody/davfs/mail59 davfs rw,noexec,nosuid,noauto,nodev,user 0 0
Once in /etc/fstab, the user can do the following - using example:
mount /home/ajcody/davfs/mail59
fusedav
On Ubuntu, easy as doing a :
apt-get install fusedav .
Then vi /etc/fuse.conf and set the option for - user_allow_other .
Example from CLI:
[as ajcody] mkdir -p /home/ajcody/davfs/mail59 fusedav -u username -p password http://192.168.0.59/av/ajcody /home/ajcody/davfs/mail59
wdfs
If you have wdfs installed, then you can add a line like this in /etc/fstab:
wdfs#http://192.168.0.59/dav/ajcody /home/ajcody/davfs/mail59 fuse rw,noexec,nosuid,noauto,nodev,user,owner 0 0
Gnome Apps
Evolution And WebDAV
- Contacts Via WebDAV and Evolution
- "RFE: Official Support For Full Functionality Of Evolution Mail Client"
- http://bugzilla.zimbra.com/show_bug.cgi?id=47097
- I'll probably need to rename this as private comments on the RFE actually impact Evolution and KDE based apps in regards to webdav use.
- http://bugzilla.zimbra.com/show_bug.cgi?id=47097
Nautilus - Gnome File Manager
To setup a webdav connection:
- File > Connect to Server
- Service Type = Webdav (HTTP) or Secure Webdav (HTTPS)
- Server = [example] 192.168.0.59 or mail59.zimbra.DOMAIN.com
- Port = 80 for HTTP , 443 for HTTPS
- Folder = dav/ajcody or dav/ajcody/Briefcase
- User Name = [example] ajcody or ajcody@mail59.zimbra.DOMAIN.com
- Check the "Add bookmark" box
- Bookmark name = this is the name that will show up under Nautilus Places sidebar
The path that is establish if you were to manual type it in the path/url box would be:
dav://ajcody@192.168.0.59/dav/ajcody
Which can also be witnessed with the CLI command for gvfs mounts:
$ gvfs-mount -l Drive(0): CD/DVD/HDDVD Drive Type: GProxyDrive (GProxyVolumeMonitorGdu) Mount(0): WebDAV on 192.168.0.59 -> dav://ajcody@192.168.0.59/dav/ajcody Type: GDaemonMount
This seems to work for file opening and browsing.
Nautilus For KDE Workstation
For example, if you use Kubuntu, just install nautilus like:
sudo apt-get install nautilus
You can then launch from the cli to test out by doing:
nautilus
Here's a little bit that is sorta stupid. Nautilus will actually cache the webdav data in this location [for example] :
/home/ajcody/.gvfs/WebDAV on 192.168.0.59/
I can use konquerer/dolphin to then browse to /home/ajcody/.gvfs/WebDAV on 192.168.0.59/Briefcase/ and open files that would normally fail if I mounted the webdav location directly in dolphin. At least for a jpg file I'm testing against. For .doc files, I found OOo would fail but I could use Abiword just fine. This is probably tied to cache files OOo is trying to make vs Abiword not creating them.
Nautilus - GVFS And FUSE - FUSEDAV
Basic Package names: gvfs-fuse fuse fusedav
Reference to tracks bugs tied to GVFS-FUSE, GVFS-Webdav Backend:
- gvfs + fuse open bug listing
- gvfs + webdav backend open bug listing
- Gnome GVFS Bug 605619 - Problem when copying folders with Nautilus using WebDAV
There are some gvfs commands you can use as well to trouble shoot issues with gvfs/fuse-dav and Nautilus:
gvfs-cat gvfs-ls gvfs-mount gvfs-rm gvfs-tree gvfs-copy gvfs-mkdir gvfs-move gvfs-save gvfs-info gvfs-monitor-dir gvfs-open gvfs-set-attribute gvfs-less gvfs-monitor-file gvfs-rename gvfs-trash
KDE Apps
Bugs Against KDE
- "Webdav not compatible with KDE's webdav kioslave"
- http://bugzilla.zimbra.com/show_bug.cgi?id=23422
- Note, KDE bugzilla reference:
- "webdav kioslave unable to delete folder on apache dav_fs enabled webdav folder"
- https://bugs.kde.org/show_bug.cgi?id=209508
- Note, KDE bugzilla reference:
- http://bugzilla.zimbra.com/show_bug.cgi?id=23422
- "RFE: Official Support For Full Functionality Of KDE Kontact Mail Client [webdav]"
KNetAttach
Manual for KNetAttach , which can do webdav mounts. The manual states: "Network folders show up in a special location of Konqueror and Dolphin called a virtual folder. This virtual folder is accessed by typing remote:/ in the location bar or by selecting Network from the Places panel. You will then be able to see any folders which have been previously added and you also will be able to use the wizard to add new ones."
To use with Dolphin, you'll see on the left an object/icon called Network. Click on that and in the main body of Dolphin you should see an object called "Add Network Folder" now to configure a webdav mount.
To use with Konqueror, enabled the sidebar. Settings > Sidebar or F9 . On the left, look for the icon called Network. The, from the top bar - Go > Network Folders . You'll see the "Add Network Folder" object now to configure a webdav mount.
KDE3 - Konqueror3
This works for read access, but not write. Though it does seems to allow you to create a "New Folder" when you right-click on the page and do Select New > Folder. The other options - Text File, etc. - fail as well as drop and drag.
- Open Konquerer
- Put in the appropriate url and authenticate when popup shows
- webdav/s://SERVERNAME/dav/USERNAME/Briefcase
KDE4 - Konqueror4
This doesn't seem to work with reads or writes.
- Open Konquerer
- Put in the appropriate url and authenticate when popup shows
- webdav/s://SERVERNAME/dav/USERNAME/Briefcase
- webdav/s://SERVERNAME/service/dav/USERNAME/Briefcase
- http/s://SERVERNAME/dav/USERNAME/Briefcase
- http/s://SERVERNAME/service/dav/USERNAME/Briefcase
The http will output the properties of the destination but doesn't actually give a file listing.
Firefox
These various syntax tests either gives an error about access, page not found, or does output an xml type page listing details about the briefcase/notebook contents. It does not however, give it in a usable format - page indexing - where one could download the file so one could view it.
http/s ://$MAILSERVER/service/dav/$USERNAME http/s ://$MAILSERVER/service/dav/$USERNAME@$DOMAIN http/s ://$MAILSERVER/service/dav/$USERNAME%40$DOMAIN http/s ://$MAILSERVER/dav/$USERNAME/Briefcase/ http/s ://$MAILSERVER/dav/$USERNAME/Notebook/ webdav/s ://$MAILSERVER/dav/$USERNAME/Briefcase/ webdav/s ://$MAILSERVER/dav/$USERNAME/Notebook/
RSS
Notable Bugs And RFE's
- "Expand documentation in admin guide/wiki for RSS topics"
Zimbra And Alfresco
Providing reference URL for those that might be interesting in doing this. This wouldn't be an issue that Zimbra support handles though, just trying to be helpful to those that aren't afraid to go it alone.
- Alfresco + Zimbra LDAP
Testing & Debugging Tools
Actual Testing & Debugging Homepage
Please see Ajcody-Testing-Debugging
Other Debug References
When Was A ZCS Service Enabled Or Disabled=
See Ajcody-Notes-Archive-Discovery#When_Was_A_ZCS_Service_Enabled_Or_Disabled
Ajcody Logging Wiki
See Ajcody-Logging
Other Zimbra Wiki Pages
RFE's Related To Better Logging And Historical Data Of Systems
- Bug 93954 : RFE: admin console services status page and service enable/disable page improvements
- Bug 93950 : RFE: History files written when changes to individual ZCS services have taken place
- Bug 93784 : Proxy check script
- Bug 93656 : zmdiaglog to collect history or diffs of server variables changes via backups
- Bug 90598 : RFE: Add option to let the MTA log a mail's subject
- Bug 90400 : Provide User's Mailbox Summary/Reporting for the end user in ZWC/email report
- Bug 90032 : zmdiaglog: don't attempt to process mailboxd collection activities on non-mailbox nodes
- Bug 89860 : gather additional system information in zmdiaglog
- Bug 89700 : proxy logs goto logger host
- Bug 89450 : Include postfix-logwatch and amavis-logwatch
- Bug 89008 : Source Port Logging in Jetty For EU Legal Requirements
- This bug was initially created as a clone of Bug #85073, include client (source) ports for all mail related connections, in addition to client IP. Need this done for Jetty, nginx and postfix already complete.
- Bug 88414 : expand 3rd party performance and diagnostic software support
- Bug 88412 : virtualization env. specific checks and diagnostic scripts
- Bug 88410 : zmdiaglog gets moved to it's own package, zimbra-diagnostics and we include public package repro to update
- Bug 88408 : expand zmrcd commands and/or zmdiaglog to do more specific level of log collection
- Bug 88406 : proxy diagnostic script
- Bug 88404 : post-installer check script
- Bug 85523 : Individual logs for zmlocalconfig edits and all services start, stop, restarts
- Bug 85520 : All backups should include the localconfig.xml
- Bug 85073 : Source Port Logging For EU Legal Requirements
- Bug 84313 : zmcontrol should give more details when a service isn't in a running state
- Bug 84273 : Expand zmdiaglog to better fit as a support case creation tool
- I have a lot of other bugs/rfe's references in this one also.
- Private Bug
- Bug 84272 : Host our own ZCS instance for customers support accounts
- Private Bug
- Bug 84271 : Integration of account logins for ftp for support cases
- Private Bug
- Bug 84142 : Add script for detecting compromised accounts
- Bug 83779 : zmdiaglog: include all ZCS configs which are different from the default value
- Bug 83227 : *.trace.log - what's it for, how to control growth, better retention policy
- Bug 81201 : zmcontrol start --debug offer verbose logging for all services and notes logs
- Bug 79883 : Expand Monitor > MobileSync Statistics reporting
- Bug 78251 : RFE: Display Date And Time for each command
- Bug 77076 : All Ext. Auth options should have documentation about limitations and expectations
- Bug 77075 : Flowchart of ext. auth and where log events occur
- Bug 76540 : RFE: Priovide ability to view major logs, edit configs and terminal login within admin console
- Bug 75502 : Logging should state if any stop/start/restart of service is manual or automated
- Bug 62315 : RFE: integrate log analysis project - AWstats or Analog, Webalizer, etc.
- Bug 62095 : Incorrect graphs in Administration Monitoring Server Statistics message count, message volume
- Bug 57871 : RFE: replace the current syslog + flash based log system to a Hyperic HQ + HTML5 based log solution
- Bug 54034 : RFE: zimbra-attrs.xml & zmprov desc state if immediate, zmprov fc , or service restart needed
- Bug 51100 : RFE: zmprov variable edits should output if a service restart is need or flush cache
- Bug 49557 : RFE: Login Format Option other than user@domain ,ex. user+domain
- Bug 45383 : RFE: Enhance documentation as to where connection logging should be expected - esp Active Sync
- Bug 28934 : More logging options via admin console
- Bug 12196 : support for real time counters (snmp, jmx, etc)
Various Errors And Log Items One Might See
Logging
See Ajcody-Logging for more complete logging information.
mailbox.log and zimbra.log
- Zimbra Support will likely need to see the /opt/zimbra/log/mailbox.log file and possibly /var/log/zimbra.log from the system. These files contain the basic operation data from the system, and can tell us if the server has something seriously wrong with it. They can also be correlated against other collected data to give a complete picture of the workings of the system.
mysql_error.log and myslow.log
- These files are both located in /opt/zimbra/log/ on the mail store server. They contain information about the health of the mysql database. If there is data corruption or another problem causing direct mysql errors, events will be logged in mysql_error.log. If certain search requests are taking longer to complete than others, they will be logged in myslow.log.
zmmailboxd.out
- This file contains startup information for mailboxd and thread dumps created whenever mailboxd is shut down. If a server goes completely nonresponsive and is restarted, the thread dump captured here will tell us if there are certain threads blocking other threads' access to critical data elements. Frequently, slow behavior can be caused by these thread locks.
- In some cases, it may be necessary to monitor garbage collection and other operations at the Java VM level. To enable this logging, add the following to mailboxd_java_options.
- To get current options:
zmlocalconfig mailboxd_java_options
- Then appended the following to your current set of options - "-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime". PLEASE NOTE FOR BELOW, "YOUR EXISTING OPTIONS" should be replaced with the output above.
zmlocalconfig -e mailboxd_java_options="YOUR EXISTING OPTIONS -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime"
Connections And Or Threads Mentioned In mailbox.log
Tuning Variables Related To Threads And Connections
Resources to "tune" the below variables:
zimbraHttpNumThreads
- ZCS 5.0 has a single thread pool for both HTTP and HTTPS
- To see current setting
- zmprov gs `zmhostname` zimbraHttpNumThreads
- To modify setting
- zmprov ms `zmhostname` zimbraHttpNumThreads 500
- To see current setting
zimbraPop3NumThreads
- This setting of 300 is able to support a few 10s of thousands of users checking mailing every 8 minutes.
- To see current setting
- zmprov gs `zmhostname` zimbraPop3NumThreads
- To modify setting
- zmprov ms `zmhostname` zimbraPop3NumThreads 300
- To see current setting
zimbraImapNumThreads
- IMAP thread settings.
- To see current setting
- zmprov gs `zmhostname` zimbraImapNumThreads
- To modify setting
- zmprov ms `zmhostname` zimbraImapNumThreads 500
- To see current setting
zimbraLmtpNumThreads
- LMTP thread settings.
- To see current setting
- zmprov gs `zmhostname` zimbraLmtpNumThreads
- To modify setting
- zmprov ms `zmhostname` zimbraLmtpNumThreads 40
- To see current setting
mailboxd_java_options
- JVM (java Virtual Machine options) ZCS 5.0 and later
- To see current setting
- zmlocalconfig mailboxd_java_options
- To modify setting
- zmlocalconfig mailboxd_java_options="-client -XX:NewRatio=2 -Djava.awt.headless=true -XX:MaxPermSize=128m -XX:SoftRefLRUPolicyMSPerMB=1"
- The above is using the "default" options as an example to modify.
- zmlocalconfig mailboxd_java_options="-client -XX:NewRatio=2 -Djava.awt.headless=true -XX:MaxPermSize=128m -XX:SoftRefLRUPolicyMSPerMB=1"
- Other various configurations for mailboxd_java_options. See sections related to heap and thread dumps below for more details.
- mailboxd_java_options="-client -XX:NewRatio=2 -XX:MaxPermSize=128m -Djava.awt.headless=true -XX:SoftRefLRUPolicyMSPerMB=1 -XX:+UseParallelGC"
- mailboxd_java_options="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/some/directory/that/exists/and/is/zimbra/writable"
- To see current setting
mailboxd_java_heap_memory_percent
- JVM Cache setting
- To see current setting
- zmlocalconfig mailboxd_java_heap_memory_percent
- To modify setting
- zmlocalconfig -e mailboxd_java_heap_memory_percent=40
- To see current setting
zimbraMessageCacheSize
- Mailbox message cache. The mailbox server maintains a cache of message bodies (blobs on disk from /opt/zimbra/store). This cache speeds up retrieval of message content for mailapps such as mail.app
- To see current setting
- zmprov gs `zmhostname` zimbraMessageCacheSize
- To modify setting
- zmprov ms `zmhostname` zimbraMessageCacheSize 104857600
- To see current setting
zimbra_mysql_connector_maxActive
- zimbra_mysql_connector_maxActive
- To see current setting
- zmlocalconfig zimbra_mysql_connector_maxActive
- To modify setting
- zmlocalconfig -e zimbra_mysql_connector_maxActive=100
- To see current setting
Example Of mailbox.log event:
dbconn - Connection pool is 75% utilized (88 connections out of a maximum of 100 in use). Turn on debug logging for zimbra.dbconn to see stack traces of connections not returned to the pool.
zimbraXMPPEnabled
There is a potential problem with XMPP [Instant Messaging]. XMPP is a beta feature and is not supported. If you're not using XMPP anyways, disabling it is strongly recommended, and XMPP is a beta feature and not supported in a production environment today.
- Confirm if variable is set to False or True
- zmprov gacf zimbraXMPPEnabled
- Set to FALSE to turn off
- zmprov mcf zimbraXMPPEnabled FALSE
- Then at some point restart the mailstores.
Bug reference:
- "IM service connection leak,memory leak"
Zimbra Tools And Commands
zmstat and zmstat-chart
Resources:
Down And Dirty Example zmstat
Zmstats runs constantly on all current Zimbra systems. It monitors various Zimbra components as well as the whole system to give a good picture of how the system is performing over time. Stats charts are extremely useful for troubleshooting performance issues and can often point to a bottleneck on the system itself or to specific problems in the mailboxd Java VM.
To generate a stat chart, run the following:
zmstat-chart -d <output directory> -s /opt/zimbra/zmstat/<date> --title "<server> <date>"
The date is in the format 'YYYY-MM-DD'. Generally when troubleshooting a performance problem, Zimbra Support will need to see several days' worth of statistics data, as well as the log files and possibly thread dumps from the same time period.
zmdiaglog
See also : Zmdiaglog
Running a full zmdiaglog would take many stats such as zmstats, thread dump, lsof, core dump, and heap dump. The heap dump portion could take longer than the other steps and is invasive on the system. If possible, it should only be ran if it is late in the evening or if the server is not responding anyway. Prior to 5.0.16, this tool may run a heap dump, which would be interruptive on a running server. From 5.0.16, a heap dump requires "zmdiaglog -a". Also, you can CTRL-C to exit out of the zmdiaglog during the heap dump process without affecting the rest of the dump. You will just need to compress the files manually.
As root:
/opt/zimbra/libexec/zmdiaglog
This will create a zip file with allot of relevant data for troubleshooting issues.
zimdialog To Get Full Heap Dump
In order to get a full heap dump, one has to make sure that HeapDumps have been enabled in the JVM options. In order to do this, please first check your mailboxd_java_options on that server:
# su - zimbra $ zmlocalconfig mailboxd_java_options
Then, append these additional arguments:
-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/some/directory/that/exists/and/is/zimbra/writable
It is very important that the HeapDumpPath chosen is large enough to dump a multi-GB core file and heap dump. It is also very important to make sure that these options are appended to your existing options.
Example [You might need to adjust based upon your existing mailboxd_java_options]:
$ zmlocalconfig mailboxd_java_options mailboxd_java_options = -client -XX:NewRatio=2 -Djava.awt.headless=true -XX:MaxPermSize=128m -XX:SoftRefLRUPolicyMSPerMB=1
$ zmlocalconfig -e mailboxd_java_options="-client -XX:NewRatio=2 -Djava.awt.headless=true -XX:MaxPermSize=128m -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp"
Then restart ZCS on that mailstore.
Caution: If /var/tmp does not enough space, it could fill the filesystem/server and cause very bad problems!
Then, if any problem with performance occurs again, run the zmdiaglog as root:
[Prior to 5.0.16]
/opt/zimbra/libexec/zmdiaglog
[For 5.0.16 and forward]
/opt/zimbra/libexec/zmdiaglog -a
Note: there are optional options to zmdiaglog that allows it to write to locations other than /tmp, again very important not to fill up the filesystem. Run zmdiaglog -h to get the list of options:
# /opt/zimbra/libexec/zmdiaglog -h Usage: zmdiaglog [-dth] -d - Log destination (Default /tmp/zmdiaglog.PID) -t - Timeout in seconds for hanging commands (Default 120) -h - This help message
This may take a few minutes to complete, as it is writing a multiple GB file. Once complete, then restart mailboxd:
# su - zimbra $ zmmailboxdctl restart
You will then provide to Zimbra, via FTP, the zmdiaglog information that is written to /tmp. Our developers will most likely need to debug the information provided rather than support staff. Please be patient, support will provide feedback as they hear the conclusions that the developers provide.
There are lots of steps requested here, but this information is critical if we are to diagnose this reported problem. The data provided by zmdiaglog is very important.
Using JDK 1.6 For zmdiaglog
See Ajcody-Testing-Debugging#jmap_And_JDK_1.6
Note About Memory And Linux
In general, memory used by the JVM can occasionally spike above the maximum heap size under normal usage, but we would expect that it would go back down again eventually. Also, please note that all Linux systems will utilize all memory available on the system - this is because the kernel will cache any data it can in order for possible reuse; this is perfectly normal, and in the case that memory is needed, older cache will be deleted from memory and memory freed for other use. No Linux system will show much memory as "free" after being up for some period of time, for example when using top or vmstat. All memory will get utilized by the system, and this is a good thing for performance - you paid for the memory, might as well use it. Where it could be a problem is if one process grows uncontrolled, or if swapping is occurring.
Thread Dumps And Specific Commands To Use
A thread dump is a printout of the status of all the running threads in the mailboxd process at a specific point in time. A thread dump allows Zimbra engineers to see how the system is operating, what each thread is doing, and what data elements are being accessed by individual threads. If a performance bottleneck is not identifiable from the stat charts alone, it may be necessary to generate a periodic thread dump.
zmthrdump - ZCS 5.0.10+
The thread dump from zmthrdump will show which threads are currently in process within the jvm. zmthrdump is useful when you want to grab several thread dumps in short intervals in order to track and look for locked threads. Typically, the thread dumps are useful for tracking what is waiting on what in the thread stack. If the server is also experiencing GC [garbage collection] issue, it is generally better to just have a full zmdiaglog (with valid java memory heap dump) along with the rest of the data it collects.
Resources:
- zmthrdump help
- zmthrdump [-h] [-i] [-t <timeout seconds>] [-p <pid file>] [-f <file>] [-o <out-file>]
zmmailboxdmgr And Threaddump - Prior to 5.0.10
Run this command as root
for i in 1 2 3; do /opt/zimbra/libexec/zmmailboxdmgr threaddump; sleep 30; done
/opt/zimbra/log/zmmailboxd.out will reflect the thread dumps.
Thread Dump Script Example 1
Usage of the below script:
- $0 FILENAME -dl [-frames #_ stack_frames] [-id REGEXP] [-all]
- $0 FILENAME -dt [-frames #_ stack_frames] [-sort id|state] [-stack REGEXP] [-id REGEXP] [-state REGEXP] [-waiting NUMBER]
- frames: controls the # lines of stack trace included in the output
- id: only include where the id matches REGEXP
- all: include ALL locks (not just ones that other threads are blocked-on)
- sort: controls the order threads are printed out (locks always printed in lock ID order)
- stack: only include threads where the thread's stack output matches REGEXP
- state: only include threads where the thread state (e.g. RUNNABLE) matches REGEXP
- waiting: only inclure threads are blocking other threads
Examples:
- $0 threads.txt -dt -stack MailboxIndex.java -state RUNNABLE -f 20
- -- dumps all RUNNABLE threads with MailboxIndex.java on the stack (1st 20 lines of the stack)
- $0 threads.txt -dt -stack MailboxIndex.java -state RUNNABLE -f 20
- $0 threads.txt -dl -f 0
- -- dumps the list of locks in the system that are blocking at least one thread
- $0 threads.txt -dl -f 0
- $0 threads.txt -dt -f 0 -sort state
- -- dumps a list of all the threads in the system and tells you their run state
- $0 threads.txt -dt -f 0 -sort state
- $0 threads.txt -dt -f 20 -w 1
- -- dumps a list of all the threads in the system that are blocking other threads (quick way to hunt for contention)
- $0 threads.txt -dt -f 20 -w 1
Working Example:
- threadDumpAnalyzer.pl zmmailboxd.out -dl -f 100 ~/Desktop
threadDumpAnalyzer.pl Script
#!/usr/bin/perl -w # # ***** BEGIN LICENSE BLOCK ***** # # Zimbra Collaboration Suite Server # Copyright (C) 2004, 2005, 2006, 2007 Zimbra, Inc. # # The contents of this file are subject to the Yahoo! Public License # Version 1.0 ("License"); you may not use this file except in # compliance with the License. You may obtain a copy of the License at # http://www.zimbra.com/license. # # Software distributed under the License is distributed on an "AS IS" # basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. # # ***** END LICENSE BLOCK ***** # use strict; use warnings; use Getopt::Long; my %locks; # by LockID - hash w/ 'owner','type' my %threads; # by threadId - hash w/ 'state' 'waitingOnLock' my $filename = $ARGV[0]; my ($dumpLocks, $dumpThreads, $searchThreadStack, $searchThreadId, $stackFrames, $sort, $filterByState, $allLocks); my ($waiting); $stackFrames = 10; $sort = "state"; GetOptions( "dl" => \$dumpLocks, "dt" => \$dumpThreads, "frames=s" => \$stackFrames, "sort=s" => \$sort, "id=s" => \$searchThreadId, "state=s" => \$filterByState, "stack=s" => \$searchThreadStack, "waiting=s" => \$waiting, "all" => \$allLocks, ); sub dumpLocks(); sub handleThread(); sub dumpThreads(); sub usage(); sub readFile($); sub getBlockedThreads($); if (!defined $filename) { usage(); } readFile($filename); if (defined $dumpLocks) { dumpLocks(); } elsif (defined $dumpThreads) { dumpThreads(); } else { usage(); } exit(0); sub usage() { my $usage = <<END_OF_USAGE; USAGE: $0 FILENAME -dl [-frames #_ stack_frames] [-id REGEXP] [-all] $0 FILENAME -dt [-frames #_ stack_frames] [-sort id|state] [-stack REGEXP] [-id REGEXP] [-state REGEXP] [-waiting NUMBER] frames: controls the # lines of stack trace included in the output id: only include where the id matches REGEXP all: include ALL locks (not just ones that other threads are blocked-on) sort: controls the order threads are printed out (locks always printed in lock ID order) stack: only include threads where the thread's stack output matches REGEXP state: only include threads where the thread state (e.g. RUNNABLE) matches REGEXP waiting: only inclure threads are blocking other threads Examples: $0 threads.txt -dt -stack MailboxIndex.java -state RUNNABLE -f 20 -- dumps all RUNNABLE threads with MailboxIndex.java on the stack (1st 20 lines of the stack) $0 threads.txt -dl -f 0 -- dumps the list of locks in the system that are blocking at least one thread $0 threads.txt -dt -f 0 -sort state -- dumps a list of all the threads in the system and tells you their run state $0 threads.txt -dt -f 20 -w 1 -- dumps a list of all the threads in the system that are blocking other threads (quick way to hunt for contention) END_OF_USAGE die $usage; } sub mySort($$) { my ($a, $b) = @_; if ($sort eq "state") { my $state1 = $threads{$a}{state}; my $state2 = $threads{$b}{state}; if (!defined $state1) { return 1; } elsif (!defined $state2) { return -1; } return $state1 cmp $state2; } else { return $a cmp $b; } } sub padToWidth($$) { my ($str, $width) = @_; if (!defined($str)) { $str = ""; } return sprintf "%-*s", $width, $str; } sub formatStackTrace($$) { my ($stack, $indent) = @_; my $ret; if ($stackFrames == 0) { return ""; } my $num = $stackFrames; foreach my $line (split /\n/, $stack) { if ($num <= 0) { return $ret; } $ret .= $indent.$line."\n"; $num--; } return $ret; } sub formatLock($) { my $lockId = shift; my $output = "$lockId - "; my $numData = 0; while (my ($k,$v) = each %{ $locks{$lockId}}) { $output .= " $k=$v"; $numData++; } if ($numData == 0) { $output .= " UNKNOWN"; } return $output; } sub formatThread($) { my $threadId = shift; if (!defined $threadId) { return ""; } my $foo = padToWidth($threadId, 50); if (!defined($foo) || $foo eq "") { $foo = "ASDF"; } my $bar = $threads{$threadId}{state}; if (!defined($bar) || $bar eq "") { $bar = "HJKL"; } my $ret = $foo.$bar."\n"; if (defined $threads{$threadId}{waitingOnLock}) { $ret .= "\tWaiting for: ".formatLock($threads{$threadId}{waitingOnLock})."\n"; } my @blockedThreads = getBlockedThreads($threadId); for my $blockedThread (sort @blockedThreads) { $ret .= "\t$blockedThread is waiting on this thread\n"; } $ret .= formatStackTrace($threads{$threadId}{stack}, "\t "); if ($stackFrames > 0) { $ret .= "\n"; } return $ret; } # given a lockId, return a list of the threads that are blocked on it sub getLockWaiters($) { my $lockId = shift; my @ret; while (my ($threadId,$thread) = each %threads ) { if (defined $thread->{waitingOnLock} && $thread->{waitingOnLock} eq $lockId) { push @ret, $threadId; } } return @ret; } sub getLockOwner($) { my $lockId = shift; if (defined $locks{$lockId} && defined $locks{$lockId}{owner}) { return $locks{$lockId}{owner}; } else { return ""; } } # given a threadId, get a list of all other threads that are blocked # on locks it is holding sub getBlockedThreads($) { my $threadId = shift; my @ret; foreach my $lockId ( sort keys %locks ) { if (getLockOwner($lockId) eq $threadId) { # a lock we own my @blockedThreads = getLockWaiters($lockId); foreach my $blockedThread (@blockedThreads) { push @ret, $blockedThread; push @ret, getBlockedThreads($blockedThread); #recurse! } } } return @ret; } sub dumpLocks() { foreach my $lockId ( sort keys %locks ) { my $ret = ""; my $numWaiters = 0; $ret .= "LOCK: $lockId "; while (my ($k,$v) = each %{ $locks{$lockId}}) { $ret .= "$k=$v, "; } $ret .= "\n"; while (my ($threadId,$thread) = each %threads ) { if (defined $thread->{waitingOnLock} && $thread->{waitingOnLock} eq $lockId) { $ret .= "\tThread $threadId is waiting for this lock\n"; $numWaiters++; } } $ret .= formatStackTrace($threads{getLockOwner($lockId)}{stack}, "\t"); if (!defined $searchThreadId || $lockId =~ /$searchThreadId/) { if ($numWaiters > 0 || defined $allLocks) { if ($stackFrames > 0) { $ret .= "\n"; } print $ret; } } } } sub dumpThreads() { foreach my $threadId ( sort { mySort($a, $b) } keys %threads ) { if (!defined $threadId) { # continue } elsif (defined $searchThreadStack && $threads{$threadId}{stack} !~ /$searchThreadStack/) { # continue } elsif (defined $searchThreadId && $threadId !~ /$searchThreadId/) { # continue } elsif (defined $filterByState && $threads{$threadId}{state} !~ /$filterByState/) { # continue } elsif (defined $waiting && getBlockedThreads($threadId) < $waiting) { # continue } else { print formatThread($threadId); } } } sub readFile($) { my $filename = shift; open IN, '<', $filename or die "couldn't open $filename"; my @curThread; while (<>) { chomp; s/\r//g; if ($_ eq "") { if (@curThread) { my $threadId; my @locksHeld; my $waitingOnLock; my $threadState; my $output; my $firstLineState; # 1stline my $line = shift @curThread; $output .= "$line\n"; if ($line =~ /"(.*)"/) { $threadId = $1; if ($line =~/nid=0x[0-9a-f]+\s([a-zA-Z\s\.()]+)/) { $threads{$threadId}{state} = $1; } } else { $threadId = $line; } if ($threadId eq "") { $threadId = "none"; } # 2nd line $line = shift @curThread; if (defined $line) { $output .= "$line\n"; if ($line =~ /State: ([A-Z_]+)/) { $threadState = $1; $threads{$threadId}{state} = $1; } foreach $line (@curThread) { $output .= "$line\n"; if ($line =~ /locked <(0x[0-9a-f]+)>\s?(.*)?/) { push @locksHeld, $1; $locks{$1}{owner} = $threadId; $locks{$1}{type} = $2; } elsif ($line =~ /- waiting to lock <(0x[0-9a-f]+)>/) { $waitingOnLock = $1; $threads{$threadId}{waitingOnLock} = $1; } elsif ($line =~ /- waiting to lock <(0x[0-9a-f]+)>/) { $waitingOnLock = $1; $threads{$threadId}{waitingOnLock} = $1; } } } else { $threads{$threadId}{state} = "unknown"; } $threads{$threadId}{stack} = $output; @curThread = (); } } else { push @curThread, $_; } } close IN; }
Thread Dump Script Example 2
The following script generates five thread dumps within one minute:
#!/bin/bash # # Dump 5 threads and proc stats for mailboxd Java PID in one minute. # Daily output written to /tmp/zmperiodic-YYYYMMDD # # Execute the script with cron # crontab: * * * * * /tmp/zmperiodic.sh DUMPDIR="/tmp/zmperiodic-$(date '+%Y%m%d')" if [ ! -d $DUMPDIR ] then mkdir $DUMPDIR fi for ((i=0; i<5; i++)) do echo "" > /opt/zimbra/log/zmmailboxd.out STAMP=$(date '+%Y%m%d.%H%M%S') JPID=$(cat /opt/zimbra/log/zmmailboxd_java.pid) kill -3 $JPID sleep 1 cp /opt/zimbra/log/zmmailboxd.out $DUMPDIR/zmmailboxd.out-$STAMP cat /proc/$JPID/task/*/stat > $DUMPDIR/proc-stats-$STAMP if [ $i -ne 4 ] then sleep 11 else exit fi done
It should be placed in cron so that it runs every minute. The output data will be placed in a directory named /tmp/zmperiodic-YYYYMMDD. This data will consist of thread dumps along with thread data from /proc.
Heap Dump Specific Commands
jmap And JDK 1.6
Go here:
http://java.sun.com/javase/downloads/index.jsp
Select:
- Java SE Development Kit (JDK)
- JDK 6 Update 13
Choose Example Is Using x64 Binary - Adjust If Using x32:
- Linux x64
- jdk-6u13-linux-x64.bin *DO NOT choose RPM download*
Place jdk-6u13-linux-x64.bin in /tmp on the ZCS server.
Install JDK 1.6 as root. Executing the bin file prompt you to agree to a license agreement. The installer will create a directory named jdk1.6.0_13 and extract contents to that directory. This process does not interfere with ZCS.
cd /opt/zimbra /tmp/jdk-6u13-linux-x64.bin chown -R zimbra:zimbra jdk1.6.0_13
Next, stop ZCS services with zmcontrol, destroy the java symlink and symlink JDK 1.6 to java.
su - zimbra zmcontrol stop exit rm -rf java ln -s jdk1.6.0_13 java zmcontrol start
You can now run your zmdiaglog with greater efficiency. Alternatively, you can generate heap dumps manually with jmap (as zimbra this time).
su - zimbra -c '/opt/zimbra/java/bin/jmap -dump:live,file=/tmp/heapdump.live $(cat /opt/zimbra/log/zmmailboxd_java.pid)'
Rollback at any time by shutting down services with zmcontrol and pointing the java symlink back to jdk1.5.
zmdbintegrityreport
Resources:
Trouble Shooting Memory Issues
Basic Commands To Gather Data
PS
Try adjusting the ps output:
ps -eo pid,ppid,rss,vsize,pcpu,pmem,cmd -ww --sort=pid
- RSS (Resident Set Size) - The portion of a process that exists in physical memory (RAM). The rest of the program exists in swap. If the computer has not used swap, this number will be equal to VSIZE. RSS per unit time will show how much RAM the process is using over time.
- VSIZE (Virtual memory SIZE) - The amount of memory the process is currently using. This includes the amount in RAM and the amount in swap. VSIZE per unit time will show how large the process is over time.
Free
Using Free:
free -o
- [ -o ] Disables the display of a "buffer adjusted" line. If the -o option is not specified, free subtracts buffer memory from the used memory and adds it to the free memory reported.
- "Memory Used" per unit time will show the "Memory Used" asymptotically approach the total amount of memory in the system under heavy use. This is normal, as RAM unused is RAM wasted.
- "Memory Used - Memory Buffered - Memory Cached" per unit time will give a good sense of the memory use of your applications minus the effects of your operating system. As you start new applications, this value should go up. As you quit applications, this value should go down. If an application has a severe memory leak, this line will have a positive slope.
- "Swap Used" per unit time will display the swap usage. When the system is low on RAM, a program called kswapd will swap parts of process if they haven't been used for some time. If the amount of swap continues to climb at a steady rate, you may have a memory leak or you might need more RAM.
You could also use watch with free to poll it:
watch free
- or
watch -n 1 -d free
Top
Top is nice, put you want to save output. Do the [-b batch] and [-n number of iterations] variables:
top -b -n1 > /tmp/top-output.txt
One could add into cron if you want to get top output every 5 minutes like this for example [Would would need to confirm the directory structure and perms to write in it our correct]:
crontab -e ** add the following line
*/5 * * * * top -b -n1 > /opt/zimbra/support/data/tops/top-`date +\%F-\%H-\%M`.txt
This is a nice addition to include in the Ajcody-Logging#Like_To_Have_zmstat-chart_Data_Integrated_With_Zimbra setup. Where you would place the data underneath the root http directory.
Vmstat
Gives a good overview as well:
vmstat
- The process-related fields are:
- r — The number of runnable processes waiting for access to the CPU
- b — The number of processes in an uninterruptible sleep state
- w — The number of processes swapped out, but runnable
- The memory-related fields are:
- swpd — The amount of virtual memory used
- free — The amount of free memory
- buff — The amount of memory used for buffers
- cache — The amount of memory used as page cache
- The swap-related fields are:
- si — The amount of memory swapped in from disk
- so — The amount of memory swapped out to disk
- The I/O-related fields are:
- bi — Blocks sent to a block device
- bo— Blocks received from a block device
- The system-related fields are:
- in — The number of interrupts per second
- cs — The number of context switches per second
- The CPU-related fields are:
- us — The percentage of the time the CPU ran user-level code
- sy — The percentage of the time the CPU ran system-level code
- id — The percentage of the time the CPU was idle
Polling data with vmstat:
vmstat 1 10
- Displays one new line per second, but only for the next ten seconds
Meminfo
See below for description of the output for meminfo:
Featured Article: /proc/meminfo Explained
Do the following:
cat /proc/meminfo
Non-Zimbra Tools
Caution : None of this is supported by the Zimbra Support team. However it might prove useful, and depending on how much I can document these tools - who knows, maybe we can start using them more directly in the support process.
Debugging Tools
- Opensourcetesting.org - General Site with News and Tools
- MAT - Memory Analyzer Tool
- MAT w/ Eclipse
- http://www.eclipse.org/mat/
- Adding this to the startup settings of zimbra
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/opt/zimbra/log/dumps/
- dumps out of mem heap. That file you can then load into MAT.
- Adding this to the startup settings of zimbra
- http://www.eclipse.org/mat/
- MAT w/ Eclipse
- Hat - the Haskell Tracer
- Linux
/proc/sys/kernel/core_pattern
- By setting
/proc/sys/kernel/core_pattern
, one can run a user-level program before a process dies to gather additional debugging information. - A pipe is opened and attached to your script. The kernel writes the coredump to the pipe. Your script can read it and write it to storage.
- Your script can attach GDB, get a backtrace, run pstack, dump open file descriptors, etc.
- For detailed explanation + script: http://timetobleed.com/an-obscure-kernel-feature-to-get-more-info-about-dying-processes/
- By setting
Automate Testing / Load Testing
- Mstone - is a multi-protocol stress and performance measurement tool. Mstone can test multiple protocols (e.g. POP and SMTP) simultaneously and measures the performance of every transaction. The performance can be graphed throughout the duration of the test.
- Selenium - is a suite of tools to automate web app testing across many platforms.
- Postal - to benchmark SMTP servers.
Disk Full Issues
Actual Disk Full Issues Homepage
Please see Ajcody-Disk-Full-Issues
Read First - Zimbra Support Is Not Your Appropriate Support Contact For Storage Modifications
The information provided on this wiki page is provided for those that are comfortable doing the activity by themselves and will use proper testing and DR strategies prior to making changes on the production machine.
Modifying your disk/partitions/storage is outside of the supported responsibilities of the Zimbra Support team. Your proper vendor support contact for this activity is your OS, Virtualization, and Storage vendors and any other contractor or consultant resources you use.
The only exception to the above rule is for customers that are running the EOL [End of Life] ZCA product of ours. Those customers should be upgrading to the Network Edition of ZCS and after that upgrade, their support contacts for storage modifications would fall to Vmware or the OS vendor.
as root:
cat /etc/fstab cat /proc/mounts df -hT
Let's identify your larger directories , as root :
for i in `find /opt/zimbra -maxdepth 1 -type d`; \ do export sum=`find $i -printf %k"\n" | awk '{ sum += $1 } END { print sum kb }'`; \ echo -e "$sum kb\t$i"; export sum=; done | sort -rn | head -n 20 [example output below] 6007764 kb /opt/zimbra 1966620 kb /opt/zimbra/db 837160 kb /opt/zimbra/backup 680932 kb /opt/zimbra/jetty-distribution-7.6.12.v20130726 387140 kb /opt/zimbra/data 286160 kb /opt/zimbra/jdk-1.7.0_45 211080 kb /opt/zimbra/store 207172 kb /opt/zimbra/zmstat 178628 kb /opt/zimbra/logger 162280 kb /opt/zimbra/mta 155700 kb /opt/zimbra/bdb-5.2.36 116520 kb /opt/zimbra/aspell-0.60.6.1 98820 kb /opt/zimbra/mysql-standard-5.5.32-pc-linux-gnu-i686-glibc23 79408 kb /opt/zimbra/zimbramon 72608 kb /opt/zimbra/lib 66940 kb /opt/zimbra/keyview-10.13.0.0 66488 kb /opt/zimbra/clamav-0.97.8 64676 kb /opt/zimbra/httpd-2.4.4 47408 kb /opt/zimbra/store2 47164 kb /opt/zimbra/index
If you have stats for the server, this will give us trending data:
[zimbra@]$ tar cvf /tmp/df.tar `find /opt/zimbra/zmstat -name df.csv\* -print | sort -r | head -n 20` tar: Removing leading `/' from member names /opt/zimbra/zmstat/df.csv /opt/zimbra/zmstat/2014-04-06/df.csv.gz /opt/zimbra/zmstat/2014-04-05/df.csv.gz /opt/zimbra/zmstat/2014-04-04/df.csv.gz /opt/zimbra/zmstat/2014-04-03/df.csv.gz /opt/zimbra/zmstat/2014-04-02/df.csv.gz /opt/zimbra/zmstat/2014-04-01/df.csv.gz /opt/zimbra/zmstat/2014-03-31/df.csv.gz /opt/zimbra/zmstat/2014-03-30/df.csv.gz /opt/zimbra/zmstat/2014-03-29/df.csv.gz /opt/zimbra/zmstat/2014-03-28/df.csv.gz /opt/zimbra/zmstat/2014-03-27/df.csv.gz /opt/zimbra/zmstat/2014-03-26/df.csv.gz /opt/zimbra/zmstat/2014-03-25/df.csv.gz /opt/zimbra/zmstat/2014-03-24/df.csv.gz /opt/zimbra/zmstat/2014-03-23/df.csv.gz /opt/zimbra/zmstat/2014-03-22/df.csv.gz /opt/zimbra/zmstat/2014-03-21/df.csv.gz /opt/zimbra/zmstat/2014-03-20/df.csv.gz /opt/zimbra/zmstat/2014-03-19/df.csv.gz [zimbra@zcs806 ~]$ ls -ls /tmp/df.tar 80 -rw-r----- 1 zimbra zimbra 81920 Apr 7 11:40 /tmp/df.tar
As the zimbra user, Note - If your zimbraBackupTarget variable uses something different than /opt/zimbra/backup then ALSO do the three ls commands below with that path :
su - zimbra zmprov -l gs `zmhostname` | egrep 'Back|Redo' du -sh /opt/zimbra/redolog ls -latr /opt/zimbra/backup ls -latr /opt/zimbra/backup/tmp ls -latr /opt/zimbra/backup/sessions crontab -l | grep -i back zmbackupquery
and then with a user account, replacing user@domain.com below with a valid account :
zmprov ga user@domain.com |grep -i Lifetime
you can also do these with the COS you use:
zmprov gac
and then:
zmprov -l gc [cos name] | grep Lifetime
You probably are only using the default COS, so:
zmprov -l gc default | grep Lifetime
Notable Bugs In ZCS That Cause Unnecessary Disk Growth Or Consumption
Large /opt/zimbra/logger/db/data/rrds Directory
- This bug was reported in 8.0.5 and is slated to be resolved with 8.0.7
- zmlogger causes extreme rrd file growth
Large /opt/zimbra/data/amavisd/.spamassassin Directory
You find you have a large /opt/zimbra/data/amavisd/.spamassassin directory because bayes_toks.expire* files are not being purged via the cronjob. To check your crontab :
su - zimbra crontab -l | grep sa-learn
You should have something like this in your crontab [It's all one line below if you want to manually run it from the CLI as the zimbra user]:
/opt/zimbra/libexec/sa-learn -p /opt/zimbra/conf/salocal.cf --dbpath \ /opt/zimbra/data/amavisd/.spamassassin --siteconfigpath \ /opt/zimbra/conf/spamassassin --force-expire --sync > /dev/null 2>&1
That should be cleaning up those files. If not, and you have anti-spam off, I would recommend moving them to a temp location or compressing them [just in case]. Give it a night and if nothing is amiss, then remove them from your filesystem.
Large /opt/zimbra/data/ldap/mdb/db/ Directory Because Of The data.mdb File
If your data.mdb file actually is consuming up GB's of space and isn't no longer a sparse file, you most likely did a move, cp, or rsync of this data/directory improperly. With ZCS 8, this is a sparse file and has to be treated differently.
For example:
[root@zcs806 db]# pwd /opt/zimbra/data/ldap/mdb/db [root@zcs806 db]# ls -lh total 1.5M -rw------- 1 zimbra zimbra 15G Apr 7 10:10 data.mdb -rw------- 1 zimbra zimbra 8.0K Apr 7 13:03 lock.mdb [root@zcs806 db]# du -c -h data.mdb 1.5M data.mdb 1.5M total
Notice that the data.mdb file says 15G on the ls but is actually only 1.5M in size with the du output.
Reference:
- About the changes to data.mdb in ZCS 8, see:
- To correct a problem like this where it's no longer a sparse file, see the following:
Very Fast Growing zimbra.log And mail.* in /var/log Directory
If you find, especially after an upgrade, that zimbra.log and the mail.* logs in /var/log are growing in size extremely fast please check your syslog/rsyslog configuration files.
- Endless loop of logging from rsyslog 60-zimbra.conf
Dumpster Issues
Confirm if you have dumpster enabled and then if it's actually purging messages like it should, see the following:
Adding A New Primary Store Volume How-To
Your existing primary volume is using the default path of /opt/zimbra/store . You'll create and mount with a new ext3/4 partition , for example, to /opt/zimbra/store2 . Make sure it's properly placed also in the /etc/fstab - at the end of the file probably will work just fine.
[as root]
mkdir /opt/zimbra/store2 chown zimbra:zimbra /opt/zimbra/store2 chmod 755 /opt/zimbra/store2 mount /opt/zimbra/store2 [Now confirm a write/delete test] su - zimbra touch /opt/zimbra/store2/testfile rm /opt/zimbra/store2/testfile
You can then use the admin console to add the new zimbra message volume for /opt/zimbra/store2 . Assuming you set it to be the active one, the transition to now use that volume for new blobs will be immediate. The old blobs will stay where they are [ /opt/zimbra/store ] . HSM or a secondary volume is different in that, it runs a job that you setup in the crontab that tells it to move messages X days/weeks/months/etc old for all accounts from the primary message volumes into it's own volume path.
You can monitor the /opt/zimbra/store2/ directory and you'll see sub-directories being made as the new blobs/messages come in. Please note, the sub-directories will not have 751 perms. They will be like drwxr-x--- [750] .
- References
- Using zmvolume from the command line
Adding A New HSM Volume How-To - 1 Total
If this is your first time using HSM , please review the complete table of contents at Ajcody-HSM-Notes .
See the following:
Adding A Second HSM Volume How-To - Having One Active HSM and on Inactive HSM Volume - 2 Total
You currently have a HSM volume but it's getting close to being full.
/dev/sdc1 2.0T 1.6T 292G 85% /opt/zimbra/hsm
Let's say you create a new partition for HSM and mount it as /opt/zimbra/hsm2 . You'll need to have it owned by zimbra [ chown zimbra:zimbra /opt/zimbra/hsm2 ]
You would then:
su - zimbra zmhsm -u [Confirm hsm is not currently running] crontab -e [Comment out the hsm run in cron if that's how you have it setup] [We don't want hsm running during the change] zmvolume -a -n hsm2-volume -t secondaryMessage -p /opt/zimbra/hsm2 [You might want to adjust -n hsm2-volume depending on how you named the other hsm volume] [ -p /opt/zimbra/hsm2 is the path for the new volume, adjust if needed] zmvolume -l [Get the volume id of the new volume] [You should see that the new volume isn't listed as current, the old one still is] zmvolume -sc -id ## [Replace ## with the volume id for the hsm2 volume] [This will set the new hsm volume to be the current one, msg's will go there on the next hsm run] [There can only be one "current" volume for each type of volume type {index, primary, secondary] zmvolume -l [confirm the volumes] crontab -e [Uncomment the hsm job in cron if that's how you ran it.]
- See the following for zmvolume options:
- Other references:
- http://wiki.zimbra.com/wiki/Ajcody-HSM-Notes#Create_The_HSM_Volume
- 1.3.1.1 Create The HSM Volume
- 1.3.1.2 Set HSM Volume To Current
- 1.3.1.3 Starting HSM For First Time
- http://wiki.zimbra.com/wiki/Ajcody-Server-Misc-Topics#Volumes_.26_zmvolume
- 1.9 Volumes & zmvolume
- 1.9.1 Basic Concepts
- 1.9.2 Notable RFEs
- 1.9.3 How To Move A User's Data To Another Volume
- 1.9.3.1 Using zmsoap Example
- 1.9.4 How To Go About Changing Volume Paths
- 1.9.4.1 To Modify Volume From CLI After Data Move
- 1.9 Volumes & zmvolume
- http://wiki.zimbra.com/wiki/Ajcody-HSM-Notes#Create_The_HSM_Volume
Adding Additional Storage - Vmware Virtual Machine Example
See the following:
Growing The vmdk Disk And Expanding The LVM Filesystem
See the following references and contact Vmware Support for additional help [Note - doing a snapshot of the vm prior to this seems wise] :
- http://v-reality.info/2010/06/working-with-linux-volumes-n-vsphere/
- http://www.rootusers.com/how-to-increase-the-size-of-a-linux-lvm-by-expanding-the-virtual-machine-disk/
Moving Zimbra To New Partitions For the /opt/zimbra And backups Directory - Vmware Virtual Machine Example
See the following:
ZCA - Zimbra Appliance - Manually Adding And Expanding The Disk Partitions
Note - Ideally the link below should work for you and be the best method to adding/increasing disk space on ZCA
See the following also for a more manual way if needed:
Checking Your Dumpster Settings And Purging
If the user empties trash, those are deleted from trash. If you have dumpster enabled though, they will still reside on the file system until it hit the DumpsterLifetime of 30 days. See Ajcody-Server-Misc-Topics#Dumpster_Specific for more details on dumpster variables and how they work.
To see if you have dumpster enabled, you have to check your COS. This example below is against the COS named default:
su - zimbra zmprov getCos default|grep -i dumpster zimbraDumpsterEnabled: TRUE # it must be TRUE value zimbraMailDumpsterLifetime: 30d #
Notice the Enabled flag.
The below command should empty the dumpster data regardless of the lifetime variable:
zmmailbox -z -A -m user@domain.com emptyDumpster
- Reference
- Some default values
- zimbraMailDumpsterLifetime: 30d
- zimbraMailMessageLifetime: 0
- zimbraMailSpamLifetime: 30d
- zimbraMailTrashLifetime: 30d
- Some default values
Ajcody Client Topics
Actual Client Topics Homepage
Please see Ajcody-Client-Topics
Official End User Guides And How-To's
Please see the following:
http://www.zimbra.com/community/end_user_guide_and_how_to.html
Bugs-RFE's
- "Support user creation and moving of nested calendars" subfolders , sub-folders
- http://bugzilla.zimbra.com/show_bug.cgi?id=27211
- "Appointments created in shared sub-calendar does not show in ZWC"
- http://bugzilla.zimbra.com/show_bug.cgi?id=27211
Small Screen Resolution Issues - 800x600
Please see the following:
- "Support for 800x600 Screen Resolution on Web Client"
- "Horizontal scroll bar is present when screen resolution is 800*600"
- "UI issues in HTML client when screen resolution is 800*600"
Section 508 , VPAT, ADA, And Accessibility
New
- VMware Collaboration Product VPATs For Zimbra
- 2014 update - dead link now that Zimbra is no longer a part of Vmware.
- Zimbra Support staff - please see the following private bug comment:
About section 508 and Accessibility:
- US Federal Agency Home Page For "accessibility for people with disabilities".
- IAccessible2 Overview
- Voluntary Product Accessibility Template
Various Zimbra RFE's related to topic:
- "Ability to adjust transparency of Calendar view"
- "508 and Accessibility issues with HTML client"
- http://bugzilla.zimbra.com/show_bug.cgi?id=28516
- This one is marked private, but it just referenced the materials within Vmware - which no longer applies.
- "AJAX accessibility - section 508"
- "Accessibility And Section 508 Documentation"
- http://bugzilla.zimbra.com/show_bug.cgi?id=45706
- BLYNX: Lynx Support Files Tailored for Blind and Visually Handicapped Users, see rfe comment:
- "I18N - "Translate" and tooltips "Translate currently opened email's " & "Converts the selected message's text to speech" not translated"
- http://bugzilla.zimbra.com/show_bug.cgi?id=49720
- Mentions the use of Google Language with a zimlet. Note though, Google language also offers text-to-speech.
- "RFE: text-to-speech and speech-to-text"
- http://bugzilla.zimbra.com/show_bug.cgi?id=52174
- Experimental zimlet for text-to-speech, see comment in rfe:
- http://bugzilla.zimbra.com/show_bug.cgi?id=52174
Zimbra Web Client
Setting ZWC As Default Mail Client - Mailto
Seeing What Is Currently Set For Mailto Handler In Various Browsers
- Firefox: Seeing what mailto is currently configured as:
- In url field , about:config and hit enter.
- Search for mailto
- In url field , about:config and hit enter.
- Chrome: Seeing what mailto is currently configured as:
- In url field , chrome://settings/handlers and hit enter.
- Will pop up a box, if empty, then nothing is set.
ZCS 8 And RFE To Set ZWC As Default Mail Client
Per the following RFE, with ZCS 8 a user should be prompted upon first login if they want to set ZWC as their default mail client.
- "Cannot Make Zimbra Web Client the Default "mailto" Client"
Most Complete Solution For Windows - Full Mailto & MAPI Calls Mapping via Zimbra Notifier
First, though it's unsupported directly by the Zimbra Support team, most users will prefer to use this because of its full functionality:
- Zimbra Notifier
The other option is setting up a registry key within windows.
Windows Mailto To Use ZWC Advance Ajax Client or Basic HTML Client Interface
Note, the Windows examples below use the url string to use the ZWC Advanced Ajax client, if you want to use the ZWC Basic HTML one you'll use a string like the following instead :
https://mail.server.com/h/search?action=compose&to=%1
Windows 7 Mailto Registry Key
Save the following to a file ending with a name of .reg, and modify the following items below to suit the user. Save the file after you made the changes. Run the registry editor [Start > Run > regedit ] and do a File > Import - selecting the file you had saved.
Windows Registry Editor Version 5.00 [HKEY_CURRENT_USER\SOFTWARE\Classes\mailto] "URL Protocol"="" [HKEY_CURRENT_USER\SOFTWARE\Classes\mailto\shell] [HKEY_CURRENT_USER\SOFTWARE\Classes\mailto\shell\open] [HKEY_CURRENT_USER\SOFTWARE\Classes\mailto\shell\open\command] @="C:\\Path\\to\\Program.exe https://mail.server.com?username=user@example.com&view=compose&to=%1"
- Change the C:\\Path\\to\\Program.exe to the full path of browser executable to be used or to "rundll32.exe url.dll,FileProtocolHandler" [This will use the default registered browser of the user] .
- Note - When importing a reg key, you have to use \\ in the path rather than just one\ . Once imported, the key value will only show one \ per directory.
- Firefox example
@="C:\\Program Files (x86)\\Mozilla Firefox\\firefox.exe https://mail.domain.com?username=user@domain.com&view=compose&to=%1"
- Chrome example
@="C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe https://mail.domain.com?username=user@domain.com&view=compose&to=%1"
- IE example
@="C:\\Program Files (x86)\\Internet Explorer\\iexplore.exe https://mail.domain.com?username=user@domain.com&view=compose&to=%1"
- Change the mail server name and the user name :
- mail.server.com?username=user@example.com
- for example mycompany.com?username=john@mycompany.com
Windows Xp And Windows 98 MailTo Setup
See the following Microsoft KB :
You'll need to adjust the "Application used to perform action:" to be like one of the examples below:
- Firefox example
- "C:\Program Files (x86)\Mozilla Firefox\firefox.exe" "https://mail.domain.com?username=user@domain.com&view=compose&to=%1"
- Chrome example
- "C:\Program Files (x86)\Google\\Chrome\Application\chrome.exe" "https://mail.domain.com?username=user@domain.com&view=compose&to=%1"
- IE example
- "C:\Program Files (x86)\Internet Explorer\iexplore.exe" "https://mail.domain.com?username=user@domain.com&view=compose&to=%1"
Conversation Views
Here's an RFE I made you might want to check out:
- "RFE: Converstation View - Cos/User Customization"
Read & Delivery Confirmation Receipt
Please see RFE:
- "Support read receipts and delivery reports in webmail"
Web Client Search
The webclient offers a powerful search tool, the problem is knowing how to use it.
Please see this to learn the proper syntax's to get your searches right:
[Web Client Search - Power tip]
Is some particular state [ unread, read, flagged, unflagged, sent, draft, received, replied, unreplied, forwarded, unforwarded, anywhere [show messages anywhere], local, sent, solo [haven't replied, without a conversation] ]:
is:unread
In A Folder (Inbox):
In:Inbox
Unread and in Inbox:
In:Inbox is:uread
To someone:
to:user@domain.com
From someone:
from:user@domain.com
CC'd reference:
cc:user@domain.com
Subject reference:
subject:word-string
Has something [attachment , phone , url]
has:attachement
A particular 'something':
attachment:word
Size of mesage [{larger,smaller} : ## "b","kb","mb"]
larger:12kb
By tag:
tar:MyStuff
Email To Attachment Or Saved To Desktop
Send Email As Attachment
You can do a "forward" on an email and then in the new window, select "Options". This drop down will give you the options to "Include Orginal As Attachment".
Attachments Saved To Desktop And Other Options
Some RFE's are more expansive functionality:
- "save email message from web client to desktop"
- "RFE: expand the Add Attachment picker to attach emails"
- "When forward an email as attachement, the addresses in the forwarded email are not "usable"
- "Save email attachement to a Notebook or page in Documents"
A non-support zimlet that could be used:
- http://www.zimbra.com/forums/zimlets/16789-zimlet-save-email-txt-file.html
- http://gallery.zimbra.com/gallery.php?act=viewProd&productId=73
Date Format date_zimlet Issues
Most likely, your issues are addressed in the following:
- "Date formats not localized"
- "All date format should be supported for date zimlet"
- "RFE: date_zimlet - general improvement for functionality, customization and support of"
General CALDAV Issues
Only Seeing 1 Month Back and 1 Year Ahead In CalDAV
Please see this bug:
http://bugzilla.zimbra.com/show_bug.cgi?id=28713
Lightning & Thunderbird
News Bugs/RFE's To Track That Might Update Situation As Described Below
- "Calendar not configurable via Caldav/Lightening"
- "Lightning can not write to calendar."
- "lightning plugin for thunderbird no longer displays free busy w/5.0.18"
- "CalDAV: Events deleted on Lightning aren't deteted on the server"
- "Conduit for Thunderbird"
Lightning does not support Caldav + Free/Busy
Free/Busy support is only available for the Sun Cal Server.
Maybe we could "fake" this out within our Ldap configuration. Like RFE for Apple OD:
http://bugzilla.zimbra.com/show_bug.cgi?id=26619
Address/username lookup doesn't occur within a new appointment
Unfortunately, I've not figured a way around this at this time.
Zimbra CalDav doesn't allow user to dismiss alarms from lightning
Please see bug from Mozilla:
https://bugzilla.mozilla.org/show_bug.cgi?id=432540
Ajcody Apple Mac Issues
Actual Apple/Mac Issues Homepage
Please see Ajcody-Apple-Mac-Issues
App To Help Enable Debugging Options
You might want to check out this app:
Outlook 2011 For Mac And EWS Setup
Follow these steps to set up Outlook to sync using Exchange Web Services:
- Open Outlook.
- Open the Outlook menu and select Preferences.
- Click the Accounts icon under the Person Settings section.
- Click the Exchange icon which will open a new window.
- Enter your email address in the E-mail address field.
- Set the Method to "User Name and Password."
- Enter your full email address (e.g. user@example.com) as the username.
- Enter your password in the password field.
- Uncheck the "Configure automatically" check-box.
- In the Server field, enter Zimbra Collaboration Server's hostname - (e.g. mail.example.com)
- Click Add Account.
Note, if you end up customizing the other elements be aware of the following:
- ZimbraEWS uses BASIC Auth for validating the user request.
With Screenshots
Please see Ajcody-Outlook_2011_For_Mac_And_EWS_Setup#Outlook_2011_For_Mac_And_EWS_Setup_With_Screenshots
Contact Related Items
EWS Configuration And ZCS 8.5+
Example is from OS X 10.9.5 . The ZCS server must be licensed for EWS and your account needs to have the EWS feature enabled. This can be enabled in the admin console either checking the box for EWS under Features in the COS that is set for the account or under the individuals user configuration for Features. It can also be set on the CLI - zimbraFeatureEwsEnabled TRUE [COS or user].
Note - ZCS 8.5 targeted EWS support ONLY with Outlook for Mac's. There was no testing or expectation that the native mac apps would work with the EWS configuration type.
- Launch Contacts.app
- Click on "Contacts" from the menu screen at the top of your screen.
- Select "Add Account.."
- Select "Exchange" and then "Continue"
- Name : your full name
- Email Address : user@domain
- Password : your password
- Once filled out, click "Continue" . You'll need move onto another configuration screen.
- Description : I would recommend filling this out to avoid any confusion from other accounts you might setup.
- Server Address : the FQDN/hostname of your ZCS server. You can also use an ip address here.
- Once filled out, click "Continue" .
- You should now see a summary of your configuration, click "Continue".
- You'll now be asked to select what apps to use, your choices will be:
- Contacts
- Calendar
- Notes
- Reminders
- Once you've checked the ones you want, click "Done".
This will create two resources within Contacts.app. You'll see a header called "Exchange" with your account contact folders under it [ZCS defaults will be - Contacts, Emailed Contacts] and also a line that says "All Exchange". There will also be a section now called "Directories", listed under it will be "All Directories" and "Exchange Global Address List".
Verify Certificate Prompts
If the ZCS server isn't using commericial certificates, you'll end up getting a pop up window on your Mac called "Verify Certificate". To get rid of this pop up window, you'll want to do the following.
Click on the "Show Certificate" button. You'll then have an option to check a box for "Always trust ....". Then click on the "Continue" button.
Log Event Showing EWS
Possible log files to check for in /opt/zimbra/log/ are below with some examples from my logging into Contacts.App and creating a new contact. You can increase the logging details if needed , please see Using_log4j_to_Configure_mailboxd_Logging#Predefined_log4j_Categories_in_ZCS :
- access_log.[date]
192.168.1.73 - - [10/Oct/2014:16:01:30 +0000] "POST /EWS/Exchange.asmx HTTP/1.1" 200 907 "-" "Mac OS X/10.9.5 (13F34); ExchangeWebServices/4.0 (193); Contacts/8.0 (1371.2)" 16 192.168.1.73 - - [10/Oct/2014:16:05:55 +0000] "GET /home/user1@mail2.zimbra.DOMAIN.com/Contacts?fmt=cf&t=2&all=all HTTP/1.1" 200 - "https://192.168.1.172/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36" 42 192.168.1.73 - - [10/Oct/2014:16:28:11 +0000] "POST /EWS/Exchange.asmx HTTP/1.1" 401 0 "-" "Mac OS X/10.9.5 (13F34); ExchangeWebServices/4.0 (193); Contacts/8.0 (1371.2)" 7
- audit.log
2014-10-10 12:01:28,976 INFO [qtp509886383-107:https://192.168.1.172:443/EWS/Exchange.asmx] [] security - cmd=Auth; account=user1@mail2.zimbra.DOMAIN.com; protocol=zsync; 2014-10-10 12:05:50,227 INFO [qtp509886383-113:http://127.0.0.1:80/service/soap/AuthRequest] [name=user1@mail2.zimbra.DOMAIN.com;oip=192.168.1.73;ua=zclient/8.5.0_GA_3042;] security - cmd=Auth; account=user1@mail2.zimbra.DOMAIN.com; protocol=soap;
- ews.log
2014-10-10 12:28:11,361 INFO [qtp509886383-117:https://192.168.1.172:443/EWS/Exchange.asmx] [] ews - Start syncFolderHierarchy 2014-10-10 12:28:11,371 INFO [qtp509886383-117:https://192.168.1.172:443/EWS/Exchange.asmx] [] ews - User: user1 has been successfully authorized. 2014-10-10 12:28:11,371 INFO [qtp509886383-117:https://192.168.1.172:443/EWS/Exchange.asmx] [name=user1@mail2.zimbra.DOMAIN.com;ip=192.168.1.73;ua=MacOSX/(F)ExchangeWebServices/()Contacts/();] ews - The account: 7277b33a-c7ee-4b03-bb4b-f24814c7cea3is true 2014-10-10 12:28:11,371 INFO [qtp509886383-117:https://192.168.1.172:443/EWS/Exchange.asmx] [name=user1@mail2.zimbra.DOMAIN.com;ip=192.168.1.73;ua=MacOSX/(F)ExchangeWebServices/()Contacts/();] ews - Folder Id was not provided with request, defaulting to 1 2014-10-10 12:28:11,427 INFO [qtp509886383-117:https://192.168.1.172:443/EWS/Exchange.asmx] [name=user1@mail2.zimbra.DOMAIN.com;ip=192.168.1.73;ua=MacOSX/(F)ExchangeWebServices/()Contacts/();] ews - End syncFolderHierarchy 2014-10-10 12:28:11,449 INFO [qtp509886383-118:https://192.168.1.172:443/EWS/Exchange.asmx] [] ews - Start createItem version 1.0.01 2014-10-10 12:28:11,450 INFO [qtp509886383-118:https://192.168.1.172:443/EWS/Exchange.asmx] [] ews - Start: createItem 2014-10-10 12:28:11,457 INFO [qtp509886383-118:https://192.168.1.172:443/EWS/Exchange.asmx] [] ews - User: user1 has been successfully authorized. 2014-10-10 12:28:11,457 INFO [qtp509886383-118:https://192.168.1.172:443/EWS/Exchange.asmx] [name=user1@mail2.zimbra.DOMAIN.com;ip=192.168.1.73;ua=MacOSX/(F)ExchangeWebServices/()Contacts/();] ews - The account: 7277b33a-c7ee-4b03-bb4b-f24814c7cea3is true 2014-10-10 12:28:11,580 INFO [qtp509886383-118:https://192.168.1.172:443/EWS/Exchange.asmx] [name=user1@mail2.zimbra.DOMAIN.com;ip=192.168.1.73;ua=MacOSX/(F)ExchangeWebServices/()Contacts/();] ews - End: createItem 2014-10-10 12:28:11,580 INFO [qtp509886383-118:https://192.168.1.172:443/EWS/Exchange.asmx] [name=user1@mail2.zimbra.DOMAIN.com;ip=192.168.1.73;ua=MacOSX/(F)ExchangeWebServices/()Contacts/();] ews - End createItem
- mailbox.log
2014-10-10 12:28:11,522 INFO [qtp509886383-118:https://192.168.1.172:443/EWS/Exchange.asmx] [name=user1@mail2.zimbra.DOMAIN.com;ip=192.168.1.73;ua=MacOSX/(F)ExchangeWebServices/()Contacts/();] mailop - adding contact test2@domain.com: id=269, folderId=7, folderName=Contacts.
- trace_log.[date]
12:28:11.323:qtp509886383-117:https://192.168.1.172:443/EWS/Exchange.asmx REQUEST 192.168.1.73 POST null; Mac OS X/10.9.5 (13F34); ExchangeWebServices/4.0 (193); Contacts/8.0 (1371.2) 12:28:11.330:qtp509886383-117:https://192.168.1.172:443/EWS/Exchange.asmx RESPONSE 401 text/xml; charset=UTF-8 12:28:11.354:qtp509886383-117:https://192.168.1.172:443/EWS/Exchange.asmx REQUEST 192.168.1.73 POST null; Mac OS X/10.9.5 (13F34); ExchangeWebServices/4.0 (193); Contacts/8.0 (1371.2) 12:28:11.429:qtp509886383-117:https://192.168.1.172:443/EWS/Exchange.asmx RESPONSE 200 text/xml; charset=UTF-8 12:28:11.446:qtp509886383-118:https://192.168.1.172:443/EWS/Exchange.asmx REQUEST 192.168.1.73 POST null; Mac OS X/10.9.5 (13F34); ExchangeWebServices/4.0 (193); Contacts/8.0 (1371.2) 12:28:11.581:qtp509886383-118:https://192.168.1.172:443/EWS/Exchange.asmx RESPONSE 200 text/xml; charset=UTF-8
AddressBook
Logging-Debug for Addressbook
First shutdown Addressbook app.
Launch terminal app.
defaults write com.apple.addressbook ABShowDebugMenu -bool true
Turn it off by just reversing it:
defaults write com.apple.addressbook ABShowDebugMenu -bool false
CardDAV
Contact Groups
- Contact groups do not sync via CardDAV
- https://bugzilla.zimbra.com/show_bug.cgi?id=46359#c30
- Resolved for ZCS 8+
10.6 - Snow Leopard Notes
Basic Setup Notes
Some notes I have against testing CardDAV I shared on a case. The following was done with CardDAV [OSX 10.6.4] & Addressbook [Version 5.0.2 (870)] .
Primary references I found related to :
- CardDAV fails in 6.0.8 for Evolution and Mac Address Book
- CardDAV do not work on the non-default domains in Zimbra
- WebDAV: Add support for CardDav
To setup, launch Addressbook.
- Addressbook > Preferences
- Accounts > click on + sign to add new account
- Account type = CardDav
- User Name = user
- bug/rfe's mentions the use of user@domain vs. just user will most likely cause Addressbook to attempt a DNS SRV record lookup.
- Password = accounts password
- Server Address = http://servername:80 or https://servername:443
- might need to adjust that for proxy setup or if you have load balancers/F5's/etc..
- Accounts > click on + sign to add new account
Additional Details And Troubleshooting Steps - Proxy Issues Most Likely
I was able to get my carddav account to work using my virtual hostname that points to my two servers that run ldap/proxy/mta . I have them setup in DNS as well.
$ host rr608.zimbra.DOMAIN.com rr608.zimbra.DOMAIN.com has address 192.168.0.42 rr608.zimbra.DOMAIN.com has address 192.168.0.41 rr608.zimbra.DOMAIN.com mail is handled by 10 rr608.zimbra.DOMAIN.com.
One thing I did hit though, was the way OSX handles it's odd resolution stuff. For example, initially it failed when I had just modified the /etc/resolv.conf to use my internal DNS. Doing :
host rr608.zimbra.DOMAIN.com
resolved properly but doing:
ping rr608.zimbra.DOMAIN.com
was resolved to the external ip and was being hampered by my firewall. Once host and ping were resolving correctly, the carddav account work.
The whole reason I mention this is that I think we should walk our way back from the mailstore hostname in the configuration.
First, setup a carddav account using the mailstore name and confirm it works and then close/exit the Address Book. Then do the following. In a terminal, be in the users home directory and cd to this subdirectory - for example:
$ pwd /Users/[username]/Library/Application Support/AddressBook/Sources/
Do a ls there and find the most recent directory that was made and cd into it, for example:
$ ls -latr total 0 drwx------ 7 ajcody staff 238 Nov 17 06:50 000C3E7F-3DE5-40C8-8820-AF96214D6AF0 drwx------ 4 ajcody staff 136 Nov 17 06:58 . drwx------ 6 ajcody staff 204 Nov 17 07:07 56892F5B-9F55-437E-9890-A8489A63EDAA drwx------@ 8 ajcody staff 272 Nov 17 07:07 .. Adams-MacBook-Pro:Sources username$ cd 56892F5B-9F55-437E-9890-A8489A63EDAA/ Adams-MacBook-Pro:56892F5B-9F55-437E-9890-A8489A63EDAA username$ ls AddressBook-v22.abcddb Configuration.plist Metadata SyncLockFile
We can now manually edit the Configuration.plist file with various test hostnames/ip addresses.
The two keys we're interested in will be like this:
<key>name</key> <string>ajcody@rr608.zimbra.DOMAIN.com</string> <key>servername</key> <string>http://rr608.zimbra.DOMAIN.com:80/principals/users/ajcody@rr608.zimbra.DOMAIN.com</string>
What I'd like you to attempt is adjusting the line for the servername using a hostname/ip address that will bypass the F5 if possible. You'll modify the file and save it. Launch Address Book and confirm/deny if the new hostname/ip address works.
- Use a static ip address of one of the zcs proxy servers. Please confirm it is pingable from the machine first.
- Use the hostname of the actual host of the same proxy server.
I'm assuming you can't use [your mailstore hostname] without going first to the F5, if the test above work you might want to setup a test BIND server you could use with the mac machine to resolve the [your mailstore hostname] using the various hostnames/ip's of the zcs proxy servers.
Use LDAP directory from ZCS for Mac Addressbook / Mail.App / iCal for Mac 10.4
Setting Up LDAP With Authentication - ZCS 8+
As an example, you would fill out the setup fields in the client to be like :
Username: uid=joe,ou=people,dc=abccompany,dc=com password: _zimbra_password_ Authentication Type: simple
Changing uid=joe to match the users email address [joe@abccompany] . You would also adjust the dc=abccompany above to match the users domain name in their email address.
To see and test what type of returns this setup will give, you can do the following from the ZCS server's CLI:
ldapsearch -x -H ldap://ldap.abccompany.com:389 -D uid=joe,ou=people,dc=abccompany,dc=com -w supp0rt -LLL -b 'ou=people,dc=abccompany,dc=com'
You would adjust the uid= , dc= , strings to match the user@domain for the user and also adjust "-w supp0rt" to be the user's password.
Older Reference Prior To ZCS 8
This is generally for those that aren't using Z-iSync.
First, test on shell that you have ldap connectivity.
ldapsearch -x -H ldap://ZCSserver-name:389 "uid=ajcody"
That returned successfully.
Then Addressbook Preferences
LDAP + (for new) Name: anything really Server: ip/hostname that was used successfully in the ldapsearch string Search Base: [leave blank] Port: 389 Scope: Subtree No auth needed * Save
I have no idea what the "Auto Update LDAP Cards" does.
Then setup Mail.App for IMAP to the Zimbra server. Check preferences to use LDAP - it will use the Address Book configuration that you did prior and it should auto-resolve to a username when you do a new message.
iCal will only use address that are in the LOCAL addressbook when you use the attendee field for a new event.
One way around this is to do a wildcard search in the Addressbook and drag those items to the local addressbook.
Goto to Address Book, selected Directories my ldap server and put in the below for the search field:
*.
This search result can then be dragged to the "All" folder in the Group column. It creates a "Last Import" item in the Group column now - it might warn about updating entries that all ready match.
Entourage And Calendars
Can't Sync Calendars Between Entourage and Zimbra
Bug/RFE filed to correct current public documents on issue:
- "Clean up references to having general support for Entourage and Calendars"
Bug/RFE filed to include comment in Release Notes:
- "Document 10.5 cannot sync entourage to zcs"
Summary of the situation:
Update [05/2010] : Many of the bugs/rfe's filed against iSync will now reference this:
- We are EOL'ing support for iSync in ZCS 7.0. See bug 42448 for more details,
- but support of CalDav and CardDav with MacOSX 1.6+ for ZSC 6.0.x+ will be the
- solution going forward.
- MacOS 10.4
- MacOS 10.4 still has the iSync/Entourage integration. Entourage integration worked via iSync rather than CalDAV (10.4 can't use CalDAV, only iSync). It would create a local calendar called Entourage within iCal and then the iSync service (via Zimbra Connector) would post updates to the users Zimbra calendar called Calendar (the default). I believe it would also check the box in Entourage under Preferences > Sync Services > Synchronize events and tasks with iCal and .Mac
- MacOS 10.5
- Apple's SyncServices issues make iSync unreliable and buggy so we (developers) now favor CalDAV.
- See bug 27380 "Leopard - CalDav only"
- Pre-5.0.6 Zimbra Connector
- The connectors before 5.0.6 release allowed you to select iSync rather than CalDav on MacOS 10.5.
- For the brave, you could always use connectors before the 5.0.6 release (on MacOS 10.5) . Since this would end in an unsupported situation [support staff and developers would tell a customer in a "support case" to upgrade to a new release] it would be best to encourage your users to keep a very clean and simple calendar as to avoid needed support. This might buy some time until the bugs mentioned are resolved.
- The connectors before 5.0.6 release allowed you to select iSync rather than CalDav on MacOS 10.5.
- 5.0.6+ Zimbra Connectors
- 5.0.6+ Zimbra connectors on MacOS 10.5 will only be able to use CalDAV and iCal for calendaring features. You will not have the option to use iSync for calendars and therefor the Entourage integration is lost. You can't even drag/drop items to/from Entourage and iCal.
- Entourage doesn't support CalDAV, only WebDAV.
- See RFE bug 24502 "Support for Entourage WebDav"
- Enable snow leopard messaging features, Outlook for Mac 2010
- http://bugzilla.zimbra.com/show_bug.cgi?id=38631
- This was marked as a dup of the above:
- Support for Outlook for Mac 2010
- http://bugzilla.zimbra.com/show_bug.cgi?id=41106
- 5.0.6+ Zimbra connectors on MacOS 10.5 will only be able to use CalDAV and iCal for calendaring features. You will not have the option to use iSync for calendars and therefor the Entourage integration is lost. You can't even drag/drop items to/from Entourage and iCal.
- Apple's SyncServices issues make iSync unreliable and buggy so we (developers) now favor CalDAV.
- Forum thread
For The Stubborn - Getting Old Z-iSync Connector
Please don't kill the messenger, I'm putting this up here for those that are desperate and would rather deal with the iSync bugs on their own versus moving to the iCal + CalDAV supported option. Don't expect support for this configuration. There's an internal discussion about making the latest connector available directly - I'll update this page depending on the outcome. Until that discussion is resolved, I don't feel comfortable posting the individual connector by itself.
Grab the 5.0.5 tar ball from here:
http://www.zimbra.com/products/downloads_previous.html
The connectors are in the zimbra-store*.rpm , you'll find it in the packages directory once you untar the zimbra tarball.
Copy this over to temp. Find the path of the file. Extract it.
mkdir /tmp/isync cp packages/zimbra-store*.rpm /tmp/isync/ cd /tmp/isync rpm2cpio zimbra-store*.rpm | cpio -iv --make-directories `rpm2cpio zimbra-store*.rpm | cpio -t | grep -i zimbra-isync`
You'll see the new directories and within them you'll see the dmg file for the connector.
iCal General Issues
Clearing iCal cache
Make and run script:
#!/bin/sh rm -rf ~/Library/Application\ Support/SyncServices/Local/clientdata/com.apple.iCal rm -rf ~/Library/Calendars rm -rf ~/Library/Preferences/com.apple.iCal.plist rm -rf ~/Library/Preferences/IcalExternalSync.plist rm -rf ~/Library/Preferences/ByHost/com.apple.iCal.helper.* rm -rf ~/Library/Caches/com.apple.iCal rm -rf ~/Library/Caches/Metadata/iCal
Clearing ALL of iCal
Careful, This Will Remove All Calendar Data. Have all iSync related programs shutdown [addressbook, ical, iMail]
You will also need to redo your CalDAV account setup for the Zimbra iSync Connector. It might also be necessary to uninstall and reinstall the Zimbra Connector. Make and run script:
#!/bin/sh rm -rf ~/Library/Application\ Support/SyncServices/Local rm -rf ~/Library/Application\ Support/SyncLocalCopy rm -rf ~/Library/Application\ Support/iSync/SyncLocalCopy rm -rf ~/Library/Application\ Support/iCal rm -rf ~/Library/Calendars rm -rf ~/Library/Preferences/com.apple.iCal.* rm -rf ~/Library/Preferences/IcalExternalSync.plist rm -rf ~/Library/Preferences/ByHost/com.apple.iCal.helper.* rm -rf ~/Library/Caches/com.apple.iCal rm -rf ~/Library/Caches/Metadata/iCal
You might be seeing this:
http://bugzilla.zimbra.com/show_bug.cgi?id=23671
Comment #30 give some instructions on a case they can reproduce.
zmprov gd <your domain name> zimbraPublicServiceHostname zmprov gs <your server name> zimbraServiceHostname
If zimbraPublicServiceHostname is not set, or set to a different name (such as an alias of the machine or a load balancer), you are hitting this issue.
Calendar events are displaying the wrong time
Apple expects the year of 1971 for the DTSTART variable within an ICS file. There is no standard that dictates this.
Zimbra (prior to version 5.0.5) was using 1601, this is in /opt/zimbra/conf/timezones.ics .
Please see bug for more details (resolved 5.0.5):
http://bugzilla.zimbra.com/show_bug.cgi?id=22808
Alarm & Calendar Notification Issues with iCal
These issues are getting worked out for the 5.0.7 release.
Please review:
http://bugzilla.zimbra.com/show_bug.cgi?id=28057
Some more details:
http://bugzilla.zimbra.com/show_bug.cgi?id=28883
http://bugzilla.mozilla.org/show_bug.cgi?id=432540
iCal calendar color changing modifies calendar name and/or doesn't retain color
Fixed in 5.0.5 . See bugs for details:
http://bugzilla.zimbra.com/show_bug.cgi?id=26627
http://bugzilla.zimbra.com/show_bug.cgi?id=26625
Colors for Calendar & Appointments (Non-client specific)
Here's some RFE's/Bugs you might be interested in:
- "Need more colors on calendars"
- "Making visual difference between meeting status is not easy."
- "Shared calendar should auto-select an unused color"
- "Support tagging of appointments in calendar"
- http://bugzilla.zimbra.com/show_bug.cgi?id=2769
- "Tags not synced for appointments and tasks"
- "Color-coding of appointments"
- http://bugzilla.zimbra.com/show_bug.cgi?id=2769
Calendar Ordering Issues
Please see RFE I made.
- "Allow the reordering of Calendars on webclient to match the order on iCal"
- http://bugzilla.zimbra.com/show_bug.cgi?id=32197
- Depends On:
- "allow changing the order/position of calendars in schedule view"
Directory Access configuration on Macs
File that configures [system wide] for Address Book resolution
/Library/Preferences/DirectoryService/DSLDAPv3PlugInConfig.plist
Calendar Invites can't be imported into iCal
If you look at the ics file, you might see an extra return character at the end.
You'll see this if an Outlook client send an invite to a user using a Mac with a thick client [Mail.app/Entourage].
This was resolved in the 5.0.5 release. Please see bug for more details:
http://bugzilla.zimbra.com/show_bug.cgi?id=26487
Mac clients cause spamming of invitation events when they adjust calendar events
This, at it's root, is caused by other bugs listed here. Usually this will get resolved with the proper upgrades that the bugs require (ZCS 5.0.5+)
There is a RFE/Bug though that will also resolve this, by allowing the "action" of an appointment to be set.
Please see bug for details [scheduled for 5.0.7]:
http://bugzilla.zimbra.com/show_bug.cgi?id=10536
Calendar.app 10.9.5 With ZCS 8.5+
EWS - Exchange Account Topics
Configuring EWS And Calendar.app
Note - currently, with ZCS 8.5 it is not working with calendars. See the following bug:
- Calendar.app via EWS setup doesn't work with ZCS Calendar
Configuring an EWS account on a Mac gives you the option to enabled multiple services. See EWS Configuration And ZCS 8.5 on setting this up.
CalDAV Account Topics
Delegate Issues
- can't create delegate from ical via caldav
iCal 10.5 - CalDAV Issues
Logging-Debug for CalDAV+iCAL
If you are using CalDAV account with ZCS, please help us investigating this problem. On your Mac please do the following and gather some data.
First shutdown iCal app.
Launch terminal app.
You can turn on the debug options in iCal by:
defaults write com.apple.iCal IncludeDebugMenu 1
Turn them off by just reversing it:
defaults write com.apple.iCal IncludeDebugMenu 0
Furthermore, you can log all of the HTTP transaction to the console via:
defaults write com.apple.iCal LogHTTPActivity yes
Turn them off by just reversing it:
defaults write com.apple.iCal LogHTTPActivity no
Logging will show up in /var/log/system.log
Launch iCal app.
Select the shared folder, right click, refresh.
Only Seeing 1 Month Back and 1 Year Ahead In CalDAV
Please see this bug: http://bugzilla.zimbra.com/show_bug.cgi?id=28713
Set to be resolved in 5.0.9
Not seeing Calendar's In iCAL/CALDAV
Space In Calendar Name's And Other "Special Characters"
For space's in Calendar names, this is resolved for the 5.0.7 release. (Need bug # for this)
There have also been reports in the forums that "special characters" can cause an issue where Calendars aren't displayed - even those without the special character name. Note, in one case, the "special character" where Norwegian character.
Server's With Multiple Domains - Effect Users Are In Default Domain
Bug filed:
http://bugzilla.zimbra.com/show_bug.cgi?id=30263
- Conditions:
- On ZCS 5.0.7
- Mac Clients using Zimbra Connector (provided from ZCS 5.0.7) and setting up iCal/CalDAV via the Connector.
- ZCS has multiple domains
- Error:
- Default domain accounts can't see calendars
- Reproduce:
- Clear iCal with no Accounts configured (CALDAV)
- Setup Accounts via Zimbra Connector
- User username@defaultdomain.com as format for the User Name Field
- Work Around:
- Clear iCal with no Accounts configured (CALDAV)
- Setup Accounts via Zimbra Connector
- User username as format for the User Name field
- Non-Default Domains - Works As:
- Clear iCal with no Accounts configured (CALDAV)
- Setup Accounts via Zimbra Connector
- User username@domain.com as format for the User Name Field
- Note, these accounts need @domain, will not work with just username.
iCal And Entourage Use for Calendars - Leopard/10.5 users
Update The 5.0.6+ Connectors will no longer work with Entourage. Your only option for Calendars will be using CalDAV with iCal.App. You still will see an option about Entourage in the Connector but it will be grayed out.
Because of some core issues with Apples SyncServices, we recommend that Entourage and iCal users use iCALv3 (Mac 10.5) configured for CALDAV. Please use the Zimbra Connector instructions to setup iCal for CALDAV.
For Entourage, this recommendation stems from the fact that Microsoft decided to use WebDAV rather than CALDAV. If MS decides later to include CALDAV support for Entourage, we'll have another option.
Please see bug for details:
- RFE: "Leopard - CalDav only" [FIXED]:
- RFE: "Support for Entourage WebDav" [WONTFIX]:
Snow Leopard , Outlook/Entourage , Exchange Web Services
There is no target milestone for it as of today [2010/02/23]. This topic applies to the following references or terms: Exchange auto discovery feature, Exchange Autodiscover service or anything else that relies on Exchange Web Services [ EWS ] on the Mac. Another Apple reference on the topic [pdf warning] : Mac OS X Snow Leopard: Integrating the Mac into an Exchange 2007 Environment .
Please see the following in regards to Zimbra's work and stance on the issue:
- Zimbra has open RFE's for the 'newer' Outlook for Mac versions that use Exchange Web Services [EWS] and will be bypassing the older Entourage versions that use WebDAV.
- "Support for Entourage WebDav" - marked as WONTFIX
- http://bugzilla.zimbra.com/show_bug.cgi?id=24502
- "entourage 2008 integration" - marked as a duplicate of bug 24502 above.
- "Support for Entourage WebDav" - marked as WONTFIX
- RFE: "Enable snow leopard messaging features, Outlook for Mac 2010"
- Assigned but no committed target release, you'll want to include your votes/comments to increase priority.
- http://bugzilla.zimbra.com/show_bug.cgi?id=38631
- Duplicate of bug 38631 above - RFE: "Support for Outlook for Mac 2010"
- http://bugzilla.zimbra.com/show_bug.cgi?id=41106
- In the meantime, Snow Leopard has 3 applications that will work like the following:
- Mail > IMAP [working]
- iCal > CalDAV [working]
- Address Book > CardDav
- We are waiting for Apple to have general support for CardDav and to allow 3rd party servers
- See the following dev comments in bug 22008 - "WebDAV: Add support for CardDav":
- We are waiting for Apple to have general support for CardDav and to allow 3rd party servers
Can't see Calendars after configuring Z-iSync for CALDAV - 10.5 users
You didn't follow the directions most likely. Goto iCal Preferences Accounts.
There should be NO configured accounts prior to configuring Z-iSync for CALDAV.
Remove the current Account listed in iCal and redo configuration for CALDAV with Z-iSync.
Trick to work around address lookup for the Attendee field in iCal Events
Configure the Addressbook using one of the other recommendations on this page.
You can now drag entries from your Addressbook into the Attendee field of iCal.
Attendee Lookups for iCALv3/MacOS10.5
iCal3 [webdav] needs to use the Apple Directory Access utility for attendees (It's on 10.5). It will only work against Apple OD/CAL servers.
RFE filed to include necessary LDAP attributes to provide ical oattendee lookups
See bug [scheduled for 5.0.7]:
http://bugzilla.zimbra.com/show_bug.cgi?id=26619
Free/Busy Lookup not working as expected
The Free/Busy lookup requires auto-attendee lookup to be working.
This requires bug 26619 [ http://bugzilla.zimbra.com/show_bug.cgi?id=26619 ] to be resolved for iCalv3/Mac 10.5.
iCal-CalDAV - server doesn't retain color information from iCal
Summary of issue: Zimbra only supports the preset colors on iCal, which are blue, green, red, orange, pink, and purple.
Please see the following:
- "CalDAV: server does not retain color information from iCal"
- http://bugzilla.zimbra.com/show_bug.cgi?id=26627
- I've added a private comment to this to see if another RFE is needed to expand this and if there's any show stopper reasoning with why it can't be done. I'll update this here when I hear back on it.
I created a bug for this issue, please see:
- "iCal - CalDAV used to create new calendars sets up share permissions on server"
Can't Select Which Calendars To Sync - I Could With 10.4 Though
This is because of the change from iSync to CalDAV. iCal always syncs all the calendars in a given CalDAV account together. The CalDAV spec allows syncing each Calendar, but iCal chooses to sync them all.
There is no known work around at this time. Nor is there an RFE made because the dev's have stated it's an iCal issue. If this changes, I'll update this entry.
iTunes & iPhone configuration for CALDAV
Please see main wiki page on iPhone http://wiki.zimbra.com/index.php?title=IPhone
If you follow the instructions about configuring the Zimbra iSync Connector for CALDAV use, there should be no issues about sync'ing with your iPhone.
At this time, new events created on a iPhone/CALDAV setup will write the event to a local calendar in iCal.app . There's no way around this yet. It's a limitation at this time with Apple's software. You can change the events calendar assignment though later via iCal.app.
Please see Screenshot of iTunes & iPhone
iPhone 3.0 will use port 8443 as default when setting up Caldav, you most likely need to change this to 443.
iCal 10.4 - iSync Issues
Attendee Lookup for iCal/Mac 10.4
iCal.app on Mac 10.4 only uses local entries in Apples Addressbook for address/username lookup for new appointments.
Addressbook+LDAP configuration requires a copy of an "all/*" search in ldap into the local addressbook. Sync doesn't seem to work.
Please see this bug comment and the 10.4 section for more details:
http://bugzilla.zimbra.com/show_bug.cgi?id=26619#c3
New Calendars don't sync unless you select "all". Mac 10.4/Z-isync
Please see bug for details:
http://bugzilla.zimbra.com/show_bug.cgi?id=26653
Sync To Do/Tasks Items in Leopard
RFE filed, please see http://bugzilla.zimbra.com/show_bug.cgi?id=12917
No target date, please vote for this RFE.
Outlook
General Outlook Issues
Getting Older ZCO Versions
First check to see if the release version you want is available on the support portal, you'll need to login to the support portal.
You can download all previous releases from here:
The connectors are in the zimbra-store package. For example:
[root@mail59 zimbra]# find . -name *.msi -print ./jetty-6.1.22.z6/webapps/zimbra/downloads/ZimbraOlkConnector-6.0.6_GA_2324_6.0.5902.6.msi [root@mail59 zimbra]# rpm -qf ./jetty-6.1.22.z6/webapps/zimbra/downloads/ZimbraOlkConnector-6.0.6_GA_2324_6.0.5902.6.msi zimbra-store-6.0.6_GA_2324.RHEL5_64-20100406133038
With a rpm based distro, you'll do something like the following reference shows to extract the files from the rpm using rpm2cpio & cpio vs. needing to "install" the rpm:
Basic gist:
[root@mail59 zimbra]# pwd /tmp/zco-rpm [root@mail59 zimbra]# cp /path/to-untar-zcstarball/packages/zimbra-store* /tmp/zco-rpm/ [root@mail59 zimbra]# rpm2cpio zimbra-store*.rpm | cpio -iv --make-directories `rpm2cpio zimbra-store*.rpm | cpio -t | grep -i zimbraolkconnector`
Getting Older PST Import Tool
For rpm example, see above about getting older ZCO version. This example will be using deb, on ubuntu.
If you don't have ar, install it [as root]
apt-get install binutils
Get ZCS 7.2.7 , for example :
wget https://files2.zimbra.com/downloads/7.2.7_GA/zcs-NETWORK-7.2.7_GA_2942.UBUNTU10_64.20140314190301.tgz
Unpack it:
root@dell2-vm2:/tmp# tar --wildcards -xzvf zcs-NETWORK-7.2.7_GA_2942.UBUNTU10_64.20140314190301.tgz '*store*' zcs-NETWORK-7.2.7_GA_2942.UBUNTU10_64.20140314190301/packages/zimbra-store_7.2.7_GA_2942.UBUNTU10_64_amd64.deb
Then us ar to extract the deb file
root@dell2-vm2:/tmp# ar vx zcs-NETWORK-7.2.7_GA_2942.UBUNTU10_64.20140314190301/packages/zimbra-store_7.2.7_GA_2942.UBUNTU10_64_amd64.deb x - debian-binary x - control.tar.gz x - data.tar.gz
Now you can then get the PST import tool:
root@dell2-vm2:/tmp# tar --wildcards -xzvf data.tar.gz '*PST*' ./opt/zimbra/jetty-6.1.22.z6/webapps/zimbra/downloads/ZCSPSTImportWizard-7.2.7.1022.zip
Import & Export Of Outlook Categories As Zimbra Tags
Under ZCO
Older versions of ZCO documentation had:
- "If categories have been assigned to messages and contacts, these are converted to tags in the user’s Zimbra mailbox."
Newer ZCO releases and documentation now state:
- "Zimbra Tags are synchronized with Outlook Categories"
Import & Export Notes & Journal
Notes & Journal Via PST
In order to keep their Notes and Journal entries, users should save these Outlook items in pst format before syncing and then import the pst file after the initial sync is performed. This would need to be done prior to installing and configuring the Zimbra Outlook Connector. Also, this "import" is via the normal Outlook pst process - NOT via the Zimbra pst import tool.
Related Bugs And Resources
- "Outlook Notes not visible in webmail UI after migration."
- http://bugzilla.zimbra.com/show_bug.cgi?id=5017
- "Support import/migration/sync of outlook notes"
- "Inconsistent handling of "Notes" folder"
- ZCO Support for Multi-Service MAPI Profiles
Into Zimbra Documents
Notes can be exported from Outlook to a CSV file. You can the manipulate this with Excel/Word to a format that would work for you within Zimbra Documents.
In Excel - Add borders to the cells, Adjust fonts, Adjust text spacing and then highlight the data. Ctrl+C .
In Zimbra web client - Goto Documents > New .
Click in body of New Document and Ctrl+V . You should see your Notes now in a Table.
Upload HTML
See:
- "View as HTML and Import as HTML in documents"
Following above example except you format your old Notes to become a HTML file. You could export/save-as using whatever application you like.
Then, in Zimbra web client - Goto Documents > New Drop Down Arrow and select Upload File
This, unfortunately, will get saved under "Briefcase"
Upload File Type For Conversion To Zimbra Documents
Update - This Will Be True For Upcoming Releases
See (Zimbra Employees) - On DF
- "import/export word/excel to documents"
- Internal (Private) RFE - 10124
Notes can be exported from Outlook to a CSV file. Open this file with Excel and make any adjustments you would like and then save as an Excel file.
From the Zimbra webclient, log in with your account details.
Goto the Documents tab.
Click on Import. You'll be able to select Excel files as an import option.
Into Zimbra Tasks?
Initial thoughts.
If one does something like this : http://ZIMBRASERVER/home/ajcody/Tasks
It will give you your tasks in an ics format (iCal). Here's what one tasks would look like:
BEGIN:VCALENDAR PRODID:Zimbra-Calendar-Provider VERSION:2.0 METHOD:PUBLISH BEGIN:VTIMEZONE TZID:(GMT-05.00) Eastern Time (US & Canada) BEGIN:STANDARD DTSTART:19710101T020000 TZOFFSETTO:-0500 TZOFFSETFROM:-0400 RRULE:FREQ=YEARLY;WKST=MO;INTERVAL=1;BYMONTH=11;BYDAY=1SU END:STANDARD BEGIN:DAYLIGHT DTSTART:19710101T020000 TZOFFSETTO:-0400 TZOFFSETFROM:-0500 RRULE:FREQ=YEARLY;WKST=MO;INTERVAL=1;BYMONTH=3;BYDAY=2SU END:DAYLIGHT END:VTIMEZONE BEGIN:VTODO UID:0118908d-2c50-4586-bed1-b135388afcc6 SUMMARY:Tasks Exporting DESCRIPTION:These are my notes I'm putting in for my task. \n LOCATION:My Home PRIORITY:5 PERCENT-COMPLETE:30 ORGANIZER;CN=Adam J. Cody:mailto:ajcody@mail3.internal.homeunix.com DTSTART;VALUE=DATE:20080821 DUE;VALUE=DATE:20080831 STATUS:IN-PROCESS CLASS:PUBLIC DTSTAMP:20080821T170223Z SEQUENCE:1 END:VTODO END:VCALENDAR
As you can see, it would be a difficult task [no pun intended] to convert the data from Outlook Notes into ics format to allow importing to Zimbra Tasks.
If you do find a way, the import process from CLI would be:
curl -u USERNAME:password --data-binary @/tmp/Tasks.ics http://server/service/home/USERNAME/Tasks?fmt=ics
See User_Migration for more details.
ZCO Zimbra Outlook Connector
Supported Versions Of Windows, Outlook, ZCO
I created the following RFE:
- "ZCO matrix table showing supported versions"
Current document for this would be:
ZCO 5.0.10 Release
Local Failure Notice In My Inbox
This is new with this release of the ZCO.
- "Better error reporting for sync issues. A copy of local failures/server failure error messages will be delivered to the inbox."
- "UI: User not alerted of certain errors in a noticeable way"
- http://bugzilla.zimbra.com/show_bug.cgi?id=25884
What To Do When You Get This If You Want To Contact Support
Add these details/steps to the creation of your support case.
- Send us the failure messages.
- Easiest way is to create a temp folder on your desktop, drag all these messages to the folder. (This will create .msg files inside the folder.) Then zip up the folder and send the zip file to us through a support case.
- For each type of errors with an id, get the "Show Original" of one of the failures.
- To get the "Show Original", use an URL like this:
- Format: http(s)://<server>/service/home/<user@domain.com>/~/?fmt=sync&id=<id>
- Example: https://zimbra.foo.com/service/home/user1@foo.com/~/?fmt=sync&id=123456
- Then, select all and copy to a .txt file.
- To get the "Show Original", use an URL like this:
If You Want To Turn This Feature Off
Just create a DWORD value in the registry of HKEY_LOCAL_MACHINE\SOFTWARE\Zimbra\turnOffInboxFailures and set it to 1. Then the local failures won't go to the Inbox. They'll just go to the local failures folder, as before.
ZCO 5.0.7+
ZCO 5.0.7 and higher have new features and also should only be used with ZCS server 5.0.7 or higher.
Check The ZCO 5.0.6 Bug Backport Binary
Login to the support portal and check Zimbra Utility and Beta Downloads.
Or try this direct url:
https://support.zimbra.com/node/71
It will list out the bugs that were back ported.
The complete troubling-shooting guide (dev info requests)
- Reproduce
- ZCO install logs
- msiexec /i zco-installer.msi /lv 00022083-zco-install.log
- Get zco logs
- See Using Logging Control for Troubleshooting; within the below article :
- Check for core dumps - mandatory for all crashes
- Generating Core Dumps :
- Creating a Core Dump from a Running Process using WinDbg :
- Winhttp trace (issues where they can&amp;amp;amp;amp;#39;t connect - check zco logs first). Run this on the machine with Outlook.
- WinHttpTraceCfg -e 1 [prefix] -d 0
- When WinHttpTraceCfg is executed, try sending a message with an attachment. This time a log should be created.
- WinHttpTraceCfg Documentation
- DbgViewHttpTrace :
- WinHttpTraceCfg -e 1 [prefix] -d 0
- Are there any local failures/server failures?
- Winmsd (dump of the local environment configuration)
- Start Run winmsd
- This might take awhile to bring up the application
- From app [System Information titled] , File Export
- What about msinfo32.exe ?
- Start Run winmsd
- External factors (add-ins, other mapi clients)
- Network topology (firewall, proxy, etc)
- Server topology (single/multi-node, clustering, nginx, 3rd party lb)
- Search bugzilla
- Search old support cases
- Can we get remote access?
- Can we get an account on the server?
- Can we access the account in question?
- Get the id of the item
- get the original of the item from the server (mime representation)
- get the .msg of the item from outlook
- Quick analysis of zco logs
- sync - change record creation - CHANGE RECORD
- follow soap requests, look for soap faults
- look for the word - exception
- follow soap traces
- ignore MAPI_E_NOT_FOUND
Performance Issues
Sources:
http://msexchangeteam.com/archive/2007/12/17/447750.aspx
http://blogs.msdn.com/outlook/
http://support.microsoft.com/kb/940226/
http://technet.microsoft.com/en-us/library/bb738147.aspx
The skinny is this, [per Microsoft]
- ...recommended max OST size (2GB)
- ...we strongly recommend storing no more than 5,000 items in core folders, such as the Inbox and Sent Items folders. Creating more top-level folders, or subfolders underneath the Inbox and Sent Items folders, greatly reduces the costs associated with this index creation, so long as the number of items in any one folder does not exceed 5,000.
Can't Send Via Send To -> Mail Recipient
Please see this bug:
http://bugzilla.zimbra.com/show_bug.cgi?id=26730
You shouldn't have this problem with connectors from 5.0.6+
Saving Current Profile Out To Then Re-Import
With Outlook closed/shutdown, you can then copy/rename the *.zdb file under your profile to a .pst file.
Then you can later run the PST Import Wizard to import the file into a new Zimbra profile you setup. One of the options on the Import Wizard let's you import only mail after a specified date - to avoid duplications.
Thunderbird , Lightning, Firefox
Debug Logging For Thunderbird
To create a log file with all POP, SMTP, or IMAP traffic in Mozilla Thunderbird:
For Windows
- Make sure Mozilla Thunderbird is not running.
- Select All Programs | Accessories | Command Prompt from the Start menu.
- Type "set NSPR_LOG_MODULES=" followed immediately by
- "POP3:4" for POP,
- "IMAP:4" for IMAP and
- "SMTP:4" for SMTP
- Press Enter.
- You can enable logging for multiple protocols by separating them with commas (',').
- To log both POP and SMTP traffic, for example, type "set NSPR_LOG_MODULES=POP3:4,SMTP:4" (excluding the quotation marks).
- To log only IMAP traffic, type "set NSPR_LOG_MODULES=IMAP:4", for instance.
- You can enable logging for multiple protocols by separating them with commas (',').
- Type "set NSPR_LOG_FILE=%HOMEDRIVE%%HOMEPATH%\Desktop\tbird_log.txt".
- Press Enter.
- Type "start thunderbird".
- Press Enter again.
- Perform the desired email actions in Mozilla Thunderbird.
- Quit Mozilla Thunderbird and take a look at "tbird_log.txt" on your Desktop.
For Mac OS X
- Open a Terminal window.
- Type "set NSPR_LOG_MODULES=" followed immediately by
- "POP3:4" for POP,
- "IMAP:4" for IMAP and
- "SMTP:4" for SMTP
- Press Enter.
- You can enable logging for multiple protocols by separating them with commas (',').
- To log both POP and SMTP traffic, for example, type "set NSPR_LOG_MODULES=POP3:4,SMTP:4" (excluding the quotation marks).
- To log only IMAP traffic, type "set NSPR_LOG_MODULES=IMAP:4", for instance.
- Type "set NSPR_LOG_FILE=~/Desktop/tbird.log".
- Press Enter.
- Now type "/Applications/Thunderbird.app/Contents/MacOS/thunderbird-bin".
- Press Enter again.
- Perform the desired email actions in Mozilla Thunderbird.
- Quit Mozilla Thunderbird and take a look at "tbird.log" on your Desktop.
For Linux
- Open a Terminal window.
- Type "set NSPR_LOG_MODULES=" followed immediately by
- "POP3:4" for POP,
- "IMAP:4" for IMAP and
- "SMTP:4" for SMTP
- Press Enter.
- You can enable logging for multiple protocols by separating them with commas (',').
- To log both POP and SMTP traffic, for example, type "set NSPR_LOG_MODULES=POP3:4,SMTP:4" (excluding the quotation marks).
- To log only IMAP traffic, type "set NSPR_LOG_MODULES=IMAP:4", for instance.
- Type "set NSPR_LOG_FILE=~/tbird.log.txt".
- Press Enter.
- Type "thunderbird".
- Press Enter again.
- Perform the desired email actions in Mozilla Thunderbird.
- Quit Mozilla Thunderbird and take a look at "tbird.log.txt" in your Home directory.
Thunderbird Contact Sync for Zimbra
Please see the following as a possible solution:
Gnome Evolution And KDE Kontact
Some RFE's I've made:
- "RFE: Official Support For Full Functionality Of Evolution Mail Client [webdav]"
- "RFE: Official Support For Full Functionality Of KDE Kontact Mail Client [webdav]"
Zimbra Desktop [ZD]
ZD References
- ZD Homepage
- ZD FAQ
- ZD Feature Comparison with Competing "Desktop" Products
- ZD Forum Support Page
- Blog About The Internals Of ZD
Using ZD To Import Zimbra TGZ Export User Data
See :
- Problems with importing tgz files - couple of items to fix/clean up
- ZD Installer/Account Setup to allow "Local Only" for mail account setup
IM - Instant Messaging Clients
Spark
Please see:
- Spark_and_Zimbra
- Spark Project Page
SparkWeb
Don't believe this will currently work. I'm still investigating it though.
RFE I made in the meantime:
- "IM server to support Sparkweb / Shockwave - Flash need for crossdomain.xml type file"
Please see:
- SparkWeb Project Page
- SparkWeb How-To
Setting Up Apache For The crossdomain.xml File
Ok, this is just down and dirty. You'll need to handle security issues yourself.
mkdir /opt/zimbra/spark-web vi /opt/zimbra/spark-web/crossdomain.xml
Putting in this,
<?xml version="1.0"?> <!DOCTYPE cross-domain-policy SYSTEM "http://www.macromedia.com/xml/dtds/cross-domain-policy.dtd"> <cross-domain-policy> <allow-access-from domain="*" to-ports="5222" /> </cross-domain-policy>
Set permissions:
chown -R zimbra:zimbra /opt/zimbra/spark-web
Let's configure apache to server this file out and listen on port 5229.
Note, you could also set this up within a Virtual Hosts.
vi /opt/zimbra/conf/httpd.conf Towards the start of the file, add another Listen line for port 5229 under the Listen 80
Listen 5229
Towards the end, add the following to include spark-web directory
Include /opt/zimbra/httpd/conf/extra/spark-web.conf
Let's now make the apache conf file for zmstat-chart.conf
vi /opt/zimbra/httpd/conf/extra/spark-web.conf
Alias /zmstat-chart "/opt/zimbra/spark-web" <Directory "/opt/zimbra/spark-web"> Options Indexes IndexOptions FancyIndexing VersionSort AllowOverride None Order allow,deny Allow from all </Directory>
Restart apache so the changes take effect:
zmapachectl stop zmapachectl start
To test your changes, with a web browser goto:
http://yourapacheserverhostname:5229/crossdomain.xml
You should see the xml file you made.
Now setup the spark-web software on the same box. SparkWeb HOW-TO will guide you but use the /opt/zimbra/spark-web directory you already setup. I'll leave the particulars to this to the HOW-TO referenced. Just remember that when you setup the spark-web client you'll modify it's config file like below.Don't forget to put in your IM servers hostname for "youropenfireserver":
<script type="text/javascript"> function jive_sparkweb_getConfig() { return { server: "youropenfireserver", connectionType: "socket", port: "5222", autoLogin: "false", policyFileURL: "xmlsocket://youropenfireserver:5229" };
Pidgin - Formally GAIM
Please see Pidgin_and_Zimbra
Pidgin Groupchat
Configure For Chat Room:
- Buddies
- Join A Chat
- Account = Zimbra IM Account
- Room
- Either A New One - which will prompt you for setup. Check persistent if you want it to be.
- Enter An Existing Conference Room Name
- Server
- Enter conference.SERVERNAME , Example: conference.example.com
- conference is a default variable that will be added to the primary zimbra hostname/domainname.
- Enter conference.SERVERNAME , Example: conference.example.com
See Public Chat Rooms Listed:
- Buddies
- Join A Chat
- Account = Zimbra IM Account
- Server
- Enter conference.SERVERNAME , Example: conference.example.com
- conference is a default variable that will be added to the primary zimbra hostname/domainname.
- Enter conference.SERVERNAME , Example: conference.example.com
- Click On The "Room List" button
Adium
Please see Adium_and_Zimbra
Adium Groupchat
Configure For Chat Room:
- File
- Join Group Chat
- Set Account To Your Zimbra XMPP/Jabber Account
- Chat Room Name
- Either A New One - which will prompt you for setup. Check persistent if you want it to be.
- Enter An Existing Conference Room Name
- Server
- Enter conference.SERVERNAME , Example: conference.example.com
- conference is a default variable that will be added to the primary zimbra hostname/domainname.
- Enter conference.SERVERNAME , Example: conference.example.com
See Public Chat Rooms Listed:
- File
- Highlight Your Zimbra Account until the options show on the right.
- Discovery Browser
Digsby
Just discovered this project, though I haven't used it much. Might be worth checking out. It's available for Windows [Beta] and coming "soon" for Mac and Linux. Does IM, Email notification, and Social Networking stuff.
- Digsby Home Page
- http://www.digsby.com/
- Features:
- digsby is a multiprotocol IM client that lets you chat with all your friends on AIM, MSN, Yahoo, ICQ, Google Talk, and Jabber with one simple to manage buddy list.
- digsby is an email notification tool that alerts you of new email and lets you perform actions such as 'Delete' or 'Report Spam' with just one click.
- digsby is a social networking tool that alerts you of events like new messages and gives you a live Newsfeed of what your friends are up to.
- Features:
- Items to Notes:
- Where is group chat?
- We had some bugs with group chat so we launched Digsby beta without it. However, we will bring it back in the near future.
- Requires account setup with Digsby. This might not be acceptable for some customers for security reasons.
- Where is group chat?
- http://www.digsby.com/
Empath
Please see Empathy_and_Zimbra
Other Topics
PGP , GPG , Domain Keys, Encryption
Just references as I find them:
- "GPG/PGP support"
- The last two bugs look to be held up because of amavis, we are waiting for a release from them that will support what we need to do.
- "DKIM/DK signing support in Amavisd"
- Duplicate of above
- "Domain Keys and signed inbound/outbound mail"
Third Party Option:
- The "Zimbra OpenPGP Zimlet"
Mobility
Help Me Understand the Mobility Options
Let's break it down to three forms of access for mobile devices.
Mobile Web Client - http://www.zimbra.com/products/mobile_web_browser.html
A. Setups the web access to the Zimbra server to be viewable from a mobile device. This is accessing mail over the "web browser" on the mobile phone.
A1. http://servername.com/zimbra/m/
B. Built in - no license or additional software needed.
Zimbra Mobile - http://www.zimbra.com/products/mobile_smartphone.html
A. This allows two-way, over-the-air synchronization between the mobile device and Zimbra server.
A1. Think of this as "fat client" mail sync'ing on a mobile device
B. This page will describe that process in better detail - http://wiki.zimbra.com/index.php?title=Moble_Device_Setup
C. License required
Blackberry - http://www.zimbra.com/products/mobile_blackberry.html
A. Blackberry "two-way, over-the-air synchronization" requires the use of a Blackberry server. We use a "connector" on the Blackberry server that allows BES device to use their "fat client" for mail sync'ing.
B. This option requires a separate server, a Windows box that will run the Blackberry server software.
C. Zimbra Mobile license required. Windows Blackberry server require with appropriate Blackberry licenses. Installation of Zimbra BlackberryConnector on the Blackberry server.
D. This is still in BETA
Zimbra Mobile/Blackberry requires the license file to enable it. This cost can be discussed with your sales contact. It's only the Blackberry configuration that requires a "software" installation - so to speak. Enabling "Zimbra Mobility/option 2 above" is a license requirement - not a software installation one - and then configuration for the user. "You enable Zimbra Mobile in the ZCS COS or for individual Accounts."