Ajcody-Server-Topics: Difference between revisions

mNo edit summary
Line 267: Line 267:




==LDAP Topics==


===Working With ldap Commands - The Easy Way===
Tired of working with long ldap strings on the CLI with ldapmodify, ldapsearch, ldapdelete.
zimbra$ source ~/bin/zmshutil
zimbra$ zmsetvars
You can now use this string syntax to make things a little easier.
ldapCOMMAND -x -H $ldap_master_url -D $zimbra_ldap_userdn -w $zimbra_ldap_password ACTION
===Change LDAP Port On Zimbra===
It is recommend that there is no other LDAP service running on the ZCS besides Zimbra's. But if you find the need to change it, here's the steps.
You can modify the default [389] port ldap uses to port 390 by doing the following [change LDAPHOSTNAME.com to your server name]:
zmlocalconfig -e ldap_port=390
zmlocalconfig -e ldap_master_url=ldap://LDAPHOSTNAME.com:390
zmlocalconfig -e ldap_url=ldap://LDAPHOSTNAME.com:390
Note you need to do all three of these or things will break.  Once you've run those commands, you will need to restart ZCS:
zmcontrol stop
zmcontrol start
Once this is completed, Zimbra will run with LDAP on port 390 rather than port 389 (the default).  If you'd like to use a port other than 390, substitute that port for 390 and use the same process.
===Can't Setup/Install LDAP Replica===
Please see bug against documentation:
* "Update ch5 multi-server replication instructions"
** http://bugzilla.zimbra.com/show_bug.cgi?id=26117
*** Updates from the bug seem to be mostly going into the Multi-Server documentation.
**** http://www.zimbra.com/docs/os/latest/multi_server_install/LDAP%20Replication.6.1.html
Checking /tmp/zmsetup.log will expose errors with installation configuration, for example below, others are listed in my notes farther down:
<pre>
: Thu Aug 21 16:00:12 2008 *** Running as zimbra user: /opt/zimbra/bin/zmlicense -c
Error: exception occurred: system failure: ZimbraLdapContext
when running 'zmcontrol start':
[zimbra@mail1b ~]$ zmcontrol start
Host mail1b.DOMAINNAME
Unable to determine enabled services from ldap.
Unable to determine enabled services. Cache is out of date or doesn't exist.
</pre>
Make sure you are using the right passwords. If the ldap replica can't authenticate to the master LDAP server it will not be able to pull down the ldap configurations it needs - one of which is the license data.
====My Notes From The Case That I Made Bug 26117 For====
=====Summary=====
Action Items:
*Update doc's to reflect ldap repli password setting during install (all ref docs miss this step)
** If this is done during install, no other action is needed except logger and sshkeys (zmupdateauthkeys)
* Update doc's to reflect recovery steps on an improper install for ldap replication server
* Make comments consistent. See [[LDAP|LDAP wiki page]]
* Update installer to require ldap repli password rather default to random set
password.
References:
* "Update ch5 multi-server replication instructions"
** http://bugzilla.zimbra.com/show_bug.cgi?id=26117
* [[LDAP|Main LDAP Wiki Page]]
** [[LDAP#LDAP_replication|LDAP Replication Section]]
* http://www.zimbra.com/docs/os/latest/multi_server_install/LDAP%20Replication.6.1.html
* http://www.zimbra.com/elqNow/elqRedir.htm?ref=http://files.zimbra.com/website/docs/Zimbra%20NE%20Multi-Server%20Install.pdf
LDAP replication has some comments that aren't in other places
* Set the master LDAP password to the correct value (run zmlocalconfig -s ldap_root_password on the master to determine this value)
* Note: In order to install an LDAP replica server with no MBS (Mailbox Server), set zimbra_zmprov_default_to_ldap to true, using the following command:
** <pre>zmlocalconfig -e zimbra_zmprov_default_to_ldap=true.</pre>
** If you later add an MBS to your LDAP replica server, set zimbra_zmprov_default_to_ldap to false.
* After the installation is complete, run /opt/zimbra/libexec/zmldapenablereplica on the replica server. This will enable replication in the LDAP server and will force an LDAP restart to begin pulling data from the master. zmldapenablereplica must be run on both the master and replica for LDAP replication to work.
=====The Work=====
Duplicated on ZCS 5.0.2 - OS Redhat.
Current Servers For Test:
::* Setup LDAP MAIN server - zldap1  (ldap only configured server)
::** setting passwords to "PASSWORD"
::* Mailstore server - zmail2  (mailstore only config)
::* MTA server      - zmta1    (mta only config)
::* Post install
::** enable logger
::** enable ssh keys
::*** run zmupdateauthkeys on servers
::** created two test accounts and confirmed email delivery between the two.
::** On LDAP MAIN (zldap1)- zmldapenablereplica
Install LDAP repli - zldap3
::* Doc show only to set
::** ldap master host
::** Master ldap server password
::** Create Domain to no
::* It fails to include
::**  Replication Password (part of "Ldap configuration")
::*** If this is not done, install will finish but you get these errors.
::::<pre>
:::: [zimbra@zldap3 ~]$ zmcontrol status
:::: Cannot determine services - exiting
:::: zldap3.DOMAIN.com...ERROR: service.FAILURE (system failure: getDirectContext)
:::: (cause: javax.naming.AuthenticationException [LDAP: error code 49 - Invalid Credentials])</pre>
=====To Correct Without Reinstalling=====
If this has happened and you want to correct without a reinstall. Do the following, please adjust for HOSTNAMES for LDAP MAIN and LDAP REPLICA.
* confirm ssh keys are updated
:: run zmupdateauthkeys on all servers
* confirm current settings on LDAP MAIN
::<pre>
:: zmlocalconfig -s | grep replication
:: grep credentials /opt/zimbra/conf/slapd.conf</pre>
* confirm current settings on LDAP REPLI
::<pre>
:: grep credentials /opt/zimbra/conf/slapd.conf
:: zmlocalconfig -s | grep replication</pre>
* Now reconfigure on LDAP REPLI
::<pre>zmldappasswd -l [passwd from ldap main slapd.conf]</pre>
** This does not dynamically update the slapd.conf variable, it puts it in the zmconfig file /opt/zimbra/libexec/zmldapenablereplica
** You'll see that it updates the slapd.conf
** There might be one auth error, until the slapd.conf is reconfigured.
** zmprov gacf will fail with auth errors
* Confirm again settings on LDAP REPLI
::<pre>
:: grep credentials /opt/zimbra/conf/slapd.conf
:: zmlocalconfig -s | grep replication
:: zmlocalconfig | grep ldap_url</pre>
** Should have repli first and ldap main second
* Now to confirm it worked
::<pre>
:: zmcontrol stop
:: zmcontrol start
:: zmcontrol status</pre>
** Create a test account on LDAP MAIN (example uses test3)
** On LDAP REPLI - replace hostname and test account below
*** <pre>ldapsearch -x -H ldap://LDAPREPLICA.DOMAIN.com:389 | grep test3</pre>
===Disable LDAP Replica===
References:
http://wiki.zimbra.com/index.php?title=Promoting_Replica_to_LDAP_Master
http://www.zimbra.com/docs/os/latest/multi_server_install/LDAP%20Replication.6.1.html
====Remove LDAP Replica From All Active Servers====
On each member server, including the replica itself, verify the ldap_url value.
zmlocalconfig ldap_url
Modify the ldap_url to only include enabled ZCS LDAP servers.  The master LDAP server should always be at the end of the ldap_url string value.
zmlocalconfig -e ldap_url="ldap://<replica-server-host> ldap://<master-server-host>"
====Disable LDAP On The Replica====
Stop Zimbra services.
zmcontrol stop
The - in front of zimbraServiceEnabled is [off], rather than a + for [on].
zmprov -l ms `zmhostname` -zimbraServiceEnabled ldap
To enable the ldap service, prefix the zimbraServiceEnabled attribute with a "+".
zmprov -l ms `zmhostname` +zimbraServiceEnabled ldap
If other services are enabled on this host, start them.
zmcontrol start
====Additional Steps for MTA hosts====
After updating the ldap_url with zmlocalconfig, rerun /opt/zimbra/libexec/zmmtainit. This rewrites the Postfix configuration with the updated ldap_url.
====Disabling Replication On The Master====
Use only if disabling replication entirely across all nodes.
Edit /opt/zimbra/conf/slapd.conf.in adding the following comments.  Please use the precise number of hash marks (#) shown.
change
include /opt/zimbra/conf/master-accesslog.conf
to
###include /opt/zimbra/conf/master-accesslog.conf
change
overlay syncprov
syncprov-checkpoint 20 10
syncprov-sessionlog 500
include /opt/zimbra/conf/master-accesslog-overlay.conf
to
#overlay syncprov
#syncprov-checkpoint 20 10
#syncprov-sessionlog 500
###include /opt/zimbra/conf/master-accesslog-overlay.conf
Restart the master LDAP server
ldap stop; ldap start


==Logger Issues==
==Logger Issues==

Revision as of 21:59, 16 November 2008

Template:NotOfficial

Server Topics

Actual Server Topics Homepage

Please see Ajcody-Server-Topics

Issues Being Investigated

License Issues

What Should Count Against License

Real accounts, as listed in admin console under Addresses > Accounts. The admin account will count against this but the ham, spam, and wiki ones will not.

Accounts or entries listed under : Aliases , Distribution Lists, Resources should not. See below for bug about Resources though.

Deleted Accounts Still Show In Use

Please see:

Resources Counting Against License

Please see:

Workaround is :

  • zmcontrol stop
  • zmcontrol start

Tested on 5.0.9

  • license count 5
    • created two resources
  • license count 7
    • zmcontrol stop
    • zmcontrol start
  • license count 5

Performance Issues When Using Mini-Cal And You Have zimbraMailCanonicalAddress Set To Domains You Don't Have

Background Bugs:

Do you have any user's with the variable zimbraMailCanonicalAddress set using a domain that is not within your Zimbra infrastructure? There was case that had that set for a particular user to a domain they didn't have within Zimbra and the symptom showed as a performance issue within the mini-calendar & calendar. The root cause was actually the ldap lookups occurring in the background (those against the zimbraMailCanonicalAddress domain).

  • One work around was setting:
    • zmlocalconfig -e ldap_starttls_supported=0
      •  ldap stop
      •  ldap start
    • zmlocalconfig -e zimbra_require_interprocess_security=0
    • To update the postfix configuration files.
      • /opt/zimbra/libexec/zmmtainit
    • To update amavis config files.
      • /opt/zimbra/libexec/zmmtaconfig amavis
    • Then restart the system. Still need to double check this will be necessary.
  • The other workaround was to remove the zimbraMailCanonicalAddress variable.

5.0.7+ Performance & Hanging Issues

Administrators might or might not catch this events being tied to calendars or ics data. Here's what I've gather from other cases so far about the issue, there's about 5 of them I've seen. None are resolved at this time (July 23, 08), so use with caution.

1. bug: http://bugzilla.zimbra.com/show_bug.cgi?id=29596 The resolution for this bug would involve an upgrade to 5.0.8 .

  • One customer has reported the upgrade to 5.0.8 has resolved their issue so far. They also confirm that the ics files were being processed with much faster times as logged in mailbox.log
  • Second customer has confirmed upgrade to 5.0.8 has resolved their issue.

2. Check a the thread dump if the message is getting stuck during an invite email delivery to a conference room. You can guess the calendar object based on the emails in the conference room's Inbox.

3. Also check their recurrence expansion configuration in LDAP with:

"zmprov gacf | grep zimbraCalendarRecurrence". 

On a clean install you should see:

zimbraCalendarRecurrenceDailyMaxDays: 730
zimbraCalendarRecurrenceMaxInstances: 0
zimbraCalendarRecurrenceMonthlyMaxMonths: 360
zimbraCalendarRecurrenceOtherFrequencyMaxYears: 1
zimbraCalendarRecurrenceWeeklyMaxWeeks: 520
zimbraCalendarRecurrenceYearlyMaxYears: 100

If these are set to 0, the sysadmin enabled near-infinite expansion on purpose. If these are missing, it's an upgrade problem. The code will default the values to 0 and thus infinite loop. Set them to the above values to avoid long expansions.If these are set to 0, please set to the above values to avoid long exp.

  • One customer has reported that the variables weren't set and they set them to the defaults. Restarted zimbra and issues appear to be resolved. They are holding off on 5.0.8 upgrade at this point.

4. Also you are might hitting bug ( http://bugzilla.zimbra.com/show_bug.cgi?id=28397 - this is a private bug) or something similar like this caused by an offending appointment. You can also find out the mailbox (conference room/user) and put it into maintenance mode to keep the mails flowing. Then try to flush the queue.

Upper Case Hostname Causes Problems With Install/Upgrade

I believe this is new for version 5.0.8+. Until you adjust the case, the installer script will not continue. It's usually picking up the upper case hostname from the server's /etc/hosts entry. Please don't do this, use upper case in your hosts file ... Unix is not Windows.


Ajcody Notes Server Planning

   KB 2494        Last updated on 2008-11-16  




0.00
(0 votes)
24px ‎  - This is Zeta Alliance Certified Documentation. The content has been tested by the Community.


Server Performance Planning & Infrastructure - The Big Picture

Actual Server Planning Homepage

Please see Ajcody-Notes-ServerPlanning

Initial Comments

These are just my random thoughts. They have not been peer-reviewed yet.

Actual DR and Server restore issues are listed in my Ajcody-Server-Topics page.

Items For Your Review


Redundancy - High Availability - And Other Related Questions

One might ask, "Is there any other way to have redundant zimbra email servers?"

The short answer is No, but if you have millions of dollars you can get really close.

Remember, redundancy always comes with a dollar amount cost. And this dollar amount will determine your range of options to implement redundant technologies or processes.

Redundancy isn't magical and all choices have a "time" for the operation to failover and the application being used can restrict what can be done or increase the length of "time" of the operation. Timing of the operation is where the service/server has a client side impact of being "unavailable".

So you break down the server and services into components and put them in a graph or excel sheet. This will help to make sure your not "over engineering" the system.

For example, disk and data.

Disks can use different raid systems to give redundancy. Channels to the disk (remote) can also have redundant paths. The disk chassis can have redundant power supplies which goto different UPS's on different circuit breakers. This data can also be sent to tape, rysnc'd/copied to another disk subsystem on another server, or "flashed" to another location if your filesystem or SAN unit supports it. There's allot that has to fail for you to completely use the data. When exploring these items, you want to have multiple channel paths to make sure copies, rsync's, flashing occurs differently as compared to the "production" path. Have your network backup occur on a different ethernet device.

Two big picture objectives for "redundancy" are "data redundancy" for DR situations and "application" redundancy for service availability. Our "cluster" situation is an active-passive "redundancy" for the purposes of the mailstore application layer. The raid level of the disks on the san servers the "redundancy" in regards to data recovery for DR.

When you introduce the objective of having off-site redundancy the costs and issues become huge. Remote locations introduce speed issue for data transfers which will also impact performance for most applications as it tries to stay in sync between the two machines for write and roll-back purposes.

My intent wasn't so much to give specific answers to this question but rather demonstrate that to answers these questions you have to get down to the real specifics - it's simply impossible to answer the broad question of "how do I make redundant servers". It would take a book to fully explore all the issues and possibilities but it still comes down to - how much money you have to spend on the project. I use to work with High Performance Clusters and HA Linux+Oracle+SAP global deployments with TB's of data - this issue would arise daily for us in those environments.

HA Clustering Software

From the Single Node Cluster Installation Guide - Rev 1 June 2008:

For ZCS 5.0.7 to ZCS 7. With ZCS 8, only Vmware HA is support by Zimbra. RHCS still can work but it would be hardware only monitoring and support should be directed to Redhat for the specifics.
For cluster integration to provide high availability, Zimbra Collaboration Suite (ZCS) can integrate with either of the following:
  • Red Hat® Enterprise Linux® Cluster Suite version 4, Update 5 or later update release. In the single-node cluster implementation, all Zimbra servers are part of a cluster under the control of the Red Hat Cluster Manager.
Note: Red Hat Cluster Suite consists of Red Hat Cluster Manager and Linux Virtual Server Cluster. For ZCS, only Red Hat Cluster Manager is used. In this guide, Red Hat Cluster Suite refers only to Cluster Manager.
  • Veritas™ Cluster Server by Symantec (VCS) version 5.0 with maintenance pack 1 or later.

References:

CPU And Motherboards

Other references: Performance_Tuning_Guidelines_for_Large_Deployments#RAM_and_CPU

This is your mail server, there is no higher profile application in your environment most likely than this box. You'll want to think 3 to 5 years down the road when you spec out the box. Make sure to confirm:

  • It will scale or offer:
    • Hotpluggable technology
      • You'll need to confirm the OS can handle this
    • Redundant and Hot-swap power supplies
      • You might need to upgrade your power supplies depending on the "accessories" you put in it.
    • CPU's (There is NO reason to use a 32bit chip - the memory limitations will kill you)
      • By default, a 32-bit Linux kernel only allows each process to address 2GB of space. Through PAE (Process Address Extension), a feature available in some CPUs, and a special 32-bit kernel that supports large address space for processes, it is possible to get a 32-bit mode kernel that really uses > 4GB of RAM, and get a per process 3-4GB address range
    • Memory
      • What are your onboard cache options?
      • How many slots are available? Does it force you to buy the large memory sticks to max out total memory - this increases cost?
      • Can you mix & match different memory size sticks? (This will increase costs when you goto scale)
      • Understand how memory interacts with multi-cpu's if your server has them
      • Understand the front side bus (FSB)
      • Does the motherboard offline for bad memory detection
    • Have growth for expansion cards and the right kind
      • What is PCI Express?
      • How PCI Express Works?
      • Know what slots you need for your network card, raid card, san card, etc.
        • Then do a sanity check against the motherboard and understand if the cpu and channels can allow the full throughout from all channels.
      • Is there room for redundancy with your expansion cards?

Memory

Other references: Performance_Tuning_Guidelines_for_Large_Deployments#RAM_and_CPU

When I was working with HPC, I found there was never a good reason to NOT at least start with 32GB's of RAM. Now, a mail server isn't a HPC compute node - I understand that. But I would still try to spec out a system that has at least 8 memory slots (usually a ratio of x slots per cpu on system) but allows me to use 4 4GB DIMMS giving me 16GB of memory and allows an upgrade path choice of 4 more 4GB or 8GB DIMMS. Get the higher speed DIMMS, you can't mixed memory of different speeds.

Chart Of Memory Speeds
Memory Interconnect Buses Bit Bytes
PC2100 DDR-SDRAM (single channel) 16.8 Gbit/s 2.1 GB/s
PC1200 RDRAM (single-channel) 19.2 Gbit/s 2.4 GB/s
PC2700 DDR-SDRAM (single channel) 21.6 Gbit/s 2.7 GB/s
PC800 RDRAM (dual-channel) 25.6 Gbit/s 3.2 GB/s
PC1600 DDR-SDRAM (dual channel) 25.6 Gbit/s 3.2 GB/s
PC3200 DDR-SDRAM (single channel) 25.6 Gbit/s 3.2 GB/s
PC2-3200 DDR2-SDRAM (single channel) 25.6 Gbit/s 3.2 GB/s
PC1066 RDRAM (dual-channel) 33.6 Gbit/s 4.2 GB/s
PC2100 DDR-SDRAM (dual channel) 33.6 Gbit/s 4.2 GB/s
PC2-4200 DDR2-SDRAM (single channel) 34.136 Gbit/s 4.267 GB/s
PC4000 DDR-SDRAM (single channel) 34.3 Gbit/s 4.287 GB/s
PC1200 RDRAM (dual-channel) 38.4 Gbit/s 4.8 GB/s
PC2-5300 DDR2-SDRAM (single channel) 42.4 Gbit/s 5.3 GB/s
PC2-5400 DDR2-SDRAM (single channel) 42.664 Gbit/s 5.333 GB/s
PC2700 DDR-SDRAM (dual channel) 43.2 Gbit/s 5.4 GB/s
PC3200 DDR-SDRAM (dual channel) 51.2 Gbit/s 6.4 GB/s
PC2-3200 DDR2-SDRAM (dual channel) 51.2 Gbit/s 6.4 GB/s
PC2-6400 DDR2-SDRAM (single channel) 51.2 Gbit/s 6.4 GB/s
PC4000 DDR-SDRAM (dual channel) 67.2 Gbit/s 8.4 GB/s
PC2-4200 DDR2-SDRAM (dual channel) 67.2 Gbit/s 8.4 GB/s
PC2-5300 DDR2-SDRAM (dual channel) 84.8 Gbit/s 10.6 GB/s
PC2-5400 DDR2-SDRAM (dual channel) 85.328 Gbit/s 10.666 GB/s
PC2-6400 DDR2-SDRAM (dual channel) 102.4 Gbit/s 12.8 GB/s
PC2-8000 DDR2-SDRAM (dual channel) 128.0 Gbit/s 16.0 GB/s
PC2-8500 DDR2-SDRAM (dual channel) 136.0 Gbit/s 17 GB/s
PC3-8500 DDR3-SDRAM (dual channel) 136.0 Gbit/s 17 GB/s
PC3-10600 DDR3-SDRAM (dual channel) 165.6 Gbit/s 21.2 GB/s
PC3-12800 DDR3-SDRAM (dual channel) 204.8 Gbit/s 25.6 GB/s

Bus For Expansion Cards [Peripheral buses]

Typical uses are for network, san, raid cards. I'll deal with them separately under each section below.

Chart Of Bus Speeds
Interconnect Max speed (MB/s) Comments
PCI 2.0 132.0 MB/s
PCI 2.1 264.0 MB/s
PCI 2.2 528 MB/s
PCI-X 1.0 1 GB/s
PCI-X 2.0 4 GB/s
PCI-E (Express) Ver. 1.1 250 MB/s x2 - bi-directional These speeds are bi-directional per "lane". Meaning that they are the same going both ways and not shared.
Ver. 1.1 @ 1x 256 MB/s x2 - bi-directional PCI-E Ver. 1.1 notes.
Ver. 1.1 @ 2x 512 MB/s x2 - bi-directional PCI-E Ver. 1.1 notes.
Ver. 1.1 @ 4x 1 GB/s x2 - bi-directional PCI-E Ver. 1.1 notes.
Ver. 1.1 @ 8x 2 GB/s x2 - bi-directional PCI-E Ver. 1.1 notes.
Ver. 1.1 @ 16x 4 GB/s x2 - bi-directional PCI-E Ver. 1.1 notes.
PCI-E (Express) Ver. 2.0 400 MB/s x2 - bi-directional 500 MBs but there's a 20% overhead hit. These speeds are bi-directional per "lane". Meaning that they are the same going both ways and not shared.
Ver. 2 @ 1x 400 MB/s x2 - bi-directional PCI-E Ver. 2 notes.
Ver. 2 @ 4x 1600 MB/s x2 - bi-directional PCI-E Ver. 2 notes.
Ver. 2 @ 8x 3200 MB/s x2 - bi-directional PCI-E Ver. 2 notes.
Ver. 2 @ 16x 6400 MB/s x2 - bi-directional PCI-E Ver. 2 notes.
PCI-E (Express) Ver. 3.0 1 GB/s x2 - bi-directional The final spec is due in 2009. These speeds are bi-directional per "lane". Meaning that they are the same going both ways and not shared.
Front-side Bus replacements below
HyperTransport 1.x 6.4 GB/s x2 - bi-directional Bidirectional per 32 bit link at 800MHz. Front-side bus replacement, see HyperTransport for more details .
HyperTransport 2.0 11.2 GB/s x2 - bi-directional Bidirectional per 32 bit link at 1.4GHz. Front-side bus replacement, see HyperTransport for more details .
HyperTransport 3.0 20.8 GB/s x2 - bi-directional Bidirectional per 32 bit link at 2.6GHz/ Front-side bus replacement, see HyperTransport for more details .
HyperTransport 3.1 25.6 GB/s x2 - bi-directional Bidirectional per 32 bit link at 3.2GHz. Front-side bus replacement, see HyperTransport for more details .
QuickPath Interconnect 12.8 GB/s x2 - bi-directional Intel competitor to HyperTransport. Everything you wanted to know about QuickPath (QPI)

Network Infrastructure

Network Cards

Most motherboards will have integrated ethernet ports. If they are the same chipset then you might want to just channel bond these. What I have done in the past, is used them for management ports and added in network cards for production activity.

Ideally, I would buy two cards that had two ports (Gb). I would then channel bond 2 of the ports across the cards, for 2 separate bondings. I would use one of those bonds for "front facing" traffic and the other for "backup" traffic. Remember to consider the bus infrastructure and other cards when deciding on what ones to get.

Channel Bonding

This will require you to confirm your switches can do this. There are different choices when it comes to channel bonding and some of them require the switch to support it - it usually involves "bonding" on the switch side by configuring the ports in question.

The Production Channel Bond

The "production" bond is more in regards to failover. Make sure this happens as expected through the application layer. That proxies, firewall, and so forth allow on of the ports in the channel bond to go down without any end-user impact.

The Backup Channel Bond

The backup port is for throughput. You'll want to map out the network path and switches to the end host that data is being moved to. You'll also need to confirm the network path actually gives you a gain in performance by doing the channel bonding.

This will give you an excellent way to off load your tape backups, rsyncs (as shown at Ajcody-Notes-ServerMove ), and maybe nfs mounts if your using them.

You'll want to map out the network path and switches to the end host the data is being moved to. You'll also need to confirm the network path actually gives you a gain in performance by doing the channel bonding.

Disk And Data Infrastructure

Other references: Performance_Tuning_Guidelines_for_Large_Deployments#Disk

Disk Types


References


Description of Items Used Below For Specific HDD's:

  • Notable Items
    • Benefits
    • Downsides
  • Bus Interface
    • Types:
    • Maximum Devices:
    • Bus Speeds:
  • Performance
    • I/Os per second & Sustained Transfer Rate:
    • Spindle Speed: This is the speed the drives disk actually spins at. This ranges from 5400rpm to 15,000rpm. The higher the speed the more often the data on the disk will be in the right position to be read by the drive heads, and the faster data can be transferred.
    • Average Access Time: Time. This is the average time is takes to position the heads so that data can be read. The faster the better.
    • Cache Size: This is the size of the cache on board the disk drive itself. This does reach a point where doubling the size generates a very small boost in performance and is not worth the cost, but generally the bigger the better. It also assists in small bursts of data where you may actually achieve near maximum performance as data is read from cache rather than the drive.
    • Internal Transfer Rate: This is the speed that data can be transferred within the drive. This speed will be higher than the actual transfer rate of the drive as there is some overhead for protocol handling as data is transferred to the SCSI or IDE bus.
    • Latency: the time it takes for the selected sector to be positioned under the read/write head. Latency is directly related to the spindle speed of the drive and such is influenced solely by the drive's spindle characteristics.
  • Reliability (MTBF or unrecoverable read error rate)
  • Capacity
  • Price
Ultra DMA ATA HDD's
  1. Notable Items
    • Benefits
    • Downsides
  2. Bus Interface
    • Types : ATA
  3. Performance
    • I/Os per second & Sustained transfer rate:
      1. Ultra DMA ATA 33 - 264 Mbit/s 33 MB/s
      2. Ultra DMA ATA 66 - 528 Mbit/s 66 MB/s
      3. Ultra DMA ATA 100 - 800 Mbit/s 100 MB/s
      4. Ultra DMA ATA 133 - 1,064 Mbit/s 133 MB/s
    • Spindle Speed:
    • Average Access Time:
    • Cache Size:
    • Internal Transfer Rate:
    • Latency:
  4. Reliability (MTBF or unrecoverable read error rate)
  5. Capacity
  6. Price
SATA HDD's
  1. Notable Items
    1. Benefits
      • SATA drives typically draw less power than traditional SAS HDDs due to slower RPM speeds.
      • SATA drives have the best dollar per gigabyte compared to SAS drives.
      • SATA HHDs can work on a SAS interface.
    2. Downsides
      • SATA HDDs are single port and not capable of being utilized in dual port environments without the addition of an interposer designed for this purpose.
  2. Bus Interface Type
    • Types: SATA , SAS
  3. Performance
    • I/Os per second & Sustained transfer rate:
      1. Serial ATA (SATA-150) - 1,500 Mbit/s 187.5 MB/s
        • Real speed: 150 MB/s
      2. Serial ATA (SATA-300) - 3,000 Mbit/s 375 MB/s
        • Real speed: 300 MB/s
        • (alternate names: SATA II or SATA2)
      3. Serial ATA (SATA-600) - 4,800 Mbit/s 600 MB/s
        • I've seen this listed as well though, SATA 6.0 Gbit/s (SATA 6Gb/s).
        • Standard is expected to be available before the end of 2008.
    • Spindle Speed: 7200 RPM , 5400 RPM
    • Average Access Time:
    • Cache Size:
    • Internal Transfer Rate:
    • Latency:
  4. Reliability (MTBF or unrecoverable read error rate)
  5. Capacity
  6. Price
SCSI (Parallel SCSI) HDD's
  1. Notable Items
    • Benefits
    • Downsides
  2. Bus Interface Type
    • Types: SCSI
    • Maximum Devices (On Single Channel):
      • Ultra Wide SCSI - 16
      • Ultra2 SCSI - 8
      • Ultra2 Wide SCSI - 16
      • Ultra3 SCSI - 16
        • (alternate names: Ultra-160, Fast-80 wide)
      • Ultra-320 SCSI - 16
        • (alternate name: Ultra4 SCSI)
      • Ultra-640 SCSI - 16
    • Bus Speeds:
      • The total amount of data that can be transferred throughout the whole channel.
      • Data transfers will step down to the rate speed of the drive. If you have a Ultra160 HDD on a Ultra320 controller it operate at 160 MB/s.
      • SCSI-3 : Also known as Ultra SCSI and fast-20 SCSI. Bus speeds at 20 MB/s for narrow (8 bit) systems and 40 MB/s for wide (16-bit).
      • Ultra-2 : Also know as LVD SCSI. Data transfer to 80 MB/s.
      • Ultra-3 : Also known as Ultra-160 SCSI. Data transfer to 160 MB/s.
      • Ultra-320 : Data transfer to 320 MB/s.
      • Ultra-640 : Also known as Fast-320. Data transfer to 640 MB/s.
  3. Performance
    • I/Os per second & Sustained Transfer Rate:
      • Ultra Wide SCSI 40 (16 bits/20MHz) - 320 Mbit/s 40 MB/s
        • Real speed: 40 MB/s
      • Ultra2 SCSI
        • Real speed: 40 MB/s
      • Ultra2 Wide SCSI 80 (16 bits/40 MHz) - 640 Mbit/s 80 MB/s
        • Real speed: 80 MB/s
      • Ultra3 SCSI 160 (16 bits/80 MHz DDR) - 1,280 Mbit/s 160 MB/s
        • (alternate names: Ultra-160, Fast-80 wide)
        • Real speed: 160 MB/s
      • Ultra-320 SCSI (16 bits/80 MHz DDR) - 2,560 Mbit/s 320 MB/s
        • (alternate name: Ultra4 SCSI)
        • Real speed: 320 MB/s
      • Ultra-640 SCSI (16 bits/160 MHz DDR) - 5,120 Mbit/s 640 MB/s
        • Real speed: 640 MB/s
    • Spindle Speed:
    • Average Access Time:
    • Cache Size:
    • Internal Transfer Rate:
    • Latency:
  4. Reliability (MTBF or unrecoverable read error rate)
  5. Capacity
  6. Price
SAS (Serial Attached SCSI) HDD's
  1. Notable Items
    1. Benefits
      • SAS HDDs are true dual port, full duplex devices. This means SAS HDDs can simultaneously process commands on both ports.
      • All SAS HDDs are hot-swap capable. Users can add or remove an HDD without disrupting the enterprise environment.
      • SAS HDDs can support online firmware update (check vendor). This allows users to update firmware on the SAS HDD without having to schedule downtime.
    2. Downsides
      • SAS HDDs cannot be used on the older architecture SCSI backplanes or cables.
      • SAS HDDs typically draw more power than the equivalent SATA counterparts.
  2. Bus Interface Type
    • Types: SAS
    • Maximum Devices:
      • SAS 16,256
        • 128 devices per port expanders
    • Bus Speeds:
  3. Performance
    • I/O per second & Sustained transfer rate:
      • Serial Attached SCSI (SAS) - 3,000 Mbit/s 375 MB/s
        • Real Speed: 300 MB/s (full duplex , per direction)
      • Serial Attached SCSI 2 - 6,000 Mbit/s 750 MB/s
        • Planned
    • Spindle Speed: 15000 RPM , 10000 RPM , 7200 RPM
    • Average Access Time:
    • Cache Size:
    • Internal Transfer Rate:
    • Latency:
  4. Reliability (MTBF or unrecoverable read error rate)
    • 10K & 15K SAS HDDs has been rated at 1.6 million hours MTBF
  5. Capacity
  6. Price
Fibre Channel - Arbitrated Loop (FC-AL) HDD's
  1. Notable Items
    1. Benefits
      • FC-AL HDDs are dual ported, providing two simultaneous input/output sessions
      • FC-AL HDDs are hot-swap capable so users can add and remove a hard drives without interrupting system operation.
    2. Downsides
      • FC-AL HDDs are typically utilized in unique environments and are not compatible to be used on SAS or SATA interfaces
      • Long term, SAS is projected to replace FC-AL HDDs within the IT industry
  2. Bus Interface Type
    • Types: FC-AL
    • Maximum Devices:
      • FC-AL in a private loop with 8-bit ID's - 127
      • FC-AL in a public loop with 24-bit ID's - +16 million
    • Bus Speeds:
  3. Performance
    • I/Os per second & Sustained transfer rate:
      • FC-AL 1Gb 100 MB/s (full duplex , per direction)
      • FC-AL 2Gb 200 MB/s (full duplex , per direction)
      • FC-AL 4GB 400 MB/s (full duplex , per direction)
    • Spindle Speed:
    • Average Access Time:
    • Cache Size:
    • Internal Transfer Rate:
    • Latency:
  4. Reliability (MTBF or unrecoverable read error rate)
  5. Capacity
  6. Price

Raid Cards (Not Raid Level)


  • If your raid chip on your motherboard goes out, what is your expect downtime to resolve it?
    • Isn't it easier to buy a hot space raid card and replace it in case of a raid card failing?

SAN Typologies - Interfaces


General Questions
San Cards
  • Have you planned on multipathing?
    • You do have more than one port/card, right?
      • If not (because of cost), you do have an open slot on motherboard to allow another one?
      • Remember to consider this with your SAN switch purchase
    • Confirm that your HBA's, HBA drivers, SAN switches allow for the options you want?
    • multipathing can be for failover
    • multipathing can increase throughput for same partition mount
iSCSI Interface

So far, physical devices have not featured native iSCSI interfaces on a component level. Instead, devices with SCSI Parallel Interface or Fibre Channel interfaces are bridged by using iSCSI target software, external bridges, or controllers internal to the device enclosure. Source

  • iSCSI over Fast Ethernet 100 Mbit/s 12.5 MB/s
  • iSCSI over Gigabit Ethernet 1,000 Mbit/s 125 MB/s
  • iSCSI over 10G Ethernet (Very few products exist) 10,000 Mbit/s 1,250 MB/s
  • iSCSI over 100G Ethernet (Planned) 100,000 Mbit/s 12,500 MB/s
iSCSI Performance Topics

One will find the following recommendations in regards to iscsi:

  • Private network for iscsi traffic
  • Jumbo frames enabled with MTU size of 9000
    • All equipment must support and be enabled for this from point to point.
      • Remember, point to point meaning the "server" to the host serving the iscsi disk array.
iSCSI References
Serial Attached SCSI (SAS) Interface
  1. Notable Items
    1. Benefits
      • SAS interface protocol supports both SAS and SATA Hard Disk Drives allowing tiered storage management.
      • SAS supports seamless scalability through port expansion enabling customers to daisy chain multiple storage enclosures.
      • SAS supports port aggregation via a x4 wide-link for a full external connection bandwidth of up to 12.0 Gbps (1200MBps) on a single cable and a single connection.
      • SAS is a point-to-point interface allowing each device on a connection to have the entire bandwidth available. The current bandwidth of each SAS port is 3Gb/sec. with future generations aimed at 6Gb/sec and beyond.
    2. Downsides
      • SAS is not backwards compatible with U320 SCSI or previous SCSI generations
Fibre Channel - Arbitrated Loop (FC-AL)
  1. Notable Items
    1. Benefits
      • FC-AL devices can be dual ported, providing two simultaneous input/output sessions that doubles maximum throughput
      • FC-AL enables "hot swapping," so you can add and remove hard drives without interrupting system operation, an important option in server environments.
    2. Downsides
      • FC-AL adapters tend to cost more than SAS adapters
      • FC-AL is currently the fastest interface at 4Gb but is expected to be passed in maximum bandwidth by the next generation of SAS interface at 6Gb

Multipathing


General References
Multipathing And SAN Persistent Binding
Multipathing And LVM
Multipathing - Redundant Disk Array Controller [RDAC] (LSI mpp driver)

References:

Multipathing - Fibreutils - sg3_utils (QLogic)

References:

Multipathing - lpfc (Emulex)

References:

Multipathing - device-mapper [dm] & multipath-tools (Linux OSS driver & tools)

DM is the open source multipath driver.

References:

Multipathing - Veritas (Symantec)

References:

Multipathing - HP & Linux

References:

Multipathing - NetApp

References:

Multipathing - EMC (PowerPath)

References:

Raid Levels For Disks


General References

References:

Raid 10

RAID 10 (or 1+0) uses both striping and mirroring.

Reference - http://bugzilla.zimbra.com/show_bug.cgi?id=10700 - I fixed statement based upon a private comment in this RFE.

Zimbra recommends Raid 10 for the :
Mailstore and Logger [MySQL] databases - /opt/zimbra/db & /opt/zimbra/logger
Indexing Volume [Lucene] database - /opt/zimbra/index
Raid 10 is NOT Raid 0+1
RAID 0+1 takes 2 RAID0 volumes and mirrors them. If one drive in one of the underlying RAID0 groups dies, that RAID0 group is unreadable. If a single drive simultaneously dies in the other RAID0 group, the whole RAID 0+1 volume is unreadable.
RAID 1+0 takes "mirrored slices" and build a RAID0 on top of them. A RAID10 volume can lose a maximum of half of its underlying disks at once without data loss (as long as only one disk from each "slice pair" is lost.)
Raid 5

RAID 5 (striped disks with parity) combines three or more disks in a way that protects data against loss of any one disk; the storage capacity of the array is reduced by one disk.

Reference - http://bugzilla.zimbra.com/show_bug.cgi?id=10700

Zimbra does NOT recommend RAID 5 for the MySQL database and Lucene volumes (RAID 5 has poor write performance, and as such is generally not recommended by MySQL, Oracle, and other database vendors for anything but read-only datastores. We have seen order of magnitude performance degradation for the embedded Zimbra database running on RAID 5!).
Mailstore and Logger [MySQL] databases - /opt/zimbra/db & /opt/zimbra/logger
Indexing Volume [Lucene] database - /opt/zimbra/index
Raid 6

RAID 6 (less common) can recover from the loss of two disks. Basically an extension of Raid 5.

Filesystem Choices


Other references:

Bugs/RFE's that reference filesystems:

Most configuration go with ext3 because of it's implied default. My own experience has me using xfs always except for the root ( / ) partition.

Filesystem Feature And Option Descriptions
Journaling
Inodes

To see existing inode use:

# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/VolGroup00-LogVol00
                     19005440  158715 18846725    1% /
/dev/sda1              26104      39   26065    1% /boot
Block Size
Volume Managers

LVM / LVM2 (Linux)
EVMS (IBM's Enterprise Volume Management System)
Veritas Volume Manager
Regular File Systems

XFS

Supported

  • 1. Bugs and RFE's
  • 2. Journaling
    • A. "XFS uses what is called a metadata journal. Basically, this means that every disc transaction is written in a journal before it is written to the disc and then marked as "done" in the journal when it finishes. If the system crashes during the writing of the journal entry, that incomplete entry can be ignored since the data on the disc has not been touched yet and if the journal entry is not marked done, then that operation can be rolled back to preserve disc integrity. Its a very nice system. As stated above, XFS practices a type of journaling called "metadata journaling." This means only the inodes are journaled, not the actual data. This will preserve the integrity of the file system, but does not preserve the integrity of the data." reference: Filesystem Design Part 1 : XFS
    • B.
  • 3. Inodes
    • A. "FS considers dynamic allocation of inodes and keeps track of such inodes using B+Trees. Each allocation group uses a B+Tree to index the locations of inodes in it. This allows to create millions of inodes in each allocation group and thus supporting large number of files" reference: PDF Warning - Failure Analysis of SGI XFS File System
    • B.
  • Resources:
EXT3 - EXT4

Supported

Reiser
vxfs - Veritas
  • 1.
    • A.
    • B.
  • 2.
    • A.
    • B.
  • 3. Dynamic inode allocation
    • A. A description of sorts of what this means, "Vxfs is dynamic, so the inodes are basically created on the fly. However, it's a bit inaccurate to actually state that inodes are dynamic in VxFS and leave it at that (somewhat confusing I know). Vxfs really doesn't use them per se. It creates them to be compatible with UFS. VxFS uses extent-based allocation rather than the block-based UFS controlled by inodes. So, the question of how many inodes for vxfs, is as many as it needs."
    • B. http://www.docs.hp.com/en/B3929-90011/ch02s04.html
  • Resources:
BTRFS - Oracle's Better File System for Linux (with Redhat, Intel, HP)


ZFS
  • 1.
    • A.
    • B.
  • 2.
    • A.
    • B.
  • Resources:
Network File Systems

NFS
  • 1. Only supported for the Zimbra backup directory or when used internally to Vmware vSphere at this time. Please see Bug/RFE and comments below.
  • 2.
    • A.
    • B.
  • Resources:
  • Bugs/RFE's:
    • "Need clarity on supporting nfs mounted zimbra directories - report error/msg if nfs mount is present"
    • "Zimbra on NFS Storage through VMware ESX"
      • http://bugzilla.zimbra.com/show_bug.cgi?id=50635
      • Note - I asked this rfe to become private since it is really an internal request to do testing/qa'ing of nfs with vSphere. I asked another rfe for public viewing to be made that will let customers know when that can deploy with it under production use.

Purposed inclusion to release notes from Bug/RFE above:

ZCS & NFS:
Zimbra will support customers that store backups (e.g. /opt/zimbra/backup) on an NFS-mounted partition. Please note that this does not relieve the customer from the responsibility of providing a storage system with a performance level appropriate to their desired backup and restore times. In our experience, network-based storage access is more likely to encounter latency or disconnects than is equivalent storage attached by fiber channel or direct SCSI.
Zimbra continues to view NFS storage of other parts of the system as unsupported. Our testing has shown poor read and write performance for small files over NFS implementations, and as such we view it unlikely that this policy will change for the index and database stores. We will continue to evaluate support for NFS for the message store as customer demand warrants.
When working with Zimbra Support on related issues, the customer must please disclose that the backup storage used is NFS.
Samba - SMB/CIFS
Global , Distributed , Cloud, Cluster-Type Filesystems - Unsupported

Currently, Zimbra does not support or recommend the use of the various filesystems listed under this section. Please see the specific section to see if I've identified any existing bugs/RFE's against them. One general RFE for this topic is:


GFS
Lustre [Acquired by Sun]
Hadoop Distributed File System (HDFS) - Cluster Filesystem Project From Apache & Yahoo!
SGI CXFS
IBM GPFS
Veritas Storage Foundation Cluster File System
Google Computing

Unsupported at this time by Zimbra Support.

References:

Amazon S3 - And Amazon EC2 Information

Unsupported at this time by Zimbra Support.

References:

Still going to try this when it's not supported? See the following if you run into troubles and if your doing a new setup:

Other Products - Applications (DRBD, Veritas Volume Replicator, etc.)

Unsupported at this time by Zimbra Support.


Background Items To Review Before Drawing Up A Plan


When you start putting all this together you'll find there's allot of "exceptions" to work out based upon your needs, equipment, kernel version, distro choice, SAN equipment, HBA's, OSS drivers vs. vendor ones, and so forth. You must TEST all your assumptions when your planning on using methods that will provide online filesystem growth. Don't deploy just on an assumption. Some situations will not allow a true online filesystem growth.

References to review:

Online Resizing Of LUN Example 1 - Multipathing Required
Warning, this process reminds me of what I was doing over a year ago [before zimbra]. I don't have the necessary hardware or my old notes to go through the steps and confirm.
  1. Resize the SAN volume.
  2. Reinitialize the HBA, e.g. using sg_reset or some module specific method.
  3. Rescan the SCSI device:
    • echo 1 > /sys/block/[sdx]/device/rescan.
  4. Now confirm that /proc/partitions should contain updated values. The fdisk and sfdisk commands may still (most likely) see the old values.
  5. Remove and readd the SCSI Devices.
    • Warning: Without multipathing you are going to loose access to your disk!! Multipathing gives you several paths to the volume, so you will not loose access. Make sure multipathd is up and running.
      • echo "scsi remove-single-device <Host> <Channel> <SCSI ID> <LUN>" > /proc/scsi/scsi
      • echo "scsi add-single-device <Host> <Channel> <SCSI ID> <LUN>" > /proc/scsi/scsi
    • This will also issue new device names!
      • Multipathing updated it's device size automatically after I reloaded the last path-device.
  6. Run:
      • If you have LVM: vgresize, lvresize if you have a lvm setup.
        • Need to double check for the need of pvresize & vgextend use first.
      • Your filesystem online grow commands if you have just a filesystem.
Q&A
  • Q. The use of multipathing failover has the block device being changed in size, potentially during a write (this was, after all, online). What happens if data is being written to the device while you're doing the disconnect/reconnect operations? Do you have two conflicting pieces of information about the device?
    • A. The write is being done at the "filesystem" level, which is unaware of the block device size change.

Example Filesystem Layouts


Local (Server) Disk Configuration

I would have 3 disk subsystems at a minimum. Meaning, three distinction group of disks on their own disk i/o backplane, if possible. For example:

  1. Two disks using the motherboard provided SATA/ATA ports. (OS)
  2. Multiple disks accessed through a dedicated SAN or SCSI card. Maybe these are the hot swap disk array available on the front of your server.(Backup)
  3. Multiple disks accessed through another dedicated SAN or SCSI card. Maybe these are the disks available from an external disk array on your server or to external disks via SAN/iSCSI. (zimbra)
OS - /boot , swap , /tmp , /
Your first disk is referred to as disk1 or sda. Your second disk is referred to as disk2 or sdb. Partitions will start at 1 , rather than 0, throughout my examples below.
For the OS partitions, there's no reason to use anything but "default" for the mounting options in the /etc/fstab. If you reviewed the Performance_Tuning_Guidelines_for_Large_Deployments#File_System wiki page, those settings would apply to a dedicated partition(s) used for zimbra - not the root OS level partitions.
  1. For OS ( /boot , swap, /tmp , / )
    • A. This could be whatever for disks (SATA, SCSI, etc.).
      • 1. Two Disk minimum for mirroring. Four Disks would allow raid 10.
        • A. I prefer to use software raid, because then the "raid" will move with the disks with less complications if my server dies. I can simply move the disks into another server chassis.
        • B. Most motherboards will have at least two SATA ports and drive slots. Make sure you put your drive on different channels if you have the options rather than being on the same.
    • B. /boot partition
      • 1. I would setup a /boot partition (128MB or 256MB) on drive. This would be my first partition on each drive.
        • A. Disk 1 > partition type linux > partition 1 for 128MB or 256MB > filesystem ext3 > no "raiding" > NO LVM > mount point /boot
        • B. Disk 2 > partition type linux > partition 1 for 128MB or 256MB > filesystem ext3 > no "raiding" > NO LVM > mount point /boot2
        • 2. After OS install, I would then "rsync/copy" the contents of /boot into /boot2
        • 3. Configure grub or your bootloader to have the option to boot from /boot2 (disk1/partition0)
          • A. This will give you a failover in case something goes wrong with /boot (disk0/partition0). Bad kernel upgrade or a wacked partition maybe.
        • 4. After any changes to /boot (disk1/partition1) confirm everything works right (confirm reboots fine), you would then do another manual rsync/copy of /boot to /boot2 (disk2/partition1).
    • C. swap
      • 1. Determine how much swap you "need". I will use an example of 2GB's below. Notice both "swap" partitions will get 2 GB's.
      • 2. Setup a swap partition on disk1/partition2 and disk2/partition2 for 2 GB's.
        • A. Disk 1 > partition type - swap > partition 2 for 2GB > filesystem swap > no "raiding" > NO LVM > mount point swap
        • B. Disk 2 > partition type - swap > partition 2 for 2GB > filesystem swap > no "raiding" > NO LVM > not set to mount
      • 3. *Default Suggestion* Configure the OS to only use/mount swap on disk1/partition2. You can configure /etc/fstab for the other partition but just comment out the line for now.
      • 4. Reasoning for this.
        • A. swap drives that are "mirror" can cause undue complications.
        • B. I configured a "same" sized swap on the other drive really for the third partition - that for /. This way the blocking/sizes are as close as possible for the / mirror.
        • C. It isn't really a "lose" of space, but rather for adjustments you might need later.
          • 1. It can server as a "failure" over swap partition incase things go bad on disk0/partition1.
          • 2. It can server for more "production" swap if you find you need it.
          • 3. It allows for a complete disk failover or simplicity in case you need to move the "one" disk to another box.
        • 5. You could also configure the two swap partitions into a raid0 if you would like. Swap partitions can be turned on/off (swapon , swapoff). It's easy enough to reformat them as well, if something goes wrong, after a swapoff.
    • D. /tmp
      • 1. I generally never setup a partition for /tmp , but if you did decide to do this make sure it's following the swap partitions. ( IMO )
        • A. If you do setup /tmp , I would go with ext3 or xfs and put it within LVM. See notes below about LVM use.
    • E. / - Introduction of LVM
      • 1. You'll now place the rest of the "free" disks under a "software" raid1 / mirror partition.
        • A. You should then see this "mirror" as a new disk/partition to be used within the OS installer.
      • 2. Place the "mirror" under LVM - this will be a "partition" type. Let's assume this new mirror is now /dev/md0.
      • 3. Now configure the LVM partition for LVM. I'm assuming that a partition wasn't made for /tmp
        • A. General concepts of what happens with the LVM setup and recommendations on the naming scheme. Sda3 is disk1/partion3 and sdb3 is disk1/partition3
          • 1. pvcreate md0
          • 2. vgcreate -s 128M rootvg md0
            • A. vgchange -a y rootvg
          • 3. lvcreate -L 150GB -n rootlv rootvg
            • A. I put in 150GB above as an example. I would normally put in about 90% of the available space left and leave me some room to create a new partition/mount point if it becomes necessary. For example, let's say / kept filling up because of lazy admins not cleaning up in /root. I could create a new vg group called roothomevg and mount it as /root , restricting the messy admins from effecting / .
          • 4. Now you would make the filesystem.
            • A. Example of setting up ext3 filesystem using defaults.
              • 1. mkfs.ext3 /dev/rootvg/rootlv
          • 5. Now you need to setup the new filesystem in /etc/fstab to mount. I'll use /root as an example, considering the / would of been done through the installer and you would of most likely used the GUI tools to have done all of this.
            • A. mkdir /root (if it didn't exist - this example would of of course)
            • B. vi /etc/fstab
              • 1. /dev/rootvg/rootlv /root defaults 1 2
                • A. Adjust for your system defaults in regards to the device naming convention that's being used.
      • 4. So why did I bother with LVM if I used all the disk space anyways?
        • 1. If you setup the / with LVM at the start, even if you use all the disk space, it allows you in the future to add more disk to the underlying LVM setup to grow the / filesystem - online.
          • A. For example, let's say I have 2 open drive bays that weren't initially used when I setup my server. And two years later I find my / becoming very close to 100%. I can throw in two new drives into those bays (assuming hot-swap drives). Setup a mirror (mdadm) between the two drives. Set the new mirror partition type to LVM. Then run through the pvcreate , vgextend , lvresize , and then online grow the filesystem (ext2online/resize2fs , xfs_growfs , etc.)
Backup
  1. /opt/zimbra/backup
    • A. I would make sure the disk I/O is separate from /opt/zimbra . This way you minimize performance hits to your end-users. Do a review of the disk i/o bus path is clean as possible to the cpu/memory. Motherboard spec's should tell you what "slots" are on shared buses. Make sure your maximizing your raid/san cards performance to the bus path to the cpu.
    • B. I would make this a multiple of some degree of the /opt/zimbra space that your spec'ing. This default backup schedule is to purge sessions older than one month. This means you'll have 4 fulls and 24 incremental sessions.
      • 1. Make sure you investigate the auto-group backup method as well and the zip option. This could have a huge effect on your disk space requirements for the backup partition.
    • C. If disks are local
      • 1. Raid Choices
        • A. If you'll be sending this to tape as well, you could go with a raid0 for performance.
        • B. If you don't plan on sending to tape or another remote storage system, maybe raid5.
      • 2. LVM
        • A. Please encapsulate the backup partition under LVM. This will give you some flexibility later in case you need more space.
      • 2. Other Topics
    • D. If disks are on SAN
      • 1. Raid Choices
        • A. If you'll be sending this to tape as well, you could go with a raid0 for performance.
        • B. If you don't plan on sending to tape or another remote storage system, maybe raid5.
        • C. Note, if you have allot of disks to construct your raid with you still can achieve good performance with raid5. This way you don't have to lose all the "disks" when doing raid10. It would be worth bench marking your SAN using x number of disks configured as raid10 vs. the same x number of disks are raid5. Remember to consider the i/o backplanes involved in what "disks" you choose to use throughout your different disk chassis. Going up and down your disk rack vs left to right.
      • 2. LVM
        • A. Please encapsulate the backup partition under LVM. This will give you some flexibility later in case you need more space.
        • B. If your NAS/SAN system is going to do block level snap-shots, the choice to use LVM or not becomes more complicated. A block level snap-shot across multiple LUN's will generally not work when the top level filesystem is using LVM. If you plan on only using one LUN and growing that same LUN as needed, then LVM will still prove useful if your also using the SAN/NAS block level snap-shots.
      • 2. Other Topics
Zimbra
  1. /opt/zimbra
    • A. Remember you having logging data in here as well. If this partition becomes full, zimbra will hang and could cause database corruption as well.
    • C. If disks are local
      • 1. Raid Choices
        • A. Raid 0 or Raid 10
        • B.
      • 2. LVM
        • A. Please encapsulate the backup partition under LVM. This will give you some flexibility later in case you need more space.
        • B.
    • D. If disks are on SAN / NAS
      • 1. Raid Choices
        • A. Raid 0 or Raid 10
        • B. Note, if you have allot of disks to construct your raid with you still can achieve good performance with raid5. This way you don't have to lose all the "disks" when doing raid10. It would be worth bench marking your SAN using x number of disks configured as raid10 vs. the same x number of disks are raid5. Remember to consider the i/o backplanes involved in what "disks" you choose to use throughout your different disk chassis. Going up and down your disk rack vs left to right.
      • 2. LVM
        • A. Please encapsulate the partitions under LVM. This will give you some flexibility later in case you need more space.
        • B. If your NAS/SAN system is going to do block level snap-shots, the choice to use LVM or not becomes more complicated. A block level snap-shot across multiple LUN's will generally not work when the top level filesystem is using LVM. If you plan on only using one LUN and growing that same LUN as needed, then LVM will still prove useful if your also using the SAN/NAS block level snap-shots.
      • 2. Other Topics
More Details About LVM Use

The notes below were gathered from the "Zimbra Admins in Universities" mailing list.

Contents of Post by Matt on Date: Mon, 27 Oct 2008 14:24:50 -0500

We've found however that we are able to grow storage on the fly with LVM.  
It basically works like this for us...

    *  Grow the LUN on the SAN
          o wait for the LUN to finish growing by checking the 'Jobs / Current Jobs' 
            display until the "Volume Initialization" job is finished. 
    * reboot the host to see the new partition size.
          o (partprobe -s is supposed to do this, but it doesn't) 
    * find the device name:
          o pvs | grep VolumeName? 
    * grow the Volume Group:
          o pvresize -v /dev/XXX 
    * Verify the new size:
          o pvs | grep VolumeName? 
    * grow the logical volume:
          o Grow by a specific size: lvextend --size +NNg /dev/VgName/LvName
          o Grow to use all free space: lvextend -l +100%FREE /dev/VgName/LvName 
    * grow the file system:
          o Online method (dangerous?)
                + ext2online /dev/VgName/LvName 
          o Offline method (safer?)
                + umount /mountpoint
                + e2fsck -f /dev/Vgname/Lvname
                + resize2fs /dev/Vgname/LvName
                + mount /dev/VgName/LvName /mountpoint 
    * Verify new filesystem size:
          o df -h /mountpoint 

I've always used the online method (marked "dangerous?" by one of my cohorts) and 
never had a problem.  One other thing we've been able to do with LVM that has been 
a benefit is migrating data to a new LUN...

   1.  Find the new physical volume that is associated with the correct LUN#. On 
       the Zimbra servers you can use this MPP (linuxrdac) tool.

      # /opt/mpp/lsvdev

   2. Prepare the physical volume with PVCREATE.

      # pvcreate /dev/sdX

   3. Extend the logical volume to the new physical volume with VGEXTEND.

      # vgextend /dev/VolGroupName /dev/sdX

   4. Use LVDISPLAY to make sure you are moving from the right physical volume.

      # lvdisplay /dev/VolGroupName/LogVolName -m

      Example Results
      ===========
        --- Logical volume ---
        LV Name                /dev/VgMbs03Backup/LvMbs03Backup
        VG Name                VgMbs03Backup
        LV UUID                0vZQx3-5A22-a4ZO-4VmV-2naM-jwoi-yc6r6k
        LV Write Access        read/write
        LV Status              available
        # open                 0
        LV Size                580.00 GB
        Current LE             148479
        Segments               1
        Allocation             inherit
        Read ahead sectors     0
        Block device           253:6
         
        --- Segments ---
        Logical extent 0 to 148478:
          Type                linear
          Physical volume     /dev/sdab
          Physical extents    0 to 148478

   5. Move the Volume Group to the new physical volume using PVMOVE.

      # pvmove -i 60 -v /dev/sdZ /dev/sdX

      -i 60     : Show progress every 60 seconds
      -v         : Verbose
      /dev/sdZ     : Physical volume we are moving from
      /dev/sdX     : Physical volume we are moving to

   6. When the move is completed use VGREDUCE to reduce the volume group down to 
      the new physical volume.

      # vgreduce /dev/VolGroupName /dev/sdZ

A reply was given to the post above, noting the issue of the reboot in the steps above. Rich, Mon, 27 Oct 2008 15:24:06 -0500, write:

Any process including the term "reboot" isn't "on the fly." :-)

Current proprietary OSes can rescan and use expanded LUNs on the fly while 
filesystems are mounted. Apparently, so can the latest development Linux kernel,
but stacked device-mapper and LVM layers will need major changes, so don't 
expect to see this capability in enterprise linices for 2 years.

You can save some time, though, by replacing "reboot" with the minimum steps 
required to clear all holders of the device:

umount /file/system
vgchange -an VG
service multipathd stop
multipath -ll # note the physical devices involved, here assuming sd[fg]
multipath -f mpathVG
echo 1 > /sys/block/sdf/device/rescan # partprobe -s might also do this
echo 1 > /sys/block/sdg/device/rescan
multipath -v2
service multipathd start

...and continue with the pvresize. But simply adding a new LUN and marking it 
active with the admin console (or zmvolume) can be done with zero downtime, 
so that's my new model.

SAN Layout As Recommend For Clusters

This is from Multi Node Cluster Installation Guide - PDF

Preparing the SAN

You can place all service data on a single volume or choose to place the service data in multiple volumes. Configure the SAN device and create the partitions for the volumes.

  • Single Volume SAN Mount Point - /opt or /opt/zimbra
    • If you select to configure the SAN in one volume with subdirectories, all service data goes under a single SAN volume.
  • Multiple Volumes For SAN Mount Points
    • If you select to partition the SAN into multiple volumes, the SAN device is partitioned to provide the multiple volumes for each Zimbra mailbox server in the cluster. Example of the type of volumes that can be created follows.
      • /opt Volume for ZCS software (or really, /opt/zimbra/ )
        • Directories under /opt/zimbra/
          • conf Volume for the service-specific configuration files
          • log Volume for the local logs for Zimbra mailbox server
          • redolog Volume for the redo logs for the Zimbra mailbox server
          • db/data Volume for the MySQL data files for the data store
          • store Volume for the message files
          • index Volume for the search index files
          • backup Volume for the backup files
          • logger/db/data Volume for the MySQL data files for logger service’s MySQL instance
          • openldap-data Volume for OpenLDAP data
  • Note, for a multi-volume SAN Cluster, you'll actually create the directory path differently. [ /opt/zimbra-cluster/mountpoints <clusterservicename.com> ] Please see the Cluster Installation Guides for the full planning recommendations and steps if this is what your going to do.

Zimbra Directory Layout & FHS (Filesystem Hierarchy Standard)

FHS

References:

Bugs/RFE's

Community Feedback
  • Feedback Reference One
We are following FHS[1] standard for our deployments (or at least trying 
our best to follow it). It would be nice to reflect on the possibilities 
of mostly FHS-compliant Zimbra deploy. Here's what we've came up with so 
far:

/etc/opt/zimbra for configs
/opt/zimbra - binaries
/var/opt/zimbra - message store, OpenLDAP db, MySQL db's etc.
/var/log/zimbra - logs

Going by the FHS standards (in our case) means deploying a well-documented
system and that its layout is consistent across the board. 

Benefits:
* A paranoid type setup like mounting /opt read-only, and /var as no-exec. 
* For the uber-paranoia, including /etc as read-only. 
* You could tune each FS for specific needs which are consistent across 
  the board. 
** Different FS / or differently tuned FS/ used for each generic case.
* Migrations would be fairly simple, as it's easy to rip out configs (/etc) 
  or data (/var) or logs (/var/log) and copy/move it someplace else. 
* It opens the door to possibility of mounting volume with binaries on 
  multiple machines that only have local configs and data (not that we plan
  on it at the moment).

Disk Or Disk Layout Performance Testing

hdparm - Read or set the hard drive parameters

References:

Using DD

1GB file test

sync ; time bash -c "(dd if=/dev/zero of=largefile bs=1024 count=1000000; sync)"

Now time the removal of the large file.

sync ; time bash -c "(rm largefile; sync)"
Bonnie++

References:

dbench - generate load patterns

References:

IOzone

References:

Stress

References:

Postmark (Netapp)

References:

LTP - Linux Test Project

This suite of tools has filesystem performance tests.

References:

Clustering Software

Please see Ajcody-Clustering

Virtualization

Please see Ajcody-Virtualization

What About Backups? I Need A Plan

Please Review Other Backup-Restore As Well

Please review the Ajcody-Backup-Restore-Issues section as well.

What Might Be Wrong?

First thing to check is the log file and see if anything notable stands out.

grep -i backup /opt/zimbra/log/mailbox.log

Use Auto-Group Backups Rather Than Default Style Backup

Having trouble completing that entire full backup during off-hours? Enter the hybrid auto-grouped mode, which combines the concept of full and incremental backup functions - you’re completely backing up a target number of accounts daily rather than running incrementals.

Auto-grouped mode automatically pulls in the redologs since the last run so you get incremental backups of the remaining accounts; although the incremental accounts captured via the redologs are not listed specifically in the backup account list. This still allows you to do a point in time restore for any account.

Please see the following for more detailed information:

Need To Write Fewer Files - Add The Zip Option To Your Backup Commands

Using the zip option will compress all those thousands of single files that exist under a user's backup, decreasing performance issues that arise from writing out thousands of small files as compared to large ones. This is often seen when one is :

  • Using nfs for the backup directory
  • Copying/rsyncing backups to a remote server
  • Are using some third party backup software (to tape) to archive/backup the zimbra backup sessions.

Please see the following for more information about using the Zip option:

SAN Snapshots For Backups

Please see:

Cloud Backups

Please see:

Tape Backups

I would then use the non-rsync network ports for your traditional network backup software to run over to dump the data to tape. This way that activity doesn't effect prod performance at all. All full DR would use the backup/ data anyways (offsite DR). I've created another section that will deal with this in more details - specifically handling the hard links that are used by Zimbra.

Please see:

Test Environments And Managing Customizations

I have some suggestions on this in the RFE below. The first comment has a recommended layout for your test/qa/dev environments:

Using Vmware ESX For A DEV/QA/Prod Test Environment

Please see Ajcody-Virtualization#Using_VMWare_ESX_For_ZCS_Test_Servers_-_How-To

Creating A Hot-spare Server

Setup along the same lines... though you could cut out some of the HA/performance items if you only see this box being used for "short-term" use. Rsync's will occur over backup network port.

Need to do a sanity check in regards to ldap data. With a normal DR restore, one would do a zmrestoreldap. zmrestoreldap looks to a backup session, there is no real option in regards to a "redolog" directory. Things that are "ldap" only are COS's, DL's, etc..

  1. Setup the hot-spare according to http://wiki.zimbra.com/index.php?title=Ajcody-Notes-ServerMove
    • Basically install the zimbra packages. You'll want to do this with upgrades to production as the system evolves.
  2. I would do an initial rsync of /opt/zimbra (remember to use nice flag to not effect prod to much)
  3. I would then setup 2 daily rsync jobs (following the same wiki instructions)
    1. rsync /opt/zimbra/backup
      • This could be intergrated within the backup cron job so it kicks off after the backup is done. You'll need to monitor the times of backups and also the time for sync's so you can make sure you make it within the window of rotation - backup , rsync , next backup. Times will be different on diff and full backups.
      • rsync other necessary files:
        • /opt/zimbra/conf
        • /opt/zimbra/redolog
        • /opt/zimbra/log
      • This will give some "sanity" in case issues arise with the full restore. This part could use some better feedback from more experience Zimbra staff. I can think of some circumstances where this effort would prove useful.

A Real Hot-spare DR RFE

Note: Please add your votes to this RFE!

If It All Blows Up, Now What?

References:

http://wiki.zimbra.com/index.php?title=Ajcody-Notes-ServerMove

http://wiki.zimbra.com/index.php?title=Network_Edition_Disaster_Recovery




Ajcody Notes Server Move

   KB 2494        Last updated on 2008-11-16  




0.00
(0 votes)
Attention.png - This article describes the steps to move a ZCS server to a new physical or virtual server. Technically, this is not 100% supported as it is not a method that the developers or QA teams test against. But because of customer demands and needs, this method is often preferred compared to Network_Edition_Disaster_Recovery.

THIS IS THE ONLY OTHER DOCUMENTED METHOD BESIDES THE Network_Edition_Disaster_Recovery WIKI FOR A ZCS SERVER MOVE PROCESS THAT ZIMBRA WILL SUPPORT AND ACCEPT SUPPORT CASES FOR.

Support cases should reference this wiki page and include a copy of all the steps and output of them as you work through this how-to. Then noting the issue and step in the how-to your stuck at.

Discussion on this document is located here:

Server Move Notes

Actual Server Move Home Page

Please see: Ajcody-Notes-Server-Move

Server Move To Same Platform (32/64bit) And OS Type & Version


References

Please review the main reference for this page. It will have additional information you should be aware of before following these steps.

To do a server move via zmrestoreoffline, see:

OS Changes Also Happening

Please see these wiki article:

Related RFE's

The following, if and when done, should also address directly changes in the OS or architecture of the "new" server as compared to the old production server you were moving from.

Assumptions


Basic Assumptions

  • 1. OS Type & Version
    • This article makes the assumption your moving to the same OS type & version. For example, your OLDPROD machine is running RHEL4-64bit. So your new machine would be running the same and brought to the same patch level as well.
    • Note This wiki has been used many times to successfully move to a new server that was running a different OS than the OLDPROD. The one condition that does stay true though is the ZCS version must be the same between the two serves. Any ZCS "upgrades" should occur after the migration has been successfully done to the new server using the same ZCS version as the old one. Experience has shown it's generally a non-issue, but I would advise doing a 'test' move first to confirm this prior to attempting a real production server switch over. I can't address all the various issues/tweaks that might or might not need to be addressed with an OS change in this article though - hence, my strong recommendation of a proper "test" before you schedule your production server switch over. ---Ajc , Jan 27, 2015
  • 2. Same HOSTNAME
    • New server is setup with the same HOSTNAME information as OLDPROD but it will use a different IP until OLDPROD can be shutoff/reconfigured (if needed)
  • 3. Different IP Issues - Either For Testing Or As A Permanent Change
    • Problems that are caused by different ip's being in use related to zcs variables will usually show up on the NEWHOST where services [usually mailboxd] can't successfully start.
      • Note about command examples below.
        • I used `zmhostname` rather than typing out an explicit hostname.
        • I also included the -l option for the zmprov ms examples since the NEWHOST most likely will only be able to start the ldap service until you have these variables changed.
      • One example variable to look for:
        • zmprov gs `zmhostname` zimbraLmtpBindAddress
        • To modify on NEWHOST - as zimbra
          • zmprov -l ms `zmhostname` zimbraLmtpBindAddress "XXX.XXX.XXX.XXX"
            • replace XXX.XXX.XXX.XXX with the ip address on the NEWHOST
      • Another example variable to look for - this would be an issue if the NEWHOST is on a different subnet than OLDPROD :
        • zmprov gs `zmhostname` zimbraMtaMyNetworks
        • An examaple to modify on the NEWHOST - as zimbra
          • zmprov -l ms `zmhostname` zimbraMtaMyNetworks '127.0.0.0/8 10.10.130.0/24'
    • Since there are many possible variable that might include a specific ip reference, it would be best to do something like the following on the OLDPROD and review the possible variables you might have set that are using an explicit ip address.
su - zimbra
ip addr
ifconfig -a
[ NOTE THE IP's BEING USED - You'll grep for the first octet below ]
[ For example, my server's eth0 is using 192.168.0.71 ]
zmlocalconfig | grep 192
[192 is based upon my example]
zmprov gacf | grep 192
zmprov gs `zmhostname` | grep 192
  • 4. SAME AMOUNT OF MEMORY
    • If your moving from 32bit to 32bit and the new system has more than 4GB of memory and the older one didn't you will most likely need to adjust mailboxd_java_heap_memory_percent . This problem will show up as the mailbox stop starting and /opt/zimbra/log/zmmailboxd.out logging errors about JVM memory heap. Try the following:
su - zimbra
zmlocalconfig -e mailboxd_java_heap_memory_percent=25
zmmailboxdctl restart
If You're Attempting A Switch From 32 to 64

Update - 02/01/2014 - I created https://wiki.zimbra.com/wiki/Ajcody-Notes-Server-Move-32-To-64 so it's less confusing to do the steps on this page when you have a 32>64 transition. - Adam Cody

Note - If you skipped this section for whatever reason and run into the problem later as
your attempting to start/install on the new box you can still fix it. Use rpm2cpio to extra the
needed data for the zimbramon/ directory. Move the existing one at /opt/zimbra/ to a tmp
location and then copy/move the one from your rpm2cpio extraction.
Get the path to the zimbra-core package within the extracted tarball you have for the zimbra installation :
ex: /root/zcs-NETWORK-6.0.6_GA_2324.RHEL5_64.20100406133038/packages
Change directory to a temporary location that you'll extract the data to :
ex: rpm2cpio /root/zcs-NETWORK-6.0.6_GA_2324.RHEL5_64.20100406133038/packages/zimbracore-6.0.6_GA_2324.RHEL5_64-20100406133038.x86_64.rpm | cpio -idmv
You'll be able to find the zimbramon directory under there now to copy over to the /opt/zimbra path.


The details of this document assume that both servers are of the same chip type - 32bit or 64bit. Below are notes to "tweak" this how-to to handle a switch as well from 32 to 64.

The two exceptions to this how-to's instructions would be handling the zimbramon direcotry and the inclusion of the 32>64 ldap export/import steps listed in Network_Edition:_Moving_from_32-bit_to_64-bit_Server :

  1. On the "new server" , install the 64-bit version of ZCS that is the same version that your old server is running. The only difference would be it's 64 rather than 32-bit tar ball of ZCS.
  2. Copy /opt/zimbra/zimbramon of the 64bit installed stuff to a tmp dir
    • [as root] cp -rp /opt/zimbra/zimbramon /opt/zimbramon-64bit
  3. On the "old server" do the zmldapcat steps listed in Network_Edition:_Moving_from_32-bit_to_64-bit_Server
  4. Proceed with the steps listed in the main body of this how-to
    1. When you start step 7 in Ajcody-Notes-Server-Move#The_Big_Day_-_PROD_Downtime_For_Switch section stop.
  5. Copy the zimbramon-tmp over the /opt/zimbra/zimbramon dir
    • [as root] cp -rp /opt/zimbramon-64bit/* /opt/zimbra/zimbramon 
      • This is to bypass an reported error, below, during the ./install.sh part . Reports so far have been against Ubuntu.
        • Can't load '/opt/zimbra/zimbramon/lib/i486-linux-gnu-thread-multi/auto/IO/IO.so' for module IO:
          /opt/zimbra/zimbramon/lib/i486-linux-gnu-thread-multi/auto/IO/IO.so: wrong ELF class:
          ELFCLASS32 at /usr/lib/perl/5.8/XSLoader.pm line 70.
          at /opt/zimbra/zimbramon/lib/i486-linux-gnu-thread-multi/IO.pm line 11
  6. Save a copy of your .saveconfig directory and /opt/zimbra/conf/localconfig.xml in case you need to fall back on it. It seems the ./install.sh -s removes the files in .saveconfig and alter's localconfig.xml , which means when you goto do the ./install.sh later it will not see your configuration variable for your server.
    • [as root] cp -rp /opt/zimbra/.saveconfig /opt/saveconfig-bak
    • [as root] cp -p /opt/zimbra/conf/localconfig.xml /opt/localconfig.xml 
  7. Do a ./install.sh -s from the zimbra installation extract directory as root and then copy back your files from the saveconfig-bak directory.
    • [as root] cp -rp  /opt/saveconfig-bak/* /opt/zimbra/.saveconfig/
    • [as root] cp -p /opt/localconfig.xml /opt/zimbra/conf/ 
  8. Then follow the steps listed in the 32>64 bit wiki page about restoring the ldap data
    • Either the 5.x or 6.x section entitled:
      • "5.0.x or previous LDAP setup: 1. Restore the LDAP data to the 64-bit server. As zimbra, type"
      • "6.0.x and later LDAP setup: 1. Restore the LDAP data to the 64-bit server. As zimbra, type"
    • Stop at the point you just finished doing the
      • "/opt/zimbra/openldap/sbin/slapadd -q -b "" -F /opt/zimbra/data/ldap/config -cv -l /backup/ldap.bak"
  9. Now proceed with the step 7 section of the server move wiki, which is the full install/upgrade -- ./install.sh .
Moving To A New VMware Image Using Different OS

A short overview I wrote:

Optional Uses


For DR Prior To Upgrades

You could also use this method for a "copy" of your old version prior to doing an upgrade.

For Local Disk Partition Changes Rather Than A Server Move

Let's say your going to add addition storage and you want to move /opt/zimbra which was on the root partition of / . You could use the rsync steps listed here to make that move and decrease the downtime window versus shutting down zimbra to do a move or copy operation to the new partition. Remember to adjust your /etc/fstab after this otherwise zimbra might not come up after your next server restart.

Typical example of this situation is customer also wanting to change the underlying OS/distro , need to move the zimbra directory off of the / partition onto a new partition they now have available, etc..

Note - if you switch the OS type/version, you'll need to include the steps below about getting the right version of the ZCS binaries on the system and so forth since your 'rsync' copy would have the older version. For the re-install to work, you need the correct binaries - mostly so ldap can run during the installation/upgrade.

Other Sync Method Processes


BackupPC

Some customers have been using BackupPC for server moves and roll backs from ZCS upgrades.

Please see:

Important Advise About Testing Or Practicing This How-To


Attention.png We strongly encourage customers to 'test' this how-to prior to scheduling
your final downtime for the production server to be moved.

If you plan accordingly, you'll only need two downtime windows on the production server to include this testing. These downtime windows will be much shorter using rsync versus other means to do your server move.

To allow for this recommend testing, please adjust the how-to steps below with the following information. Since at some point you'll have to shutdown zimbra on the production server to get your final rsync, you should preserve the rsync'd data on the NEWHOST if you have the available disk space.

Alternation to the rsync targets:

  1. rsync OLDPROD:/opt/zimbra data to NEWHOST:/opt/zimbra-BACKUP rather than NEWHOST:/opt/zimbra .
  2. Then rsync or cp [with appropriate option flags] to NEWHOST:/opt/zimbra .
    • Leaving the NEWHOST:/opt/zimbra-BACKUP intact for later reuse for testing and etc..

For retesting purposes, you have two possible methods.

  1. First - jut rsync over the NEWHOST:/opt/zimbra directory
    • You would use the rsync command that includes the --delete option, since we want NEWHOST:/opt/zimbra to be exactly the same as NEWHOST:/opt/zimbra-BACKUP .
  2. Second - remove NEWHOST:/opt/zimbra and then rsync NEWHOST:/opt/zimbra-BACKUP [copying anything over again into a new and empty NEWHOST:/opt/zimbra ]
    1. run the uninstall option via the zimbra installation script - install.sh -u - on the NEWHOST.
    2. [NEWHOST] rm -rf /opt/zimbra . We want to confirm all files are gone.
    3. Remove any 'zimbra' files in /tmp
    4. And now restart the how-to with a clean zcs installation followed by another rsync of NEWHOST:/opt/zimbra-BACKUP to NEWHOST:/opt/zimbra .
  3. Note - I just used /opt/zimbra above, you'll need to include other data/zmvolume paths as well if you have them.

Final word of advise, please consider installing bind or some other DNS service on your NEWHOST so you can handle resolution locally on that server to resolve to the different ip address the NEWHOST is using. Consult Ajcody-Hostname-DNS and Split_DNS for more details.

The Actual Steps


Preparing NEWHOST Server


Attention.png Please Note - DO NOT USE CRON WHEN SYNCING YOUR SERVER!
You should always run the rsync command manually. If you fail to do so, and then complete the migration without removing the cron job that did the sync's, and you leave the old server running, you will lose data. The sync will then kick off again when cron runs and overwrite your production data! This will very likely corrupt your installation and leave you with an unstable system!


  1. Install Newer (supported) Operating System that matches OLDPROD
  2. Set up newer ZCS Server’s Hostname as it was on the older server
  3. Configure BIND locally on NEWHOST to handle resolution issues (A, MX, etc.)
    • Attention.png Please Note - The NEWHOST MUST RESOLVE THE ZMHOSTNAME TO IT'S OWN IP ADDRESS!
      • If dns resolution of the zmhostname on the NEWHOST resolves to the OLDPROD's ip address - this will most likely cause a production outage on OLDPROD since all zm commands ran on NEWHOST will actually be sent to OLDPROD. You will most likely need to do a DR recovery on OLDPROD if you don't confirm the ip resolution is correct on the NEWHOST.
    • On NEWHOST confirm /etc/hosts , /etc/resolv.conf is set correctly to resolve to NEWHOSTs ip address and NOT the OLDPROD's ip address.
  4. Download the EXACT version of ZCS that your OLDPROD is using - meaning the ZCS release number [ex. 8.0.7]; for some, it might be a different distro OS release of 8.0.7 [ex ZCS version].
  5. ZIMBRA User And UID Match
    • On OLDPROD as ZIMBRA:
      zmlocalconfig zimbra_uid
    • On OLDPROD, note the zimbra entry in /etc/passwd and /etc/group.
    • On NEWHOST configure the /etc/passwd to be the same UID for zimbra as the OLDPROD had.
    • On NEWHOST configure the entry for zimbra in the OLDPROD's /etc/group to match for zimbra as well.
  6. On NEWHOST as ROOT: Run the installer with the -s option:
    ./install.sh -s
    • This tells the installer to only install the software, and not to configure the installation.
    • To see what is installed and enabled on the PROD server, do the following on the PROD server:
    • zmprov gs `zmhostname` | grep zimbraService
    • Save this list, you'll need it also when you rerun the installer later on the NEWHOST to confirm the "upgrade" does the right package upgrades/installs.
  7. On NEWHOST as Root: Remove the dummy install:
    rm -rf /opt/zimbra ; mkdir /opt/zimbra
  8. On NEWHOST, make any other mounts or directories you'll need as to match the OLDPROD server.
    • Secondary mailstores, alternative backup directory paths, etc.
    • On OLDPROD, double check for these additional mounts by doing:
      • reviewing /etc/fstab and output of the df command.
      • run, as zimbra, the following: zmvolume -l and review the output and directory paths.
Sync OLDPROD Data While OLDPROD Is Still In Production Use


Attention.png Please Note - DO NOT USE CRON WHEN SYNCING YOUR SERVER!
You should always run the rsync command manually. If you fail to do so, and then complete the migration without removing the cron job that did the sync's, and you leave the old server running, you will lose data. The sync will then kick off again when cron runs and overwrite your production data! This will very likely corrupt your installation and leave you with an unstable system!


Preliminary Comments
Rsync options used below.
I've added the -H option to the rsync command to preserve hard links.
-H, --hard-links , preserves hard links
-a, --archive . This is equivalent to -rlptgoD .
From man page:
It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when --files-from is specified, in which case -r is not implied. Note that -a does not preserve hardlinks, because finding multiply-linked files is expensive. You must separately specify -H.
-z, --compress
From man page:
With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection.
You should upgrade rsync on both servers to version 3+. It addresses some performance issues. Please see the following for more details.
http://www.samba.org/rsync/FAQ.html#4
Nice and the numeric ranger, from RHEL5 nice man page:
Nicenesses range from -20 (most favorable scheduling) to 19 (least favorable).
-n, --adjustment=N add integer N to the niceness (default 10)


  1. First initial sync of OLDPROD to NEWHOST
    • on OLDPROD as Root
      nice -n +19 rsync -avzH -e ssh --progress /opt/zimbra/ root@NEWHOSTIP:/opt/zimbra
      • For ZCS8+ , YOU HAVE TO EXCLUDE THE LDAP DB DIRECTORY [default path is /opt/zimbra/data/ldap ] in your RSYNC command.
      • On my test box, Ubuntu 12.10 , the syntax for rsync to exclude the default ldap path would be :
      • Do the same with other paths you might have - i.e. secondary volumes.
        • [as zimbra] zmvolume -l
      • Some distro's, might want this format for nice :
        nice +19 rsync ....
  2. Sync daily until schedule downtime is available
    • On OLDPROD as Root
      nice -n +19 rsync -avzH -e ssh --progress /opt/zimbra/ root@NEWHOSTIP:/opt/zimbra
      • For ZCS8+ , YOU HAVE TO EXCLUDE THE LDAP DB DIRECTORY [default path is /opt/zimbra/data/ldap ] in your RSYNC command.
      • On my test box, Ubuntu 12.10 , the syntax for rsync to exclude the default ldap path would be :
        • nice -n +19 rsync -avzH --exclude='data/ldap/*' -e ssh --progress /opt/zimbra/ root@NEWHOSTIP:/opt/zimbra
      • Do the same with other paths you might have - i.e. secondary volumes.
      • Some distro's, might want this format for nice :
        nice +19 rsync ....
    • Extra Point Comment
      • One could also use a more aggressive nice value to start a job after hours but then include a renice command via cron prior to your normal business hours the next day. You would need to script this out though since it would need the process # of the command you wanted to renice.
    • Copy your crontab over to new system - should be able to use scp or rsync. /var/spool/cron/zimbra
      • You can then compare the 'old' one to the new one that will be created with the dummy install. You might have customization's that aren't in the new one provided by the basic install - ie, backup schedule, hsm runs, etc.
The Big Day - OLDPROD Downtime For Switch


  1. Commercial Certificates
    • Are you prepared for the impact this server move has for your commercial certificate? This document doesn't address that.
  2. Block client access to the old server's IP address with firewall rules
    • If your remote, make sure to keep your access port open. We are just trying to prevent any changes to the machines while they are being reconfigured.
  3. Shut down Zimbra on OLDPROD
    • On OLDPROD as ZIMBRA
      su - zimbra
      zmcontrol stop
  4. Attention.png Please Note - Prior To Final Rsync
    • YOU MUST PERFORM THIS STEP WITH ZIMBRA DOWN!
      Please ensure that you have the Zimbra server STOPPED before performing the final rsync. If you're using this method to test your server move, you must schedule downtime to stop the server before performing the final sync. This is required, and failure to do so will result in corruption of your mysql/ldap databases on the test server.
  5. Get LDAP in a good state prior to the final rsync [as recommend by our LDAP developer - 12/1/2012]
    1. For Versions Prior to ZCS8 , use db_recover - see the following reference for your version:
      1. http://wiki.zimbra.com/wiki/Ajcody-LDAP-Topics#Attempt_To_Cover_Versions_Higher_Than_ZCS5_-_I.27ve_yet_to_confirm_the_below
    2. For ZCS8 , db_recover isn't needed. You must include the -S option though for rsync.
  6. Last rsync of OLDPROD to NEWHOST
    • On Prod as ROOT
      nice -n -20 rsync -avzHS -e ssh --delete --progress /opt/zimbra/ root@NEWHOSTIP:/opt/zimbra 
      UPDATE : I've added the -H option to the rsync command to preserve hard links.
      UPDATE : I've added the -S option to the rsync command to handle sparse files. With ZCS8's ldap setup, this is necessary. See OpenLDAP_Performance_Tuning_8.0#Notes_on_MDB for details. 12/1/2012
      • Do the same with other paths you might have - i.e. secondary volumes.
      • Some distro's, might want this format for nice :
        nice -20 rsync ....
  7. Fix permissions on NEWHOST
    • On NEWHOST as ROOT run
      /opt/zimbra/libexec/zmfixperms --verbose --extended
  8. Turn off OLDPROD and reconfigure NEWHOST
    • This is a good time to turn off OLDPROD.
      • Reconfigure the network interfaces so if someone turns on OLDPROD later, it will not use the ip addresses that will now be used on NEWHOST.
      • Reconfigure any mounts (san, nfs, iscsi, etc.) so it will not mount anything that should only be mounted on our NEWHOST. Again, in case the machine is powered on accidentally later.
    • Reconfigure NEWHOST to take over ip addresses of OLDPROD.
      • Reconfigure the following now to use the production ip address etc:
        • The network interface configuration files per your distro
        • /etc/hosts
        • /etc/resolve.conf to use your production DNS servers
          • disabled BIND on this server if you set it up temporary until you could use the production DNS servers
      • Make any firewall or other network changes that are necessary.
        • Remember about arp tables.
    • Reconfigure for any mounts that were on OLDPROD that will be needed for NEWHOST.
  9. Install of Zimbra on NEWHOST
    • If you have a Split DNS install or use private LAN addresses on the server with a firewall front-ending the public addresses, you'll want to verify logical hostname resolution and hostname resolution
      • In some cases, you can move Zimbra to another server with a different hostname but keep the logical hostname the same. The logical hostname is what users know this server as, and it doesn't necessarily have to match the actual hostname.
        • For example, you might have "mail.mydomain.com" as the DNS name for the server, but the hostname is "web11233"
        • You need to have the server itself resolve the logical hostname, the old hostname and the new hostname as the internal private LAN address.
        • host `hostname`
        • host (logical hostname)
        • nslookup `hostname`
        • nslookup (logical hostname)
        • nslookup (old hostname)
        • nslookup (old hostname)
      • You want all these to look the same. You can follow instructions at Split_dns. Essentially what that does is have a local copy of bind (named) running that resolves just those names and forwards all other lookups to your normal DNS servers.
    • On NEWHOST as ROOT, rerun the installer without the -s option. Again, this is using the EXACT version of ZCS that your OLDPROD is using - meaning the ZCS release number [ex. 8.0.7]; for some, it might be a different distro OS release of 8.0.7 [ex ZCS version]. Any attempt to upgrade to a newer version of ZCS must be done AFTER you've completed this wiki page in full.
      ./install.sh --skip-activation-check   #OR This If your version doesn't support the skip activation option#  ./install.sh
      • It will detect ZCS already installed, and ask if you want to upgrade. Select Yes.
      • You can confirm the package installation/upgrade selection by comparing the output from the OLDPROD host by running this on the OLDPROD host
        • zmprov gs `zmhostname` | grep zimbraService
      • Attention.png Note - If in the configuration screen it looks like it doesn't have your old values, then the localconfig.xml in /opt/zimbra/conf must of been altered. To fix:
        • Stop/cancel the install - you'll see an option for it from the list of options.
        • Copy from your OLDPROD's /opt/zimbra/conf/ directory the localconfig.xml file to the NEWHOST's /opt/zimbra/conf/ directory.
        • Restart the the installation again with ./install.sh --skip-activation-check or ./install.sh , depending on your version
        • If that failed to fixed the install configuration screen situation, redo the above but also remove /opt/zimbra/.saveconfig from NEWHOST and copy that directory from OLDPROD.
      • Within the installation script, you might want to choose the option that tells Zimbra to NOT automatically start upon completion of upgrade/install.
      • Confirm your license is valid and activated:
        •  zmlicense -c ; zmlicense -p | grep -A4 Activation
  10. Post-Install on NEWHOST
    • This document assumes you were going to get the same hostname and ip address once the final move was done. In case this isn't true, below are some follow up issues you might want to check. You might of done some of these already.
      • Do you need to make adjustments for commercial certificates?
      • Reconfigure any network interface/ip information that you need because of hardware move.
      • Make necessary adjustments you might need because of hostname changes. ( see ZmSetServerName )
      • Adjust any firewall settings
        • If ip address is going to be different, make sure you know the settings you'll need to adjust within Zimbra (if any).
        • If ip address is going to be the same, remember your network will take awhile to see change as the new MAC address gets updated to other devices arp table.
          • If you can, you can speed this along with changes on your switches.
  11. Start zimbra once you think everything is ready.
    • Do some client access tests within your LAN.
    • If testing looks good, the remove any firewall rules you might of done to block access from outside. Then confirm outside access and functionality.
      • Remember to check those mobile devices, certificates, and other access software/devices besides just the Zimbra webclient.

Things To Check Before Going Production Again

See the post checks from the DR page:

Additional Topics


Zimbra Domains Into An Existing Zimbra Server

See Ajcody-Migration-Notes . There's a couple of possibilities being worked out there.

Verified Against: unknown Date Created: 7/23/2008
Article ID: https://wiki.zimbra.com/index.php?title=Ajcody-Server-Topics Date Modified: 2008-11-16



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »





Ajcody Notes Multi Server Restore Disaster Recovery (DR)

   KB 2494        Last updated on 2008-11-16  




0.00
(0 votes)
24px ‎  - This is Zeta Alliance Certified Documentation. The content has been tested by the Community.


Actual Multi-Server Restore-DR Notes Homepage

Please see Ajcody-Notes-Multi-Server-Restore-DR

Bugs I've Filed That Might Apply

Issues I Still Need To Confirm

As reported by one customers experience.

"The zmmyinit command does not work when used as directed. I found that for it to work
the mailstore has to be started then stopped then reinitialize the db, this sets some
configs somewhere allowing zmmyinit to create/start the db and connect. After that the
rest of the restore procedure works."

Multi-Server Restore-DR How-To

Introduction

This document is for disaster recovery, where the /opt/zimbra data is destroyed [hardware failure], missing, or completely inaccessible for some reason. There are better options if it’s a matter of some data being corrupt or some function not working properly.

Full restores can take a while, so plan accordingly.

More To Do

  • Section different between LDAP+MTA and LDAP only and MTA only
    • Include (need to get details still) MTA only steps
  • I still need to adjust this document to allow administrators to practice a DR situation.
    • Special consideration to take when production system is still live
    • Can production multi-server be DR'd to a single server?
      • Commands needed to adjust configurations for this situation
    • Private network
    • Internal DNS server for practice DR servers to use
    • How to handle different hostnames if one wanted/needed to.

Note, whenever an application is moved to a different physical host one must account for DNS TTL for hostname/ip addresses and also arp table caching on the clients and network devices in regards to the old MAC address of the network interfaces. Down the rode, I hope to add another wiki write up on this topic and the different solutions & practices to handle this.

Pre-Install

Confirm base OS restore/configuration details.

  • Hostname  :
    cat /etc/hostname & /etc/hosts
  • IP configuration  :
    ip ad
    • or  :
      ifconfig -a
  • DNS resolution  :
    cat /etc/resolv.conf ; host –a `hostname –f`
  • Routing information  :
    netstat –rn
  • Local and remote filesystems :
    cat /etc/fstab ; df –k
  • Access to backup data

The information you need from the old server including:

  • Installation packages - options to the yes/no questions
  • License file
  • Admin account name and password
  • Spam training and non-spam training user account names
  • Exact domain name
  • The global document account name
  1. If the server is not running.
    1. Block client access to the server IP address with firewall rules.
    2. Find the latest full ZCS backup session to use.
  2. If ZCS is still running, to prepare the move to the new server.
    1. Block client access to the server’s IP address with firewall rules.
    2. Run a full backup of the old service, or if the backup is recent, run an incremental backup to get the most current incremental backup session.
    3. Run zmcontrol stop, to stop ZCS. In order to restore to the most current state, no new mail should be received after the last incremental backup has run.
    4. Change the hostname and IP address on the old server to something else. Do not turn off the server.

LDAP Server (Everything but Mailstore)

  1. Reinstall/Restore the Zimbra Collaboration Suite
    1. Ensure the host hostname and MX DNS records resolve to the new server.
    2. Untar the matching version of the old server or restore the /opt/zimbra directory
    3. If this is NOT a new server, then :
      1. sudo – u zimbra /opt/zimbra/bin/zmcontrol stop
      2. rpm –qa | grep zimbra
      3. rpm –e zimbra-<package-name>.rpm
    4. Copy the License file (ZCSLicense.xml) to a path on your server
    5. Find and run the install.sh script
      1. Choose all options as original install for modules
        1. Mailstore shouldn’t be selected or MTA if you had a separate MTA server
      2. Make sure the same domain, hostname, passwords as on the old server are used.
      3. Configure the SMTP server to the LDAP server hostname or a separate MTA server you might be using.
      4. Disable auto-backup and starting of servers after configuration in the configuration menu
      5. Apply configuration changes
  2. Confirm ZCS isn’t running.
    1. su – zimbra ; /opt/zimbra/bin/zmcontrol stop
  3. Copy the backup files to /opt/zimbra/backup if not already present.
  4. Restore the LDAP server (zmrestoreldap -lb <latest_session_label>)
    1. Note: To find the LDAP session label to restore, type zmrestoreldap –lbs
    2. If you are restoring large number of accounts, you may want to run this command with nohup so that the session does not terminate. (Observe whether the LDAP server is started successfully after the restore, it must be running for the next steps).
    3. Note: The zmrestoreldap script included in ZCS 4.5.7 through ZCS 4.5.10 and ZCS 5.0 through ZCS 5.0.1 is broken. This is being tracked as Bug 23644: zmrestoreldap not taking accesslog db into consideration. The fix will be included in ZCS 4.5.11 and ZCS 5.0.2. You can also download an updated script with the fix from these links:
      1. ZCS 4.5.x: http://files.zimbra.com/downloads/4.5.10_GA/zmrestoreldap_4511
      2. ZCS 5.0.x: http://files.zimbra.com/downloads/5.0.1_GA/zmrestoreldap_502
    4. Type zmconvertctl start. This is required before running zmrestoreoffline.
    5. Because some ZCS services are running at this point, type zmcontrol stop to stop all services.
    6. Remove any old backup sessions because these sessions are no longer valid.
      1. Type
        rm -rf /opt/zimbra/redolog/* /opt/zimbra/backup/*
    7. Start the Zimbra server (zmcontrol startup)
    8. Confirm and get the root LDAP password for the mailstore setup.
      1. zmlocalconfig –s ldap_root_password
      2. Go to the Zimbra administration console to verify that the accounts are set to active.
        1. (Accounts>General tab)

Mailstore Server (Nothing but Mailstore option in the installer)

  1. Reinstall/Restore the Zimbra Collaboration Suite
    1. Ensure the host hostname and MX DNS records resolve to the new server.
    2. Untar the matching version of the old server or restore the /opt/zimbra directory
    3. If this is not a new server, then :
      1. sudo – u zimbra /opt/zimbra/bin/zmcontrol stop
      2. rpm –qa | grep zimbra
      3. rpm –e zimbra-<package-name>.rpm
    4. Copy the License file (ZCSLicense.xml) to a path on your server
    5. Find and run the install.sh script
      1. Choose Mailstore for module to install
        1. Install other modules you might have had for your Mailstore server.
      2. Make sure the same domain, hostname, passwords as on the old server are used.
      3. Configure LDAP using the LDAP server hostname and LDAP ROOT password found with:
        1. zmlocalconfig –s ldap_root_password
      4. Configure the SMTP server to the LDAP server hostname or a separate MTA server you might be using.
      5. Disable auto-backup and starting of servers after configuration in the configuration menu
      6. Apply configuration changes
  2. Confirm ZCS isn’t running.
    1. su – zimbra ; /opt/zimbra/bin/zmcontrol stop
  3. Re-initialize mysql
    1. rm –rf /opt/zimbra/db/data/*
    2. /opt/zimbra/libexec/zmmyinit
  4. Copy the backup files to /opt/zimbra/backup if not already present.
  5. Type
    zmconvertctl start
    . This is required before running zmrestoreoffline.
  6. Run the zmrestoreoffline to restore all account mailbox and messages
    1. To start the offline restore, type
      zmrestoreoffline -sys -a all -c -br
      . You may want to run nohup here also. To watch the progress, tail /opt/zimbra/log/mailbox.log. Note: Use –c on the command line so that accounts will be restored even if some accounts encounter errors during the offline restore process.
    2. Because some ZCS services are running at this point, type zmcontrol stop to stop all services.
    3. Remove any old backup sessions because these sessions are no longer valid.
    4. Type
      rm -rf /opt/zimbra/redolog/* /opt/zimbra/backup/*
    5. Start the Zimbra server (zmcontrol startup)

Post-Server setup

  1. Remove the firewall rules and allow client access to the new server.
Verified Against: Zimbra Collaboration 8.0, 7.0 Date Created: 04/16/2014
Article ID: https://wiki.zimbra.com/index.php?title=Ajcody-Server-Topics Date Modified: 2008-11-16



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »





Ajcody Disaster Recovery Specific Notes

   KB 2494        Last updated on 2008-11-16  




0.00
(0 votes)
24px ‎  - This is Zeta Alliance Certified Documentation. The content has been tested by the Community.


Actual Disaster Recovery Specific Notes Homepage

Please see: Ajcody-Disaster-Recovery-Specific-Notes

Bugs Filed That Might Apply

DR Is Taking A Very Long Time - Too Long

Please see:

Missing accounts.xml File

Please see:

Multi-server DR / Restore Documentation

Please see:

ZSC to ZSC Online Sync For DR

Please see:

Please add your votes to this RFE!

My Disaster Recovery Failed

Main Reference

Please see: Network_Edition_Disaster_Recovery#Restoring_to_the_new_server

What To Do Or Check

Time Issues

First, make sure your TIME is set right! See Time_Zones_in_ZCS#The_server_OS

Authentication Password Issues

Though I'm still investigating why this is happening for our customers, the root issues seem to be resolved by the following. Some of the auth errors will be logged to /var/log/zimbra.log and /opt/zimbra/log/mailbox.log

Put the output of this command in a text file:

zmlocalconfig -s | grep password

These should match what's in /opt/zimbra/conf/localconfig.xml

Now, compare the passwords with what is in your restore. Put in your path of the restore specific directory.

vi /opt/zimbra/backup/sessions/YOUR_DIR/sys/localconfig.xml

Do a /password or the full variable in vi to see what the old passwords are. You'll need to adjust the following below. Remember to copy a backup.

cp /opt/zimbra/conf/localconfig.xml /opt/zimbra/conf/localconfig.xml.DR
accounts.xml file - restore error

mailbox.log or restore command will output errors about

  • accounts don't actually get restore though restore command appears to completed without error.
zimbra@xxxx:~/log$ zmrestore -all -rf
Error occurred: invalid request: invalid account email address: ll
zimbra@xxxx:~/log$ zmrestore -a bdavis@XXXXXXX.com -rf
Error occurred: no such account: Account ID for bdavis@XXXXXXXXX.com not found in backup
  • no such session backup label errors
com.zimbra.cs.backup.BackupServiceException: backup full-20080920.050014.406 not found: 
/opt/zimbra/backup/sessions/full-20080920.050014.406 not found ExceptionId:main:1222811246316:7615eea74222489d

If your missing the /opt/zimbra/backup/accounts.xml file, grab it from your saved backup directory or from the backup session your using (/opt/zimbra/backup/sessions/full-XXXXXX/sessions.xml). If you need/want to change the referenced session label in the accounts.xml file - just vi it and use the global search and replace string (:%s/search_string/replacement_string/g).

Errors about mysql or ldap not running

This will most likely be encounter if you have to adjust some items in regards to the restore command not working. Run or confirm the following:

su - zimbra
zmconvertctl stop
mysql.server start
ldap start
zmconvertctl start

Then try the restore command again.

Verified Against: Zimbra Collaboration 8.0, 7.0 Date Created: 04/16/2014
Article ID: https://wiki.zimbra.com/index.php?title=Ajcody-Server-Topics Date Modified: 2008-11-16



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »





Migration Issues

   KB 2494        Last updated on 2008-11-16  




0.00
(0 votes)
24px ‎  - This is Zeta Alliance Certified Documentation. The content has been tested by the Community.


Actual Migration Notes Homepage

Please see Ajcody-Migration-Notes

RFE/Bug's For Migration Tools

I'll try to keep track of items here when I find them.

Server Migrations Terminology

Staged (Migration Over Time Rather Than Single Instance)

Staged describes a migration will take place over time (days, weeks, months). Where users are moved to the "new" server over these time frames. This process might involve a Split-Domain or Sub-Domain configuration or the use of LDAP replica's that would integrate a new ZCS mailstore server.
Reasons For Staged Migrations

Normally, the split-domain option is used for one of these two reasons:

  • Very large users base and can't migrate everyone over is one downtime window. Must migrate over the course of multiple downtime windows spanning across weeks or months.
    • One of the mailserver might be over a remote network link that prevents migration data to be moved fast enough in one downtime window.
  • The "old" mail system must stay up because some users will not be migrated to Zimbra.

Split-Domain (Shared Mail Domain Between Multiple Email Servers)

Split-Domain means that your going to have mail servers from different vendors (Zimbra, MS Exchange, Sendmail, Postfix, etc.). One of these servers+vendor-type will be authoritative (Primary) for the email domain. The other servers (Secondaries) will still be in the same email domain, rather than a sub-domain, and use the Primary server for authoritative type functions.
Reasons For Split-Domains
  • Allows the highest possible merger between the two systems to minimize the impact to the end users.
  • Doesn't require a final domain rename as a final step to the server migration.
    • This can cause some disruption to some user items like - calendars, appointments, client reconfiguration, etc.

Sub-Domain (New Email Domain To Integrate Into Existing Mail Environment - Multiple Email Servers)

Sub-Domain means we'll add a new email sub-domain to the primary email domain to migrate users over to. It doesn't have to be an actual dns sub-domain, i.e. company.com > sub.company.com . This configuration will use forwarding & domain masquerading rules between the two email systems to allow users to move to the new email system over time. Once everyone is moved over, it can be reconfigured to use the primary domain only and drop the use of the sub-domain.
Reasons For Sub-Domains
  • Administrator wants to avoid the possibility of very complex configurations of the other options.
  • Can be setup fairly fast and easily.
  • Good for testing since it requires very little modifications to the existing system.

Required Reading Overview & Resources

Review the following articles because the how-to's in this area will require understanding of the content within them.

Zimbra And Linux Resources

  • Other Zimbra Related Steps That Might Be Necessary
    • Specifically "Domain Masquerading"
    • Special Note About These Variables And Calendar Invites & Email Replies
      • Please confirm replies are working correctly as well as calendar invites acceptance to those outside your zimbra environment.
      • [http://bugzilla.zimbra.com/show_bug.cgi?id=9545 See RFE 9545 - calendar organizer shows as account name, not "reply-to" address or zimbraMailCanonicalAddress"
        • CLI commands that might be necessary to modify for the situation:
          • zmprov ma user@domain.com zimbraMailCanonicalAddress otheraddess@domain.com
          • zmprov ma user@domain.com zimbraPrefReplyToAddress otheraddess@domain.com

MS Exchange Resources

Different Email Server Type/Vendor To ZCS - Using Split-Domain To Migrate

Split-Domain Example 1 - Zimbra Primary & Exchange Secondary

My attempt to rewrite Split_Domain for customers that are having problems following it for whatever reasons.

Rough Notes - Working On It Now

Hostnames used for server configurations when referencing each other MUST be using internal ip addresses/hostnames for resolution. This will avoid unnecessary routing out of the local network and then back in through your firewall, thus getting a new IP address.


Details of Firewal and/or External Mail Filter :

  • Inbound email goes first to:
    • Firewall : homename =
      • external ip's =
      • internal ip's =
    • External Mail Filter Server : hostname = mailfilter.YYY.state.XX.us
      • external ip's =
      • internal ip's =
    • Will then forward email to primary.YYY.state.XX.us via ip address after implementation


Configuring The New Zimbra Server As The Primary :

  1. Primary Mail Server ("authoritative" system for domain YYY.state.XX.us)
    • Zimbra Server
      • Hostname is primary.YYY.state.XX.us
      • mail domain = YYY.state.XX.us
    • The primary MTA must
      1. Be aware of, and accept mail for, all accounts on the domain
        1. Create ALL accounts for Zimbra that are on existing Exchange box
          • $ zmprov ca secondary-user@YYY.state.XX.us <some_random_password>
          • Review wiki on Bulk_Create . Don't forget about system accounts that you have configured for email
      2. Forward mail to the secondary MTA for users hosted there.
        1. Setup Transport for users to point to the Exchange Server
          • $ zmprov ma secondary-user@YYY.state.XX.us zimbraMailTransport smtp:secondary.YYY.state.XX.us:25
          • Again, review wiki on Bulk_Create to this easier.
      3. Reject on RCPT TO or configure a catchall account for those accounts that do not exist in either system
      4. LAST STEP - Configure DNS, MX records, firewall to make Zimbra server authoritative for domain.
        1. You will bounce mail if you make this change before configuring the entire system and testing that mail flow is working as desired.)
        2. Change your MX record so mail from the internet flows into the Zimbra MTA first.


Re-configuring The Old Exchange Server As The Secondary :

  1. Seconardy Mail Server
    • Exchange Server
      • Hostname is secondary.YYY.state.XX.us
      • mail domain = YYY.state.XX.us
    • The secondary MTA must
      1. Accept mail for accounts that are hosted on the secondary
        1. See Ajcody-Migration-Notes#MS_Exchange_Resources for now. I'll add specific steps as I get confirmation of them.
      2. We also highly recommend that you turn off DNS lookups and internet wide message routing from the secondary host and route all mail through the primary.
        1. See Ajcody-Migration-Notes#MS_Exchange_Resources for now. I'll add specific steps as I get confirmation of them.
      3. Forward mail to primary.YYY.state.XX.us for accounts hosted on the primary server
        1. See Ajcody-Migration-Notes#MS_Exchange_Resources for now. I'll add specific steps as I get confirmation of them.

Migrating Users To Be On the Zimbra Server :

  • When you are ready to move a user from the old system to the new system just run this command
    1. (where zimbra.YYY.state.XX.us is the name of your Zimbra server):
    2. $ zmprov ma secondary-user@YYY.state.XX.us zimbraMailTransport lmtp:primary.YYY.state.XX.us:7025


If there is no account on either server, the mail will bounce back to Zimbra.YYY.state.XX.us to catchall@YYY.state.XX.us


Outbound email:

Either of the mail servers will be able to send email out independent of the other by allowing outbound smtp traffic from their respective ip addresses through the firewall via policy rule.

Split-Domain Example 2 [Mnichols Write Up] - Exchange Primary & Zimbra Secondary

Please see the following for excellent write ups on doing this with MS Exchange acting as the primary. I'll leave my older write up below for historical reasons.

Split-Domain Example 2 [My Attempt] - Exchange Primary & Zimbra Secondary

My attempt to rewrite Split_Domain for customers that are having problems following it for whatever reasons.

Rough Notes - Working On It Now

Hostnames used for server configurations when referencing each other MUST be using internal ip addresses/hostnames for resolution. This will avoid unnecessary routing out of the local network and then back in through your firewall, thus getting a new IP address.


Details of Firewal and/or External Mail Filter :

  • Inbound email goes first to:
    • Firewall : homename =
      • external ip's =
      • internal ip's =
    • External Mail Filter Server : hostname = mailfilter.YYY.state.XX.us
      • external ip's =
      • internal ip's =
    • Will then forward email to primary.YYY.state.XX.us via ip address after implementation


Configuring The Old Exchange Server As The Primary :

  1. Primary Mail Server ("authoritative" system for domain YYY.state.XX.us)
    • Exchange Server
      • Hostname is primary.YYY.state.XX.us
      • mail domain = YYY.state.XX.us
    • The primary MTA must
      1. Be aware of, and accept mail for, all accounts on the domain
        1. See Ajcody-Migration-Notes#MS_Exchange_Resources for now. I'll add specific steps as I get confirmation of them.
        2. You'll have to grant relay permission on the Exchange server [Primary Server] to the Zimbra Server [Secondary Server] in the Exchange SMTP properties so outbound email from Zimbra going through the Exchange will work.
      2. Forward mail to the secondary MTA for users hosted there.
        1. Setup Transport for "migrated" [if any are done] users to point to the Zimbra Server
      3. Reject on RCPT TO or configure a catchall account for those accounts that do not exist in either system
        1. Determine if you want this on the Exchange or Zimbra Server.


Configuring The Zimbra Server As The Secondary :

  1. Seconardy Mail Server
    • Zimbra Server
      • Hostname is secondary.YYY.state.XX.us
      • mail domain = YYY.state.XX.us
    • The secondary MTA must
      1. Accept mail for accounts that are hosted on the secondary
        1. On the Zimbra Server, run these commands (domain already exists in below example of course):
          • The first two commands (in combination) tell the Zimbra postfix to accept all addresses in the @example.com domain as valid addresses.
            • $ zmprov md YYY.state.XX.us zimbraMailCatchAllAddress @YYY.state.XX.us
            • $ zmprov md YYY.state.XX.us zimbraMailCatchAllForwardingAddress @YYY.state.XX.us
      2. But must forward all other mail for accounts on this domain to the primary system
        1. On the Zimbra Server, run this command (domain already exists in below example of course).
          • This third command establishes default mail routing for the domain. Any users that do not exist on the Zimbra system will have their mail routed according to this rule.
            • $ zmprov md YYY.state.XX.us zimbraMailTransport smtp:primary.YYY.state.XX.us:25
      3. We also highly recommend that you turn off DNS lookups and internet wide message routing from the secondary host and route all mail through the primary.
        1. On the Zimbra Server you would:
          • Relay mail to Primary with:
            • $ zmprov mcf zimbraMtaRelayHost primary.YYY.state.XX.us:25
              • At the end, 25 is the port number for smtp on the targeted system. Adjust this number if you changed the smtp port.
          • Turn off DNS lookups with:
            • $ zmprov mcf zimbraMtaDnsLookupsEnabled FALSE
              • If you disable DNS Lookups (under the MTA tab of the admin console, or with zmprov), Zimbra will end up using (according to the postconf man page) the "gethostbyname() system library routine which normally also looks in /etc/hosts" (based on the entries on the "hosts" line in /etc/nsswitch.conf). If you do this but don't also specify an SMTP relay host (typically your ISP's SMTP server), which will take care of checking DNS, you will reverse your ability to send mail: suddenly you can send mail to other users on the Zimbra server, but you can't send to the internet (though you can still receive mail from the internet either way).
      4. accept mail relayed by primary.YYY.state.XX.us
      5. forward mail to primary.YYY.state.XX.us for accounts hosted on the primary server
      6. After configuration changes, restart services/server if needed.
        1. On the Zimbra Server, run these commands (as zimbra):
          • $ postfix stop
          • $ postfix start


If there is no account on either server, the mail will bounce back to a catchall account if you configured one.

Outbound email:

Either of the mail servers will be able to send email out independent of the other by allowing outbound smtp traffic from their respective ip addresses through the firewall via policy rule.

As You Migrate Accounts From Exchange To Zimbra

Need to determine if these steps are conditional to certain Exchange versions or to all versions.

After an account is migrated to the Zimbra server from the Exchange server, you might need to delete the mailbox of the migrated account on the Exchange server so it will forward the mail to the Zimbra Server. You might also need to "Run Cleanup Agent" in the Exchange System Manager Console as well.

Unsetting Split-Domain Configurations (Migration is done and Split-Domain is no longer needed)

Simply null out the values/return them to their previous:

zmprov md domain.com zimbraMailCatchAllAddress 
zmprov md domain.com zimbraMailCatchAllForwardingAddress 
zmprov md domain.com zimbraMailTransport lmtp:primary.YYY.state.XX.us:7025

Accounts should already of been adjusted through the migration steps:

zmprov ma secondary-user@YYY.state.XX.us zimbraMailTransport lmtp:primary.YYY.state.XX.us:7025

Unset Mail Relay and DNS Lookups False:

zmprov mcf zimbraMtaRelayHost 
zmprov mcf zimbraMtaDnsLookupsEnabled TRUE

Different Email Server Type/Vendor To ZCS - Using Sub-Domain To Migrate

Yet to be written.

Sub-Domain Example 1 - Exchange Primary Domain & ZCS With Sub-Domain

Yet to be written.

Sub-Domain Example 1 - ZCS Primary Domain & Exchange With Sub-Domain

Yet to be written.

ZCS User to Another ZCS Server - With Rest & TGZ

Before Import - Confirm Your TGZ Is Good

On the server you'll be importing to, do the following.

  • Have a copy of the tgz file on the system, for example : /tmp/user_domain.tgz
  • Confirm the following doesn't give any errors
    • tar tzf /tmp/user_domain.tgz

Rest And The FMT= Option

See also:

Please consult /opt/zimbra/docs/rest.txt on your server for any version differences, the below is from 6.0.8.

  • fmt="tar", "tgz", "zip"
    • stream multiple items in an archive format. These forms are a superset of most other REST formatters .
  • Download options: export item data in raw or interchange format
    • body=0
      • only include msg headers, not bodies
    • charset=name
      • name character set for tar file and directory names as well as ical or vcard text. UTF-8 is default
    • emptyname=name
      • filename to set in content-disposition header if no data items are found in query instead of returning HTTP 204 error
    • file=name
      • filename to set in content-disposition header
    • lock=1
      • lock mailbox before running query to assure a complete, consistent snapshot
    • meta=
      • meta=1 (default)
        • item metadata from the DB is JSON encoded as .meta files. Associated data blobs follow immediately after
      • meta=0
        • data items saved in common interchange formats including eml for mail, ical for appointments and vcard for contacts. Briefcase files saved unmodified
  • Upload options: POST to the original folder or a subfolder in raw or MIME encoded data stream. If .meta files are included, data will be imported with original details. Otherwise, eml, ics and vcards are imported the same as with other formatters and all other files are imported to the briefcase
    • callback=name
      • name of javascript function used to return status when doing a direct browser upload.
      • window.parent.name("exception string", "exception class name", exception_code) is called when the upload completes.
      • plain html is returned by default if callback unspecified
    • charset=name
      • character set for archive file and directory names as well as ical or vcard text
    • resolve=[modfy|replace|reset|skip]
      • how to handle duplicate data conflicts
        • modify modify old item in place
        • replace delete old item and recreate
        • reset reset folder by deleting all item before importing
        • skip (default) skip duplicate items completely
    • subfolder=name
      • create subfolder under the destination to import items into
    • timestamp=0
      • do not use the archive entry date as the received date for msgs or the creation date for documents. Msg dates will be inferred from the Date header if available
    • timeout=mscec
      • update brower client with a text newline to prevent upload timeouts

The Basics

Please see [ZCS-to-ZCS Migrations] for use of this new tool option set.

Teaser, to export an entire mailbox with::

/opt/zimbra/bin/zmmailbox -z -m user@domain.com getRestURL “//?fmt=tgz” > /tmp/account.tgz

Create account on new server to import data to and then:

/opt/zimbra/bin/zmmailbox -z -m user@domain.com postRestURL “//?fmt=tgz&resolve=reset” /tmp/account.tgz

Note, this ends with "skip" - skip is default option. Resolve is how to handle duplicate data conflicts, options being: modify, replace, reset, skip . Please know what you want to do with the resolve= option or be testing against a test account rather than a production one. See /opt/zimbra/docs/rest.txt or Ajcody-Migration-Notes#Rest_And_The_FMT.3D_Option for more information on the fmt= options.

Time Out Errors

If you see errors about timing out, especially with large tgz file exports, then you'll want to include the -t 0 option for zmmailbox. T is for timeout and the 0 tells it to NOT have a timeout period. For example:

$ zmmailbox -z -m USER@DOMAIN -t 0 getRestURL "//?fmt=tgz" > /tmp/USER@DOMAIN-20100825.tgz

Problems Importing Data From Large Accounts

You might want to use the addMessage option instead of the postRestURL. For example:

tar zxvf user001.tgz
cd Inbox
zmmailbox -z -m user001@domain.com addMessage /Inbox *.eml
cd Sent
zmmailbox -z -m user001@domain.com addMessage /Sent *.eml

Work Around For Possible Proxy Or PreAuth Problems

Change With ZCS6+ vs ZCS5 - If Account Is Closed

This seems to be a ZCS 6+ change in behavior, when an account is in the closed status [zimbraAccountStatus: closed] and the zmmailbox steps to export above don't work - auth failure type errors. Work around, change account status or use the curl steps listed here.

Example of error you'll see:

$ zmmailbox -z -m USER@DOMAIN getRestURL "//?fmt=tgz" > /tmp/USER@DOMAIN-20100825.tgz

ERROR: service.FAILURE (system failure: GET from 
https://www.DOMAIN/home/USER@DOMAIN/?fmt=tgz failed, status=401.  must authenticate)
Using Curl To EXPORT TGZ Data

This is a work around if you see the following error when attempting the export via zmmailbox -- "status=401. must authenticate" . For admin:PASSWORD below you could use admin or a domain admin username and the appropriate password. You can also either use your 'global' url or the specific mailstore the user resides on. Notice the use of https and the :7071 , this is critical to use an admin or domain admin account to do the export.

curl -k -u admin:PASSWORD https://MAILSTORE-FQDN:7071/service/home/USER@DOMAIN/?fmt=tgz > ./USER-account.tgz
Using WGET To EXPORT TGZ Data

This is a work around if you see the following error when attempting the export via zmmailbox -- "status=401. must authenticate" . For admin:PASSWORD below you could use admin or a domain admin username and the appropriate password. You can also either use your 'global' url or the specific mailstore the user resides on. Notice the use of https and the :7071 , this is critical to use an admin or domain admin account to do the export.

wget --no-check-certificate -O ./account.tgz https://admin:PASSWORD@MAILSTORE-FQDN:7071/service/home/USER@DOMAIN/?fmt=tgz
Using ZWC Via Admin Port To EXPORT TGZ Data

In your browser, do something like the following:

https://admin:PASSWORD@MAILSTORE-FQDN:7071/service/home/USER@DOMAIN/?fmt=tgz
Using Curl To IMPORT TGZ Data

To import a tgz file into zimbra using curl, use the format example shown below:

curl -k -u admin:PASSWORD --data-binary @/path/to/USER-account.tgz "https://MAILSTORE-FQDN:7071/service/home/USER@DOMAIN/?fmt=tgz&resolve=skip"

Note, this ends with "skip" - skip is default option. Resolve is how to handle duplicate data conflicts, options being: modify, replace, reset, skip . Please know what you want to do with the resolve= option or be testing against a test account rather than a production one. See /opt/zimbra/docs/rest.txt or Ajcody-Migration-Notes#Rest_And_The_FMT.3D_Option for more information on the fmt= options.

Too Large Or Timeout Errors With Proxy

Update - rest_request_max_upload_size does not exist in code. This key was removed with change 399480 as part of a code revert. See bug :

Apply the LC rest_request_max_upload_size instead of zimbraFileMaxUploadSize for imports when using nginx proxy or if your using curl [ZCS 7.2.1+] .

See:

  • "unable to import .tgz using ImportUI through the Proxy "
    • https://bugzilla.zimbra.com/show_bug.cgi?id=74622#c35
      • Introduce LC config "rest_request_max_upload_size" with default value of 1073741824 (1GB). Limit the file uploads of tar/zip/tgz/ics formatters to this size.
      • To set to 1GB [as zimbra] :
        • zmlocalconfig -e rest_request_max_upload_size=1073741824

See also:

To Just Export & Import Contents Of "Something"

To export/import just the Inbox:

zmmailbox -z -m user@domain.com getRestURL "//?fmt=tgz&query=in:inbox/inboxsub" > /tmp/userinboxsubin.tgz
zmmailbox -z -m userinboxsubin@domain.com postRestURL "//?fmt=tgz&resolve=replace" /tmp/userinboxsubin.tgz

To export/import Inbox and Subdirectories. If the query is [ in: ] , it just gets from that folder. If it's [ under: ] , it'll grab anything including and beneath that folder.

zmmailbox -z -m user@domain.com getRestURL "//?fmt=tgz&query=under:inbox/inboxsub" > /tmp/userinboxsubunder.tgz
zmmailbox -z -m userinboxsubunder@domain.com postRestURL "//?fmt=tgz&resolve=replace" /tmp/userinboxsubunder.tgz

Export Using Query String - Before And After Dates

Note - Make sure you scroll web page to the sides to see the complete command examples!


Tested against ZCS 8. This is an example [user@`zmhostname` should be replace below with the targeted user@domain information] :

**Updated to include this new example at the top.**

zmmailbox -z -m user@`zmhostname` gru '//?fmt=tgz&query=under:/ after:"4/8/15" AND before:"4/11/15"' > /tmp/test_export/test_export.tgz

Or:

 zmmailbox -z -m user@`zmhostname` getRestURL "//?fmt=tgz&query="after:2/1/2014\ before:3/15/2014\ type:message\ meta=0"" > /tmp/test_export/test_export.tgz

Or :

zmmailbox -z -m user@`zmhostname` gru '/?fmt=tgz&query=after:"10/01/14" and before:"10/15/14"' > /tmp/filename.tgz

Or:

zmmailbox -z -m user@`zmhostname` gru -u https://localhost '/?fmt=tgz&query=after:"4/30/13" and before:"8/1/13"' > /tmp/filename.tgz

Using ZD To Import Zimbra TGZ Export User Data

See :

Old ZCS To New ZCS - Using LDAP Replica To Migrate

If your moving users to a new mailstore, please see King0770-Notes#Preferred_Method_Moving_Users_To_New_Machine for a possible solution. This process implements a new LDAP replica's with the new ZCS mailstore. Over time you migrate the account to new ZCS with zmmailboxmove. Once the accounts have all been moved, you prompt the LDAP Replica configuration on the new ZCS server to be the LDAP master. Adjust for the MTA configuration and then retire the old box.

Old ZCS To New ZCS - Using Rsync Method To Migrate

The rsync method is described in Ajcody-Notes-Server-Move would require downtime.

Old ZCS Email Domains To Another ZCS With Existing Domains

LDAP Method - Untested & Not Finished

The process would involve migrating the LDAP data for the domain from the old zimbra server into the other existing Zimbra system (using slapcat or ldapsearch). You'll need to get the server entries, any customer COS entries, the domain entry, and all users under that domain. You would then load the information (ldif) into the existing Zimbra server with slapadd. You could then reconfigured the old server to become an ldap slave to the existing Zimbra box and to server simply as a mailstore for that domain data (users). One could also proceed with zmmailboxmove if you needed to migrate the user data onto the other box as well now.

Adjustments would also need to be done in regards to auth keys and server certs.

Sub-Domain Method - Untested & Not Finished

Situation

Let's say you own a hosting company and a new customer says they are running Zimbra Network edition (same version as yours) and wants you to host them. They only have one domain and want to minimize downtime. Their server is in another country or state. They have 50 users.

Realistically, during the migration time frame, those accounts that will be using the temporary sub-domain until the switch is complete should only use the zimbra web client. Phones, thick clients, and so forth will introduce issues about synchronization and so forth that would be better to simply avoid.

Split-Domain Setup

Setup a split domain that you can use through the migration time frame.

They are company.com

You'll be migrate.company.com

See Ajcody-Migration-Notes#Zimbra_And_Linux_Resources about Split-Domains and Domain Masquerading. Other adjustments might need to be applied.

The situation will be this:

  • All emails maintain their company.com domain and email comes and goes through customers ZCS server.
  • Customers ZCS will be using forward/alias for account to goto @migrate.company.com
    • User Forward/Alias
      • Need to figure out the right way to do this for this situation still.
    • Relaying/Domain Forwarding
      • zmprov
      • md example.com zimbraMailCatchAllAddress @migrate.company.com
      • md example.com zimbraMailCatchAllForwardingAddress @migrate.company.com
      • md example.com zimbraMailTransport smtp:mta.HOSTING-COMPANY.com
  • Accounts migrated to Hosting company server will have rewrite rules. So migrated accounts will not be exposed to "outside" world.
    • zmprov md migrate.company.com zimbraMailCatchAllAddress @migrate.company.com zimbraMailCatchAllCanonicalAddress @company.com
    • Need instructions to configure for this particular domain mta rules do basically do the below WITHOUT effecting the other domains with the hosting companies ZCS infrastructure.
      • $ zmprov mcf zimbraMtaRelayHost mta.COMPANY.com
      • $ zmprov mcf zimbraMtaDnsLookupsEnabled FALSE
Break Up User Migration

Determine group of users that make sense to migrate based upon their functions and mailbox sizes. Some items might not work 100% as expect while in transition - i.e. Calendars, Resources, Wiki, etc..

Set Accounts To Maintenance

Set the targeted accounts to Maintenance status. They should stay this way after the backup is completed and will stay like that until the migration to the new server is complete and the split-domain is setup.

Backup

On the customers ZCS server, do a backup of the selected users. Be careful of where you direct the backup to and that you'll have space.

mkdir /tmp/backup-accounts
zmbackup -f -a userA@company.com userB@company.com userC@company.com -t /tmp/backup-accounts
Problem With Backup

Ideally, we would then do something like this:

zmbackup -i -a userA@company.com userB@company.com userC@company.com -t /tmp/backup-accounts

But this will fail, with:

Error occurred: system failure: Customer backup target is not allowed for incremental backup

And if you don't use the customer target it seems like the incremental picks up all the accounts. The point of wanting this would be to send a full via mail if needed and to allow the incrementals to be small enough to transfer over the network between the customer site and the hosting company.

Set Company Domain Onto Hosting Server

You'll need to setup the companies domain on your server before the zmrestore will work. You might as well configure the COS to make what the customer is using.

If your using a non-default COS.

  • zmprov cc companyCOS
    • adjust in web admin console if needed
  • zmprov gc companyCOS | grep zimbraId
    • zimbraId: 420447db-ea43-4d92-b62d-0164767f516d
  • zmprov cd company.com
    • adjust in web admin console if needed
    • zmprov md company.com zimbraDomainDefaultCOSId 420447db-ea43-4d92-b62d-0164767f516d

Set this domain to Maintenance mode.

Restore Account Onto Hosting Server
zmrestore -a userA@company.com userB@company.com userC@company.com -t /tmp/backup-accounts
Move Restored Accounts Into Split-Domain

If split domain is already setup:

zmprov ra userA@company.com userA@migrate.company.com

If not, your could do a domainrename:

zmprov -l rd company.com migrate.company.com
Forward Accounts That Were Moved

On the customers server, you'll now setup those accounts to forward to the split-domain.

Email for userA@company.com forwards to userA@migrate.company.com

Here's another problem though. We need to find a way to switch to a forward/alias on the companies ZCS so that emails don't get bounced with "no such account". You can't setup an alias as long as the account exists on the box.

Split-Domain Method - Untested & Not Finished

Summary Of Idea
I'm sure there's a way you could have both servers hosting the company.com domain at the same time with separate users. Where you had all mail going to a shared non-zimbra MTA host that would direct delivery to either the Hosting or Client ZCS server depending on where the user was at. Of course, you would need to have DNS setup to handle this type of setup (both internal and external). Could your Postini, MailProtector, or other such device do this? Then you would need to figure out a way to handle internal domain delivery between the two systems. The defaults, would have lmtp rejecting the delivery thinking the account doesn't exists because it believes it's authoritative for the domain. Maybe the use of subdomains for aliases/forwarding work be the work around for this.
Diagram Of Flow
                            ALL Company.com
                           emails to and from

                               MTA HEAD
                     lookups username to determine
                          server to deliver to

                                either/or
 userA@company.com is                             userB@company.com is
 userA@hosting.company.com                       userB@client.company.com
                    (above is done on MTA device)

 hosting server has:                             client server has:

 hosting.company.com has a                        client.company.com has a
 domain alias pointing to company.com             domain alias pointing to company.com
  * see http://wiki.zimbra.com/index.php?title=ManagingDomains#Creating_a_Domain_Alias *

 hosting server knows (MX record)                client server knows (MX record)
 Eternity server for delivery of                 hosting server for delivery of  
 eternity.company.com accounts (alias)           volaria.company.com accounts (alias)
  * also see http://wiki.zimbra.com/index.php?title=ManagingDomains#Relaying.2FDomain_Forwarding *

 Summary of hosting.company.com ZCS server
 Has a domain called company.com
 Has a domain alias called hosting.company.com that points to company.com
 All users are setup within company.com
 Migrate user(s) via zmrestore command into the domain (company.com)
 Non-migrated users will have alias setup to forward to username@client.company.com
 Will have mail-relaying/forwarding setup for client.company.com

 Summary of client.company.com ZCS server
 Has a domain called company.com
 Has a domain alias called client.company.com that points to company.com
 All users all ready exist within company.com
 Non-migrated user(s) will remain untouched
 Migrate user(s) via zmbackup with individual or list of users as flag
 Migrated users will have alias setup to forward to username@hosting.company.com
 Will have mail-relaying/forwarding setup for client.company.com

I've only just started to think this one possible solution through, maybe you can pick up what I have and go with it. To see if it might or might not work. Of course there's going to be a performance overhead on this in regards to mail delivery - because of the MTA Head being needed for routing issues. And there's also any security (spam as well) issues to think through.

Some Zimbra Users From DomainA to DomainB On Same Zimbra Server

All Accounts From DomainA to DomainB

You would use [renameDomain]:

zmprov -l rd domainA.com domainB.com

Please review domainrename though, there's some outstanding issues with this depending on your ZCS version.

Account From DomainA to DomainB

You want to use [renameAccount]:

zmprov ra userA@domainA.com userA@domainB.com

This is same command to rename account within the same domain:

zmprov ra userA@domainA.com userB@domainA.com

Non-Zimbra IMAP Accounts To Zimbra

IMAPSYNC References

General IMAPSYNC References:

IMAPSYNC with admin login

Example of using admin login:

imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates \
--host1 server1.test.org --user1 yourAccount --password1 yourPassword \
--host2 zimbra.test.org --user2 yourZimbraAccount --authuser2 admin \
--password2 adminZimbraPassword --authmech2 LOGIN

You can also try --authmech2 PLAIN

I found this description in one of the imapsync files:

"You may authenticate as one user (typically an admin user), but be authorized as someone else, which means you don't need to know every user's personal password. Specify --authuser1 "adminuser" to enable this on host1. In this case, --authmech1 PLAIN will be used, but otherwise, --authmech1 CRAM-MD5 is the default. Same behavior with the --authuser2 option."

Imapsync During A Certain Time Frame

You might want to check out the imapsync options of --maxage and --minage

Performance Issues With IMAPSYNC

MS Exchange , IMAPSYNC, And ExMerge [Microsoft Exchange Server Mailbox Merge Wizard] For Email Other Data

See Ajcody-Migration-Notes#IMAPSYNC_References for IMAPSYNC references.

ExMerge References:

MS Exchange To Zimbra By Way Of Zimbra's Migration Wizard for Exchange

References:

Office 365 Specifics

We also have these two RFE's requests:

PST Import Options

References:

Zimbra Data Back To Another Server (Exchange)

I understand there are numerous reasons why a company might need to change the mail server software, most of them are probably not technical reasons. To help where I can for this situation, I'll start documenting the information, tricks, and tools I discover to make this process more easier for you - unfortunate that it is.

The Zimbra Support team doesn't support exporting data into a non-zimbra server. You would normally get support from the company of the non-zimbra server, since you would be using their tools and processes for importing. We can only offer the references below and also point out different "export" abilities and the standard formats within Zimbra. The one migration wiki article goes over those examples.

  • Exchange Server - We don't have anything specific about moving data from Zimbra to Exchange, but the following should help or apply.
    • If you enable IMAP on Exchange, you could use IMAPSYNC. This tool is mentioned in the below url.
      • There's also the possibility of exporting the data to PST/OST files from Outlook as well.
    • You'll most likely need to manually recreate all users, resources, and dl's in Exchange - unless you can work through a way to import the ldap data from the zimbra server into the exchange/active directory server. Calendar data can be export out in the ical format - I think there might be third party tools for Exchange to import this.

I did a number of example's about migrations in general on the wiki, these could be adjusted for your situation. They also include url resources on the Exchange configuration.

I would highly recommend you reaching out to the sales group and exploring other options with them as well. Contact Information



Clustering Topics

   KB 2494        Last updated on 2008-11-16  




0.00
(0 votes)
24px ‎  - This is Zeta Alliance Certified Documentation. The content has been tested by the Community.


Actual Clustering Topics Homepage


Please see Ajcody-Clustering

My Other Clustering Pages


RFE I made based upon the experience above:

Clustering For ZCS 8 And Above


ZCS 8 Specifics

From the ZCS 8.0.5 Release Notes:

Red Hat Cluster Suite is not available with ZCS 8.0 To streamline support efforts, we will only test and certify in house availability solutions. Today VMware offers clustering failover and automated recovery through VMware HA. Zimbra integrates with the VMware HA cluster infrastructure to heartbeat Zimbra application services and provide automated recover in the event of a service failure. (Bugs 72215/72216)

If you use Zimbra Clustering

Zimbra Clustering is no longer available for ZCS 8.0. VMware provides integrated high availability between Zimbra and VMware HA for automated recovery of critical Zimbra application services and server components in the event of an application or infrastructure failure. Third party solutions such as network load balancers, storage mirroring, or OS clustering solutions like Red Hat Cluster Suite may be used in your deployment, but are not specifically tested or certified by VMware. (Bug75821)

Statement On Zimbra Support And "Clustering" For ZCS 8+ [In My Own Words - Ajcody]

I've sent this off to our PM team to review and to provide formal comments and documentation updates on it. - Ajcody

Official and Full support are NOT terms Zimbra Support applies to Redhat RHCS or Vmware's Clustering options since our team does not support either of those two options directly. We do not support customers in configuring or trouble-shooting the clustering components for Redhat's RHCS or Vmware's Clustering.

Will and can RHCS and Vmware's Clustering work with ZCS, yes. Does our team offer support in configuring, trouble-shooting, or diagnosing the clustering component of the setup - no. You would need to contact Redhat if you went with RHEL and RHCS or Vmware and your Linux OS distribution support channel if you went with Vmware Clustering.

Note - The zimbra clustering zimlets are completely removed with ZCS 8.6 . The Vmware clustering monitoring scripts are a separate issues that falls outside of general clustering support.

Future Releases After ZCS 8

Please see the following blog about our Always ON, Carrier Grade Architecture HA features we are hoping to have in future releases:

Vmware Virtualization and Clustering


Overview

From the ZCS 8.0.5 Release Notes:

Red Hat Cluster Suite is not available with ZCS 8.0 To streamline support efforts, we will only test and certify in house availability solutions. Today VMware offers clustering failover and automated recovery through VMware HA. Zimbra integrates with the VMware HA cluster infrastructure to heartbeat Zimbra application services and provide automated recover in the event of a service failure. (Bugs 72215/72216)

VMware Heartbeat Service

ZCS utilizes the VMware application programming interface that allows software providers to deploy application monitoring components inside a VMware guest OS and inform VMware HA when problems arise. The VMware-heartbeat service provides information to the VMware HA components on the health and availability of the ZCS. If you have VMware HA components installed, the VMware HA service is shown as enabled on the Server>Services page.

General Description of VMware HA

VMware HA provides a simple, reliable way to increase the availability of virtual machines hosting critical applications. VMware HA is a visualization- based distributed infrastructure service of VMware vSphere 4.1+, which monitors the health of virtual machines and the VMware ESX® hosts upon which they reside. If a fault is detected, the virtual machine is automatically restarted on another ESX host with adequate capacity to host it. VMware HA is included in all vSphere editions and can be enabled on a VMware cluster with a single check box. As VMware HA utilizes the storage and network connectivity already in place to support vMotion, enabling high availability is as simple as ensuring you have adequate server capacity to handle failure of one or more ESX hosts.

ZCS Scripts For Vmware-HA

The three scripts for vmware-ha are:

  • /opt/zimbra/bin/zmhactl
  • /opt/zimbra/libexec/vmware-heartbeat
  • /opt/zimbra/libexec/vmware-appmonitor

We have an open documentation RFE to better explain these :

Some older bugs describing a little more about the vmware-heartbeat check:

Enable Vmware-HA Service

To confirm the service is available to enable:

[As the zimbra user]

zmprov gs `zmhostname` zimbraServiceInstalled | grep vmware-ha
zimbraServiceInstalled: vmware-ha

To enable vmware-ha on a server:

[As the zimbra user]

su - zimbra
zmprov ms `zmhostname` +zimbraServiceEnabled vmware-ha
zmhactl start

Vmware Performance Recommendations For Vmware Clustering

See

VMware Host Based Replication (HBR)

Questions should be addressed to Vmware Support. This feature happens outside of Zimbra and is not QA tested by us.

Private Bug [Referenced here in case it's later made public]:


Other References

See also:

Webinar On Virtualizing Zimbra with VMware vSphere and NetApp NFS

Learn how the combination of VMware vSphere and NetApp file-based storage provides full support for virtualization and all of vSphere's advanced features (vMotion, Storage vMotion, DRS), simplifies high availability through the use of VMware HA and server consolidation.

View Webinar Now (Duration: 45 minutes)

Critical Bugs/RFE's - False Restarts And So Forth

Log Rotation Causes Cluster Failover

See:

Recommendations to work around bug until fix is released - 2 methods:

  • First Method - disable log rotation
    • Remove the zmmtaconfigctl restart from the /etc/logrotate.d/zimbra file so that it will not attempt to restart the service that is being detected as down. We can remove the line that reads:
su - zimbra -c "/opt/zimbra/bin/zmmtaconfigctl restart"
    • That will stop these failures, but will affect the logging for zmmtaconfigctl, probably causing it to write to a nonexistent file. We haven't seen problems in zmmtaconfig for a long time, so this is a pretty low risk workaround.
  • Second Method - disable software monitoring in general

Software Monitoring Causes Problems

  • To Disable software monitoring.
    • This will prevent failover if zmcluctl finds a service down. It will not prevent failover if there is a hardware fault detected by the cluster software.
      • $zmcluctl mode hardware
      • or
      • $zmlocalconfig -e zimbra_cluster_mode=hardware
      • Note from bug/rfe - Added switch for software or hardware only based fail-over implemented as a lc zimbra_cluster_mode attribute. default mode is software meaning any non-zero exit status from zmcontrol status will trigger a failover. hardware mode means that zmcluctl status will always return a zero exit status and will not consult zmcontrol status for the state.
    • To also increase the chance of getting more information in the log on what might be going on:
      • In /opt/zimbra-cluster/bin/zmcluctl you should see a line like:
        • my @output = `su - zimbra -c 'zmcontrol status'`;
      • Change that to:
        • my @output = `su - zimbra -c 'date >> /opt/zimbra/log/zmcluster-status.log ; zmcontrol status >> /opt/zimbra/log/zmcluster-status.log 2>> /opt/zimbra/log/zmcluster-status.log'`;
      • That should give us more logging. I believe zmcluctl is read every time from disk when it does the check, so no restart of services should be needed.

RHEL 5 Clusters And Cisco Switches

Please see the following:

Mysql Related Items Impacting Cluster

Please see:

Other Misc Bug/RFEs

Other bugs/rfe's you might be interested in looking at:

Failover Occurring Prior To Mysql Shutting Down - RHCS

Here is the summary of the situation that was sent for a case on the matter.

Regarding RHCS desired configuration -
In particular, it looks like mysqld can at times take quite a long time to shutdown, especially in situations when write IO is limited on the mysql db partition, and mysqld must write cache data to disk safely before shutting down. We believe this should be reflected in the "rc=1" output from "zmcontrol stop" - if mysqld does not stop within the 60 seconds via /opt/zimbra/bin/zmcontrol (via /opt/zimbra/bin/mysql.server), and then what appears to be happening is that RHCS either tries to unmount the filesystem and/or just kill -KILL the running process, which in turns causes mysqld to die ungracefully and cause the mysqld startup on the other box to go through a long log replay. The net result is one of two scenarios:
(a) mysqld takes longer to startup than it would have taken to shutdown (normal shutdown can take 0-10 minutes if mysqld has to flush a bunch of data to write out)
(b) mysqld is still running when RHCS tries to unmount the filesystem, which then fails, and therefore the failover fails
What we'd like to be able to recommend is a more robust failover logic, such as:
1. Run zmcontrol stop, check output
2. If failed, wait X seconds (e.g., 120 seconds)
3. Run zmcontrol stop again
4. If failed, wait X seconds
5. Run zmcontrol third time
6. If failed third time, try a kill -KILL and force unmount
If RHCS can be configured to do this, it would be the best way to handle this situation. The problem with modifying zmcontrol to have this logic internally is that it should not be granted the decision to wait indefinitely, in the case that mysqld is having a critical problem and is not shutting down. Having an external process manage this logic allows for additional triangulation on the determination of the situation, and allows additional capabilities such as monitoring the files in /opt/zimbra/db/data to confirm that files are changing. For example, if mysqld is still running and no files have changed in /opt/zimbra/db/data for a significant amount of time, it may indicate that mysqld is locked up, and other steps may be required. zmcontrol would not be able to handle all of this logic internally due to the needs for additional external monitoring outside of the zmcontrol process, including return code from the script.

RFE that was made about Zimbra adjusting the our script and the wait time:

Good Summary For RHEL Clustering


This is a good solid summary about RHEL clustering:

http://www.linuxjournal.com/article/9759

Active-Active Clustering


There is a bug(rfe) for active-active configuration. Please see:

http://bugzilla.zimbra.com/show_bug.cgi?id=19700

Non-San Based Fail Over HA/Cluster Type Configuration


This RFE covers issues when your wanting a "copy" of the data to reside on an independent server - LAN/WAN.

Please see:

RFE's/Bug Related To Supporting Clustering Options


Other Clustering RFE's And Bugs


HA-Linux (Heartbeat)



HA-Linux How-To For Testing And Educational Use

References:


Actual HA-Linux How-To For Testing And Educational Use Homepage

Please see Ajcody-Notes-HA-Linux-How-To

Motive Behind How-To

I hope this gives an easy way to setup through some clustering concepts for an administrator to gain some real-world experience when they currently have none. I plan on walking through each "function" that is behind clustering rather than jumping to an end setup (Linux-HA, Shared Storage, And Zimbra).

The structure will be:

  • Setup two machines (physical or virtual)
    • Emphasis physical hostname / ip vs. the hostname and ip address that will be for HA.
  • Setup virtual hostname and ip address for HA.
    • Explain and do ip failover between the two machines.
  • Setup a disk mount, we'll use probably use a nfs export from a third machine.
    • This will give us an example of expanding the HA conf files to move beyond the ip address failover.
    • Adjust HA conf's to now export via nfs a local directory from each server. This will not be a shared physical disk of course.
  • Setup a shared disk between the two servers and include it in the HA conf files.
    • Can use drbd or maybe figure out a way to share a virtual disk between the two vm's.
  • Setup a very simple application to include between the two machines. Something like apache or cups.
  • Go back and now readjust all variables between monitoring type (automatic) failover and simple manually initiated.



Virtualization Issues

   KB 2494        Last updated on 2008-11-16  




0.00
(0 votes)
24px ‎  - This is Zeta Alliance Certified Documentation. The content has been tested by the Community.



Actual Virutalization Issues Homepage


Please see Ajcody-Virtualization

Vmware - ESX - Performance & Support Resources

Internal Resources:


External Resources:

Vmware And Clustering

Please see Ajcody-Clustering#Vmware_Virtualization_and_Clustering

Introduction To Using VMWare ESX For ZCS Test Servers

References

References I ended up consulting to resolve and develope my notes below. There appears to be many alterations on how to do this depending on the version of the Vmware software you are using.

Introduction To Notes

I just did this and I plan on going through it and improving it once I have time.

  • I believe the hard coded reference to the volumes could use the sym-link for the path:
    • /vmfs/volumes/Storage vs. /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af
  • Setup /etc/hosts file on first vmimage to have entries for other machines
  • Configure first vmimage to use ESX server for DNS [/etc/resolv.conf] , NTP , and other network services
  • Configure vm configuration files, *.vmx , to use manual set MAC addresses and had last number matched number used in hostname.
  • Configure ESX server to act as DNS & DHCP server using to auto-allocate ip address and hostnames and set each hostname to have proper A, PTR, and MX records using hostname of image equally domainname for mail.
    • for example:
    • I would create three groups of images, if possible:
      • host.dev.DOMAIN.com
        • DEV is for pure testing
      • host.qa.DOMAIN.com
        • QA is for testing of changes prior to roll out to production
      • host.prod.DOMAIN.com
        • PROD would be replica's of production and used to test against production issues
    • Images configured as:
      • image hostname = centos5-30.dev.DOMAIN.com
      • server centos5-31.dev.DOMAIN.com has ip address of 192.168.0.31 - set A and PTR record
      • MX equals centos5-31.dev.DOMAIN.com / 192.168.0.31
    • Use more expansive descriptions in hostnames for standards and predictability.
      • Note, underscores ARE NOT vaild DNS hostnames per RFC 1034 - http://www.ietf.org/rfc/rfc1034.txt
      • Add the DEV - QA - PROD if your using that as well.
      • Primary Domain = example.com
        • LDAP Master hostname = ldap-1.DOMAIN.com
        • LDAP Replica hostnames = ldap-2.DOMAIN.com , ldap-3.DOMAIN.com , etc.
        • First MTA hostname = mta-1.DOMAIN.com
        • Other MTA hostnames = mta-2.DOMAIN.com , mta-3.DOMAIN.com , etc.
        • First Mailstore hostname = mailstore-1.DOMAIN.com
        • Other Mailstore hostnames = mailstore-2.DOMAIN.com , mailstore-3.DOMAIN.com , etc.
      • On proxy server hostname =
        • Proxy on MTA's: mta-1.DOMAIN.com , mta-2.DOMAIN.com , etc.
          • Though the proxy package can be installed on other servers, usually it is installed on the mta's or on it's own.
        • Proxy on it's own: proxy-1.DOMAIN.com , proxy-2.DOMAIN.com , etc.
      • On archive [A&D] hostname:
        • First Archive hostname = archive-1.DOMAIN.com
        • Other Archive hostnames = archive-2.DOMAIN.com , archive-3.DOMAIN.com , etc.
  • Configure proxy/firewall to be route point for primary domain and then pass to appropriate server for subdomain mail.
    • All external requess for *.zimbra.DOMAIN.com and zimbra.DOMAIN.com route to ESX server which in turns routes to appropriate vm server for subdomain.
  • Ports open on firewalls to allow vsphere connection to esx server:
  • Add post-clone script to the "gold image" to reconfigure the cloned image for things like hostname & network information. [Thanks Tony for the idea]

How-To Setup ESX For ZCS Test Servers

Creating Initial x64 VM

Initial Setup Of Guest Image In vSphere
  1. Right Click on ESX server name in left column listing in vSphere Client.
    1. Select "New Virtual Machine"
      1. Select "Typical"
      2. Name - ex. "Centos5-x64-30"
        1. Format being - Distro & Distro Version - platform x32 or x64 - last octet of ip address
      3. Datastore - your local esx data storage you'll be using
      4. Guest OS > Linux > RHEL5 64bit - RHEL for Centos [using my example]
      5. Create a disk
        1. Defaults to 8GB, but this isn't enough for ZCS because ZCS requires at least 5GB's free. The OS will take about 4-6+GB's between the swap partition, /boot , and /. Using installer defaults that is.
        2. I make the root image for 12GB. This will be enough to get a basic ZCS install done. One can add more later if needed.
          1. For a 'test' environment, I would recommend leaving the other options UNCHECKED.
            1. Leave UNCHECKED - Allocate and commit space on demand
            2. Leave UNCHECKED - Support clustering features such as Fault Tolerance.
      6. Ready to Complete - Check the box that says:
        1. check - "Edit the virtual machine settings before completion"
          1. Memory > Adjust Memory to be 1024 MB or 1 GB
          2. CD/DVD Drive 1 >
            1. Check "Connected at power on"
            2. Check "Datastore ISO File" and select the iso image for your OS - ex. CentOS-5.4-x86_64-bin-DVD.iso.
              1. Centos default disk partitioning will do:
              2. Also, see - need wiki section - about how to prep your server to host iso images via the datastore.
          3. The rest can be left as the defaults for now.
Initial Setup Of Guest Image Operating System - CentOS/RHEL Example
  1. In the vSphere Client and right click on Guest image, ex. "Centos5-x64-30", and select power on.
    1. Prep our 'base image' prior to cloning it.
    2. The image should give you your Distro's installer screen. Go through the OS installation.
      1. Note about CentOS example
        1. First stage of installer
          1. The partitioning example if using the defaults will give you:
            • /dev/sda1  /boot       ext3      101 MB
            • /dev/sdb2  VolGroup00  LVM PV  12182 MB
              • LogVol00  /  ext3  10144 MB
              • LovVol01     swap   2016 MB
          2. The Network Devices
            1. Network Devices > click on edit
              1. Leave CHECKED - "Enabled IPv4 support"
              2. UNCHECKED "Enable IPv6 support"
          3. Package Selection
            1. Check the "Server" option and then click "Next"
              1. Installer will now finish the first stage of the installation and reboot.
        2. Second state of installation.
          1. Firewall > Disable firewall.
          2. SELinux > Disable SELinux
          3. Date and Time
            1. Network Time Protocol - Enable NTP if you'll be able to reach the NTP servers.
              1. Check the "Synchronize system clock before starting service"
          4. Create User > No need to create additional users for ZCS purposes.
          5. Finish installation.
        3. Post-Installation
          1. Note - Ctrl+Alt allows you to switch the focus of the mouse in and out of the VM guest image showing in vSphere's client.
          2. Note - The client 'screen' is under the "Console" tab.
          3. Install VMwareTools
            1. In the vSphere Client and right click on Guest image, ex. "Centos5-x64-30", and select Guest > Install\Upgrade VMware Tools.
            2. Double click on the VMwareTools rpm or install it via CLI.
            3. You can also run the vmware-tool configuration script
            4. vmware-config-tools.pl
            5. Allows you to adjust the screen resolution and cut-n-paste between workstation and the guest vm.
          4. Launch a terminal. Applications > Accessories > Terminal
            1. Disable sendmail from being used. Note - CentOS/RHEL uses sendmail as default, for another distro you might need to disable postfix from being used.
              1. In terminal, paste [if you installed vmware tools] or type the following command.
              2.  chkconfig sendmail off ; /etc/init.d/sendmail stop
            2. Install some prereq packages that you might not already have installed - CentOS example:
              1. yum install compat-libstdc++-33 compat-libstdc++-296 sysstat
            3. Umount the vmware-tools mount - example
              1.  umount /media/VMware\ Tools/ 
              2. confirm with df -h
            4. Reboot server now - the guest OS image.
          5. See also Ajcody-Virtualization#RHEL_or_CentOS for adding 'disk/partitions' to an image.

Manual Cloning of x32 Setup

My Version of Vmware ESX does not have a cloning option - I was actually given the ESX box rebuilt. So here's what I did to 'clone' an image the manual way. See Ajcody-Virtualization-Named-DNS about setting up BIND/DNS for this example.

# Vmware ESX 4.0.0 Build 208167
# vSphere Client 4.0.0. Build 208111
#
# Copy some iso files to the server, here's is what I used. 
# Please note the vmx files below in my examples to my iso name for the centos dvd.
#   [ /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/iso/CentOS/CentOS-5.4-i386-bin-DVD.iso ]
# Create your first VM - I called my Centos5-30 and that standard and naming convention is used throughout.
# I left the hostname just to use the localhost and installed the vmware tools.
# I then shutdown the image and did a snapshot of it.

pwd
  /vmfs/volumes

ls -la
  total 1024
  drwxr-xr-x 1 root root  512 Apr  9 23:59 .
  drwxrwxrwt 1 root root  512 Apr  9 00:39 ..
  drwxr-xr-t 1 root root 4200 Apr  9 23:20 4bbaf57f-6230127f-d432-00101849e4af
  lrwxr-xr-x 1 root root   35 Apr  9 23:59 Storage1 -> 4bbaf57f-6230127f-d432-00101849e4af

cd Storage1/

ls
  Centos5-30/      esxconsole-4bbb574e-eae0-2eb3-9d5b-00101849e4af/ 	iso/

# Download and copy the various OS iso's you'll want and copy them underneath your storage volume, 
# I made an iso directory for them all.
# Also, I used Firefox and the add-on DownThemAll and went to the Zimbra download pages
# http://www.zimbra.com/downloads/ne-downloads.html
# http://www.zimbra.com/downloads/ne-downloads-previous.html
# And downloaded all the versions of ZCS that I wanted to test against. I then used K3B under Linux
# to make iso files of the tar balls. Copy the iso files to your ESX server under the iso directory
# you made. If you want to mount them remotely to confirm they are ok to or to review what they have:
# ex.  mount -t iso9660 ./zcs-x32-installs.iso /mnt/cdrom -o ro,loop
# Having the Zimbra iso files like that will allow your images to easily install Zimbra .

ls -R iso/
  CentOS/  openSUSE/  SLES/  Ubuntu/  ZCS/
  ./CentOS:
    CentOS-5.4-i386-bin-DVD.iso  CentOS-5.4-x86_64-bin-DVD.iso
  ./openSUSE:
    openSUSE-11.2-Addon-NonOss-BiArch-i586-x86_64.iso  openSUSE-11.2-DVD-i586.iso  
    openSUSE-11.2-DVD-x86_64.iso
  ./SLES:
    SLES-10-SP3-DVD-i386-GM-DVD1.iso  SLES-10-SP3-DVD-x86_64-GM-DVD1.iso  SLES-11-DVD-i586-GM-DVD1.iso
    SLES-10-SP3-DVD-i386-GM-DVD2.iso  SLES-10-SP3-DVD-x86_64-GM-DVD2.iso  SLES-11-DVD-i586-GM-DVD2.iso
  ./Ubuntu:
    ubuntu-8.04.4-desktop-amd64.iso  ubuntu-8.04.4-desktop-i386.iso  
    ubuntu-8.04.4-server-amd64.iso  ubuntu-8.04.4-server-i386.iso
  ./ZCS:
    zcs-x32-606-02_5023-018.iso  zcs-x64-5018-23.iso  zcs-x64-603-06.iso

vmware-cmd -l
  /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-30/Centos5-30.vmx

pwd
  /vmfs/volumes/Storage1

# Make more directories for you to copy the initial image vitrual disks to

mkdir Centos5-{31..49}

# Now you'll do a copy operation of your first image to the new directories you made

pwd
  /vmfs/volumes/Storage1

vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-31/Centos5-31.vmdk
vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-32/Centos5-32.vmdk
vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-33/Centos5-33.vmdk
vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-34/Centos5-34.vmdk
vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-35/Centos5-35.vmdk
vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-36/Centos5-36.vmdk
vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-37/Centos5-37.vmdk
vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-39/Centos5-39.vmdk
vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-39/Centos5-39.vmdk
vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-40/Centos5-40.vmdk
vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-41/Centos5-41.vmdk
vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-42/Centos5-42.vmdk
vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-43/Centos5-43.vmdk
vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-44/Centos5-44.vmdk
vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-45/Centos5-45.vmdk
vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-46/Centos5-46.vmdk
vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-47/Centos5-47.vmdk
vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-48/Centos5-48.vmdk
vmkfstools -i ./Centos5-30/Centos5-30.vmdk ./Centos5-49/Centos5-49.vmdk

pwd
  /vmfs/volumes/Storage1

# Now you'll copy the inital images configuration file to the directory above the image directories 
# so we can modify it and make it a template file

cp Centos5-30/Centos5-30.vmx ./Centos5-31.vmx

vi Centos5-31.vmx
  change variables to be more generic

cat Centos5-31.vmx 
  #!/usr/bin/vmware
  .encoding = "UTF-8"
  config.version = "8"
  virtualHW.version = "7"
  pciBridge0.present = "TRUE"
  pciBridge4.present = "TRUE"
  pciBridge4.virtualDev = "pcieRootPort"
  pciBridge4.functions = "8"
  pciBridge5.present = "TRUE"
  pciBridge5.virtualDev = "pcieRootPort"
  pciBridge5.functions = "8"
  pciBridge6.present = "TRUE"
  pciBridge6.virtualDev = "pcieRootPort"
  pciBridge6.functions = "8"
  pciBridge7.present = "TRUE"
  pciBridge7.virtualDev = "pcieRootPort"
  pciBridge7.functions = "8"
  vmci0.present = "TRUE"
  nvram = "Centos5-31.nvram"
  deploymentPlatform = "windows"
  virtualHW.productCompatibility = "hosted"
  unity.customColor = "|23C0C0C0"
  tools.upgrade.policy = "useGlobal"
  powerType.powerOff = "soft"
  powerType.powerOn = "default"
  powerType.suspend = "hard"
  powerType.reset = "soft"
  displayName = "Centos5-31"
  extendedConfigFile = "Centos5-31.vmxf"
  floppy0.present = "TRUE"
  scsi0.present = "TRUE"
  scsi0.sharedBus = "none"
  scsi0.virtualDev = "lsilogic"
  memsize = "1024"
  scsi0:0.present = "TRUE"
  scsi0:0.fileName = "Centos5-31.vmdk"
  scsi0:0.deviceType = "scsi-hardDisk"
  ide1:0.present = "TRUE"
  ide1:0.clientDevice = "FALSE"
  ide1:0.deviceType = "cdrom-image"
  ide1:0.startConnected = "TRUE"
  floppy0.startConnected = "FALSE"
  floppy0.clientDevice = "TRUE"
  ethernet0.present = "TRUE"
  ethernet0.networkName = "VM Network"
  ethernet0.addressType = "generated"
  guestOSAltName = "Red Hat Enterprise Linux 5 (32-bit)"
  guestOS = "rhel5"
  uuid.location = "56 4d 41 ed 87 cc a2 03-77 97 48 8c 65 16 7e ed"
  uuid.bios = "56 4d 41 ed 87 cc a2 03-77 97 48 8c 65 16 7e ed"
  vc.uuid = "52 9e a1 5c 02 25 0f c2-20 03 f8 bb 21 93 74 c7"
  ethernet0.generatedAddress = ""
  tools.syncTime = "FALSE"
  cleanShutdown = "TRUE"
  replay.supported = "FALSE"
  sched.swap.derivedName = "/vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-31/Centos5-31-d6823091.vswp"
  scsi0:0.redo = ""
  vmotion.checkpointFBSize = "4194304"
  pciBridge0.pciSlotNumber = "17"
  pciBridge4.pciSlotNumber = "21"
  pciBridge5.pciSlotNumber = "22"
  pciBridge6.pciSlotNumber = "23"
  pciBridge7.pciSlotNumber = "24"
  scsi0.pciSlotNumber = "16"
  ethernet0.pciSlotNumber = "32"
  vmci0.pciSlotNumber = "33"
  ethernet0.generatedAddressOffset = "0"
  vmci0.id = "-876333085"
  hostCPUID.0 = "0000000b756e65476c65746e49656e69"
  hostCPUID.1 = "000106a500100800009ce3bdbfebfbff"
  hostCPUID.80000001 = "00000000000000000000000128100800"
  guestCPUID.0 = "0000000b756e65476c65746e49656e69"
  guestCPUID.1 = "000106a500010800809822010febfbff"
  guestCPUID.80000001 = "00000000000000000000000128100800"
  userCPUID.0 = "0000000b756e65476c65746e49656e69"
  userCPUID.1 = "000106a500100800009822010febfbff"
  userCPUID.80000001 = "00000000000000000000000128100800"
  evcCompatibilityMode = "FALSE"
  ide1:0.fileName = "/vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/iso/CentOS/CentOS-5.4-i386-bin-DVD.iso"
  floppy0.fileName = "/dev/fd0"
  debugStub.linuxOffsets = "0xa356145,0xffffffff,0xfc052084,0xffffffff,0x0,0x0,0xa356150,0x0,0xa35616f,0x0,0xfc052088,0xffffffff,0x0,0x0"
  tools.remindInstall = "TRUE"
  sched.cpu.affinity = "all"
  sched.swap.hostLocal = "disabled"

# Now we'll copy this template to the other image directories

cp Centos5-31.vmx Centos5-31/Centos5-31.vmx
cp Centos5-31.vmx Centos5-32/Centos5-32.vmx
cp Centos5-31.vmx Centos5-33/Centos5-33.vmx
cp Centos5-31.vmx Centos5-34/Centos5-34.vmx
cp Centos5-31.vmx Centos5-35/Centos5-35.vmx
cp Centos5-31.vmx Centos5-36/Centos5-36.vmx
cp Centos5-31.vmx Centos5-37/Centos5-37.vmx
cp Centos5-31.vmx Centos5-38/Centos5-38.vmx
cp Centos5-31.vmx Centos5-39/Centos5-39.vmx
cp Centos5-31.vmx Centos5-40/Centos5-40.vmx
cp Centos5-31.vmx Centos5-41/Centos5-41.vmx
cp Centos5-31.vmx Centos5-42/Centos5-42.vmx
cp Centos5-31.vmx Centos5-43/Centos5-43.vmx
cp Centos5-31.vmx Centos5-44/Centos5-44.vmx
cp Centos5-31.vmx Centos5-45/Centos5-45.vmx
cp Centos5-31.vmx Centos5-46/Centos5-46.vmx
cp Centos5-31.vmx Centos5-47/Centos5-47.vmx
cp Centos5-31.vmx Centos5-48/Centos5-48.vmx
cp Centos5-31.vmx Centos5-49/Centos5-49.vmx

# You'll want to swap the references of the initial image name [ Centos5-30 ] to be the directories image name 

vi Centos5-31/Centos5-31.vmx 
vi Centos5-32/Centos5-32.vmx 
vi Centos5-33/Centos5-33.vmx 
vi Centos5-34/Centos5-34.vmx 
vi Centos5-35/Centos5-35.vmx 
vi Centos5-36/Centos5-36.vmx 
vi Centos5-37/Centos5-37.vmx 
vi Centos5-38/Centos5-38.vmx 
vi Centos5-39/Centos5-39.vmx 
vi Centos5-40/Centos5-40.vmx 
vi Centos5-41/Centos5-41.vmx 
vi Centos5-42/Centos5-42.vmx 
vi Centos5-43/Centos5-43.vmx 
vi Centos5-43/Centos5-43.vmx 
vi Centos5-44/Centos5-44.vmx 
vi Centos5-45/Centos5-45.vmx 
vi Centos5-46/Centos5-46.vmx
vi Centos5-47/Centos5-47.vmx
vi Centos5-48/Centos5-48.vmx
vi Centos5-49/Centos5-49.vmx

# Now we can register those new images and they'll show up in your vSphere Client

vmware-cmd Centos5-31/Centos5-31.vmx register
vmware-cmd Centos5-32/Centos5-32.vmx register
vmware-cmd Centos5-33/Centos5-33.vmx register
vmware-cmd Centos5-34/Centos5-34.vmx register
vmware-cmd Centos5-35/Centos5-35.vmx register
vmware-cmd Centos5-36/Centos5-36.vmx register
vmware-cmd Centos5-37/Centos5-37.vmx register
vmware-cmd Centos5-38/Centos5-38.vmx register
vmware-cmd Centos5-39/Centos5-39.vmx register
vmware-cmd Centos5-40/Centos5-40.vmx register
vmware-cmd Centos5-41/Centos5-41.vmx register
vmware-cmd Centos5-42/Centos5-42.vmx register
vmware-cmd Centos5-43/Centos5-43.vmx register
vmware-cmd Centos5-44/Centos5-44.vmx register
vmware-cmd Centos5-45/Centos5-45.vmx register
vmware-cmd Centos5-46/Centos5-46.vmx register
vmware-cmd Centos5-47/Centos5-47.vmx register
vmware-cmd Centos5-48/Centos5-48.vmx register
vmware-cmd Centos5-49/Centos5-49.vmx register

# Confirm they all are registered

vmware-cmd -l | sort
  /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-30/Centos5-30.vmx
  /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-31/Centos5-31.vmx
  /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-32/Centos5-32.vmx
  /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-33/Centos5-33.vmx
  /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-34/Centos5-34.vmx
  /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-35/Centos5-35.vmx
  /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-36/Centos5-36.vmx
  /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-37/Centos5-37.vmx
  /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-38/Centos5-38.vmx
  /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-39/Centos5-39.vmx
  /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-40/Centos5-40.vmx
  /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-41/Centos5-41.vmx
  /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-42/Centos5-42.vmx
  /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-43/Centos5-43.vmx
  /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-44/Centos5-44.vmx
  /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-45/Centos5-45.vmx
  /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-46/Centos5-46.vmx
  /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-47/Centos5-47.vmx
  /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-48/Centos5-48.vmx
  /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af/Centos5-49/Centos5-49.vmx

# You'll see your images now in your vSphere Client. 
# When you first start them, do a console view of it because you'll see in that view 
# you need to select an option before it will start up.
# Choose :  "I copied it"
# Once image is running, login as root and within a shell confirm your network interface has
# an unique IP address as well as a unique MAC address for the interface
# You then might want to suspend the image at this point.

Manual Cloning of x64 Setup

If you can to x64 with your VMware, here's the additions steps for that [cut-n-paste]

Make a new image for x64 rhel5 called Centos5-x64-50 and once it's all done, shutdown and halt it.

Short and sweet update for ESXi 5.5

[ Working Dir is /vmfs/volumes/datastore1 ]

# ls -la /vmfs/volumes/ | grep data
lrwxr-xr-x    1 root     root            35 Aug 24 21:35 datastore1 -> 53f864a6-1d3302c1-1b97-00101849e4af

# mkdir linuxsrv

# vmkfstools -i Centos\ 6.5\ Base/Centos\ 6.5\ Base.vmdk linuxsrv/linuxsrv.vmdk

# cp Centos\ 6.5\ Base/Centos\ 6.5\ Base.vmx linuxsrv/linuxsrv.vmx

# vi linuxsrv/linuxsrv.vmx
   ESC Key
   :%s/Centos 6.5 Base/linuxsrv/g
   ESC Key
  # make the following variables blank , ethernet0.generatedAddress = "00:0c:29:44:16:8d"
   ethernet0.generatedAddress = ""
   :wq!

# Note - if you have snapshots against your source image, you might need to adjust this var
# scsi0:0.fileName = "850-Wapp2-0000104.vmdk" , for example. The numbers after the hostname get removed.
# 

# vim-cmd solo/registervm /vmfs/volumes/datastore1/linuxsrv/linuxsrv.vmx
10

For ESX 4


pwd
  /vmfs/volumes/4bbaf57f-6230127f-d432-00101849e4af

mkdir Centos5-x64-{51..69}

vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-51/Centos5-x64-51.vmdk
vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-52/Centos5-x64-52.vmdk
vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-53/Centos5-x64-53.vmdk
vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-54/Centos5-x64-54.vmdk
vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-55/Centos5-x64-55.vmdk
vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-56/Centos5-x64-56.vmdk
vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-57/Centos5-x64-57.vmdk
vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-58/Centos5-x64-58.vmdk
vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-59/Centos5-x64-59.vmdk
vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-60/Centos5-x64-60.vmdk
vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-61/Centos5-x64-61.vmdk
vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-62/Centos5-x64-62.vmdk
vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-63/Centos5-x64-63.vmdk
vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-64/Centos5-x64-64.vmdk
vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-65/Centos5-x64-65.vmdk
vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-66/Centos5-x64-66.vmdk
vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-67/Centos5-x64-67.vmdk
vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-68/Centos5-x64-68.vmdk
vmkfstools -i Centos5-x64-50/Centos5-x64-50.vmdk Centos5-x64-69/Centos5-x64-69.vmdk

# Setup the vmx file like a template like you did for the 32bit ones and then copy to the other x64 directories.

cp Centos5-x64-50/Centos5-x64-50.vmx ./
cp Centos5-x64-50/Centos5-x64-50.vmx ./Centos5-x64-60.vmx

# make this like be empty quoted - ethernet0.generatedAddress = ""

vi Centos5-x64-50.vmx

# Edit this for the ethernet and change to 50 to 60 -- make for editing easier for the 60 dir's

vi Centos5-x64-60.vmx

# copy to other directories.

cp Centos5-x64-50.vmx Centos5-x64-51/Centos5-x64-51.vmx
cp Centos5-x64-50.vmx Centos5-x64-52/Centos5-x64-52.vmx
cp Centos5-x64-50.vmx Centos5-x64-53/Centos5-x64-53.vmx
cp Centos5-x64-50.vmx Centos5-x64-54/Centos5-x64-54.vmx
cp Centos5-x64-50.vmx Centos5-x64-55/Centos5-x64-55.vmx
cp Centos5-x64-50.vmx Centos5-x64-56/Centos5-x64-56.vmx
cp Centos5-x64-50.vmx Centos5-x64-57/Centos5-x64-57.vmx
cp Centos5-x64-50.vmx Centos5-x64-58/Centos5-x64-58.vmx
cp Centos5-x64-50.vmx Centos5-x64-59/Centos5-x64-59.vmx

cp Centos5-x64-60.vmx Centos5-x64-60/Centos5-x64-60.vmx
cp Centos5-x64-60.vmx Centos5-x64-61/Centos5-x64-61.vmx
cp Centos5-x64-60.vmx Centos5-x64-62/Centos5-x64-62.vmx
cp Centos5-x64-60.vmx Centos5-x64-63/Centos5-x64-63.vmx
cp Centos5-x64-60.vmx Centos5-x64-64/Centos5-x64-64.vmx
cp Centos5-x64-60.vmx Centos5-x64-65/Centos5-x64-65.vmx
cp Centos5-x64-60.vmx Centos5-x64-66/Centos5-x64-66.vmx
cp Centos5-x64-60.vmx Centos5-x64-67/Centos5-x64-67.vmx
cp Centos5-x64-60.vmx Centos5-x64-68/Centos5-x64-68.vmx
cp Centos5-x64-60.vmx Centos5-x64-69/Centos5-x64-69.vmx

# Now edit them, replacing the 50 in the hostname reference to match the ## of the directory.

vi Centos5-x64-51/Centos5-x64-51.vmx
vi Centos5-x64-52/Centos5-x64-52.vmx
vi Centos5-x64-53/Centos5-x64-53.vmx
vi Centos5-x64-54/Centos5-x64-54.vmx
vi Centos5-x64-55/Centos5-x64-55.vmx
vi Centos5-x64-56/Centos5-x64-56.vmx
vi Centos5-x64-57/Centos5-x64-57.vmx
vi Centos5-x64-58/Centos5-x64-58.vmx
vi Centos5-x64-59/Centos5-x64-59.vmx

# Now edit them, replacing the 60 in the hostname reference to match the ## of the directory.

vi Centos5-x64-61/Centos5-x64-61.vmx
vi Centos5-x64-62/Centos5-x64-62.vmx
vi Centos5-x64-63/Centos5-x64-63.vmx
vi Centos5-x64-64/Centos5-x64-64.vmx
vi Centos5-x64-65/Centos5-x64-65.vmx
vi Centos5-x64-66/Centos5-x64-66.vmx
vi Centos5-x64-67/Centos5-x64-67.vmx
vi Centos5-x64-68/Centos5-x64-68.vmx
vi Centos5-x64-69/Centos5-x64-69.vmx

# Register the x64 images

vmware-cmd Centos5-x64-51/Centos5-x64-51.vmx register
vmware-cmd Centos5-x64-52/Centos5-x64-52.vmx register
vmware-cmd Centos5-x64-53/Centos5-x64-53.vmx register
vmware-cmd Centos5-x64-54/Centos5-x64-54.vmx register
vmware-cmd Centos5-x64-55/Centos5-x64-55.vmx register
vmware-cmd Centos5-x64-56/Centos5-x64-56.vmx register
vmware-cmd Centos5-x64-57/Centos5-x64-57.vmx register
vmware-cmd Centos5-x64-58/Centos5-x64-58.vmx register
vmware-cmd Centos5-x64-59/Centos5-x64-59.vmx register

vmware-cmd Centos5-x64-60/Centos5-x64-60.vmx register
vmware-cmd Centos5-x64-61/Centos5-x64-61.vmx register
vmware-cmd Centos5-x64-62/Centos5-x64-62.vmx register
vmware-cmd Centos5-x64-63/Centos5-x64-63.vmx register
vmware-cmd Centos5-x64-64/Centos5-x64-64.vmx register
vmware-cmd Centos5-x64-65/Centos5-x64-65.vmx register
vmware-cmd Centos5-x64-66/Centos5-x64-66.vmx register
vmware-cmd Centos5-x64-67/Centos5-x64-67.vmx register
vmware-cmd Centos5-x64-68/Centos5-x64-68.vmx register
vmware-cmd Centos5-x64-69/Centos5-x64-69.vmx register

# You'll see your images now in your vSphere Client. 
# When you first start them, do a console view of it because you'll see in that view 
# you need to select an option before it will start up.
# Choose :  "I copied it"
# Once image is running, login as root and within a shell confirm your network interface has
# an unique IP address as well as a unique MAC address for the interface
# You then might want to suspend the image at this point.

Other Useful Steps To Get Zimbra Installed And Running

RHEL or CentOS
chkconfig sendmail off
/etc/init.d/sendmail stop

Some prereq packages that you might not already have installed:

yum install compat-libstdc++-33 compat-libstdc++-296 sysstat
Adding Additional Storage

Adding additional storage for Zimbra installation requirement

  • Add new virtual disk - requires at least 5GB's of free space on available partition
  • Power Off VM And then create a new disk giving it at least 5GB's of space
  • Power On the VM
  • You can then run this to see/confirm the new "disk"
    •  fdisk -l
    • In my example here, my new disk is /dev/sdb
      • Partition the new disk
        •  fdisk /dev/sdb 
        • Select "n" for new partition
        • Select "p" for primary partition
        • Select "1" for partition number
        • Select default of "1" for first cylinder
        • Select default, which should be the highest number given in range. This will change based upon the size of the virtual disk you made.
        • You new disk is now partition.
        • Hit "p" to print out the partition table to confirm.
        • Hit "w" to write table to disk and exit fdisk.
      • Create new filesystem for the new disk/partition
        • This example uses ext3 and the example partition path of /dev/sdb1
        •  mkfs.ext3 /dev/sdb1 
      • Setup /etc/fstab to mount the new partition for zimbra use
        •  mkdir /opt/zimbra 
        •  vi /etc/fstab 
        • And now add a line like the following:
        • /dev/sdb1       /opt/zimbra        ext3   defaults  1 1
Moving Zimbra To New Partitions For zimbra and backups

Adding additional storage for Zimbra installation requirement

  • Disable zimbra from starting via init.d - how this is done depends on your distro.
  • Power Off VM And then create two new virtual disks. Spacing is based upon your current usage and what you expect you'll need going forward. Recommend the disk is allocated now vs allowing it to dynamically grow.
  • Power On the VM
  • Confirm zimbra isn't running, stop it if it is running:
    •  su - zimbra ; zmcontrol stop 
  • You should now confirm the new "disks" exist
    •  fdisk -l
    • In my example here, my new disks are /dev/sdb and /dev/sdc
      • Partition the new sdb disk
        •  fdisk /dev/sdb 
        • Select "n" for new partition
        • Select "p" for primary partition
        • Select "1" for partition number
        • Select default of "1" for first cylinder
        • Select default, which should be the highest number given in range. This will change based upon the size of the virtual disk you made.
        • You new disk is now partition.
        • Hit "p" to print out the partition table to confirm.
        • Hit "w" to write table to disk and exit fdisk.
      • Partition the new sdc disk
        •  fdisk /dev/sdc 
        • Select "n" for new partition
        • Select "p" for primary partition
        • Select "1" for partition number
        • Select default of "1" for first cylinder
        • Select default, which should be the highest number given in range. This will change based upon the size of the virtual disk you made.
        • You new disk is now partition.
        • Hit "p" to print out the partition table to confirm.
        • Hit "w" to write table to disk and exit fdisk.
      • Create new filesystem for the new disks/partitions
        • This example uses ext3 and the example partition path of /dev/sdb1 and /dev/sdc1
        •  mkfs.ext3 /dev/sdb1 
        •  mkfs.ext3 /dev/sdc1 
      • Move zimbra data to a temporary location
        • as root
        •  mv /opt/zimbra /opt/zimbra_old 
      • Setup /etc/fstab to mount the new partition for zimbra use
        •  vi /etc/fstab 
        • And now add a line like the following, they must be below the entry for your / partition:
        • /dev/sdb1       /opt/zimbra        ext3   defaults  1 1
        • /dev/sdc1       /opt/zimbra/backup ext3   defaults  1 1
        • Save the file.
        • We first need to get /opt/zimbra mounted before the backup partition.
        •  mkdir /opt/zimbra 
        •  mount /opt/zimbra 
        • Now we can do the backup one.
        •  mkdir /opt/zimbra/backup 
        •  chown zimbra:zimbra /opt/zimbra/backup 
        •  mount /opt/zimbra/backup 
      • Now we sync the data over onto our new partitions.
        •  rsync -avzHS --progress /opt/zimbra_old/ /opt/zimbra 
        • Once you confirm your rsync is done and is correct, you could remove the old data.
        •  rm -rf /opt/zimbra_old 
      • You should now be able to start zimbra.
        • su - zimbra ; zmcontrol start
      • Remember to re-enable zimbra to start from init.d now also if you disabled it at the beginning.

Final Look

Comment About Hyper-threading And CPU Performance

TheE5520 has 4 cores, which do show up in vSphere as 4 available proc's but the cpuinfo just shows the one. Difference comes when Intel VT [vmx flag in cpuinfo] or AMD-V [svm flag in cpuinfo] is enabled while hyperthreading [ht flag in cpuinfo] is turned off. Turning on HT in the bios I think would show 8 proc in vm and report 4 cores in the cpuinfo output from base OS [don't want to reboot and reconfigure at the moment to double check]. Note, cpuinfo will show HT if the cpu's support it - not just if it's enabled. You still need to enable it and confirm APCI is enabled in the BIOS if your not seeing the 'virtual cpu's'. VT is about seeing the physical cores and individual cpu's. HT is about virtualizing each cpu or core into 2 virtual cpu's.

Spec

ESX Server

[root@vmware-server ~]# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU E5520 @ 2.27GHz
stepping : 5
cpu MHz : 2266.688
cache size : 8192 KB
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi
 mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl 
 vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips : 4535.95
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: [8]

[root@vmware-server ~]# cat /proc/meminfo
MemTotal: 356684 kB
MemFree: 30392 kB
Buffers: 9068 kB
Cached: 127804 kB
SwapCached: 14180 kB
Active: 264444 kB
Inactive: 41320 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 356684 kB
LowFree: 30392 kB
SwapTotal: 730916 kB
SwapFree: 671960 kB
Dirty: 208 kB
Writeback: 0 kB
AnonPages: 168356 kB
Mapped: 36188 kB
Slab: 12448 kB
PageTables: 3160 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 909256 kB
Committed_AS: 533636 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 25036 kB
VmallocChunk: 34359705099 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB
MachineMem: 12580415 kB

[root@vmware-server ~]# uname -a
Linux vmware-server.zimbra.homunix.com 2.6.18-128.ESX #1 
 Thu Oct 15 16:11:16 PDT 2009 x86_64 x86_64 x86_64 GNU/Linux

[root@vmware-server ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.1 (Tikanga)

[root@vmware-server ~]# fdisk -l

Disk /dev/sda: 750.1 GB, 750156374016 bytes
255 heads, 63 sectors/track, 91201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 140 1124518+ 83 Linux
/dev/sda2 141 154 112455 fc VMware VMKCORE
/dev/sda3 155 91201 731335027+ 5 Extended
/dev/sda5 155 91201 731334996 fb VMware VMFS

Disk /dev/sdb: 8095 MB, 8095006720 bytes
255 heads, 63 sectors/track, 984 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 91 730926 82 Linux swap / Solaris
/dev/sdb2 92 346 2048287+ 83 Linux
/dev/sdb3 347 984 5124735 5 Extended
/dev/sdb5 347 984 5124703+ 83 Linux


I've setup the vm's to have 1 cpu and 1024MB RAM. I have 9 running right now. Here's one of my 64bit image details:

[root@mail59 ~]# cat /proc/meminfo
MemTotal: 1026932 kB
MemFree: 85816 kB
Buffers: 14884 kB
Cached: 87132 kB
SwapCached: 117444 kB
Active: 705916 kB
Inactive: 143664 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 1026932 kB
LowFree: 85816 kB
SwapTotal: 2064376 kB
SwapFree: 1453736 kB
Dirty: 644 kB
Writeback: 0 kB
AnonPages: 734740 kB
Mapped: 34284 kB
Slab: 38884 kB
PageTables: 33304 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 2577840 kB
Committed_AS: 3257840 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 263932 kB
VmallocChunk: 34359473927 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB
[root@mail59 ~]# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU E5520 @ 2.27GHz
stepping : 5
cpu MHz : 2266.631
cache size : 8192 KB
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 
 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc up ida 
 nonstop_tsc pni cx16 popcnt lahf_lm
bogomips : 4533.26
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: [8]

[root@mail59 ~]# uname -a
Linux mail59.zimbra.homeunix.com 2.6.18-164.el5 #1 SMP Thu 
 Sep 3 03:28:30 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

[root@mail59 ~]# cat /etc/redhat-release
CentOS release 5.4 (Final)

[root@mail59 ~]# fdisk -l

Disk /dev/sda: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 1305 10377990 8e Linux LVM

Disk /dev/sdb: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 1044 8385898+ 83 Linux

[root@mail59 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
7.7G 3.5G 3.9G 48% /
/dev/sda1 99M 13M 82M 14% /boot
tmpfs 502M 0 502M 0% /dev/shm
/dev/hdc 6.8G 6.8G 0 100% /media/zcs-x64-603-06
/dev/sdb1 7.9G 1.8G 5.8G 24% /opt/zimbra

Here's my esxtop output, which is ran on the base os of your ESX server. You'll see the running vm's I mentioned.

[root@vmware-server ~]# esxtop

 9:20:07pm up 32 days 20:43, 139 worlds; CPU load average: 0.14, 0.15, 0.16
PCPU USED(%):   7.7   7.3   6.8   6.8 AVG:   7.1
PCPU UTIL(%):  10.6  10.4  10.0   9.9 AVG:  10.2
CCPU(%):   0 us,   2 sy,  98 id,   0 wa ;       cs/sec:    229

     ID    GID NAME             NWLD   %USED    %RUN    %SYS   %WAIT    %RDY   %IDLE  %OVRLP   %CSTP  %MLMTD  %SWPWT
      1      1 idle                4  367.57  370.04    0.00    0.00   34.02    0.00    0.00    0.00    0.00    0.00
      2      2 system              6    0.01    0.01    0.00  600.00    0.00    0.00    0.00    0.00    0.00    0.00
      6      6 helper             58    0.01    0.01    0.00 5800.00    0.00    0.00    0.00    0.00    0.00    0.00
      7      7 drivers             9    0.00    0.00    0.00  900.00    0.00    0.00    0.00    0.00    0.00    0.00
      8      8 vmotion             4    0.00    0.00    0.00  400.00    0.00    0.00    0.00    0.00    0.00    0.00
     10     10 console             2    1.93    1.89    0.01  200.00    0.08   99.04    0.03    0.00    0.00    0.00
     15     15 vmkapimod           9    0.00    0.00    0.00  900.00    0.00    0.00    0.00    0.00    0.00    0.00
     17     17 FT                  1    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00    0.00    0.00
     18     18 vobd.4231           8    0.00    0.00    0.00  800.00    0.00    0.00    0.00    0.00    0.00    0.00
     19     19 net-cdp.4239        1    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00    0.00    0.00
     20     20 vmware-vmkauthd     1    0.00    0.00    0.00  100.00    0.00    0.00    0.00    0.00    0.00    0.00
    114    114 Centos5-x64-59      4    4.43    5.17    0.01  398.60    0.29   95.33    0.29    0.00    0.00    0.00
    125    125 Centos5-x64-50      4    2.64    3.49    0.00  400.00    0.30   97.30    0.27    0.00    0.00    0.00
    126    126 Centos5-x64-51      4    2.38    3.13    0.00  400.00    0.31   97.63    0.28    0.00    0.00    0.00
    127    127 Centos5-x64-52      4    2.70    3.54    0.00  400.00    0.31   97.21    0.28    0.00    0.00    0.00
    128    128 Centos5-x64-53      4    2.65    3.50    0.00  400.00    0.32   97.25    0.28    0.00    0.00    0.00
    139    139 Centos5-x64-54      4    2.36    3.12    0.00  400.00    0.29   97.63    0.27    0.00    0.00    0.00
    140    140 Centos5-x64-55      4    2.36    3.13    0.00  400.00    0.28   97.65    0.29    0.00    0.00    0.00
    141    141 Centos5-x64-56      4    2.37    3.11    0.00  400.00    0.29   97.66    0.27    0.00    0.00    0.00
    142    142 Centos5-x64-57      4    2.44    3.21    0.00  400.00    0.29   97.57    0.29    0.00    0.00    0.00
Screenshot

Esx-screenshot.jpg

Optional Setups To Enhance Your Test Environment

Using RINETD To Redirect To Your Various VM For External HTTP Access

RINETD Setup

rinetd redirects TCP connections from one IP address and port to another. rinetd is a single-process server which handles any number of connections to the address/port pairs specified in the file /etc/rinetd.conf.


Here's an example /etc/rinetd.conf on an internal server [192.168.0.16] that will redirect to the various vm's to port 80. The 'firewall', a cheap one like I have, only allows me to do very basic port redirection to an ip address -- NOT an internal ip address AND a different port. So my cheap firewall is setup to redirect port 80## to the rinetd server - 192.168.0.16 and then that server redirects to the vm and the right port - 80.


If your using a DynDNS service this works out very well. As I can now use the DynDNS domain name and simply add a port number to get redirected to the various ZCS servers. For example:

  • http://EXTERNAL-DOMAIN:8059 would get redirected to my vm server that is using 192.168.0.59 and to port 80 for the http ZWC login page.

Here's the /etc/rinetd.conf I have setup:

192.168.0.16 8030 192.168.0.30 80                               
192.168.0.16 8031 192.168.0.31 80                               
192.168.0.16 8032 192.168.0.32 80                               
192.168.0.16 8033 192.168.0.33 80                               
192.168.0.16 8034 192.168.0.34 80
192.168.0.16 8035 192.168.0.35 80
192.168.0.16 8036 192.168.0.36 80
192.168.0.16 8037 192.168.0.37 80
192.168.0.16 8038 192.168.0.38 80
192.168.0.16 8039 192.168.0.39 80
192.168.0.16 8040 192.168.0.40 80
192.168.0.16 8041 192.168.0.41 80
192.168.0.16 8042 192.168.0.42 80
192.168.0.16 8043 192.168.0.43 80
192.168.0.16 8044 192.168.0.44 80
192.168.0.16 8045 192.168.0.45 80
192.168.0.16 8046 192.168.0.46 80
192.168.0.16 8047 192.168.0.47 80
192.168.0.16 8048 192.168.0.48 80
192.168.0.16 8049 192.168.0.49 80
192.168.0.16 8050 192.168.0.50 80
192.168.0.16 8051 192.168.0.51 80
192.168.0.16 8052 192.168.0.52 80
192.168.0.16 8053 192.168.0.53 80
192.168.0.16 8054 192.168.0.54 80
192.168.0.16 8055 192.168.0.55 80
192.168.0.16 8056 192.168.0.56 80
192.168.0.16 8057 192.168.0.57 80
192.168.0.16 8058 192.168.0.58 80
192.168.0.16 8059 192.168.0.59 80
192.168.0.16 8060 192.168.0.60 80
192.168.0.16 8061 192.168.0.61 80
192.168.0.16 8062 192.168.0.62 80
192.168.0.16 8063 192.168.0.63 80
192.168.0.16 8064 192.168.0.64 80
192.168.0.16 8065 192.168.0.65 80
192.168.0.16 8066 192.168.0.66 80
192.168.0.16 8067 192.168.0.67 80
192.168.0.16 8068 192.168.0.68 80
192.168.0.16 8069 192.168.0.69 80
logcommon
logfile /var/log/rinetd.log
Setting Up A Mailhub

Continuing with the example of using 192.168.0.16 with a hostname of mail3.[SUB].[DOMAIN].com as my mail hub which the firewall routes all smtp traffic to and also is running rinetd. This assumes you've setup DNS like I outlined in Ajcody-Virtualization-Named-DNS which I mentioned at the top of this wiki page.


Install postfix [centos5 example]

yum install postfix
rpm -q postfix
 postfix-2.3.3-2.1.el5_2

Configure /etc/postfix/main.cf

/etc/init.d/postfix stop
cd /etc/postfix/
mv main.cf main.cf-backup

Now paste in the contents:

## paste the below into file and adjust for your setup the following variables
## myhostname & mydomain with my example of mail3.[SUB].[DOMAIN].com
## might need to also adjust : mynetworks = 192.168.0.0/24
## You might want to compare our default main.cf below and adjust for different paths, etc
## that your distro or postfix version might be setup to use
## and then save -- :wq!

vi main.cf

queue_directory = /var/spool/postfix
command_directory = /usr/sbin
daemon_directory = /usr/libexec/postfix
mail_owner = postfix

myhostname = mail3.[SUB].[DOMAIN].com
mydomain = mail3.[SUB].[DOMAIN].com
myorigin = $mydomain
mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain
mynetworks = 192.168.0.0/24, 127.0.0.0/8
relay_domains = $mydestination, $mynetworks, hash:/etc/postfix/relay-domains

transport_maps = hash:/etc/postfix/transport
local_transport = local

alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases

debug_peer_level = 2
debugger_command =
         PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin
         xxgdb $daemon_directory/$process_name $process_id & sleep 5

sendmail_path = /usr/sbin/sendmail.postfix
newaliases_path = /usr/bin/newaliases.postfix
mailq_path = /usr/bin/mailq.postfix
setgid_group = postdrop
html_directory = no
manpage_directory = /usr/share/man
sample_directory = /usr/share/doc/postfix-2.3.3/samples
readme_directory = /usr/share/doc/postfix-2.3.3/README_FILES

You might need to make the aliases db. Path is referenced in the main.cf above. This command below will end up creating /etc/aliases.db

postmap /etc/aliases

Now we'll setup the relay-domains file. Adjust my example for your domain using substitution in vi [ ESC key and then :%s/SUB.DOMAIN/With Your Domain Details/g hit return key].

vi /etc/postfix/relay-domains

mail30.SUB.DOMAIN.com OK                     
mail31.SUB.DOMAIN.com OK
mail32.SUB.DOMAIN.com OK
mail33.SUB.DOMAIN.com OK
mail34.SUB.DOMAIN.com OK
mail35.SUB.DOMAIN.com OK
mail36.SUB.DOMAIN.com OK
mail37.SUB.DOMAIN.com OK
mail38.SUB.DOMAIN.com OK
mail39.SUB.DOMAIN.com OK
mail40.SUB.DOMAIN.com OK
mail41.SUB.DOMAIN.com OK
mail42.SUB.DOMAIN.com OK
mail43.SUB.DOMAIN.com OK
mail44.SUB.DOMAIN.com OK
mail45.SUB.DOMAIN.com OK
mail46.SUB.DOMAIN.com OK
mail47.SUB.DOMAIN.com OK
mail48.SUB.DOMAIN.com OK
mail49.SUB.DOMAIN.com OK
mail50.SUB.DOMAIN.com OK
mail51.SUB.DOMAIN.com OK
mail52.SUB.DOMAIN.com OK
mail53.SUB.DOMAIN.com OK
mail54.SUB.DOMAIN.com OK
mail55.SUB.DOMAIN.com OK
mail56.SUB.DOMAIN.com OK
mail57.SUB.DOMAIN.com OK
mail58.SUB.DOMAIN.com OK
mail59.SUB.DOMAIN.com OK
mail61.SUB.DOMAIN.com OK
mail62.SUB.DOMAIN.com OK
mail63.SUB.DOMAIN.com OK
mail64.SUB.DOMAIN.com OK
mail65.SUB.DOMAIN.com OK
mail66.SUB.DOMAIN.com OK
mail67.SUB.DOMAIN.com OK
mail68.SUB.DOMAIN.com OK
mail69.SUB.DOMAIN.com OK

Now build the db, the command will end up creating /etc/postfix/relay-domains.db

postmap /etc/postfix/relay-domains

Now we'll create a transport map for each domain. Adjust my example for your domain using substitution in vi [ ESC key and then :%s/SUB.DOMAIN/With Your Domain Details/g hit return key].

vi /etc/postfix/transport

mail30.SUB.DOMAIN.com      smtp:mail30.SUB.DOMAIN.com
mail31.SUB.DOMAIN.com      smtp:mail31.SUB.DOMAIN.com
mail32.SUB.DOMAIN.com      smtp:mail32.SUB.DOMAIN.com
mail33.SUB.DOMAIN.com      smtp:mail33.SUB.DOMAIN.com
mail34.SUB.DOMAIN.com      smtp:mail34.SUB.DOMAIN.com
mail35.SUB.DOMAIN.com      smtp:mail35.SUB.DOMAIN.com
mail36.SUB.DOMAIN.com      smtp:mail36.SUB.DOMAIN.com
mail37.SUB.DOMAIN.com      smtp:mail37.SUB.DOMAIN.com
mail38.SUB.DOMAIN.com      smtp:mail38.SUB.DOMAIN.com
mail39.SUB.DOMAIN.com      smtp:mail39.SUB.DOMAIN.com
mail40.SUB.DOMAIN.com      smtp:mail40.SUB.DOMAIN.com
mail41.SUB.DOMAIN.com      smtp:mail41.SUB.DOMAIN.com
mail42.SUB.DOMAIN.com      smtp:mail42.SUB.DOMAIN.com
mail43.SUB.DOMAIN.com      smtp:mail43.SUB.DOMAIN.com
mail44.SUB.DOMAIN.com      smtp:mail44.SUB.DOMAIN.com
mail45.SUB.DOMAIN.com      smtp:mail45.SUB.DOMAIN.com
mail46.SUB.DOMAIN.com      smtp:mail46.SUB.DOMAIN.com
mail47.SUB.DOMAIN.com      smtp:mail47.SUB.DOMAIN.com
mail48.SUB.DOMAIN.com      smtp:mail48.SUB.DOMAIN.com
mail49.SUB.DOMAIN.com      smtp:mail49.SUB.DOMAIN.com
mail50.SUB.DOMAIN.com      smtp:mail50.SUB.DOMAIN.com
mail51.SUB.DOMAIN.com      smtp:mail51.SUB.DOMAIN.com
mail52.SUB.DOMAIN.com      smtp:mail52.SUB.DOMAIN.com
mail53.SUB.DOMAIN.com      smtp:mail53.SUB.DOMAIN.com
mail54.SUB.DOMAIN.com      smtp:mail54.SUB.DOMAIN.com
mail55.SUB.DOMAIN.com      smtp:mail55.SUB.DOMAIN.com
mail56.SUB.DOMAIN.com      smtp:mail56.SUB.DOMAIN.com
mail57.SUB.DOMAIN.com      smtp:mail57.SUB.DOMAIN.com
mail58.SUB.DOMAIN.com      smtp:mail58.SUB.DOMAIN.com
mail59.SUB.DOMAIN.com      smtp:mail59.SUB.DOMAIN.com
mail60.SUB.DOMAIN.com      smtp:mail60.SUB.DOMAIN.com
mail61.SUB.DOMAIN.com      smtp:mail61.SUB.DOMAIN.com
mail62.SUB.DOMAIN.com      smtp:mail62.SUB.DOMAIN.com
mail63.SUB.DOMAIN.com      smtp:mail63.SUB.DOMAIN.com
mail64.SUB.DOMAIN.com      smtp:mail64.SUB.DOMAIN.com
mail65.SUB.DOMAIN.com      smtp:mail65.SUB.DOMAIN.com
mail66.SUB.DOMAIN.com      smtp:mail66.SUB.DOMAIN.com
mail67.SUB.DOMAIN.com      smtp:mail67.SUB.DOMAIN.com
mail68.SUB.DOMAIN.com      smtp:mail68.SUB.DOMAIN.com
mail69.SUB.DOMAIN.com      smtp:mail69.SUB.DOMAIN.com

Now build the db, the command will end up creating /etc/postfix/transport.db

postmap /etc/postfix/transport

We should be all set now. Start postfix and monitor the log file as you do your testing.

/etc/init.d/postfix start
tail -f /var/log/maillog
Configuring Your ZCS VM's To Use The Mailhub

Note, another 'trick' you might need to do on your vm's to get them working in this type of situation from the outside is to setup one server [I used my 192.168.0.16 machine] to redirect and send emails from the other zcs servers. This is assuming your using a DynDNS setup where you can wildcard for a base domain and also wildcard it for MX requests as well. This is also necessary when your firewall device can only route all port 25 requests to a single ip address internally to your server. This setup assumes that server is running a mta that can then do MX lookups internally for the domainname being used for routing information. The adjustments below setup the ZCS vm's to accept the mail relay from that internal server.


Adjust in the Configuration > Global Settings > MTA tab and under Configuration > Servers > Your server > MTA tab on the other ZCS servers to use the one server for:

  • Web mail MTA Port 25
  • Relay MTA for external delivery
  • Inbound SMTP Host name
  • zmcontrol stop and then zmcontrol start on the server/s




Ajcody Notes Archive Discovery

   KB 2494        Last updated on 2008-11-16  




0.00
(0 votes)
24px ‎  - This is Zeta Alliance Certified Documentation. The content has been tested by the Community.


Actual Archive & Discovery Notes Homepage

Ajcody-Notes-Archive-Discovery

References for A&D

Updated A&D Documentation In Admin Manual

Please see:

Zimbra ZSC Version 8.x

Admin Guide

The version 7.x manual is more comprehensive. Version 8 Documentation is missing a lot of content available in Version 7 documentation.

Zimbra ZSC Version 7.x

Admin Guide

Older Notes About ZAD

Important Note:

I think we need to clean up the documentation that refers to this "download" as no longer being necessary as it's bundled in the zcs-network edition tar ball now. I found an old email to the development team where I inquired about this - I didn't get a response from on it at the time.

If this assumption is true [Like I see with the 5.x setups I've helped with] then I think we need to adjust the documents for 4.5.x stuff to state:

You'll need to download your version of 4.5.x zcs network edition tarball, untar it, and then run the install script . Which effectively will allow you to now select the "archive" package which will also include the search zimlet [zmbxsearch].


The two main A&D references are :

http://wiki.zimbra.com/index.php?title=ZAD

http://www.zimbra.com/pdf/Zimbra%20Archiving%20and%20Discovery%20Release%20Notes.pdf

But I Just Want Some BCC's To Happen - Not All This A&D Stuff

Please see Ajcody-MTA-Postfix-Topics#Automatic_BCC instead then.

But We Have A Non-Zimbra MTA - zimbraSmtpHostname & zimbraMtaRelayHost Pointed To External Device

Please see this RFE I made:

  • "RFE: Ability to handle A&D when another MTA systems is being specified"

Notable Bugs Or RFE's

Update Jan 22, 2015 . I filed these while testing against ZCS 8.6 .


Older Bug/RFE's I've noted:

Archive Stop Working After Upgrade Or Disabling Anti-Spam - Anti-Virus

Prior to JP/8.5 , the archiving feature required the anti-spam or anti-virus service to be enabled. If you disabled both of them, then amavis as a whole was disabled. This is required for archiving to work. The following bug resolves this for JP/8.5 - it makes amavis its own service:

Also note, my testing on ZCS 8.0.7 did confirm that disabling av/as [therefore, amavis] will cause archiving to not work. This is very confusing since zmcontrol status AND the admin console will show that the service for "archiving" is running - though archiving is not actually working since amavis is disabled.

Update Jan 22, 2015. New bug filed to deal with other upgrade issues we discovered.

First - Short & Sweet How-To On CLI With Single ZCS 8.6 Server

In this example, I'll turn off the antivirus and antispam service in 8.6 and then enable amavis and archiving. Create an archive cos and then archive user. This was a single ZCS 8.6 server with all services installed during the installation - just using default values. This was NOT an upgrade to ZCS 8.6 .

[zimbra@ldap2 ~]$ zmcontrol -v 
Release 8.6.0_GA_1153.RHEL6_64_20141215151258 RHEL6_64 NETWORK edition. 

[zimbra@ldap2 ~]$ zmprov gs `zmhostname` | grep Service | egrep 'amavis|antivirus|antispam|archiving' 
zimbraServiceEnabled: amavis 
zimbraServiceEnabled: antivirus 
zimbraServiceEnabled: antispam 
zimbraServiceEnabled: archiving 
zimbraServiceInstalled: amavis 
zimbraServiceInstalled: antivirus 
zimbraServiceInstalled: antispam 
zimbraServiceInstalled: archiving 

[zimbra@ldap2 ~]$ zmprov ms `zmhostname` +zimbraServiceInstalled archiving +zimbraServiceEnabled archiving 

[zimbra@ldap2 ~]$ zmprov ms `zmhostname` -zimbraServiceEnabled antispam -zimbraServiceEnabled antivirus 

[zimbra@ldap2 ~]$ zmprov gs `zmhostname` | grep Service | egrep 'amavis|antivirus|antispam|archiving' 
zimbraServiceEnabled: amavis 
zimbraServiceEnabled: archiving 
zimbraServiceInstalled: amavis 
zimbraServiceInstalled: antivirus 
zimbraServiceInstalled: antispam 
zimbraServiceInstalled: archiving 


!! Note - zmmtactl restart didn't cause the amavisd.conf file to change, hence why I then went with zmcontrol restart. 

[zimbra@ldap2 ~]$ zmcontrol restart 

[zimbra@ldap2 ~]$ grep archive_quar /opt/zimbra/conf/amavisd.conf 
archive_quarantine_method => undef, # Don't run archiving a second time 
$archive_quarantine_method = 'smtp:[127.0.0.1]:10025'; 
$archive_quarantine_to = undef; 


[zimbra@ldap2 ~]$ zmprov cc archive 
71f9d7f4-54cc-4bf3-a6e1-94a1fc38a129 

[zimbra@ldap2 ~]$ zmarchiveconfig enable user1@`zmhostname` archive-cos archive 

[zimbra@ldap2 ~]$ vi /tmp/email.txt 

[zimbra@ldap2 attrs]$ cat /tmp/email.txt
To: user1@ldap2.zimbra.DOMAIN.com
Subject: Test For Archive - CLI
From: admin@ldap2.zimbra.DOMAIN.com
test

[zimbra@ldap2 ~]$ /opt/zimbra/postfix/sbin/sendmail -t < /tmp/email.txt 

[zimbra@ldap2 ~]$ grep archive /var/log/zimbra.log 

Jan 15 16:09:43 ldap2 amavis[29283]: (29283-01) h9nw0WdCl1Tw(h9nw0WdCl1Tw) SEND from <> -> <user1-20150115@ldap2.zimbra.DOMAIN.com.archive>,
ENVID=AM.h9nw0WdCl1Tw.20150115T210943Z@ldap2.zimbra.DOMAIN.com BODY=7BIT 250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued 
as 2DE506445C 
Jan 15 16:09:43 ldap2 amavis[29283]: (29283-01) Passed CLEAN {RelayedInbound,Archived}, <zimbra@ldap2.zimbra.DOMAIN.com> -> 
<user1@ldap2.zimbra.DOMAIN.com>, quarantine: user1-20150115@ldap2.zimbra.DOMAIN.com.archive, Message-ID: 
<20150115210943.087106438E@ldap2.zimbra.DOMAIN.com>, mail_id: h9nw0WdCl1Tw, Hits: -, size: 351, queued_as: 3049C64466, 138 ms 
Jan 15 16:09:45 ldap2 postfix/lmtp[3478]: 2DE506445C: to=<user1-20150115@ldap2.zimbra.DOMAIN.com.archive>, 
relay=ldap2.zimbra.homeunix.com[192.168.1.172]:7025, delay=2.2, delays=0/0.02/0.17/2, dsn=2.1.5, status=sent (250 2.1.5 Delivery OK)

When Was A ZCS Service Enabled Or Disabled

Note - single ZCS server deployment example. Have to double check on multi-server deployment if the command needs to be tweaked.

Check Current Status Of Services Enabled-Disable and Installed-NotInstalled

Example is done on the host in question:

zmprov -l gs `zmhostname` | egrep 'zimbraServiceEnabled|zimbraServiceInstall'

The -l is used in case mailboxd is the service in question, -l makes the call to ldap to check the configuration of the server.

Check Backups And What LDAP Has Stored

Do the following, adjust the backup path if your not using the default path:

 for i in `find /opt/zimbra/backup/sessions/ -name ldap.bak.gz -print`;  \
 do echo $i; zgrep zimbraServiceEnabled $i; done

Will show you something like :

zimbraServiceEnabled: antivirus
zimbraServiceEnabled: antispam
zimbraServiceEnabled: opendkim
zimbraServiceEnabled: logger
zimbraServiceEnabled: mailbox
zimbraServiceEnabled: mta
zimbraServiceEnabled: convertd
zimbraServiceEnabled: stats
zimbraServiceEnabled: snmp
zimbraServiceEnabled: ldap
zimbraServiceEnabled: spell
zimbraServiceEnabled: archiving
Check The History Of The Service Status In zimbra-stats.log

In /var/log/ you'll see zimbra-stats.log and archives of that file, for example - zimbra-stats.log-20140714 . Here's an example of a zgrep for the archive service:

[zimbra@zcs807 log]$ zgrep archiv zimbra-stats* | tail
zimbra-stats.log:Jul 22 08:24:08 zcs807 zimbramon[27645]: 27645:info: 2014-07-22 08:24:01, 
 STATUS: zcs807.us.zimbralab.com: archiving: Running
zimbra-stats.log:Jul 22 08:26:09 zcs807 zimbramon[28689]: 28689:info: 2014-07-22 08:26:01, 
 STATUS: zcs807.us.zimbralab.com: archiving: Running
zimbra-stats.log-20140714:Jul 15 14:40:19 zcs807 zimbramon[21519]: 21519:info: 2014-07-15 14:40:01, 
 STATUS: zcs807.us.zimbralab.com: archiving: Stopped
zimbra-stats.log-20140714:Jul 15 14:42:06 zcs807 zimbramon[24998]: 24998:info: 2014-07-15 14:42:01, 
 STATUS: zcs807.us.zimbralab.com: archiving: Stopped
zimbra-stats.log-20140714:Jul 15 14:44:09 zcs807 zimbramon[29250]: 29250:info: 2014-07-15 14:44:01, 
 STATUS: zcs807.us.zimbralab.com: archiving: Running
zimbra-stats.log-20140714:Jul 15 14:46:08 zcs807 zimbramon[32188]: 32188:info: 2014-07-15 14:46:01, 
 STATUS: zcs807.us.zimbralab.com: archiving: Running
zimbra-stats.log-20140714:Jul 15 14:48:26 zcs807 zimbramon[3174]: 3174:info: 2014-07-15 14:48:07, 
 STATUS: zcs807.us.zimbralab.com: archiving: Running
zimbra-stats.log-20140714:Jul 15 14:50:15 zcs807 zimbramon[6466]: 6466:info: 2014-07-15 14:50:01, 
 STATUS: zcs807.us.zimbralab.com: archiving: Running
zimbra-stats.log-20140714:Jul 15 14:52:10 zcs807 zimbramon[10260]: 10260:info: 2014-07-15 14:52:02, 
 STATUS: zcs807.us.zimbralab.com: archiving: Running
zimbra-stats.log-20140714:Jul 15 14:54:10 zcs807 zimbramon[19004]: 19004:info: 2014-07-15 14:54:01, 
 STATUS: zcs807.us.zimbralab.com: archiving: Running
Check The Logger/RRD Data And Service History Uptime

Let's pull the stats from the logger database going back to Jan 1st of this year. Here's an example on how to do that and what to expect. Please send me a copy of the output also. Notice in the example below the lines that have ",," for column 9 - which is for archiving. This is how I can tell when the service was first enabled and running. Ref for timestamp: http://www.epochconverter.com/

Note: The columns are identified by:

timestamp,opendkim,zmconfigd,mailbox,proxy,spell,logger,antivirus,archiving,snmp,ldap,memcached,
  convertd,stats,antispam,mta

Please adjust the commands below to suit your needs. I use `zmhostname` below in the command rather than typing out the localhosts hostname. Also, see https://www.gnu.org/software/coreutils/manual/html_node/Examples-of-date.html for examples on the --date string.


For this example, I'm looking to see when the "archiving" service was running 100%. Archiving is the 9th column - '$9 == 1' and the 1 means it was available 100% for the time period. I searched a year's worth of data by using this part below in the examples : "-s `date +%s --date="12 month ago"` -e `date +%s --date="1 minute ago"`"

 /opt/zimbra/libexec/zmrrdfetch -f zmstatuslog -h `zmhostname` -s `date +%s \
 --date="12 month ago"` -e `date +%s --date="1 minute ago"` | awk -F, '$9 == 1'

timestamp,opendkim,zmconfigd,mailbox,proxy,spell,logger,antivirus,archiving,snmp,ldap,memcached,
  convertd,stats,antispam,mta
 [cut]
1405454400,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405458000,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405461600,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405465200,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405468800,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405472400,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405476000,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405479600,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
 [cut]

For this example, I'm looking to see when the "archiving" service was NOT enabled or running at all. Archiving is the 9th column - '$9 == ""' and the absence of a number value means it was UNAVAILABLE 100% for the time period.

 /opt/zimbra/libexec/zmrrdfetch -f zmstatuslog -h `zmhostname` -s `date +%s \
 --date="12 month ago"` -e `date +%s --date="1 minute ago"` | awk -F, '$9 == ""' | head

timestamp,opendkim,zmconfigd,mailbox,proxy,spell,logger,antivirus,archiving,snmp,ldap,memcached,
  convertd,stats,antispam,mta
 [cut]
1388538000,,,,,,,,,,,,,,,
1388541600,,,,,,,,,,,,,,,
1388545200,,,,,,,,,,,,,,,
1388548800,,,,,,,,,,,,,,,
1388552400,,,,,,,,,,,,,,,
1388556000,,,,,,,,,,,,,,,
1388559600,,,,,,,,,,,,,,,
1388563200,,,,,,,,,,,,,,,
1388566800,,,,,,,,,,,,,,,
1388570400,,,,,,,,,,,,,,,
 [cut]

For this example, I'm looking to see when the "zmconfigd" service was NOT 100% available during the time period BUT was greater than 0% of the time. Zmconfigd is the 3rd column - '$3 ~ 0' and by using ~ [NOT] 0 , I can see when it was greater than 0 [100% UNAVAILABLE] but not 1 [100% available].

 /opt/zimbra/libexec/zmrrdfetch -f zmstatuslog -h `zmhostname` -s `date +%s \
 --date="12 month ago"` -e `date +%s --date="1 minute ago"` | awk -F, '$3 ~ 0'

timestamp,opendkim,zmconfigd,mailbox,proxy,spell,logger,antivirus,archiving,snmp,ldap,memcached,
  convertd,stats,antispam,mta
1400670000,0.926591142901235,0.882009858611111,0.962549540740741,0.962548086419753,0.926603197839506,
  1,,,0.964051042283951,1,0.962549647839506,0.962515237345679,0.292491091358025,,0.926590777777778
1400756400,1,0.968276889722222,1,1,1,1,,,1,1,1,1,1,,1
1401678000,0.9323037375,0.937399301388889,1,0.964109132777778,0.932308961111111,1,,,0.932305973888889,
  1,0.964109800277778,1,0.932309575,,0.932302677222222
1401692400,0.999979394166667,0.9999799825,1,0.9999993925,0.999979395277778,1,,,0.999979393333333,1,
  0.9999993975,1,0.999979395555556,,0.999979393333333
1401966000,1,0.964610447777778,1,1,1,1,,,1,1,1,1,1,,1
1402052400,0.999603703008394,0.999722196864111,0.999615163422937,0.999602498779275,0.999597924831049,
 0.999619443248889,,,0.99959995491499,0.999626231525528,0.99961180945684,0.999632334525742,
 0.999591523017581,,0.999607517027221
1402765200,1,0.966623691358025,1,1,0.966623606481481,1,1,,0.966623565432099,1,1,1,0.966623647530864,
 1,0.966623437345679
1402768800,0.999471892261905,0.999587652037617,0.999491736309524,0.999464918452381,0.999464358928571,
 0.999495648214286,0.999508635119048,,0.999464445238095,0.999496001785714,0.999483839285714,
 0.999503813095238,0.999461091071429,0.99951724702381,0.999478128571429
1402776000,0.9999910775,0.999972398888889,0.999993851388889,0.999991085555555,0.999972394722222,
 0.999991074166667,0.999991234166667,,0.999972394722222,0.999995651111111,0.999991075833333,
 0.999991235277778,0.999972394722222,0.999991233333333,0.999972392777778
1403175600,0.999841834656085,0.999841587301587,0.999841991402116,0.999841788690476,0.999841692791005,
 0.999842046957672,0.999842196097884,,0.999841743386243,0.999842097222222,0.999841937830688,
 0.99984214484127,0.999841638227513,0.999842260251323,0.999841883267196
1405076400,1,0.964354991388889,1,1,1,1,1,,1,1,1,1,1,1,1
1405080000,1,0.946143796944445,1,1,1,1,1,,1,1,1,1,1,1,1
1405105200,1,0.987032061666667,1,1,1,1,1,,1,1,1,1,1,1,1
1405116000,1,0.982143184166667,1,1,1,1,1,,1,1,1,1,1,1,1
1405130400,1,0.967078515555556,1,1,1,1,1,,1,1,1,1,1,1,1
1405141200,1,0.920520648333333,1,1,1,1,1,,1,1,1,1,1,1,1
1405144800,1,0.9539186725,1,1,1,1,1,,1,1,1,1,1,1,1
1405148400,1,0.981826185277778,1,1,1,1,1,,1,1,1,1,1,1,1
1405162800,1,0.996330343333333,1,1,1,1,1,,1,1,1,1,1,1,1

An example of a zmcontrol restart

 /opt/zimbra/libexec/zmrrdfetch -f zmstatuslog -h `zmhostname` -s `date +%s \
 --date="15 minute ago"` -e `date +%s --date="1 minute ago"`

timestamp,opendkim,zmconfigd,mailbox,proxy,spell,logger,antivirus,archiving,snmp,ldap,memcached,
  convertd,stats,antispam,mta
1405621920,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405621950,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405621980,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405622010,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405622040,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405622070,0.105026416666667,0.1050323,1,1,0.105029991666667,1,0.105016,0.105017466666667,0.105028733333333,
  1,1,1,0.10503115,0.105014483333333,0.105025008333333
1405622100,0.105026416666667,0.1050323,1,1,0.105029991666667,1,0.105016,0.105017466666667,0.105028733333333,
  1,1,1,0.10503115,0.105014483333333,0.105025008333333
1405622130,0.105026416666667,0.1050323,1,1,0.105029991666667,1,0.105016,0.105017466666667,0.105028733333333,
  1,1,1,0.10503115,0.105014483333333,0.105025008333333
1405622160,0.105026416666667,0.1050323,1,1,0.105029991666667,1,0.105016,0.105017466666667,0.105028733333333,
  1,1,1,0.10503115,0.105014483333333,0.105025008333333
1405622190,,0,,,,,,,,,,,,,
1405622220,,0,,,,,,,,,,,,,
1405622250,,0,,,,,,,,,,,,,
1405622280,,0,,,,,,,,,,,,,
1405622310,,0,,,,,,,,,,,,,
1405622340,,0.893621444444444,,,,,,,,,,,,,
1405622370,,0.893621444444444,,,,,,,,,,,,,
1405622400,,0.893621444444444,,,,,,,,,,,,,
1405622430,,1,,,,,,,,,,,,,
1405622460,,1,,,,,,,,,,,,,
1405622490,,1,,,,,,,,,,,,,
1405622520,,1,,,,,,,,,,,,,
1405622550,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
1405622580,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
Check The audit.log File For Service Enable Changes

There is the /opt/zimbra/log/audit.log file that will note a change in a service - for example [ use: zgrep zimbraService /opt/zimbra/log/audit* ]:

audit.log:2014-07-21 12:15:44,745 INFO 
 [qtp1097575009-2632:https://127.0.0.1:7071/service/admin/soap/ModifyServerRequest]
 [name=zimbra;ip=127.0.0.1;ua=zmprov/8.0.7_GA_6029;] security - cmd=ModifyServer; 
  name=zcs807.us.DOMAIN.com; -zimbraServiceEnabled=archiving;
audit.log:2014-07-21 12:16:51,609 INFO 
 [qtp1097575009-2640:https://127.0.0.1:7071/service/admin/soap/ModifyServerRequest]
 [name=zimbra;ip=127.0.0.1;ua=zmprov/8.0.7_GA_6029;] security - cmd=ModifyServer; 
  name=zcs807.us.DOMAIN.com; +zimbraServiceEnabled=archiving;
Check The zmsetup logs For Changes During Installation/Re-installs

In regards to the /opt/zimbra/log/zmsetup.* log files, here's an example from a test machine :

[Someone rerunning the installer/zmsetup, archiving was enabled - greping' for archiv below]

zmsetup.07152014-150104.txt:Tue Jul 15 14:51:43 2014 enabled packages zimbra-logger zimbra-store 
  zimbra-mta zimbra-convertd zimbra-core zimbra-apache zimbra-archiving zimbra-proxy zimbra-snmp 
  zimbra-ldap zimbra-spell
zmsetup.07152014-150104.txt:Tue Jul 15 14:52:02 2014 archiving is enabled
zmsetup.07152014-150104.txt:Tue Jul 15 14:52:29 2014 checking isEnabled zimbra-archiving
zmsetup.07152014-150104.txt:Tue Jul 15 14:52:29 2014 zimbra-archiving is enabled
zmsetup.07152014-150104.txt:Tue Jul 15 14:54:43 2014 Updating cached config attribute for Server 
  zcs807.us.DOMAIN.com: zimbraServiceInstalled=archiving
zmsetup.07152014-150104.txt:Tue Jul 15 14:54:43 2014 *** Running as zimbra user: /opt/zimbra/bin/zmprov 
  -r -m -l ms zcs807.us.DOMAIN.com zimbraServiceInstalled 'antivirus' zimbraServiceInstalled 'antispam'
  zimbraServiceInstalled 'opendkim' zimbraServiceInstalled 'archiving' zimbraServiceInstalled 'logger' 
  zimbraServiceInstalled 'mailbox' zimbraServiceInstalled 'mta' zimbraServiceInstalled 'convertd' 
  zimbraServiceInstalled 'stats' zimbraServiceInstalled 'proxy' zimbraServiceInstalled 'snmp' 
  zimbraServiceInstalled 'ldap' zimbraServiceInstalled 'spell'
zmsetup.07152014-150104.txt:Tue Jul 15 14:54:46 2014 Updating cached config attribute for Server 
  zcs807.us.DOMAIN.com: zimbraServiceEnabled=archiving
zmsetup.07152014-150104.txt:Tue Jul 15 14:54:46 2014 *** Running as zimbra user: 
  /opt/zimbra/bin/zmprov -r -m -l ms zcs807.us.DOMAIN.com zimbraServiceEnabled 'antivirus' 
  zimbraServiceEnabled 'archiving' zimbraServiceEnabled 'antispam' zimbraServiceEnabled 'opendkim' 
  zimbraServiceEnabled 'logger' zimbraServiceEnabled 'mailbox' zimbraServiceEnabled 'mta' 
  zimbraServiceEnabled 'convertd' zimbraServiceEnabled 'stats' zimbraServiceEnabled 'proxy' 
  zimbraServiceEnabled 'snmp' zimbraServiceEnabled 'ldap' zimbraServiceEnabled 'spell'
zmsetup.07152014-150104.txt:Tue Jul 15 14:55:30 2014 *** Running as zimbra user: /opt/zimbra/bin/zmprov 
  -r -m -l mcf +zimbraComponentAvailable archiving
zmsetup.07152014-150104.txt: Stopping archiving...Done.
zmsetup.07152014-150104.txt: Starting archiving...Done.
zmsetup.07152014-150104.txt:Tue Jul 15 15:00:06 2014 com_zimbra_archive...
zmsetup.07152014-150104.txt:Tue Jul 15 15:00:06 2014 *** Running as zimbra user: 
  /opt/zimbra/bin/zmzimletctl -l deploy zimlets-network/com_zimbra_archive.zip
zmsetup.07152014-150104.txt:[] INFO: Deploying Zimlet com_zimbra_archive in LDAP.
zmsetup.07152014-150104.txt:[] INFO: Installing Zimlet com_zimbra_archive on this host.
zmsetup.07152014-150104.txt:[] INFO: Upgrading Zimlet com_zimbra_archive to 7.1.0
zmsetup.07152014-150104.txt:[] INFO: Enabling Zimlet com_zimbra_archive

Showing the ldap backup data in regards to the archiving service being disabled and then enabled after the 07/15 setup :

$ zgrep "zimbraServiceEnabled:" full-20140716.070025.112/ldap/* incr-20140714.070016.382/ldap/*

  :from the day after the re-install:
full-20140716.070025.112/ldap/ldap.bak.gz:zimbraServiceEnabled: opendkim
full-20140716.070025.112/ldap/ldap.bak.gz:zimbraServiceEnabled: archiving <<
full-20140716.070025.112/ldap/ldap.bak.gz:zimbraServiceEnabled: logger
full-20140716.070025.112/ldap/ldap.bak.gz:zimbraServiceEnabled: mailbox
full-20140716.070025.112/ldap/ldap.bak.gz:zimbraServiceEnabled: mta
full-20140716.070025.112/ldap/ldap.bak.gz:zimbraServiceEnabled: convertd
full-20140716.070025.112/ldap/ldap.bak.gz:zimbraServiceEnabled: stats
full-20140716.070025.112/ldap/ldap.bak.gz:zimbraServiceEnabled: proxy
full-20140716.070025.112/ldap/ldap.bak.gz:zimbraServiceEnabled: snmp
full-20140716.070025.112/ldap/ldap.bak.gz:zimbraServiceEnabled: ldap
full-20140716.070025.112/ldap/ldap.bak.gz:zimbraServiceEnabled: spell

  :from the day before the re-install:
incr-20140714.070016.382/ldap/ldap.bak.gz:zimbraServiceEnabled: logger
incr-20140714.070016.382/ldap/ldap.bak.gz:zimbraServiceEnabled: mailbox
incr-20140714.070016.382/ldap/ldap.bak.gz:zimbraServiceEnabled: mta
incr-20140714.070016.382/ldap/ldap.bak.gz:zimbraServiceEnabled: convertd
incr-20140714.070016.382/ldap/ldap.bak.gz:zimbraServiceEnabled: stats
incr-20140714.070016.382/ldap/ldap.bak.gz:zimbraServiceEnabled: snmp
incr-20140714.070016.382/ldap/ldap.bak.gz:zimbraServiceEnabled: ldap
incr-20140714.070016.382/ldap/ldap.bak.gz:zimbraServiceEnabled: spell
incr-20140714.070016.382/ldap/ldap.bak.gz:zimbraServiceEnabled: proxy
incr-20140714.070016.382/ldap/ldap.bak.gz:zimbraServiceEnabled: opendkim
Check The Zimbra Users BASH History File and .zmprov_history

You could check your .bash_history file to see if the command is still referenced there - if it goes back long enough for your purposes. I think the default lines the history file will remember is 500.

[zimbra@zcs807 ~]$ zmprov ms `zmhostname` -zimbraServiceEnabled archiving 
[zimbra@zcs807 ~]$ zmcontrol status | grep -i arch 
[zimbra@zcs807 ~]$ zmprov gs `zmhostname` | grep Enabled | grep archiving 
[zimbra@zcs807 ~]$ zmprov ms `zmhostname` +zimbraServiceEnabled archiving 
[zimbra@zcs807 ~]$ zmcontrol status | grep -i arch 
archiving Running 
[zimbra@zcs807 ~]$ zmprov gs `zmhostname` | grep Enabled | grep archiving 
zimbraServiceEnabled: archiving 

[zimbra@zcs807 ~]$ grep archiving .bash_history 
zmprov ms `zmhostname` +zimbraServiceInstalled archiving +zimbraServiceEnabled archiving 
zmprov ms `zmhostname` -zimbraServiceEnabled archiving 
zmprov gs `zmhostname` | grep Enabled | grep archiving 
zmprov ms `zmhostname` +zimbraServiceEnabled archiving 
zmprov gs `zmhostname` | grep Enabled | grep archiving 
grep archiving .bash_history 

Note, there is also a /opt/zimbra/.zmprov_history you can also review

Getting Duplicate Emails In The Archive Account

Please see the following:

Archiving Issues When Using 3rd Party MTA's

Please see the following:

Pre-Deployment Type Questions

How To Add OLD Data Or Retroactively Add Data To A&D Account

You'll want to have A&D all setup and then you can use imapsync to 'sync' the old data from the one account into the A&D account.

Recommendations:

  • Confirm imap is enabled on the mailstores
  • Setup a sub-folder in A&D account to hold data that existed prior to the A&D account being in use.
  • Test the various imapsync flags you'll want to use first before running against all your accounts.

See also:

Can we set the archive settings for only sent mail?

Question: Can we set the archive settings for only sent mail?

Answer: No, archiving is done for any message that passes through the MTA for an account that has archiving enabled. You can setup separate MTA's for in/out bound delivery and enable archiving for only the inbound servers to achieve this goal.

Can we set the archive settings for only form port 25?

Question: Can we set the archive settings for only form port 25?

Answer: No, there are no per port level configurations. You can configure which accounts have archiving enabled and you can enable/disable the service on a per server level.

If Zimbra implemented an archive feature, is something in the following MTA files changes?

Question: If Zimbra implemented an archive feature, is something in the following MTA files changed? [ zmmta.cf , main.cf , main.cf.default , master.cf.in , master.cf ]

Answer: Nothing is specifically changed in the files you listed but settings in zmmta.cf determine how the postfix content filter is configured when archiving is enabled. Archiving functionality is handled by the amavis process on the Zimbra MTA.

Using One Archive Account For Multiple Accounts

You could also include other variable's here if they are needed, ex. - archive-cos COS_NAME

zmarchiveconfig enable user1@domainname archive-address user-archives@domainname.archive
zmarchiveconfig enable user2@domainname archive-address user-archives@domainname.archive archive-create FALSE
zmarchiveconfig enable user3@domainname archive-address user-archives@domainname.archive archive-create FALSE

You'll see the mapping in the primary email account settings that state what archive account is being used:

zmprov ga user1@example.com zimbraArchiveAccount
 zimbraArchiveAccount: user-archives@domainname.archive
zmprov ga user2@example.com zimbraArchiveAccount
 zimbraArchiveAccount: user-archives@domainname.archive
zmprov ga user3@example.com zimbraArchiveAccount
 zimbraArchiveAccount: user-archives@domainname.archive


External Host For Archive Accounts Setup

I installed 5.0.11 on a test box, selecting the archiving package during the installation.

The main domain was "zimbra.INTERNAL.com" and the servername is mail3.zimbra.INTERNAL.com .

'Note, I did have an external MTA relay server setup for zimbra. I was doing this test from home.

Once it was done, I then created a test account - ajcody@zimbra.INTERNAL.com

I then ran this from the CLI [on a multi-server environment, this is ran on the mta server]:

zmprov ms mail3.zimbra.INTERNAL.com +zimbraServiceInstalled archiving  +zimbraServiceEnabled archiving

I confirm that I can send emails to the external account that I will be using with the zimbra admin account:

  • Logged into zimbra admin web console , accounts > admin > View Mail
    • Composed new message for ajcody@EXTERNAL-DOMAIN.com

I confirmed the external account received the email and did a reply back. Again, confirming now that the admin account gets the email as well from the external account.

Created the archive account to be used:

zmarchiveconfig enable ajcody@zimbra.INTERNAL.com archive-address ajcody@EXTERNAL-DOMAIN.com archive-create false

I then stopped/started zimbra.

zmcontrol stop
zmcontrol start

Confirmed that amavis (antispam & antivirus) and archive show up as running process:

[zimbra@mail3 ~]$ zmcontrol status
Host mail3.zimbra.INTERNAL.com
	antispam                Running
	antivirus               Running
	archiving               Running
	ldap                    Running
	logger                  Running
	mailbox                 Running
	mta                     Running
	snmp                    Running
	spell                   Running
	stats                   Running

Checked my zimbra account to confirm the archive variables:

[zimbra@mail3 ~]$ zmprov ga ajcody@`hostname -d` | egrep -i archive  
amavisArchiveQuarantineTo: ajcody@EXTERNAL-DOMAIN.com
zimbraArchiveAccount: ajcody@EXTERNAL-DOMAIN.com
zimbraArchiveAccountDateTemplate: yyyyMMdd
zimbraArchiveAccountNameTemplate: ${USER}-${DATE}@${DOMAIN}.archive

Tests I then performed:

  • I sent an email from the zimbra admin account to the ajcody@zimbra.INTERNAL.com account. Confirmed that the external account [ajcody@EXTERNAL-DOMAIN.com] received a copy.
  • I sent an email to ajcody@zimbra.INTERNAL.com from my ajcody@BUSINESS.com account and confirmed a copy went to ajcody@EXTERNAL-DOMAIN.com .
  • I sent an email from ajcody@zimbra.INTERNAL.com to the zimbra admin account, again confirming a copy went to ajcody@EXTERNAL-DOMAIN.com .

What didn't work as expected:

  • Messages to/from the two accounts don't create copies. If ajcody@zimbra.INTERNAL.com sends to ajcody@EXTERNAL-DOMAIN.com a "copy" will not be generated and vis-versa .
  • The subject lines weren't altered as stated in the official A&D document.
    • p3 , "When a message is received to a mailbox with archiving enabled, a copy of the message is sent to the archive mailbox with the text “to be archived” added to the subject line."

I've sent off a request about these two items to the developers to get their feedback on them.

Multi-Server & New Mailstore A&D Setup

I also created a RFE for documentation on this.

http://bugzilla.zimbra.com/show_bug.cgi?id=25135

The following is a very rough draft document I made for multi-server / new mailstore A&D setups.

Ajcody Multi-Server & New Mailstore A&D Setup Homepage

   KB 2494        Last updated on 2008-11-16  




0.00
(0 votes)
24px ‎  - This is Zeta Alliance Certified Documentation. The content has been tested by the Community.


Actual Multi-Server & New Mailstore A&D Setup Homepage

Please see Ajcody-Notes-Archive-Discovery-Mailstore-Setup

Issues That Have Caused Confusion

What Gets Installed Where?
RFE To Clear Up The Confusion
zimbra-archive package/rpm - Mailstores

zimbra-archive (the package/rpm you see from the installer) should be installed on all mailstores which you want to use for cross mailbox search. This also sets the zimbraComponentAvailable archiving config attribute which allows the mta(s) to turn on archiving. zimbra-archive is not installed directly on the mta, it's just enabled.

Note, you install zimbra-archive on a mailbox server but the service runs on the mta node.

MTA's - Require Configuration

If you add zimbra-archiving to an existing install you need to :

  • Install zimbra-archiving on one or more of your mailbox servers
  • Then set zimbraServiceInstalled archiving and zimbraServiceEnabled archiving on all the mta servers
  • Restart the mta services

For example:

zmprov ms mta.example.com +zimbraServiceInstalled archiving +zimbraServiceEnabled archiving

On the mta server:

zmmtactl restart

To confirm the /opt/zimbra/conf/amavisd.conf was modified correctly, you should see on the mta:

#$archive_quarantine_method = 'smtp:[127.0.0.1]:10025'; 

Was uncommented out:

$archive_quarantine_method = 'smtp:[127.0.0.1]:10025';

You'll be able to then notice in the /var/log/zimbra.log file if the redirect to the A&D account is happening [once A&D accounts are setup that is]. Example uses example.com.archive as the archive domain I setup for the A&D accounts :

grep "example.com.archive" /var/log/zimbra.log
 Dec 11 13:38:52 mta-server amavis[1978]: (01978-19) SEND via SMTP: <> -> 
    <user-20081211@example.com.archive>,ENVID=AM.8ISxcrQG8uAj.20081211T193852Z@mailstore.example.com 
    BODY=7BIT 250 2.6.0 Ok, id=01978-19, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 
    5ADF8F120C4
 Dec 11 13:38:52 mta-server postfix/lmtp[21864]: 5ADF8F120C4: 
    to=<user-20081211@example.com.archive>, relay=archive.example.com[X.X.X.93]:7025, 
    delay=0.07, delays=0/0/0/0.06, dsn=2.1.5, status=sent (250 2.1.5 OK)
Enabling Amavis And Archiving With 8.5+ While Antispam And AntiVirus Are Disabled

With 808 and 8.5 , archiving should be able to run without as/av being enabled.

For example, disabling antispam and antivirus but enabling amavis [required] and archiving on a mta server [note - this server had the full mta package already installed on it and had antivirus, antispam, [amavis], and postfix running on it] :

[zimbra@850-mta1 ~]$ zmcontrol status | egrep 'amavis|antispam|antivirus|archiving'
	amavis                  Running
	antispam                Running
	antivirus               Running

## NOTE , the below command will disable antispam & anitvirus for this "test". ##

[zimbra@850-mta1 ~]$ zmprov ms `zmhostname` +zimbraServiceEnabled archiving \
 +zimbraServiceEnabled amavis -zimbraServiceEnabled antispam -zimbraServiceEnabled antivirus

[zimbra@850-mta1 ~]$ zmcontrol restart                
Host 850-ldap1.zimbra.homeunix.com
	Stopping vmware-ha...Done.
[cut]
	Stopping ldap...Done.
Host 850-ldap1.zimbra.homeunix.com
	Starting ldap...Done.
	Starting zmconfigd...Done.
	Starting dnscache...Done.
	Starting logger...Done.
	Starting convertd...Done.
	Starting mailbox...Done.
	Starting memcached...Done.
	Starting proxy...Done.
	Starting amavis...Done.
	Starting opendkim...Done.
	Starting archiving...Done.
	Starting snmp...Done.
	Starting spell...Done.
	Starting mta...Done.
	Starting stats...Done.
	Starting service webapp...Done.
	Starting zimbra webapp...Done.
	Starting zimbraAdmin webapp...Done.
	Starting zimlet webapp...Done.
[zimbra@850-ldap1 ~]$ zmcontrol status | egrep 'amavis|antispam|antivirus|archiving'
	amavis                  Running
	archiving               Running
zimbra_xmbxsearch zimlet

For 5.x installs, this zimbra_xmbsearch zimlet will get configured on each mailstore that you install the zimbra-archive package on. The documentation in various places might cause confusion on this matter, because for the 4.x releases it was a separate step.

You should find the zimlet already located at /opt/zimbra/zimlets-network/zimbra_xmbxsearch.zip

After the installation, you should see when you go to the admin web console on the mailstore you install the zimbra-archive package on that the cross-mailbox search zimlet is there. It shows up in two locations:

  • Left Pane : Configuration > Admin Extensions > zimbra_xmbxsearch
  • Left Pane : Tools > Search Mail

If you wanted this zimlet to also be available on a server that didn't have the zimbra-archiving packaged installed you could then deploy it on that server.

cd /opt/zimbra/zimlets-network/
zmzimletctl deploy zimbra_xmbxsearch.zip
## ls the directory and confirm the full name - you might need this:
zmzimletctl deploy com_zimbra_xmbxsearch.zip

The How-To

Reference Documents

http://www.zimbra.com/docs/ne/latest/multi_server_install/multi-server_install.5.1.html

http://www.zimbra.com/docs/ne/latest/administration_guide/Archiving.16.1.html

Assumptions

This install how-to assumes you have an existing LDAP/Mailstore/MTA server(s) for your normal production environment, the Zimbra license and logger are installed on the primary ZCS server(s), and that you are NOT running the proxy module.

Example archive mailstore hostname is : archive.example.com

Example primary ZCS hostname is : mail.example.com

Preliminary Items

Things to do or check before install:

  • DNS entry for new mailstore and primary ZCS server(s) can resolve to it.
  • DNS configured properly on mailstore server.
  • Master Root LDAP Server mail.example.com
  • Master Root LDAP Password
    • On LDAP server do : su – zimbra ; zmlocalconfig –s | grep ldap_root_password
  • Master LDAP port – default is 389
  • SMTP Server
Installation Of New Mailstore That Will Have A&D
Install Modules
  • Type y to install the zimbra-store, zimbra-archiving and zimbra-spell (optional) packages.
    • Do Not Install MTA! These Instructions Do Not Take That Into Account.
    • When zimbra-spell is installed the zimbra-apache package is also installed.
  • Installing: zimbra-core zimbra-store zimbra-apache zimbra-spell
Modify Configuration

Press Enter to modify the system. The selected packages are installed on the server.

At this point the Main menu displays the default entries for the Zimbra component you are installing.

To expand the menu to see the configuration values type x and press Enter.

To navigate the Main menu, select the menu item to change. You can modify any of the defaults.

  • Common Configuration
    • LDAP
      • Ldap master host: [set this to the FQDN of your LDAP server]
      • Ldap port: 389 [set this if your LDAP server isn’t using default]
      • Ldap Admin password: [this is your LDAP servers Root LDAP password]
        • On LDAP server do : su – zimbra ; zmlocalconfig –s ldap_root_password
      • TimeZone: [set this]
  • For zimbra-store
    • Set the Admin Password
      • +License filename: UNSET [if you see this, then something is wrong with your
    • LDAP configuration. It should of pulled the license info from the LDAP server.
    • Set the SMTP host

Type r to return to the Main menu, if you aren’t there already.

When the mailbox server is configured, return to the Main menu and type a to apply the configuration changes.

Press Enter to save the configuration data.

When Save Configuration data to a file appears, press Enter.

The next request is where to save the files. To accept the default, press Enter.

To save the files to another directory, enter the directory and then press Enter.

When “The system will be modified - continue?” appears type y and press Enter.

The server is modified.

Installing all the components and configuring the server can take a few minutes.

When Installation complete - press return to exit displays, press Enter.

The installation of the mailbox server is complete.

After Install

Confirm server status

su – zimbra ; zmcontrol status

Populate the ssh keys, on each server in your environment

su - zimbra ;  zmupdateauthkeys 

The key is updated on /opt/zimbra/.ssh/authorized_keys.

Upgrading A Zimbra Server For An Archive & Discovery Mailstore
Adding Package For A&D

This will retain your current settings for the system. Your server will experience downtime during the upgrade.

untar zcs*.tar that matches your existing system

 cd zcs-version-directory
 ./install
  choose upgrade
  select zimbra-archiving

The upgrade of the mailbox server is complete.

After Upgrade

Confirm server status

su – zimbra ; zmcontrol status

Note, zimbra-archiving only runs as a service on a MTA server.

Populate the ssh keys, on each server in your environment

su-zimbra ;  zmupdateauthkeys 

The key is updated on /opt/zimbra/.ssh/authorized_keys.

Configure Zimbra For Use Of The New Mailstore and A&D

Example A&D mailstore hostname is : archive.example.com

  • Go to your primary admin console url. [https://[example.com]:7071/zimbraAdmin]
  1. Confirm you see the new mailstore under Configuration > Servers
    1. Under Configuration > Servers > [MTA servername(s)] > Services
      1. [each MTA server needs this]
      2. You’ll see a box for Archiving and Discovery
        1. Check this to enable the MTA server(s) for Archiving and Discovery. If this is grayed out, run the command below (modified for your server) on your one of your mailstores.
          • This effectively does:
          • zmprov ms mta.example.com +zimbraServiceInstalled archiving  +zimbraServiceEnabled archiving
            • Remember, zmprov uses the variable below. A mta only server can't be set for localhost, change it to point to a mailstore.
            •  [root@mta ~]# zmlocalconfig | grep zmprov
            •     zimbra_zmprov_default_soap_server = localhost
            •  [root@mta ~]# zmlocalconfig -e zimbra_zmprov_default_soap_server=mailstore.example.com
Configuring Your COS's - Normal COS's and Archiving COS

It's recommended that the archive accounts be created on a dedicated mailstore. You can limit what mailstores are used for new account creation by restricting what mailstores are used under "Server Pool" within a COS. Your normal COS's should excluded your archive mailstores and your archive COS should only have archive mailstores selected.

You'll also want to make sure your archive COS isn't using any 'features' that aren't necessary for archive accounts. No point in consuming certain license features when they aren't needed.

  1. Go to Configuration > Class of Service > default [or your primary domain] > Server Pool
    1. You’ll want to make sure it’s limited to the correct server pools
      1. Your new mailstore for A&D should be unchecked.
    2. Click on New for a new Class of Server (COS)
      1. Call it archive or something similar
        1. Under Server Pool > Limit > have only the new mailstore checked
    3. Make sure your not using licensed features [EWS for example] that aren't necessary for your archive accounts.
Setup Initial A&D With First Account - Creation Of The Archive Domain
Revisit To COS - Naming Scheme Of Archive Accounts

When archive accounts are created they use the zimbraArchiveAccountNameTemplate variable from the COS. The default is:

$ zmprov gc default | grep -i archive
  zimbraArchiveAccountDateTemplate: yyyyMMdd
  zimbraArchiveAccountNameTemplate: ${USER}-${DATE}@${DOMAIN}.archive

I, personally, don't like the use of the $DATE variable in this. I change my ARCHIVE COS to use the normal username but retain the .archive for the domain.

zmprov mc archive zimbraArchiveAccountNameTemplate '${USER}@${DOMAIN}.archive'

Bug to be aware of:

The Creation

On server with zmarchiveconfig (most likely mailstore you installed A&D on) and as zimbra (su – zimbra) do the following to setup your first A&D account.

format : zmarchiveconfig –s servername enable user@example.com archive-cos <cos>

example :

zmarchiveconfig –s archive.example.com enable account@example.com archive-cos archive

NOTE

If the above command doesn't seem to create the archive account/domain. Drop the use of [ -s servername ]. Basically, just run this on the A&D mailstore:
zmarchiveconfig enable account@example.com archive-cos archive

The above command will create the mail domain for the archive accounts using the template defaults, user@example.com to make example.com.archive

On your main ldap server or where ever you usually access the zimbra admin web console, login to the admin web console.

  1. Confirm the archive domain was setup.
    1. Configuration > Domains > [domainname].archive > General
    2. Confirm or adjust the archive domain to use the right COS
      1. Configuration > Domains > [domainname].archive > General Information
        1. Change “Default Class of Service” to your COS [archive], if needed for your configuration.
  2. Now check for the new archive account you made
    1. Address > Accounts
    2. Click on account and hit the edit button
    3. In the top summary section you'll be able to confirm the COS and Mail Server being used for the account.
      1. NOTE, if it's showing the account is on the primary mailstore and NOT the A&D mailstore.
        1. Remove the A&D account
          •  zmprov ra [user]@[domainname].archive
        2. Add the account back again using the zmarchiveconfig command
          • zmarchiveconfig enable account@example.com archive-cos archive
        3. Now confirm, as above, that the account is using the A&D mailstore.
          • This might be a bug related to the archive domain being created for the first time.

Send the primary account a test email and then shortly afterwards do a "View Mail" within the admin console for the archive account. You should see the archive message in the archive account.

Error: unknown document: EnableArchiveRequest

If you get this error when trying to create the archive account "Error: unknown document: EnableArchiveRequest" you most likely needed to install a new license for A&D and have not restart the mailboxd services . Updating the license is not enough, you'll need to restart ZCS on the mailstores also.

See the following bug:

RFE's On Archive Accounts
Testing Of Archive Mail Flow

Send the primary account a test email and then shortly afterwards do a "View Mail" within the admin console for the archive account. You should see the archive message in the archive account.

You should confirm mail flow copies occur with the following:

  1. Inbound
    1. External Account (email) to the primary zimbra account setup for archive.
    2. A zimbra account that ISN'T the archive account in question to the primary account setup for archive.
  2. Outbound
    1. With primary account setup for archive, send an email to an external email address.
    2. With primary account setup for archive, send an email to another internal zimbra email address.
Archive Account Isn't Getting Email

Let's double check everything was done correctly up above.

Assumption on syntax of account creation:

zmarchiveconfig enable user@example.com archive-cos archive

Let's check what actually was done:

zmprov ga user@example.com | grep -i archive
 amavisArchiveQuarantineTo: user-20081211@example.com.archive
 zimbraArchiveAccount: user-20081211@example.com.archive
 zimbraArchiveAccountNameTemplate: ${USER}-${DATE}@${DOMAIN}.archive

It should reference an account that's like, if you are using the archive templates:

user-[date]@example.com.archive

that account should exist and reference lmtp, rather than smtp, for the transport:

zmprov ga user-20081211@example.com.archive | grep -i trans
  zimbraMailTransport: lmtp:archive.example.com:7025
Checking Logs For Archive Operations

On the mta-server, you should find a reference to the archive account in /var/log/zimbra.log

grep archive /var/log/zimbra.log
 Dec 11 13:38:52 mta-server amavis[1978]: (01978-19) SEND via SMTP: <> -> 
    <user-20081211@example.com.archive>,ENVID=AM.8ISxcrQG8uAj.20081211T193852Z@mailstore.example.com 
    BODY=7BIT 250 2.6.0 Ok, id=01978-19, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 
    5ADF8F120C4
 Dec 11 13:38:52 mta-server postfix/lmtp[21864]: 5ADF8F120C4: 
    to=<user-20081211@example.com.archive>, relay=archive.example.com[X.X.X.93]:7025, 
    delay=0.07, delays=0/0/0/0.06, dsn=2.1.5, status=sent (250 2.1.5 OK)

On the archive-server, you should find reference to the delivery in /opt/zimbra/log/mailbox.log

grep archive /opt/zimbra/log/mailbox.log
 2008-12-11 14:45:32,923 INFO  [LmtpServer-9] 
  [name=user-20081211@example.com.archive;mid=7;] mailop - Adding Message: id=257,
  Message-ID=<1350363939.41021229024728317.JavaMail.root@EXTERNAL-MTA.DOMAIN.com>, parentId=-1,
  folderId=2, folderName=Inbox.
Mass Accounts Configuration
Update 5/28/15, I filed the following RFE:
  • "Redesign archive account creation process [autocreate, mass creation/enabling, etc]"

https://bugzilla.zimbra.com/show_bug.cgi?id=99710

Update, our 6.0 release will have a zmarchiveconfig -f command for batch processing from a file input.

CHECK YOUR AVAILABLE LICENSES BEFORE YOU PROCEED!!

One could put all the accounts in a txt file and then use a for-loop to process the account@example.com variable.

zmprov -l gaa > /tmp/accounts.txt

Remove any accounts you've already done and those not necessary for archiving (ex. admin, ham, spam, etc.)

You can give gaa other options, look at zmprov help account. For example, you could also narrow this down to a dump of accounts in a domain:

zmprov -l gaa [DOMAIN] > /tmp/accounts.txt

Note, the below uses the above setup for A&D - You'll need to modify for your environment.

for i in `cat /tmp/accounts.txt`
do
zmarchiveconfig –s archive.example.com enable $i archive-cos archive
sleep 3
done

You can be tailing /opt/zimbra/log/mailbox.log on the archive server to watch the progress.

Searches After Configuration Is Done

Please see Ajcody-Server-Misc-Topics#Cross_Mailbox_Searches_and_Tracing

Searches Limited To 500 or 1000 Maximum Results

See Ajcody-Server-Misc-Topics#Searches_Limited_To_500_or_1000_Maximum_Results


Special Circumstance Case Scenarios

Need Primary Account Going To Two "Archive" Targets

Currently it isn't possible, I have submitted an RFE for this though:

I also submitted an RFE to clarify the difference/intention between amavisArchiveQuarantineTo and zimbraArchiveAccount

Using Legal Intercept For Secondary Target

Note - since we are setting the legal intercept against the ARCHIVE account and not the production account, we avoid what might be an issue with legal intercept's limitations. See :

Other references:

Primary Archive Account Setup
[zimbra@zcs807 ~]$ zmarchiveconfig create archive-test@zcs807.us.DOMAIN.com archive-address archive-test-manual@zcs807.us.DOMAIN.com.archive

Setup For The Legal Intercept One We'll Use - You could also use the 'normal' account creation process, but those defaults also will show up in the GAL.
[zimbra@zcs807 ~]$ zmarchiveconfig create archive-test2@zcs807.us.DOMAIN.com archive-address archive-test-manual@zcs807.us.DOMAIN.com.archive

[zimbra@zcs807 ~]$ zmprov ga archive-test@zcs807.us.DOMAIN.com | egrep -i 'archive|amavisArchiveQuarantine'      
 [cut - notes] 
confirm that amavisArchiveQuarantineTo & zimbraArchiveAccount are using only ONE of the 'archive' accounts, the primary archive account. Not the one we'll setup later to be the legal intercept target.

[zimbra@zcs807 ~]$ zmprov ma archive-test@zcs807.us.DOMAIN.com amavisArchiveQuarantineTo archive-test-manual@zcs807.us.DOMAIN.com.archive
[zimbra@zcs807 ~]$ zmprov ma archive-test@zcs807.us.DOMAIN.com zimbraArchiveAccount archive-test-manual@zcs807.us.DOMAIN.com.archive
 [note] by using just the variable name and not including a + or - in front of it, we'll set it to only be one variable if there were multiple ones set.

[zimbra@zcs807 ~]$ zmprov ga archive-test@zcs807.us.DOMAIN.com | egrep -i 'archive|amavisArchiveQuarantine'                                                       # name archive-test@zcs807.us.DOMAIN.com
amavisArchiveQuarantineTo: archive-test-manual@zcs807.us.DOMAIN.com.archive
cn: archive-test
mail: archive-test@zcs807.us.DOMAIN.com
sn: archive-test
uid: archive-test
zimbraArchiveAccount: archive-test-manual@zcs807.us.DOMAIN.com.archive
zimbraArchiveAccount: archive-test-manual2@zcs807.us.DOMAIN.com.archive
zimbraArchiveAccountDateTemplate: yyyyMMdd
zimbraArchiveAccountNameTemplate: ${USER}-${DATE}@${DOMAIN}.archive
zimbraArchiveEnabled: TRUE
zimbraMailDeliveryAddress: archive-test@zcs807.us.DOMAIN.com

Set Up your Legal Intercept Variable Now To The Production Account
[zimbra@zcs807 ~]$ zmprov ma archive-test@zcs807.us.DOMAIN.com zimbraInterceptAddress archive-test-manual2@zcs807.us.DOMAIN.com.archive

For Testing - I'm purging the Inbox on all three test accounts.
[zimbra@zcs807 ~]$ zmmailbox -z -m archive-test-manual2@zcs807.us.DOMAIN.com.archive ef /Inbox
[zimbra@zcs807 ~]$ zmmailbox -z -m archive-test-manual@zcs807.us.DOMAIN.com.archive ef /Inbox
[zimbra@zcs807 ~]$ zmmailbox -z -m archive-test@zcs807.us.DOMAIN.com ef /Inbox

Create An Email That We Can Send From The CLI
[zimbra@zcs807 ~]$ cat /tmp/email-archive.txt
To: archive-test@zcs807.us.DOMAIN.com
From: admin@zcs807.us.DOMAIN.com
Subject: Test message
test

[zimbra@zcs807 ~]$ /opt/zimbra/postfix/sbin/sendmail -t < /tmp/email-archive.txt  ; tail -f /var/log/zimbra.log | grep archive-test                                  Jul 18 11:10:27 zcs807 amavis[31986]: (31986-02) ESMTP:[127.0.0.1]:10024 /opt/zimbra/data/amavisd/tmp/amavis-20140718T105435-31986-BmFt0jbd: <zimbra@zcs807.us.DOMAIN.com> -> <archive-test@zcs807.us.DOMAIN.com> SIZE=341 Received: from zcs807.us.DOMAIN.com ([127.0.0.1]) by localhost (zcs807.us.DOMAIN.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP for <archive-test@zcs807.us.DOMAIN.com>; Fri, 18 Jul 2014 11:10:27 -0600 (MDT)
Jul 18 11:10:27 zcs807 amavis[31986]: (31986-02) Checking: qeJtNl1F-X9w <zimbra@zcs807.us.DOMAIN.com> -> <archive-test@zcs807.us.DOMAIN.com>
Jul 18 11:10:27 zcs807 amavis[31986]: (31986-02) SEND from <> -> <archive-test-manual@zcs807.us.DOMAIN.com.archive>,ENVID=AM.qeJtNl1F-X9w.20140718T171027Z@zcs807.us.DOMAIN.com BODY=7BIT 250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as E7236E05EA
Jul 18 11:10:27 zcs807 amavis[31986]: (31986-02) FWD from <zimbra@zcs807.us.DOMAIN.com> -> <archive-test@zcs807.us.DOMAIN.com>,BODY=7BIT 250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as EDAEBE0B7C
Jul 18 11:10:27 zcs807 amavis[31986]: (31986-02) Passed CLEAN {RelayedInbound,Archived}, <zimbra@zcs807.us.DOMAIN.com> -> <archive-test@zcs807.us.DOMAIN.com>, quarantine: archive-test-manual@zcs807.us.DOMAIN.com.archive, Message-ID: <20140718171027.C8881E0B7B@zcs807.us.DOMAIN.com>, mail_id: qeJtNl1F-X9w, Hits: -, size: 341, queued_as: EDAEBE0B7C, 122 ms
Jul 18 11:10:27 zcs807 postfix/smtp[12085]: C8881E0B7B: to=<archive-test@zcs807.us.DOMAIN.com>, relay=127.0.0.1[127.0.0.1]:10024, delay=0.22, delays=0.09/0/0/0.12, dsn=2.0.0, status=sent (250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as EDAEBE0B7C)
Jul 18 11:10:28 zcs807 postfix/lmtp[12088]: E7236E05EA: to=<archive-test-manual@zcs807.us.DOMAIN.com.archive>, relay=zcs807.us.DOMAIN.com[10.137.27.38]:7025, delay=0.15, delays=0.02/0.01/0/0.12, dsn=2.1.5, status=sent (250 2.1.5 Delivery OK)
Jul 18 11:10:28 zcs807 postfix/lmtp[12090]: EDAEBE0B7C: to=<archive-test@zcs807.us.DOMAIN.com>, relay=zcs807.us.DOMAIN.com[10.137.27.38]:7025, delay=0.21, delays=0.01/0.01/0/0.19, dsn=2.1.5, status=sent (250 2.1.5 Delivery OK)
Jul 18 11:10:28 zcs807 postfix/smtp[12093]: 1C4ADE05EA: to=<archive-test-manual2@zcs807.us.DOMAIN.com.archive>, relay=127.0.0.1[127.0.0.1]:10030, delay=0.13, delays=0.05/0.01/0.01/0.06, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 2B480E0B9F)
Jul 18 11:10:28 zcs807 postfix/smtp[12096]: 2B480E0B9F: to=<archive-test-manual2@zcs807.us.DOMAIN.com.archive>, relay=127.0.0.1[127.0.0.1]:10025, delay=0.09, delays=0.06/0.01/0/0.02, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 3B6C6E05EA)
Jul 18 11:10:28 zcs807 postfix/lmtp[12088]: 3B6C6E05EA: to=<archive-test-manual2@zcs807.us.DOMAIN.com.archive>, relay=zcs807.us.DOMAIN.com[10.137.27.38]:7025, delay=0.15, delays=0.02/0/0/0.13, dsn=2.1.5, status=sent (250 2.1.5 Delivery OK)

Let's Confirm That All Three Accounts Show A New Message
[zimbra@zcs807 ~]$ zmmailbox -z -m archive-test-manual@zcs807.us.DOMAIN.com.archive gaf | grep Inbox                                                                         2  mess           1           1  /Inbox
[zimbra@zcs807 ~]$ zmmailbox -z -m archive-test-manual2@zcs807.us.DOMAIN.com.archive gaf | grep Inbox
  2  mess           1           1  /Inbox
[zimbra@zcs807 ~]$ zmmailbox -z -m archive-test@zcs807.us.DOMAIN.com gaf | grep Inbox
  2  mess           1           1  /Inbox

[zimbra@zcs807 ~]$ zmmboxsearch -m archive-test@zcs807.us.DOMAIN.com,archive-test-manual@zcs807.us.DOMAIN.com.archive,archive-test-manual2@zcs807.us.DOMAIN.com.archive -q "Test Message"
1)
ID: 12d3b4cf-61b8-4ba7-805f-be1d9ab6c0c2:283
Subject: Test message
Address:
 [f]  <admin@zcs807.us.DOMAIN.com>
Date: 1405703428000
Size: 1041
Conv: -283
Flags: u
Folder: 2
Fragment: test

2)
ID: 0b283721-857e-4bdc-967a-c5f9e7ec7db2:257
Subject: Test message
Address:
 [f]  <admin@zcs807.us.DOMAIN.com>
Date: 1405703428000
Size: 1269
Conv: -257
Flags: u
Folder: 2
Fragment: test

3)
ID: 3cb67670-1ad9-48c0-8f9b-372adaa2fe3f:258
Subject: Intercepted message for archive-test@zcs807.us.DOMAIN.com: Test message
Address:
 [f] Postmaster <postmaster@zcs807.us.DOMAIN.com>
Date: 1405703428000
Size: 2769
Conv: -258
Flags: au
Folder: 2
Fragment: Intercepted message for archive-test@zcs807.us.DOMAIN.com. Operation=add message, folder=Inbox, folder ID=2.
Using Another Archive Account Against the Primary Archive Account

To Do

Domain Admin Searches For ZCS6

For setting up domain admin rights, please see the following : User:Vladimir-DomainAdminSetup-v6

And you'll see about half way down a screen shot of the "Rights" panel view -- the screen shot shows the ACL right called :

adminConsoleCrossMailboxSearchRights --- Admin Console Cross Mailbox Search Rights

You'll also need to make sure domain admins have the following ACL right :

AdminLoginAs

service.UNKNOWN_DOCUMENT Error When Trying Search

The service.UNKNOWN_DOCUMENT error generally means that there is no handler for specified document. Try redeploying the zimlet, restarting the mailboxd service and let us know how it goes. The xmbxsearch zimlet is located in /opt/zimbra/zimlets-network directory.

 su - zimbra
 cd /opt/zimbra/zimlets-network
 zmzimletctl undeploy zimbra_xmbxsearch.zip
 zmzimletctl deploy zimbra_xmbxsearch.zip
 zmmailboxdctl restart

How To Turn Off / Disable ALL A&D Processing From Admin Web Console

From a "server" level, you would uncheck the "Archiving and Discovery" box in the admin web console under:

Configurations > Servers > [MTA's] > right pane > Services tab

This would stop the mta's from splitting the emails for archive.

How To Turn Off / Disable A&D For An Account

Disabling A&D at a user level, requires it to be done via the CLI:

zmarchiveconfig disable user@domainname

What Accounts Are Set For Archive And Checking Against License Usuage

Note - in regards to closing the parent account but leaving the "archive" accounts or multiple archive accounts enabled, the actual archive account doesn't count against the license - either the normal or archive limit. It's consumed based upon the variables and status of the parent email address. See :

This example below is done on a ZCS 8.6 server. The license usage is based upon the 'parent' account having amavisArchiveQuarantineTo set to something.  :

$ zmprov -l gaa
admin@ldap2.zimbra.DOMAIN.com
spam.gktt7hym36@ldap2.zimbra.DOMAIN.com
ham.0zrc899s@ldap2.zimbra.DOMAIN.com
virus-quarantine.o944yi8wwl@ldap2.zimbra.DOMAIN.com
galsync.cs_37n5wp@ldap2.zimbra.DOMAIN.com
user1@ldap2.zimbra.DOMAIN.com
user1-archive1@ldap2.zimbra.DOMAIN.com.archive
user1-archive2@ldap2.zimbra.DOMAIN.com.archive
galsync@test2.zimbra.DOMAIN.com
testuser2@test2.zimbra.DOMAIN.com

$ zmprov -l sa "amavisArchiveQuarantineTo=*"
user1@ldap2.zimbra.DOMAIN.com

$ zmprov ga user1@`zmhostname` | egrep -i "zimbraArchiveEnabled|amavisArchiveQuarantineTo|\
zimbraArchiveAccount:|zimbraIsSystemResource"

amavisArchiveQuarantineTo: user1-archive1@ldap2.zimbra.DOMAIN.com.archive, user1-archive2@ldap2.zimbra.DOMAIN.com.archive
zimbraArchiveAccount: user1-archive1@ldap2.zimbra.DOMAIN.com.archive
zimbraArchiveAccount: user1-archive2@ldap2.zimbra.DOMAIN.com.archive
zimbraArchiveEnabled: TRUE
zimbraIsSystemResource: FALSE

$ zmprov -l sa "zimbraIsSystemResource=TRUE"
galsync.cs_37n5wp@ldap2.zimbra.DOMAIN.com
galsync@test2.zimbra.DOMAIN.com
ham.0zrc899s@ldap2.zimbra.DOMAIN.com
spam.gktt7hym36@ldap2.zimbra.DOMAIN.com
user1-archive1@ldap2.zimbra.DOMAIN.com.archive
user1-archive2@ldap2.zimbra.DOMAIN.com.archive
virus-quarantine.o944yi8wwl@ldap2.zimbra.DOMAIN.com

$ zmprov -l sa "zimbraIsSystemResource=FALSE"
user1@ldap2.zimbra.DOMAIN.com

$ zmprov fc -a all ; zmsoap -z GetLicenseRequest | egrep 'TotalAccounts|ArchivingAccounts'
    <attr name="ArchivingAccountsLimit">10000</attr>
    <attr name="TotalAccounts">3</attr>
    <attr name="ArchivingAccounts">1</attr>

Need To Find A&D Accounts On Wrong MailStore To Move Them

Searching For Accounts On A Particular Mailstore

To do ldapsearch for the account, and see if there are any related entries in ldap.

su - zimbra
source ~/bin/zmshutil
zmsetvars
ldapsearch -x -H $ldap_master_url -D $zimbra_ldap_userdn -w $zimbra_ldap_password "zimbraMailHost=YOURMAILSTORE.COM" | grep "archive.domain.com"

Renaming Production Domain And Want To Also Rename Archive Domain

This was tested against ZCS 608p1.

There is no 'automatic' process to rename the associated targeted archive email addresses of the user or the archive domain when changing/renaming the domain.

  1. Get all the account details from the production users in regards to the archive acct they are using
    • zimbraMailDeliveryAddress & amavisArchiveQuarantineTo
    • Example via CLI. My initial production domain was primary.DOMAIN.com and the archive domain was primary.DOMAIN.com.archive :
      • $ zmprov ga arch-prod@primary.DOMAIN.com zimbraArchiveAccount
        • [Returns] zimbraArchiveAccount: arch-prod@primary.DOMAIN.com.archive
      • $ zmprov ga arch-prod@primary.DOMAIN.com amavisArchiveQuarantineTo
        • [Returns] amavisArchiveQuarantineTo : arch-prod@primary.DOMAIN.archive
  2. Prepare a script that will redo those variables, you'll run it after the domain renames.
    • Example via CLI. My initial archive domain was primary.zimbra.homeunix.com.archive :
      • $ zmprov ma arch-prod@primary71.DOMAIN.com zimbraArchiveAccount arch-prod@primary71.DOMAIN.com.archive
      • $ zmprov ma arch-prod@primary71.DOMAIN.com amavisArchiveQuarantineTo arch-prod@primary71.DOMAIN.archive
    • Since this is ran after the domain renames, both references in the command about a domain will use the 'new' domainnames.
    • Ref: Bulk_Provisioning
  3. Put domains into maintenance mode during a schedule downtime window
  4. Rename/forward the production domain
  5. Rename/forward the archive domain
  6. Run the script you had prepared earlier that will reset the user variable to their new archive domain.
  7. Put domains into active mode:
    • Example [notice I have the 'new' domainname here] :
      • $ zmprov md primary71.DOMAIN.com zimbraDomainStatus active
      • $ zmprov md primary71.DOMAIN.com.archive zimbraDomainStatus active
  8. Send a test message to a prod address and you can see/confirm the transaction in the /var/log/zimbra.log on the mta .


Confirmation of things working [examples]:

### From /var/log/zimbra.log on the MTA ###
Nov 17 14:42:27 mail71 amavis[7210]: (07210-01) SEND via SMTP: <> -> <arch-prod@primary71.DOMAIN.com.archive>,
  ENVID=AM.BYPyb4z8Cl0N.20101117T204227Z@mail71.DOMAIN.com BODY=7BIT 250 2.0.0 Ok, id=07210-01, 
  from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as C54F623E0E
Nov 17 14:42:27 mail71 postfix/smtpd[7446]: DDA5323E10: client=localhost[127.0.0.1]
Nov 17 14:42:27 mail71 postfix/cleanup[7262]: DDA5323E10: 
 message-id=<201011172042.oAHKgKG1024723@mail3.zimbra.homeunix.com>
Nov 17 14:42:27 mail71 postfix/qmgr[7530]: DDA5323E10: from=<root@mail3.DOMAIN.com>, 
 size=1644, nrcpt=1 (queue active)
Nov 17 14:42:27 mail71 amavis[7210]: (07210-01) FWD via SMTP: <root@mail3.DOMAIN.com> -> 
 <arch-prod@primary71.DOMAIN.com>,BODY=7BIT 250 2.0.0 Ok, id=07210-01, 
 from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as DDA5323E10

You can also use the CLI to confirm delivery into the archives & production accounts Inbox. For example:

$ zmmailbox -z -m arch-prod@primary71.DOMAIN.com gaf | grep Inbox
         2  mess           3           3  /Inbox

$ zmmailbox -z -m arch-prod@primary71.DOMAIN.com.archive gaf | grep Inbox
         2  mess           3           3  /Inbox

On Why You Have To Change The User Variables


I confirmed in my test that you do have to reset the user variables rather than just relying on the archive domain having a global forwarding. This is what shows up in /var/log/zimbra.log if you just use the domain rename/forwarding and don't reconfigure the user variable.

 
Nov 17 14:27:05 mail71 postfix/error[26043]: 8E5DD23E0E: to=<arch-prod@primary.DOMAIN.com.archive>, 
 relay=none, delay=0.04, delays=0/0/0/0.03, dsn=5.0.0, status=bounced (User unknown in virtual alias table)

sending directly to archive account, works:

Nov 17 14:37:28 mail71 amavis[7209]: (07209-01) FWD via SMTP: <root@mail3.DOMAIN.com> -> 
 <arch-prod@primary71.DOMAIN.com.archive>,BODY=7BIT 250 2.0.0 Ok, id=07209-01, 
from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 1FCA523E0E
Verified Against: Zimbra Collaboration 8.0, 7.0 Date Created: 04/16/2014
Article ID: https://wiki.zimbra.com/index.php?title=Ajcody-Server-Topics Date Modified: 2008-11-16



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »





Ajcody HSM Or Hierarchical Storage Management

   KB 2494        Last updated on 2008-11-16  




0.00
(0 votes)
24px ‎  - This is Zeta Alliance Certified Documentation. The content has been tested by the Community.


Actual HSM Or Hierarchical Storage Management Homepage

Please see Ajcody-HSM-Notes

General Q&A

What's It Look Like - Big Picture

HSM requires a new "volume" on the mailstore(s), i.e. the mailbox server. The Zimbra mailbox server(s) starts with dedicated volumes for the [Default paths listed]:

Reference is Zimbra Mailbox Server

  • Message Store [/opt/zimbra/store]
    • All email messages reside, including the message body and any file attachments.
      • Messages are stored in MIME format.
    • Each mailbox has a dedicated directory named after its internal Zimbra mailbox ID.
      • Note: Mailbox IDs are unique per server, not system-wide.
    • HSM Message Store (optional) [there is no default path, you use whatever partition you make for it]
      • Hierarchical Storage Management (HSM) allows you to configure storage volumes for older messages.
      • To manage your email storage resources, you can implement a different HSM policy for each message server.
        • Messages and attachments are moved from a primary volume to the current secondary volume based on the age of the message.
        • The messages are still accessible.
    • Single-Copy Message Storage
      • Single copy storage allows messages with multiple recipients to be stored only once in the file system.
  • Index Store [/opt/zimbra/index]
  • Data (MySQL) Store [/opt/zimbra/db]
  • Backup [/opt/zimbra/backup]
  • Log files [/opt/zimbra/log].

Backup And HSM

Please see Bugs/RFE:

HSM Impact With Backups

Zimbra mailbox servers cannot see, read, or write to another Zimbra server. The HSM store data is integrated into the mailbox servers normal backup process - they are NOT separate processes. If you find the HSM disks are having an impact on your backup times and you need to get the backup times shorter please look at the following:

HSM Impact To Server Performance

HSM currently iterates all mailboxes without pausing. This can result in a big disk/CPU hit. Please see below for more details.

HSM Running During ZCS Restarts

I created this RFE:

Customer reported that mailboxd was manually restart during their hsm operation and it resulted in an extended downtime during the zimbra start processes because the hsm activity was in the redologs and had to be processed that way before mailboxd was fully running.

How Does HSM Determine When To Move Message?

Taken from an internal thread within bugzilla.

Question is:

What date/timestamp does HSM use to determine when to move the message? The time a message is injected or the date/time in the Date: header?

Answer is:

HSM uses the date that's stored in the database. This is either the time that the message was added or the value specified in the X-Zimbra-Received header.

HSM And Attachments - Any Options?

If you would like to remove attachments (to another type of storage), please see the RFE/bug below. Comment #9 says:

Today most customers use our built-in HSM to allow for very large mailboxes but
use cheaper storage. We don't have a way to strip attachments but are looking
at some options of providing a way to move large attachments optionally to
either online or offline storage.

Please see the RFE/bug below. Vote on this if you like it.

Aging Policy Options For HSM Data

This is needed otherwise your HSM volume would grow indefinitely.

Please see the following RFE and vote on it.

What Doesn't Get HSM'd?

It's basically messages that get HSM'd. Here's some RFE/bug's I've found for missing items.

Wiki Items

RFE filed, please see:

Document & Wiki Version Items

Please see:

Briefcase Items

RFE filed, please see:

Does The Mailbox Go Into Maintenance Mode?

This was fixed in 5.0.3. Please see the following bug.

HSM Logging

I found this RFE, it might prove useful.

More Than One HSM Volume (Secondary Message Store)

There is a RFE for this, please see the following:

HSM/Secondary Volume for Spam & Junk

I'm not sure exactly what the details and dependencies are with this bug. I added a comment for clarity.

Please see the follow:

Consistency Checking Tool For HSM

This is available with zmblobchk, which checks messages in general.

Please see the RFE that it was built for for more information:

What If HSM Volume Becomes Full?

Q: What would happen if the HSM volume filled up while the HSM process was moving messages from the primary store to HSM store? Would it detect the full volume and abort the transaction(s) or would it keep trying? Is it possible that any mail would be lost if the HSM store filled up during an HSM run?

A. It's transactional, so it will fail gracefully. More specifically, if anything goes wrong during the file copying process, we delete any copied files and abort. The volume id of messages processed before the failure remains the same. HSM runs one mailbox at a time, so the rollback only happens for the last mailbox.

Restoring HSM Volumes - RFE

Please see:

A How-To Example - CLI

Introduction

This is a testing example. You should adjust these steps to use a REAL PARTITION and NOT A DIRECTORY in the steps listed below.

Create The HSM Volume

Normally you would have the HSM volume as a separate partition on the server - the lower performance disks when compared to the partition the mailstore is using.

To do a "dry" test, I did the following.

As root.

[root@mail3 ~]mkdir /opt/zimbra/hsm
[root@mail3 ~]chown zimbra:zimbra /opt/zimbra/hsm

Then switch to zimbra.

 [zimbra@mail3 ~]su - zimbra
 [zimbra@mail3 ~]zmvolume -a -n hsm-volume -t secondaryMessage -p /opt/zimbra/hsm
 [zimbra@mail3 ~]zmvolume -l
   Volume id: 2
        name: index1
        type: index
        path: /opt/zimbra/index
  compressed: false
     current: true

   Volume id: 1
        name: message1
        type: primaryMessage
        path: /opt/zimbra/store
  compressed: false
     current: true

   Volume id: 3
        name: hsm-volume
        type: secondaryMessage
        path: /opt/zimbra/hsm
  compressed: false
     current: false
Set HSM Volume To Current

Now let's set the hsm-volume to "current". Otherwise, if you try to run zmhsm you'll get a error of "invalid request: None of the secondary message volumes are marked as current."

 [zimbra@mail3 ~]$zmvolume -sc -id 3
 [zimbra@mail3 ~]$zmvolume -l
   Volume id: 2
        name: index1
        type: index
        path: /opt/zimbra/index
  compressed: false
     current: true

   Volume id: 1
        name: message1
        type: primaryMessage
        path: /opt/zimbra/store
  compressed: false
     current: true

   Volume id: 3
        name: hsm-volume
        type: secondaryMessage
        path: /opt/zimbra/hsm
  compressed: false
     current: true
Starting HSM For First Time

This is example is on a test server of mine. I don't have any messages older than 30 days. The default global configuration for HSM Age is 30 days.

[zimbra@mail3 ~]$ ls /opt/zimbra/hsm/
[zimbra@mail3 ~]$ crontab -l | grep -i hsm
[zimbra@mail3 ~]$ zmhsm -t
HSM process started.
[zimbra@mail3 ~]$ zmhsm -u
Last HSM Session Stats
----------------------
Start time: Fri Aug 15 15:48:58 EDT 2008
End time:   Fri Aug 15 15:48:58 EDT 2008
Not currently running.
Moved 0 blobs dated earlier than Wed Jul 16 15:48:58 EDT 2008
  to volume 3.
Mailboxes processed: 7 out of 7.
[zimbra@mail3 ~]$ ls /opt/zimbra/hsm/
[zimbra@mail3 ~]$ zmprov gacf | grep zimbraHsmAge
zimbraHsmAge: 30d

So, it worked but didn't have anything to Age.

Adjusting the zimbraHsmPolicy variable - ZCS 6+

Default policy [ZCS 8.6] :

$ zmprov gacf zimbraHsmPolicy
zimbraHsmPolicy: message,document:before:-30days
$ zmprov desc -a zimbraHsmPolicy
zimbraHsmPolicy
    The policy that determines which mail items get moved to secondary
    storage during HSM. Each value specifies a comma-separated list of
    item types and the search query used to select items to move. See the
    spec for <SearchRequest> for the complete list of item types and
    query.txt for the search query spec.

               type : string
              value :
           callback :
          immutable : false
        cardinality : multi
         requiredIn :
         optionalIn : globalConfig,server
              flags : serverInherited
           defaults : message,document:before:-30days
                min :
                max :
                 id : 1024
    requiresRestart :
              since : 6.0.0_BETA2
    deprecatedSince :
Adjusting the zimbraHsmAge variable - Deprecated since: 6.0.0_BETA2.
$ zmprov desc -a zimbraHsmAge
zimbraHsmAge
    Deprecated since: 6.0.0_BETA2. deprecated in favor for
    zimbraHsmPolicy. Orig desc: Minimum age of mail items whose filesystem
    data will be moved to secondary storage.. Must be in valid duration
    format: {digits}{time-unit}. digits: 0-9, time-unit: [hmsd]|ms. h -
    hours, m - minutes, s - seconds, d - days, ms - milliseconds. If time
    unit is not specified, the default is s(seconds).

               type : duration
              value :
           callback :
          immutable : false
        cardinality : single
         requiredIn :
         optionalIn : globalConfig,server
              flags : serverInherited
           defaults : 30d
                min : 0
                max :
                 id : 8
    requiresRestart :
              since :
    deprecatedSince : 6.0.0_BETA2


zimbraHsmAge must be a valid duration of: nnn[hsmd]

[zimbra@mail3 ~]$ zmprov mcf zimbraHsmAge 30d
### example initially had to reproduce issue for my RFE - please don't use 1d ###
###  [zimbra@mail3 ~]$ zmprov mcf zimbraHsmAge 1d ###
[zimbra@mail3 ~]$ zmprov gacf | grep zimbraHsmAge
zimbraHsmAge: 1
[zimbra@mail3 ~]$ zmhsm -t
HSM process started.
[zimbra@mail3 ~]$ zmhsm -u
Last HSM Session Stats
----------------------
Start time: Fri Aug 15 15:56:39 EDT 2008
End time:   Fri Aug 15 15:56:40 EDT 2008
Not currently running.
Moved 63 blobs dated earlier than Fri Aug 15 15:56:38 EDT 2008
  to volume 3.
Mailboxes processed: 7 out of 7.
[zimbra@mail3 ~]$ ls /opt/zimbra/hsm/
0
[zimbra@mail3 ~]$ ls /opt/zimbra/hsm/0/
1  14  15  3

The above commands modified the zimbraHsmAge as a global setting. You can also modify it on a server basis.

[zimbra@mail3 ~]$ zmprov gs mail3.internal.homeunix.com | grep zimbraHsmAge
zimbraHsmAge: 1d
[zimbra@mail3 ~]$ zmprov gacf | grep zimbraHsmAge
zimbraHsmAge: 1d
[zimbra@mail3 ~]$ zmprov ms mail3.internal.homeunix.com zimbraHsmAge 30d
[zimbra@mail3 ~]$ zmprov gs mail3.internal.homeunix.com | grep zimbraHsmAge
zimbraHsmAge: 30d
[zimbra@mail3 ~]$ zmprov gacf | grep zimbraHsmAge
zimbraHsmAge: 1d
Moving zimbraHsmAge Back To A Lower Number

Continuing after the above steps, let's see what happens if we now run zmhsm.

[zimbra@mail3 ~]$ zmhsm -t
HSM process started.
[zimbra@mail3 ~]$ zmhsm -u
Last HSM Session Stats
----------------------
Start time: Fri Aug 15 16:09:05 EDT 2008
End time:   Fri Aug 15 16:09:05 EDT 2008
Not currently running.
Moved 0 blobs dated earlier than Fri Aug 15 16:08:35 EDT 2008
  to volume 3.
Mailboxes processed: 7 out of 7.

Did the message files get moved back?

 [zimbra@mail3 ~]$ find /opt/zimbra/store/0/15 -name *.msg -print
 [zimbra@mail3 ~]$ find /opt/zimbra/hsm/0/15 -name *.msg -print
 /opt/zimbra/hsm/0/15/msg/0/269-21.msg
 /opt/zimbra/hsm/0/15/msg/0/263-10.msg
 /opt/zimbra/hsm/0/15/msg/0/268-14.msg
 /opt/zimbra/hsm/0/15/msg/0/261-8.msg
 /opt/zimbra/hsm/0/15/msg/0/266-13.msg
 /opt/zimbra/hsm/0/15/msg/0/259-4.msg
 /opt/zimbra/hsm/0/15/msg/0/265-12.msg
 /opt/zimbra/hsm/0/15/msg/0/257-2.msg

Nope. I'm not sure what else to add here. Don't know if there's actually a way to move them back.

RFE - To Move Msg's Back To Primary Store

Please see:

It is possible to move blobs between volumes using the zmsoap command for ZCS 6 and above. Please read the source documents on this on your ZCS server in /opt/zimbra/docs - the two files are soap-admin.txt and soap.txt . The reference your looking for is MoveBlobsRequest, in the soap-admin.txt guide. To build the proper query, you'll also want to consult the soap.txt guide. As of ZCS 8, it reads as:

<MoveBlobsRequest types="{types}" sourceVolumeIds="{volume-ids}" destVolumeId="{volume-id}" [maxBytes="{n}"]>
  [<query>{search-query}</query>]
</MoveBlobsRequest>

<MoveBlobsResponse numBlobsMoved="{n}" numBytesMoved="{n}" totalMailboxes="{n}"/>

Moves blobs between volumes.  Unlike HsmRequest, this request is synchronous,
and reads parameters from the request attributes instead of zimbraHsmPolicy.

types: a comma-separated list of item types, or "all" for all types.  See the spec for
  <SearchRequest> for details.
volume-ids: a comma-separated list of volume ids.
query: if specified, only items that match this query will be moved.
maxBytes: Limit for the total number of bytes of data to move.  Blob move will abort
  if this threshold is exceeded.

Some examples are listed below, please check your volume id's with zmvolume -l - you will NOT want to run these examples without updating the volume id's. The HSM policy and MoveBlobsRequest are search based, so you can add a query like this:

zmsoap -z MoveBlobsRequest @types=all @sourceVolumeIds=3 @destVolumeId=4 query=is:anywhere

An "is:anywhere" query will look in all folders, including /Trash for messages to move.


A query just in junk would be like the following below. This will move every blob for every mailbox located in volume 1 to volume 3 that is under the junk folder.

zmsoap -z MoveBlobsRequest @sourceVolumeIds=1 @destVolumeId=3 query=in:junk


Or you can specify a date:

zmsoap -z MoveBlobsRequest @sourceVolumeIds=1 @destVolumeId=3 query=before:1/1/2012


MoveBlobsRequest will move the blob and update mysql mail_item table accordingly.


Old Note Below


The desperate, could review the following. This would be an unsupported procedure. I'll try to engage developers on this and get some feedback though:

Now What? Place In Cron

You will have to manually put in the zimbra crontab file a line to run the zmhsm command.

Something like the following at the end of the crontab [ su - zimbra ; crontab -e ] :

# HSM
0 6 * * * /opt/zimbra/bin/zmhsm -t

This will run every morning at 0600. The question of each administrator is what is the right time to run this. I'm initial thought is to try and kick it off after your backups. One might want to have two entries for the zmhsm command, one after your daily incremental and another time frame for you full backup days.

Checking For The HSM Zimlet

In the Admin web console, you'll see the reference to the HSM zimlets in this path:

Configuration > Admin Extensions > com_zimbra_hsm

Deploying The HSM Zimlet

To deploy the zimlet for HSM

su - zimbra
zmzimletctl deploy /opt/zimbra/zimlets-network/com_zimbra_hsm.zip

Log into the admin web console - fresh session.

What Can I Do For HSM In The Admin Web Console?

Confirm the HSM zimlet is installed first.

Setup The HSM Volume

Configuration > Servers > [Server For HSM Volume]

Then select the Volumes tab on the right-hand section.

Clicking on "Add", you'll be given a drop down chooser for "Volume Type". For HSM, you would select "Secondary Message".

The "Assign Current Volumes" section will show you what volume is in use for what function.

You will also see "HSM" on this page to set the zimbraHsmAge variable for the SERVER - rather than globally.

Adjust The zimbraHsmAge Date

Per Server

Configuration > Servers > [Server For HSM Volume]

Then select the Volumes tab on the right-hand section.

You will see "HSM" on this page to set the zimbraHsmAge variable for the SERVER - rather than globally.

Global

Configuration > Global Settings

The select the HSM tab on the right-hand section.

This will set the global (default) message age for HSM.

Starting & Stopping HSM (zmhsm)

Configurations > Servers > [Server With HSM Volume]

The on the right-hand section, above the details area for the server you'll see a button/tab HSM. Click on this and you'll be given the option to Start HSM Session. It will output progress details.


References From Official Documentation

zmhsm - command for HSM

Please see:

CLI_zmhsm_Network_Edition_only

zmvolume - command for volumes

Please see:

http://wiki.zimbra.com/index.php?title=CLI_zmvolume

Global HSM Session Setting

Reference: http://www.zimbra.com/docs/ne/latest/administration_guide/Managing_ZCS.10.1.html#1111022

Global Settings HSM (Hierarchical Storage Management) sets the default message age threshold to 30 days. The HSM global setting is the default unless you change the schedule in the Server configuration. See “Scheduling HSM Sessions” .

Scheduling HSM Sessions

Reference: http://www.zimbra.com/docs/ne/latest/administration_guide/Managing_ZCS.10.1.html#1111045

HSM can be configured for secondary storage volumes for older messages. Messages and attachments are moved from a primary volume to the current secondary volume based on the age of the message. Users are not aware of any change and do not see any noticeable difference when opening an older message that has been moved.

To manage your email storage resources, you can implement a different HSM policy for each mailbox server. The message age threshold for HSM is set globally on the HSM tab or for individual servers from the Server Volume tab. The default is 30 days. The thresholds configured on individual servers override the threshold configured as the global setting. Sessions to move messages to the secondary volume are scheduled in your cron table. From the administration console, when you select a server, you can manually start a session, monitor sessions, and abort sessions that are in progress from the Volumes tab.

When you abort a session and then restart the process, the HSM session looks for entries in the primary store that meet the HSM age criteria. Any entries that were moved in the previous run would be excluded, as they would no longer exist in the primary store.


Verified Against: Zimbra Collaboration 8.0, 7.0 Date Created: 08/15/2008
Article ID: https://wiki.zimbra.com/index.php?title=Ajcody-Server-Topics Date Modified: 2008-11-16



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »





Attention.png - This article is NOT official Zimbra documentation. It is a user contribution and may include unsupported customizations, references, suggestions, or information.

A Proxy On You!

Actual Proxy Notes Homepage

Please see Ajcody-Proxy-Notes

Very Rough Draft For Proxy Trouble Shooting Script - More of steam of consciousness really

Do the following on one of your ZCS servers:

$ su - zimbra
$ touch /tmp/proxy_support.sh
$ chmod +x /tmp/proxy_support.sh

Now just vi /tmp/proxy_support.sh again and cut and paste the contents below and save the file. Make sure your not getting odd line wrapping. If you do, try clicking on the view source option at the top of the wiki page and see if that allows your paste into your shell/vi to work better. Note - I removed the leading spaces one would normally use to make the script an easier read since it causes a paste to be a horrible format.

To Do Items: Have a table formatted output, like "Variable Name | Global | ServerA | ServerB | Server C | Server D | Server E | Description of Var | Recommendations" and to include a * next to a variable if we know it should be set to something else when compared to other vars in the environment.

#!/bin/bash

printf "\nProxy Trouble Shooting Script For ZCS 8+ Servers\n\n"
printf "\n\n Reference: https://wiki.zimbra.com/wiki/Ajcody-Proxy-Notes#Very_Rough_Draft_For_Proxy_Trouble_Shooting_Script_-_More_of_steam_of_consciousness_really.\n\n"
### Global Configurations

printf "\n\n## Getting Some Global Variables ##\n\n"
printf "## GARPU [getAllReverseProxyURLs] - Used to publish into nginx.conf what servers should be used for reverse proxy lookup.\n"
printf "## Output from zmprov garpu ##\n\n `zmprov garpu` \n\n"
printf "## GARPB [getAllReverseProxyBackends]  - no description at this time.\n"
printf "## Output from zmprov garpb ##\n\n `zmprov garpb` \n\n"
printf "## GARPD [getAllReverseProxyDomains] - no description at this time.\n"
printf "Output from zmprov -l garpd ##\n\n `zmprov -l garpd` \n\n"
printf "## GAMCS [getAllMemcachedServers] - Used to list memcached servers (for nginix use).\n"
printf "Ouput from zmprov gamcs ##\n\n `zmprov gamcs` \n\n" ;
printf "## Some Global Configs from the following command: ##\n"
printf "## [zimbra@]$ zmprov gcf | egrep 'zimbraWebClientLogoutURL|zimbraWebClientLoginURL|zimbraMailURL|zimbraMailReferMode|zimbraReverseProxySSLToUpstreamEnabled|zimbraMailProxyReconnectTimeout|zimbraMailProxyMaxFails' \n\n"
zmprov gacf | egrep 'zimbraWebClientLogoutURL|zimbraWebClientLoginURL|zimbraMailURL|zimbraMailReferMode|zimbraReverseProxySSLToUpstreamEnabled|zimbraMailProxyReconnectTimeout|zimbraMailProxyMaxFails'
printf "## [For ZCS8+] If zimbraReverseProxySSLToUpstreamEnabled is set to TRUE then zimbraMailMode on the server should NOT be set to http. It should be either https or both. \n\n"
printf "## For details on zimbraMailProxyReconnectTimeout & zimbraMailProxyMaxFails , see https://bugzilla.zimbra.com/show_bug.cgi?id=80135 . Recommendation [ZCS8+] is to set to 0 , when encountering 502/504 timeout errors. \n\n"
printf "\n\n Continuing To The Specific Server Variables...\n\n"
zmlicense -p

### Server Configuration

for server_hostname in `zmprov gas`
do printf "## Need IP Information For Server $server_hostname . Using /etc/resolv.conf on local server to resolve.##\n\n"
printf "`host $server_hostname` \n\n"
printf "## Checking Enabled Services For Server $server_hostname ##\n\n"
zmprov gs $server_hostname | grep ServiceEnabled
printf "\n## Checking Running Services For $server_hostname ##\n\n"
zmcontrol -H $server_hostname status
printf "\n\n## Checking for Some Proxy Variables On $server_hostname ##\n\n"
zmprov gs $server_hostname | egrep 'zimbraReverseProxyHttpEnabled|zimbraReverseProxyMailEnabled|zimbraMailReferMode|zimbraMailMode|zimbraReverseProxySSLToUpstreamEnabled|zimbraMailProxyReconnectTimeout|zimbraMailProxyMaxFails'
printf "## [For ZCS8+] If zimbraReverseProxySSLToUpstreamEnabled is set to TRUE then zimbraMailMode on the server should NOT be set to http. It should be either https or both. \n\n"
printf "## For details on zimbraMailProxyReconnectTimeout & zimbraMailProxyMaxFails , see https://bugzilla.zimbra.com/show_bug.cgi?id=80135 . Recommendation [ZCS8+] is to set to 0 , when encountering 502/504 timeout errors. \n\n"
printf "\n\n## Checking for Some Proxy Port Variables On $server_hostname ##\n\n"
zmprov gs $server_hostname | grep Port
printf "\n\n## Checking for zimbraReverseProxyLookupTarget ; if TRUE then you should be able to telnet to port 7072 on that server. ##\n"
printf "## telnet $server_hostname:7072 \n"
printf "## If $server_hostname shouldn't be a part of the lookup pool. You can remove it by doing:\n"
printf "## [zimbra@]$ zmprov ms $server_hostname zimbraReverseProxyLookupTarget FALSE \n"
printf "## Then check the output of zmprov garpu:\n"
printf "## [zimbra@]$ zmprov garpu \n"
printf "## Then rebuild the configs on all the PROXIES and restart (one at a time is fine):\n"
printf "## [zimbra@]$ zmnginxctl restart \n"
printf "## After doing this, check that $server_hostname is no longer listed in any proxy configs on all proxies:\n"
printf "## [zimbra@]$ grep $server_hostname /opt/zimbra/conf/nginx/includes/*\n"
printf "## $server_hostname Currently has zimbraReverseProxyLookupTarget set to:\n\n"
zmprov gs $server_hostname | grep zimbraReverseProxyLookupTarget
printf "\n\n Finished Getting Variables for $server_hostname.\n\n"
done

### Domain Configuration

for domain in `zmprov gad`
do printf "## Getting Some Domain Information.##\n\n"
printf "Getting Information for $domain \n\n"
printf "## Getting zimbraVirtualHostName And zimbraVirtualIPAddress Information for $domain ##\n\n"
zmprov gd $domain | grep zimbraVirtual
printf "## Getting zimbraPublicServiceHostname, zimbraPublicServicePort And zimbraPublicServiceProtocol for $domain ##\n\n"
zmprov gd $domain | grep zimbraPublicService
printf "\n\n Finished Getting Variables for $domain.\n\n"
done
printf "\n\n Script Complete.\n\n"

Not Finished Part

Not Finished - Need to now adjust how to handle non-zcs commands to be executed on remote hosts . I would like to be able to add to this script but can't with a default multi-server ZCS environment.

Notes on limitations:

  • The below isn't effective for what we need because of the limitation noted after it.
    • ssh -i .ssh/zimbra_identity -o strictHostKeyChecking=no zimbra@server.domain.com -p 22
  • cat /opt/zimbra/.ssh/authorized_keys will give you something like:
    • command="/opt/zimbra/libexec/zmrcd" ssh-dss AAAAB3NzaC1kc3MAA [cut] A== zcs806.us.zimbralab.com
  • zmrcd restricts what commands can be ran, cat /opt/zimbra/libexec/zmrcd to see.
## cut - ifconfig output instead of using host $server_hostname
for server_hostname in `zmprov gas`
do printf "## Need IP Information For Server $server_hostname ##\n\n"
printf "`ifconfig -a` \n\n"
## cut
printf "\n\n## Checking for Some Proxy Port Variables On $server_hostname ##\n\n"
zmprov gs $server_hostname | grep Port
printf "\n\n## Confirmig Ports Are Available ## \n\n"
ZCSPXYPORTS=\'`zmprov gs $server_hostname | grep Port | cut -d':' -f2 | tr '\n' '|' | tr -d ' ' | xargs echo`\'
netstat -plnt | egrep `echo $ZCSPXYPORTS`
## cut

UPDATE PLEASE SEE THESE FIRST

No Route To Host

[working draft]

If your getting "No route to host" errors in /opt/zimbra/log/nginx.log files on the proxy servers, you should check:

  1. The resolution [DNS] to the host from the proxy servers is correct and working. This might happen more when you've deployed a new ZCS server in your environment.
  2. Following step one, you also have to confirm there is port level access to between proxy servers and the server it's trying to reach. memcache port is 11211 .
  3. The server/s are too busy to serve the request.
  4. You have a server in the list that shouldn't be in the lookup pool or you should remove the trouble server from the pool to avoid any more customer issues to deescalate the situation.

50X Errors

  • 500 Internal Server Error
    • A generic error message, given when an unexpected condition was encountered and no more specific message is suitable.
  • 502 Bad Gateway
    • The server was acting as a gateway or proxy and received an invalid response from the upstream server.
  • 503 Service Unavailable
    • The server is currently unavailable (because it is overloaded or down for maintenance). Generally, this is a temporary state. Sometimes, this can be permanent as well on test servers.
  • 504 Gateway Timeout
    • The server was acting as a gateway or proxy and did not receive a timely response from the upstream server.


These following configuration changes for an upstream mailbox server can improve proxy behavior related to timeouts:

  • This will configure proxy to immediately reconnect to all mailbox server on any failure
    • $ zmprov mcf zimbraMailProxyReconnectTimeout 0 
    • If necessary, for each mailbox server
      • $ zmprov ms <mailbox server hostname> zimbraMailProxyReconnectTimeout 0 
  • This will configure proxy to ignore failures in regards to disconnects to all mailbox servers
    • $ zmprov mcf zimbraMailProxyMaxFails 0 
    • If necessary, for each mailbox server
      • $ zmprov ms <mailbox server hostname> zimbraMailProxyMaxFails 0 
  • Then, restart all the proxies to regenerate the nginx config files with the updated values for 'fail_timeout' & 'max_fails':
  •  $ zmproxyctl restart 


See the following bug for more details on this recommendation:

Admin Guide Update

Our Administration Guide now has everything you should need to setup your proxy server.

Please see:

How To Enable Proxy

And Also Another More Current How-To Enable Proxy here:

My Own Proxy How-To For Setup

Note: We are working on improving the official documentation. I haven't seen the document yet for the 5.0.11 release, but I've been told it will have many improvements.

Here's my attempt at writing something better in the meantime.

High Level Overview

One Package

First, there's only ONE package or rpm in the installer for "proxy".

Where Does It Get Installed?

It can be installed with any of your Zimbra servers (a complete ZCS install, ZCS MTA, ZCS LDAP, ZCS Mailstore) or on its own server. ( Reference )

Other proxy documentation might give the impression it needs to go onto a zimbra server that is running some other function - mta, ldap, or mailstore.

Three Components To The Package

This package, can be broken down into THREE components.

Nginx

Nginx is a high performance IMAP/POP3 proxy server which handles all incoming POP/IMAP requests. In addition to this, it is also able to reverse proxy HTTP requests to the right backend (mailstore) server. This helps to hide the hostnames of the backend mailbox servers from end users. The steps involved in this lookup are described in more detail under the Flow_Process_For_Proxy_Target_Lookups below.

Memcached

Memcached is a high performance, distributed memory object caching system (an open source distributed in-memory hashtable). Route information is cached for further use in order to increase performance.

Memcache does not have authentication and security features so the servers should have a firewall set up appropriately. The default port is 11211 and is controlled by zimbraMemcacheBindPort conf setting in zimbraserver.

Note: Memcached will be split into its own service in the future. Meaning, for example, that you'll be able to configure a server to solely perform this function of providing the cache results. This will give more flexibility in managing performance issues, as related to the proxy environment. Rather than the same server having to share cpu/memory resources with nginx. It will also allow you to configure your different servers running nginx to quiry against a "shared" cached table of memcached. Currently, each nginx will have unique results in their memcached tables.

What is being cached, are the prior results of pop and imap lookups of users and their corresponding mailbox server they should be routed to. The process is described below in Flow_Process_For_Proxy_Target_Lookups.

Zimbra Proxy Route Lookup Handler

Zimbra Proxy Route Lookup Handler is a servlet located on the ZCS mailbox servers. This servlet handles queries for the user account route information (the server and port number where the user account resides). This is more about a difference in functionality when the proxy package is being used rather than a "new" software package being installed. The functionality difference is coming from the "software" related to nginx and memcached. The process is described in more detail in Flow_Process_For_Proxy_Target_Lookups.

NOTE - Again, don't let the reference to "mailbox server" confuse you. The mailbox servers don't need the proxy package installed on them directly to see this effect. The proxy server (whatever one you install it on, will make configuration changes global to handle this function. The referenced servlet is already on a base mailbox server install. This is referring to how the proxy package components will or can interact with that pre-existing servlet on the mailbox server.
Flow Process For Proxy Target Lookups

Here's the flow of the process for pop and imap lookups:

  1. Nginx will query the LOCAL memcached service first to see if the lookup result has a cached result. [FYI - Memcache runs over port 11211]
    • Note: when memcached is allowed to run on another server besides the one where nginx is at, you'll need to account for this port 11211 access between the servers. For now, it's all occurring locally to the server. If you can't access this memcache process, confirm you don't have any firewall/port level blocking occur for the local machine communicating over port 11211.
  2. If the initial lookups by nginx against the LOCAL memcached service have no results, nginx will then proceed to query against the "Zimbra Proxy Route Lookup Handler" services.
    • By default, all mailbox servers participate in this lookup process via the "Zimbra Proxy Route Lookup Handler".
      • This "where" to lookup is determined by the zimbraReverseProxyLookupTarget server attribute on servers running the mailbox service.
    • The lookups are performed in a round-robin fashion across configured mailbox servers.
    • Now that there is a successful lookup, the results are then cached in LOCAL memcached service and future lookups will bypass the "Zimbra Proxy Route Lookup Handler" steps for that specific lookup.
      • Meaning, the lookups will solely operate between nginx and LOCAL memcached service and no longer involve the "Zimbra Proxy Route Lookup Handler"/mailbox stores.

Here's the flow of the process for reverse proxy http lookups:

  1. Nginx will query against the "Zimbra Proxy Route Lookup Handler" services.
    • By default, all mailbox servers participate in this lookup process via the "Zimbra Proxy Route Lookup Handler".
      • This "where" to lookup is determined by the zimbraReverseProxyLookupTarget server attribute on servers running the mailbox service.
    • The lookups are performed in a round-robin fashion across configured mailbox servers.
    • Ngninx will perform this lookup for every http request, it does not involve the memcached server for cache results.
Three Functions That Can Be Done With Proxy Package

There are THREE functions or options within Zimbra that relate to "proxy".

IMAP - POP Proxy

IMAP/POP Proxy is about the use of an IMAP/POP proxy server routes users of these services to the Zimbra mailbox server on which their mailbox resides.

For example, proxying allows users to enter imap.example.com as their IMAP server. The proxy running on imap.example.com inspects their IMAP traffic, does a lookup to determine which backend mailbox server a user’s mailbox lives on (mbs1.example.com, for example), and transparently proxies the connection from user’s IMAP client to the correct mailbox server.

Flow Of The Login Process: When an IMAP or POP3 client logs in through the proxy, the following takes place:

  1. The proxy analyzes the login sequence
  2. Extracts the user name of the user trying to login
  3. Does a HTTP lookup on a mailbox server to find out which server the mailbox of the user attempting to login lives on
    • This lookup service runs on mailbox servers on port 7072, and this port on mailbox servers should be available from all proxy servers.
      • Which mailbox servers participate in this lookup is determined by the zimbraReverseProxyLookupTarget server attribute on servers running the mailbox service.
        • By default all mailbox servers participate in this lookup.
      • Lookup is performed round-robin across configured mailbox servers.
      • The result of the login name to mailbox server lookup are cached in memcached (an open source distributed in-memory hashtable).
        • The memcached process is run alongside all IMAP/POP proxy services (nginx). Meaning wherever the proxy package was install and is configured to run.
  4. Also see Flow_Process_For_Proxy_Target_Lookups, if you have not.

Port Changes For IMAP/POP Proxying: When the proxy server is configured, the service ports on backend Zimbra mailbox server are changed to alternate ports. The proxy now services the standard ports for these protocols. This change is applied even if the proxy services are run on their own independent hosts, in order to distinguish and avoid confusion between the services.

  1. Here's the changes that will be made on the mailbox servers and what ports will now be handled by the proxy server(s).
    • Note: If you have any other services running on these ports, turn them off.
      • Standard Ports now server or handled by Proxy server(s)
        • IMAP proxy port 143
        • IMAP SSL proxy port 993
        • POP proxy port 110
        • POP proxy port 995
      • Mailbox Servers will now work on these alternate ports
        • IMAP port 7072
        • IMAP SSL port 7143
        • POP port 7110
        • POP SSL port 7995
        • Route Lookup Handler (new port) 7072
        • This was described above in regards to the "Flow Of The Login Process".
Reverse HTTP Proxy
  1. Reverse HTTP Proxy
    • Using an nginx-based reverse proxy for HTTP helps to hide names of backend mailbox servers from end users. For example, users can always use their web browser to visit the proxy server at http://mail.example.com. The connection from users whose mailbox lives on mbs1. example.com will be proxied to mbs1.example.com by the proxy running on the mail.example.com server. In addition to the ZCS web interface, clients such as REST and CalDAV clients, Zimbra Connector for Outlook and Zimbra Mobile Sync devices are also supported by the proxy.
      • HTTP reverse proxy routes requests as follows:
        1. If the request has an auth token cookie (ZM_AUTH_TOKEN), the request is routed to the backend mailbox server of the authenticated user.
        2. If the requesting URL can be examined to determine the user name, then the request is routed to the backend mailbox server of the user in the URL. REST, Ca lDAV, and Zimbra Mobile Sync are supported through this mechanism.
        3. If the above methods do not work, the IP hash method is used to load balance the requests across the backend mailbox servers which are able to handle the request or do any necessary internal proxying.
Virtual Hosting

Please see Ajcody-Server-Misc-Topics#Proxy_.5BNGINX.5D_Virtual_Hosting

Proxy References

NGINX Open-Source Project Homepage

Proxy documentation BUG/RFE's:

From Install Guide':

From Wiki:

zmproxyinit Help Output [Oct 16, 2008 - From 5.0.9 I believe]:

[zimbra@mail2 libexec]$ ./zmproxyinit 
Usage: ./zmproxyinit [-h] [-o] [-m] [-w] [-d [-r] [-s] [-a w1:w2:w3:w4] [-i p1:p2:p3:p4] [-p p1:p2:p3:p4] [-x mailmode]] 
[-e [-a w1:w2:w3:w4] [-i p1:p2:p3:p4] [-p p1:p2:p3:p4] [-x mailmode]] [-f] -H hostname
        -h: display this help message
        -H: Hostname of server on which enable/disable proxy functionality.
        -a: Colon separated list of Web ports to use. Format: HTTP-STORE:HTTP-PROXY:HTTPS-STORE:HTTPS-PROXY (Ex: 8080:80:
8443:443)
        -d: disable proxy
        -e: enable proxy
        -f: Full reset on memcached port and search queries and POP/IMAP throttling.
        -i: Colon separated list of IMAP ports to use. Format: IMAP-STORE:IMAP-PROXY:IMAPS-STORE:IMAPS-PROXY (Ex: 7143:14
3:7993:993)
        -m: Toggle mail proxy portions
        -o: Override enabled checks
        -p: Colon separated list of POP ports to use. Format: POP-STORE:POP-PROXY:POPS-STORE:POPS-PROXY (Ex: 7110:110:799
5:995)
        -r: Disable reverse proxy lookup target for store server.  Only valid with -d.  Be sure that you intend for all p
roxy function for the server to be disabled
        -s: Set cleartext to FALSE (secure mode) on disable
        -w: Toggle Web proxy portions
        -x: zimbraMailMode to use on disable (Default: http)
hostname is the value of the zimbra_server_hostname LC key for the server being modified.
Required options are -f by itself, or -f with -d or -e.
Note that -d or -e require one or both of -m and -w.
Note that -i or -p require -m.
Note that -a requires -w.
Note that -x requires -w and -d for store.
Note that -x requires -w for proxy.
The following are the defaults for -a, -i, -p, and -x if they are not supplied as options.
-a default on enable: 8080:80:8443:443
-a default on disable: 80:0:443:0
-i default on enable: 7143:143:7993:993
-i default on disable: 143:7143:993:7993
-p default on enable: 7110:110:7995:995
-p default on disable: 110:7110:995:7995
-x default on store disable: http
-x default on proxy enable/disable: http

What's Wrong?

Did you check in /opt/zimbra/log/nginx.log ?

Notable Proxy Bugs

Unresolved At This Time (Sept 25, 2008)

Unsorted

  • "ZCS Mailbox Server Proxy"
    • See: http://bugzilla.zimbra.com/show_bug.cgi?id=9469
    • From release notes, "HTTP proxy is supported (Beta). This includes REST sharing, Caldav, and Zimbra mobile. For Zimbra Mobile, ZCS redirect/forwards to the correct host. (Bug 9469)"
      • Please note, that ZCS HTTP Proxy (http reverse proxy) is Beta, first released in 5.0.6.
    • Though, as of 5.0.8,""For activesync connection to go thru the proxy, they'll have to configure reversed proxy. However, there're already a few known bugs about sharing broken with reversed proxy enabled. They're better off waiting for 5.0.9 than to try to configure reversed proxy now."
  • "need to support multi-server (proxy) for mobile clients"

Resolved In 5.0.9

Those That Are Sort Of Client Side Impacting
Configuration / Initial Setup Issues

Resolved in 5.0.7

Resolved in 5.0.6

Public Service Hostname

Resolved in 5.0.5

  • Nginx: http proxy does not work with REST
    • Note, this isn't "completely" resolved, as you'll notice by other targeted bugs with 5.0.5+ releases.
    • See: Nginx: http proxy does not work with REST

Single ZCS Server

Need To Disable Pop/Imap Proxy And Use POP/IMAP Normally

Sometimes, people install/setup proxy services on their single ZCS server and they don't need them. Here's how you would disable the proxy stuff and get imap/pop working over the default ports.

do a zmprov -l gs `zmhostname` | grep -i port

get the ports, then set variables to port 0:

zmprov ms `zmhostname` zimbraImapProxyBindPort 0
zmprov ms `zmhostname` zimbraImapSSLProxyBindPort 0
zmprov ms `zmhostname` zimbraPop3ProxyBindPort 0
zmprov ms `zmhostname` zimbraPop3SSLProxyBindPort 0

then, set the non "Proxy" ports to the desired standard ports

zmprov ms `zmhostname` zimbraImapBindPort 143
zmprov ms `zmhostname` zimbraImapSSLBindPort 993
zmprov ms `zmhostname` zimbraPop3BindPort 110
zmprov ms `zmhostname` zimbraPop3SSLBindPort 995

once complete:

zmprov ms `zmhostname` -zimbraServiceEnabled memcached
zmprov ms `zmhostname` -zimbraServiceEnabled imapproxy

zmproxyctl stop
zmmemcachedctl stop
zmmailboxdctl stop
zmmailboxdctl start 

Proxy Guide Rewrite Scratch Pad

Intro

Moved to Ajcody-Proxy-Guide-Rewrite-Project



Ajcody Backup & Restore Issues

   KB 2494        Last updated on 2008-11-16  




0.00
(0 votes)
24px ‎  - This is Zeta Alliance Certified Documentation. The content has been tested by the Community.


Actual Backup & Restore Issues Homepage

Please see Ajcody-Backup-Restore-Issues

Update - May, 2017

I'll be updating this page after the ZCS 8.8 release is done to only reflect what is true in that version and future versions. It will also include information about the Zimbra_Suite_Plus modules where appropriate. If you currently are using the backup module for ZSP, please see Zimbra_Suite_Plus/Zimbra_Backup_Plus for now.

Backup-Restore Training Material Rough Drafts I Wrote

Bug/RFE's I Filed Against ZCS 8.6

Miscellaneous Bugs & RFE's

For those that address other specific issues described on this page:

RFE For Live DR Restore Option
Backups And Upgrades And Prior Versions

Information To Provide When Submitting A Support Case For Backup Issues

Basic Backup Information To Submit To Support

Disk Space Usage Issues
Trend Data

If there is concerns about disks/partitions getting full, this command would be helpful for trending data on your server. Send support the resulting df.tar file . Note - adjust the tail command if you want more than 20 day's worth of trending data, the -n 20 option.

[zimbra@zcs806 tmp]$ /tmp

[zimbra@zcs806 tmp]$ tar cvf /tmp/df.tar `find /opt/zimbra/zmstat -name df.cs\* | sort | tail -n 20`
tar: Removing leading `/' from member names
/opt/zimbra/zmstat/2014-03-10/df.csv.gz
/opt/zimbra/zmstat/2014-03-11/df.csv.gz
 [cut - Ajcody]
/opt/zimbra/zmstat/2014-03-27/df.csv.gz
/opt/zimbra/zmstat/2014-03-28/df.csv.gz
/opt/zimbra/zmstat/df.csv

[zimbra@zcs806 tmp]$ ls -lah /tmp/df.tar
-rw-r----- 1 zimbra zimbra 80K Mar 29 06:44 /tmp/df.tar

[zimbra@zcs806 tmp]$ tar tvf /tmp/df.tar
-rw-r----- zimbra/zimbra  2566 2014-03-11 00:00 opt/zimbra/zmstat/2014-03-10/df.csv.gz
-rw-r----- zimbra/zimbra  2553 2014-03-12 00:00 opt/zimbra/zmstat/2014-03-11/df.csv.gz
 [cut - Ajcody]
-rw-r----- zimbra/zimbra  2513 2014-03-28 00:00 opt/zimbra/zmstat/2014-03-27/df.csv.gz
-rw-r----- zimbra/zimbra  2531 2014-03-29 00:00 opt/zimbra/zmstat/2014-03-28/df.csv.gz
-rw-r----- zimbra/zimbra  8013 2014-03-29 06:40 opt/zimbra/zmstat/df.csv
Directory Sizes In /opt/zimbra

Please see the following and provide the output to support. Note, even though this method is faster than doing a du it still can take awhile.

* Ajcody-Server-Misc-Topics#Faster_Way_To_Get_Directory_Size_On_Filesytem_-_find_vs_du
Adjusting The Disk Alert Threshold

Note - zmlocalconfig smtp_notify must return yes if you want to receive the notifications.

If you just need to adjust the disk alert threshold, then see the following:

See current values:

 zmlocalconfig | grep zmdisklog

Example adjustment:

 su - zimbra
 zmlocalconfig -e zmdisklog_critical_threshold=98
 zmlocalconfig -e zmdisklog_warn_threshold=95
 zmstatctl

To exclude a partition from the checks [example of two being excluded]:

 su - zimbra
 zmlocalconfig -e zmstat_df_excludes="/mount/point:/mount/point2"
 zmstatctl

They might be a bug on this, where you'll keep getting email until a logrotate happens [zimbra.log?].

Some things to do to confirm and share with support or in bug. As zimbra

su - zimbra

ls -la /var/log/zimbra.log

df -h
 /dev/mapper/vg_rhel664-lv_root
                      5.5G  3.5G  1.7G  68% /
 tmpfs                 939M     0  939M   0% /dev/shm
 /dev/sda1             485M   79M  381M  18% /boot
 /dev/sdb1              30G  6.2G   23G  22% /opt

date

zmlocalconfig | grep zmdisklog
 zmdisklog_critical_threshold = 80
 zmdisklog_warn_threshold = 85  

zmlocalconfig -e zmdisklog_critical_threshold=95

zmlocalconfig -e zmdisklog_warn_threshold=90

zmlocalconfig | grep zmdisklog
 zmdisklog_critical_threshold = 95
 zmdisklog_warn_threshold = 90

zmstatctl restart

date

ps -eaf | grep zmstat-df

ls -la /var/log/zimbra.log

date ; grep "Disk warning" /var/log/zimbra* ; zmmailbox -z -m admin@`zmhostname` s -l 100 -t message "Subject: Disk and after:yesterday"

##Note - Emails by default go out every 10 minutes - for example:

[zimbra@zcs803 ~]$ date ; grep "Disk warning" /var/log/zimbra* ; zmmailbox -z -m admin@`zmhostname` s -l 100 -t message "Subject: Disk and after:yesterday"
Thu May 22 09:40:08 PDT 2014
/var/log/zimbra.log:May 22 08:30:00 zcs803 zimbramon[18826]: 18826:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
/var/log/zimbra.log:May 22 08:40:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
/var/log/zimbra.log:May 22 08:50:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
/var/log/zimbra.log:May 22 09:00:00 zcs803 zimbramon[22970]: 22970:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
  ## Note - I had readjusted the variable to not warn during this time segment ##
/var/log/zimbra.log:May 22 09:20:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
/var/log/zimbra.log:May 22 09:30:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
/var/log/zimbra.log:May 22 09:40:00 zcs803 zimbramon[8322]: 8322:err: Disk warning: zcs803.DOMAIN.com: / on device /dev/mapper/vg_rhel664-lv_root at 82%
num: 7, more: false

     Id  Type   From                  Subject                                             Date
   ----  ----   --------------------  --------------------------------------------------  --------------
1.  328  mess   admin                 Disk / at 82% on zcs803.DOMAIN.com:           05/22/14 09:40
2.  327  mess   admin                 Disk / at 82% on zcs803.DOMAIN.com:           05/22/14 09:30
3.  326  mess   admin                 Disk / at 82% on zcs803.DOMAIN.com:           05/22/14 09:20
  ## Note - I had readjusted the variable to not warn during this time segment ##
4.  325  mess   admin                 Disk / at 82% on zcs803.DOMAIN.com:           05/22/14 09:00
5.  324  mess   admin                 Disk / at 82% on zcs803.DOMAIN.com:           05/22/14 08:50
6.  323  mess   admin                 Disk / at 82% on zcs803.DOMAIN.com:           05/22/14 08:40
7.  320  mess   admin                 Disk / at 82% on zcs803.DOMAIN.com:           05/22/14 08:31

Continue to monitor your zmmailbox search results for an hour.

The Basic Information Support Needs

as root:

  • cat /etc/fstab
    • Shows us what is mounted upon boot
  • cat /proc/mounts
    • Shows us what is currently mounted and its status - you can see if a mount is read-only here.
  • df -hT
    • Lists current mounts using human-readable size information and also notes the filesystem type.

as zimbra:

  • zmprov -l gs `zmhostname` | egrep 'Back|Redo'
    • Will show us a number of variables related to backup and redologs. Also tell us if your using auto-group or the default method.
  • du -sh /opt/zimbra/redolog
    • Will might notice your redolog logs aren't rolling over, causing a possible issue.
  • ls -latr /opt/zimbra/backup
    • This is the default backup target, please adjust this path here and below if you are using a different zimbraBackupTarget value.
    • zmprov gs `zmhostname` zimbraBackupTarget
    • We'll be able to confirm permissions are right.
  • ls -latr /opt/zimbra/backup/tmp
    • This will show us if you have failed backup jobs and confirm tmp is being cleaned appropriately after the backup is done.
  • ls -latr /opt/zimbra/backup/sessions
    • This will show us what backup sessions are available and confirm permissions are correct.
    • Adjust path if your zimbraBackupTarget value is not the default path.
      • su - zimbra
      • zmjava com.zimbra.cs.backup.util.GetVersion
      • cd /opt/zimbra/backup/sessions/full-[YOUR MOST RECENT FULL]/
      • head -n6 session.xml
      • cd /opt/zimbra/backup/sessions/incr-[YOUR MOST RECENT INCREMENTAL]/
      • head -n6 session.xml
  • Some directory sizes in the backup directory:
    • Default path first
      • du -sh `find /opt/zimbra/backup -maxdepth 2 -type d`
    • If your using a different backup target, check that directory also. Replace /opt/zimbra/backup above with your backup path.
  • zmbackupquery
    • This should match what's in the sessions directory and it will also tell us if status of each backup and how many accts were done.
  • crontab -l | grep -i back
    • This will show use when backups are support to run and with what options they are running with.
  • zmlocalconfig | grep -i back
    • This is useful to see a number of backup options not exposed in the crontab, things related to the zip options.
  • zmvolume -l
    • This is useful to see how many volumes are being used, if HSM is being used, and if compression is being done at the volume level.
Additional Log Files Support Might Need

And send the following logs:

  • /var/log/messages
    • Filesystem issues often times are noted here and also in syslog. This might explain an interruption in the backup process. Server restarts, filesystem going full, filesystem going read-only, etc.
  • /var/log/syslog
  • /opt/zimbra/log/mailbox.log
    • The backup activity is logged here.
    • And any other mailbox.log file that would cover the event

Additional Checks For Performance Specific Issues

If Your Using a SAN or NFS For Your Backup Target - Please Check Your IOWait

Ideally, you would compare iowait and performance data from the target backup host as well as the stats available on the ZCS servers. To get graphs and stats on this from ZCS, please see Ajcody-Testing-Debugging#zmstat_and_zmstat-chart . You should submit this data and iowait conclusions if you still need to submit a support case about backup performance issues.

Is HSM Running During Your Backup Window
Are You Using --zipStore

--zipStore zips the blobs vs. keeping the blobs as individual files. --zipStore does not use compression either. For most circumstances, this will give the best performance, especially with NFS. This should be the default behavior of the backups, the following RFE is when it became the default [ZCS6+] :

To see if zip's are being used for backups for example, in the backup/session directory you'll know if it is by seeing .zip files:

mail:~/backup/sessions/full-20080820.160003.770/accounts/115/988/11598896-a89b-4b9d-bedb-1ed1afcb6c87/blobs 
zimbra$ ls blobs-1.zip    blobs-2.zip    blobs-3.zip    blobs-4.zip

To see if the zip file is using compression [-Z option for unzip will indicate whether or not the archive is actually compressed] :

unzip -Z blobs-4.zip
293 files, 5982984 bytes uncompressed, 5982984 bytes compressed:  0.0%

Also, if your zmvolume has compression enabled the blobs will remain compressed within the zip also upon backup. The point being, they are uncompressed to be then put into a zip file when the backup is using --zipStore.

Restore Compatibility Between ZCS Versions

Please see the following for details. In summary, user level restores should work against older ZCS backup data.

Other related bug/rfe's:

NFS Use For Backups

Please see this RFE:

This is the proposed statement to be included in the release notes following the RFE:

ZCS & NFS:
Zimbra will support customers that store backups (e.g. /opt/zimbra/backup) on an NFS-mounted partition. Please note that this does not relieve the customer from the responsibility of providing a storage system with a performance level appropriate to their desired backup and restore times. In our experience, network-based storage access is more likely to encounter latency or disconnects than is equivalent storage attached by fiber channel or direct SCSI.
Zimbra continues to view NFS storage of other parts of the system as unsupported. Our testing has shown poor read and write performance for small files over NFS implementations, and as such we view it unlikely that this policy will change for the index and database stores. We will continue to evaluate support for NFS for the message store as customer demand warrants.
When working with Zimbra Support on related issues, the customer must please disclose that the backup storage used is NFS.

Things To Check

  • Check the /var/log/messages on both the zimbra server and the nfs server for nfs related errors during the time frame of your backup.
  • Check /opt/zimbra/log/mailbox.log for error messages about folders/files not being able to be written or missing directory errors.
  • Is root_squash configured on the nfs server? If it's changed to no_root_squash , does the behavior of the backup change?
  • Is the */backup directory owned by zimbra:zimbra with at least 750 or 755 permissions?
    • This parent directory as given in:
      • zmprov gs `zmhostname` zimbraBackupTarget
  • Does zimbraBackupTarget have at least the subdirectories of : sessions and tmp  : and are owned by zimbra:zimbra with 750 or 755 permissions?
    • If not, try manually creating them and then running a test backup.
  • IF USING A NAS - MAKE SURE YOUR NOT USING EXTENDED ACLS OR THAT YOU HAVE THEM CONFIGURED PROPERLY
    • If your backup session directory shows something like : drwxrwx---+ 2 zimbra zimbra 4096 Sep 14 00:00 TO_DELETE-full-XXXXXX , that + sign indicated extended acls are in use.

Debugging Example

Steps I wrote for one customer, where saving out the information as you walk through all the commands would give enough information [hopefully] to submit a good rfe/bug:

 1. make a test partition on nfs server - /nfs-test

 2. mount on zimbra server
 2A. mkdir /nfs-test
 2B. chmod 755 /nfs-test
 2C. mount nfs-server:/nfs-test /nfs-test
 2D. ls -la /nfs-test
 2E. mkdir /nfs-test/backup
 2F. chown zimbra:zimbra /nfs-test/backup
 2G. chmod 755 /nfs-test/backup
 2H. su - zimbra ; touch /nfs-test/backup/testfile 
 2I. ls -laR /nfs-test/
 2J. rm /nfs-test/backup/testfile

 3. Set zimbraBackupTarget
 3A. zmprov ms `zmhostname` zimbraBackupTarget /nfs-test/backup

 4. Run a full backup against one account
 4A. ex. zmbackup -f -a user@domain.com

 5. ls -laR /nfs-test/

 6. If you again, run into the same problem. You could also repeat the backup after increasing the backup 
    logging variable for the account your trying to backup. If you didn't run into the same problem, it might 
    had to do with the initial setup of the nfs mount and permissions being used during the directory creation.
 6A. zmprov aal user@domain.com zimbra.backup debug
 6B. logging will show up in /opt/zimbra/log/mailbox.log
 6C. Remove account logging when your done.  zmprov ral user@domain.com zimbra.backup

 8. Change zimbraBackupTarget back to your production path.

Setup A Fast Test NOT Using NFS

A way to "avoid" the NFS issues for testing purposes would be to setup a new zimbraBackupTarget to try doing a full backup of a couple of user accounts. I DON'T recommend this if your using auto-group for zimbraBackupMode [ zmprov gs `zmhostname` zimbraBackupMode ] , only if your using Standard mode.

[as root]
** adjust your new "backup" directory path as needed - mine is just an example**
mkdir /mnt/usb1/backup-test
chown zimbra:zimbra /mnt/usb1/backup-test
chmod 750 /mnt/usb1/backup-test
su - zimbra
**confirm backup mode as standard**
zmprov gs `zmhostname` zimbraBackupMode
**if not stand, please stop**
zmprov ms `zmhostname` zimbraBackupTarget /mnt/usb1/backup-test
**Find a couple of test accounts to do a full for. Make sure you'll have space to do the backup regards to need free space.**
zmbackup -f -a user1@domain.com user2@domain.com user3@domain.com
**Watch and confirm status of backup you just started.**
zmbackupquery
**Confirm files were backed up in right location**
ls /mnt/usb1/backup-test/sessions/
**Failed backups would most likely results in left over directory in tmp directory**
ls /mnt/usb1/backup-test/tmp/

Restore For Disaster Recovery

For Full Single ZCS Server DR Restores

Please see Network_Edition_Disaster_Recovery

Some additional notes I have on it - Ajcody-Disaster-Recovery-Specific-Notes

For Multi-Server DR Restore Specifics

Along with the above references, please see Ajcody-Notes-Multi-Server-Restore-DR

To Restore Just The LDAP Date

Let's say your ldap data was 'lost/destoyed' but everything else was intact, you should look at the zmrestoreldap command.

This section should have more precaution and background information to handle this section.

The basics:

  • To find the LDAP session labels type -lbs.
    • zmrestoreldap -lbs
  • Restore the complete LDAP directory server
    • zmrestoreldap -lb full20061130135236
  • Restore LDAP data for specific accounts
    • zmrestoreldap -lb full20061130135236 -a tac@DOMAIN.com jane@DOMAIN.com

To Restore Just The Mysql DB

Option available from the following RFE work:

  • "Allow backup of only primary message volume"
    • https://bugzilla.zimbra.com/show_bug.cgi?id=35278
      • Options to exclude types of data as described in the 6.0.6 Admin Guide:
        • Search index : If you do not restore the search index data, the mailbox will have to be reindexed after the restore.
          • zmrestore <all or account> --exclude-search-index
        • Blobs : This is a useful option when all blobs for the mailbox being restored already exists.
          • zmrestore <all or account>|--exclude-blobs
        • HSM-blobs : This is useful when all HSM blobs for the mailbox being restored already exists.
          • zmrestore <all or account> --exclude-hsm-blobs

Let's say your mysql data was 'lost/destoyed' but everything else was intact. This might be the solution for the situation:

The steps below are to REPRODUCE A DR situation to test.

  1. zmcontrol stop
  2. Caution - step to reproduce DR situation to test against
    1. mv ~/db/data ~/db/data.OLD
  3. Getting old mysql passwords
    • zmlocalconfig -s mysql_root_password
    • zmlocalconfig -s zimbra_mysql_password
  4. [as zimbra] /opt/zimbra/libexec/zmmyinit
  5. ldap start
  6. zmconvertctl start
  7. mysql is already running per the zmmyinit - mysql.server status - to check.
  8. zmrestoreoffline -a all -br --systemData --excludeSearchIndex --excludeBlobs --excludeHsmBlobs
    • Note - you most likely will want to use the -br or -rf option for this situation.
      • -br,--backedupRedologsOnly : Replays the redo logs in backup only, which excludes archived and current redo logs of the system
      • -rf,--restoreFullBackupOnly : Restores to last full backup only, which excludes incremental backups.
    • -sys,--systemData : Restores global tables and local config.
    • Note the use of -a all , you could also do this for one account first to confirm operation is successful for your circumstances.
  9. If you used -rf or -br with your zmestoreoffline, you might also need to use zmplayredo to finish it up to get the most complete restore.

Performance Issues And Time To Complete

Please see Ajcody-Notes-BackupPlans#What_About_Backups.3F_I_Need_A_Plan

Also created an RFE for increase backup performance:

Understanding Option Flags For zmbackup & zmrestore

First, they don't make sense if your just reading from the help output - I will not argue this point at all.

The biggest problem with the options I point out below is that you can often include them in the command and they do nothing or you include them for a particular situation and they don't apply. Why is this a problem? Because they are silent and give no output telling you that it's not necessary, it's redundant, or it will actually cause your intended results to fail simply because you included the option.

zmrestore Options

Problems mostly revolve around these options.

To Times In The Past (If -lb Isn't Used, Implies Your Using Times/Incr/RedoSeq AFTER Last Full)
  • -restoreToIncrLabel
    • <arg> Replay redo logs up to and including this incremental backup
      • Requires: --label or -lb
  • -restoreToTime
    • <arg> Replay rodo logs until this time
      • Requires: --label or -lb
  • -restoreToRedoSeq
    • <arg> Replay up to and including this redo log sequence
Redolog Variables
  • --backedupRedologs or -br
    • Replays the redo logs in backup only, which excludes archived and current redo logs of the system
      • Only useful when restoring against latest full backup (NOT using the -lb option).
      • Will restore using incremental backup data after the last full backup as well but NOT including any redolog activity.
  • --restorefullBackup or -rf
    • Restores to the full backup only, not any incremental backups since that backup.
      • The default behavior of zmrestore in general is to always play from a "full" and "incrementals" that are associated with it UNLESS you state otherwise.
        • If you do:
          zmrestore -a user@domain.com -lb full6monthsago
        • It will playback the incremental data associated with that full from 6 months ago.
        • If you do:
          zmrestore -a user@domain.com -lb full6monthsago -rf
        • It will ONLY playback the data in the full from 6 months ago.
      • Will not progress past the data that's in the last full backup, no incremental backups after it in other words.
      • Implies NO redolog play, so there's no need to use -br.
Targets And Labels
  • --label or -lb
    • <arg> The label of the full backup to restore. Restores to the latest full backup if this is omitted.
  • --target or -t
    • <arg> Specifies the backup target location. The default is <zimbra_home>/backup.
Impact Of AutoGroup Option Being Used

Place for notes about how autogroup backup option might impact or limit command options.

zmbackup Options

Problems mostly revolve around these options:

  • --target or -t
    • <arg> Specifies the target backup location. The default is <zimbra_home>/backup.
  • --zip or -z
    • Zips email blobs in backup - using compression
  • --zipStore
    • Zips email blobs in backup - does NOT use compression
Deleting Old Backups -del

Caution You want to delete from the oldest label to newest. The -del option will automatically purge all older sessions prior to the label you used. To find out the label names, use zmbackupquery.

Format example:

zmbackup -del <oldest_backup_label>
Impact Of AutoGroup Option Being Used

Place for notes about how autogroup backup option might impact or limit command options.

Changing Default Backup Target

To find out what the current backup target is, do:

 zmprov gacf | grep zimbraBackupTarget
 zimbraBackupTarget: /opt/zimbra/backup

This is also configurable at the "server" level:

 zmprov gs `zmhostname` | grep zimbraBackupTarget
 zimbraBackupTarget: /opt/zimbra/backup

For example:

 zmprov ms `zmhostname` zimbraBackupTarget /san/mount/backup

Issues of changing the default path in regards to the admin web console. Please see:

Another way to change the backup path is described at Changing_Backup_directory_and_General_Information. I recommend reviewing it as well.

See Change Location For Backup Or Restore Source Data for non-default adjustments to CLI commands in regards to a non-default path for backups.

RFE & Bugs Concerning Option Flags For Restore And Backup

Setting Account Status After Restore Is Done

I filed an RFE for this:

Use DL COS Status To Generate User List Of Accounts

I filed an RFE for this:

An Overview Of Some Backup/Restore Items

Shared Blobs (messages)

I believe we are a little light on describing the shared blobs situation. Shared blobs can cause different corrections to a problem as compared to a normal message issue that isn't shared. I'll start some notes on this here.

From Backup and Restore

  • "When backing up shared messages, the backup process looks to see whether a Binary Large Object file (BLOB) representing a message already exists in the backup. If it does, it simply flags this object as such and does not copy its content again."
  • "Keeping the same backup target saves disk space, because shared binary large object files (BLOB) and other files do not have to be duplicated every time the backup process runs.

Bugs/RFE's

Remote copies of backup data for DR use

You will want to copy over /opt/zimbra/redolog/archive/* and /opt/zimbra/redolog/redo.log frequently in order to stay current. The redo.log file being open is not a problem since the crash recovery step can work with redo.log file in any state.

The redolog/archive/ contains logs that have not yet been backed up by an incremental (or by a full in auto-grouped mode)

The redo.log rolls over when it reaches zimbraRedoLogRolloverFileSizeKB (by default 100MB prior to ZCS 5.0.11 and 1GB after). When ZCS restarts after a crash, it seems to work through the current redo.log ok regardless of its state, if the current log really must be copied."

My Initial Thoughts On This

Start of process:

  • Weekend full on prod
  • rsync full-xxx on prod > remote sessions/
  • rsync redolog/* > remote redolog/
    • through non-full and incremental times every x about of minutes
  • weekday nights 10pm incre-xxx on prod
  • rsync incr-xxx on prod > remote sessions/
  • rsync redlog/* > remote redolog/

Created three separate rsync cron rules.

  • Full - once a week
    • Confirms full is done and then looks for latest full-xxxx and rsyncs that specific directory
  • Incre - once a night except for full schedule night
    • Confirms incre is done and then looks for latest incr-xxx and rsyncs that specific directory
  • Redolog/ - every x amount of minutes (outside of full and incr backups sessions)
    • Soes full rsync of redolog/ - probably want delete/remove option?
    • lsof will report /opt/zimbra/redolog/redo.log is open.

Somewhere we need to account for accounts.xml in this process. And also confirm what else might be missing. Also, steps on the actually restore process depending on when/where the DR event took place.

What's Needed For Later Restores

In regards to what is moved and what is needed for later restores, you must remember this "flow" of the backups:

  • Full backup files that contains all the information needed to restore mailboxes (to that point in time)
  • Incremental backup files that contains the LDAP directory server files and all the redo log transactions written since the last backup (to that point in time)
  • Redo logs that contains current and archived transactions processed by the Zimbra server since the last incremental backup (to that point in time - more about this topic is above)

Variables to be aware of in regards to backup/restore [ viewed with - zmprov gs [server-name] | grep Backup ]:

  • This is that gui path option - note, it doesn't change default to redo file path [see below]
zimbraBackupTarget

zimbraBackupMode
zimbraBackupAutoGroupedInterval
zimbraBackupAutoGroupedNumGroups
zimbraBackupAutoGroupedThrottled

zimbraBackupReportEmailSubjectPrefix
zimbraBackupReportEmailSender
zimbraBackupReportEmailRecipients
  • Variables to be aware of in regards to redo files [ viewed with - zmprov gs [server-name] | grep Redo ]:
zimbraRedoLogArchiveDir
zimbraRedoLogDeleteOnRollover
zimbraRedoLogEnabled
zimbraRedoLogFsyncIntervalslMS
zimbraRedoLogLogPath
zimbraRedoLogRolloverFileSizeKB

Mysql Table That References Most Recent Backup Session Of Users (AutoGroup Backup Mode)

During the backup we update the zimbra.mailbox table for each mailbox to record the most recent backup time. This is in the "last_backup_at" column within Mysql.

This data is used by auto-grouped backup to figure out which mailboxes to backup.

Creating A List Of Last Backup Of Users

Remove the , | head , below to get a full listing of all your accounts. Note, this reports on the users that exist on the mailstore your running the command.

[zimbra@zcs806 tmp]$ mysql zimbra -NBe 'select from_unixtime(last_backup_at), comment from mailbox' | sort | head
2014-06-05 01:00:24     dluser558@zcs806.DOMAIN.com
2014-06-05 01:00:26     dluser557@zcs806.DOMAIN.com
2014-06-05 01:00:28     dluser556@zcs806.DOMAIN.com
2014-06-05 01:00:30     dluser555@zcs806.DOMAIN.com
2014-06-05 01:00:31     dluser554@zcs806.DOMAIN.com
2014-06-05 01:00:32     dluser551@zcs806.DOMAIN.com
2014-06-05 01:00:32     dluser552@zcs806.DOMAIN.com
2014-06-05 01:00:32     dluser553@zcs806.DOMAIN.com
2014-06-05 01:00:33     dluser550@zcs806.DOMAIN.com
2014-06-05 01:00:34     dluser549@zcs806.DOMAIN.com

I include also below the results from the zmbackupquery for the first user:

[zimbra@zcs806 tmp]$ date ; zmbackupquery -a dluser558@zcs806.DOMAIN.com
Wed Jun 11 14:29:48 PDT 2014
Account: dluser558@zcs806.DOMAIN.com

    Label:   full-20140605.080023.296
    Type:    full
    Started: Thu, 2014/06/05 01:00:23.296 PDT
    Ended:   Thu, 2014/06/05 01:02:37.739 PDT
    Acct ID: 85d3c8f4-eea8-4cf2-8a84-8f1fcc87d555

    Label:   full-20140529.080016.546
    Type:    full
    Started: Thu, 2014/05/29 01:00:16.546 PDT
    Ended:   Thu, 2014/05/29 01:02:51.556 PDT
    Acct ID: 85d3c8f4-eea8-4cf2-8a84-8f1fcc87d555

    Label:   full-20140522.080015.818
    Type:    full
    Started: Thu, 2014/05/22 01:00:15.818 PDT
    Ended:   Thu, 2014/05/22 01:02:42.126 PDT
    Acct ID: 85d3c8f4-eea8-4cf2-8a84-8f1fcc87d555

    Label:   full-20140515.080016.160
    Type:    full
    Started: Thu, 2014/05/15 01:00:16.160 PDT
    Ended:   Thu, 2014/05/15 01:02:34.204 PDT
    Acct ID: 85d3c8f4-eea8-4cf2-8a84-8f1fcc87d555
Possible Issue That A Failed Or Interrupted Backup Causes

An interrupted backup can cause an issue because the table currently gets updated right off the bat rather than waiting for backup to be successfully completed.

Possible RFE: To update zimbra.mailbox.last_backup_at column for successfully backed-up mailboxes to the very end of the backup process, to either just before or just after renaming the /opt/zimbra/backup/tmp/<backup label> directory to /opt/zimbra/backup/sessions/<backup label>.

Setting To Null To Cause A New Backup For User

To undo what was done by an interrupted backup for example, you need to clear this column (set it to null) for the affected mailboxes. By clearing the column, you're forcing the next AG backup to choose these mailboxes because they look like they have never been backed up. If you don't clear this column, you have to wait until the next cycle. (7 days)

Example syntax to view:

mysql zimbra -e "select last_backup_at from mailbox where id=27"

Example syntax to change data of the last_backup_at to NULL"

mysql zimbra -e "UPDATE mailbox SET last_backup_at = NULL WHERE id = 27"
Related Bugs RFE's

Restore Requires accounts.xml File

Also, see:

Change "Location" For Backup Or Restore Source Data

Remember that zmbackup and zmrestore can take flags as well in regards to the location of items.

  • zmrestore & zmbackup can both take : -t,--target (default <zimbra_home/backup)

You can't state a different location with redo logs though. There's a command called zmplayredo [for newer versions of ZCS] and it has a variable to point to the redologs to play from [ --logfiles ]. It will replay into the default redolog directory or redolog file. The mailbox has to be stop to run zmplayredo . This is a command to manual kick off a replay of a redo log. This is normally done with the zmrestore when options about to a specific time aren't included.

Manual Removal Of Older Backup Sessions

General Situation:

  • Keep in mind every restore requires starting with data from a full backup.
  • For each account on the server, there must be at least 1 full backup after the deletion is complete.
  • You should also make sure all incremental backups made after the oldest of the remaining backups are retained.
    • This basically gets reduced to deleting only those backups that are old enough, based on your full/incremental schedule.

More Specific Issues:

  • Does the accounts.xml dependency cause issues with this?
    • No. Just don't delete it.
  • What about the contents of the backup/tmp/ data or shared blobs type references?
    • Don't touch this directory either. It is used during backup and restore. You don't want to change its content while an operation is going on, so best to leave it alone.
  • What if a zimbra server is running some type of restore of backup command while the manual removal is running on the nfs server?
    • You shouldn't remove the backups that are being used in a restore currently underway. You are responsible for avoiding the race condition.
      • Please understand you are responsible for avoiding the race condition. Make sure no backup or restore is happening at the moment, then rename the directories that will be deleted, preferably move them to another subdirectory, e.g. /opt/zimbra/backup/sessions_to_delete. Then delete.

LDAP Backup Related Items

Backup Schedule On LDAP Only Server [non-Mailstore]

If you look at the code in /opt/zimbra/bin/zmschedulebackup  :

if ($BACKUP_MODE eq 'Standard') {
    # default schedule: full backup 1am every sunday, incr backup 1am every weekday
    # deletes backups older than a month at 12am everyday
  if (isLdapOnly()) {
      @default = ("0 1 * * 6 /opt/zimbra/bin/zmbackup $target\n",
            "0 0 * * * /opt/zimbra/bin/zmbackup -del 1m\n"); 
  } else { 
      @default = ("0 1 * * 6 /opt/zimbra/bin/zmbackup -f $account $target $compress \n",
          "0 1 * * 0-5 /opt/zimbra/bin/zmbackup -i $compress\n",
            "0 0 * * * /opt/zimbra/bin/zmbackup -del 1m\n");
  }
    %commandmap = ("f", "/opt/zimbra/bin/zmbackup -f $target $compress $account",
                   "i", "/opt/zimbra/bin/zmbackup -i $compress",
                   "d", "/opt/zimbra/bin/zmbackup -del");
} else {  # Auto-Grouped mode
    # default schedule: full backup 1am everyday, no incr backup
    # deletes backups older than a month at 12am everyday
    @default = ("0 1 * * 0-6 /opt/zimbra/bin/zmbackup -f $target $compress\n",
                "0 0 * * * /opt/zimbra/bin/zmbackup -del 1m\n");
    %commandmap = ("f", "/opt/zimbra/bin/zmbackup -f $target $compress",
                   "i", "/opt/zimbra/bin/zmbackup -i $compress",
                   "d", "/opt/zimbra/bin/zmbackup -del");
}

Notice the specific check and then format for ldapOnly:

  if (isLdapOnly()) {
      @default = ("0 1 * * 6 /opt/zimbra/bin/zmbackup $target\n",
            "0 0 * * * /opt/zimbra/bin/zmbackup -del 1m\n"); 

Related Bugs To LDAP Backups

Some bugs to be aware of - most are resolved/fixed:

A Way To Verify Backup Integrity

I filed an RFE for this:

Negative Seek Offset Error & RFE

Explanation of negative seek offset error during a restore attempt and manual fixes are located here:

Auto-Group Backups Rather Than Default Method Topics

General Description And Official References

Having trouble completing that entire full backup during off-hours? Enter the hybrid auto-grouped mode, which combines the concept of full and incremental backup functions - you’re completely backing up a target number of accounts daily rather than running incrementals.

Auto-grouped mode automatically pulls in the redologs since the last run so you get incremental backups of the remaining accounts; although the incremental accounts captured via the redologs are not listed specifically in the backup account list. This still allows you to do a point in time restore for any account.

Administrative manual page:

http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.17.01.html

Compare the sections called "Standard Backup Method" & "Auto-Grouped Backup Method"

http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.17.02.html

Configuration details are here on that page:

http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.17.08.html

Good explanation:

http://www.zimbrablog.com/blog/archives/2008/08/recent-admin-tidbits-part-1.html

Simply divide your total accounts by the number of groups you choose (zimbraBackupAutoGroupedNumGroups is 7 by default) and that’s how many will get a full backup session each night. Newly provisioned accounts, and accounts whose last backup is a specified number of days older are picked first. (zimbraBackupAutoGroupedInterval is defaulted to 1d)

Think of auto-grouped mode as a full backup for the scheduled group as well as an incremental (via redologs) for the all other accounts at the same time.

Bugs - RFE's To Review For Auto-Group

Please see:

Enabling Auto-Group For Backups - Schedule In Crontab

  • You'll need to make the variable changes as listed in the administrative guide and then the following for set the crontab correctly.
  • Run zmschedulebackup --help to see a list of options.
  • Run zmschedulebackup -D , which will now set a new default schedule (crontab) that uses auto-group settings now that your variable are set for it.
  • Incremental backups aren't performed as they were.
    • You'll see there's no longer the -i option in cron.
    • Incrementals are performed, in a manner, but the redologs are just copied into the full backup session

Two bugs to look at as well:

Standard Mode Default Cron Setup
  • Here is a schedule without auto-grouped backups enabled:
    • The Default Schedule For Normal Backups [ zmschedulebackup --help ]:
 f    0 1 * * 6
 i    0 1 * * 0-5
 d 1m 0 0 * * *
    • To set the backup to standard mode and use the default schedule one would:
      • zmprov mcf zimbraBackupMode Standard
      • zmschedulebackup -D
    • The crontab would look like:
 0 1 * * 6 /opt/zimbra/bin/zmbackup -f -a all   
 0 1 * * 0-5 /opt/zimbra/bin/zmbackup -i -a all
 0 0 * * * /opt/zimbra/bin/zmbackup -del 1m
Auto-Group Mode Cron Setup
  • Here is a schedule with auto-grouped enabled:
    • The Default Schedule For Auto-Group Backups [ zmschedulebackup --help ]:
f 0 1 * * 0-6
d 1m 0 0 * * *
    • To set the backup to auto-group mode and use the default schedule one would:
      • zmprov mcf zimbraBackupMode Auto-Grouped
      • zmschedulebackup -D
    • The crontab would look like:
0 1 * * 0-6 /opt/zimbra/bin/zmbackup -f  
0 0 * * * /opt/zimbra/bin/zmbackup -del 1m

Some Variables For Auto-Group

The below might not be complete or the defaults, I just wanted to save this before I forget them. Try to get more complete details on these later.

zmprov gacf | grep Backup
zimbraBackupAutoGroupedInterval: 1d
zimbraBackupAutoGroupedNumGroups: 7
zimbraBackupAutoGroupedThrottled: FALSE
zimbraBackupMode: Auto-Grouped

Auto-group And Redologs

Please see Ajcody-Backup-Restore-Issues#Redologs_And_Auto-group_In_Regards_To_Backups

Problems Switching To Auto-Groups Because It Wants To Run A Full Against All Accounts

Please see the following bug/rfe made about problems switching over to Auto-group when the first backup run of it tries to backup ALL of he accounts. I have the full how-to within the bug. It basically manipulates the last_backup_at for each account.

Backup And Deletion Schedule - zmschedulebackup

How To Adjust The Deletion Schedule

For my example below, I first set the backup to the "default" schedule. And I then adjust that "default" to have backups delete with a 14 day interval rather than 1 month.

[zimbra@mail3 ~]$ zmschedulebackup -D
Default schedule set

Current Schedule:

        f 0 1 * * 6 -a all
        i 0 1 * * 0-5
        d 1m 0 0 * * *

[zimbra@mail3 ~]$ zmschedulebackup -q
Current Schedule:

        f 0 1 * * 6 -a all
        i 0 1 * * 0-5
        d 1m 0 0 * * *

[zimbra@mail3 ~]$ zmschedulebackup -R f "0 1 * * 6" i "0 1 * * 0-5" d 14d "0 0 * * *"
Schedule replaced

Current Schedule:

        f 0 1 * * 6 -a all
        i 0 1 * * 0-5
        d 14d 0 0 * * *

[zimbra@mail3 ~]$ zmschedulebackup -q
Current Schedule:

        f 0 1 * * 6 -a all
        i 0 1 * * 0-5
        d 14d 0 0 * * *

[zimbra@mail3 ~]$ crontab -l | grep backup
0 1 * * 6 /opt/zimbra/bin/zmbackup -f   -a all
0 1 * * 0-5 /opt/zimbra/bin/zmbackup -i  
0 0 * * * /opt/zimbra/bin/zmbackup -del 14d

The Zip - Compression Option For Backups

Using the zip option will compress all those thousands of single files that exist under a user's backup, decreasing performance issues that arise from writing out thousands of small files as compared to large ones. This is often seen when one is :

  • Using nfs for the backup directory
  • Copying/rsyncing backups to a remote server
  • Are using some third party backup software (to tape) to archive/backup the zimbra backup sessions.

Optional Tweaks To The Zip Options

Please see this comment and those underneath it within this RFE:

  • "Use zip files for shared blobs of a full backup made with --zip option"
    • http://bugzilla.zimbra.com/show_bug.cgi?id=26624#c6
      • Use zmlocalconfig. To set:
        • $ zmlocalconfig -e key=val
      • To unset:
        • $ zmlocalconfig -u key
      • Once you set the key you will be able to view it.
        • $ zmlocalconfig
      • backup_zip_copier_private_blob_zips
        • How many zip files to distribute a mailbox's private (unshared) blobs over; default 4 (blobs-1.zip through blobs-4.zip); range 1 to 10,000
      • backup_zip_copier_copy_buffer_size
        • File copy buffer size; default 16384 (16KB); range 4KB to 1MB
      • backup_zip_copier_queue_capacity
        • Each zip file gets a queue of files to add. This key sets the queue size. Default is 10. Range is 1 to 10,000.
      • backup_zip_copier_deflate_level
        • Compression level. Default is -1. (same as in java.util.zip.ZipOutputStream). -1 is same as level 6. This behavior comes from zlib library which the JVM uses to implement zip. Other than the special default value, the level can range from 0 to 9. 0 means no compression. 1 means fastest compression and 9 means best compression.
      • backup_disable_shared_blobs
        • This one isn't limited to zip backups. When this is set to true, all blobs are backed up as private backups. Default is false.
      • backup_debug_use_old_zip_format
        • If true, backup will behave like ZCS 5.0.4 and earlier. Shared blobs are never zipped, and private blobs are added to a single blobs.zip file in zip backup. Default is false.

Need To Write Fewer Files - Add The Zip Option To Your Backup Commands

RFE to make zip option the default for backups:

There is very little details in the official documentation on this option unfortunately. This does have a really good explanation though:

http://www.zimbrablog.com/blog/archives/2008/08/recent-admin-tidbits-part-1.html

From the administrative manual on the Backup section:

http://www.zimbra.com/docs/ne/latest/administration_guide/10_Backup_Restore.15.1.html

It says,

"-zip can be added to the command line to zip the message files during backup. Zipping these can save backup storage space."

It's implied that instead of having all the individual message files in the backup that it will bunch them all together into zip files. The body of a shared blob is added once to a shared-blobs zip file, then a small pointer-only entry is added to a mailbox's zip file. Same effect as in non-zipped case. This will be useful when the number of message files is causing disk i/o issues. Maybe your trying to rsync the backup session directories off to another server or your running a third party backup on it to save to tape. The default use of -zip will use compression, if this also causes overhead that you need to avoid you can use the -zipStore option.

Note about -zipStore:

"when used with the -zip option, it allows the backup to write fewer files (-zip), but not incur the compression overhead as well"

The zip options effect backups that are in blob formats (full's). Incremental backups are bascially redologs, not the full message store of the user. In summary, the zip option will not impact the increment type backups. Auto-group backups are a mixture of both fulls and incrementals.

How To Use As A Default Option?

You'll add the options to the zimbra crontab file. This can be done with the zmschedulebackup command.

Run zmschedulebackup with help option:

zmschedulebackup --help

You'll see:

-z: compress - compress email blobs with zip

It appears that you'll need to manually add the options about -zipStore , if you want it, to the crontab file.

See bug :

http://bugzilla.zimbra.com/show_bug.cgi?id=30981

What Does It Look Like When I Use Zip?

Shared blobs are zipped and blobs (messages) are zipped per root store directory.

mail:~/backup/sessions/full-20080820.160003.770/accounts/115/988/11598896-a89b-4b9d-bedb-1ed1afcb6c87/blobs 
zimbra$ ls blobs-1.zip    blobs-2.zip    blobs-3.zip    blobs-4.zip

General Backup & Restore Debugging

You'll be monitoring the /opt/zimbra/log/mailbox.log file

Include the -d / --debug option on the CLI for either zmrestore or zmbackup .

To increasing logging for backup/restore-related logs - /opt/zimbra/log/mailbox.log . Enable DEBUG log level for "zimbra.backup" logger in :

  • /opt/zimbra/conf/log4j.properties for "temporary" change - until next restart. This could take a couple of minutes before jetty "sees" the changes.
  • /opt/zimbra/conf/log4.properties.in for "permament" change that will stick after restart. A restart of jetty/mailbox would be required for this change - zmmailboxctl restart .
   log4j.logger.zimbra.backup=DEBUG


For incremental backups, this should log each redolog being copied to the backup and also log which ones will be deleted out of archive directory. Those not deleted are kept because they are newer than 1 hour (default). The kept logs are deleted (but not copied again) during the next incremental backup.

Redolog Files


Redologs Copied To Backup Session And When Deleted

Archived logs that are less than an hour old at the time of incremental backup are copied to the backup but aren't deleted to support post-crash waitset reinitialization mechanism. The interval is set in localconfig key backup_archived_redolog_keep_time, which is in seconds, default=3600.

An Outline Of The Step
  1. /opt/zimbra/redolog/redo.log (starts and then grows to zimbraRedoLogRolloverFileSizeKB size - default 100MB)
  2. This flushes to /opt/zimbra/redolog/archive/[file] upon hitting the zimbraRedoLogRolloverFileSizeKB.
  3. 1 & 2 keep repeating when zimbraRedoLogRolloverFileSizeKB is hit.
  4. When a backup is done, the archive/* files are copied. The redo.log file is not moved.
    1. When the backup processes archive/* logs, it first figures out the last sequence copied to backup. All newer logs are copied to the current backup. Then, all logs are deleted except those that are too new, determined by localconfig parameter backup_archived_redolog_keep_time, which defaults to 1 hour. (This is part of the waitset feature.)
      • In standard backup mode, only incremental backups move the redologs.
      • In auto-grouped mode, every backup is a hybrid of full and incremental and thus redologs are moved.
Redologs And Auto-group In Regards To Backups

http://www.zimbra.com/forums/administrators/21360-auto-grouped-backups.html

Think of auto-grouped mode as a full backup for the scheduled group as well as an incremental (via redologs) for the all other accounts at the same time. Auto-grouped mode automatically pulls in the redologs since the last run, so you get incremental backups of the remaining accounts. Although the incremental accounts captured via the redologs are not listed specifically in the backup account list. This still allows you to do a point in time restore for any account.

If You Have Older Redologs Not Being Deleted

According to the code, only archived logs newer than 1 hour old (default for backup_archived_redolog_keep_time) should remain after an incremental backup. It is a bug if you are seeing older logs sticking around. If so, look at mailbox.log and see if any error was logged. If you enable DEBUG logging for "zimbra.backup" logger in log4j.properties you will see log statements for each copy and deletion.

The zimbraRedoLogDeleteOnRollover variable

zimbraRedoLogDeleteOnRollover shouldn't have an effect on "If you have older redologs not being deleted". By default it's FALSE and affects whether or not stuff makes it into /opt/zimbra/redolog/archive at all. With it set to TRUE there's just /opt/zimbra/redolog/redo.log and it's deleted/not rolled over into archive. As discussed above old redologs are deleted after the incremental; thus if you don't take incremental backups you should set this value to TRUE or periodically script manual deletion of /opt/zimbra/redolog/archive. (And with zimbraRedoLogEnabled FALSE there's no redo.log at all.)

If You Don't Run Incremental Backups Or Don't Need Archive Redologs

You would set zimbraRedoLogDeleteOnRollover to TRUE.

(Auto-Grouped backups you can still leave this to the default of FALSE.)

Redolog Sequence And The Backup Session

Redologs will exist in the incremental backup sessions. The zmbackupquery command will reference the redologs associated with the backup. For example"

[zimbra@mail3 ~]$ ls /opt/zimbra/backup/sessions/incr-20080925.224528.230/redologs/
redo-20080925.213136.165-seq53.log  redo-20080925.220726.521-seq55.log  redo-20080925.224209.287-seq57.log
redo-20080925.215516.450-seq54.log  redo-20080925.221749.133-seq56.log

[zimbra@mail3 ~]$ zmbackupquery -lb incr-20080925.224528.230
Label:   incr-20080925.224528.230
Type:    incremental
Status:  completed
Started: Thu, 2008/09/25 18:45:28.230 EDT
Ended:   Thu, 2008/09/25 18:45:39.099 EDT
Redo log sequence range: 53 .. 57
Number of accounts: 2

In the above example, we see the sequence range of "53 .. 57" is referring to the files in the backup session directory called redologs.


RedoLog Variables

Changing Redolog File Size And Location

The /opt/zimbra/redolog/redo.log rolls over when it reaches zimbraRedoLogRolloverFileSizeKB (by default 100mb).

The "roll overs" then goto /opt/zimbra/redolog/archive/

zmprov gacf | grep Redo
zimbraRedoLogArchiveDir: redolog/archive
zimbraRedoLogDeleteOnRollover: FALSE
zimbraRedoLogEnabled: TRUE
zimbraRedoLogFsyncIntervalMS: 10
zimbraRedoLogLogPath: redolog/redo.log
zimbraRedoLogRolloverFileSizeKB: 102400
Need To Move Redologs Because Partition Getting Full

Let's say you have a partition getting full and you need to move the redolog to another partition or nfs mount temporary to deal with the potential crisis that will happen when the partition becomes full. You'll need to reallocate the complete redolog/ directory and the archive subdirectory to the same partition because the roll over from redo.log to the archive directory happens with a rename function within the java code. This will require downtime since you'll need to move the actual redo.log file and zimbra can't be running while you do this. You can use a symlink to your new partition path. For example:

su - zimbra
zmcontrol stop
su -
 ** as root
mkdir /data/redolog
chown zimbra:zimbra /data/redolog
mount /dev/sdb1 /data/redolog
mv /opt/zimbra/redolog/* /data/redolog/
 ** or use rsync
rmdir /opt/zimbra/redolog
ln -s /opt/zimbra/redolog /data/redolog
ls -laR /data/redolog
** confirm ownership is with zimbra and double check zimbra can write in this directory
su - zimbra
touch /data/redolog/testfile
rm /data/redolog/testfile
zmcontrol start
Automatic Deleting Of Redo Logs On Rollover

This variable (zimbraRedoLogDeleteOnRollover) is set TRUE or FALSE.

zmprov gacf | grep zimbraRedoLogDeleteOnRollover

To modify it

zmprov mcf zimbraRedoLogDeleteOnRollover TRUE

Want To See What's In Redolog Files

This is for older versions of ZCS - newer versions should use zmredodump if it's available.

If you suspect there's too much redolog activity during a time window or have another need to inspect the contents of the redolog, dump it and examine it:

$ zmjava com.zimbra.cs.redolog.util.RedoLogVerify /opt/zimbra/redolog/redo.log > out.file

Pick the right redolog file, either redo.log or one of the files under archive/, based on timestamp.

zmplayredo And zmredodump

zmplayredo - Replaying Content From Any Redolog File

zmplayredo is a newer command, first introduced in 5.0.5 I believe. The mailbox has to be stop to run zmplayredo.

The help output from 6.0.8:

$ zmplayredo --help
usage: zmplayredo <options>
    --fromSeq <arg>         Replay from this redolog sequence (inclusive)
    --fromTime <arg>        Replay from this time (inclusive)
 -h,--help                  Show help (this output)
    --logfiles <arg>        Replay these logfiles, in order
    --mailboxId <arg>       Replay for this mailbox only
    --queueCapacity <arg>   Queue capacity per player thread; default=100
    --stopOnError           Stop replay on any error
    --threads <arg>         Number of parallel redo threads; default=50
    --toSeq <arg>           Replay to this redolog sequence (inclusive)
    --toTime <arg>          Replay to this time (inclusive)

Specify date/time in one of these formats:

    2010/11/19 13:55:08
    2010/11/19 13:55:08 802
    2010/11/19 13:55:08.802
    2010/11/19-13:55:08-802
    2010/11/19-13:55:08
    20101119.135508.802
    20101119.135508
    20101119135508802
    20101119135508

Specify year, month, date, hour, minute, second, and optionally millisecond.
Month/date/hour/minute/second are 0-padded to 2 digits, millisecond to 3 digits.
Hour must be specified in 24-hour format, and time is in local time zone.
zmredodump - Replaying Content From Any Redolog File

zmredodump is a newer command and very useful. It does not require mailboxd to be stopped like zmplayredo does.

The help output from 6.0.8:

$ zmredodump --help
usage: zmredodump [options] <redolog file/directory> [...]
where [options] are:

 -h,--help        show this output
    --m <arg>     one or more mailbox ids separated by comma or white
                  space.  The entire list must be quoted if using space as separator.  If
                  this option is given, only redo ops for the specified mailboxes are
                  dumped.  Omit this option to dump redo ops for all mailboxes.
    --no-offset   don't show file offsets and size for each redo op
 -q,--quiet       quiet mode.  Only print the log filename and any errors.
                  This option can be used to verify the integrity of redologs with minimal
                  output.
    --show-blob   show blob content.  Item's blob is printed, surrounded
                  by <START OF BLOB> and <END OF BLOB> markers.  The last newline before end
                  marker is not part of the blob.

Multiple log files/directories can be specified.  For each directory, all
redolog files directly under it are processed, sorted in ascending redolog
sequence order.
Using zmredodump To Get Message Blobs To Inject With zmlmtpinject - RFE

Please see:

Expand Ability To Parse Redologs - RFE

Please see:

Getting A Sequence or Time Variable For Restore Or Replay

You can see the changes within the redo logs with the command below. You can point it to any redolog.

zmjava com.zimbra.cs.redolog.util.RedoLogVerify /opt/zimbra/redolog/redo.log > out.file

You'll get output like this:

VERIFYING: redo.log
HEADER
------
sequence: 59
open: 1
filesize: 512
serverId: d5c5d6a7-b82f-4c29-b0cd-91818057196b
firstOpTstamp: 1222385426273
lastOpTstamp:  1222385426273
version: 1.22

------
txn 1222383600.1 [PurgeOldMessages] ver=1.22, tstamp=1222385426273, change=20200, mailbox=1
txn 1222383600.1 [CommitTxn] ver=1.22, tstamp=1222385426329, mailbox=1, txnType=PurgeOldMessages
txn 1222383600.2 [PurgeOldMessages] ver=1.22, tstamp=1222385486337, change=13500, mailbox=3
txn 1222383600.2 [CommitTxn] ver=1.22, tstamp=1222385486351, mailbox=3, txnType=PurgeOldMessages
txn 1222383600.3 [PurgeOldMessages] ver=1.22, tstamp=1222385546357, change=20201, mailbox=1
txn 1222383600.3 [CommitTxn] ver=1.22, tstamp=1222385546383, mailbox=1, txnType=PurgeOldMessages
txn 1222383600.4 [PurgeOldMessages] ver=1.22, tstamp=1222385606391, change=13501, mailbox=3
txn 1222383600.4 [CommitTxn] ver=1.22, tstamp=1222385606404, mailbox=3, txnType=PurgeOldMessages
txn 1222383600.5 [PurgeOldMessages] ver=1.22, tstamp=1222385666416, change=20202, mailbox=1
txn 1222383600.5 [CommitTxn] ver=1.22, tstamp=1222385666428, mailbox=1, txnType=PurgeOldMessages
txn 1222383600.6 [PurgeOldMessages] ver=1.22, tstamp=1222385726435, change=13502, mailbox=3
txn 1222383600.6 [CommitTxn] ver=1.22, tstamp=1222385726459, mailbox=3, txnType=PurgeOldMessages
txn 1222383600.7 [PurgeOldMessages] ver=1.22, tstamp=1222385786476, change=20203, mailbox=1
txn 1222383600.7 [CommitTxn] ver=1.22, tstamp=1222385786486, mailbox=1, txnType=PurgeOldMessages
txn 1222383600.8 [PurgeOldMessages] ver=1.22, tstamp=1222385846493, change=13503, mailbox=3
txn 1222383600.8 [CommitTxn] ver=1.22, tstamp=1222385846506, mailbox=3, txnType=PurgeOldMessages
txn 1222383600.9 [PurgeOldMessages] ver=1.22, tstamp=1222385906739, change=20204, mailbox=1
txn 1222383600.9 [CommitTxn] ver=1.22, tstamp=1222385906775, mailbox=1, txnType=PurgeOldMessages
txn 1222383600.10 [PurgeOldMessages] ver=1.22, tstamp=1222385966944, change=13504, mailbox=3
txn 1222383600.10 [CommitTxn] ver=1.22, tstamp=1222385966963, mailbox=3, txnType=PurgeOldMessages
txn 1222383600.11 [PurgeOldMessages] ver=1.22, tstamp=1222386026972, change=20205, mailbox=1
txn 1222383600.11 [CommitTxn] ver=1.22, tstamp=1222386026990, mailbox=1, txnType=PurgeOldMessages
...

How Do I Figure Out Which Sequence or Time Variable To Use For Restore Or Replay

In 5.0.10+ we'll have a CLI wrapper (zmredodump) with a slightly different command line syntax, but the below long syntax works in earlier versions.

To locate the correct restore-to time, you have to start with an approximate time the message was added/deleted. Look at the redolog files. The filename contains the GMT time when the file was rolled over, which is roughly the tstamp of the last operation in the file. If your time data is accurate you can find the specific file. Or you have a range of files to examine.

Use the redolog verify tool to dump the contents into text form, the -m / --message option to show message body data:

zmjava com.zimbra.cs.redolog.util.RedoLogVerify -m <filename or directory> ... > out.file

If the message was deleted and you don't know the id, you must go by some other clue such as the subject. Search the file to locate your message. You can cut/paste the message and lmtp-inject it to recover the message. No need to go through with a restore if this is all you needed.

Are You Messages Really Gone - Things To Check If zmplayredo Isn't Doing What You Expect

Here's something I found out testing zmplayredo for a customer case. Testing on a ZCS 6.0.8 single ZCS server.

Created a test account and sent it one message that is in the Inbox. I delete the msg in zwc but don't purge the Trash - msg is in Trash now.

Log events of above action:

2010-10-27 15:07:13,375 INFO [btpool0-3://192.168.0.71/service/soap/ConvActionRequest] 
[name=ajcody@mail71.DOMAIN.com;mid=17;ip=192.168.0.17;ua=ZimbraWebClient 
- FF3.0 (Linux)/6.0.8_GA_2661;] mailop - Moving VirtualConversation (id=-257) to 
Folder Trash (id=3). Affected message ids: 257.

Stop mailboxd so I can use zmplayredo and then start mailboxd back up after zmplayredo is finished :

 zmmailboxdctl stop
 zmplayredo --mailboxId 17 --logfiles /opt/zimbra/redolog/archive/* --fromTime 20101027090000 --toTime 20101027140000
 zmmailboxdctl start

Log event for above:

2010-10-27 15:07:50,383 INFO [main] [] mailop - Adding Message: id=257, Message-ID=
<1604937630.920784.1288201208382.JavaMail.root@corp.zimbra.com>, parentId=-1, folderId=2, folderName=Inbox.
2010-10-27 15:07:50,404 INFO [main] [] RedoableOp - Message 257 is already in mailbox 17

Log into ZWC with the test account. The msg is not in the Inbox, but it's still in Trash folder. I purge it from Trash.

Log event for deletion.

2010-10-27 15:09:38,761 INFO [btpool0-2://192.168.0.71/service/soap/ConvActionRequest]
[name=ajcody@mail71.DOMAIN.com;mid=17;ip=192.168.0.17;ua=ZimbraWebClient 
- FF3.0 (Linux)/6.0.8_GA_2661;] mailop - Deleting Message (id=257).

Then I redo the stop/start of mailboxd and zmplayredo again.

 zmmailboxdctl stop 
 zmplayredo --mailboxId 17 --logfiles /opt/zimbra/redolog/archive/* --fromTime 20101027090000 --toTime 20101027140000 
 zmmailboxdctl start 

Log event for the above:

2010-10-27 15:10:29,192 INFO [main] [] mailop - Adding Message: id=257, Message-ID=
<1604937630.920784.1288201208382.JavaMail.root@corp.zimbra.com>, parentId=-1, 
folderId=2, folderName=Inbox.

Log back into ZWC with test account and now I can confirm that the msg is not in Trash and it is now showing in Inbox.

Gap In Redo Log

The error message from either a backup or restore command:

"Error occurred: Found gap in redo log sequence; missing 5965 through 6149;
To avoid future restore problems, discard all existing backups and take a
full backup of all accounts; If this error occurred during restore,
try the --ignoreRedoErrors option"

The output is pretty accurate in how to handle the situation.

If you get the error during a backup, the recommendation is to move your old backups out. The directories in /opt/zimbra/backup/sessions/* . You'll want to keep them around just in case and then proceed to do a full backup.

If you get the error during a restore, you would add the flag --ignoreRedoErrors to your restore command.

Another possible related issue is if your /tmp or /opt/zimbra/redolog/ is filling up.

Error Executing redoOp

Errors with restores that involve the message 'error executing redoOp' will not show up in the admin console but will when you attempt the restore from CLI. This can also be the cause when you use the RestoreToTime option from the admin console and it doesn't seem to work correctly - the restore stopping prematurely from the specified date/time.

I created the following RFE in regards to the admin console issue:

  • "Include --ignoreRedoErrors option and error feedback in Admin Console for restore"
    • http://bugzilla.zimbra.com/show_bug.cgi?id=52358
      • This could explain why your restore to time isn't working in the Admin Console but does from CLI when you see an error about redologs and then reattempt restore with the --ignoreRedoErrors and it works.

Another RFE that was made but marked as 'WONTFIX' that gives a background story to the issue is:

When you hit this error with your backup data during restore attempts, there's basically only a couple of options to handle this that are recommended by support:

  1. Try to get full backups of your accounts or the accounts in question and then test against them and the preceding incrementals after the full.
  2. Attempt restores via the CLI using the additional option of : --ignoreRedoErrors
  3. Do your restore against the latest full backup of the account in question and then use the zmplayredo command against the redologs in the incrementals and/or the /opt/zimbra/redologs/* directory . This will give you more control to walk the restored account up to the point in time you want it at. One should really read through the whole section above, Ajcody-Backup-Restore-Issues#Redolog_Files , to understand the whole concept of redologs and then the use of zmplayredo.

Generally, "fixing" the redolog itself is not an option.

Why Do My Fulls Not Report All Accounts?

Are you sure it was a full backup that was ran or just a full session that was generated from your incremental backup job? When an incremental is ran, it will create a "full" session for any new accounts it discovers after the last actual full backup job.

For example, here's a full session that was created by an incremental backup job:

Label: full-20081010.060126.559
Type: full
Status: completed
Started: Fri, 2008/10/10 01:01:26.559 CDT
Ended: Fri, 2008/10/10 01:01:28.988 CDT
Redo log sequence range: 705 .. 705
Number of accounts: 1

Label: incr-20081010.060009.420
Type: incremental
Status: completed
Started: Fri, 2008/10/10 01:00:09.420 CDT
Ended: Fri, 2008/10/10 01:01:26.413 CDT
Redo log sequence range: 700 .. 704
Number of accounts: 392


[zimbra@servername ~]$ du -sh /opt/zimbra/backup/sessions/full-20081010.060126.559
1.2M /opt/zimbra/backup/sessions/full-20081010.060126.559
[zimbra@servername ~]$ du -sh /opt/zimbra/backup/sessions/incr-20081003.060010.622
452M /opt/zimbra/backup/sessions/incr-20081003.060010.622

Notice the Start and End times, this will show that the full is related to the incremental job.

You'll want to run zmbackupquery against your full labels to see your "main" full backup session - assuming you can't simply guess based upon the cron entry for it [ su - zimbra ; crontab -l | grep backup ]. For example, to see all your fulls from today's date back to October 01, 2008 and the accounts within each session - you would do:

zmbackupquery -v --from 20081001.000000 --type full

The -v flag outputs the accounts, the --from uses YearMonthDay.HourMinuteSecond , and the --type can be full or incremental. To just see one particular sessions date, you would use the lb [label] flag:

zmbackupquery -v -lb [your full label, ex. full-20081001.000000]

Issues After /opt/zimbra/backup Became Full

Bugs/RFE's on this issue:

See alsoAjcody-Backup-Restore-Issues#Possible_Issue_That_A_Failed_Or_Interrupted_Backup_Causes and the Bugs & RFE's under that section.

You can run into numerous issues if you allow your backup directory to become full.

Confirm your /opt/zimbra/backup/accounts.xml is being updated after a backup. You might see that the newer account.xml* file is accounts.xml.new . This is a sign of problems.

Confirm that the files in /opt/zimbra/backup/tmp/* don't have 0 byte lengths. There might be files like 1.xml and 3.xml in there. If they show 0 bytes, you need to remove them. The backup/restore commands if the file exist and they are empty. Your errors might look like this:

[zimbra@mailb ~]$ zmrestore -a USER@DOMAIN -ca -pre restore_
Error occurred: system failure: Unable to parse XML file /opt/zimbra/backup/tmp/restore/shared_blobs/1.xml

If you tried doing restores (redirected -ca -pre) before to clear up the above issues, you might find you can't do a successful restore AND you can't delete the account afterwards.

If you get errors like :

zmprov da restore3_tester3@XXXXXX.edu
ERROR: service.FAILURE (system failure: writing new mailbox row for account 89e7d9f4-013e-4cf1-a352-7b2f0a00d5af)

zmprov gmi restored_USER
ERROR: service.FAILURE (system failure: writing new mailbox row for account 56a7f654-f85b-45cc-931a-81d9bb9076bf)

You'll need to delete the account via a ldapdelete command.

Query And Stopping A Backup or Restore In Progress

Please see the url below for Backup In Progress - topic name "Aborting Full Backup In Progress":

  • A matter of using zmbackupquery & zmbackupabort .

Please see url below for Restore In Progress - topic name "Stopping a Restore Process" :

A -del Delete In Progress

Please see the bug / rfe's I filed

zmbackupabort syntax

From ZCS 6.0.8

$ zmbackupabort -h
usage: zmbackupabort <options>
 -d,--debug          Display diagnostics for debugging purposes.
 -h,--help           Displays this help message.
 -lb,--label <arg>   Label of full backup set to abort.
 -r,--restore        Abort the restore in progress.
 -s,--server <arg>   Mail server hostname. Default is localhost.
 -t,--target <arg>   Backup target location (default
                     <zimbra_home>/backup).

zmbackupquery syntax

From ZCS 6.0.8

$ zmbackupquery -h
usage: zmbackupquery <options>
 -a,--account <arg>   Account email addresses seperated by white space or
                      "all" for all accounts.
 -d,--debug           Display diagnostics for debugging purposes.
    --from <arg>      List backups whose start date/time is at or after
                      this date/time.
 -h,--help            Displays this help message.
 -lb,--label <arg>    The label of full backup to query.
 -s,--server <arg>    Mail server hostname. Default is localhost.
 -t,--target <arg>    Backup target location (default
                      <zimbra_home>/backup).
    --to <arg>        List backups whose start date/time is at or before
                      this date/time.
    --type <arg>      Backup set type to query.  "full" or "incremental";
                      both if unspecified.
 -v,--verbose         Show account list in each backup.

Specify date/time in one of these formats:

    2010/11/19 14:06:22
    2010/11/19 14:06:22 923
    2010/11/19 14:06:22.923
    2010/11/19-14:06:22-923
    2010/11/19-14:06:22
    20101119.140622.923
    20101119.140622
    20101119140622923
    20101119140622

Specify year, month, date, hour, minute, second, and optionally millisecond.
Month/date/hour/minute/second are 0-padded to 2 digits, millisecond to 3 digits.
Hour must be specified in 24-hour format, and time is in local time zone.

Restore To Time Problems

There's seems to be some syntax issues when using this variable. Please review the following to confirm your syntax.

http://wiki.zimbra.com/index.php?title=CLI_zmrestore_restoreToTime_Network_Edition_only

The gist, you MUST use the -lb full-200xxxxxx option when your trying to restore anything that ISN'T meant to include the latest information for the mailbox. The -lb argument should specify a full backup that took place prior to the time of the backup you wish to restore.

Find Out What Backup Session Labels You Need First

To find out what backups are associated with a particular account, you would do the following :

zmbackupquery -a user@domain

You'll want to note what is the first full that occurs before the point in time you want to restore. And then the incremental that follows right after your point in time.

Backup label (-lb) for fulls can be found by doing [include the -v option if you want to see a listing of the user accounts within the backups] :

zmbackupquery --type full

Backup labels (-restoreToIncrLabel) for incrementals can be found by:

zmbackupquery --type incremental

Command Syntax Example For Restores On The CLI

Example to full label and stop :

zmrestore -a USER@DOMAIN.com -lb full-20080726.050017.306 -rf -ca -pre restore_

Example to incremental label and stop :

zmrestore -a USER@DOMAIN.com -restoreToIncrLabel incr-20080731.060007.644 -lb full-20080726.050017.306 -br -ca -pre restore_

Example to specific time and stop :

zmrestore -a USER@DOMAIN.com -restoreToTime 20080801011800 -lb full-20080726.050017.306 -br -ca -pre restore_


Important Options You Might Want Or Need To Include

--ignoreRedoErrors : If you attempt a restore and you see an error about problems related to playing the redolog, you'll want to run the restore command again and include this option.


--skipDeletes : Please see http://bugzilla.zimbra.com/show_bug.cgi?id=31824#c5 for details on this.


-t /path/to/backup_dir : If you are restoring from another backup directory besides your current default path.


Variables that are asking for TIME rather than LABELS should follow this syntax (from zmrestore --help):

Specify date/time in one of these formats:

    2008/08/06 09:55:50
    2008/08/06 09:55:50 572
    2008/08/06 09:55:50.572
    2008/08/06-09:55:50-572
    2008/08/06-09:55:50
    20080806.095550.572
    20080806.095550
    20080806095550572
    20080806095550

Specify year, month, date, hour, minute, second, and optionally millisecond.
Month/date/hour/minute/second are 0-padded to 2 digits, millisecond to 3 digits.
Hour must be specified in 24-hour format, and time is in local time zone.

Bugs And RFE's To Review

Update - Bug/RFE's I Filed Against ZCS 8.6

Older Bug/RFE's

Restore An Individual Message

The zmrestore command is at a mailbox level.

An RFE was filed already to expand this. It is currently targeted for the Helix release.

A way around the current limitations would be to use lmtpinject. Please see the following for details on that:

The difficultly would be determining the message your trying to find within your backups that was "deleted" in prod.

User Deleted A Bunch Of Data And Notified You Hours Later Wanting It Restored

  1. To determine the time of the delete, use the zmredodump command.
    • You'll use this "time" for the restore command.
    • Example:
      • $ zmprov gmi USER@DOMAIN.com  ::: gives me mailboxid 17
      • $ zmredodump --show-blob --m 17 /opt/zimbra/redolog/ | grep Delete  ::: returns:
        • [0000f311 - 0000f350: 64 bytes; tstamp: 2010/11/19 11:21:30.852 CST] txn 1290033504.2544 [DeleteItem] ver=1.28, tstamp=1290187290852, change=3913, mailbox=17, ids=[304, 308], type=5
        • [0000f351 - 0000f382: 50 bytes; tstamp: 2010/11/19 11:21:30.869 CST] txn 1290033504.2544 [CommitTxn] ver=1.28, tstamp=1290187290869, mailbox=17, txnType=DeleteItem
  2. See Ajcody-Backup-Restore-Issues#Restore_To_Time_Problems about setting up the proper restore syntax.
    • You'll want to include the -ca -pre restore_ options since we'll first get the old data into a temporary account.
    • Remember to use zmbackupquery -a USER@DOMAIN.com --type full to find what full label will be needed for the restore.
    • Also, if you need/want the incremental label, do a zmbackupquery -a USER@DOMAIN.com --type incremental .
      • Example:
        • $ zmbackupquery -a USER@DOMAIN.com --type full  ::: shows the latest as being:
          • Label: full-20101119.070018.607
          • Type: full
          • Started: Fri, 2010/11/19 01:00:18.607 CST
          • Ended: Fri, 2010/11/19 01:00:29.751 CST
          • Acct ID: f33d6daf-8875-4496-8bee-6df345f295e7
    • Do the restore now with the proper information for the variable flags.
      • Note: You might need to include -br , -rf , or neither depending on the time frames involved.
      • Example:
        • $ zmrestore -a USER@DOMAIN.com -restoreToTime 20101119112000 -lb full-20101119.070018.607 -ca -pre restore_
      • It would be best to log into the restore_USER@DOMAIN.com account to confirm the data is as you expect it.
  3. Use zmmailbox with the getRestUrl option against the "restore_USER" account now to export the data.
    • Examples:
      • Export ALL the 'old' data from the restored account
        • $ /opt/zimbra/bin/zmmailbox -z -m restore_USER@DOMAIN.com getRestURL "//?fmt=tgz" > /var/tmp/user-export.tgz
      • You can also export SOME of the 'old' data from the restore account using other options. One option is with the before and after variables. NOTE - We have to set the query string as a variable to get around some of the shell issues.
      • For example:
        • $ query='before:11/20/2010 after:11/1/2010'
          • ZCS5 might require you to have a %20 rather than the actual space character.
        • $ /opt/zimbra/bin/zmmailbox -z -m restore_USER@DOMAIN.com getRestURL "//?fmt=tgz&query=$query" > /var/tmp/user-export.tgz
          • Note': the $query above will be the expansion of the variable you set prior in the shell, query='before:11/20/2010 after:11/1/2010 .
  4. Use zmmailbox with the postRestUrl option to IMPORT the RESTORE_USER account into the "USER" account now with appropriate options.
    • Examples:
      • $ /opt/zimbra/bin/zmmailbox -z -m USER@DOMAIN.com postRestURL “//?fmt=tgz&resolve=replace” /var/tmp/user-export.tgz
    • Note : A critical option in the above command is the &resolve=replace one. There are various ways you can handle the importing of data. Please review the following to determine what is best for you needs.

Restore Deleted Items - skipDeletes Option - ZCS6+

From the RFE comment:

        Added new option --skipDeletes to zmrestore.  If specified, skip 
        over delete operations during redo replay.  Delete ops are:

        DeleteItem (hard delete)
        DeleteMailbox
        EmptyFolder
        MoveItem, if moving item to Trash folder
        PurgeImapDeleted
        PurgeOldMessages

        Skipping these deletes can lead to other problems later, such as 
        conflicting paths, but it is assumed the priority is recovering as 
        much data as possible when using this option.

Restore Account Not Yet In Backups

Please see:

How To Restore To Events Within The Same Day

Work in progress and investigation.

Users Trash Items

User Ability To Recover Trash Purge

Please see RFE:

Retention Policy About Purges

Please see Mailbox_Purge

Can't Restore Or Find An Account That Was Renamed

When an account is "renamed", the old account name will no longer be "found" is your "default" type restore or backup queries. This can cause some confusions when one needs to restore to a time frame of when the account was under it's older name.

Restoring From CLI

First, identify what sessions labels hold the differing account names:

zmbackupquery -a USER@DOMAIN
zmbackupquery -a USER-RENAMED@DOMAIN

You should know have what you need to do follow the steps in the Ajcody-Backup-Restore-Issues#Restore_To_Time_Problems section.

Restoring From Admin Console GUI

I'll give a detail explanation of the situation when working around restores of renamed accounts in the admin console [web GUI]:

  1. If you in the GUI goto the "Restore" button, it first asks for an account rather than giving an option for date/time/session. I think you already stated, that "renamed" accounts don't show up in this query window. Therefore, one wouldn't really progress to the next window that would allow you to change the backup session label.
  2. They way you get around this is, you actually double click on the full session listing that you see on the backup page in the admin UI. This will bring you to another page, that is specific to that session. In there, you should see the old account name prior to the rename. You can then highlight that account listing and click on the "Restore" button. This will bring up the restore dialog, which will now have the date/time/session label auto-filled out.

Quota Is Stopping A Redirected Restore

Update

Reasoning for need, maybe the msg files coming from the restore are no longer "shared message blobs" and therefore increase the mailbox to a size it wasn't in the past. Changes to HSM maybe?

I think I'll need to create a RFE about adding an option to the zmrestore command to also include an option to set the COS value on a created account. Until then... Create a new COS and set it up to NOT have any quota. Once you kick off the backup and you see the account is created you can then apply the COS to that account. Call the cos something like no-quota. Here's the steps below.

To copy your cos (assuming your quota cause is the default one, change default to match your production cos your using).

To see all cos's

zmprov gac

Copies cos called default to cos named no-quota

zmprov cpc default no-quota

To remove quota to the new cos no-quota

zmprov mc no-quota zimbraMailQuota 0

To confirm

zmprov gc no-quota | grep -i quota

Now you would start your redirected restore and once you see the account is created, run the below in a separate shell. (example)

zmrestore -a USER@DOMAIN.com -restoreToTime 20080801011800 -lb full-20080726.050017.306 -br -ca -pre restore-

Once restore has kicked off - To apply the no-quota cos to the restored account:

zmprov sac restore-USER@DOMAIN.com no-quota

Restore Of Non-Account Items - Example - COS DL Etc

Cos and DL's are ldap entries basically.

From Backup and Restore

  • zmrestoreldap. This command restores the complete LDAP directory server, including accounts, domains, servers, COS and other data."

You'll see in a DR process, Network_Edition_Disaster_Recovery , that the zmrestoreldap is done before the zmrestoreoffline .

  • zmrestoreldap doesn't have options that allow specific items to be restored (COS, DL's, etc.). It only has option for named accounts (-a). One could try a ldapadd with a ldif of the COS or DL details. One could also take the information on the COS or DL within the ldap file in the backup session to at least have all the variables to manually add it back (via the zmprov command). Your looking at the backups on the LDAP master if your in a multi-server configuration.
/opt/zimbra/backup/sessions/full-xxxxxx/ldap/ldap.bak

Start of ldap entry example to search for:

  • Cos example
    • dn: cn=default,cn=cos,cn=zimbra
  • DL example
    • dn: uid=dl-group,ou=people,dc=mail,dc=domain,dc=com

To compare a current DL with past details, just save out the ldap entry from the backup to a txt file. And then do:

zmprov gdl maillist@domain.com

Make the necessary changes after comparing the two.

Restoring A Calendar (ics)

There seems to be a bug or odd expectation on how this command is currently working. If the appointment exists in the Calendar and the time is different with the same appointment in the ics file your importing - the time of the appointment will not change to the imported ics one. If you delete the event first, then the imported appointment will reflect the correct time.

Here's what I did to reproduce this situation. It seems this has been true for sometime, customer was on 4.5.11 and I was on 5.0.8

Created test account and made two appointments on friday - 9am and 4pm.
 Did a full backup.
 Restored test account to restore_user
 Ran :
     zmmailbox -z -m user@domain.com gru /Calendar > /tmp/calendarA.ics
     zmmailbox -z -m restore_user@domain.com gru /Calendar > /tmp/calendarB.ics
 And then
     diff /tmp/calendarA.ics /tmp/calendarB.ics [no differences]

 Now some tests.
 As user, I deleted the two appointments and then:
     zmmailbox -z -m user@domain.com pru /Calendar /tmp/calendarB.ics
 Refreshed Calendar as User in webclient.
     9am and 4pm appointment shows up.
 I then moved in the webclient the 9am appointment to 11am
 Did another restore:
     zmmailbox -z -m user@domain.com pru /Calendar /tmp/calendarB.ics
 Refreshed Calendar as User in webclient.
     11am and 4pm appointment shows up.
     ** The restore did not move the 11am appointment back to the 9am slot as in /tmp/calendarB.ics
     ** Assumption, this process will not over-write an appointment if it's there - it does not look to the time.
        Let's do a diff of the state of the calendar
        zmmailbox -z -m user@domain.com gru /Calendar > /tmp/calendarC.ics
        diff /tmp/calendarB.ics /tmp/calendarC.ics
          The DTSTAMP and SEQUENCE shows the difference in the time.
 If I delete the 11am appointment and then do the calendarB.ics restore the appointment shows up again at 9am.

I see this same behavior if I also use the web interface to export/import the calendar between the restored account and user one. Even when I import it into a NEW calendar, it even changes the two appointments to reflect the new calendar rather than the default one.

One Fix

One fix, if the situation allows, is to purge the current Calendar and then import the full ics file. This would be done like this:

zmmailbox -z -v -m ajcody@mail3.internal.homeunix.com ef /Calendar
* ef is for emptyFolder  zmmailbox help folder
*webclient shows all events gone but Calendars are still listed.
zmmailbox -z -v -m ajcody@mail3.internal.homeunix.com pru /Calendar /tmp/calendarB.ics
*webclient show all events with times as expected.

Second Fix

Haven't tried yet, but someone said you should be able to adjust the SEQUENCE number in the appointment and the import process will use that data (data/time) of the newer sequence number appointment.

Archiving User Accounts Out Of Production Use In Zimbra

Backup - Restore Method

To "archive" the user data with the ability of Zimbra later restoring if needed, one would rely on the backup/restore tools.

For example:

zmbackup -f -a user@domain

If you want to make this more "intelligent" later, one could do this:

mkdir /nfs-mount/archived-users/[user-name]
zmbackup -f -a userA@domain -t /nfs-mount/archived-users/userA-domain/

This would give some "intelligent" information for recovery later. Confirms it was an archived account, shows the "real" username, the date of the full backup, and so forth.

ls /nfs-mount/archived-users/userA-domain/
  full-20081104.131643.006
ls /nfs-mount/archived-users/userA-domain/full-20081104.131643.006
  accounts/ session.xml shared_blobs sys

If one would need to "restore" the account later, it would consume a license if the account was "deleted".

zmrestore -a userA@domain -ca -pre restored- -t /nfs-mount/archived-users/userA-domain/ -lb full-20081104.131643.006
Setup A Secondary Zimbra Box For Restores Of Archive Accounts

Your Zimbra license can be installed on multiple machines. One idea that might prove useful in handling these "archive" accounts for those situations when you need to investigate something is to setup a "archive" Zimbra box. You'll want to isolate this box from any "production" use. It will need to be configured to have the "domains" of the archive accounts. You can then use this box to restore the "archive" account and then use the administration tools to investigate the user data.

Use Of REST And Other Tools For Specific User Data

The following page, User_Migration , will shows numerous examples of how to export different types of data from a user account into a neutral file format that one could use for "archive" purposes.

Use Of The REST Command To Export ALL User Data - Version Dependant, 5.0.9+ [I believe]

Example of the commend syntax:

 /opt/zimbra/bin/zmmailbox -z -m user@domain.com getRestURL “//?fmt=tgz” > /tmp/account.tgz

Backup And HSM

Please see Bugs/RFE:

Using zmprov -l gaa To Create User Listing for zmbackup & zmrestore -a option

For example, see below. Note the use of the egrep -v option below, this would remove any matches from the grep. Useful if you want to only backup certain domains for example.

[zimbra@ldap2 sessions]$ zmbackup -f -a `zmprov -l gaa | egrep -v "ham|spam|virus|admin|galsync"`
full-20150130.210821.124

[zimbra@ldap2 sessions]$ zmbackup -f -a `zmprov -l gaa`
full-20150130.210922.637

[zimbra@ldap2 sessions]$ zmbackupquery -lb full-20150130.210821.124 -v
Label:   full-20150130.210821.124
Type:    full
Status:  completed
Started: Fri, 2015/01/30 16:08:21.124 EST
Ended:   Fri, 2015/01/30 16:08:50.181 EST
Redo log sequence range: 23 .. 23
Number of accounts: 3 out of 3 completed
Accounts:
  archive-search-results@ldap2.zimbra.DOMAIN.com: completed
  user1-20150115@ldap2.zimbra.DOMAIN.com.archive: completed
  user1@ldap2.zimbra.DOMAIN.com: completed

Total space: 14020MB
 Free space: 5977MB

[zimbra@ldap2 sessions]$ zmbackupquery -lb full-20150130.210922.637 -v
Label:   full-20150130.210922.637
Type:    full
Status:  completed
Started: Fri, 2015/01/30 16:09:22.637 EST
Ended:   Fri, 2015/01/30 16:09:33.083 EST
Redo log sequence range: 23 .. 23
Number of accounts: 9 out of 9 completed
Accounts:
  admin-20150122@ldap2.zimbra.DOMAIN.com.archive: completed
  admin@ldap2.zimbra.DOMAIN.com: completed
  archive-search-results@ldap2.zimbra.DOMAIN.com: completed
  galsync.tfnapjb9@ldap2.zimbra.DOMAIN.com: completed
  ham.yeygcogdd2@ldap2.zimbra.DOMAIN.com: completed
  spam.dpxyqjnm6t@ldap2.zimbra.DOMAIN.com: completed
  user1-20150115@ldap2.zimbra.DOMAIN.com.archive: completed
  user1@ldap2.zimbra.DOMAIN.com: completed
  virus-quarantine.wfkd4vvm1g@ldap2.zimbra.DOMAIN.com: completed

Total space: 14020MB
 Free space: 5977MB

Scripting Out Individual Backups Of Accounts

If you want to do individual backups of accounts using a for-loop, for example, you might want to include the -sync option from zmbackup. zmbackup without this will normal give an error as it passing the next zmbackup command stating that there's a current backup in progress.

Example command in some for-loop script is [without the -sync option]:

sudo -u zimbra /opt/zimbra/bin/zmbackup --fullBackup --account "$acct" --server "$host" --target "$dest"

zmbackup --help shows:

--sync  -sync Runs full backup synchronously. 

The admin guide mentions:

"Full backup is usually run asynchronously. When you begin the full backup, the label of the ongoing backup process is immediately displayed. The backup continues in the background. You can use the zmbackupquery command to check the status of the running backup at any time."

I couldn't find any other indication beyond that to explain in more details the purpose of that flag. But from what is stated above, it does look like the -sync flag will resolve the issues of "backup in progress" when scripting out multiple zmbackup commands.

If not, you could query for "Status: in progress" from the zmbackupquery command.

You can give the zmbackupquery command flags for date/time, label, account, -t target, and so forth [Do a zmbackupquery --help to see the options format].

Finding Message Blob In Users Backup

Get the zimbraId of the user, this is what is used in the backup session directory layout:

$ zmprov ga admin@`zmhostname` zimbraId
# name admin@zcs806.DOMAIN.com
zimbraId: e46a828b-cdda-4635-98ab-31b8ac0129b6

Get the mailboxId, this is what is used in the zmvolume primary message volume directory layout:

$ zmprov gmi admin@`zmhostname`
mailboxId: 1
quotaUsed: 77794874

Finding a message in the primary message volume:

$ cd /opt/zimbra/store/0/1/msg/9/

$ ls -la 40511-49376.msg
-rw-r----- 1 zimbra zimbra 1372 Mar 16 02:01 40511-49376.msg

Finding the users directory in a full backup session directory layout:

$ find /opt/zimbra/backup/sessions/full-20140426.080019.153/ -name e46a828b-cdda-4635-98ab-31b8ac0129b6 -print
/opt/zimbra/backup/sessions/full-20140426.080019.153/accounts/e46/a82/e46a828b-cdda-4635-98ab-31b8ac0129b6

Locating the message in the backup session - this example is where the backups are using the zip option:

$ cd /opt/zimbra/backup/sessions/full-20140426.080019.153/accounts/e46/a82/e46a828b-cdda-4635-98ab-31b8ac0129b6/blobs/

$ ls
blobs-1.zip  blobs-2.zip  blobs-3.zip  blobs-4.zip

$ for i in `ls /opt/zimbra/backup/sessions/full-20140426.080019.153/accounts/e46/a82/e46a828b-cdda-4635-98ab-31b8ac0129b6/blobs/`; do echo $i ; unzip -l $i | grep 40511-49376.msg; done

blobs-1.zip
     1372  04-26-2014 01:00   1/9/sha256_bKH8aeWJSh9bhzv5zsXrweA7+jZ88NUllkfV+7m9cJo=_40511-49376.msg1

Backing Up Backups - 3rd Party Tools And Software - Dealing With Directories With Hard Links

Description Of Hard Links

Zimbra uses hard links and special attention needs to be given to this fact. See hard links if your not familiar with hard links and their difference to symbolic links. Not all 3rd party backup software will handle or respect hard links. Many unix commands will need special flags to maintain hard links. When hard links are respected and also "copied" to the new location you could find your data usage become a large multiplication of the original size.

An good thread I found on the topic of preserving hard links for copy/move/backup operations is here:

I'll be summarizing the comments and including additional information I find on the topic below, based upon the command or software being used.

Zimbra and Single Instance Storage - Hard Links

If hard links are possible, we use them. The message must be identical and on to users on the same mailstore and stored on zmvolumes where hard linking is possible. Hard links exist only on the same partition. Postfix has a veriable, default_destination_recipient_limit , which will cause large recipient emails to be delivered in a way where they aren't identical [ see Ajcody-Hardlinks-And-Postfix-default_destination_recipient_limit for more details ].

Mike gives a good description of it here:

Redoing Hard Links RFE

For 8.0.2+ , see this option - zmdedupe :

Easy Way To SEE Hard Links In Use

I sent a test email with 5 accounts listed in the To field, that is shown below with the inode listing of 2133394. the first column of the ls [because of the i option] is listing the inode number of the file. I included the -type f because the . and .. will show directories using the same inode as the 'name' of the directory as listed.

# pwd
   /opt/zimbra/store2

# find . -links +1 -type f -exec ls -lai '{}' \; | sort
2133387 -rw-r----- 2 zimbra zimbra 1600 May  7 05:30 ./0/13/msg/0/281-4320.msg
2133387 -rw-r----- 2 zimbra zimbra 1600 May  7 05:30 ./0/27/msg/0/261-3021.msg
2133394 -rw-r----- 5 zimbra zimbra 1789 May  7 05:53 ./0/13/msg/0/282-4322.msg
2133394 -rw-r----- 5 zimbra zimbra 1789 May  7 05:53 ./0/14/msg/0/412-8476.msg
2133394 -rw-r----- 5 zimbra zimbra 1789 May  7 05:53 ./0/15/msg/0/284-9052.msg
2133394 -rw-r----- 5 zimbra zimbra 1789 May  7 05:53 ./0/25/msg/0/260-4021.msg
2133394 -rw-r----- 5 zimbra zimbra 1789 May  7 05:53 ./0/27/msg/0/268-3033.msg
2133404 -rw-r----- 2 zimbra zimbra 1518 May  7 05:22 ./0/13/msg/0/280-4318.msg
2133404 -rw-r----- 2 zimbra zimbra 1518 May  7 05:22 ./0/27/msg/0/260-3018.msg
Why Does A Message Blob With The Same Message-ID Have Multiple Inodes

Most likely, the message has a large recipient list and because of a variable in postfix, default_destination_recipient_limit , causes multiple deliveries of the message to have slight differences between them - for example, the times in the headers.

Please see the following:

Hard Links Used Within Zimbra Backup Directory - Sessions

Note, this describes an example when zimbraBackupMode equals Standard rather than Auto-Grouped


In regards to the backup sesions, hard links are only used for data that is in the shared_blobs directories from the various full backup sessions you have.

If you goto [below is an example used throughout]:

cd /opt/zimbra/backup/sessions/full-[some full dir]/shared_blobs/[some path]/t9RRjTIdwAZ3k,iJSWo0DxFKCbs=/

Then do a:

ls -li blob.dat
3869621 -rw-r----- 2 zimbra zimbra 5419 Jul 15 01:00 blob.dat

The first number is the inode of the file, which will be the same for all other hard links in use.

cd /opt/zimbra/backup/sessions
find . -inum 3869621 -print
  /opt/zimbra/backup/sessions/full-[some full dir #1]/shared_blobs/[some path]/t9RRjTIdwAZ3k,iJSWo0DxFKCbs=/blob.dat
  /opt/zimbra/backup/sessions/full-[some full dir #2]/shared_blobs/[some path]/t9RRjTIdwAZ3k,iJSWo0DxFKCbs=/blob.dat

Notice also, that the directory path naming scheme is also used for the user backup paths. The directory name that the blob.dat is in, will also be used in the user paths.

For example:

/opt/zimbra/backup/sessions/full-[some full dir]/shared_blobs/[some path]/t9RRjTIdwAZ3k,iJSWo0DxFKCbs=/blob.dat

if I then goto and do:

cd /opt/zimbra/backup/sessions/full-[some full dir]/
find . -name t9RRjTIdwAZ3k* print

I'll see matches in various user backups, with:

full-[some full dir]/accounts/c2e/23b/[zimbra ID of user]/blobs/3/0/t9RRjTIdwAZ3k,iJSWo0DxFKCbs=260-2041.msg4

Note though, those "user" files are not links [hard or soft] and they are of zero length in size.

Reference To Bugs Tied To Backups And Links:

Bacis Unix Commands

cp

cp [copy] Man page:

  • http://linux.die.net/man/1/cp
    • -a, --archive : same as -dpR
      • -d : same as --no-dereference --preserve=link
      • -p : same as --preserve=mode,ownership,timestamps
      • -R, -r, --recursive : copy directories recursively
  • -l, --link : link files instead of copying
  • -L, --dereference : always follow symbolic links
  • -P, --no-dereference : never follow symbolic links
  • -p : same as --preserve=mode,ownership,timestamps
    • --preserve[=ATTR_LIST] : preserve the specified attributes (default: mode,ownership,timestamps) and security contexts, if possible additional attributes: links, all
    • --no-preserve=ATTR_LIST : don't preserve the specified attributes
  • -s, --symbolic-link : make symbolic links instead of copying

No explicit mention of "hard links".

Formats from blog url above [I'll test these later and confirm/deny]:

  • cp -rpv
  • cp -av --preserve=all . /mnt/new
  • Simple `cp -a` using cp (GNU coreutils) 5.97 on my debian does the job quite nicely, I just checked. No need for the --preserve=all option, -a implies --preserve=link. It didn't seem to take too long either, but I would be surprised if it was very much better than rsync. Much easier to remember though.
mv

mv [move] Man page:

tar

tar [tape archive] Man page:

  • http://linux.die.net/man/1/tar
    • --check-links : warn if number of hard links to the file on the filesystem mismatch the number of links recorded in the archive
    • -h, --dereference : don't dump symlinks; dump the files they point to
      • No explicit reference to "hard links".

Please see the following on hard-links and GNU/Tar:

dd

dd [copy and covert - cc was reserved for the C complier] Man page:

No explicit mention of "hard links" in Man page.

rsync and nice

rsync and nice Man page:

A reasonable syntax to use for rsync is [ taken from Ajcody-Notes-Server-Move#The_Actual_Steps ]:

nice +19 rsync -avzH -e ssh --progress /opt/zimbra/ root@NEWHOSTIP:/opt/zimbra

or

nice -20 rsync -avzH -e ssh --delete --progress /opt/zimbra/ root@NEWHOSTIP:/opt/zimbra 
Notice the use of --delete in the last one.
find And cpio

find and cpio Man pages:

Possible syntax use to try [recursive copy of current working directory] :

mkdir /path/to/dest
cd /path/to/DATA
find . -print | cpio -Bpdumv /path/to/dest

Issue to note: "...cpio didn't properly preserve timestamps on directories."

dump And restore

dump and restore Man pages:

ditto - For Mac

ditto Man page:

xfs_copy

xfs_copy Man page:

LVM Tricks
LVM Snapshots

See Back Up (And Restore) LVM Partitions With LVM Snapshots

SAN Snapshot

Possible note of caution I've seen from someone, "When implementing snapshots for ZCS, you should do the snapshot across all ZCS LUNs for a single host at the same time using a consistency group (for netapp, I believe this means cg-start/cg-commit)."

Please see:

Cloud Backups
Amazon S3 , Amazon EC2 , SecoBackup And/Or Tar

I've not used Amazon S3 or SecoBackup. I have no idea about the pricing structure of Amazon S3 and how differing solutions might cause price differences. What I think would be a reasonable approach:

  • Adjust zimbra cron to:
    • run zmbackup as normally scheduled but then include:
      • tar and gzip "new" backup that was made to a "staging" partition.
      • Setup Secobackup [CLI method for cron] to then copy this tar'd/gzip'd file to the Amazon S3 cloud.
      • Remove local tar'd/gzip'd file from staging partition.

I purpose the tar'd/gzip'd step because I doubt there's a way to avoid the hard link issue with SecoBackup/Amazon S3. Why pay multiple times for the same data?

Some information a customer reported to me:

S3 does not work as a normal filesystem and you cannot mount it; hence 
it wouldn't normally work.

However, there are various projects out there which let you use S3 as 
a local POSIX-compliant file system.

Possible options:

s3fs-fuse
jungle disk
subcloud
persistentfs
Amazon EBS

To cut a long story short, PersistentFS came out on tops - it worked 
extremely well - however did not work with Zimbra at all once I set it 
up as the store (/opt/zimbra/store)

The problem is that the filesystem while it is POSIX compliant does not 
have support for hard linking (Which is what Zimbra does with tmp incoming 
messages to the store).  -- [bug 43019 below]

So, overall it's not really possible to do it right now with S3.

They should have hard link support soon.

References:

3rd Party Backup Tools And Software - Generally Not Apart Of The Basis OS

Amanda And Zmanda

Zmanda Home Page:

Amanda Home Page - Advanced Maryland Automatic Network Disk Archiver:

Found this, "Hard links. Maintains the integrity of hard links during backup."

Arkeia

Arkeia Home Page:

Found this, "The Arkeia solution accommodates full and incremental backups, scheduled or on demand, and preserves directory structure, registry, symbolic links and special attributes."

Also, a customer supplied what Arkeia's support sent them to the question. Here is that response:

When Arkeia encounters a hard link (a regular file with more than one reference to it):
  • if it is the first time we see this link, we backup the file and keep the inode number and the path to the file in a memory hash table.
  • if we have already seen the file, we backup the fact that it is a hard link and the target of the link. The files data are not backed up again.

Backup Exec - Symantec / Veritas

Backup Exec Home Page:

Unable to find any reference about the Linux/Unix Agent and the Backup Exec server being able to handle or not symbolic and hard links.

Backup Exec "server" is only available on Windows. One might inquire with Symantec if you can "swap out" your current investment in "Backup Exec" and use their NetBackup product. This supports hard links and the "server" can run on Windows or other *nixes. See Ajcody-Backup-Restore-Issues#NetBackup_-_Veritas.2FSymantec

BackupPC
Bacula

Bacula Home Page:

Found this:

hardlinks=yes|no
When enabled (default), this directive will cause hard links to be backed up. However, the File daemon keeps track of hard linked files and will backup the data only once. The process of keeping track of the hard links can be quite expensive if you have lots of them (tens of thousands or more). This doesn't occur on normal Unix systems, but if you use a program like BackupPC, it can create hundreds of thousands, or even millions of hard links. Backups become very long and the File daemon will consume a lot of CPU power checking hard links. In such a case, set hardlinks=no and hard links will not be backed up. Note, using this option will most likely backup more data and on a restore the file system will not be restored identically to the original.

Source:

BRU - TOLIS Group

BRU Home Page:

Found this in one of their manuals, you should confirm with them based upon the product version you'll be using:

Special Files - BRU will save and restore all types of filesystems and files with their proper ownership, access attributes, creation dates, and modification dates. BRU can be used to move an entire directory hierarchy from one system to another, with all files, including directories, block special files, character special files, fifos, hard links, and symbolic links reproduced with all attributes intact.
Lone-Tar - Lone Star Software Corp.

Lone-Tar Home Page:

Found the following references, though it's not explicit in stating "hard links" :

BACKUP FEATURES - Backs up everything including device files, empty directories, links, symbolic links, Virtual Files and NFS mounted file systems.
RESTORE FEATURES - Fast Seek File restore of files, sym-links, device files, and linked files.

Source:

NetBackup - Veritas/Symantec

NetBackup Home page:

Found the following:

Hard links to directories :
On most UNIX systems, only the root user can create a hard link to a directory. Some systems do not permit hard links and many vendors recommend that these links be avoided. NetBackup does not back up and restore hard-linked directories in the same manner as files:
  • During a backup, if NetBackup encounters hard-linked directories, the directories are backed up once for each hard link.
  • During a restore, NetBackup restores multiple copies of the hard-linked directory contents if the directories do not already exist on the disk. If the directories exist on disk, NetBackup restores the contents multiple times to the same disk location.
Hard links to files :
A hard link differs from a symbolic link in that a hard link is not a pointer to another file. A hard link is two directory entries that point to the same inode number.
If the backup selection list includes hard-linked files, the data is backed up only once during a backup. NetBackup uses the first file name reference that is found in the directory structure. If a subsequent file name reference is found, it is backed up as a link to the name of the first file. Backup up only the link means that only one backup copy of the data is created, regardless of the number of hard links. Any hard link to the data works.
For more information and examples, see “Hard links to files (NTFS volumes or UNIX)” on page 173.

Source:

NetVault - ORBiT

NetVault Home page:

Found this in their "NV Backup Administrators Guide" - pdf only:

[screen shot of GUI check box] The Attempt to Restore Hard Links’ option as revealed in the Restore Options tab on a Linux/UNIX-based version of the File System Plugin.
Attempt to Restore Hard Links (Linux/UNIX-based O/S, ONLY) - During a backup, when the first occurrence of a hard link is found, the complete data will be backed up. For all other occurrences, only the link is backed up. This data can only be restored when the first occurrence exists; trying to restore subsequent occurrences without the presence of the first causes the job to fail. Selecting this option will attempt to locate the full sequence so that all occurrences of the hard link will be restored.
rsnapshot

rsnapshot Home page:

Tivoli - IBM

Tivoli Home Page:

Found the following:

Understanding how hard links are handled
When you back up files that are hard-linked, Tivoli Storage Manager backs up each instance of the linked file. For example, if you back up two files that are hard-linked, Tivoli Storage Manager will back up the file data twice.
When you restore hard-linked files, Tivoli Storage Manager attempts to reestablish the links. For example, if you had a hard-linked pair of files, and only one of the hard-linked files is on your workstation, when you restore both files, they will be hard-linked. The files will also be hard-linked even if neither of the files exists at the time of restore, if both of the files are restored together in a single command. The one exception to this procedure occurs if you back up two files that are hard-linked and then break the connection between them on your workstation. If you restore the two files from the server, Tivoli Storage Manager will respect the current file system and not re-establish the hard link.
Attention: If you do not back up and restore all files that are hard-linked at the same time, problems will occur. To ensure that hard-linked files remain synchronized, back up all hard links at the same time and restore those same files together.


Source:

Other Related Items

freedup

freedup Man page:

freedup Homepage:

I've never used this tool, but from the description it seems it might come in handy for some circumstances.

"Establishes hard or symbolic links between identical files. Search all given file system trees for identical files and link them to the most frequently referenced inode or if equally referenced to the inode of the first file tree. If the devices differ a symbolic link is used instead of a hard link. Symbolic links will not replace files, when at least one of the directory trees is not starting with a '/'."
Tape Devices

Many times, "drivers" aren't needed for tape devices for linux, but many administrators are unaware of this and never give it a test. Instead, they just assume the device doesn't work and the tape vendor isn't supporting it because they "didn't publish" a driver for it.

Quantum

Resource:



NO_SUCH_BLOB Errors

   KB 2494        Last updated on 2008-11-16  




0.00
(0 votes)
24px ‎  - This is Zeta Alliance Certified Documentation. The content has been tested by the Community.


Actual No Such Blob Homepage

Please see Ajcody-Notes-No-Such-Blob

Other Resource Overviews

It would be good to review this wiki page before you start any "changes":

http://wiki.zimbra.com/index.php?title=Account_mailbox_database_structure

And great forum posting:

http://www.zimbra.com/forums/administrators/19811-solved-missing-blob-errors-zimbra-4-5-10-a.html

Check Permissions First

If you recently did a server move or similar type of operation, make sure zimbra:zimbra permissions are applied correctly to store directory. You could run the following to double check permissions (as root):

/opt/zimbra/libexec/zmfixperms --verbose --extended

Perl To Find MailboxID's That Have Problem

And down and dirty way to see what mailboxes have this error:

grep -B2 NO_SUCH_BLOB /opt/zimbra/log/mailbox.lo* |grep mailbox= |sed -r 's/.*mailbox=([0-9]*).*$/\1/' |sort -u

Mapping MailboxId to User Account

Remember, mailboxId is unique to the mailstore - it's not a global user variable that's unique.

Stolen from forum post:

http://www.zimbra.com/forums/administrators/16885-mailbox_id-email-address.html

su - zimbra
mysql
use zimbra;
select comment from mailbox where id=257;
And you'll get back:
+----------------------+
| comment |
+----------------------+
| user@domain.com |
+----------------------+
1 row in set (0.00 sec)

If you wanted to see all the info for that account:
select * from mailbox where id=257;
And you'll get back:
+-----+----------+--------------------------------------+-----------------+--------------------+---------------+-----------------+-------------------+---------------+---------------+----------------+----------------------+------------------+--------------+--------------------+
| id | group_id | account_id | index_volume_id | item_id_checkpoint | contact_count | size_checkpoint | change_checkpoint | tracking_sync | tracking_imap | last_backup_at | comment | last_soap_access | new_messages | idx_deferred_count |
+-----+----------+--------------------------------------+-----------------+--------------------+---------------+-----------------+-------------------+---------------+---------------+----------------+----------------------+------------------+--------------+--------------------+
| 257 | 71 | xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxxxxxxx | 2 | 257 | 0 | 1832 | 100 | 0 | 0 | NULL | user@domain.com | 1209652908 | 0 | 0 |
+-----+----------+--------------------------------------+-----------------+--------------------+---------------+-----------------+-------------------+---------------+---------------+----------------+----------------------+------------------+--------------+--------------------+
1 row in set (0.00 sec)

Zmblobchk for 5.0.6+ Systems

On 5.0.6+ there's a script in /opt/zimbra/bin called zmblobchk . This will list out information about the NO_SUCH_BLOB errors and other inconsistencies.

zmblobchk - RFE's

zmblobchk currently just lists "issues" - for the most part. Please see the following RFE's, some will give details about resolutions based upon what zmblobchk reported.

Example Output On A System With No Issues

su - zimbra
ls /opt/zimbra/zmblobchk.jar
  zmblobchk.jar
java -jar zmblobchk.jar
Retrieving volume information
Retrieving mboxgroup list
Spooling item list to /tmp/mailitems53650.lst
Retrieving items from mboxgroup1
Retrieving items from mboxgroup2
Retrieving items from mboxgroup3
Retrieving items from mboxgroup4
Retrieving items from mboxgroup5
Retrieving items from mboxgroup6
Retrieving items from mboxgroup7
Retrieving items from mboxgroup8
Retrieving items from mboxgroup9
Retrieving items from mboxgroup10
/
Processing BLOB store
-
/tmp/mailitems53650.lst: size 28141
Processed 329 items in 1223ms
No inconsistencies found

Example Output On A System With Issues

Last login: Thu Oct 30 12:03:12 on ttys000
COLDMILE-LM-3:~ ajcody$ ssh root@mail3.internal.homeunix.com
root@mail3.internal.homeunix.com's password: 
Last login: Sat Nov  1 14:25:50 2008 from 192.168.0.13
[root@mail3 ~]# su - zimbra
[zimbra@mail3 ~]$ zmblobchk 
Retrieving volume information
Retrieving mboxgroup list
Spooling item list to /tmp/mailitems39690.lst
Retrieving items from mboxgroup1
Retrieving items from mboxgroup2
Retrieving items from mboxgroup3
Retrieving items from mboxgroup4
Retrieving items from mboxgroup5
Retrieving items from mboxgroup6
Retrieving items from mboxgroup7
Retrieving items from mboxgroup8
Retrieving items from mboxgroup9
Retrieving items from mboxgroup10
Retrieving items from mboxgroup11
Retrieving items from mboxgroup12
Retrieving items from mboxgroup13
Retrieving items from mboxgroup14
Retrieving items from mboxgroup15
Retrieving items from mboxgroup16
Retrieving items from mboxgroup17
Retrieving items from mboxgroup18
Retrieving items from mboxgroup19
Retrieving items from mboxgroup20
-
Processing BLOB store
/
/tmp/mailitems39690.lst: size 470853
Processed 5873 items in 18462ms
11844 inconsistencies found
 * MailboxGroup=1, mailbox=1, item=303: wrong volume, expected at /vol1/0/1/msg/0/303-2062.msg, found at /opt/zimbra/store/0/1/msg/0/303-2062.msg (move to correct volume)
 * MailboxGroup=1, mailbox=1, item=304: wrong volume, expected at /vol1/0/1/msg/0/304-2064.msg, found at /opt/zimbra/store/0/1/msg/0/304-2064.msg (move to correct volume)
 * MailboxGroup=1, mailbox=1, item=305: wrong volume, expected at /vol1/0/1/msg/0/305-2077.msg, found at /opt/zimbra/store/0/1/msg/0/305-2077.msg (move to correct volume)
 * MailboxGroup=1, mailbox=1, item=306: wrong volume, expected at /vol1/0/1/msg/0/306-2091.msg, found at /opt/zimbra/store/0/1/msg/0/306-2091.msg (move to correct volume)

## CUT OF 1000's OF SIMILAR LINES ##

 * MailboxGroup=1, mailbox=1, item=6480: file not found: /vol1/0/1/msg/1/6480-20100.msg (delete associated metadata)
 * MailboxGroup=3, mailbox=3, item=400: wrong volume, expected at /vol1/0/3/msg/0/400-11287.msg, found at /opt/zimbra/store/0/3/msg/0/400-11287.msg (move to correct volume)
 * MailboxGroup=3, mailbox=3, item=425: wrong volume, expected at /vol1/0/3/msg/0/425-12939.msg, found at /opt/zimbra/store/0/3/msg/0/425-12939.msg (move to correct volume)
 * MailboxGroup=3, mailbox=3, item=427: wrong volume, expected at /vol1/0/3/msg/0/427-12943.msg, found at /opt/zimbra/store/0/3/msg/0/427-12943.msg (move to correct volume)
 * MailboxGroup=3, mailbox=3, item=440: file not found: /vol1/0/3/msg/0/440-13400.msg (delete associated metadata)
 * /opt/zimbra/store/0/3/msg/0/363-6194.msg: no associated metadata (delete blob)
 * /opt/zimbra/store/0/3/msg/0/360-5000.msg: no associated metadata (delete blob)
 * /opt/zimbra/store/0/3/msg/0/350-3129.msg: no associated metadata (delete blob)
 * /opt/zimbra/store/0/3/msg/0/320-2272.msg: no associated metadata (delete blob)
 
## CUT OF 1000's OF SIMILAR LINES ##

 * /opt/zimbra/store/0/2/msg/0/304-6907.msg: no associated metadata (delete blob)
 * /opt/zimbra/store/0/2/msg/0/315-6918.msg: no associated metadata (delete blob)
 * /opt/zimbra/store/0/2/msg/0/312-6915.msg: no associated metadata (delete blob)
 * /opt/zimbra/store/0/2/msg/0/308-6911.msg: no associated metadata (delete blob)
 * /opt/zimbra/store/0/2/msg/0/303-6906.msg: no associated metadata (delete blob)
 * /opt/zimbra/store/0/2/msg/0/301-6904.msg: no associated metadata (delete blob)
 *** 11844 faults found
Report saved to: /tmp/zmblobc39689.rpt
[zimbra@mail3 ~]$ 

To see the reported output again:

zmblobchk -l /tmp/zmblobc39689.rpt


Error - Not Enough Memory To Run zmblobchk

If the "Processing BLOB store" part errors out with Java out of memory, you could try adjusting the variable for it. The default is set to 30%. This would require a mailstore/jetty restart though [zmmailboxdctl restart]. If the mailstore doesn't come back up, check /opt/zimbra/log/zmmailbox.log - you most likely exceeded the threshold for your box. Remember, this is a percentage and a 32bit machine with more than 4GB can cause issues with this setting since it could try to allocate more than it actually can to the thread.

# ZCS 5.0 and later
$ zmlocalconfig -e mailboxd_java_heap_memory_percent=40

And zmmailboxdctl restart when it's appropriate.

BLOB Issue Script

I've commented out the two delete lines and also the "$msghash &= $fmask;" one.

If you're using HSM, this script isn't smart enough to detect it. It will flag and remove entries that are in the HSM store because it doesn't see them in the primary store. You can modify the output though to use the actual path to the HSM store. I had one case where this was the case and copying the missing blobs to the HSM path worked.

Comment from other support staff member on a case that used this.

Once you've gotten a list of the 'NOT OK' files generated by this tool, you'll want to check the store directories to see if the message blobs exist with an incorrect change number. That's the second number in the filename. If the first number matches the ID it's looking for but the second does not, you can rename the file to the new change number and get the message blob back without losing data. If not, then the blob is completely missing. If that's the case, we should try to figure out what happened to it by going through the mailbox.log files. You should be able to grep for the missing message ID and the user's account ID to find a log entry showing something happening to it. It's possible it was deleted and the deletion was never written to mysql for some reason.

#!/usr/bin/perl

# On 5.0.6+ there's a script, /opt/zimbra/bin/zmblobchk , that you can use instead of this.
# This script is compatible with Zimbra version 4.5.x and 5.x .
# OK, there's 2 MAILBOX_*_BITS values in the VOLUME table.
# Take the mailbox ID, right-shift it by MAILBOX_BITS, and take the lowest MAILBOX_GROUP_BITS of the result.
# That's your mailbox hash.
# Take the message ID, right-shift it by FILE_BITS, and take the lowest FILE_GROUP_BITS of the result.  That's your msgid hash.
# I think.
# <mbx-hash>/<mbx-id>/msg/<msgid-hash>/<msgid>-<mod_content>.msg

my ($fbits, $fgbits, $mbits, $mgbits, $basepath) = split (' ',`echo "select file_bits,file_group_bits,mailbox_bits,mailbox_group_bits,path from volume where type='1'" | mysql -N zimbra`);
my $ARGV = shift;

chomp $basepath;

my $mbmask = "1" x $mgbits;
my $fmask = "1" x $fgbits;

foreach (`echo "select id, group_id, account_id, comment from mailbox where comment = '$ARGV'" | mysql -N zimbra`) {
	chomp;
	my $path = "$basepath/";
	my ($id, $grid, $aid, $nm) = split;
	my $mbhash = $id >> $mbits;
	$mbhash &= $mbmask;
	$path .= $mbhash."/".$id."/msg/";
	foreach my $msgstuff (`echo "select id, mod_content, type from mail_item where blob_digest is not null and mailbox_id=${id};" | mysql -N mboxgroup${grid}`) {
		chomp $msgstuff;
		next if $msgstuff eq "";
		my ($msgid, $modContent, $type) = split (' ',$msgstuff);
		my $msghash = $msgid >> $fbits;
#		$msghash &= $fmask;
		my $nm = $msgid;
		if ($modContent) {$nm .= "-$modContent";}
		my $npath = $path.$msghash."/".$nm.".msg";
		if (-e $npath) {print $npath."\t\tOK\n";} 
		else {
		   print $npath."\t\tNOT OK\n";
		   # not ok, remove the entry from the database so it is not a nuisance
		   print "Delete from mail_item where MSGID is ${msgid} and MAILBOXID is ${id} \n";
		   print "Uncomment line below me in script to have me delete. \n";
#		   `echo "delete from mail_item where id=${msgid} and mailbox_id=${id}" | mysql -N mboxgroup${grid}`;
		   if ($type eq "11") {
	       print "Delete from appointment where MSGID is ${msgid} and MAILBOX_ID is ${id} \n";
	       print "Uncomment line below me in script to have me delete. \n";
#              `echo "delete from appointment where item_id=${msgid} and mailbox_id=${id}" | mysql -N mboxgroup${grid}`;
		   }
		}
#		print $npath."\n";
	}
}

If this doesn't run, make sure perl is installed and in the correct path.

You run this as the zimbra user.

What I did was the following.

Copied the script to /opt/zimbra/bin/ as root and called it blob-check.pl . chmod +x /opt/zimbra/bin/blob-check.pl

su - zimbra
mkdir /tmp/BLOB-CHECKS
for i in `zmprov gaa`; do blob-check.pl $i | grep "NOT OK" >> /tmp/BLOB-CHECKS/$i.txt; done
ls -la /tmp/BLOB-CHECKS

The ls will help identify the more troubled accounts. You'll end up with something like this:

[zimbra@mail3 ~]$ ls -la /tmp/BLOB-CHECKS/
total 16
drwxr-x--- 2 zimbra zimbra 4096 Jul 10 23:40 .
drwxrwxrwt 8 root   root   4096 Jul 10 23:46 ..
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 admin@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 ajcody2@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra   49 Jul 10 23:40 ajcody@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 forward@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 ham.bidiob2mm@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 secondary@secondary.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 spam.rormmtcyy@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 wiki@mail3.internal.homeunix.com.txt
-rw-r----- 1 zimbra zimbra    0 Jul 10 23:40 wiki@secondary.internal.homeunix.com.txt

And the output of one that shows some size:

$ cat /tmp/BLOB-CHECKS/ajcody@mail3.internal.homeunix.com.txt
/opt/zimbra/store/0/3/msg/0/360-6070.msg		NOT OK

Remember this from above, "you'll want to check the store directories to see if the message blobs exist with an incorrect change number. That's the second number in the filename. If the first number matches the ID it's looking for but the second does not, you can rename the file to the new change number and get the message blob back without losing data."

One can uncomment the 2 lines in the script with delete and run the script again. This will remove the reference to the blob.

To re-index the users mailbox [this can take start, cancel, status]:

zmprov rim user@domainname start

Running the script again shouldn't show "NOT OK" lines.

FYI - I still need to gather more information about this situation and how to guide one to make a decision to delete, re-index, and so forth.

BLOB Script To Copy From Restore

You might need to use these variables with the restore command. I need to find a way for someone to figure out what restore to goto for a particular blob:

-restoreToTime <arg> - Replay the redo logs until the time specified.

-restoreToIncreLabel <arg> - Replay redo logs up to and including this incremental backup.

-restoreToRedoSeq <arg> - Replay up to and including this redo log sequence

-br - Replays the redo logs in backup only, which excludes archived and current redo logs of the system.

-rf - Restores to the full backup only, does not include any incremental backups since that backup

This is a walk through to test the proceedure.

$ zmrestore -ca -a ajcody@mail3.internal.homeunix.com -pre restored-
$ zmprov gmi restored-ajcody@mail3.internal.homeunix.com
mailboxId: 12
quotaUsed: 251513
$ zmprov gmi ajcody@mail3.internal.homeunix.com
mailboxId: 3
quotaUsed: 251512
$ diff /opt/zimbra/store/0/12/msg/0/257-25.msg /opt/zimbra/store/0/3/msg/0/257-25.msg
$ rm /opt/zimbra/store/0/3/msg/0/257-25.msg
rm: remove regular file `/opt/zimbra/store/0/3/msg/0/257-25.msg'? y
$ cp /opt/zimbra/store/0/12/msg/0/257-25.msg /opt/zimbra/store/0/3/msg/0/257-25.msg
$ grep Subject /opt/zimbra/store/0/3/msg/0/257-25.msg
Log into main account and confirm I can see/use that mail.

The script to look at the blob-check.pl output and do the copies after a redirected restore is here:

#!/bin/bash

#START OF VARIABLES#
#CHANGEME to existing user account
USER=ajcody@mail3.internal.homeunix.com

#CHANGEME to the restored account name
# zmrestore -ca -a ajcody@mail3.internal.homeunix.com -pre restored-
RESTOREDUSER=restored-ajcody@mail3.internal.homeunix.com

# To get FILE, blob-check.pl ajcody@mail3.internal.homeunix.com > /tmp/BLOB-CHECKS/ajcody.txt
# blob-check.pl located at http://wiki.zimbra.com/index.php?title=Ajcody-Notes#BLOB_Issue_Script
FILE=/tmp/BLOB-CHECKS/ajcody.txt #CHANGEME

#CHANGEME, if needed, to the mailstore. /opt/zimbra/store is default
STOREDIR=/opt/zimbra/store

#END VARIABLES#

USERUID=`zmprov gmi $USER | grep mailboxId | cut -f2 -d: | cut -c 2-10`
USERGID=`perl -e 'print $USERUID >> 12 ; print "\n"'`
USERPATH=`echo $USERGID/\$USERUID`

RESTOREDUID=`zmprov gmi $RESTOREDUSER | grep mailboxId | cut -f2 -d: | cut -c 2-10`
RESTOREDGID=`perl -e 'print $RESTOREDUID >> 12 ; print "\n"'`
RESTOREDPATH=`echo $RESTOREDGID/\$RESTOREDUID`

echo USERUID $USERUID
echo USERGID $USERGID
echo USERPATH $USERPATH
echo RESTOREDUID $RESTOREDUID
echo RESTOREDGID $RESTOREDGID
echo RESTOREDPATH $RESTOREDPATH

cd $STOREDIR
echo "Run ECHO first to confirm copies look right and then uncomment COPY in script"
for ROOTBLOBPATH in `grep "NOT OK" $FILE | cut -f7-9 -d/ | cut -f1`
do
# Comment out echo and uncomment copy after dry run
echo $RESTOREDPATH/$ROOTBLOBPATH $USERPATH/$ROOTBLOBPATH
#cp -uv $RESTOREDPATH/$ROOTBLOBPATH $USERPATH/$ROOTBLOBPATH
done

BLOB Script To Copy From Restore (HSM Issue)

One customer ran into an issue that the blob references were really expecting them in the HSM mailstore path.

Example:

Default mailstore for PROD : /opt/zimbra/mail/

HSM mailstore path : /opt/zimbra/mail0 which was another partition mount, compared to /opt/zimbra/mail

So, what this script did was used the output from the blob-check.pl script which has the missing blobs formatted for the PROD path rather than the HSM path. The blob-check.pl script doesn't handle HSM data. The script then finds the blobs in the full backup directory of the user and copies them into the HSM path that the WEBCLIENT is expecting. This whole situation with this customer might of been a fluke. So don't consider these steps as applying to your situation unless you do your research. This case took 9 days to resolve and figure out.

Notice, if you use this script below you really need to review it in details and adjust it for your situation. There's more to adjust beyond the base variables.

#!/bin/bash

# Steps
# Make sure directory path exists for the copy command
# Get mailboxId - zmprov ga USER@DOMAIN | grep zimbraId:
# mkdir /opt/zimbra/mail0/0/[mailboxId #]/msg/{1..15}  or how what ever # you need.
# Change USEREMAIL
# Get path for BACKUPFILES and uncomment for use
# Run find command with echo and confirm. Then rerun using the find with cp rather than echo

# Put in user's email for USEREMAIL
USEREMAIL="USER@DOMAIN"
ZIMBRAID=`zmprov ga $USEREMAIL | grep zimbraId: | awk '{print $2}'`
# See below
# ROOTBACKUPPATH="/opt/zimbra/mail0/BACKUP/sessions/full-20080707.004336.789"

#The line below takes to long, see instructions underneath
#BACKUPFILES=`find $ROOTBACKUPPATH -name *$ZIMBRAID*`
# To run manually, which will increase speed of script if you then paste full path in for BACKUPFILES var
# zmprov ga USER@DOMAIN | grep zimbraId: | awk '{print $2}' 
# Then place the out in this find command, leave the wildcards on each side of it
# find /opt/zimbra/mail0/BACKUP/sessions/full-20080707.004336.789 -name *OUTPUTHERE* 
# This should give you path like the exampe below, paste it between the quotes and uncomment the variable BACKUPFILES
#EXAMPLE FOR BACKUPFILES="/opt/zimbra/mail0/BACKUP/sessions/full-20080707.004336.789/accounts/a07/1b5/a071b5ad-b341-4de4-b6ee-4463f322873b"
# BACKUPFILES="PASTE_IN_PATH_HER"

for BLOBPATH in `blob-check.pl $USEREMAIL | grep "NOT OK" | awk '{print $1}' | sed 's/store/mail0/'`
do
MSGID=`echo $BLOBPATH | awk -F/ '{print $NF}'`
# Run echo first to make sure copy path seems correct, ctrl-c to cancel. And then uncomment the other find command with the cp. Comment the echo one.
find $BACKUPFILES -name *$MSGID* -exec  echo {} $BLOBPATH \;
#find $BACKUPFILES -name *$MSGID* -exec cp -uv {} $BLOBPATH \;
done

The Really Bad Situation, Blob References Gone & Mail Still Missing

For now, this is all I have to offer.

Method One - zmlmtpinject

See forum post:

http://www.zimbra.com/forums/installation/12617-recover-data-mbox-folders.html#post64962

Method Two - Use IMAPSYNC

I haven't tested this and would just be guessing, but I believe IMAPSYNC could be used.

From the IMAPSYNC notes, "imapsync is the adequate tool because it reduces the amount of data transferred by not transferring a given message if it is already on both sides. Same headers, same message size and the transfer is done only once. All flags are preserved, unread will stay unread, read will stay read, deleted will stay deleted."

This would involve having imap setup on ZCS and then doing a redirected user restore (which would consume a license).

See the following on imapsync details:

Imapsync During A Certain Time Frame

You might want to check out the imapsync options of --maxage and --minage

Proposed Steps
  • Confirm these steps are necessary
  • Install imapsync
  • Do a redirected restore of the account
    • This is an example format of the redirected restore:
      • zmrestore -ca -a prod-user@servername.com -pre restored- -rf -lb full-'date of full'
      • The "new" account would be restored-userPROD@servername.com . The -rf flags should look only at the backup data in regards to the restore and not catch the "production" references to missing blobs and what not.
  • Confirm in webclient that msg's are there
  • Do a IMAPSYNC
    • Two ways, unsure what is the best method
      • From PROD to NEW
        • imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates --host1 servername.com --user1 prod-user --password1 test123 -user2 restored-prod-user --host2 servername.com --password2 test123 --authmech1 PLAIN --authmech2 PLAIN
      • From NEW to PROD
        • imapsync --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates --host1 servername.com --user1 restored-prod-user --password1 test123 -user2 prod-user --host2 servername.com --password2 test123 --authmech1 PLAIN --authmech2 PLAIN
  • Reindex restored-prod-user
  • Rename prod-user to prod-user-old
    • zmprov ra prod-user@servername.com prod-user-old@servername.com
  • Rename restored-prod-user to prod-user
    • zmprov ra restored-prod-user@servername.com prod-user@servername.com
  • Do a full backup of each account
    • zmbackup -f - <servername.com> -a <newaccountname@servername.com>

Notes On - No Associated Metadata

Tentative Research At This Point - Haven't Proven Out Yet

This is for the errors you see from the blobchk like this:

"* /opt/zimbra/store/0/1303/msg/0/1059-1726.msg: no associated metadata  (delete blob) " 

You could use the following command to bring the blob back in and regenerate the metadata -- zmmailbox am along -d . The am flag details are : addMessage(am) [opts] {dest-folder-path} {filename-or-dir} [{filename-or-dir} ...] . But the -d flag states : -d/--date <arg> received date (msecs since epoch) . To get the epoch time in seconds using the same receive data, for example :

ls -l --time-style=+%s /opt/zimbra/store/0/1303/msg/0/1059-1726.msg

And then remember the -d requires milliseconds [a thousandth (1/1,000) of a second] so you'll need to convert it. Or, you could just use something like this below. This with the cut would work well for a for loop script against a number of msg files.

ls -l --time-style=+%s000 --si /opt/zimbra/store/0/1303/msg/0/1059-1726.msg | cut -d" " -f6
  1272642001000
ls -l --time-style=+%s000 --si /opt/zimbra/store/0/1303/msg/0/1059-1726.msg
  -rw-r--r-- 1 zimbra zimbra 0 1272642001000 

Another suggestion for scripting was:

date=$(stat -c +%Y /opt/zimbra/store/0/1303/msg/0/1059-1726.msg); echo $(($date*1000));

Mysql Queries To See What Is Shown For Missing Message ID

References

See also :

Getting User Variables we need to query MySQL

Please see Ajcody-Mysql-Topics#Getting_User_Variables_We_Need_To_Query_MySQL

Mysql Query For A Users Specific Blob-Data ID

Please see Useful Mysql Query Examples



SSL Certificate Issues

   KB 2494        Last updated on 2008-11-16  




0.00
(0 votes)
24px ‎  - This is Zeta Alliance Certified Documentation. The content has been tested by the Community.


Actual SSL Certificate Issues Homepage

Please see Ajcody-Notes-SSLCerts

Resources For SSL Certificates

Bug & RFE's Related To SSL

Multiple SSL Certificates Aren't Supported On One Server

The -subjectAltNames option is the way to do, if your certificate provider doesn't support then you should inquire into another provider who does.

Recreating Self-Signed SSL Certificates

Other references:

Steps I've used for a single ZCS 6 server that WAS NOT using commercial certificates. Also, this is for recreating self-signed certificates and not changing them. This documentation was done specifically for the issue when the self-signed certificates expired and caused upgrade issues.

** have zimbra running - ldap at least - need to double check this
** run as root
** Prep work
mv /opt/zimbra/ssl/zimbra /opt/zimbra/ssl/zimbra_old
** I had used the below, but think the above step is *better*
* mv /opt/zimbra/ssl /root/ssl_old
* mkdir /opt/zimbra/ssl
* chown zimbra:zimbra /opt/zimbra/ssl
** Creating new certs
/opt/zimbra/bin/zmcertmgr createca -new 
/opt/zimbra/bin/zmcertmgr deployca -localonly 
** others have just used: /opt/zimbra/bin/zmcertmgr deployca 
/opt/zimbra/bin/zmcertmgr createcrt self -new 
** others have just used: /opt/zimbra/bin/zmcertmgr createcrt -new
/opt/zimbra/bin/zmcertmgr verifycrt self 
/opt/zimbra/bin/zmcertmgr deploycrt self
** run as zimbra
zmcontrol stop
zmcontrol start

Keystore Password Errors - Server Move Stuff

Seen this with user attempting the 32 > 64 server move wiki page, mailboxd wouldn't start. Errors like:

keytool error: java.io.IOException: Keystore was tampered with, or password was incorrect
*
Zimbra server reserving server socket port=995 bindaddr=null ssl=true
Fatal error: exception while binding to ports
java.net.SocketException: Unbound server sockets not implemented
*
Exception in thread "main" java.io.IOException: Keystore was tampered with, or password was incorrect

And when I attempted to recreate the self-sign cert using the steps above, it ended with this error:

# /opt/zimbra/bin/zmcertmgr deploycrt self
** Saving server config key zimbraSSLCertificate...failed.
** Saving server config key zimbraSSLPrivateKey...failed.
** Installing mta certificate and key...done.
** Installing slapd certificate and key...done.
** Installing proxy certificate and key...done.
** Creating pkcs12 file /opt/zimbra/ssl/zimbra/jetty.pkcs12...done.
** Creating keystore file /opt/zimbra/mailboxd/etc/keystore...failed.

Exception in thread "main" java.io.IOException: Keystore was tampered with, or password was incorrect
	at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:771)
	at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:38)
	at java.security.KeyStore.load(KeyStore.java:1185)
	at com.zimbra.cert.MyPKCS12Import.main(MyPKCS12Import.java:98)
Caused by: java.security.UnrecoverableKeyException: Password verification failed
	at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:769)
	... 3 more

** Installing CA to /opt/zimbra/conf/ca...done.

And the above recreating of self-cert fails because of issues about keystore password.

Fix was to :

[as root]  mv /opt/zimbra/mailboxd/etc/keystore /root/keystore.old 

And then run the recreating self-certs above.

Note: this also appears to happen on 8.6, with self-signed certs after rsync migration. Moving the old keystore and redeploying ca/certs corrects the issue.

Contrib from Baylink

(Here's an actual script you can copy to a file and run, by Baylink 16:08, 25 March 2010 (UTC))

echo "Backing up old certs..."
mv /opt/zimbra/ssl/zimbra /opt/zimbra/ssl/zimbra.newcert-`timestamp`

echo "Creating new CA..."
/opt/zimbra/bin/zmcertmgr createca -new

echo "Deploying new CA..."
/opt/zimbra/bin/zmcertmgr deployca -localonly

# added 12 Apr 2010 per tonyp@zimbra.com
echo "Creating new CSR..."
/opt/zimbra/bin/zmcertmgr createcsr self -new -subject "/C=US/ST=N\/A/L=N\/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration Suite/CN=server.company.com" -subjectAltNames "server.company.com,altname.company.com,othername.company.com"

echo "Creating new cert..."
/opt/zimbra/bin/zmcertmgr createcrt -new -subject "/C=US/ST=N\/A/L=N\/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration Suite/CN=server.company.com" -subjectAltNames "server.company.com,altname.company.com,othername.company.com"

echo "Verifying new cert..."
/opt/zimbra/bin/zmcertmgr verifycrt self

echo "Deploying new cert..."
/opt/zimbra/bin/zmcertmgr deploycrt self

echo "Restarting Zimbra server"
su - zimbra -c 'zmcontrol stop; zmcontrol start'

Note that if you're running Exchange ActiveSync with iPhones, the iPhones require that the "server.company.com" name (the primary name) must

  • be the primary name on the SSL cert (I've tested this) and
  • resolve to the same IP both inside and outside your firewall which
    • requires a firewall that can do "hairpin" inbound-NAT

So, if your mailbox server has a "real" name, then that, and any "role" name you use for ZWC user access must be secondary names, and the Active Sync name (I chose "async") must be primary.

That script calls my 'timestamp' script, which is just:

date +%Y%m%d-%H%M%S

(end contrib)

Ldap And-Or MTA Doesn't Start After Cert Changes Or Upgrade

Brief summary of issues:

  • The ca directory contained extra links to different certificates. This seemed to not bother 5.0.9 but under 5.0.10 postfix has a fit if there are more then 3 files in that directory.

Example of LDAP error:

[zimbra@server-01 ~]$ zmcontrol start
Host server-01.DOMAIN.com
        Starting ldap...Done.
FAILED
Failed to start slapd.  Attempting debug start to determine error.
TLS: error:0906D06C:PEM routines:PEM_read_bio:no start line pem_lib.c:647
TLS: error:0906D06C:PEM routines:PEM_read_bio:no start line pem_lib.c:647
TLS: error:0906D06C:PEM routines:PEM_read_bio:no start line pem_lib.c:647
TLS: error:0906D06C:PEM routines:PEM_read_bio:no start line pem_lib.c:647
TLS: error:0906D06C:PEM routines:PEM_read_bio:no start line pem_lib.c:647
TLS: error:0906D06C:PEM routines:PEM_read_bio:no start line pem_lib.c:647
TLS: error:0906D06C:PEM routines:PEM_read_bio:no start line pem_lib.c:647
TLS: error:0906D06C:PEM routines:PEM_read_bio:no start line pem_lib.c:647
TLS: error:02001002:system library:fopen:No such file or directory bss_file.c:352
TLS: error:20074002:BIO routines:FILE_CTRL:system lib bss_file.c:354
main: TLS init def ctx failed: -1

The fix is described in:

Details as related to MTA/Postfix - Error_(MTA):_Unable_to_set_STARTTLS

More details as shared by a customer:

I moved the "ca" directory to "ca.BAK" to back it up, made a new ca directory, and then went through the steps detailed in that post.
Once done, zimbra started up without a problem:

1) Clear all the contents of the /opt/zimbra/conf/ca directory by backing them up
somewhere on disk.
2) Copy the /opt/zimbra/ssl/zimbra/commercial/commercial.key /opt/zimbra/conf/ca/ca.key
3) Copy /opt/zimbra/ssl/zimbra/commercial/commercial.crt /opt/zimbra/conf/ca/ca.pem
4) Create the hash value
ln -f -s ca.pem /opt/zimbra/conf/ca/`openssl x509 -hash -noout -in
/opt/zimbra/conf/ca/ca.pem`.0
5) Chmod 644 /opt/zimbra/conf/ca/*
6) Restart the zmcontrol


A sample of what the /opt/zimbra/conf/ca directory looked like:

lrwxrwxrwx  1 root root   20 Apr  9 20:29 0bb21872.0 -> commercial_ca_26.pem
lrwxrwxrwx  1 root root   20 Apr  9 20:29 0c364b2d.0 -> commercial_ca_14.pem
lrwxrwxrwx  1 root root   20 Apr  9 20:29 0e82f83a.0 -> commercial_ca_36.pem
lrwxrwxrwx  1 root root   20 Apr  9 20:29 11f154d6.0 -> commercial_ca_49.pem
lrwxrwxrwx  1 root root   19 Apr  9 20:29 128b9c8d.0 -> commercial_ca_9.pem
lrwxrwxrwx  1 root root   19 Apr  9 20:29 1a147d5b.0 -> commercial_ca_5.pem
lrwxrwxrwx  1 root root   20 Apr  9 20:29 1bb6c7e0.0 -> commercial_ca_24.pem
lrwxrwxrwx  1 root root   20 Apr  9 20:29 1c647a6d.0 -> commercial_ca_21.pem
lrwxrwxrwx  1 root root   20 Apr  9 20:29 256fd83b.0 -> commercial_ca_33.pem
lrwxrwxrwx  1 root root   20 Apr  9 20:29 25f0cbee.0 -> commercial_ca_23.pem
lrwxrwxrwx  1 root root   20 Apr  9 20:29 2d047263.0 -> commercial_ca_28.pem

There we many more of the above entries, all of which pointed to .pem files that didn't exist.

There was also:
lrwxrwxrwx  1 root root    6 Apr 13 11:43 555ebb99.0 -> ca.pem
lrwxrwxrwx  1 root root   17 Apr  9 20:29 8e6e2991.0 -> commercial_ca.pem
lrwxrwxrwx  1 root root   17 Apr 13 11:43 c33a80d4.0 -> commercial_ca.pem
lrwxrwxrwx  1 root root   20 Apr  9 20:29 c527e4ab.0 -> commercial_ca_57.pem
-rw-r--r--  1 root root  887 Apr 13 11:43 ca.key
-rw-r--r--  1 root root  989 Apr 13 11:43 ca.pem
-rw-r--r--  1 root root 1155 Apr 13 11:43 commercial_ca_1.pem
-rw-r--r--  1 root root 1156 Apr 13 11:43 commercial_ca.pem

zmcertmgr output:

./zmcertmgr deploycrt comm /opt/zimbra/certs/server-01.DOMAIN.com.crt /opt/zimbra/certs/ca_chain-server-01.DOMAIN.com.crt
** Verifying /opt/zimbra/certs/server-01.DOMAIN.com.crt against /opt/zimbra/ssl/zimbra/commercial/commercial.key
Certificate (/opt/zimbra/certs/server-01.DOMAIN.com.crt) and private key (/opt/zimbra/ssl/zimbra/commercial/commercial.key) match.
Valid Certificate: /opt/zimbra/certs/server-01.DOMAIN.com.crt: OK
** Copying /opt/zimbra/certs/server-01.DOMAIN.com.crt to /opt/zimbra/ssl/zimbra/commercial/commercial.crt
** Appending ca chain /opt/zimbra/certs/ca_chain-server-01.DOMAIN.com.crt to /opt/zimbra/ssl/zimbra/commercial/commercial.crt
** Saving server config key zimbraSSLCertificate...done.
** Saving server config key zimbraSSLPrivateKey...done.
** Installing mta certificate and key...done.
** Installing slapd certificate and key...done.
** Installing proxy certificate and key...done.
** Creating pkcs12 file /opt/zimbra/ssl/zimbra/jetty.pkcs12...done.
** Creating keystore file /opt/zimbra/mailboxd/etc/keystore...done.
** Installing CA to /opt/zimbra/conf/ca...done.

Moving Your Certificates To New Or Another Server

Please see Transfer_SSL_certificates_between_servers

Commercial Cert Error - Subject Does Not Start With /

As reported by a customer to me:

When creating a commercial cert for a server the zmcertmgr will fail if you don't supply a subjectAltName ---- Took me awhile to figure this out since the error isn't correctly describing the problem. It says that "Subject does not start with '/'." Which is incorrect. Subject does start with "/" , it's the subjectAltName that was needed. After I supplied this name, it generated the csr. Here are my commands for your own reference.

I modified the Some* entries below.
/opt/zimbra/bin/zmcertmgr createcsr comm -new -subject "/C=US/ST=SomeState/L=SomeCity/O=Some Community College/OU=ITS/CN=zimbra.somecommunitycollege.edu" -subjectAltNames zimbra.somecommunitycollege.edu

then I went to thawte and applied for a ssl cert.

How To Setup Certs With CACert.org - Free Certs

CaCert.Org References

Free Certs with http://www.cacert.org/

How-To (tested on 5.0.2)

Note, the following :

su - root ; cd /opt/zimbra/ssl/zimbra/commercial/

included in all steps in case someone is skipping through instructions.

Clean up and start fresh
su - root
cd /opt/zimbra/ssl/zimbra/commercial/
tar -czvf /tmp/ssl.commercial.backup.tar.gz *
rm -rf *
Generate new csr
[ su - root ; cd /opt/zimbra/ssl/zimbra/commercial/ ]
/opt/zimbra/bin/zmcertmgr createcsr comm -new

This uses the defaults, note the items to change.

/opt/zimbra/bin/zmcertmgr createcsr comm -new "/C=US/ST=TX/L=Somewhere/O=Company, Inc./OU=ITDepartment/CN=mail.CHANGEME.com"
Confirm
[ su - root ; cd /opt/zimbra/ssl/zimbra/commercial/ ]
ls -la

There should only be two files and time/date should match - commercial.csr & commercial.key

cat /opt/zimbra/ssl/zimbra/commercial/commercial.csr
 -----BEGIN CERTIFICATE REQUEST-----
[delete]CCAWwCAQAwgZkxCzAJBgNVBAYTAlVTMQwwCgYDVQQIEwNOL0ExDDAKBgNV
[delete]4vQTEjMCEGA1UEChMaWmltYnJhIENvbGxhYm9yYXRpb24gU3VpdGUxIzAh
[delete]AsTGlppbWJyYSBDb2xsYWJvcmF0aW9uIFN1aXRlMSQwIgYDVQQDExttYWls
[delete]nRlcm5hbC5ob21ldW5peC5jb20wgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJ
[delete]AOri9/m6RtM1vASBROPgLvkUYybwf2WDI2xTdKUuAMI0rTpMH1IzjPRP/J+m
[delete]RQTiJe1mRX3rJCy3qVooVzsLe2yJ1+rs3FzLSfQhazK6PqMD8GhpqHO0Y75
[delete]LEA/qdOCrTFjosO9C3j3WPCW8lutTxf/QsoKGkIVs5tjAgMBAAGgKTAnBgkq
[delete]0BCQ4xGjAYMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgXgMA0GCSqGSIb3DQEB
[delete]A4GBAKMLVFilRjI9xvU/vZmP69yReVZyxa5YVpF/cEvwFwbOU6E4USkdONGT
[delete]DRj1XxfzYD+CDf8TVuTY4tapaLvKPRUtdd/mM1PidY5t126QAObyKjHBRzy
[delete]RJFQeP+0ktxcYJ99+sfiescwR/qzPJM58i6daqmMamQBZi
  -----END CERTIFICATE REQUEST-----
Sign up for cacert.org use

Goto http://www.cacert.org/

Sign up - https://www.cacert.org/index.php?id=1

Verify the email the confirmation email.

Add a domain , it will send an email to some "admin" account to the domain. Make sure you can get it before you do this.

Get New Server Certificate

Now do a New Server Certificate from your administration page at http://www.cacert.org/

You'll copy in the /opt/zimbra/ssl/zimbra/commercial/commercial.csr contents in window [all of it].

It'll generate your cert. on the webpage.

Copy this onto the server [paste in cert details]:

vi /opt/zimbra/ssl/zimbra/commercial/commercial.crt
-----BEGIN CERTIFICATE-----
[delete]TCCAl2gAwIBAgIDBPRRMA0GCSqGSIb3DQEBBQUAMHkxEDAOBgNVBAoTB1Jv
[delete]0ExHjAcBgNVBAsTFWh0dHA6Ly93d3cuY2FjZXJ0Lm9yZzEiMCAGA1UEAxMZ
[delete]2VydCBTaWduaW5nIEF1dGhvcml0eTEhMB8GCSqGSIb3DQEJARYSc3VwcG9y
[delete]NlcnQub3JnMB4XDTA4MDQxMTIwMjQzNloXDTA4MTAwODIwMjQzNlowJjEk
[delete]1UEAxMbbWFpbDMuaW50ZXJuYWwuaG9tZXVuaXguY29tMIGfMA0GCSqGSIb3
[delete]AQUAA4GNADCBiQKBgQDq4vf5ukbTNbwEgUTj4C75FGMm8H9lgyNsU3SlLgDC
[delete]B9SM4z0T/yfpoZc3yUUE4iXtZkV96yQst6laKFc7C3tsidfq7Nxcy0n0IWs
[delete]BoaahztGO+ZgsfCxAP6nTgq0xY6LDvQt491jwlvJbrU8X/0LKChpCFbOb
[delete]QABo4HcMIHZMAwGA1UdEwEB/wQCMAAwNAYDVR0lBC0wKwYIKwYBBQUHAwIG
[delete]QUFBwMBBglghkgBhvhCBAEGCisGAQQBgjcKAwMwCwYDVR0PBAQDAgWgMDMG
[delete]QUFBwEBBCcwJTAjBggrBgEFBQcwAYYXaHR0cDovL29jc3AuY2FjZXJ0Lm9y
[delete]QYDVR0RBEowSIIbbWFpbDMuaW50ZXJuYWwuaG9tZXVuaXguY29toCkGCCsG
[delete]wgFoB0MG21haWwzLmludGVybmFsLmhvbWV1bml4LmNvbTANBgkqhkiG9w0B
[delete]AAOCAgEANzr/jRcEd5BF2QqF+X8deq4Xxp1tN9lFgji38C5ixNJ+Busq9Sk5
[delete]O7YYJQbSf5K14eZyC1jaNEOEwqgzFiM1HRWL1HCca3EM7TXUoH8sMXS1Ng
[delete]M5oyfQcFWZYa22CtKQANQEX5l7EYNkr0yvD/YnP02l3hk1jZr+3pszCW6Iw
[delete]vabHMYcAXus+iOGgws788QsMaqzoZwla1AaacZ98s0lFAR0xdRiuXCHUFz
[delete]meS5sK+med95/z+Mb6ShJzC7KAi1nfZk9CoNHUHVxMis5Cr+GT7MoIvhQ
[delete]8fkiANQQoEgam37lyHezPKyc6iLxW4ag2PWKrZa2+3pyTg/6aHKxZR325z
[delete]kcdwKYo/eUGaN1tNmsY638N4hCz01FHHKr97W0m4u5wtwKBo4/5Gy9e5nG6
[delete]khOyjfOz6VYvZHNqDaqGJwsxitxSGGDc8bA+9d73RCOFuztwVrKYg5OJ
[delete]Ei5C9gWzee7AmoGpgxOrYjgBrx4nuBw71EFzgKSOZqxUxSNiLuGAx+oVd2
[delete]Z4EAPsa90ZNb0mLGagAuTAdccekOqPVnyZrqiINelY7fpAAUvO9rgTSB9A
[delete]RxUydTgY1jyBtoXjp59HMVbCkAtOtX43NqIhPYJNPeSoyw/5SU=
-----END CERTIFICATE-----
Get Root CA's

Root CA certs are found here  : https://www.cacert.org/index.php?id=3

Do the following on the server:

[ su - root ; cd /opt/zimbra/ssl/zimbra/commercial/ ]
wget http://www.cacert.org/certs/root.crt
mv root.crt commercial_ca.crt
Verify

Let's verify all is good.

[ su - root ; cd /opt/zimbra/ssl/zimbra/commercial/ ]
/opt/zimbra/bin/zmcertmgr verifycrt comm commercial.key commercial.crt

Giving something like this:

** Verifying commercial.crt against commercial.key
Certificate (commercial.crt) and private key (commercial.key) match.
Valid Certificate: commercial.crt: OK
Errors - Double check time
  • Date [is it wrong?]
    • Install ntpd if it's not
    • Stop ntpd  :
      • /etc/init.d/ntpd stop
    • Set time with  :
      • ntpdate us.pool.ntp.org
    • Confirm time change  :
      • date
    • Confirm hardware clock time  :
      • hwclock
    • Sync hardware clock time  :
      • hwclock --systohc
    • Confirm hardware clock time  :
      • hwclock
    • Start ntpd now  :
      • /etc/init.d/ntpd start

Deploy CA

[ su - root ; cd /opt/zimbra/ssl/zimbra/commercial/ ]

/opt/zimbra/bin/zmcertmgr deploycrt comm commercial.crt commercial_ca.crt

Restart the webserver

su - zimbra
zmmailboxdctl restart


GAL (Server) Topics

   KB 2494        Last updated on 2008-11-16  




0.00
(0 votes)
24px ‎  - This is Zeta Alliance Certified Documentation. The content has been tested by the Community.

Actual GAL (Server) Topics Notes Homepage

Please see Ajcody-Notes-ServerGAL

GAL Behavior Bugs/RFE's

GAL And Alises

Please see the following:

Canonical Addresses - Beyond GAL Issues Really

Please see the following:

External GAL Source

Please see:

Setting/Checking GAL Varaibles

CLI To Sync Gal On Server

Truthfully, I'm not sure of the reasoning behind the command but this might be useful in trouble shooting connector issues with GAL sync issues.

zmprov syg DOMAINNAME.com

4.5.x versions will need quotes at the end.

zmprov syg DOMAINAME.com ""

My Gal & LDAP Settings For A Domain

To see your setting, do the following - replacing with domainname with the domain in question.

su - zimbra
zmprov gd [domainname] zimbraGalLdapSearchBase
zmprov gd [domainname] zimbraGalSyncLdapSearchBase

You'll see more GAL/LDAP related variables with:

zmprov gd domainname | egrep -i 'ldap|gal'

They are set using:

zmprov md [domainname] zimbraGalLdapSearchBase variable
zmprov md [domainname] zimbraGalSyncLdapSearchBase variable

Want My GAL To See All Domains

The default of a domain GAL [zimbraGalInternalSearchBase] is to see only it's domain. To have the GAL for a domain to see all domains on the server you need to set the variable to ROOT.

To see the existing setting:

zmprov gd [domainname] zimbraGalLdapSearchBase

To change the variable for the domain:

zmprov md [domainname] zimbraGalInternalSearchBase ROOT

All global change would be done with:

zmprov mcf zimbraGalInternalSearchBase ROOT
Some Other Ideas - Especially With Multiple Domains And ZCS Servers

This is just some notes I made for a specific case where there were multiple ZCS installs across sister companies. They didn't have a multi-server install and weren't sharing LDAP data therefore.

Here's some ideas I'm considering for this situation:

  1. Dump GAL data from each server and combine date to either:
    1. an external LDAP server you can then configure the ZCS servers to use as an external GAL
      • admin console > domain > GAL > Configure GAL > Both or External
    2. reformat GAL data so it can then be imported as an address book via the CLI into a sharead adddressbook on each server. Setup command to run from cron.
  2. Just setup an external Openldap server (which would have replica's at each site) that can be used as an external GAL - make this authoritative in your company. Then setup the ZCS to use it as an external GAL.
  3. To actually move your infrastructure of ZCS servers into one primary ZCS configuration. Instead of having each site/domain having it's own primary LDAP/ZCS server they would become LDAP replica servers with their domain mailstore's being local to each site. Configure each site/domain to use it's own MTA - if you want/need. And then internally to Zimbra you could set the GAL to be ROOT rather than DOMAIN.

Some bugs to review that touch on this:

I'm sure there's some other possibilities.

GAL Related Attributes Usage

This is from earlier 5.x versions, newer version might include more variables.

Attribute                                 scope                  For              GAL op
                                                                 - zimbra         - autocomplete
                                                                 - external       - serarch
                                                                 - both           - sync
                                                                                  - all
======================================================================================================================================================================
zimbraHideInGal                           account,DL,CR          zimbra           all
zimbraFeatureGalEnabled                   account,cos            both             search,sync
zimbraFeatureGalAutoCompleteEnabled       account,cos            both             autocomplete
zimbraPrefGalAutoCompleteEnabled          account,cos            both             autocomplete
zimbraGalMode                             domain                 both             all
zimbraGalLdapFilterDef                    globalConfig           both             all
zimbraGalLdapAttrMap                      domain,globalConfig    both             all
zimbraGalMaxResults                       domain,globalConfig    both             autocomplete, search (for sync, system does not specify a max when searching LDAP)
zimbraGalTokenizeAutoCompleteKey          domain                 both             autocomplete
zimbraGalTokenizeSearchKey                domain                 both             search

zimbraGalLdapPageSize                     domain,globalConfig    both             autocomplete, search, (sync if zimbraGalSyncLdapPageSize is not set)
zimbraGalSyncLdapPageSize                 domain,globalConfig    both             sync

----------------------------------------------------------------------------------------------------------------------------------------------------------------------
zimbraGalInternalSearchBase               domain,globalConfig    zimbra           autocomplete, search, (sync if zimbraGalSyncInternalSearchBase is not set)
zimbraGalSyncInternalSearchBase           domain,globalConfig    zimbra           sync

----------------------------------------------------------------------------------------------------------------------------------------------------------------------
zimbraGalLdapURL                          domain                 external         autocomplete, search, (sync if zimbraGalSyncLdapURL is not set)
zimbraGalSyncLdapURL                      domain                 external         sync

zimbraGalLdapSearchBase                   domain                 external         autocomplete, search, (sync if zimbraGalSyncLdapSearchBase is not set)
zimbraGalSyncLdapSearchBase               domain                 external         sync

zimbraGalLdapAuthMech                     domain                 external         autocomplete, search, (sync if zimbraGalSyncLdapAuthMech is not set)
zimbraGalSyncLdapAuthMech                 domain                 external         sync

zimbraGalLdapBindDn                       domain                 external         autocomplete, search, (sync if zimbraGalSyncLdapBindDn is not set)
zimbraGalSyncLdapBindDn                   domain                 external         sync

zimbraGalLdapBindPassword                 domain                 external         autocomplete, search, (sync if zimbraGalSyncLdapBindPassword is not set)
zimbraGalSyncLdapBindPassword             domain                 external         sync

zimbraGalLdapKerberos5Principal           domain                 external         autocomplete, search, (sync if zimbraGalSyncLdapKerberos5Principal is not set)
zimbraGalSyncLdapKerberos5Principal       domain                 external         sync

zimbraGalLdapKerberos5Keytab              domain                 external         autocomplete, search, (sync if zimbraGalSyncLdapKerberos5Keytab is not set)
zimbraGalSyncLdapKerberos5Keytab          domain                 external         sync

zimbraGalAutoCompleteLdapFilter           domain,globalConfig    external         autocomplete
zimbraGalLdapFilter                       domain                 external         search, (sync if zimbraGalLdapSyncFilter is not set)
zimbraGalSyncLdapFilter                   domain                 external         sync




Zimlets

   KB 2494        Last updated on 2008-11-16  




0.00
(0 votes)
24px ‎  - This is Zeta Alliance Certified Documentation. The content has been tested by the Community.
Warning: You are looking at legacy Zimlet documentation. For Zimbra Modern UI Zimlet development go to: https://wiki.zimbra.com/wiki/DevelopersGuide#Zimlet_Development_Guide.


Actual Zimlet Notes Homepage

Please see Ajcody-Zimlet-Notes

Zimlets In ZCS 8+

List Of Supported Zimlets

$ find /opt/zimbra/zimlets -name *.zip -print
/opt/zimbra/zimlets/com_zimbra_url.zip
/opt/zimbra/zimlets/com_zimbra_tooltip.zip
/opt/zimbra/zimlets/com_zimbra_clientuploader.zip
/opt/zimbra/zimlets/com_zimbra_srchhighlighter.zip
/opt/zimbra/zimlets/com_zimbra_viewmail.zip
/opt/zimbra/zimlets/com_zimbra_ymemoticons.zip
/opt/zimbra/zimlets/com_zimbra_date.zip
/opt/zimbra/zimlets/com_zimbra_email.zip
/opt/zimbra/zimlets/com_zimbra_adminversioncheck.zip
/opt/zimbra/zimlets/com_zimbra_bulkprovision.zip
/opt/zimbra/zimlets/com_zimbra_attachmail.zip
/opt/zimbra/zimlets/com_zimbra_cert_manager.zip
/opt/zimbra/zimlets/com_zimbra_webex.zip
/opt/zimbra/zimlets/com_zimbra_phone.zip
/opt/zimbra/zimlets/com_zimbra_attachcontacts.zip
/opt/zimbra/zimlets/com_zimbra_proxy_config.zip


$ find /opt/zimbra/zimlets-network -name *.zip -print
/opt/zimbra/zimlets-network/com_zimbra_backuprestore.zip
/opt/zimbra/zimlets-network/com_zimbra_archive.zip
/opt/zimbra/zimlets-network/com_zimbra_click2call_mitel.zip
/opt/zimbra/zimlets-network/com_zimbra_voiceprefs.zip
/opt/zimbra/zimlets-network/com_zimbra_click2call_cisco.zip
/opt/zimbra/zimlets-network/com_zimbra_smime_cert_admin.zip
/opt/zimbra/zimlets-network/com_zimbra_convertd.zip
/opt/zimbra/zimlets-network/com_zimbra_smime.zip
/opt/zimbra/zimlets-network/com_zimbra_license.zip
/opt/zimbra/zimlets-network/com_zimbra_delegatedadmin.zip
/opt/zimbra/zimlets-network/com_zimbra_mobilesync.zip
/opt/zimbra/zimlets-network/com_zimbra_xmbxsearch.zip
/opt/zimbra/zimlets-network/com_zimbra_cluster.zip
/opt/zimbra/zimlets-network/com_zimbra_hsm.zip
/opt/zimbra/zimlets-network/com_zimbra_ucconfig.zip

Undeploy All Zimlets And Redeploy Supported Zimlets

Let's flush the cache in regards to zimlets before we start:

zmprov fc zimlet

Create a text file of your currently deployed zimlets . Note - save this for future reference so you can recall what was removed.

/opt/zimbra/bin/zmzimletctl listZimlets

and

ls /opt/zimbra/zimlets-deployed 

Add the zimlets to a file - This Is An Example Below - Be Sure Your Text File Lists YOUR Zimlets:

vi /tmp/list
com_zimbra_adminversioncheck
com_zimbra_apptsummary
com_zimbra_attachcontacts
com_zimbra_attachmail
com_zimbra_backuprestore
com_zimbra_bulkprovision
com_zimbra_cert_manager
com_zimbra_coloredemails
com_zimbra_contactcleaner
com_zimbra_contactorganizer
com_zimbra_convertd
com_zimbra_date
com_zimbra_delegatedadmin
com_zimbra_dnd
com_zimbra_email
com_zimbra_emailreminder
com_zimbra_emailtemplates
com_zimbra_hsm
com_zimbra_license
com_zimbra_linkedin
com_zimbra_local
com_zimbra_meebo
com_zimbra_mobilesync
com_zimbra_smime
com_zimbra_social
com_zimbra_srchhighlighter
com_zimbra_stickynotes
com_zimbra_tracking
com_zimbra_url
com_zimbra_webex
com_zimbra_xmbxsearch
com_zimbra_ycurrency
com_zimbra_yfinance
com_zimbra_ymaps
com_zimbra_ymemoticons

Use the file to undeploy:

  for i in `cat /tmp/list`; do ; zmzimletctl undeploy $i ; done

To deploy the basic supported zimlets:

vi /tmp/install-list
/opt/zimbra/zimlets/com_zimbra_url.zip
/opt/zimbra/zimlets/com_zimbra_tooltip.zip
/opt/zimbra/zimlets/com_zimbra_clientuploader.zip
/opt/zimbra/zimlets/com_zimbra_srchhighlighter.zip
/opt/zimbra/zimlets/com_zimbra_viewmail.zip
/opt/zimbra/zimlets/com_zimbra_ymemoticons.zip
/opt/zimbra/zimlets/com_zimbra_date.zip
/opt/zimbra/zimlets/com_zimbra_email.zip
/opt/zimbra/zimlets/com_zimbra_adminversioncheck.zip
/opt/zimbra/zimlets/com_zimbra_bulkprovision.zip
/opt/zimbra/zimlets/com_zimbra_attachmail.zip
/opt/zimbra/zimlets/com_zimbra_cert_manager.zip
/opt/zimbra/zimlets/com_zimbra_webex.zip
/opt/zimbra/zimlets/com_zimbra_phone.zip
/opt/zimbra/zimlets/com_zimbra_attachcontacts.zip
/opt/zimbra/zimlets/com_zimbra_proxy_config.zip
/opt/zimbra/zimlets-network/com_zimbra_backuprestore.zip
/opt/zimbra/zimlets-network/com_zimbra_smime_cert_admin.zip
/opt/zimbra/zimlets-network/com_zimbra_convertd.zip
/opt/zimbra/zimlets-network/com_zimbra_smime.zip
/opt/zimbra/zimlets-network/com_zimbra_license.zip
/opt/zimbra/zimlets-network/com_zimbra_delegatedadmin.zip
/opt/zimbra/zimlets-network/com_zimbra_mobilesync.zip
/opt/zimbra/zimlets-network/com_zimbra_xmbxsearch.zip
/opt/zimbra/zimlets-network/com_zimbra_hsm.zip

Those that are absent from the above list from /opt/zimbra/zimlets & /opt/zimbra/zimlets-network are:

/opt/zimbra/zimlets-network/com_zimbra_archive.zip
/opt/zimbra/zimlets-network/com_zimbra_click2call_mitel.zip
/opt/zimbra/zimlets-network/com_zimbra_voiceprefs.zip
/opt/zimbra/zimlets-network/com_zimbra_click2call_cisco.zip
/opt/zimbra/zimlets-network/com_zimbra_cluster.zip
/opt/zimbra/zimlets-network/com_zimbra_ucconfig.zip

To deploy the zimlets from /tmp/install-list :

   for i in `cat /tmp/install-list`; do ; zmzimletctl deploy $i ; done

Or to deploy ALL supported zimlets:

  cd /opt/zimbra/zimlets
  for i in `ls` ; do zmzimletctl deploy $i ; done
  cd /opt/zimbra/zimlets-network
  for i in `ls` ; do zmzimletctl deploy $i ; done

Flush the cache again in regards to zimlets:

zmprov fc zimlet

Depending on issue or further troubles, you might want to restart the mailboxd service:

zmmailboxdctl restart

Do a current listing of your installed zimlets and confirm it's what you expect:

/opt/zimbra/bin/zmzimletctl listZimlets

Zimlet Changes In ZCS 6

New Directory Path For Deployed Zimlets

From ZCS 5, it was:

zmlocalconfig zimlet_directory
zimlet_directory = ${mailboxd_directory}/webapps/service/zimlet
**where mailboxd was /opt/zimbra/jetty/**

Under ZCS 6:

zimlet_directory = zimlet_directory = /opt/zimbra/zimlets-deployed

The related bug/rfe:

Can't Deploy Zimlets - Admin Or Others

This variable & directory seems to have been dropped with 6.0.5+ .

If you can't deploy zimlets and mailbox.log is logging an error about unable to locate file, check the following exists.

drwxr-xr-x   17 zimbra  zimbra  578 Nov  4 13:55 /opt/zimbra/zimlets-properties
zmlocalconfig zimlet_properties_directory
zimlet_properties_directory = /opt/zimbra/zimlets-properties

Location Of Zimlets

Zimlets should be already located on the zimbra server in one of these directories:

/opt/zimbra/zimlets/
/opt/zimbra/zimlets-admin-extra/
/opt/zimbra/zimlets-extra/
/opt/zimbra/zimlets-experimental/
/opt/zimbra/zimlets-network/

How To List Currently Installed Zimlets

Do the following:

zmzimletctl listZimlets

You can also see them in the admin console.

Configuration > Zimlets

Configuration > Admin Extensions

How To Deploy Zimlets

To deploy a zimlet, simply cd to the directory where the zimlet is located and issue this command:

/opt/zimbra/bin/zmzimletctl deploy <zimlet_name>

Something like:

zmzimletctl deploy /opt/zimbra/zimlets-extra/com_zimbra_ycurrency.zip

You can also deploy them via the admin console.

Configuration > Zimlets

Configuration > Admin Extensions

How To Undeploy / Uninstall Zimlets

See how the zimlet is named:

zmzimletctl listZimlets

Now run the following with the naming convention used from the above output:

zmzimletctl undeploy com_zimbra_ycurrency

You can also undeploy them via the admin console.

Configuration > Zimlets

Configuration > Admin Extensions

Samba & Posix Zimlet - ZCS 6x

Main Samba & Posix How-To Reference

Please see:

Important Bugs-RFE's Related To Samba Posix Issues

Please see:

Samba & Posix Zimlet - ZCS 5x

Main Samba & Posix How-To Reference

Please see:

Important Bugs-RFE's Related To Samba Posix Issues

Please see:

  • Other Issues
    • "Have the Unix Windows LDAP Samba extensions installed and configured by default"
    • "Suggestions to improve Posix and Samba Zimlets"
      • Items requested:
        • a) Add an option to expire the Samba password to force them change the password.
        • b) In the memberuid option under Posix Groups: Could you add an option to allow the users to select a single or multiple zimbra users to fill them up quickly?
        • c) Add an additional button to display all users that belongs to this particular Posix Group.
        • d) Add an additional button in the user profile screen that displays all the groups that he/she belongs to.
      • http://bugzilla.zimbra.com/show_bug.cgi?id=18141

Samba - LDAP - Overlays

We don't [officially] support running additional overlays with OpenLDAP.

SLAPO-RWM OVERLAY RWM

slapo-rwm is known to be buggy in OpenLDAP 2.3.43 and continues to be buggy to this day in OpenLDAP 2.4. It certainly won't work with ZCS 5.0.16.

We would advise customers to avoid using it until it stabilizes, though they need to understand it's still going to be unsupported by us.

Where one places "overlay rwm" in the slapd.conf file has been known to cause issues as well. RWM has problems in the order in which it is loaded. There are at least 2 open bugs currently in the OpenLDAP ITS tracker.

Can't Manage Users After Removing Samba & Posix Zimlet

This is after you have removed the samba & posix zimlets and now can't see or manage old accounts in the admin ui. You might need to remove the samba/posix references in each user account. You'll need the nis.schema and samba.schema configured for ldap for this to work.

Untested comment, 3 things needed for this.

1. deployed samba/posix zimlet

2. add/have the samba/nis schema

3. add/have the extra oc's

zmprov mcf +zimbraAccountExtraObjectClass posixAccount 
zmprov mcf +zimbraAccountExtraObjectClass sambaSamAccount

If you do this on one of the old accounts:

zmprov ga user@domain.com

And you see:

objectClass: posixAccount
objectClass: sambaSamAccount

These steps might need to be done.

Create a file called mod.ldif . Modify the dn line - dn: uid=posix1,ou=people,dc=testdomain,dc=com - for your server and user.

# posix1, people, testdomain.com
dn: uid=posix1,ou=people,dc=testdomain,dc=com
changetype: modify
delete: objectClass
objectClass: posixAccount
-
delete: objectClass
objectClass: sambaSamAccount
-
delete: uidNumber
-
delete: gidNumber
-
delete: loginShell
-
delete: sambaAcctFlags
-
delete: sambaSID
-
delete: homeDirectory
-
delete: sambaNTPassword

# posix2, people, testdomain.com
dn: uid=posix2,ou=people,dc=testdomain,dc=com
changetype: modify
delete: objectClass
objectClass: posixAccount
-
delete: objectClass
objectClass: sambaSamAccount
-
delete: uidNumber
-
delete: gidNumber
-
delete: loginShell
-
delete: sambaSID
-
delete: homeDirectory
-
delete: sambaNTPassword

Then run a command similar to this, modify it for your environment:

ldapmodify -D uid=zimbra,cn=admins,cn=zimbra -w PassWord -H ldap://ldapmaster.hostname.com:389 -x -f /tmp/mod.ldif
Problems With The Above Steps?

Please see :

http://wiki.zimbra.com/index.php?title=King0770-Notes#LDAP_-_Export_.26_Reimport

It should be possible to modify the dump and the re-import. This has not been tested yet, though.

Steps done in one test. Please note, you'll still need to visually review the ldap file to see what lines need to be remove, this can't be scripted out.

-as zimbra-
zimbra$ libexec/zmslapcat /tmp/ldap
zimbra$ egrep -iv 'sambaSamAccount|posixAccount|uidNumber|gidNumber|loginShell|sambaAcctFlags|sambaSID|homeDirectory|sambaNTPassword' ldap.bak > ldap.bak2
zimbra$ egrep -i 'samba|posix' ldap.bak2
zimbra$ vi ldap.bak2
zimbra$ egrep -i 'loginshell|HOMEDIRECTORY|MEMBERUID' ldap.bak2
zimbra$ vi ldap.bak2
zimbra$ zmcontrol stop
zimbra$ ps -aux | grep slapd

6) su - root
7) mv /opt/zimbra/openldap-data /opt/zimbra/openldap-data.OLD
8) mkdir -p /opt/zimbra/openldap-data/logs
9) chown -R zimbra:zimbra /opt/zimbra/openldap-data/
10) su - zimbra

zimbra$ cd /opt/zimbra/openldap-data.OLD
zimbra$ cp DB_CONFIG ../openldap-data/
zimbra$ cd
zimbra$ /opt/zimbra/openldap/sbin/slapadd -f /opt/zimbra/conf/slapd.conf -l /tmp/ldap/ldap.bak2 
The first database does not allow slapadd; using the first available one (2)
is_entry_objectclass("cn=IT,ou=groups,dc=XXXXX,dc=com", "2.16.840.1.113730.3.2.6") no objectClass attribute
slapadd: dn="cn=IT,ou=groups,dc=XXXXX,dc=com" (line=11179): no objectClass attribute
zimbra$ zmcontrol start

### Output ###
assigned-72-29-183-240:~ zimbra$ libexec/zmslapcat /tmp/ldap2/
UNKNOWN attributeDescription "LOGINSHELL" inserted.
UNKNOWN attributeDescription "HOMEDIRECTORY" inserted.
UNKNOWN attributeDescription "SAMBAACCTFLAGS" inserted.
UNKNOWN attributeDescription "SAMBASID" inserted.
UNKNOWN attributeDescription "SAMBADOMAINNAME" inserted.
UNKNOWN attributeDescription "SAMBANTPASSWORD" inserted.
UNKNOWN attributeDescription "SAMBAALGORITHMICRIDBASE" inserted.
UNKNOWN attributeDescription "SAMBANEXTUSERRID" inserted.
UNKNOWN attributeDescription "SAMBAMINPWDLENGTH" inserted.
UNKNOWN attributeDescription "SAMBALOGONTOCHGPWD" inserted.
UNKNOWN attributeDescription "SAMBAMAXPWDAGE" inserted.
UNKNOWN attributeDescription "SAMBAMINPWDAGE" inserted.
UNKNOWN attributeDescription "SAMBALOCKOUTDURATION" inserted.
UNKNOWN attributeDescription "SAMBALOCKOUTOBSERVATIONWINDOW" inserted.
UNKNOWN attributeDescription "SAMBALOCKOUTTHRESHOLD" inserted.
UNKNOWN attributeDescription "SAMBAFORCELOGOFF" inserted.
UNKNOWN attributeDescription "SAMBAREFUSEMACHINEPWDCHANGE" inserted.
UNKNOWN attributeDescription "SAMBAPWDHISTORYLENGTH" inserted.
UNKNOWN attributeDescription "SAMBAGROUPTYPE" inserted.
UNKNOWN attributeDescription "MEMBERUID" inserted.
UNKNOWN attributeDescription "SAMBAPASSWORDHISTORY" inserted.
UNKNOWN attributeDescription "SAMBAPWDLASTSET" inserted.

I Lost My Users After An Upgrade - Samba Posix Zimlet

This is from a case I saw and how a customer fixed it.

In order to fix this we did.
1. Replace slapd.conf.in
2. restarted zimbra (not sure if this is necessary, but its what we did.)
3. zmprov mcf +zimbraAccountExtraObjectClass posixAccount
4. zmprov mcf +zimbraAccountExtraObjectClass sambaSamAccount
5. ldap stop && ldap start && ldap stop && ldap start
The odd thing is, I looked in the zmprov gcf originally and posixAccount and sambaSamAccount had already been added?

Upgrade Or Installation Of New Package On Zimbra Broken Samba - Another Situation

And customer was kind enough to write up a summary of our session when troubling shooting this. We believe it will be of use for others.

Brief overview of your configuration/setup

The Zimbra server is used as LDAP master server. It's easy to maintain and it's very easy to manage hybrid Zimbra/Posix/Samba user accounts. It provides LDAP service for Samba 3.0.24 and PAM on Debian Etch 4.0 servers and desktops. Additionally, I've got few LDAP replica servers which use syncrepl mechanism to get required Posix and Samba data from Zimbra server. As Posix and Samba objects are in use it's very important to keep the tweaked /opt/zimbra/conf/slapd.conf.in file the same after upgrade/ re-installation.

System spec:

  • OS is Ubuntu 7.10 with all latest patches
  • Zimbra 5.0.11
  • LDAP 2.3.43.5z (/opt/zimbra/conf/slapd.conf configuration file)
  • Zimbra zimlets-admin-extra: zimbra_posixaccount, zimbra_samba
  • native packages Samba 3.0.24 on Debian 4.0 Etch with PAM and libnss-ldap, pam-ldap
  • smbldap-tools 0.9.5 from tar file
The symptoms and what you did to confirm the issue
Operations that failed at end-user & admin

Because of the other issue with Zimbra server we had to install convertd on the box. To avoid any other unknown problems we were advised to re-run installation script from the zcs-NETWORK-5.0.11_GA_2695.UBUNTU6.20081117023813 folder on local file system.

During the process we confirmed installation of 'convertd'. So, from this point of time it's installed. The Zimbra installer restarted a few times slapd service. I'm not sure but I believe that during this process it dumps whole LDAP objects and clears it in directory. Then it loads them back to directory.

The odd thing is that THE INSTALLER RE-GENERATES the '/opt/zimbra/conf/slapd.conf.in' file to the standard one from the new package.

This is the real source of the problem for other objects than Zimbra's ones. i.e. Posix and Samba

After I restored the changes required for Posix and Samba in /opt/zimbra/conf/slapd.conf.in' file as described on Zimbra Wiki it seems that Samba and Posix attributes were unaccessible on all clients. ( UNIX_and_Windows_Accounts_in_Zimbra_LDAP_and_Zimbra_Admin_UI )

Symptoms

[ Documentation substituted 'my_corp.net' for real domain name ]


1. "Samba can't join any new box to domain MY_CORP" but this is just tip of the iceberg.

# grep machine /etc/samba/smb.conf
  ldap machine suffix = ou=machines
  add machine script = /usr/sbin/smbldap-useradd -t 0 -W "%u"

Firstly, I've tried to test existing object I know in LDAP directory via ldapsearch and it's fails giving zero results.

1A.
zimbra# ldapsearch -x -D cn=config -W -h <name-of-zimbra-server> -b ou=machines,dc=my_corp,dc=net uid=my-pc$

I'm trying to query the object itself and then it works which means that all attributes are in directory.

1B.
zimbra# ldapsearch -x -D cn=config -W  -h <name-of-zimbra-server> -b uid=my-pc$, ou=machines,dc=my_corp,dc=net
  # extended LDIF
  #
  # LDAPv3
  # base <uid=my-pc$,ou=machines,dc=my_corp,dc=net> with scope subtree
  # filter: (objectclass=*)
  # requesting: ALL
  #
  # my-pc$, machines, my_corp.net
  dn: uid=my-pc$,ou=machines,dc=my_corp,dc=net
  cn: my-pc$
  uid: my-pc$
  uidNumber: 1001
  gidNumber: 515
  loginShell: /bin/false
  description: Computer
  gecos: Computer
  objectClass: posixAccount
  objectClass: account
  objectClass: sambaSamAccount
  displayName: my-pc$
  sambaDomainName: MY_CORP
  sambaSID: S-1-5-21-XXXXXXX-XXXXXXXXXX-XXXXX-3002

1C. One more test to be sure and no results as well:

zimbra# ldapsearch -x -D cn=config -W  -h <name-of-zimbra-server> -b dc=my_corp,dc=net uid=my-pc$

1D. Then tests on Samba side.

pdc# pdbedit -Lv my-pc$

  smbldap_search_domain_info: Searching for:[(&(objectClass=sambaDomain)(sambaDomainName=MY_CORP))]
  smbldap_open_connection: connection opened
  smbldap_search_domain_info: Searching for:[(&(objectClass=sambaDomain)(sambaDomainName=MY_CORP))]
  smbldap_open_connection: connection opened
  init_sam_from_ldap: Entry found for user: my-pc$
  Unix username:        my-pc$
  NT username:          my-pc$
  Account Flags:        [W          ]
  User SID:             S-1-5-21-XXXXXXX-XXXXXXXXXX-XXXXX-3002
  pdb_get_group_sid: Failed to find Unix account for my-pc$    
 *Primary Group SID:    (NULL SID)                               # THIS FAILS as well
  Full Name:            my-pc$
  Home Directory:       \\pdc\my-pc_
  HomeDir Drive:        F:
  Logon Script:         logon.cmd
  Profile Path:
  Domain:               MY_CORP
  Account desc:         Computer
  Workstations:
  Munged dial:
  Logon time:          
  Logoff time:          Tue, 19 Jan 2038 03:14:07 GMT
  Kickoff time:         Tue, 19 Jan 2038 03:14:07 GMT
  Password last set:    Mon, 05 Jan 2009 04:20:59 GMT
  Password can change:  
  Password must change: Tue, 19 Jan 2038 03:14:07 GMT
  Last bad password   :
  Bad password count  :
  Logon hours         : FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF

So Samba finds the object but can't link it to the Posix account via PAM query.

1E. Let's check also Posix attribute on Samba server.

pdc# getent passwd my-pc$

Nothing.

After I switched the log level to 5 in Samba I'd see the communication with LDAP in log.my-pc file on Samba server.

pdc# grep "log level" /etc/samba/smb.conf
  log level = 5

pdc# grep -v "\[200" /var/log/samba/log.my-pc|less

It turns out that the Samba's search base for machines does not work (see 1A,1C tests above).

I checked also other objects in other branches and situation was similar for ou=people ,ou=machines, ou=groups. No sambaSID or uidNumber/gidNumber attributes were visible for Samba in 'dc=my_corp,dc=net' search base.

Steps To Fix Issue

To fix this problem it is necessary to 're-fresh' affected attributes.

In our case:

  • ou=people branch - "sambaSID"
  • ou=machines branch - "sambaSID,uidNumber,gidNumber"
  • ou=groups branch - "sambaSID,gidNumber"
  • ( basedn - root for above branches is dc=my_corp,dc=net)
For ou=people

For ou=people it seems very easy to fix as there is zmprov command you can use to do it.

The syntax should be like this:

zimbra# zmprov ma <full-name>@my_corp.net sambaSID <Samba-SID>

To avoid a manual method (mistype), it's possible to use ldapsearch to create a ready zmprov command list. It requires egrep, awk and sed - standard posix tools present in every Linux system.

As the 'zimbra' user on the Zimbra server I'm sending the output to /tmp/sambaSID-refresh.sh file:

zimbra# ldapsearch -x -h <zimbra-server> -D "cn=config" -W -b ou=people,dc=my-corp,dc=net sambaSID=* uid sambaSID|egrep -v "(^#|^dn)" |awk ' /uid:/ {print "zmprov ma "$2"@my_corp.net"}; /sambaSID/ {print "sambaSID " $2};'|sed '/com$/N;s/\n */ /' > /tmp/user-sambaSID-refresh.sh

It gives the possibility to edit the file before you apply it and remove some entries if not relevant.

To apply the changes simply run the file by bash.

zimbra# bash /tmp/user-sambaSID-refresh.sh

Depends on the amount of users it can take a while. For ~300 users it takes approximately 5-10 min. on busy server.

For ou-machines

For ou=machines it's not as easy and requires using ldapmodify tool and ldif file to be created and imported.

According to the ldapmodify manual we need to create file with multiple entries like the one below

dn: uid=my-pc$,ou=machines,dc=my_corp,dc=net
changetype: modify
replace: uidNumber
uidNumber: 1001
-
replace: sambaSID
sambaSID: S-1-5-21-XXXXXXX-XXXXXXXXXX-XXXXX-3002

dn: (.....)

Make sure there is empty line before next 'dn:'.

So this task could be also automated by ldap-tools. The command below will create ldif output we can forward to the /tmp/machine-posix-smb-fix.ldif file.

zimbra# ldapsearch -x -h <zimbra-server-name> -D "cn=config" -W -b ou=machines,dc=my_corp,dc=net sambaSID=* uidNumber sambaSID|egrep -v "(^#)"|awk '/dn:/ {print "\n"$0"\nchangetype: modify"}; /uidNumber:/ {print "replace: uidNumber\nuidNumber: "$2"\n-"};/sambaSID:/ {print "replace: sambaSID\nsambaSID: "$2};' > /tmp/machine-posix-smb-fix.ldif

Please review the /tmp/machine-posix-smb-fix.ldif file as this example assumes the uidNumber attribute comes first then sambaSID one.

Then using ldapmodify we can replace the existing attributes from our file. Change command if necessary.

zimbra# ldapmodify -x -h <zimbra-server-name> -D cn=config -W -f /tmp/machine-posix-smb-fix.ldif
For ou=groups

For ou=groups it is possible to use Zimbra Admin web interface(RECOMMENDED).

If you don't have to many Posix groups you can easily go to Zimbra Admin web interface and click on "Posix Groups" in the menu. Then double-click on the required group and edit the 'gidNumber' and 'sambaSID' adding one extra digit and save. Then open again and return to the previous value and save again.

But as I mentioned before it's very easy to mistype/remove something important. We can use ldapsearch and create the appropriate ldif file as in ou=machines case above.

zimbra# ldapsearch -x -h <zimbra-server-name> -D "cn=config" -W -b ou=groups,dc=my_corp,dc=net sambaSID=* gidNumber sambaSID|egrep -v "(^#)"|awk '/dn:/ {print "\n"$0"\nchangetype: modify"}; /gidNumber:/ {print "replace: gidNumber\ngidNumber: "$2"\n-"};/sambaSID:/ {print "replace: sambaSID\nsambaSID: "$2"\n"};' > /tmp/groups-posix-smb-fix.ldif

Please review the /tmp/groups-posix-smb-fix.ldif file as this example assumes the gidNumber attribute comes first then sambaSID one. Change this command if necessary.

Admin/End-User test that were performed to confirm complete resolution

Last step is to test previously failed searches on Zimbra server. (Step Symptoms 1C from above)

zimbra# ldapsearch -x -h <name-of-zimbra-server> -b dc=my_corp,dc=net uid=my-pc$

This time it gives full list of attributes for my-pc$

Then on Samba server

  pdc# pdbedit -Lv my-pc$

  smbldap_search_domain_info: Searching for:[(&(objectClass=sambaDomain)(sambaDomainName=MY_CORP))]
  smbldap_open_connection: connection opened
  smbldap_search_domain_info: Searching for:[(&(objectClass=sambaDomain)(sambaDomainName=MY_CORP))]
  smbldap_open_connection: connection opened
  init_sam_from_ldap: Entry found for user: my-pc$
  Unix username:        my-pc$
  NT username:          my-pc$
  Account Flags:        [W          ]
  User SID:             S-1-5-21-XXXXXXX-XXXXXXXXXX-XXXXX-3002
  init_group_from_ldap: Entry found for group: 515
  init_group_from_ldap: Entry found for group: 515
  Primary Group SID:    S-1-5-21-XXXXXXX-XXXXXXXXXX-XXXXX-515
  Full Name:            my-pc$
  Home Directory:       \\pdc\my-pc_
  HomeDir Drive:        F:
  Logon Script:         logon.cmd
  Profile Path:
  Domain:               MY_CORP
  Account desc:         Computer
  Workstations:
  Munged dial:
  Logon time:          
  Logoff time:          Tue, 19 Jan 2038 03:14:07 GMT
  Kickoff time:         Tue, 19 Jan 2038 03:14:07 GMT
  Password last set:    Mon, 05 Jan 2009 04:20:59 GMT
  Password can change:  
  Password must change: Tue, 19 Jan 2038 03:14:07 GMT
  Last bad password   :
  Bad password count  :
  Logon hours         : FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF

Let's check also Posix attribute on Samba server. (Step Symptoms 1E from above)

  pdc# getent passwd my-pc$
  my-pc$:*:1001:515:Computer::/bin/false

Great it works.

Now Let's see if we can join new box to the Domain

  pdc# /usr/sbin/smbldap-useradd -t 0 -W my-pc2

  pdc# pdbedit -Lv my-pc2$

  smbldap_search_domain_info: Searching for:[(&(objectClass=sambaDomain)(sambaDomainName=MY_CORP))]
  smbldap_open_connection: connection opened
  smbldap_search_domain_info: Searching for:[(&(objectClass=sambaDomain)(sambaDomainName=MY_CORP))]
  smbldap_open_connection: connection opened
  init_sam_from_ldap: Entry found for user: my-pc2$
  Unix username:        my-pc2$
  NT username:          my-pc2$
  Account Flags:        [W          ]
  User SID:             S-1-5-21-XXXXXXX-XXXXXXXXXX-XXXXX-3004
  init_group_from_ldap: Entry found for group: 515
  init_group_from_ldap: Entry found for group: 515
  Primary Group SID:    S-1-5-21-XXXXXXX-XXXXXXXXXX-XXXXX-515
  Full Name:            my-pc2$
  Home Directory:       \\pdc\my-pc2_
  HomeDir Drive:        F:
  Logon Script:         logon.cmd
  Profile Path:
  Domain:               MY_CORP
  Account desc:         Computer
  Workstations:
  Munged dial:
  Logon time:          
  Logoff time:          Tue, 19 Jan 2038 03:14:07 GMT
  Kickoff time:         Tue, 19 Jan 2038 03:14:07 GMT
  Password last set:    Mon, 05 Jan 2009 04:20:59 GMT
  Password can change:  
  Password must change: Tue, 19 Jan 2038 03:14:07 GMT
  Last bad password   :
  Bad password count  :
  Logon hours         : FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF

  pdc# getent passwd my-pc2$
  my-pc2$:*:1002:515:Computer::/bin/false

Yes, everything seems to be back up and running.

Conclusions (Of Customer)

In my personal opinion Zimbra installer should prevent changes to the /opt/zimbra/conf/slapd.conf.in file. Or if not to all file then just to the 'include /<path>/*.schema' directives.

So, "include /<path>/samba.schema" and "include /<path>/nis.schema" should be populated in the re-generated file. Any other changes seems not important as we can apply them after installation. This is required to ensure that server recognizes these attributes and won't break integrity of data.

Additionally, I'm not sure if there is any better solution. Maybe there is, but this one seem to be the quickest in terms of my knowledge about Zimbra installer behavior.

RFE To Address

Please see:




Mailing Lists, Distribution Lists, And Mailman

   KB 2494        Last updated on 2008-11-16  




0.00
(0 votes)
24px ‎  - This is Zeta Alliance Certified Documentation. The content has been tested by the Community.


Actual Mailing Lists And Mailman Notes Homepage

Please see Ajcody-MailingLists-And-Mailman

Introduction And Choices - When Email Lists Have More Than 1000+ Recipients

If you are planning to use a distro list with 1000 to 3000 or more members, you should consider using a 3rd party mailing list software machine for this. This is the recommendation of the developers according the the bug/rfe's I reference below. According to bug 19153, you have 2 options:

  1. "set up mailing list manager or change the value of virtual_alias_expansion_limit as a customization. We have never tested with > 1000 so this should be done carefully, and will pound LDAP for any messages with lots of recipients."
  2. Mailman integration or another third party mailing list software package. There is a forum thread on how to integrate zcs and mailman. This would be the preferred fix to your issue, also noted in the bug report. We probably won't see Mailman integration in ZCS at least for another year or so, not until 6.0

Main Bug/RFE's to review:

  • "Problems resolving virtual aliases for members of large distribution lists"
    • https://bugzilla.zimbra.com/show_bug.cgi?id=19153
      • Note - this is marked as a duplicate of bug 8968, "Mailman Integration".
      • Comment 3 : "We have never tested with > 1000 so this should be done carefully, and will pound LDAP for any messages with lots of recipients."
        • Note - In ZCS 8, "Release notes: If dynamic lists are used, the default is to allow lists of up to 10,000 members. This can be controlled via the localconfig key postfix_virtual_alias_expansion_limit"
          • This does NOT resolve the issues for large email lists though, the recommendation is still to be using a dedicated mailing list software package.
      • Comment 5 : the answer for large lists is to use mailman. See bug 8968 and bug 21621
  • "Mailman Integration"
  • "Test and document manual mailman integration"


Other Bug/RFE's to review to see the full conversation on this from our developers - Note , it's important to note the bugs they are making duplicates of the main ones mentioned above.


References to Maining list software options:

Restricting Who Can Send To Mailing List

These are unsupport customizations. Please be sure to make backup copies of modifications to prevent lost after upgrades/restores of Zimbra.

Please see:


Also see the following RFE:

Restrict Sending To Certain Domains

Please see:

Add Everyone To Distribution List - Admin Console

RFE was made for this:

Everyone@domain Without Manually Adding Users To A DL

Investigating if this can be done. Please don't attempt anything below, I'm simply keeping track of my "notes" here.

DON'T ATTEMPT ANY OF THIS! THIS WILL MOST LIKELY GET POSTED INTO BUGZILLA AS AN RFE

RFE Made For This:

One Possible Way Described In An External Resource:

main.cf:
there is a line to resolve aliases:
virtual_aias_maps = ldap: $ config_directory / ldap / ldap-aliases.cf

ldap-aliases.cf
that relates to filter:
query_filter = (&(objectclass=mailgroup)(|(mail=%s)(mailalternateaddress=%s)))
result_attribute = mail result_attribute = mail
special_result_attribute = uniqueMember, memberUrl

in LDAP-e for everyone is recording with such attributes:
mail = everyone@domain.ru
memberurl = ldap :///ou=People,o=organization??sub? (&(objectclass=person)(uid=*))
objectclass=mailgroup
  • Had to use the following to get it to work:
receive_override_options = no_address_mappings 
Was a global setting.

Another Possible Way, again an external resource:

Problems Resolving Virtual Aliases For Members Of Large Distribution Lists

See the following bug:

Other wiki page is here - Error_(MTA):_unreasonable_virtual_alias_maps_map_expansion_size

The default expansion that will be done is for 1000 accounts.

Solution

Note, the bug above recommends NOT going over 1000 due to ldap performance. They recommend looking into mailman as an alternative.

Increase the virtual_alias_expansion_limit Postfix parameter to the value of the highest number of distribution list members. All commands run as the zimbra user.

Check configuration

$ postconf virtual_alias_expansion_limit

Set configuration

$ postconf -e virtual_alias_expansion_limit=3000

Restart Postfix

$ postfix stop
$ postfix start

Mailman - Mailing List Manager

Please see the following:

Sympa - Mailing List Manager

This might be another option to Mailman that might even be better.

https://www.sympa.org/overview/features

Someone made this comment on the zimbra-heid-admins@sfu.ca list :

"We recently switched from mailman to sympa, which does the same thing. Sympa allows membership data sources to be defined from SQL queries, ldap queries, and more. I'm very pleased with it."

Some customers are advocating for it over Mailman on this RFE:

Duplicate Emails With Distribution Lists

Note - this has nothing to do with hard links on the filesystem.

Situation arises when a message goes out and the effected user is somehow targeted twice in the To/CC fields by either use of a DL list, multiple DL lists, and/or a direct inclusion of their email address as well.

There was a change between 4.5.x and 5.x . New variable is zimbraMessageIdDedupeCacheSize.

Setting the "zimbraMessageIdDedupeCacheSize" attribute to zero disables this feature.

su - zimbra
zmprov gacf | grep zimbraMessageIdDedupeCacheSize

If you need to change the setting.

zmprov mcf zimbraMessageIdDedupeCacheSize 3000
zmmailboxdctl restart

Related Bugs:

In regards to DL messages sent by the sender only being in their sent box:

Confirming Hard Links With Message Blobs To DL On Filesystem

Please see:



Hostname And DNS Issues

   KB 2494        Last updated on 2008-11-16  




0.00
(0 votes)
24px ‎  - This is Zeta Alliance Certified Documentation. The content has been tested by the Community.


Actual Hostname And DNS Issues Homepage

Please see Ajcody-Hostname-DNS

Hostname resolution issues and testing commands

Allot of failed installs are because administrators are skipping the steps to make sure resolution is working.

Check your /etc/hosts file. You should have the localhost similar to the one below. Your ZCS server name should be replaced with the zimbra.test.com and zimbra fields. The IP address should be the IP address bound to the network interface [ip addr or ifconfig -a to confirm].

127.0.0.1	localhost.localdomain	localhost
192.168.0.1	zimbra.test.com         zimbra

If your ZCS server is behind a firewall or is being NAT'd. Make sure that resolution for the hostname on the ZCS server returns the internal IP information rather than the external IP. To test:

hostname -f
zimbra.test.com

or

domainname -f
zimbra.test.com

As long as one of the above returns the full hostname, use the command for the following. The MX lookups depend on how you are doing your mail domain and server hostname.

host `hostname -f`
 zimbra.test.com has address 192.168.0.1
host -t MX `domainname -f`
 zimbra.test.com mail is handled by 10 zimbra.test.com.
host -t MX `domainname -d`
   test.com mail is handled by 10 zimbra.test.com.

You should also have reverse records (PTR) as well. Replace 192.168.0.1 using your internal ip address, it should return something like:

host 192.168.0.1
1.0.168.192.in-addr.arpa domain name pointer zimbra.test.com

Please review wiki page on split-DNS. This also shows the basics of actually setting up DNS/Bind on Linux:

http://wiki.zimbra.com/index.php?title=Split_DNS

The following is a more complete how-to for setting up DNS/Bind:

http://www.zimbrablog.com/blog/archives/2007/06/making-zimbra-bind-work-together.html/

lmtp_host_lookup

Also review the variable lmtp_host_lookup to see if it applies to your situation:

http://bugzilla.zimbra.com/show_bug.cgi?id=27988

su - zimbra
zmlocalconfig -e postfix_lmtp_host_lookup=native
postfix reload

If the above value doesn't exist in your ZCS version, then you would do the following [Older ZCS version]:

su - zimbra
postconf | grep host_lookup
postconf -e lmtp_host_lookup=native
postfix reload

smtp_host_lookup

Similar to above.

disable_dns_lookups

Similar to above.

Local DNS-BIND Configuration Example

The full blown example on setting up BIND is at Split_DNS article.

Detailed Example Zone Files

This example is from a CentOS5 test box I have behind a firewall. I actually setup a dynamic dns using a valid domainname, I've replaced my "real" domainname below with "example". I'm using DynDNS to do this.

Directory Structure

The default setup is a chroot configuration of BIND.

  • The root of the configuration files is /var/named/chroot/ .
  • The general bind configuration files are in /var/named/chroot/etc/
    • Files you'll have there:
      • db.cache
      • localtime
      • named.conf
      • rndc.key
  • The directory where your zone files will reside is /var/named/chroot/var/named/
    • I have in this directory the following:
      • data (a directory that's empty)
      • slaves (a directory that's empty)
      • internal.example.com.hosts (file for A, NS, MX, CNAME, etc. records)
      • 0.168.192.rev (file for PTR records)
/var/named/chroot/etc/named.conf example

Here's my named.conf file I'm using:

options {
        directory "/etc";
        pid-file "/var/run/named/named.pid";
        };

zone "." {
        type hint;
        file "/etc/db.cache";
        };

zone "internal.example.com" {
        type master;
        file "/var/named/internal.example.com.hosts";
        };
zone "0.168.192.in-addr.arpa" {
        type master;
        file "/var/named/0.168.192.rev";
        };
/var/named/chroot/var/named/internal.example.com.hosts

The _xmpp-server._tcp entries are related to jabber/xmpp and shouldn't be necessary for a default zimbra install. I'll leave them here just for a reference in case they are useful for others depending on the circumstances.

I had a second test box "mail4" at one point but I shut it down.

The one MX record with the starting blank space :

                IN      MX      10 mail3

would "assume" for the internal.example.com domain.

My only physical box is "mail3". The other names and domains are done so I can configure multiple domains on this box. The blank space MX record (for internal.example.com) is also setup on mail3. So, in total my mail3 box has zimbra configured to handle mail for these five domains:

  • internal.example.com
  • mail3.internal.example.com
  • secondary.internal.example.com
  • move.internal.example.com
  • alias.internal.example.com
$ttl 38400
internal.example.com.  IN      SOA     internal.example.com. ajcody.mail3.internal.example.com. (
                        1207910051
                        10800
                        3600
                        604800
                        38400 )
                IN      NS      mail3
                IN      MX      10 mail3
mail3           IN      MX      10 mail3
secondary       IN      MX      10 mail3
move            IN      MX      10 mail3
alias           IN      MX      10 mail3
mail4           IN      MX      10 mail4
mail3           IN      A       192.168.0.16
mail4           IN      A       192.168.0.17
_xmpp-server._tcp       SRV      5 0 5269 mail3
_xmpp-server._tcp.mail3         SRV     5 0 5269 mail3
_xmpp-server._tcp.secondary     SRV     5 0 5269 mail3
_xmpp-client._tcp       SRV      5 0 5269 mail3
_xmpp-client._tcp.mail3         SRV     5 0 5269 mail3
_xmpp-client._tcp.secondary     SRV     5 0 5269 mail3
/var/named/chroot/var/named/0.168.192.rev
$ttl 38400
$ORIGIN 0.168.192.in-addr.arpa.
@       IN      SOA     mail3.internal.example.com. ajcody.mail3.internal.example.com. (
                        1207910558
                        10800
                        3600
                        604800
                        38400 )
                                IN      NS      mail3.internal.example.com.
16                              IN      PTR     mail3.internal.example.com.
17                              IN      PTR     mail4.internal.example.com.

Split-DNS - But I Already Have Bind Running Internally?

You should investigate using ACL's and the views options for BIND to effective setup the "purpose" of split-dns situation without actually running it on a separate server.

Please see:

Other Variables That Are Effected By Lookups

zimbraDNSCHeckHostname New In 5.0.11

If your MX records point to a spam-relay or any other external non-Zimbra server, enter the name of that server in the Inbound SMTP host name field. The "Check MX" button in the admin console compares the domain's MX records in DNS against the zimbraDNSCheckHostname setting, if set. If this attribute is not set, the domain MX setting is checked against zimbraSmtpHostname.

zimbraSmtpHostname

zmprov gs `zmhostname` zimbraSmtpHostname

zimbraMtaRelayHost

zmprov gs `zmhostname` zimbraMtaRelayHost

zimbraMtaDnsLookupEnabled

zmprov gs `zmhostname` zimbraMtaDnsLookupEnabled

zimbraAdminConsoleDNSCheckEnabled

zmprov gs `zmhostname` zimbraAdminConsoleDNSCheckEnabled

lmtp_host_lookup

postconf lmtp_host_lookup
 lmtp_host_lookup = dns

smtp_host_lookup

postconf smtp_host_lookup
 smtp_host_lookup = dns


Ajcody-External-Authentication


Domain And User "Status" Changes

If you need to manually modify the status of a domain, for example - migration.

Checking Status - Mode Of Domain

If the domain is in "active" mode, the CLI output will show this.

zmprov gd domainname zimbraDomainStatus
zimbraDomainStatus: active

The variables are all lowercase by the way.

Setting A Domain To Maintenance Status - Modes

To set a domain to "maintenance" mode, you would do the following.

zmprov md domainname zimbraDomainStatus maintenance

Setting A Domain To Active Status - Modes

To set a domain to "maintenance" mode, you would do the following.

zmprov md domainname zimbraDomainStatus active

Descriptions Of Status - Modes

Here's the descriptions for each of the "status" or "modes":

  • Active. Active is the normal status for a mailbox account. Mail is delivered and users can log into the client interface.
  • Maintenance. When a mailbox status is set to maintenance, login is disabled, and mail addressed to the account is queued at the MTA. An account can be set to maintenance mode for backing up, importing or restoring the mailbox.
  • Locked. When a mailbox status is locked, the user cannot log in, but mail is still delivered to the account. The locked status can be set, if you suspect that a mail account has been hacked or is being used in an unauthorized manner.
  • Closed. When a mailbox status is closed, the login is disabled, and messages are bounced. This status is used to soft-delete an account before deleting the account from the server. A closed account does not change the account license.
  • LockOut. Users who try to log in and do not enter their correct password are locked out of their account after a specified number of consecutive failed login attempts. An account’s status is automatically changed to Lockout. How long the account is locked out is set by COS or Account configuration, but you can change the lockout status at any time.

Cross Mailbox Searches and Tracing

Cross Mailbox Search

Events/errors will show in /opt/zimbra/log/mailbox.log

Search From the CLI

From the CLI, see zmmboxsearch help page

Here's an example use of the zmmboxsearch command:

su - zimbra
mkdir /tmp/testing
zmmboxsearch -m user1@mail3.internal.homeunix.com,user2@mail3.internal.homeunix.com -q "in:inbox" -d /tmp/testing/

Message Tracing

Please see the command help page for zmmsgtrace .

Deferred Emails - Error Of Connection Refused

Your having emails deferred and you check the /opt/log/zimbra.log file and see something like:

Sep  2 05:17:56 mail postfix/qmgr[12229]: 56A793151483: to=<USER@DOMAIN.net>, relay=none, delay=29404, delays=29404/0.08/0/0, dsn=4.4.1, status=deferred (delivery temporarily suspended: connect to 127.0.0.1[127.0.0.1]: Connection refused)

Things to do to help trouble shoot the issue:

  • Please confirm that iptables and selinux aren't running or are configured properly.
    • service iptables status
    • sestatus
  • Can you telnet to the different ports for example? port 25?
  • What's your /etc/hosts look like and is the format correct - confirm localhost entry is right?
    • cat /etc/hosts
  • Check for hostname issues:
    • hostname YOUR_FQDN
    • hostname -t MX YOUR_DOMAINNAME
    • cat /etc/resolv.conf
  • Give the output of the following as well - replace YOUR_SERVER_NAME:
    • zmprov gs YOUR_SERVER_NAME | grep -i mta
  • Have you tried to disable DNS lookups? [zimbraMtaDnsLookupsEnabled]
    • zmprov ms YOUR_SERVER_NAME zimbraMtaDnsLookupsEnabled FALSE
  • What's the current status of zimbra services?
    • zmcontrol status
  • If you can restart zimbra - while starting zimbra get logging events that might be related to issue or other errors/warnings:
    • zmcontrol stop
    • tail -f /opt/zimbra/log/mailbox.log & /opt/zimbra/log/zmmailboxd.out & /var/log/zimbra.log
    • zmcontrol start
  • Have you tried to disable antivirus and antispam to see if queue flushes?
    • Amavis
      • zmamavisdctl status
      • zmamavisdctl stop
      • zmamavisdctl start
    • Clam
      • zmclamdctl stop
      • zmclamdctl start
  • Give the following output - as root:
    • netstat -lntp
    • lsof | grep TCP
  • Checking postfix for deferred and command to flush:
    • postqueue -p
    • postqueue -f


What's my time and timezone?

I wrote up the steps to make sure the ZCS server is using the correct time and timezone here:

http://wiki.zimbra.com/index.php?title=Time_Zones_in_ZCS#The_server_OS



Logger Issues

You might want to cross review these other sections as well:

Logger Not Working Or No Stats In Admin Console

Check the following log files first to gather some information:

  • Make sure the /etc/hosts details make the actual hostname and the zmhostname output - if someone recently changed the hostname variables for this box, logger can be effected if done improperly and will not log the obvious issue.
  • /tmp/logprocess.out
  • /tmp/gengraphs.out
  • /opt/zimbra/logger/db/data/<hostname>.err
  • /var/log/zimbra/log (logswatch monitors this file)
    • grep'ing for things such as:
      •  egrep 'postfix|amavis|STATUS|DISK|QUEUE' /var/log/zimbra.log
  • You might also want to look at the following directory:
    • /opt/zimbra/logger/db/work/
    • You should see data in here, gif and rrd files.

Then proceed with the following wiki pages, as ordered:

Large Logger Database Killing Performance Of ZCS

Review the following to see what might be best option for you:

Reinitializing Logger Database From Scratch

This will effectively blow away your old logger database, allowing you to start over.

zmloggerctl stop
cd /opt/zimbra/logger/db
mv db db-old
  1. or remove it if your not worried about old directory being perserved
  2. rm -rf db
/opt/zimbra/libexec/zmloggerinit
  1. logger is probably running after this, but to make sure
zmloggerctl start
  1. Need to manually start this one though
zmlogswatchctl start

Turning Off Logger

To not have logger startup [ use `hostname` or type in hostname] :

zmprov ms `hostname` -zimbraServiceEnabled logger

To manually shutdown logger

zmloggerctl stop

Mysql

Mysql Issues And Memory Usuage

My Ajcody-General-Notes#Trouble_Shooting_Memory_Issues has some stuff on memory troubleshooting - generalities.

Our performance guide for Mysql:

http://wiki.zimbra.com/index.php?title=Performance_Tuning_Guidelines_for_Large_Deployments#MySQL

Variable Within MySQL & Zimbra

These percentages below are used against physical RAM - disregard swap in the percentage division.

From the above url, we get some details.

"By default, we try to reserve 30% of system memory for use by this process, and 40% for use by MySQL."
# ZCS 5.0 and later
$ zmlocalconfig -e mailboxd_java_heap_memory_percent=40
Java - mailboxd_java_heap_memory_percent=

The command to set the java percentage is:

 zmlocalconfig -e mailboxd_java_heap_memory_percent=30
Mysql - mysql_memory_percent & my.cnf w/ innodb_buffer_pool_size

Looks like there's an issue with the command/variable (zmlocalconfig -e mysql_memory_percent=##) in regards to MySQL though.

"There is a local config variable [mysql_memory_percent] for mysql memory percent, but today my.cnf doesn't get rewritten after install, so you have to edit my.cnf for this setting if you want to change it."

In summary, this variable [ mysql_memory_percent ] for MySQL is basically not working.

Do this to get memory in bytes for division numbers:

free -b

Figure out what your % in bytes using the physical memory (Mem:) output. Now edit the /opt/zimbra/conf/my.cnf file to change the following line with your memory percentage in bytes:

vi /opt/zimbra/conf/my.cnf
#Modify this var to adjust memory usage
innodb_buffer_pool_size = xxx

This will require a restart.

A GREAT forum post on this is here:

So other background info:

Mysql And Swap

See this article first before exploring the options below.

Should you have your swap file enabled while running MySQL ?

Set swappiness Kernel Var To 0

Source of this summary is here: [http://www.mysqlperformanceblog.com/2007/09/28/heikki-tuuri-to-answer-your-in-depth-innodb-questions/#comment-174442 Swappiness To 0"

Set it on running box:

echo 0 > /proc/sys/vm/swappiness

Set it to persist after reboot - vi /etc/sysctl.conf :

vm.swappiness = 0

The value is a percentage. By setting it to 100, Linux will always swap programs out and never shrink the buffer cache.

Swap Ram Disk Instead Of Disk

Source of this summary is here:

MySQL and the Linux swap problem

Make a (or some) swap disks:

mkdir /mnt/ram0
dd bs=1024 count=14634 if=/dev/zero of=/mnt/ram0/swapfile
mkswap /mnt/ram0/swapfile
swapon /mnt/ram0/swapfile

Will look like this:

[root@mail3 ~]# mkdir /mnt/ram0

[root@mail3 ~]# dd bs=1024 count=14634 if=/dev/zero of=/mnt/ram0/swapfile
14634+0 records in
14634+0 records out
14985216 bytes (15 MB) copied, 0.246329 seconds, 60.8 MB/s

[root@mail3 ~]# mkswap /mnt/ram0/swapfile 
Setting up swapspace version 1, size = 14979 kB

[root@mail3 ~]# free
             total       used       free     shared    buffers     cached
Mem:       1555472    1461200      94272          0     159400     344328
-/+ buffers/cache:     957472     598000
Swap:      2031608        104    2031504

[root@mail3 ~]# swapon /mnt/ram0/swapfile 

[root@mail3 ~]# free
             total       used       free     shared    buffers     cached
Mem:       1555472    1476252      79220          0     159404     344376
-/+ buffers/cache:     972472     583000
Swap:      2046232        104    2046128

[root@mail3 ~]# swapon -s
Filename				Type		Size	Used	Priority
/dev/mapper/VolGroup00-LogVol01         partition	2031608	104	-1
/mnt/ram0/swapfile 
                     file		14624	0	-2
[root@mail3 ~]# cat /proc/swaps 
Filename				Type		Size	Used	Priority
/dev/mapper/VolGroup00-LogVol01         partition	2031608	104	-1
/mnt/ram0/swapfile                      file		14624	0	-2


Just add those lines to your relevant startup file, like /etc/rc.d/rc.local, and it’ll persist after reboots.

O_DIRECT On Linux And INNODB To Fix Swapping

Source reference for this part is here:

Using O_DIRECT on Linux and INNODB to Fix Swap Insanity

See also: Ajcody-Server-Topics#O_DIRECT_And_Ext3_On_Linux_As_Possible_Reason_For_Corruption

IO_DIRECT And Ext3 On Linux As Possible Reason For Corruption

This is being investigated. Below are my notes on the issue.

Mysql Database Corruption/Repair

Please see the Mysql_Crash_Recovery page rather than the normal search results that give mysql-logger details.

Manual Change Of Mysql Passwords

Please see:

http://wiki.zimbra.com/index.php?title=Issues_with_mysql_and_logmysql_passwords

Postfix - MTA

Missing main.cf Error

If you get something like:

postfix/postqueue[8739]: fatal: open /opt/zimbra/postfix-2.4.7.5z/conf/main.cf: No such file or directory 

All you need to do is create an empty main.cf file and zimbra will rewrite it.

touch main.cf

When you now do something to start the mta, it will generate the values for main.cf

zmmtactl stop
zmmtactl start

User Alias Mapping And Mail Transport with Postfix & LDAP

See User_Alias_Mapping_and_Mail_Transport_with_Postfix_&_LDAP

Automatic BCC

Please see the following:

Limiting Or Increasing Number Of Recipents / Messages

Mailing Lists - Distribution Lists

Please see Ajcody-Notes#Problems_Resolving_Virtual_Aliases_For_Members_Of_Large_Distribution_Lists

Policy Daemon

If you want to restrict messages per hour, you can look into Policy Daemon:

Postfix

Also, there are some default postfix parameters set to control sending a message to x amount recipients. The parameters you will need to look at are smtpd_recipient_limit & smtpd_recipient_overshoot_limit, these have a default value of 1000.

Postfix defines these parameters as:

  • smtpd_recipient_limit: The maximum number of recipients that the Postfix SMTP server accepts per message delivery request.
  • smtpd_recipient_overshoot_limit: The number of recipients that a remote SMTP client can send in excess of the limit specified with $smtpd_recipient_limit, before the Postfix SMTP server increments the per-session error count for each excess recipient.

From the command line you can change the default values.

su - zimbra
postconf -e smtpd_recipient_limit=<new value>
postconf -e smtpd_recipient_overshoot_limit=<new value>
postfix reload

Controlling SMTPD Client Connections

Postfix Resources At Their Site (All Clients/Connections):

Postfix Resources At Their Site (Exceptions To Clients/Connections Or Single Source):

Spam Control And Related Issues

A list of resources you'll find useful:

http://wiki.zimbra.com/index.php?title=Zimbra_MTA#Anti-Spam_Training_Filters

http://wiki.zimbra.com/index.php?title=CLI_zmtrainsa

http://wiki.zimbra.com/index.php?title=Improving_Anti-spam_system

http://wiki.zimbra.com/index.php?title=Postfix_Policyd

http://wiki.zimbra.com/index.php?title=IP_Address_whitelisting

http://wiki.zimbra.com/index.php?title=Spam_training

Wiki articles that have been assigned to the anit-spam category:

http://wiki.zimbra.com/index.php?title=Category:Anti-spam

Down to the end-user:

http://wiki.zimbra.com/index.php?title=Cool_User_Spam_Filters

http://www.zimbra.com/community/end_user_guide_and_how_to.html

Using Different SMTP Server For Webclient (ZWC), Mobiles, And ZCO

There a variable called zimbraSmtpHostname that is in the global (zmprov gacf) and server (zmprov gs `hostname`) configs. It's defaulted to 'localhost' - at least on a single ZCS configuration.

In cases where you need/want all mail to be processed by, for example, an external non-Zimbra spam filter box you could set this variable to the spam filter boxes hostname.

Global:

zmprov mcf zimbraSmtpHostname hostname-of-ext-server

Per Server:

zmprov ms servername zimbraSmtpHostname hostname-of-ext-server

Global Disclaimer Options

Here's the url to review for a "current" possibility:

And in the notes section there's a comment about multi-servers:

Please note though, "This article is a community contribution and may include unsupported customizations." Meaning, it's an unsupported customization, so please take the necessary precautions.

In regards to an official and supported way to do this, please review this RFE:

Volumes & zmvolume

How To Go About Changing Volume Paths

List your current volume details

zmvolume -l

Make your directories to the "new" volume path. For example:

mkdir /san/mount/index /san/mount/store
chown zimbra:zimbra /san/mount/index /san/mount/store

Shutdown zimbra so we can move the data.

zmcontrol stop

Now move all the old data and then make symbolic links from old to new.

mv /opt/zimbra/index/* /san/mount/index/
mv /opt/zimbra/store/* /san/mount/store/
rmdir /opt/zimbra/store /opt/zimbra/index
ln -s /san/mount/index /opt/zimbra/index
ln -s /san/mount/store /opt/zimbra/store

Start zimbra back up.

zmcontrol start
  • Goto the admin web console , Configuration > Servers > server-name > Volumes Tab.
    • Select index volume
      • Click edit
        • Modify the path to the new path
    • Select store volume
      • Click edit
        • Modify the path to the new path
  • Click on Save in the upper left hand section.

To Modify Volume From CLI After Data Move

See what the details are of your volumes:

zmvolume -l /opt/zimbra/index
and
zmvolume -l /opt/zimbra/store
   Volume id: 2
       name: index1
       type: index
       path: /opt/zimbra/index
 compressed: false
    current: true

  Volume id: 1
       name: message1
       type: primaryMessage
       path: /opt/zimbra/store
 compressed: false
    current: true

Let's say your sym links are the ones shown above:

/opt/zimbra/index
/opt/zimbra/store

And your new directory paths are:

/mnt/nas/index
/mnt/nas/store

You would run the following to modify the volumes to use the real directory paths rather than the sym links.

zmvolume -e -id 2 -p /mnt/nas/index
zmvolume -e -id 1 -p /mnt/nas/store

Convertd

From release notes:

An alternative implementation of the convertd daemon used for text extraction and conversion to HTML is now available as beta in 5.0.7. The new implementation is Apache multi-process based, as opposed to the original implementation which was multi-threaded. When libraries used to handle attachments fail or crash, the multi-process implementation allows for better availability. Since the new implementation is still in beta, the older multi- threaded implementation continues to be the default. If you are experiencing a high number of 400 (try again) error codes during LMTP delivery, or if you are seeing too many kvoop processes consuming memory, you can try the new Apache-based convertd.

  • Run /opt/zimbra/convertd/bin/upgrade_v2 to upgrade.
  • Run /opt/zimbra/convertd/bin/downgrade_v1 to downgrade.
 ps auxwww | grep -i kvoop

Run the script you want. Plan on doing a restart of zimbra.

zmcontrol stop
zmcontrol start

One might be able to get away with just restarting convertd and mailstore.

Domain Rename Issues

We have a command to rename a "domain" and the resources within it - unfortunately it doesn't handle ALL of the different components for various reasons. Specifically Documents and Calendars [more down below].

Review this RFE that was done when they implemented the renameDomain command.

http://bugzilla.zimbra.com/show_bug.cgi?id=7655

Note comment 21 & 22 (confirming what QA'd). The syntax is

zmprov -l rd testold.com testnew.com

Problem With Documents

Problems that arise with Documents are explained here, comment #1 has workaround.

http://bugzilla.zimbra.com/show_bug.cgi?id=25873

Problem With Calendar & Appointments

Now, the very unfortunate part about calendars with a domainname change/move.

The root issue here's seems to be more about the calendar standards and practices with the use of the "Organizer" field and notifications. You'll see 3 "work arounds" in comment 2 of bug 26736.

Export/Importing of the calendar data is shown here:

http://wiki.zimbra.com/index.php?title=User_Migration

This part specifically :

http://wiki.zimbra.com/index.php?title=User_Migration#Copy_Calendar_From_One_Zimbra_User_to_Another


Themes, Branding, Logos, And Other UI Customizations

Main Wiki Page: Skins

Please see:

Finding Skins Variable From CLI

What skins are available [zimbraInstalledSkin]:

[zimbra@mail3 ~]$ zmprov gacf | grep -i skin
zimbraInstalledSkin: bones
zimbraInstalledSkin: sky
zimbraInstalledSkin: lemongrass
zimbraInstalledSkin: lavender
zimbraInstalledSkin: yahoo
zimbraInstalledSkin: waves
zimbraInstalledSkin: bare
zimbraInstalledSkin: sand
zimbraInstalledSkin: steel
zimbraInstalledSkin: hotrod
zimbraInstalledSkin: beach

Can Users Change Their Skin? This depends on their COS and the COS Value [zimbraFeatureSkinChangeEnabled]:

[[zimbra@mail3 ~]$ zmprov gac
default
[zimbra@mail3 ~]$ zmprov gc default | grep -i skin
zimbraFeatureSkinChangeEnabled: TRUE
[zimbra@mail3 ~]$ zmprov gac
default
[zimbra@mail3 ~]$ zmprov gc default | grep -i skinchange
zimbraFeatureSkinChangeEnabled: TRUE

What is the default skin for the cos [zimbraPrefSkin]:

[zimbra@mail3 ~]$ zmprov gc default | grep -i prefskin
zimbraPrefSkin: beach

To Modify the default skin for a cos:

zmprov mc [cos name] zimbraPrefSkin Name_Of_Skin

Setting A Default Domain For User Login

In the admin web console goto:

Configuration > Global Settings

On the General Information tab to the right you'll see a variable called "Default domain". This will be the assumed domain for user logins as well [as well as other functionality].

If you have multiple domains and want to allow users avoid having to type in their domainname with their login credentials you should look at Virtual Hosting.

Looking for proper documentation url's for this.

So far I have this:

http://wiki.zimbra.com/index.php?title=Provide_HTTP(s)_Integration_with_Apache

http://wiki.zimbra.com/index.php?title=Multiple_SSL_Virtual_Hosts_5.0

IM - Instant Messaging

What Happen to AOL IM and MSN?

From the 5.x Release Notes:

Note: Update about Instant Messaging feature - In order to ensure the scalability of our server software and provide a quality experience to our customers, we have removed IM interoperability with AOL and MSN that was present in previous 5.0.0 beta releases. This functionality may be restored in a future release.

ZCS to ZCS IM Availability

Please review the following bug and note the DNS SRV entries I posted within it.

IM - Conference Rooms

Please see the RFE I made:

Also the bug/RFE about Conference rooms in Multi-Server Configurations

Jump to: navigation, search