ZAD

Zimbra Archiving and Discovery

   KB 2077        Last updated on 2018-01-18  




0.00
(0 votes)

Zimbra Archiving and Discovery is an optional feature that enables you archive messages that were delivered to or sent by Zimbra Collaboration and to search across mailboxes.

The installation of the archiving feature provides the ZCS discovery tool (also known as cross mailbox search) and sets the attributes that allow archiving to be enabled on the Zimbra MTAs. Archiving is configured on a per account basis. Each account enabled for archiving requires a Zimbra archive license. When archiving is enabled for an account, a copy of all email from or to that account is forked at the MTA, and a copy of the message is delivered to a predefined archive mailbox. The archiving process is transparent to account users.

Discovery allows you to conduct a search for email messages across live and archived mailboxes and copy the results to a specified mailbox.

Zimbra-archive-001.png

Getting Started

Zimbra Archiving and Discovery is available since Zimbra Collaboration 7.0 servers and above. There are two components to the product, you must have both installed to use Zimbra Archiving and Discovery:

  1. The Archiving component - this ships on the Zimbra Collaboration 7.0 and later servers.
  2. The Discovery component - this is a separate module to download for the Admin Console UI

What if I am a Zimbra Customer already?

First, contact Sales to let us know you are interested in trying Zimbra Archiving and Discovery and to discuss pricing and product details.

After working with us, assuming you have ZC already running you can download and install the Discovery module in the Admin Console. From there you can begin using the product to set up archive accounts and run cross mailbox searches.

What do I do if I do not have Zimbra yet?

Go to the Network Trial page and register for the ZC trial; indicate in the registration form you wish to try Zimbra Archiving and Discovery and our sales team will follow up with you on pricing and product questions.

Download both ZCS Network Edition and the Discovery Module for the Admin Console and follow the installation instructions.

How Archiving Works

When a message is sent or received by a user, the message is always routed through the Postfix MTA. The Postfix MTA allows integrating software that can perform actions on messages that are in flight. When archiving is enabled for the sender or the recipient of messages, Zimbra Archiving integrates with an MTA hook and the Amavisd-New utility to fork a copy of the message. The “does recipient or sender have archiving enabled” check is performed on the SMTP standard envelope and not on the From or To/Cc headers. Since checks are performed on the envelope, Bcc copies and messages sent to distribution lists are captured. For example, if User A sends a message to User B, and if User B has archiving enabled, the MTA delivers two messages — one to User B’s mailbox and one to User B’s archive mailbox. The message received in User B’s mailbox looks normal, as shown in the following example:

Received: from localhost (localhost.localdomain [127.0.0.1])…
From: userA@example.com
To:userB@example.com
Subject: New License Key
Message-ID: <015f01c717fe$70f042d1$b1d6f61d@thom>
Date: Mon, 04 Nov 2008 23:48:18 -0000
 
Hi B,
Can you send me the license key for the software again?
Thanks, A

The message received in User B’s archive mailbox contains additional X-Envelope-From and X-Envelope-To headers. These headers show the real email address the message was sent from and each of the email addresses that the message was sent to.

Received: from localhost (localhost.localdomain [127.0.0.1])…
From: userA@example.com
To:userB@example.com
Subject: New License Key
Message-ID: <015f01c717fe$70f042d1$b1d6f61d@thom>
X-Envelope-From: userA@example.com
X-Envelope-To: userB@example.com
Date: Mon, 04 Nov 2008 23:48:18 -0000
 
Hi B,
Can you send me the license key for the software again?
Thanks, A

Zimbra archiving can be set up to create archiving accounts that are maintained within Zimbra Collaboration or to work with third-party archiving systems using SMTP forwarding to send messages to a third-party archive server. For third-party archiving, Zimbra Collaboration is configured to act as the forwarding agent.

How Discovery Works

The discovery feature of Archiving and Discovery is used to search across live* and archive mailboxes for email messages and attachments. The discovery tool can be run from the administration console and the results are copied to a target mailbox that you specify.

Zimbra-discovery-001.png

  • A live mailbox is an account on the system other than archive accounts and system accounts.

You can search outgoing and incoming email by date, from, to, cc, subject, keywords, and attachments. You can also create queries to search by name, dates and time ranges, distribution list, aliases.

Example of Discovery

In this example want to search all the messages from usera to userb that are located in the archive mailbox called userbDATE@DOMAIN.archive.

  • 1.- In the Admin Console go under Tools > Search Mail (This is the Admin Zimlet that Archive & Discovery includes)
  • 2.- Click on the right settings icon and select new.
  • 3.- Then you can add the search string by yourself or create a new one doing click in the Advanced tab, adding the parameters that you want like from or to, date, etc.

In this example also select the dedicated Archive Mailbox, and for the target of the search, in this example select admin, and the folder name, you can also restrict the search per account, so you can search only in the userbDATE@DOMAIN.archive account, then click in top-right corner Run search.

Zimbra-discovery-002.png

  • 4.- Search results are placed in a target mailbox. You can organize your search results by creating different target mailboxes or by creating individual folders within a target mailbox for each search you run. X-zimbra-Source header information is added to each message header that is copied to the targeted mailbox. This header label includes the account ID, the account name, and the server that the account resides on.

You can see the results of the search by logging on to the target mailbox address.

Zimbra-discovery-003.png

Installing the Archiving Package

You can install the archiving package on an existing single-server deployment or on a multi-server deployment. If the mailbox server and the MTA server reside on the same node, you configure and enable archiving as a single process. If your mailbox and MTA servers are on separate nodes, the zimbra-archive package is installed first on at least one mailbox server and then the archiving component is enabled on each MTA in the deployment.

Note: zimbra-archive (the package/rpm you see from the installer) should be installed on all mailstores which you want to use for cross mailbox search. This also sets the zimbraComponentAvailable archiving config attribute which allows the mta(s) to turn on archiving. zimbra-archive is not installed directly on the mta, it's just enabled.

Install Archiving in a Single-Server Environment

The following scenario assumes that the LDAP, MTA, mailstore and archiving servers are on the same node.

  • 1. Refer to the Zimbra Collaboration Single Server Installation Guide to open an SSH connection to the Zimbra Collaboration server. Log on to the server as root and run the ./install.sh command to begin the upgrade process.
  • 2. Accept the license agreement and type Yes to run the upgrade.
  • 3. Type Yes for zimbra-archiving when presented with the packages to be installed.

The upgrade process begins and the archiving package is installed. At this point, the Discovery feature is installed and can be used.

  • 4. To enable archiving, switch user to zimbra and enable archiving on the server.
zmprov ms `zmhostname` +zimbraServiceEnabled archiving
  • 5. Restart the server.
zmcontrol restart

Install zimbra-archiving in a Multi-Server Environment

The following upgrade scenario is adding a new server that is dedicated as a archiving server to your Zimbra Collaboration environment. Before beginning the install process, record the following information. You need this information when you install the archiving server. Run the zmlocalconfig -s command to find the information.

LDAP Admin Password ____________
LDAP Hostname _____________
LDAP Port _____________

Refer to the Multiple-Server Installation chapter in the Zimbra Collaboration Multi-Server Installation guide for detailed steps on installing the packages.

  • 1. Open an SSH connection to the mailbox server that is being configured for archiving. Log on to the server as root and unpack the Zimbra software. Run the ./install.sh command to begin the install process.
  • 2. Type y and press Enter to install the following packages:
    • zimbra-store
    • zimbra-archiving

The zimbra-core package is installed by default.

  • 3. Type y and press Enter to modify the system.
  • 4. The Main menu displays the default entries for the Zimbra component you are installing. To expand the menu, type x and press Enter.
  • 5. Select the Common Configuration menu and configure the LDAP Hostname, LDAP password, and LDAP port.
  • 6. Select the zimbra-store menu and configure the Admin password and the License file location.

Complete the installation process following the steps in the Multi-server Installation guide, under Installing Zimbra Mailbox Server.

At this point, the Discovery feature is installed and can be used.

Manage Archiving From the Administration Console

Enable Archiving using the Admin Console

After Archiving is installed, you can set up archiving and manage it from the Administration Console. Go to the Home > Configure > Global Settings > MTA page and in the Archiving Configuration section, check Enable archiving.

Zimbra-archive-002.png

Go now to the MTA server and enable the Service in Home > Configure > Servers > MTASERVER > Services

Zimbra-archive-003.png

  • The the mailbox server and the must be restarted. Type:
zmcontrol restart

Enable Archiving using the CLI

To enable Archiving using CLI, switch user to zimbra and enable archiving on the MTA server.

  • a. To enable archiving type:
zmprov ms <zmhostname> +zimbraServiceInstalled archiving +zimbraServiceEnabled archiving
  • b. The server must be restarted. Type:
zmcontrol restart

Repeate for each MTA server

Trick to see if the Archiving is enabled

You can see if the Archiving is enabled searching inside the amavis config file /opt/zimbra/conf/amavisd.conf an seeing if was modified correctly, you should see on the mta:

more /opt/zimbra/conf/amavisd.conf | grep '$archive_quarantine_method'

$archive_quarantine_method = 'smtp:[127.0.0.1]:10025';

In case that you see #$archive_quarantine_method = 'smtp:[127.0.0.1]:10025'; with the # icon behind the line, that means that is commented and the Archiving is not enabled in the MTA.

Creating a Dedicated Archive COS

You can configure attributes in the COS to set mailbox features, quotas, and passwords, turn off spam and virus checks, and hide the archive accounts from GAL

  • 1. Go to Configure > Class of Service and in the gear icon menu select New.

Zimbra-archive-004.png

  • 2. Change Features and Preferences as required for an Archiving COS. For example:

You can remove all the Features and General Features, except the Mail and Preferences.

Zimbra-archive-005.png

  • 3. If you have a dedicated archive server, in the Server Pool page, deselect the archiver server from the list. In a multi-server deployment with a dedicated archive server, the server should be removed from the COS server pool so that the archive server is not randomly assigned to new accounts.

For example, in the archive_COS:

Zimbra-archive-007.png

But, please remove the dedicated archive server from the other COS, like the default COS, etc, to prevent that regular accounts can be created there:

Zimbra-archive-010.png

Note: These steps to remove the server from the server pool are not done in a single-server deployment. Creating a dedicated archiving COS is a good idea as this makes it easy to create archive mailboxes that are configured the same.

  • 4. Modify the options on the Advanced page if required.

Zimbra-archive-008.png

  • 5. In the Archiving page, check the Enable archiving box to make this COS an archiving cos.

Zimbra-archive-009.png

  • 6. If you want to change the format for the naming scheme for archive accounts, modify the two template fields. See the Set Up Archive Account Name section for more information.
  • 7. Click Finish.

Set Up Archive Account Name

You use attributes to create and manage the naming scheme for archive accounts. You can set up these attributes either by COS or by account. For COS, these attributes can be changed from the administration console, COS or individual account’s Archiving page.

  • Account date template. Sets the date format used in the name template. The default is yyyyMMdd. Adding the date to the account name makes it easier to roll off older data from the system to backups.
  • Account name template. Sets up how the archive mailbox name is created. The default value is ${USER} ${DATE}@${DOMAIN}.archive. The archive account address would be like this example: user-20070510@example.com.archive. If you change the default value, you must use syntax that creates a valid email address. We recommend that you add .archive to all archive accounts to create archive mailboxes in a non-routable domain to prevent spoofing of the archives.

When the template based on the zimbraArchiveAccountDateTemplate attribute is set up, amavisArchiveQuarantineAccount is updated to the new template name when zmconfigarchive is run.

Administering the archive server

The amavisd-new server process controls account archiving as well as antivirus and antispam processes. The zmarchivectl command can be used to start, stop, restart or obtain the status of the amavisd-new server process that controls account archiving. Caution should be taken when starting or stopping the archiving process as it is a shared server process between archiving, antivirus, and antispam processes. Performing actions on any of them affect any of the other services that may be enabled in your deployment.

If you want to disable archiving and not antivirus, or antispam services, disable the respective service either through the CLI or through the Administration Console.

Set Up Archiving for a Users Mailbox

Four attributes are related to the archive feature for accounts. Two that configure a mailbox and two template attributes to construct the archive account names. To set up archiving for a mailbox two attributes are configured on the primary user’s mailbox. One attributed enables archiving and the second shows where messages are being archived.

  • Currently archived to — The current archive address. Archiving is to a single account. If this is unset, archiving is not enabled.
  • Archived accounts — Any previous and current archive addresses that this mailbox was archived to. containing all the accounts that have been archived for the given account.

Archive Mailboxes

You can create an archive mailbox with or without an assigned COS. You can also forward archive email to a third-party.

Note: Accounts with archiving enabled are counted against the number of Zimbra licenses purchased for archiving. To see current license information, go to the administration console, Global Settings > License page. Archive mailboxes are listed in the administration console along with the live accounts.

Zimbra-archive-011.png

Create an archive mailbox and assign a COS

Archive accounts are created based on the Zimbra Archive name templates.

  • The attribute zimbraIsSystemResource is added to the archive account and set to TRUE.
  • The archive account is displayed in the administration console.
  • When a message is received in a mailbox with archiving enabled, a copy of the message is sent to the archive mailbox.
    • 1. Log on as zimbra.
    • 2. Type
zmarchiveconfig enable <account@example.com> archive-cos <archive>

Then if you return to the Admin Console, you can see the new userb@example.com.archive in the list:

Zimbra-archive-013.png

You can go the userB mailbox and click into the Archiving section, and you can see the Archiving feature enabled, and the Mailbox where the emails are redirecting to archiving purposes.

Zimbra-archive-012.png

Create an Archive Mailbox with No COS or Password

If the archive account is not assigned a COS, the following settings are set by default.

  • Mailbox quota is set to 0, unlimited quota.
  • Spam and virus checks are disabled.
  • Hide in GAL is enabled, so the archive account does not display in the GAL
    • 1. Log on as zimbra
    • 2. Type
zmarchiveconfig enable <user@example.com>

Enable Archive Forwarding to a Third-party Archiving Server

If the archive account is not maintained within Zimbra Collaboration, you do not need to set a password, COS, or other attributes.

    • 1. Log on as zimbra
    • 2. Type
zmarchiveconfig enable <account@example.com> archive-address account-archive@offsiteserver.com archive-create false

Searching Across Mailboxes

When the archiving and discovery feature is installed, you can search across mailboxes either from the administration console or through the command line interface.

Note: You do not need to have any archive mailboxes configured to search across mailboxes, but the Archive package must be installed. You can assign a user to run the mailbox searches from the administration console by creating a delegated administrator with rights to access the mailbox search tool.

Cross Mailbox Search from the Administration Console

The discovery tool, Search Mail, is added to Tools and Migration on the Navigation pane when the archiving package is added. To set up a cross mailbox search, in Search Mail, go to the gear icon and select New. You configure the following information.

  • Server name. The server name to be searched.
  • Target mailbox and folders. One target mailbox and folder are created automatically. You can use this mailbox for all your search results and create new folders for each search, or you can create a new target mailbox for each separate search.

A target mailbox is like any other mailbox and can have any features or preferences that are defined by the COS or by account. Target mailboxes are listed in the administration console Accounts list. You might want to give the target mailboxes account names that identifies them as target mailboxes for cross-mailbox searches and configure a COS specific for target mailboxes to be able to manage access.

  • Limit the number of messages returned by the search. The default is 500 results.
  • You can select to send an email notification when the search is completed. The email notification includes the search task ID and status on the subject line and you can specify the type of information to include in the message, such as the number of messages found, the list of addresses resulting from the search and the search query used.
  • Select which mailboxes to search. When you check Select accounts to search, you select which account addresses to search.
  • Create the search query. You can search outgoing and incoming email by date, from, to, cc, subject, keywords, and attachments. Advanced can be used to quickly create a query to search by name, dates and time ranges, distribution list, aliases.

When searching archive messages, you can search by the envelope address using the envfrom and envto query language extensions.

As the search runs, the Search Mailbox Content pane lists the search and the status. Click Refresh to update this page. Delete the search task when it is completed because it occupies server memory. When the server is restarted, past searches are deleted.

When you use the discovery feature in the administration console, the tool makes copies of messages in the target mailbox you create. The messages occupy server space, increasing the size of your server. You might want to delete these messages from the target mailbox when they are no longer needed.

Troubleshooting

Checking by CLI if the archive is enabled per account

You can run the next command to see by CLI if a user has enabled the archive, what is the archive mailbox, etc.:

zmprov ga userb@example.com | grep -i archive
amavisArchiveQuarantineTo: userb-20150604@example.com.archive
zimbraArchiveAccount: userb-20150604@example.com.archive
zimbraArchiveAccountDateTemplate: yyyyMMdd
zimbraArchiveAccountNameTemplate: ${USER}-${DATE}@${DOMAIN}.archive
zimbraArchiveEnabled: TRUE
zimbraZimletAvailableZimlets: +com_zimbra_mailarchive

The archive account should exist and reference to lmpt instead smtp:

zimbra@zimbra-sn-u14-10:~$ zmprov ga userb-20150604@example.com.archive | grep -i trans
zimbraMailTransport: lmtp:zimbra-sn-u14-11.example.com:7025

If Enable Archiving is NOT displayed in the Admin Console > Global Settings > MTA > Archiving Configuration
Make sure the com_zimbra_archive extension is installed.

cd /opt/zimbra/zimlets-network
zmzimletctl deploy com_zimbra_archive.zip
zmmailboxdctl restart

Known Issues

Bug 96467

Bug 96467 - " NO_SUCH_COS " in Web GUI if you try to assign COS to any domain/user

service.UNKNOWN_DOCUMENT Error When Trying Search

The service.UNKNOWN_DOCUMENT error generally means that there is no handler for specified document. Try redeploying the zimlet, restarting the mailboxd service and let us know how it goes. The xmbxsearch zimlet is located in /opt/zimbra/zimlets-network directory.

su - zimbra
cd /opt/zimbra/zimlets-network
zmzimletctl undeploy zimbra_xmbxsearch.zip
zmzimletctl deploy zimbra_xmbxsearch.zip
zmmailboxdctl restart


Verified Against: Zimbra Collaboration 8.6, 8.5, 8.0 Date Created: 7/3/2007
Article ID: https://wiki.zimbra.com/index.php?title=ZAD Date Modified: 2018-01-18



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search