ZCS Mailman Integration

Revision as of 13:23, 1 July 2008 by Publiski (talk | contribs)


HOWTO: Integrate Zimbra Collaboration Suite with Mailman mailing lists


Contents

  1. Overview
  2. Getting started
  3. Configuring Mailman
  4. Configuring ZCS
  5. Further integration with Zimbra
  6. Possible configurations (unsupported)
  7. ZimbraIntegration.py
  8. zimbra_mailman.patch


1. Overview

Many administrators would like to have robust mailing list management in conjunction with Zimbra. ZCS has distribution lists which works well for small groups with a central management interface; however, it does not provide support for self-help subscription, moderation or digests. Using ZCS together with Mailman is one way to fulfill the requirements, but requires a bit of massaging to work as one would expect.

One method of integration can be found on our forums at http://www.zimbra.com/forums/administrators/1380-solved-zimbra-mailman-howto.html

The above guide will allow a single-node ZCS install to coexist with Mailman on a single server, however, your mileage may vary. The referenced setup requires manual building of Mailman, integration with Zimbra/Apache-HTTPD, Zimbra-Postfix config changes and the work required to make it work does not appear to be insignificant. The guide also places restrictions on the email-domain that can be served by Mailman lists on the server.

In this guide, we will elaborate on a dual/multi-node ZCS/Mailman setup with step-by-step instructions as well provide patches to Mailman with a Python library to automatically interface with Zimbra. There will be no restriction on the domain which Mailman can serve. In addition to integrating Mailman, one can take the general ideas introduced by this document and apply it to any other list-serv of choice.


2. Getting started

Our supported configuration for using Mailman with Zimbra is to have Mailman on an external dedicated node. We do not want to have the Mailman setup interfering with the ZCS configuration.

This guide assumes knowledge of working with ZCS and the ability to get the basics of Mailman configured (such as installation, Apache HTTPD configuration, and mailing list management). Additionally, for convenience, we will assume that the setup will be performed on RHEL.

To continue, we will need:

  1. An additional server where we will run Mailman
  2. The name of the MTA for Zimbra (ZIMBRA_MTA_HOST)
  3. The name of the host on which Mailman will run (MAILMAN_HOSTNAME)
  4. The domain name for which we are receiving email (EMAIL_DOMAIN)
  5. To create an admin password for Mailman (MAILMAN_ADMIN_PASSWORD)
  6. To create a list creator password for Mailman (MAILMAN_CREATOR_PASSWORD)


3. Configuring Mailman

First, install Mailman and postfix; on RHEL 5, we do:

$ rpm -e --nodeps sendmail
$ yum install mailman postfix

which pulls down all dependencies such as httpd if it was not already present. It also forcibly removes sendmail and replaces it with postfix.

Edit /etc/mailman/mm_cfg.py set the variables:

 DEFAULT_URL_HOST = 'MAILMAN_HOSTNAME'
 DEFAULT_EMAIL_HOST = 'EMAIL_DOMAIN'

Run

$ /usr/lib/mailman/bin/update

Create the site-wide mailing list (default is 'mailman') and any other desired mailing lists:

$ /usr/lib/mailman/bin/newlist mailman

specify any valid email address as the list owner (including any address that is hosted on ZCS).


Edit and update /etc/aliases as directed by the output of the 'newlist' command. Run

$ /usr/bin/newaliases

to update the aliases table to include the mailing list addresses.


Setup the Mailman administrative passwords:

$ /usr/lib/mailman/bin/mmsitepass MAILMAN_ADMIN_PASSWORD
$ /usr/lib/mailman/bin/mmsitepass -c MAILMAN_CREATOR_PASSWORD


Configure postfix to accept email on EMAIL_DOMAIN and redirect unknown addresses to user@EMAIL_DOMAIN back into Zimbra (this will cover the cases where users on ZCS will also be included in the mailing lists)

Edit /etc/postfix/main.cf and set:

 mydomain = EMAIL_DOMAIN
 myorigin = $mydomain
 mydestination = $mydomain
 local_recipient_maps =
 fallback_transport = smtp:ZIMBRA_MTA_HOST


We are now ready to (re)start the mail, httpd and mailman services, do the following:

$ chkconfig httpd on
$ chkconfig postfix on
$ chkconfig mailman on
$ service httpd restart
$ service postfix restart
$ service mailman restart


4. Configuring ZCS

NOTE: This section can be bypassed if one follows the directions in '5. Further integration with Zimbra' below.


Now that Mailman has been properly configured, we will need to setup the addresses that will forward from ZCS to the Mailman host.

$ zmprov ca LIST_NAME@EMAIL_DOMAIN ANY_RANDOM_PASSWORD
$ zmprov ma LIST_NAME@EMAIL_DOMAIN zimbraMailTransport MAILMAN_HOSTNAME
$ zmprov ca LIST_NAME-admin@EMAIL_DOMAIN ANY_RANDOM_PASSWORD
$ zmprov ma LIST_NAME-admin@EMAIL_DOMAIN zimbraMailTransport MAILMAN_HOSTNAME
$ zmprov ca LIST_NAME-bounces@EMAIL_DOMAIN ANY_RANDOM_PASSWORD
$ zmprov ma LIST_NAME-bounces@EMAIL_DOMAIN zimbraMailTransport MAILMAN_HOSTNAME

Repeat the ca/ma commands above for the remaining -confirm, -join, -leave, -owner, -request, -subscribe, and -unsubscribe aliases.

With the above configuration complete, we may now use fully-featured mailing lists at LIST_NAME@EMAIL_DOMAIN.


5. Further integration with Zimbra

We can offer tighter integration of Mailman with ZCS using the attached ZimbraIntegration.py (to be installed in MAILMAN_HOME/Mailman). A simple patch to the Mailman installation in conjunction with the aforementioned ZimbraIntegration.py will enable us to ignore section 4 above altogether.

With the zimbra_mailman_integration.patch applied against Mailman every invocation of 'newlist', 'rmlist' and create/delete list from the web interface will automatically issue the appropriate zmprov commands to ZCS. Zimbra accounts with the zimbraMailTransport setting will be configured as necessary for the specified LIST_NAME.

To apply the integration, copy ZimbraIntegration.py into MAILMAN_HOME/Mailman, on systems where we installed Mailman using the RPM, it will be /usr/lib/mailman. Apply the patch:

$ cd /usr/lib/mailman; patch -p0 < zimbra_mailman_integration.patch

Edit /etc/mailman/mm_cfg.py, add the variables:

 OWNERS_CAN_DELETE_THEIR_OWN_LISTS = True # enable deleting lists from web UI
 ZIMBRA_ADMIN_SOAP_SERVICE = 'https://ZCSHOST:7071/service/admin/soap/'
 MAILMAN_SMTP_TRANSPORT = 'MAILMAN_HOSTNAME'
 ZIMBRA_ADMIN_USERNAME = 'YOUR_ZCS_ADMIN_USERNAME'
 ZIMBRA_ADMIN_PASSWORD = 'YOUR_ZCS_ADMIN_PASSWORD'

ZimbraIntegration.py exposes some APIs that can be re-used by non-Mailman list-servs and can serve as an example for users that might want to integrate ZCS with another list-serv.


6. Possible configurations (unsupported)

For small configurations, it is possible to run Mailman on the same host as ZCS, although strictly unsupported.

This is possible because Zimbra installs its own postfix into /opt/zimbra/postfix, the system may retain its own postfix installation and use that configuration to communicate to Mailman on the same node. The system-version of postfix must listen on a different port (ALT_SMTP_PORT) from the Zimbra-version of postfix. The zimbraMailTransport directive above would become smtp:ZIMBRA_MTA_HOST:ALT_SMTP_PORT

The remaining integration into Zimbra's Apache HTTPD configuration is left as an exercise to the reader.


6.1 Using a different mailing list manager (other than Mailman)

Following this guide presented thus-far, similar principles may be applied to any list-serv with respect to email routing and integration using ZimbraIntegration.py.

ZimbraIntegration.py

import urllib2
# try to import the ElementTree API included with py 2.5 first
# otherwise fallback on third-party install (prereq)
try:
    from xml.etree import ElementTree
except:
    from elementtree import ElementTree

# load mailman config
try:
    from Mailman import mm_cfg
except:
    print "WARNING: Cannot find mailman config"
# NOTE: The first value below should be two single quotes, without a space.  A Bug is preventing us from displaying that properly in this article.
MAILMAN_LIST_ACCOUNTS = (' ',
                         '-admin', '-bounces', '-confirm', '-join', '-leave',
                         '-owner', '-request', '-subscribe', '-unsubscribe')
ZIMBRA_SOAP_HEADERS = {
    'Content-type': 'text/xml; charset=utf-8',
}

ZIMBRA_AUTH_REQUEST = \
    '<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope">' \
    '  <soap:Body>'                                                        \
    '    <AuthRequest xmlns="urn:zimbraAdmin">'                            \
    '      <name>%s</name>'                                                \
    '      <password>%s</password>'                                        \
    '    </AuthRequest>'                                                   \
    '  </soap:Body>'                                                       \
    '</soap:Envelope>'

ZIMBRA_REQUEST = \
    '<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope">' \
    '  <soap:Header>'                                                      \
    '    <context xmlns="urn:zimbra">'                                     \
    '      <authToken>%s</authToken>'                                      \
    '      <sessionId id="%s" type="admin">%s</sessionId>'                 \
    '    </context>'                                                       \
    '  </soap:Header>'                                                     \
    '  <soap:Body>%s</soap:Body>'                                          \
    '</soap:Envelope>'

ZIMBRA_CREATE_REQUEST = \
    '<CreateAccountRequest xmlns="urn:zimbraAdmin">' \
    '  <name>%s</name>'                              \
    '  <a n="zimbraMailTransport">smtp:%s</a>'       \
    '</CreateAccountRequest>'
ZIMBRA_GET_REQUEST = \
    '<GetAccountRequest xmlns="urn:zimbraAdmin">' \
    '  <account by="name">%s</account>'           \
    '</GetAccountRequest>'
ZIMBRA_DELETE_REQUEST = \
    '<DeleteAccountRequest xmlns="urn:zimbraAdmin">' \
    '  <id>%s</id>'                                  \
    '</DeleteAccountRequest>'

class ZimbraIntegration:
    def __init__(self):
        self.__authToken = None
        self.__sessionId = None
        try:
            self.__url       = mm_cfg.ZIMBRA_ADMIN_SOAP_SERVICE
            self.__transport = mm_cfg.MAILMAN_SMTP_TRANSPORT
            self.__username  = mm_cfg.ZIMBRA_ADMIN_USERNAME
            self.__password  = mm_cfg.ZIMBRA_ADMIN_PASSWORD
        except: # testing configuration
            self.__url = "https://192.168.230.130:7071/service/admin/soap/"
            self.__transport = 'rhel5-testn.testdomain.com'
            self.__username  = 'admin@testdomain.com'
            self.__password  = 'password'

    def __envelope(self, request):
        if self.__authToken is None:
            self.__getAuthToken()

        envelope = ZIMBRA_REQUEST % (self.__authToken, self.__sessionId,
               self.__sessionId, request)
        return envelope

    def __sendRequest(self, request):
        req = urllib2.Request(self.__url, request, ZIMBRA_SOAP_HEADERS)
        try:
            resp = urllib2.urlopen(req)
            tree = ElementTree.parse(resp)
        except urllib2.HTTPError, e:
            tree = ElementTree.parse(e.fp)
            error = tree.findtext(
                    "//{http://www.w3.org/2003/05/soap-envelope}Text")
            raise ZimbraIntegrationException(error)
        return tree

    def __getAuthToken(self):
        requestBody = ZIMBRA_AUTH_REQUEST % (self.__username, self.__password)
        tree = self.__sendRequest(requestBody)
        self.__authToken = tree.findtext("//{urn:zimbraAdmin}authToken")
        self.__sessionId = tree.findtext("//{urn:zimbraAdmin}sessionId")

    def __checkAccounts(self, name, domain):
        found = False
        for i in MAILMAN_LIST_ACCOUNTS:
            address = "%s%s@%s" % (name, i, domain)
            try:
                self.getAccount(address)
                found = True
            except:
                pass
            if found is True: # can't just immediately raise above
                raise ZimbraIntegrationException(
                        "accounts already exist for: " + name)

    def createAccounts(self, name, domain):
        self.__checkAccounts(name, domain)

        try:
            for i in MAILMAN_LIST_ACCOUNTS:
                address = "%s%s@%s" % (name, i, domain)
                self.createAccount(address)
        except ZimbraIntegrationException, e:
            self.__rollbackCreation(name, domain)
            raise ZimbraIntegrationException(e.reason)

    def __rollbackCreation(self, name, domain):
        self.deleteAccounts(name, domain)

    def getAccount(self, name):
        requestBody = ZIMBRA_GET_REQUEST % name
        requestBody = self.__envelope(requestBody)
        tree = self.__sendRequest(requestBody)
        node = tree.find("//{urn:zimbraAdmin}account")
        return node.get("id")

    def deleteAccounts(self, name, domain):
        for i in MAILMAN_LIST_ACCOUNTS:
            try:
                address = "%s%s@%s" % (name, i, domain)
                id = self.getAccount(address)
                self.deleteAccount(id)
            except:
                pass

    def deleteAccount(self, id):
        requestBody = self.__envelope(ZIMBRA_DELETE_REQUEST % id)
        self.__sendRequest(requestBody)

    def createAccount(self, name):
        requestBody = ZIMBRA_CREATE_REQUEST % (name, self.__transport)
        requestBody = self.__envelope(requestBody)
        self.__sendRequest(requestBody)

class ZimbraIntegrationException(Exception):
    def __init__(self, reason):
        self.reason = reason
    def __str__(self):
        return self.reason

zimbra_mailman.patch

diff -ur orig/Mailman/Cgi/create.py Mailman/Cgi/create.py
--- orig/Mailman/Cgi/create.py	2007-01-06 00:54:44.000000000 -0800
+++ Mailman/Cgi/create.py	2008-04-20 13:07:50.000000000 -0700
@@ -31,6 +31,7 @@
 from Mailman import i18n
 from Mailman.htmlformat import *
 from Mailman.Logging.Syslog import syslog
+from Mailman.ZimbraIntegration import ZimbraIntegrationException
 
 # Set up i18n
 _ = i18n._
@@ -217,6 +218,10 @@
                 _(Some unknown error occurred while creating the list.
                 Please contact the site administrator for assistance.))
             return
+	except ZimbraIntegrationException, e:
+	    request_creation(doc, cgidata,
+	        "Unable to create the list, zimbra error: %s" % e)
+	    return
 
         # Initialize the host_name and web_page_url attributes, based on
         # virtual hosting settings and the request environment variables.
diff -ur orig/Mailman/Cgi/rmlist.py Mailman/Cgi/rmlist.py
--- orig/Mailman/Cgi/rmlist.py	2007-01-06 00:54:44.000000000 -0800
+++ Mailman/Cgi/rmlist.py	2008-04-20 12:58:14.000000000 -0700
@@ -29,6 +29,7 @@
 from Mailman import i18n
 from Mailman.htmlformat import *
 from Mailman.Logging.Syslog import syslog
+from Mailman.ZimbraIntegration import ZimbraIntegration
 
 # Set up i18n
 _ = i18n._
@@ -157,6 +158,9 @@
                        'directory %s not deleted due to permission problems',
                        dir)
 
+    zi = ZimbraIntegration()
+    zi.deleteAccounts(listname, mlist.host_name)
+
     title = _('Mailing list deletion results')
     doc.SetTitle(title)
     table = Table(border=0, width='100%')
diff -ur orig/Mailman/MailList.py Mailman/MailList.py
--- orig/Mailman/MailList.py	2007-01-06 00:54:44.000000000 -0800
+++ Mailman/MailList.py	2008-04-20 12:45:10.000000000 -0700
@@ -46,6 +46,7 @@
 from Mailman import Errors
 from Mailman import LockFile
 from Mailman.UserDesc import UserDesc
+from Mailman.ZimbraIntegration import ZimbraIntegration
 
 # base classes
 from Mailman.Archiver import Archiver
@@ -486,6 +487,11 @@
             raise Errors.BadListNameError, postingaddr
         # Validate the admin's email address
         Utils.ValidateEmail(admin)
+
+	# Integrate with Zimbra
+        zi = ZimbraIntegration()
+	zi.createAccounts(name, emailhost)
+
         self._internal_name = name
         self._full_path = Site.get_listpath(name, create=1)
         # Don't use Lock() since that tries to load the non-existant config.pck
diff -ur orig/Mailman/mm_cfg.py Mailman/mm_cfg.py
--- orig/Mailman/mm_cfg.py	2008-04-19 09:20:44.000000000 -0700
+++ Mailman/mm_cfg.py	2008-04-20 13:04:05.000000000 -0700
@@ -80,6 +80,7 @@
 
 DEFAULT_URL_HOST   = 'YOUR_MAILMAN_WEB_HOSTNAME'
 DEFAULT_EMAIL_HOST = 'YOUR_EMAIL_DOMAINNAME'
+OWNERS_CAN_DELETE_THEIR_OWN_LISTS = True # delete to disable delete from web
 
 # Because we've overriden the virtual hosts above add_virtualhost
 # MUST be called after they have been defined.
@@ -94,3 +95,8 @@
 
 # Note - if you're looking for something that is imported from mm_cfg, but you
 # didn't find it above, it's probably in Defaults.py.
+
+ZIMBRA_ADMIN_SOAP_SERVICE = 'https://YOURZCSHOST:7071/service/admin/soap/'
+MAILMAN_SMTP_TRANSPORT = 'YOUR_MAILMAN_SMTP_HOSTNAME'
+ZIMBRA_ADMIN_USERNAME = 'YOUR_ZCS_ADMIN_USERNAME'
+ZIMBRA_ADMIN_PASSWORD = 'YOUR_ZCS_ADMIN_PASSWORD'
Verified Against: unknown Date Created: 4/24/2008
Article ID: https://wiki.zimbra.com/index.php?title=ZCS_Mailman_Integration Date Modified: 2008-07-01



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search