ZCS Mailman Integration: Difference between revisions

Line 70: Line 70:
My web config is below, and it should be trivial to get it working on zimbra's httpd.
My web config is below, and it should be trivial to get it working on zimbra's httpd.


<code>
<pre>
<VirtualHost lists.keekles.org:80>
<VirtualHost lists.keekles.org:80>
         ServerName lists.keekles.org
         ServerName lists.keekles.org
Line 109: Line 109:
ScriptAlias / /usr/lib/cgi-bin/mailman/listinfo
ScriptAlias / /usr/lib/cgi-bin/mailman/listinfo
</VirtualHost>
</VirtualHost>
</code>
</pre>
 
 


=Original Method Overview=
=Original Method Overview=

Revision as of 02:55, 21 February 2015


HOWTO: Integrate Zimbra Collaboration Suite with Mailman mailing lists


Introduction

There are multiple ways to integrate Zimbra and GNU mailman. Two will be presented here, as the second (original) method does not seem to be working for newer versions of Zimbra.

Single server config

This is on an Ubuntu 14.04.1 LTS server running Zimbra 8.6.0_GA_1153.FOSS. This arrangement has the following limitations

  • A subdomain must be used for the lists, i.e. lists.example.com
  • All email to this subdomain will be processed by mailman, no regular accounts will be allowed
  • The mailman project doesn't like this method for some reason
  • it's only been tested on a single server install of Zimbra

Install mailman

First we need to install mailman using apt-get

$ apt-get install mailman

Edit /etc/mailman/mm_cfg.py set the variables:

 MAILMAN_SITE_LIST = 'mailman'
 DEFAULT_URL_HOST = 'lists.keekles.org'
 DEFAULT_EMAIL_HOST = 'lists.keekles.org'
 MTA=None 

That's all the config needed for mailman

Configure Zimbra Postfix Transport

First we will configure the transport in /opt/zimbra/postfix/conf/master.cf.in

mailman unix - n n - - pipe

  flags=FR user=list argv=/usr/lib/mailman/bin/postfix-to-mailman.py
     ${nexthop} ${user}

Next we configure a map for this in /opt/zimbra/postfix/conf/transport mapping the subdomain to mailman: transport

lists.keekles.org mailman:

Zimbra postfix needs to be configured for this aditional file

$ zmprov mcf zimbraMtaTransportMaps 'proxy:ldap:/opt/zimbra/conf/ldap-transport.cf, lmdb:/opt/zimbra/postfix/conf/transport'

Finally generate the lmdb version of the transport file

$ postmap /opt/zimbra/postfix/conf/transport

configure the list virtual domain in zimbra

This will setup the domain as a catchall and use the mailman: transport for all email via the domain

$ zmprov cd lists.keekles.org $ zmprov md lists.keekles.org zimbraMailCatchAllAddress @lists.keekles.org $ zmprov md lists.keekles.org zimbraMailCatchAllForwardingAddress @lists.keekles.org $ zmprov md lists.keekles.org zimbraMailTransport mailman:

configure the web server

I am not running the webserver using the zimbra installed httpd, but there are serveral howto guides on this.

My web config is below, and it should be trivial to get it working on zimbra's httpd.

<VirtualHost lists.keekles.org:80>
        ServerName lists.keekles.org

        ServerAdmin webmaster@localhost

DocumentRoot /var/www/lists
ErrorLog /var/log/apache2/lists-error.log
CustomLog /var/log/apache2/lists-access.log combined
 <Directory /usr/lib/cgi-bin/mailman/>
    Options ExecCGI
    AllowOverride None
    Order deny,allow
    Allow from all
    Require all granted
  </Directory>
 ScriptAlias /mailman/       /usr/lib/cgi-bin/mailman/
ScriptAlias /cgi-bin/mailman/ /usr/lib/cgi-bin/mailman/
<Directory /var/lib/mailman/archives/>
    Options FollowSymLinks
    AllowOverride None
</Directory>

Alias /pipermail/ /var/lib/mailman/archives/public/
Alias /images/mailman/ /var/lib/mailman/icons/
ScriptAlias /admin /usr/lib/cgi-bin/mailman/admin
ScriptAlias /admindb /usr/lib/cgi-bin/mailman/admindb
ScriptAlias /confirm /usr/lib/cgi-bin/mailman/confirm
ScriptAlias /create /usr/lib/cgi-bin/mailman/create
ScriptAlias /edithtml /usr/lib/cgi-bin/mailman/edithtml
ScriptAlias /listinfo /usr/lib/cgi-bin/mailman/listinfo
ScriptAlias /options /usr/lib/cgi-bin/mailman/options
ScriptAlias /private /usr/lib/cgi-bin/mailman/private
ScriptAlias /rmlist /usr/lib/cgi-bin/mailman/rmlist
ScriptAlias /roster /usr/lib/cgi-bin/mailman/roster
ScriptAlias /subscribe /usr/lib/cgi-bin/mailman/subscribe
ScriptAlias /mailman/ /usr/lib/cgi-bin/mailman/
ScriptAlias / /usr/lib/cgi-bin/mailman/listinfo
</VirtualHost>

Original Method Overview

Many administrators would like to have robust mailing list management in conjunction with Zimbra. ZCS has distribution lists which works well for small groups with a central management interface; however, it does not provide support for self-help subscription, moderation or digests. Using ZCS together with Mailman is one way to fulfill the requirements, but requires a bit of massaging to work as one would expect.

One method of integration can be found on our forums at http://www.zimbra.com/forums/administrators/1380-solved-zimbra-mailman-howto.html

The above guide will allow a single-node ZCS install to coexist with Mailman on a single server, however, your mileage may vary. The referenced setup requires manual building of Mailman, integration with Zimbra/Apache-HTTPD, Zimbra-Postfix config changes and the work required to make it work does not appear to be insignificant. The guide also places restrictions on the email-domain that can be served by Mailman lists on the server.

In this guide, we will elaborate on a dual/multi-node ZCS/Mailman setup with step-by-step instructions as well provide patches to Mailman with a Python library to automatically interface with Zimbra. There will be no restriction on the domain which Mailman can serve. In addition to integrating Mailman, one can take the general ideas introduced by this document and apply it to any other list-serv of choice.

2. Getting started

Our supported configuration for using Mailman with Zimbra is to have Mailman on an external dedicated node. We do not want to have the Mailman setup interfering with the ZCS configuration.

This guide assumes knowledge of working with ZCS and the ability to get the basics of Mailman configured (such as installation, Apache HTTPD configuration, and mailing list management). Additionally, for convenience, we will assume that the setup will be performed on RHEL.

To continue, we will need:

  1. An additional server where we will run Mailman
  2. The name of the MTA for Zimbra (ZIMBRA_MTA_HOST)
  3. The name of the host on which Mailman will run (MAILMAN_HOSTNAME)
  4. The domain name for which we are receiving email (EMAIL_DOMAIN)
  5. To create an admin password for Mailman (MAILMAN_ADMIN_PASSWORD)
  6. To create a list creator password for Mailman (MAILMAN_CREATOR_PASSWORD)


3. Configuring Mailman

First, install Mailman and postfix; on RHEL 5, we do:

$ rpm -e --nodeps sendmail
$ yum install mailman postfix

which pulls down all dependencies such as httpd if it was not already present. It also forcibly removes sendmail and replaces it with postfix.

Edit /etc/mailman/mm_cfg.py set the variables:

 DEFAULT_URL_HOST = 'MAILMAN_HOSTNAME'
 DEFAULT_EMAIL_HOST = 'EMAIL_DOMAIN'

Run

$ /usr/lib/mailman/bin/update

Create the site-wide mailing list (default is 'mailman') and any other desired mailing lists:

$ /usr/lib/mailman/bin/newlist mailman

specify any valid email address as the list owner (including any address that is hosted on ZCS).


Edit and update /etc/aliases as directed by the output of the 'newlist' command. Run

$ /usr/bin/newaliases

to update the aliases table to include the mailing list addresses.


Setup the Mailman administrative passwords:

$ /usr/lib/mailman/bin/mmsitepass MAILMAN_ADMIN_PASSWORD
$ /usr/lib/mailman/bin/mmsitepass -c MAILMAN_CREATOR_PASSWORD


Configure postfix to accept email on EMAIL_DOMAIN and redirect unknown addresses to user@EMAIL_DOMAIN back into Zimbra (this will cover the cases where users on ZCS will also be included in the mailing lists)

Edit /etc/postfix/main.cf and set:

 mydomain = EMAIL_DOMAIN
 myorigin = $mydomain
 mydestination = $mydomain
 local_recipient_maps =
 fallback_transport = smtp:ZIMBRA_MTA_HOST


We are now ready to (re)start the mail, httpd and mailman services, do the following:

$ chkconfig httpd on
$ chkconfig postfix on
$ chkconfig mailman on
$ service httpd restart
$ service postfix restart
$ service mailman restart


4. Configuring ZCS

NOTE: This section can be bypassed if one follows the directions in '5. Further integration with Zimbra' below.


Now that Mailman has been properly configured, we will need to setup the addresses that will forward from ZCS to the Mailman host.

$ zmprov ca LIST_NAME@EMAIL_DOMAIN ANY_RANDOM_PASSWORD
$ zmprov ma LIST_NAME@EMAIL_DOMAIN zimbraMailTransport smtp:MAILMAN_HOSTNAME:25
$ zmprov ca LIST_NAME-admin@EMAIL_DOMAIN ANY_RANDOM_PASSWORD
$ zmprov ma LIST_NAME-admin@EMAIL_DOMAIN zimbraMailTransport smtp:MAILMAN_HOSTNAME:25
$ zmprov ca LIST_NAME-bounces@EMAIL_DOMAIN ANY_RANDOM_PASSWORD
$ zmprov ma LIST_NAME-bounces@EMAIL_DOMAIN zimbraMailTransport smtp:MAILMAN_HOSTNAME:25

Repeat the ca/ma commands above for the remaining -confirm, -join, -leave, -owner, -request, -subscribe, and -unsubscribe aliases.

With the above configuration complete, we may now use fully-featured mailing lists at LIST_NAME@EMAIL_DOMAIN.

ZCS 7.x note: this can now be accomplished by using a resource so that you don't have to use an account that may cost you money and keeps your account list cleaner:

$ zmprov ccr LIST_NAME@EMAIL_DOMAIN ANY_RANDOM_PASSWORD displayName DISPLAY_NAME zimbraCalResType Location zimbraMailTransport smtp:MAILMAN_HOSTNAME:25

5. Further integration with Zimbra

We can offer tighter integration of Mailman with ZCS using the attached ZimbraIntegration.py (to be installed in MAILMAN_HOME/Mailman). A simple patch to the Mailman installation in conjunction with the aforementioned ZimbraIntegration.py will enable us to ignore section 4 above altogether.

With the zimbra_mailman_integration.patch applied against Mailman every invocation of 'newlist', 'rmlist' and create/delete list from the web interface will automatically issue the appropriate zmprov commands to ZCS. Zimbra accounts with the zimbraMailTransport setting will be configured as necessary for the specified LIST_NAME.

To apply the integration, copy ZimbraIntegration.py into MAILMAN_HOME/Mailman, on systems where we installed Mailman using the RPM, it will be /usr/lib/mailman. Apply the patch:

$ cd /usr/lib/mailman; patch -p0 < zimbra_mailman_integration.patch

Edit /etc/mailman/mm_cfg.py, add the variables:

 OWNERS_CAN_DELETE_THEIR_OWN_LISTS = True # enable deleting lists from web UI
 ZIMBRA_ADMIN_SOAP_SERVICE = 'https://ZCSHOST:7071/service/admin/soap/'
 MAILMAN_SMTP_TRANSPORT = 'MAILMAN_HOSTNAME'
 ZIMBRA_ADMIN_USERNAME = 'YOUR_ZCS_ADMIN_USERNAME'
 ZIMBRA_ADMIN_PASSWORD = 'YOUR_ZCS_ADMIN_PASSWORD'

ZimbraIntegration.py exposes some APIs that can be re-used by non-Mailman list-servs and can serve as an example for users that might want to integrate ZCS with another list-serv.


6. Possible configurations (unsupported)

For small configurations, it is possible to run Mailman on the same host as ZCS, although strictly unsupported.

This is possible because Zimbra installs its own postfix into /opt/zimbra/postfix, the system may retain its own postfix installation and use that configuration to communicate to Mailman on the same node. The system-version of postfix must listen on a different port (ALT_SMTP_PORT) from the Zimbra-version of postfix. The zimbraMailTransport directive above would become smtp:ZIMBRA_MTA_HOST:ALT_SMTP_PORT

The remaining integration into Zimbra's Apache HTTPD configuration is left as an exercise to the reader.


6.1 Using a different mailing list manager (other than Mailman)

Following this guide presented thus-far, similar principles may be applied to any list-serv with respect to email routing and integration using ZimbraIntegration.py.

ZimbraIntegration.py

 import urllib2
 # try to import the ElementTree API included with py 2.5 first
 # otherwise fallback on third-party install (prereq)
 try:
     from xml.etree import ElementTree
 except:
     from elementtree import ElementTree
 
 # load mailman config
 try:
     from Mailman import mm_cfg
 except:
     print "WARNING: Cannot find mailman config"
 
 MAILMAN_LIST_ACCOUNTS = ('',
                          '-admin', '-bounces', '-confirm', '-join', '-leave',
                          '-owner', '-request', '-subscribe', '-unsubscribe')
 ZIMBRA_SOAP_HEADERS = {
     'Content-type': 'text/xml; charset=utf-8',
 }
 
 ZIMBRA_AUTH_REQUEST = \
     '<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope">' \
     '  <soap:Body>'                                                        \
     '    <AuthRequest xmlns="urn:zimbraAdmin">'                            \
     '      <name>%s</name>'                                                \
     '      <password>%s</password>'                                        \
     '    </AuthRequest>'                                                   \
     '  </soap:Body>'                                                       \
     '</soap:Envelope>'
 
 ZIMBRA_REQUEST = \
     '<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope">' \
     '  <soap:Header>'                                                      \
     '    <context xmlns="urn:zimbra">'                                     \
     '      <authToken>%s</authToken>'                                      \
     '      <sessionId id="%s" type="admin">%s</sessionId>'                 \
     '    </context>'                                                       \
     '  </soap:Header>'                                                     \
     '  <soap:Body>%s</soap:Body>'                                          \
     '</soap:Envelope>'
 
 ZIMBRA_CREATE_REQUEST = \
     '<CreateAccountRequest xmlns="urn:zimbraAdmin">' \
     '  <name>%s</name>'                              \
     '  <a n="zimbraMailTransport">smtp:%s</a>'       \
     '</CreateAccountRequest>'
 ZIMBRA_GET_REQUEST = \
     '<GetAccountRequest xmlns="urn:zimbraAdmin">' \
     '  <account by="name">%s</account>'           \
     '</GetAccountRequest>'
 ZIMBRA_DELETE_REQUEST = \
     '<DeleteAccountRequest xmlns="urn:zimbraAdmin">' \
     '  <id>%s</id>'                                  \
     '</DeleteAccountRequest>'
 
 class ZimbraIntegration:
     def __init__(self):
         self.__authToken = None
         self.__sessionId = None
         try:
             self.__url       = mm_cfg.ZIMBRA_ADMIN_SOAP_SERVICE
             self.__transport = mm_cfg.MAILMAN_SMTP_TRANSPORT
             self.__username  = mm_cfg.ZIMBRA_ADMIN_USERNAME
             self.__password  = mm_cfg.ZIMBRA_ADMIN_PASSWORD
         except: # testing configuration
             self.__url = "https://192.168.230.130:7071/service/admin/soap/"
             self.__transport = 'rhel5-testn.testdomain.com'
             self.__username  = 'admin@testdomain.com'
             self.__password  = 'password'
 
     def __envelope(self, request):
         if self.__authToken is None:
             self.__getAuthToken()
 
         envelope = ZIMBRA_REQUEST % (self.__authToken, self.__sessionId,
                self.__sessionId, request)
         return envelope
 
     def __sendRequest(self, request):
         req = urllib2.Request(self.__url, request, ZIMBRA_SOAP_HEADERS)
         try:
             resp = urllib2.urlopen(req)
             tree = ElementTree.parse(resp)
         except urllib2.HTTPError, e:
             tree = ElementTree.parse(e.fp)
             error = tree.findtext(
                     "//{http://www.w3.org/2003/05/soap-envelope}Text")
             raise ZimbraIntegrationException(error)
         return tree
 
     def __getAuthToken(self):
         requestBody = ZIMBRA_AUTH_REQUEST % (self.__username, self.__password)
         tree = self.__sendRequest(requestBody)
         self.__authToken = tree.findtext("//{urn:zimbraAdmin}authToken")
         self.__sessionId = tree.findtext("//{urn:zimbraAdmin}sessionId")
 
     def __checkAccounts(self, name, domain):
         found = False
         for i in MAILMAN_LIST_ACCOUNTS:
             address = "%s%s@%s" % (name, i, domain)
             try:
                 self.getAccount(address)
                 found = True
             except:
                 pass
             if found is True: # can't just immediately raise above
                 raise ZimbraIntegrationException(
                         "accounts already exist for: " + name)
 
     def createAccounts(self, name, domain):
         self.__checkAccounts(name, domain)
 
         try:
             for i in MAILMAN_LIST_ACCOUNTS:
                 address = "%s%s@%s" % (name, i, domain)
                 self.createAccount(address)
         except ZimbraIntegrationException, e:
             self.__rollbackCreation(name, domain)
             raise ZimbraIntegrationException(e.reason)
 
     def __rollbackCreation(self, name, domain):
         self.deleteAccounts(name, domain)
 
     def getAccount(self, name):
         requestBody = ZIMBRA_GET_REQUEST % name
         requestBody = self.__envelope(requestBody)
         tree = self.__sendRequest(requestBody)
         node = tree.find("//{urn:zimbraAdmin}account")
         return node.get("id")
 
     def deleteAccounts(self, name, domain):
         for i in MAILMAN_LIST_ACCOUNTS:
             try:
                 address = "%s%s@%s" % (name, i, domain)
                 id = self.getAccount(address)
                 self.deleteAccount(id)
             except:
                 pass
 
     def deleteAccount(self, id):
         requestBody = self.__envelope(ZIMBRA_DELETE_REQUEST % id)
         self.__sendRequest(requestBody)
 
     def createAccount(self, name):
         requestBody = ZIMBRA_CREATE_REQUEST % (name, self.__transport)
         requestBody = self.__envelope(requestBody)
         self.__sendRequest(requestBody)
 
 class ZimbraIntegrationException(Exception):
     def __init__(self, reason):
         self.reason = reason
     def __str__(self):
         return self.reason

zimbra_mailman.patch

 diff -ur orig/Mailman/Cgi/create.py Mailman/Cgi/create.py
 --- orig/Mailman/Cgi/create.py	2007-01-06 00:54:44.000000000 -0800
 +++ Mailman/Cgi/create.py	2008-04-20 13:07:50.000000000 -0700
 @@ -31,6 +31,7 @@
  from Mailman import i18n
  from Mailman.htmlformat import *
  from Mailman.Logging.Syslog import syslog
 +from Mailman.ZimbraIntegration import ZimbraIntegrationException
  
  # Set up i18n
  _ = i18n._
 @@ -217,6 +218,10 @@
                  _('''Some unknown error occurred while creating the list.
                  Please contact the site administrator for assistance.'''))
              return
 +	except ZimbraIntegrationException, e:
 +	    request_creation(doc, cgidata,
 +	        "Unable to create the list, zimbra error: %s" % e)
 +	    return
  
          # Initialize the host_name and web_page_url attributes, based on
          # virtual hosting settings and the request environment variables.
 diff -ur orig/Mailman/Cgi/rmlist.py Mailman/Cgi/rmlist.py
 --- orig/Mailman/Cgi/rmlist.py	2007-01-06 00:54:44.000000000 -0800
 +++ Mailman/Cgi/rmlist.py	2008-04-20 12:58:14.000000000 -0700
 @@ -29,6 +29,7 @@
  from Mailman import i18n
  from Mailman.htmlformat import *
  from Mailman.Logging.Syslog import syslog
 +from Mailman.ZimbraIntegration import ZimbraIntegration
  
  # Set up i18n
  _ = i18n._
 @@ -157,6 +158,9 @@
                         'directory %s not deleted due to permission problems',
                         dir)
  
 +    zi = ZimbraIntegration()
 +    zi.deleteAccounts(listname, mlist.host_name)
 +
      title = _('Mailing list deletion results')
      doc.SetTitle(title)
      table = Table(border=0, width='100%')
 diff -ur orig/Mailman/MailList.py Mailman/MailList.py
 --- orig/Mailman/MailList.py	2007-01-06 00:54:44.000000000 -0800
 +++ Mailman/MailList.py	2008-04-20 12:45:10.000000000 -0700
 @@ -46,6 +46,7 @@
  from Mailman import Errors
  from Mailman import LockFile
  from Mailman.UserDesc import UserDesc
 +from Mailman.ZimbraIntegration import ZimbraIntegration
  
  # base classes
  from Mailman.Archiver import Archiver
 @@ -486,6 +487,11 @@
              raise Errors.BadListNameError, postingaddr
          # Validate the admin's email address
          Utils.ValidateEmail(admin)
 +
 +	# Integrate with Zimbra
 +        zi = ZimbraIntegration()
 +	zi.createAccounts(name, emailhost)
 +
          self._internal_name = name
          self._full_path = Site.get_listpath(name, create=1)
          # Don't use Lock() since that tries to load the non-existant config.pck
 diff -ur orig/Mailman/mm_cfg.py Mailman/mm_cfg.py
 --- orig/Mailman/mm_cfg.py	2008-04-19 09:20:44.000000000 -0700
 +++ Mailman/mm_cfg.py	2008-04-20 13:04:05.000000000 -0700
 @@ -80,6 +80,7 @@
  
  DEFAULT_URL_HOST   = 'YOUR_MAILMAN_WEB_HOSTNAME'
  DEFAULT_EMAIL_HOST = 'YOUR_EMAIL_DOMAINNAME'
 +OWNERS_CAN_DELETE_THEIR_OWN_LISTS = True # delete to disable delete from web
  
  # Because we've overriden the virtual hosts above add_virtualhost
  # MUST be called after they have been defined.
 @@ -94,3 +95,8 @@
  
  # Note - if you're looking for something that is imported from mm_cfg, but you
  # didn't find it above, it's probably in Defaults.py.
 +
 +ZIMBRA_ADMIN_SOAP_SERVICE = 'https://YOURZCSHOST:7071/service/admin/soap/'
 +MAILMAN_SMTP_TRANSPORT = 'YOUR_MAILMAN_SMTP_HOSTNAME'
 +ZIMBRA_ADMIN_USERNAME = 'YOUR_ZCS_ADMIN_USERNAME'
 +ZIMBRA_ADMIN_PASSWORD = 'YOUR_ZCS_ADMIN_PASSWORD'

I haven't tested other versions, but at least with Mailman 2.1.9 you need also to patch MAILMAN_DIR/bin/rmlist. For reasons unknown to me, MAILMAN_DIR/bin/newlist will create the Zimbra mailboxes but MAILMAN_DIR/bin/rmlist, unpatched, will not remove them.

diff -ur bin/rmlist.orig bin/rmlist
--- bin/rmlist.orig	2010-09-19 22:12:11.000000000 -0700
+++ bin/rmlist	2010-09-19 22:08:42.000000000 -0700
@@ -46,6 +46,7 @@
 from Mailman import Utils
 from Mailman import MailList
 from Mailman.i18n import _
+from Mailman.ZimbraIntegration import ZimbraIntegration
 
 try:
     True, False
@@ -113,6 +114,8 @@
     REMOVABLES = []
     if Utils.list_exists(listname):
         mlist = MailList.MailList(listname, lock=0)
+        zi = ZimbraIntegration()
+        zi.deleteAccounts(listname, mlist.host_name)
 
         # Do the MTA-specific list deletion tasks
         if mm_cfg.MTA:
Verified Against: unknown Date Created: 4/24/2008
Article ID: https://wiki.zimbra.com/index.php?title=ZCS_Mailman_Integration Date Modified: 2015-02-21



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search