Difference between revisions of "Zimbra Proxy Guide"

(Position in ZCS Runtime)
(Proxy config rewrite(zmproxyconfgen), zmproxyconfig and zmproxyctl)
Line 548: Line 548:
You will have to restart the proxy after this by running 'zmproxyctl restart'
You will have to restart the proxy after this by running 'zmproxyctl restart'
*Note: zmproxyconfig is no longer mandatory to enable the proxy. The installer already runs this and the proxy is enabled for https access by default
*Note: zmproxyconfig is no longer required to enable the proxy if you are doing a fresh installation of ZCS with proxy. The installer already runs this and the proxy is enabled for imap/pop/https access by default. Although it is needed while trying to deploy proxy in existing non-proxy environments when using the existing servers. Kindly refer http://wiki.zimbra.com/wiki/Enabling_Zimbra_Proxy#Using_existing_servers
== Troubleshooting ==
== Troubleshooting ==

Revision as of 20:33, 29 July 2014

Admin Article

Article Information

This article applies to the following ZCS versions.

ZCS 8.0 Article ZCS 8.0

Overview of proxy and related components

What is Zimbra Proxy

Zimbra Proxy (also referred in this wiki as Nginx-Zimbra or NZ in short) is an important component of ZCS. Zimbra Proxy is a high-performance proxy server that can be configured as a POP3/IMAP/HTTP proxy used to reverse proxy IMAP/POP3 and HTTP client requests to a set of backend servers. It also provides functions like GSSAPI authentication, throttle control, SSL connection with different certificates for different virtual host names etc. In a typical use case, NZ extract user login information in someway (like account id or user name) and then fetches the route to the upstream mail server or web servers’ address from “Nginx Lookup Extension” (or called NLE for short), and finally proxy the interactions between clients and upstream ZCS servers. To accelerate the speed of route lookup, memcached is introduced, which caches the lookup result. Therefore, the subsequent login with the same username will directly be proxied without looking up in NLE.

Benefits and reasons to use

1. Zimbra proxy centralizes access to Mailbox servers

Zimbra Proxy allows mailbox servers to be hidden from public internet by acting as a reverse proxy & also allowing end users to access mail system via single Login URL instead of knowing their mailbox hostnames. It acts as the first entry point for all the HTTP/IMAP/POP traffic and then intelligently routes all kind of static UI requests (HTML/CSS/JS etc) and Dynamic requests (SOAP/REST/IMAP/POP) to the appropriate upstream server. Here are some other benefits of using the proxy as well

2. Load Balancing

This is the reverse proxy function that people are most familiar with. Here the proxy routes incoming HTTP requests to a number of identical mail servers. The upstream mail server selection can be based on a simple client IP hash or round-robin algorithm. It’s such a common function that load balancing reverse proxies are usually just referred to as ‘load balancers’. There are specialized load balancing products available, but many general purpose reverse proxies also provide load balancing functionality.

3. Security

A reverse proxy can hide the topology and characteristics of your back-end servers by removing the need for direct internet access to them. You can place your reverse proxy in an internet facing DMZ, but hide your web servers inside a non-public subnet.

4. Authentication

You can use your reverse proxy to provide a single point of authentication for all HTTP requests. Although in case of ZCS, the authentication is provided by upstream mailstores where the user accounts reside.

5. SSL Termination

Here the reverse proxy handles incoming HTTPS connections, decrypting the requests and passing unencrypted requests on to the web servers.

6. Caching

Currently we use the memcached module with proxy to achieve caching of upstream routes to mailstores on a per end-client basis. This significantly reduces the route lookup time thereby improving the total time required to process the request and boost performance.

7. Centralised Logging and Auditing

Because all HTTP requests are routed through the reverse proxy, it makes an excellent point for logging and auditing.

8. URL Rewriting

Sometimes the URL scheme that a legacy application presents is not ideal for discovery or search engine optimisation. A reverse proxy can rewrite URLs before passing them on to your back-end servers.

To an external customer it appears that they are simply navigating a single website, but internally the organisation is maintaining three entirely separate sites. This approach can work extremely well for web service APIs where the reverse proxy provides a consistent single public facade to an internal distributed component oriented architecture.

Zimbra Proxy Components

Zimbra Proxy is designed to provide a HTTP/POP/IMAP proxy that is quick, reliable, and scalable. Zimbra Proxy includes the following:

  • Nginx. A high performance HTTP/IMAP/POP3 proxy server which handles all incoming HTTP/POP/IMAP requests.
  • Memcached. A high performance, distributed memory object caching system. Route information is cached for further use in order to increase performance.
  • Zimbra Proxy Route Lookup Handler. This is a servlet located on the ZCS mailbox server. This servlet handles queries for the user account route information (the server and port number where the user account resides).

Architecture and Flow

The following sequence shows the architecture and flow of Zimbra Proxy.

  1. End clients connect to Zimbra Proxy using HTTP/HTTPS/POP/IMAP ports.
  2. When Zimbra Proxy receives an incoming connection, the Nginx component sends an HTTP request to Zimbra Proxy Route Lookup Handler component.
  3. Zimbra Proxy Route Lookup Handler locates the route information for the account being accessed and returns this to Nginx.
  4. The Memcached component stores the route information for the configured period of time (by default, this time is one hour). Nginx will use this route information instead of querying the Zimbra Proxy Route Lookup Handler until the default period of time has expired.
  5. Nginx uses the route information to connect to Zimbra Mailbox.
  6. Zimbra Proxy connects to Zimbra Mailbox and initiates the web/mail proxy session. The end client behaves as if it is connecting directly to Zimbra Mailbox.

Position in ZCS Runtime

The figure to the right show the positions of NZ and its relationships to other components of ZCS.
Proxy position in ZCS runtime
With Zimbra Proxy, a general workflow of login is like this. On a client login, NZ will attempt to contact a memcached server (the exact server will be elected from the available candidates, using an round-robin algorithm). The memcached server is expected to return the upstream route information for that particular client. If the information is not present in that memcached server, then this will be a cache-miss, so Zimbra Proxy will proceed to contact an available NLE (elected by Round-Robin), to look up the upstream server information.

Once the upstream server is known, Zimbra Proxy will immediately initiate the proxy session. And then, it will cache the upstream information into the memcached server. The next time the user logs in, the memcached server will have the upstream information available in its cache, and so Zimbra Proxy will not need to contact NLE.

Deployment strategies, server specifications, impact to other non-proxy hosts

The deployment strategy Zimbra actively suggests is to use Proxy on the edge with mailbox servers behind the firewall. Please note that the Zimbra Proxy package does not act as a firewall and needs to be behind the firewall in customer deployments.

Zimbra Proxy Ports

The following ports are used either by Zimbra Proxy or by Zimbra Mailbox (if Proxy is not configured). If you have any other services running on these ports, turn them off.

End clients connect directly to Zimbra Proxy, using the Zimbra Proxy Ports. Zimbra Proxy connects to the Route Lookup Handler or Zimbra Mailbox using the Zimbra Mailbox Ports.

Zimbra Proxy Ports (External to ZCS) Port
POP3 110
POP3S (Secure POP3) 995
IMAP 143
IMAPS (Secure IMAP) 993
Zimbra Mailbox Ports (Internal to ZCS) Port
Route Lookup Handler 7072
HTTP Backend (if Proxy configured) 8080
HTTPS Backend (if Proxy configured) 8443
POP3 Backend (if Proxy configured) 7110
POP3S Backend (if Proxy configured) 7995
IMAP Backend (if Proxy configured) 7143
IMAPS Backend (if Proxy configured) 7993

Ldap attributes for the Proxy

Pre 8.5 attributes










































































To check the description, type, default value etc for any of the above attributes, simply do 'zmprov desc -a <attr>'

eg. zmprov desc -a zimbraReverseProxyWorkerConnections

   Maximum number of connections that an NGINX Proxy worker process is
   allowed to handle
              type : integer
             value : 
          callback : 
         immutable : false
       cardinality : single
        requiredIn : 
        optionalIn : server,globalConfig
             flags : serverInherited
          defaults : 10240
               min : 1
               max : 40960
                id : 725
   requiresRestart : nginxproxy
             since : 5.0.10
   deprecatedSince :

New 8.5 attributes





Proxy Configuration and Template files

Zimbra NGINX POP/IMAP/HTTP proxy configuration is generated by the zmproxyconfgen config generation script. This script reads in the proxy configuration template files, and generates the NGINX config files after performing keyword substitution on the template files with values from the LDAP configuration.

zmproxyconfgen is usually never invoked directly -- it is invoked automatically by zmproxyctl

The following sections describe the structure of the NGINX Proxy Configuration

Config Files and Config Templates

To simplify configuration, the NGINX configuration files have been split up into different config files based on functionality. The main, top-level configuration file is /opt/zimbra/conf/nginx.conf, and this file includes the main config, memcache config, mail config, and web config files. The mail config in turn includes the configuration for imap, imaps, pop3 and pop3s. The web config includes the configuration for http and https. The template files follow exactly the same inclusion hierarchy, and each configuration file has a corresponding template file from which it is generated.

Each template file resides in /opt/zimbra/conf/nginx/templates/

Each corresponding config file resides in /opt/zimbra/conf/nginx/includes/ (excluding top-level config file which is /opt/zimbra/conf/nginx/nginx.conf)

Config File Hierarchy

The symbol |_ indicates that a file is included by the one above it. Increasing levels of indentation indicate lower levels of config files

    |_ /opt/zimbra/conf/nginx/includes/nginx.conf.main
    |_ /opt/zimbra/conf/nginx/includes/nginx.conf.memcache
    |_ /opt/zimbra/conf/nginx/includes/nginx.conf.zmlookup
    |_ /opt/zimbra/conf/nginx/includes/nginx.conf.mail
       |_ /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imap.default
       |_ /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imaps.default
       |_ /opt/zimbra/conf/nginx/includes/nginx.conf.mail.pop3.default
       |_ /opt/zimbra/conf/nginx/includes/nginx.conf.mail.pop3s.default
    |_ /opt/zimbra/conf/nginx/includes/nginx.conf.web
       |_ /opt/zimbra/conf/nginx/includes/nginx.conf.web.http.default
       |_ /opt/zimbra/conf/nginx/includes/nginx.conf.web.https.default
       |_ /opt/zimbra/conf/nginx/includes/nginx.conf.web.admin.default  
       |_ /opt/zimbra/conf/nginx/includes/nginx.conf.web.sso.default 

Description of Config Files

   * /opt/zimbra/conf/nginx.conf
       Core NGINX configuration file read by NGINX Proxy
   * /opt/zimbra/conf/nginx/includes/nginx.conf.main
       Defines global parameters for all NGINX worker processes
   * /opt/zimbra/conf/nginx/includes/nginx.conf.memcache
       Defines memcache configuration, common for mail and web
   * /opt/zimbra/conf/nginx/includes/nginx.conf.mail
       Defines the common mail configuration common to IMAP and POP3
   * /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imap.default
       Defines the server block for IMAP 
   * /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imaps.default
       Defines the server block for IMAPS
   * /opt/zimbra/conf/nginx/includes/nginx.conf.mail.pop3.default
       Defines the server block for POP3
   * /opt/zimbra/conf/nginx/includes/nginx.conf.mail.pop3s.default
       Defines the server block for POP3S
   * /opt/zimbra/conf/nginx/includes/nginx.conf.web
       Defines the common web configuration common to HTTP and HTTPS
   * /opt/zimbra/conf/nginx/includes/nginx.conf.web.http.default
       Defines the server block for HTTP
   * /opt/zimbra/conf/nginx/includes/nginx.conf.web.https.default
       Defines the server block for HTTPS
   * /opt/zimbra/conf/nginx/includes/nginx.conf.web.sso.default
       Defines the server block for Single Sign on (SSO)
   * /opt/zimbra/conf/nginx/includes/nginx.conf.web.admin.default
       Defines the server block for admin user

Proxy config rewrite(zmproxyconfgen), zmproxyconfig and zmproxyctl

Proxy config generation script 'zmproxyconfgen' is usually never invoked directly -- it is invoked automatically by zmproxyctl. So 'zmproxyctl restart' would essentially stop the proxy, rewrite the proxy config files and start the proxy again.

To change multiple LDAP attributes related to the proxy at once (i.e any of the zimbraReverseProxy* attrs) instead of using zmprov command to change each attribute individually, /opt/zimbra/libexec/zmproxyconfig can be used. The usage for this script is as follows

zimbra@zdev-vmxyz:~$/opt/zimbra/libexec/zmproxyconfig help

Usage: libexec/zmproxyconfig [-h] [-o] [-m] [-w] [-d [-r] [-s] [-a w1:w2:w3:w4] [-c [-n n1:n2]] [-i p1:p2:p3:p4] [-p p1:p2:p3:p4] [-x mailmode]] [-e [-a w1:w2:w3:w4] -C] [-n n1:n2 [-i p1:p2:p3:p4] [-p p1:p2:p3:p4] [-u|-U] [-x mailmode]] [-f] -H hostname

-h: display this help message

-H: Hostname of server on which enable/disable proxy functionality.

-a: Colon separated list of Web ports to use. Format: HTTP-STORE:HTTP-PROXY:HTTPS-STORE:HTTPS-PROXY (Ex: 8080:80:8443:443)

-d: disable proxy

-e: enable proxy

-f: Full reset on memcached port and search queries and POP/IMAP throttling.

-i: Colon separated list of IMAP ports to use. Format: IMAP-STORE:IMAP-PROXY:IMAPS-STORE:IMAPS-PROXY (Ex: 7143:143:7993:993)

-m: Toggle mail proxy portions

-o: Override enabled checks

-p: Colon separated list of POP ports to use. Format: POP-STORE:POP-PROXY:POPS-STORE:POPS-PROXY (Ex: 7110:110:7995:995)

-r: Run against a remote host. Note that this requires the server to be properly configured in the LDAP master.

-s: Set cleartext to FALSE (secure mode) on disable

-t: Disable reverse proxy lookup target for store server. Only valid with -d. Be sure that you intend for all proxy function for the server to be disabled

-w: Toggle Web proxy portions

-c: Disable Admin Console proxy portions.

-C: Enable Admin Console proxy portions.

-n: Colon separated list of Admin Console ports to use. Format: ADMIN-CONSOLE-STORE:ADMIN-CONSOLE-PROXY (Ex: 7071:9071)

-x: the proxy mail mode when enable proxy, or the store mail mode when disable proxy (Both default: http).

-u: disable SSL connection from proxy to mail store.

-U: enable SSL connection from proxy to mail store.

hostname is the value of the zimbra_server_hostname LC key for the server being modified.

Required options are -f by itself, or -f with -d or -e.

Note that -d or -e require one or both of -m and -w.

Note that -i or -p require -m.

Note that -a requires -w.

Note that -c/-C requires -w, and -n requires -c/-C. When disabling web proxy, admin console proxy will be automatically disabled.

Note that -u or -U are only available when proxy is enabled by -e.

Note that -x requires -w and -d for store.

Note that -x requires -w for proxy.

Note that no matter what mail mode is set by -x and no matter proxy is enabled or disabled, admin console's mode is always https.

The following are the defaults for -a, -i, -p, and -x if they are not supplied as options.

-a default on enable: 8080:80:8443:443

-a default on disable: 80:0:443:0

-i default on enable: 7143:143:7993:993

-i default on disable: 143:7143:993:7993

-p default on enable: 7110:110:7995:995

-p default on disable: 110:7110:995:7995

-n default on enable: 7071:9071

-n default on disable: 7071:9071

-x default on store disable: http

-x default on proxy enable/disable: http, but -x default on proxy enable when upstream ssl connection is enabled: https

eg. To change the value of zimbraReverseProxySSLToUpstreamEnabled from TRUE(default) to FALSE (i.e use http for the proxy<->mailstore connections instead of https), execute this on the server running the proxy

/opt/zimbra/libexec/zmproxyconfig -e -m -w -H <hostname> -u

You will have to restart the proxy after this by running 'zmproxyctl restart'

  • Note: zmproxyconfig is no longer required to enable the proxy if you are doing a fresh installation of ZCS with proxy. The installer already runs this and the proxy is enabled for imap/pop/https access by default. Although it is needed while trying to deploy proxy in existing non-proxy environments when using the existing servers. Kindly refer http://wiki.zimbra.com/wiki/Enabling_Zimbra_Proxy#Using_existing_servers


Here are some basic troubleshooting tips for proxy related errors found in /opt/zimbra/log/nginx.log & /opt/zimbra/log/nginx.access.log

No route to host errors

If your getting "No route to host" errors in /opt/zimbra/log/nginx.log files on the proxy servers, you should check: The resolution [DNS] to the host from the proxy servers is correct and working. This might happen more when you've deployed a new ZCS server in your environment. Following step one, you also have to confirm there is port level access to between proxy servers and the server it's trying to reach. memcache port is 11211 . The server/s are too busy to serve the request. You have a server in the list that shouldn't be in the lookup pool or you should remove the trouble server from the pool to avoid any more customer issues to deescalate the situation.

5xx Errors

1. 500 Internal Server Error: A generic error message, given when an unexpected condition was encountered and no more specific message is suitable. This might be due to several reasons due to failure on one of the upstream mailstore servers.

2. 502 Bad Gateway: The server was acting as a gateway or proxy and received an invalid response from the upstream server.

3. 503 Service Unavailable: The most common reason for this is that the jetty mailboxd process is down on the mailstore server and hence is unable to process the request. Check using 'netstat -an | grep <port>' to see if the IMAP/POP/HTTP ports used by mailboxd are up and listening for incoming connections. Also check using 'ps -eaf | grep mailboxd'

4. 504 Gateway Timeout: The most common reason for this is that the upstream mailstore server took longer than the configured timeout value and hence the client closed the upstream connection. Try changing the timeout values to a higher value if its expected to take longer for specific kind of requests. If this is not the case, then we should look for reasons why it took longer for a request to get processed.

These proxy configuration changes can improve proxy behavior related to timeouts:

  • This will configure proxy to immediately reconnect on any failure
    • $ zmprov mcf zimbraMailProxyReconnectTimeout 0 
    • If necessary, on each proxy
      • $ zmprov ms `zmhostname` zimbraMailProxyReconnectTimeout 0 
  • This will configure proxy to ignore failures in regards to disconnects
    • $ zmprov mcf zimbraMailProxyMaxFails 0 
    • If necessary, on each proxy
      • $ zmprov ms `zmhostname` zimbraMailProxyMaxFails 0 
  • Then, restart the processes on all proxies:
  •  $ zmproxyctl restart 

See the following bug for more details on this recommendation:

Zimbra Proxy Command-Line Utilities

The following commands are zmprov commands that are specific to Zimbra Proxy. For more information about using zmprov, refer to the Appendix A: Command-Line Utilities in the ZCS Administrator’s Guide, located on the Zimbra Website.


zmprov [cmd]


Long Name Short Name Description
--getAllReverseProxyURLs -garpu Used to list all the upstream mailstore servers (NLEs) that should be used for reverse proxy lookup by the proxy
--getAllReverseProxyBackends -garpb Used to list all the upstream mailstore servers that are reverse-proxied by the proxy
--getAllMtaAuthURLs -gamau Used to publish into saslauthd.conf the servers that should be used for saslauthd.conf MTA auth
--getAllMemcachedServers -gamcs Used to list memcached servers (for Zimbra Proxy use)

Verified Against: ZCS 8.5.x Date Created: 6/2/2014
Article ID: https://wiki.zimbra.com/index.php?title=Zimbra_Proxy_Guide Date Modified: 2014-07-29

Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search