Zimbra Proxy Guide
|This article applies to the following ZCS versions.|
What is Zimbra Proxy
Zimbra Proxy (also referred in this wiki as Nginx-Zimbra or NZ in short) is an important component of ZCS. Zimbra Proxy is a high-performance proxy server that can be configured as a HTTP[S]/POP[S]/IMAP[S] proxy used to reverse proxy HTTP[S]/POP[S]/IMAP[S] client requests to a set of backend servers. It also provides functions like GSSAPI authentication, throttle control, SSL connection with different certificates for different virtual host names etc. In a typical use case, NZ extract user login information in someway (like account id or user name) and then fetches the route to the upstream mail server or web servers’ address from “Nginx Lookup Extension” (or called NLE for short), and finally proxy the interactions between clients and upstream ZCS servers. To accelerate the speed of route lookup, memcached is introduced, which caches the lookup result. Therefore, the subsequent login with the same username will directly be proxied without looking up in NLE.
Benefits and reasons to use
1. Zimbra proxy centralizes access to Mailbox servers
Zimbra Proxy allows mailbox servers to be hidden from public internet by acting as a reverse proxy & also allowing end users to access mail system via single Login URL instead of knowing their mailbox hostnames. It acts as the first entry point for all the HTTP[S]/POP[S]/IMAP[S] traffic and then intelligently routes all kind of static UI requests (HTML/CSS/JS etc) and Dynamic requests (SOAP/REST/IMAP/POP) to the appropriate upstream server.
2. Load Balancing
This is the reverse proxy function that people are most familiar with. Here the proxy routes incoming HTTP requests to a number of identical mail servers. The upstream mail server selection can be based on a simple client IP hash or round-robin algorithm. It’s such a common function that load balancing reverse proxies are usually just referred to as ‘load balancers’. There are specialized load balancing products available, but many general purpose reverse proxies also provide load balancing functionality.
A reverse proxy can hide the topology and characteristics of your back-end servers by removing the need for direct internet access to them. You can place your reverse proxy in an internet facing DMZ, but hide your web servers inside a non-public subnet.
You can use your reverse proxy to provide a single point of authentication for all HTTP requests. Although in case of ZCS, the authentication is provided by upstream mailstores where the user accounts reside.
5. SSL Termination
Here the reverse proxy handles incoming HTTPS connections, decrypting the requests and passing unencrypted requests on to the web servers.
Currently we use the memcached module with proxy to achieve caching of upstream routes to mailstores on a per end-client basis. This significantly reduces the route lookup time thereby improving the total time required to process the request and boost performance.
7. Centralised Logging and Auditing
Because all HTTP requests are routed through the reverse proxy, it makes an excellent point for logging and auditing.
8. URL Rewriting
Sometimes the URL scheme that a legacy application presents is not ideal for discovery or search engine optimisation. A reverse proxy can rewrite URLs before passing them on to your back-end servers.
To an external customer it appears that they are simply navigating a single website, but internally the organisation is maintaining three entirely separate sites. This approach can work extremely well for web service APIs where the reverse proxy provides a consistent single public facade to an internal distributed component oriented architecture.
Zimbra Proxy Components & Memcached
Zimbra Proxy is designed to provide a HTTP[S]/POP[S]/IMAP[S] reverse proxy that is quick, reliable, and scalable. Zimbra Proxy includes the following:
- Nginx. A high performance HTTP[S]/POP[S]/IMAP[S] proxy server which handles all incoming HTTP[S]/POP[S]/IMAP[S] requests.
- Zimbra Proxy Route Lookup Handler. This is a servlet (also named as Nginx Lookup Extension or NLE) located on the ZCS mailbox server. This servlet handles queries for the user account route information (the server and port number where the user account resides).
Memcached is a high performance, distributed memory object caching system. Route information is cached for further use in order to increase performance. zimbra-memcached is a separate package that is recommended to be installed along with zimbra-proxy.
Position in ZCS RuntimeThe figure to the right show the positions of NZ and its relationships to other components of ZCS.
Architecture and Flow
The following sequence explains the architecture and the login flow when an end client connects to Zimbra Proxy.
- End clients connect to Zimbra Proxy using HTTP[S]/POP[S]/IMAP[S] ports.
- Proxy will attempt to contact a memcached server (elected from the available memcached servers, using a round-robin algorithm) if available and with caching enabled to query the upstream route information for this particular client.
- If the route information is present in memcached, then this will be a cache-hit case and the proxy connects to the corresponding Zimbra Mailbox server right away and initiates a web/mail proxy session for this client. The Memcached component stores the route information for the configured period of time (configurable & one hour by default). Zimbra proxy will use this route information instead of querying the Zimbra Proxy Route Lookup Handler/NLE until the default period of time has expired.
- If the route information is not present in memcached, then this will be a cache-miss case, so Zimbra Proxy will proceed sending an HTTP request to an available Zimbra Proxy Route Lookup Handler/NLE (elected by Round-Robin), to look up the upstream mailbox server where this user account resides.
- Zimbra Proxy Route Lookup Handler/NLE locates the route information from LDAP for the account being accessed and returns this back to Zimbra Proxy.
- Zimbra Proxy uses this route information to connect to the corresponding Zimbra Mailbox server and initiates a web/mail proxy session. It will also cache this route information into a memcached server so that the next time this user logs in, the memcached server will have the upstream information available in its cache, and so Zimbra Proxy will not need to contact NLE.The end client is completely transparent to this and behaves as if it is connecting directly to Zimbra Mailbox server.
Deployment strategy and position with respect to other non-proxy hosts
The deployment strategy Zimbra actively suggests is to use Proxy server on the edge (either on an independent server or on the same server running LDAP/MTA) with mailbox servers behind it. In case of multiple proxies, an external load balancer can be placed in front to distribute the load evenly among the proxy servers. Please note that the Zimbra Proxy package does not act as a firewall and needs to be behind the firewall in customer deployments.
Zimbra Proxy Ports
The following ports are used either by Zimbra Proxy or by Zimbra Mailbox (if Proxy is not configured). If you have any other services running on these ports, turn them off.
End clients connect directly to Zimbra Proxy, using the Zimbra Proxy Ports. Zimbra Proxy connects to the Route Lookup Handler/NLE (which resides on Zimbra Mailbox server) using the Zimbra Mailbox Ports.
|Zimbra Proxy Ports (External to ZCS)||Port|
|POP3S (Secure POP3)||995|
|IMAPS (Secure IMAP)||993|
|Proxy Admin console||9071|
|Zimbra Mailbox Ports (Internal to ZCS)||Port|
|Route Lookup Handler||7072|
|HTTP Backend (if Proxy configured)||8080|
|HTTPS Backend (if Proxy configured)||8443|
|POP3 Backend (if Proxy configured)||7110|
|POP3S Backend (if Proxy configured)||7995|
|IMAP Backend (if Proxy configured)||7143|
|IMAPS Backend (if Proxy configured)||7993|
|Mailbox Admin console||7071|
Ldap attributes for the Proxy
Pre 8.5 attributes
To check the description, type, default value etc for any of the above attributes, simply do 'zmprov desc -a <attr>'
eg. zmprov desc -a zimbraReverseProxyWorkerConnections
zimbraReverseProxyWorkerConnections Maximum number of connections that an NGINX Proxy worker process is allowed to handle type : integer value : callback : immutable : false cardinality : single requiredIn : optionalIn : server,globalConfig flags : serverInherited defaults : 10240 min : 1 max : 40960 id : 725 requiresRestart : nginxproxy since : 5.0.10 deprecatedSince :
New 8.5 attributes
Proxy Configuration and Template files
Zimbra NGINX POP/IMAP/HTTP proxy configuration is generated by the zmproxyconfgen config generation script. This script reads in the proxy configuration template files, and generates the NGINX config files after performing keyword substitution on the template files with values from the LDAP configuration.
zmproxyconfgen is usually never invoked directly -- it is invoked automatically by zmproxyctl
The following sections describe the structure of the NGINX Proxy Configuration
Config Files and Config Templates
To simplify configuration, the NGINX configuration files have been split up into different config files based on functionality. The main, top-level configuration file is /opt/zimbra/conf/nginx.conf, and this file includes the main config, memcache config, mail config, and web config files. The mail config in turn includes the configuration for imap, imaps, pop3 and pop3s. The web config includes the configuration for http and https. The template files follow exactly the same inclusion hierarchy, and each configuration file has a corresponding template file from which it is generated.
Each template file resides in /opt/zimbra/conf/nginx/templates/
Each corresponding config file resides in /opt/zimbra/conf/nginx/includes/ (excluding top-level config file which is /opt/zimbra/conf/nginx/nginx.conf)
Config File Hierarchy
The symbol |_ indicates that a file is included by the one above it. Increasing levels of indentation indicate lower levels of config files
/opt/zimbra/conf/nginx.conf |_ /opt/zimbra/conf/nginx/includes/nginx.conf.main |_ /opt/zimbra/conf/nginx/includes/nginx.conf.memcache |_ /opt/zimbra/conf/nginx/includes/nginx.conf.zmlookup |_ /opt/zimbra/conf/nginx/includes/nginx.conf.mail |_ /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imap.default |_ /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imaps.default |_ /opt/zimbra/conf/nginx/includes/nginx.conf.mail.pop3.default |_ /opt/zimbra/conf/nginx/includes/nginx.conf.mail.pop3s.default |_ /opt/zimbra/conf/nginx/includes/nginx.conf.web |_ /opt/zimbra/conf/nginx/includes/nginx.conf.web.http.default |_ /opt/zimbra/conf/nginx/includes/nginx.conf.web.https.default |_ /opt/zimbra/conf/nginx/includes/nginx.conf.web.admin.default |_ /opt/zimbra/conf/nginx/includes/nginx.conf.web.sso.default
Description of Config Files
* /opt/zimbra/conf/nginx.conf Description: Core NGINX configuration file read by NGINX Proxy Template: /opt/zimbra/conf/nginx/templates/nginx.conf.template Includes: /opt/zimbra/conf/nginx/includes/nginx.conf.main /opt/zimbra/conf/nginx/includes/nginx.conf.memcache /opt/zimbra/conf/nginx/includes/nginx.conf.zmlookup /opt/zimbra/conf/nginx/includes/nginx.conf.mail /opt/zimbra/conf/nginx/includes/nginx.conf.web
* /opt/zimbra/conf/nginx/includes/nginx.conf.main Description: Defines global parameters for all NGINX worker processes Template: /opt/zimbra/conf/nginx/templates/nginx.conf.main.template Includes: None
* /opt/zimbra/conf/nginx/includes/nginx.conf.memcache Description: Defines memcache configuration, common for mail and web Template: /opt/zimbra/conf/nginx/templates/nginx.conf.memcache.template Includes: None
* /opt/zimbra/conf/nginx/includes/nginx.conf.mail Description: Defines the common mail configuration common to IMAP and POP3 Template: /opt/zimbra/conf/nginx/templates/nginx.conf.mail.template Includes: /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imap.default /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imaps.default /opt/zimbra/conf/nginx/includes/nginx.conf.mail.pop3.default /opt/zimbra/conf/nginx/includes/nginx.conf.mail.pop3s.default
* /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imap.default Description: Defines the server block for IMAP Template: /opt/zimbra/conf/nginx/templates/nginx.conf.mail.imap.default.template Includes: None
* /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imaps.default Description: Defines the server block for IMAPS Template: /opt/zimbra/conf/nginx/templates/nginx.conf.mail.imaps.default.template Includes: None
* /opt/zimbra/conf/nginx/includes/nginx.conf.mail.pop3.default Description: Defines the server block for POP3 Template: /opt/zimbra/conf/nginx/templates/nginx.conf.mail.pop3.default.template Includes: None
* /opt/zimbra/conf/nginx/includes/nginx.conf.mail.pop3s.default Description: Defines the server block for POP3S Template: /opt/zimbra/conf/nginx/templates/nginx.conf.mail.pop3s.default.template Includes: None
* /opt/zimbra/conf/nginx/includes/nginx.conf.web Description: Defines the common web configuration common to HTTP and HTTPS Template: /opt/zimbra/conf/nginx/templates/nginx.conf.web.template Includes: /opt/zimbra/conf/nginx/includes/nginx.conf.web.http.default /opt/zimbra/conf/nginx/includes/nginx.conf.web.https.default /opt/zimbra/conf/nginx/includes/nginx.conf.web.sso.default /opt/zimbra/conf/nginx/includes/nginx.conf.web.admin.default
* /opt/zimbra/conf/nginx/includes/nginx.conf.web.http.default Description: Defines the server block for HTTP Template: /opt/zimbra/conf/nginx/templates/nginx.conf.web.http.default.template Includes: None
* /opt/zimbra/conf/nginx/includes/nginx.conf.web.https.default Description: Defines the server block for HTTPS Template: /opt/zimbra/conf/nginx/templates/nginx.conf.web.https.default.template Includes: None
* /opt/zimbra/conf/nginx/includes/nginx.conf.web.sso.default Description: Defines the server block for Single Sign on (SSO) Template: /opt/zimbra/conf/nginx/templates/nginx.conf.web.sso.default.template Includes: None
* /opt/zimbra/conf/nginx/includes/nginx.conf.web.admin.default Description: Defines the server block for admin user Template: /opt/zimbra/conf/nginx/templates/nginx.conf.web.admin.default.template Includes: None
Proxy config rewrite(zmproxyconfgen), zmproxyconfig and zmproxyctl
Proxy config generation script 'zmproxyconfgen' is usually never invoked directly -- it is invoked automatically by zmproxyctl. So 'zmproxyctl restart' would essentially stop the proxy, rewrite the proxy config files and start the proxy again.
To change multiple LDAP attributes related to the proxy at once (i.e any of the zimbraReverseProxy* attrs) instead of using zmprov command to change each attribute individually, /opt/zimbra/libexec/zmproxyconfig can be used. The usage for this script is as follows
Usage: libexec/zmproxyconfig [-h] [-o] [-m] [-w] [-d [-r] [-s] [-a w1:w2:w3:w4] [-c [-n n1:n2]] [-i p1:p2:p3:p4] [-p p1:p2:p3:p4] [-x mailmode]] [-e [-a w1:w2:w3:w4] -C] [-n n1:n2 [-i p1:p2:p3:p4] [-p p1:p2:p3:p4] [-u|-U] [-x mailmode]] [-f] -H hostname
-h: display this help message
-H: Hostname of server on which enable/disable proxy functionality.
-a: Colon separated list of Web ports to use. Format: HTTP-STORE:HTTP-PROXY:HTTPS-STORE:HTTPS-PROXY (Ex: 8080:80:8443:443)
-d: disable proxy
-e: enable proxy
-f: Full reset on memcached port and search queries and POP/IMAP throttling.
-i: Colon separated list of IMAP ports to use. Format: IMAP-STORE:IMAP-PROXY:IMAPS-STORE:IMAPS-PROXY (Ex: 7143:143:7993:993)
-m: Toggle mail proxy portions
-o: Override enabled checks
-p: Colon separated list of POP ports to use. Format: POP-STORE:POP-PROXY:POPS-STORE:POPS-PROXY (Ex: 7110:110:7995:995)
-r: Run against a remote host. Note that this requires the server to be properly configured in the LDAP master.
-s: Set cleartext to FALSE (secure mode) on disable
-t: Disable reverse proxy lookup target for store server. Only valid with -d. Be sure that you intend for all proxy function for the server to be disabled
-w: Toggle Web proxy portions
-c: Disable Admin Console proxy portions.
-C: Enable Admin Console proxy portions.
-n: Colon separated list of Admin Console ports to use. Format: ADMIN-CONSOLE-STORE:ADMIN-CONSOLE-PROXY (Ex: 7071:9071)
-x: the proxy mail mode when enable proxy, or the store mail mode when disable proxy (Both default: http).
-u: disable SSL connection from proxy to mail store.
-U: enable SSL connection from proxy to mail store.
hostname is the value of the zimbra_server_hostname LC key for the server being modified.
Required options are -f by itself, or -f with -d or -e.
Note that -d or -e require one or both of -m and -w.
Note that -i or -p require -m.
Note that -a requires -w.
Note that -c/-C requires -w, and -n requires -c/-C. When disabling web proxy, admin console proxy will be automatically disabled.
Note that -u or -U are only available when proxy is enabled by -e.
Note that -x requires -w and -d for store.
Note that -x requires -w for proxy.
Note that no matter what mail mode is set by -x and no matter proxy is enabled or disabled, admin console's mode is always https.
The following are the defaults for -a, -i, -p, and -x if they are not supplied as options.
-a default on enable: 8080:80:8443:443
-a default on disable: 80:0:443:0
-i default on enable: 7143:143:7993:993
-i default on disable: 143:7143:993:7993
-p default on enable: 7110:110:7995:995
-p default on disable: 110:7110:995:7995
-n default on enable: 7071:9071
-n default on disable: 7071:9071
-x default on store disable: http
-x default on proxy enable/disable: http, but -x default on proxy enable when upstream ssl connection is enabled: https
eg. To change the value of zimbraReverseProxySSLToUpstreamEnabled from TRUE(default) to FALSE (i.e use http for the proxy<->mailstore connections instead of https), execute this on the server running the proxy
/opt/zimbra/libexec/zmproxyconfig -e -m -w -H <hostname> -u
You will have to restart the proxy after this by running 'zmproxyctl restart'
- Note: zmproxyconfig is no longer required to enable the proxy if you are doing a fresh installation of ZCS with proxy. The installer already runs this and the proxy is enabled for imap/pop/https access by default. Although it is needed while trying to deploy proxy in existing non-proxy environments when using the existing servers. Kindly refer http://wiki.zimbra.com/wiki/Enabling_Zimbra_Proxy#Using_existing_servers
Here are some basic troubleshooting tips for proxy related errors found in /opt/zimbra/log/nginx.log & /opt/zimbra/log/nginx.access.log
No route to host errors
If your getting "No route to host" errors in /opt/zimbra/log/nginx.log files on the proxy servers, you should check: The resolution [DNS] to the host from the proxy servers is correct and working. This might happen more when you've deployed a new ZCS server in your environment. Following step one, you also have to confirm there is port level access to between proxy servers and the server it's trying to reach. memcache port is 11211 . The server/s are too busy to serve the request. You have a server in the list that shouldn't be in the lookup pool or you should remove the trouble server from the pool to avoid any more customer issues to deescalate the situation.
1. 500 Internal Server Error: A generic error message, given when an unexpected condition was encountered and no more specific message is suitable. This might be due to several reasons due to failure on one of the upstream mailstore servers.
2. 502 Bad Gateway: The server was acting as a gateway or proxy and received an invalid response from the upstream server.
3. 503 Service Unavailable: The most common reason for this is that the jetty mailboxd process is down on the mailstore server and hence is unable to process the request. Check using 'netstat -an | grep <port>' to see if the IMAP/POP/HTTP ports used by mailboxd are up and listening for incoming connections. Also check using 'ps -eaf | grep mailboxd'
4. 504 Gateway Timeout: The most common reason for this is that the upstream mailstore server took longer than the configured timeout value and hence the client closed the upstream connection. Try changing the timeout values to a higher value if its expected to take longer for specific kind of requests. If this is not the case, then we should look for reasons why it took longer for a request to get processed.
These following configuration changes for an upstream mailbox server can improve proxy behavior related to timeouts:
- This will configure proxy to immediately reconnect to all mailbox server on any failure
$ zmprov mcf zimbraMailProxyReconnectTimeout 0
- If necessary, for each mailbox server
$ zmprov ms <mailbox server hostname> zimbraMailProxyReconnectTimeout 0
- This will configure proxy to ignore failures in regards to disconnects to all mailbox servers
$ zmprov mcf zimbraMailProxyMaxFails 0
- If necessary, for each mailbox server
$ zmprov ms <mailbox server hostname> zimbraMailProxyMaxFails 0
- Then, restart all the proxies to regenerate the nginx config files with the updated values for 'fail_timeout' & 'max_fails':
$ zmproxyctl restart
See the following bug for more details on this recommendation:
- "Improved proxy timeout defaults"
Zimbra Proxy Command-Line Utilities
The following commands are zmprov commands that are specific to Zimbra Proxy. For more information about using zmprov, refer to the Appendix A: Command-Line Utilities in the ZCS Administrator’s Guide, located on the Zimbra Website.
|Long Name||Short Name||Description|
|--getAllReverseProxyURLs||-garpu||Used to list all the upstream mailstore servers (NLEs) that should be used for reverse proxy lookup by the proxy|
|--getAllReverseProxyBackends||-garpb||Used to list all the upstream mailstore servers that are reverse-proxied by the proxy|
|--getAllMtaAuthURLs||-gamau||Used to publish into saslauthd.conf the servers that should be used for saslauthd.conf MTA auth|
|--getAllMemcachedServers||-gamcs||Used to list memcached servers (for Zimbra Proxy use)|