Zimbra Proxy Guide
Article Information |
---|
This article applies to the following ZCS versions. |
What is Zimbra Proxy
Zimbra Proxy (also referred in this wiki as Nginx-Zimbra or NZ in short) is an important component of ZCS. Zimbra Proxy is a high-performance proxy server that can be configured as a POP3/IMAP/HTTP proxy used to reverse proxy IMAP/POP3 and HTTP client requests to a set of backend servers. It also provides functions like GSSAPI authentication, throttle control, SSL connection with different certificates for different virtual host names etc. In a typical use case, NZ extract user login information in someway (like account id or user name) and then fetches the route to the upstream mail server or web servers’ address from “Nginx Lookup Extension” (or called NLE for short), and finally proxy the interactions between clients and upstream ZCS servers. To accelerate the speed of route lookup, memcached is introduced, which caches the lookup result. Therefore, the subsequent login with the same username will directly be proxied without looking up in NLE.
Benefits and reasons to use
1. Zimbra proxy centralizes access to Mailbox servers
Zimbra Proxy allows mailbox servers to be hidden from public internet by acting as a reverse proxy & also allowing end users to access mail system via single Login URL instead of knowing their mailbox hostnames. It acts as the first entry point for all the HTTP/IMAP/POP traffic and then intelligently routes all kind of static UI requests (HTML/CSS/JS etc) and Dynamic requests (SOAP/REST/IMAP/POP) to the appropriate upstream server. Here are some other benefits of using the proxy as well
2. Load Balancing
This is the reverse proxy function that people are most familiar with. Here the proxy routes incoming HTTP requests to a number of identical mail servers. The upstream mail server selection can be based on a simple client IP hash or round-robin algorithm. It’s such a common function that load balancing reverse proxies are usually just referred to as ‘load balancers’. There are specialized load balancing products available, but many general purpose reverse proxies also provide load balancing functionality.
3. Security
A reverse proxy can hide the topology and characteristics of your back-end servers by removing the need for direct internet access to them. You can place your reverse proxy in an internet facing DMZ, but hide your web servers inside a non-public subnet.
4. Authentication
You can use your reverse proxy to provide a single point of authentication for all HTTP requests. Although in case of ZCS, the authentication is provided by upstream mailstores where the user accounts reside.
5. SSL Termination
Here the reverse proxy handles incoming HTTPS connections, decrypting the requests and passing unencrypted requests on to the web servers.
6. Caching
Currently we use the memcached module with proxy to achieve caching of upstream routes to mailstores on a per end-client basis. This significantly reduces the route lookup time thereby improving the total time required to process the request and boost performance.
7. Centralised Logging and Auditing
Because all HTTP requests are routed through the reverse proxy, it makes an excellent point for logging and auditing.
8. URL Rewriting
Sometimes the URL scheme that a legacy application presents is not ideal for discovery or search engine optimisation. A reverse proxy can rewrite URLs before passing them on to your back-end servers.
To an external customer it appears that they are simply navigating a single website, but internally the organisation is maintaining three entirely separate sites. This approach can work extremely well for web service APIs where the reverse proxy provides a consistent single public facade to an internal distributed component oriented architecture.
Zimbra Proxy Components
Zimbra Proxy is designed to provide a HTTP/POP/IMAP proxy that is quick, reliable, and scalable. Zimbra Proxy includes the following:
- Nginx. A high performance HTTP/IMAP/POP3 proxy server which handles all incoming HTTP/POP/IMAP requests.
- Memcached. A high performance, distributed memory object caching system. Route information is cached for further use in order to increase performance.
- Zimbra Proxy Route Lookup Handler. This is a servlet located on the ZCS mailbox server. This servlet handles queries for the user account route information (the server and port number where the user account resides).
Architecture and Flow
The following sequence shows the architecture and flow of Zimbra Proxy.
- End clients connect to Zimbra Proxy using HTTP/HTTPS/POP/IMAP ports.
- When Zimbra Proxy receives an incoming connection, the Nginx component sends an HTTP request to Zimbra Proxy Route Lookup Handler component.
- Zimbra Proxy Route Lookup Handler locates the route information for the account being accessed and returns this to Nginx.
- The Memcached component stores the route information for the configured period of time (by default, this time is one hour). Nginx will use this route information instead of querying the Zimbra Proxy Route Lookup Handler until the default period of time has expired.
- Nginx uses the route information to connect to Zimbra Mailbox.
- Zimbra Proxy connects to Zimbra Mailbox and initiates the web/mail proxy session. The end client behaves as if it is connecting directly to Zimbra Mailbox.
Position in ZCS Runtime
The figure to the right show the positions of NZ and its relationships to other components of ZCS.
With Zimbra Proxy, a general workflow of login is like this. On a client login, NZ will attempt to contact a memcached server (the exact server will be elected from the available candidates, using an round-robin algorithm). The memcached server is expected to return the upstream route information for that particular client. If the information is not present in that memcached server, then this will be a cache-miss, so Zimbra Proxy will proceed to contact an available NLE (elected by Round-Robin), to look up the upstream server information.
Once the upstream server is known, Zimbra Proxy will immediately initiate the proxy session. And then, it will cache the upstream information into the memcached server. The next time the user logs in, the memcached server will have the upstream information available in its cache, and so Zimbra Proxy will not need to contact NLE.
Deployment strategies, server specifications, impact to other non-proxy hosts
The deployment strategy Zimbra actively suggests is to use Proxy on the edge with mailbox servers behind the firewall. Please note that the Zimbra Proxy package does not act as a firewall and needs to be behind the firewall in customer deployments.
Zimbra Proxy Ports
The following ports are used either by Zimbra Proxy or by Zimbra Mailbox (if Proxy is not configured). If you have any other services running on these ports, turn them off.
End clients connect directly to Zimbra Proxy, using the Zimbra Proxy Ports. Zimbra Proxy connects to the Route Lookup Handler or Zimbra Mailbox using the Zimbra Mailbox Ports.
Zimbra Proxy Ports (External to ZCS) | Port |
---|---|
HTTP | 80 |
HTTPS | 443 |
POP3 | 110 |
POP3S (Secure POP3) | 995 |
IMAP | 143 |
IMAPS (Secure IMAP) | 993 |
Zimbra Mailbox Ports (Internal to ZCS) | Port |
Route Lookup Handler | 7072 |
HTTP Backend (if Proxy configured) | 8080 |
HTTPS Backend (if Proxy configured) | 8443 |
POP3 Backend (if Proxy configured) | 7110 |
POP3S Backend (if Proxy configured) | 7995 |
IMAP Backend (if Proxy configured) | 7143 |
IMAPS Backend (if Proxy configured) | 7993 |
Ldap attributes for the Proxy
Pre 8.5 attributes
zimbraReverseProxyAdminEnabled
zimbraReverseProxyAdminIPAddress
zimbraReverseProxyAdminPortAttribute
zimbraReverseProxyAuthWaitInterval
zimbraReverseProxyAvailableLookupTargets
zimbraReverseProxyCacheEntryTTL
zimbraReverseProxyCacheFetchTimeout
zimbraReverseProxyCacheReconnectInterval
zimbraReverseProxyClientCertCA
zimbraReverseProxyClientCertMode
zimbraReverseProxyConnectTimeout
zimbraReverseProxyDefaultRealm
zimbraReverseProxyDnsLookupInServerEnabled
zimbraReverseProxyDomainNameAttribute
zimbraReverseProxyDomainNameQuery
zimbraReverseProxyDomainNameSearchBase
zimbraReverseProxyErrorHandlerURL
zimbraReverseProxyExternalRouteIncludeOriginalAuthusername
zimbraReverseProxyGenConfigPerVirtualHostname
zimbraReverseProxyHttpEnabled
zimbraReverseProxyHttpPortAttribute
zimbraReverseProxyHttpSSLPortAttribute
zimbraReverseProxyIPLoginLimit
zimbraReverseProxyIPLoginLimitTime
zimbraReverseProxyImapEnabledCapability
zimbraReverseProxyImapExposeVersionOnBanner
zimbraReverseProxyImapPortAttribute
zimbraReverseProxyImapSSLPortAttribute
zimbraReverseProxyImapSaslGssapiEnabled
zimbraReverseProxyImapSaslPlainEnabled
zimbraReverseProxyImapStartTlsMode
zimbraReverseProxyInactivityTimeout
zimbraReverseProxyIpThrottleMsg
zimbraReverseProxyLogLevel
zimbraReverseProxyLookupTarget
zimbraReverseProxyMailEnabled
zimbraReverseProxyMailHostAttribute
zimbraReverseProxyMailHostQuery
zimbraReverseProxyMailHostSearchBase
zimbraReverseProxyMailImapEnabled
zimbraReverseProxyMailImapsEnabled
zimbraReverseProxyMailMode
zimbraReverseProxyMailPop3Enabled
zimbraReverseProxyMailPop3sEnabled
zimbraReverseProxyPassErrors
zimbraReverseProxyPop3EnabledCapability
zimbraReverseProxyPop3ExposeVersionOnBanner
zimbraReverseProxyPop3PortAttribute
zimbraReverseProxyPop3SSLPortAttribute
zimbraReverseProxyPop3SaslGssapiEnabled
zimbraReverseProxyPop3SaslPlainEnabled
zimbraReverseProxyPop3StartTlsMode
zimbraReverseProxyPortQuery
zimbraReverseProxyPortSearchBase
zimbraReverseProxyRouteLookupTimeout
zimbraReverseProxyRouteLookupTimeoutCache
zimbraReverseProxySSLCiphers
zimbraReverseProxySSLToUpstreamEnabled
zimbraReverseProxySendImapId
zimbraReverseProxySendPop3Xoip
zimbraReverseProxyUpstreamConnectTimeout
zimbraReverseProxyUpstreamPollingTimeout
zimbraReverseProxyUpstreamReadTimeout
zimbraReverseProxyUpstreamSendTimeout
zimbraReverseProxyUpstreamServers
zimbraReverseProxyUseExternalRoute
zimbraReverseProxyUseExternalRouteIfAccountNotExist
zimbraReverseProxyUserLoginLimit
zimbraReverseProxyUserLoginLimitTime
zimbraReverseProxyUserNameAttribute
zimbraReverseProxyUserThrottleMsg
zimbraReverseProxyWorkerConnections
zimbraReverseProxyWorkerProcesses
To check the description, type, default value etc for any of the above attributes, simply do 'zmprov desc -a <attr>'
eg. zmprov desc -a zimbraReverseProxyWorkerConnections
zimbraReverseProxyWorkerConnections Maximum number of connections that an NGINX Proxy worker process is allowed to handle type : integer value : callback : immutable : false cardinality : single requiredIn : optionalIn : server,globalConfig flags : serverInherited defaults : 10240 min : 1 max : 40960 id : 725 requiresRestart : nginxproxy since : 5.0.10 deprecatedSince :
New 8.5 attributes
zimbraReverseProxyAcceptMutex
zimbraReverseProxyUpstreamEwsServers
zimbraReverseProxyUpstreamLoginServers
zimbraReverseProxyExactServerVersionCheck
Proxy Configuration and Template files
Zimbra NGINX POP/IMAP/HTTP proxy configuration is generated by the zmproxyconfgen config generation script. This script reads in the proxy configuration template files, and generates the NGINX config files after performing keyword substitution on the template files with values from the LDAP configuration.
zmproxyconfgen is usually never invoked directly -- it is invoked automatically by zmproxyctl
The following sections describe the structure of the NGINX Proxy Configuration
Config Files and Config Templates
To simplify configuration, the NGINX configuration files have been split up into different config files based on functionality. The main, top-level configuration file is /opt/zimbra/conf/nginx.conf, and this file includes the main config, memcache config, mail config, and web config files. The mail config in turn includes the configuration for imap, imaps, pop3 and pop3s. The web config includes the configuration for http and https. The template files follow exactly the same inclusion hierarchy, and each configuration file has a corresponding template file from which it is generated.
Each template file resides in /opt/zimbra/conf/nginx/templates/
Each corresponding config file resides in /opt/zimbra/conf/nginx/includes/ (excluding top-level config file which is /opt/zimbra/conf/nginx/nginx.conf)
Config File Hierarchy
The symbol |_ indicates that a file is included by the one above it. Increasing levels of indentation indicate lower levels of config files
/opt/zimbra/conf/nginx.conf |_ /opt/zimbra/conf/nginx/includes/nginx.conf.main |_ /opt/zimbra/conf/nginx/includes/nginx.conf.memcache |_ /opt/zimbra/conf/nginx/includes/nginx.conf.zmlookup |_ /opt/zimbra/conf/nginx/includes/nginx.conf.mail |_ /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imap.default |_ /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imaps.default |_ /opt/zimbra/conf/nginx/includes/nginx.conf.mail.pop3.default |_ /opt/zimbra/conf/nginx/includes/nginx.conf.mail.pop3s.default |_ /opt/zimbra/conf/nginx/includes/nginx.conf.web |_ /opt/zimbra/conf/nginx/includes/nginx.conf.web.http.default |_ /opt/zimbra/conf/nginx/includes/nginx.conf.web.https.default |_ /opt/zimbra/conf/nginx/includes/nginx.conf.web.admin.default |_ /opt/zimbra/conf/nginx/includes/nginx.conf.web.sso.default
Description of Config Files
* /opt/zimbra/conf/nginx.conf Description: Core NGINX configuration file read by NGINX Proxy Template: /opt/zimbra/conf/nginx/templates/nginx.conf.template Includes: /opt/zimbra/conf/nginx/includes/nginx.conf.main /opt/zimbra/conf/nginx/includes/nginx.conf.memcache /opt/zimbra/conf/nginx/includes/nginx.conf.zmlookup /opt/zimbra/conf/nginx/includes/nginx.conf.mail /opt/zimbra/conf/nginx/includes/nginx.conf.web
* /opt/zimbra/conf/nginx/includes/nginx.conf.main Description: Defines global parameters for all NGINX worker processes Template: /opt/zimbra/conf/nginx/templates/nginx.conf.main.template Includes: None
* /opt/zimbra/conf/nginx/includes/nginx.conf.memcache Description: Defines memcache configuration, common for mail and web Template: /opt/zimbra/conf/nginx/templates/nginx.conf.memcache.template Includes: None
* /opt/zimbra/conf/nginx/includes/nginx.conf.mail Description: Defines the common mail configuration common to IMAP and POP3 Template: /opt/zimbra/conf/nginx/templates/nginx.conf.mail.template Includes: /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imap.default /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imaps.default /opt/zimbra/conf/nginx/includes/nginx.conf.mail.pop3.default /opt/zimbra/conf/nginx/includes/nginx.conf.mail.pop3s.default
* /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imap.default Description: Defines the server block for IMAP Template: /opt/zimbra/conf/nginx/templates/nginx.conf.mail.imap.default.template Includes: None
* /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imaps.default Description: Defines the server block for IMAPS Template: /opt/zimbra/conf/nginx/templates/nginx.conf.mail.imaps.default.template Includes: None
* /opt/zimbra/conf/nginx/includes/nginx.conf.mail.pop3.default Description: Defines the server block for POP3 Template: /opt/zimbra/conf/nginx/templates/nginx.conf.mail.pop3.default.template Includes: None
* /opt/zimbra/conf/nginx/includes/nginx.conf.mail.pop3s.default Description: Defines the server block for POP3S Template: /opt/zimbra/conf/nginx/templates/nginx.conf.mail.pop3s.default.template Includes: None
* /opt/zimbra/conf/nginx/includes/nginx.conf.web Description: Defines the common web configuration common to HTTP and HTTPS Template: /opt/zimbra/conf/nginx/templates/nginx.conf.web.template Includes: /opt/zimbra/conf/nginx/includes/nginx.conf.web.http.default /opt/zimbra/conf/nginx/includes/nginx.conf.web.https.default /opt/zimbra/conf/nginx/includes/nginx.conf.web.sso.default /opt/zimbra/conf/nginx/includes/nginx.conf.web.admin.default
* /opt/zimbra/conf/nginx/includes/nginx.conf.web.http.default Description: Defines the server block for HTTP Template: /opt/zimbra/conf/nginx/templates/nginx.conf.web.http.default.template Includes: None
* /opt/zimbra/conf/nginx/includes/nginx.conf.web.https.default Description: Defines the server block for HTTPS Template: /opt/zimbra/conf/nginx/templates/nginx.conf.web.https.default.template Includes: None
* /opt/zimbra/conf/nginx/includes/nginx.conf.web.sso.default Description: Defines the server block for Single Sign on (SSO) Template: /opt/zimbra/conf/nginx/templates/nginx.conf.web.sso.default.template Includes: None
* /opt/zimbra/conf/nginx/includes/nginx.conf.web.admin.default Description: Defines the server block for admin user Template: /opt/zimbra/conf/nginx/templates/nginx.conf.web.admin.default.template Includes: None
Proxy config rewrite(zmproxyconfgen), zmproxyconfig and zmproxyctl
Proxy config generation script 'zmproxyconfgen' is usually never invoked directly -- it is invoked automatically by zmproxyctl. So 'zmproxyctl restart' would essentially stop the proxy, rewrite the proxy config files and start the proxy again.
To change multiple LDAP attributes related to the proxy at once (i.e any of the zimbraReverseProxy* attrs) instead of using zmprov command to change each attribute individually, /opt/zimbra/libexec/zmproxyconfig can be used. The usage for this script is as follows
zimbra@zdev-vmxyz:~$/opt/zimbra/libexec/zmproxyconfig help
Usage: libexec/zmproxyconfig [-h] [-o] [-m] [-w] [-d [-r] [-s] [-a w1:w2:w3:w4] [-c [-n n1:n2]] [-i p1:p2:p3:p4] [-p p1:p2:p3:p4] [-x mailmode]] [-e [-a w1:w2:w3:w4] -C] [-n n1:n2 [-i p1:p2:p3:p4] [-p p1:p2:p3:p4] [-u|-U] [-x mailmode]] [-f] -H hostname
-h: display this help message
-H: Hostname of server on which enable/disable proxy functionality.
-a: Colon separated list of Web ports to use. Format: HTTP-STORE:HTTP-PROXY:HTTPS-STORE:HTTPS-PROXY (Ex: 8080:80:8443:443)
-d: disable proxy
-e: enable proxy
-f: Full reset on memcached port and search queries and POP/IMAP throttling.
-i: Colon separated list of IMAP ports to use. Format: IMAP-STORE:IMAP-PROXY:IMAPS-STORE:IMAPS-PROXY (Ex: 7143:143:7993:993)
-m: Toggle mail proxy portions
-o: Override enabled checks
-p: Colon separated list of POP ports to use. Format: POP-STORE:POP-PROXY:POPS-STORE:POPS-PROXY (Ex: 7110:110:7995:995)
-r: Run against a remote host. Note that this requires the server to be properly configured in the LDAP master.
-s: Set cleartext to FALSE (secure mode) on disable
-t: Disable reverse proxy lookup target for store server. Only valid with -d. Be sure that you intend for all proxy function for the server to be disabled
-w: Toggle Web proxy portions
-c: Disable Admin Console proxy portions.
-C: Enable Admin Console proxy portions.
-n: Colon separated list of Admin Console ports to use. Format: ADMIN-CONSOLE-STORE:ADMIN-CONSOLE-PROXY (Ex: 7071:9071)
-x: the proxy mail mode when enable proxy, or the store mail mode when disable proxy (Both default: http).
-u: disable SSL connection from proxy to mail store.
-U: enable SSL connection from proxy to mail store.
hostname is the value of the zimbra_server_hostname LC key for the server being modified.
Required options are -f by itself, or -f with -d or -e.
Note that -d or -e require one or both of -m and -w.
Note that -i or -p require -m.
Note that -a requires -w.
Note that -c/-C requires -w, and -n requires -c/-C. When disabling web proxy, admin console proxy will be automatically disabled.
Note that -u or -U are only available when proxy is enabled by -e.
Note that -x requires -w and -d for store.
Note that -x requires -w for proxy.
Note that no matter what mail mode is set by -x and no matter proxy is enabled or disabled, admin console's mode is always https.
The following are the defaults for -a, -i, -p, and -x if they are not supplied as options.
-a default on enable: 8080:80:8443:443
-a default on disable: 80:0:443:0
-i default on enable: 7143:143:7993:993
-i default on disable: 143:7143:993:7993
-p default on enable: 7110:110:7995:995
-p default on disable: 110:7110:995:7995
-n default on enable: 7071:9071
-n default on disable: 7071:9071
-x default on store disable: http
-x default on proxy enable/disable: http, but -x default on proxy enable when upstream ssl connection is enabled: https
eg. To change the value of zimbraReverseProxySSLToUpstreamEnabled from TRUE(default) to FALSE (i.e use http for the proxy<->mailstore connections instead of https), execute this on the server running the proxy
/opt/zimbra/libexec/zmproxyconfig -e -m -w -H <hostname> -u
You will have to restart the proxy after this by running 'zmproxyctl restart'
- Note: zmproxyconfig is no longer mandatory to enable the proxy. The installer already runs this and the proxy is enabled for https access by default
Troubleshooting
Here are some basic troubleshooting tips for proxy related errors found in /opt/zimbra/log/nginx.log & /opt/zimbra/log/nginx.access.log
No route to host errors
If your getting "No route to host" errors in /opt/zimbra/log/nginx.log files on the proxy servers, you should check: The resolution [DNS] to the host from the proxy servers is correct and working. This might happen more when you've deployed a new ZCS server in your environment. Following step one, you also have to confirm there is port level access to between proxy servers and the server it's trying to reach. memcache port is 11211 . The server/s are too busy to serve the request. You have a server in the list that shouldn't be in the lookup pool or you should remove the trouble server from the pool to avoid any more customer issues to deescalate the situation.
5xx Errors
1. 500 Internal Server Error: A generic error message, given when an unexpected condition was encountered and no more specific message is suitable. This might be due to several reasons due to failure on one of the upstream mailstore servers.
2. 502 Bad Gateway: The server was acting as a gateway or proxy and received an invalid response from the upstream server.
3. 503 Service Unavailable: The most common reason for this is that the jetty mailboxd process is down on the mailstore server and hence is unable to process the request. Check using 'netstat -an | grep <port>' to see if the IMAP/POP/HTTP ports used by mailboxd are up and listening for incoming connections. Also check using 'ps -eaf | grep mailboxd'
4. 504 Gateway Timeout: The most common reason for this is that the upstream mailstore server took longer than the configured timeout value and hence the client closed the upstream connection. Try changing the timeout values to a higher value if its expected to take longer for specific kind of requests. If this is not the case, then we should look for reasons why it took longer for a request to get processed.
These proxy configuration changes can improve proxy behavior related to timeouts:
- This will configure proxy to immediately reconnect on any failure
$ zmprov mcf zimbraMailProxyReconnectTimeout 0
- If necessary, on each proxy
$ zmprov ms `zmhostname` zimbraMailProxyReconnectTimeout 0
- This will configure proxy to ignore failures in regards to disconnects
$ zmprov mcf zimbraMailProxyMaxFails 0
- If necessary, on each proxy
$ zmprov ms `zmhostname` zimbraMailProxyMaxFails 0
- Then, restart the processes on all proxies:
$ zmproxyctl restart
See the following bug for more details on this recommendation:
- "Improved proxy timeout defaults"
Zimbra Proxy Command-Line Utilities
The following commands are zmprov commands that are specific to Zimbra Proxy. For more information about using zmprov, refer to the Appendix A: Command-Line Utilities in the ZCS Administrator’s Guide, located on the Zimbra Website.
Syntax
zmprov [cmd]
Description
Long Name | Short Name | Description |
---|---|---|
--getAllReverseProxyURLs | -garpu | Used to list all the upstream mailstore servers (NLEs) that should be used for reverse proxy lookup by the proxy |
--getAllReverseProxyBackends | -garpb | Used to list all the upstream mailstore servers that are reverse-proxied by the proxy |
--getAllMtaAuthURLs | -gamau | Used to publish into saslauthd.conf the servers that should be used for saslauthd.conf MTA auth |
--getAllMemcachedServers | -gamcs | Used to list memcached servers (for Zimbra Proxy use) |