Ajcody-Proxy-Guide-Rewrite-Project

Overview And Planning For Zimbra Proxy

Source: http://wiki.zimbra.com/wiki/Zimbra_Proxy_Guide

What Is Zimbra Proxy

Source:

* Needs to be proofed.

* Make references to pop,imap,http as IMAP[S]/POP[S]/HTTP[S]

Zimbra Proxy is a high-performance reverse proxy service for passing IMAP[S]/POP[S]/HTTP[S] client requests to other internal ZCS services. A reverse proxy server is an Internet-facing services that protects and manages client connections to your internal services. It can also provide functions like: GSSAPI authentication, throttle control, SSL connection with different certificates for different virtual host names, and other features described later below. When discussing reverse proxy topics, you'll see references to downstream and upstream. Downstream is in reference to the end-user client and the software/computer they are using. Upstream is in reference to the actually server or service that actually handles the request - often times this is the Zimbra mailstore or webapp server the users account resides on.


The Zimbra Proxy services allows other zimbra services [mailbox, webapp, etc.] that can be hidden from the Internet by acting as a reverse proxy for the other services and allows end-users to access the mail system via a single Login URL [for example, userA and userB can both use mail.domain.com] instead of knowing their specific mailbox hostnames [for example, userA uses mail12.domain.com and userB is uses mail13.domain.com]. It acts as the first entry point for all the HTTP/IMAP/POP traffic and then intelligently routes all static UI (HTML/CSS/JS etc) and dynamic (SOAP/REST/IMAP/POP) requests to the appropriate upstream server. The proxy configuration options for POP/IMAP allows end users to configure their POP/IMAP clients to use this single mail server hostname. The proxy configuration options around HTTP[S] allows end-users to use this single hostname for ZWC, REST, CalDAV, Zimbra Connector for Outlook, Zimbra Connector for BES, Zimbra Mobile Sync connections and so forth. The Zimbra Proxy service will also do URL rewriting so these internal hostnames that are providing the various services aren't exposed to the public or end-clients. For example, a Zimbra briefcase share from userA on mail12.domain.com to userB on mail13.domain.com will simply have mail.domain.com [per our example above] within the url. [ see bug 82236 ]


With ZCS 8.5, the proxy components are installed by default and therefor would be enabled on a single ZCS server deployment. Normally though, the proxy services would be installed on Internet-facing servers and the other specific ZCS services [zimbra-store for example] would be on installed on internal servers. Generally, these packages are installed on dedicated Internet-facing servers intended to just do the proxy services or those that also run the zimbra-mta package. When the Zimbra Proxy package is installed, the proxy feature is enabled with default values that normally require no modification.


The Zimbra Proxy installation components consists of the zimbra-proxy and zimbra-memcached options during the installation package choice menu. The zimbra-proxy packages is Zimbra’s modified version of Nginx [1], pronounced like “engine-ex”. And the zimbra-memcached packages is our modified version of memcached [2], pronounced like “memcache-dee”. A third component needed for our proxy environment to work is the Zimbra Proxy Route Lookup Handler or called the “Nginx Lookup Extension” (NLE for short) if some other references. This is a java servlet that is installed from the zimbra-store package on the Zimbra mailbox servers. This servlet handles nginx queries for the user account route information (the server and port number where the user account resides).


In a typical use case, the proxy services extracts the user login details and then fetches the route to the upstream mail server or web servers’ address from the Zimbra Proxy Route Lookup Handler [the NLE servlet], and finally proxies the interactions between clients and upstream ZCS servers. To accelerate the speed of future route lookups for a user, memcached caches the lookup results. Therefore, the subsequent login with the same username will directly be proxied without calling to the Zimbra Proxy Route Lookup Handler.

Benefits And Reasons To Use

Source http://wiki.zimbra.com/wiki/Zimbra_Proxy_Guide#Benefits_and_reasons_to_use

* Needs to be proofed.

* Make references to pop,imap,http as IMAP[S]/POP[S]/HTTP[S]

  1. Zimbra Proxy centralizes access to Mailbox servers
    • Zimbra Proxy allows mailbox servers to be hidden from public internet by acting as a reverse proxy & also allowing end users to access mail system via single Login URL instead of knowing their mailbox hostnames. It acts as the first entry point for all the HTTP/IMAP/POP traffic and then intelligently routes all kind of static UI requests (HTML/CSS/JS etc) and Dynamic requests (SOAP/REST/IMAP/POP) to the appropriate upstream server. Here are some other benefits of using the proxy as well
  2. Load Balancing
    • This is the reverse proxy function that people are most familiar with. Here the proxy routes incoming HTTP requests to a number of identical mail servers. The upstream mail server selection can be based on a simple client IP hash or round-robin algorithm. It’s such a common function that load balancing reverse proxies are usually just referred to as ‘load balancers’. There are specialized load balancing products available, but many general purpose reverse proxies also provide load balancing functionality.
  3. Security
    • A reverse proxy can hide the topology and characteristics of your back-end servers by removing the need for direct internet access to them. You can place your reverse proxy in an internet facing DMZ, but hide your web servers inside a non-public subnet.
  4. Authentication
    • You can use your reverse proxy to provide a single point of authentication for all HTTP requests. Although in case of ZCS, the authentication is provided by upstream mailstores where the user accounts reside.
  5. SSL Termination
    • Here the reverse proxy handles incoming HTTPS connections, decrypting the requests and passing unencrypted requests on to the web servers.
  6. Caching
    • Currently we use the memcached module with proxy to achieve caching of upstream routes to mailstores on a per end-client basis. This significantly reduces the route lookup time thereby improving the total time required to process the request and boost performance.
  7. Centralized Logging and Auditing
    • Because all HTTP requests are routed through the reverse proxy, it makes an excellent point for logging and auditing.
  8. URL Rewriting
    • Sometimes the URL scheme that a legacy application presents is not ideal for discovery or search engine optimization. A reverse proxy can rewrite URLs before passing them on to your back-end servers. To an external customer it appears that they are simply navigating a single website, but internally the organisation is maintaining three entirely separate sites. This approach can work extremely well for web service APIs where the reverse proxy provides a consistent single public facade to an internal distributed component oriented architecture.

Zimbra Proxy Components

Source: http://wiki.zimbra.com/wiki/Zimbra_Proxy_Guide#Zimbra_Proxy_Components

* Needs to be proofed.

* Make references to pop,imap,http as IMAP[S]/POP[S]/HTTP[S].

* Use style guide for hostname references.

Zimbra Proxy is designed to provide a HTTP/POP/IMAP proxy that is quick, reliable, and scalable. Zimbra Proxy includes the following:

  • Nginx. A high performance HTTP/IMAP/POP3 proxy server which handles all incoming HTTP/POP/IMAP requests.
  • Memcached. A high performance, distributed memory object caching system. Route information is cached for further use in order to increase performance.
  • Zimbra Proxy Route Lookup Handler. This is a servlet, Nginx Lookup Extension (NLE), located on the ZCS mailbox server - accessed by http://<host>:7072/service/extension/nginx-lookup . This servlet handles queries for the user account route information (the server and port number where the user account resides).

Architecture and Flow

Source: http://wiki.zimbra.com/wiki/Zimbra_Proxy_Guide#Architecture_and_Flow

* Needs to be proofed.

* Make references to pop,imap,http as IMAP[S]/POP[S]/HTTP[S].

* Merge the two sources of information into one.

* Reference customer dependencies - Firewall, load-balancer , round-robin DNS to make single-zimbra-hostname use all the proxy servers in question either by ip or dns .

The following sequence shows the architecture and flow of Zimbra Proxy.

  1. End clients connect to Zimbra Proxy using HTTP/HTTPS/POP/IMAP ports.
  2. When Zimbra Proxy receives an incoming connection, the Nginx component sends an HTTP request to Zimbra Proxy Route Lookup Handler component.
  3. Zimbra Proxy Route Lookup Handler locates the route information for the account being accessed and returns this to Nginx.
  4. The Memcached component stores the route information for the configured period of time (by default, this time is one hour). Nginx will use this route information instead of querying the Zimbra Proxy Route Lookup Handler until the default period of time has expired.
  5. Nginx uses the route information to connect to Zimbra Mailbox.
  6. Zimbra Proxy connects to Zimbra Mailbox and initiates the web/mail proxy session. The end client behaves as if it is connecting directly to Zimbra Mailbox.

Source : Admin Guide Draft , 'Configure Zimbra HTTP Proxy'

HTTP reverse proxy routes requests as follows:

  1. If the requesting URL can be examined to determine the user name, then the request is routed to the backend mailbox server of the user in the URL. REST, Ca lDAV, and Zimbra Mobile Sync are supported through this mechanism.
  2. If the request has an auth token cookie (ZM_AUTH_TOKEN), the request is routed to the backend mailbox server of the authenticated user.
  3. If the above methods do not work, the IP hash method is used to load balance the requests across the backend mailbox servers which are able to handle the request or do any necessary internal proxying.

Position In ZCS Runtime

Source: http://wiki.zimbra.com/wiki/Zimbra_Proxy_Guide#Position_in_ZCS_Runtime

* Needs to be proofed.

* Get graphic working.

* Reference customer dependencies - Firewall, load-balancer , round-robin DNS to make single-zimbra-hostname use all the proxy servers in question either by ip or dns . Expand graphic to include this reference.

The figure to the right show the positions of NZ and its relationships to other components of ZCS.

Proxy position in ZCS runtime. Click to see full sized : Proxy_position_in_ZCS-2.png

With Zimbra Proxy, a general workflow of login is like this. On a client login, NZ will attempt to contact a memcached server (the exact server will be elected from the available candidates, using an round-robin algorithm). The memcached server is expected to return the upstream route information for that particular client. If the information is not present in that memcached server, then this will be a cache-miss, so Zimbra Proxy will proceed to contact an available NLE (elected by Round-Robin), to look up the upstream server information.


Once the upstream server is known, Zimbra Proxy will immediately initiate the proxy session. And then, it will cache the upstream information into the memcached server. The next time the user logs in, the memcached server will have the upstream information available in its cache, and so Zimbra Proxy will not need to contact NLE.

Zimbra Proxy Ports

* Need reference to memcached port , admin ports , zimbraPublicServicePort .

* Do we clarify "open" ports on the firewall also? --Adam

Source: http://wiki.zimbra.com/wiki/Zimbra_Proxy_Guide#Zimbra_Proxy_Ports

The following ports are used either by Zimbra Proxy or by Zimbra Mailbox (if Proxy is not configured). If you have any other services running on these ports, turn them off.

End clients connect directly to Zimbra Proxy, using the Zimbra Proxy Ports. Zimbra Proxy connects to the Route Lookup Handler or Zimbra Mailbox using the Zimbra Mailbox Ports.


Zimbra Proxy Ports (External to ZCS) Port
HTTP 80
HTTPS 443
POP3 110
POP3S (Secure POP3) 995
IMAP 143
IMAPS (Secure IMAP) 993
Zimbra Mailbox Ports (Internal to ZCS) Port
Route Lookup Handler 7072
HTTP Backend (if Proxy configured) 8080
HTTPS Backend (if Proxy configured) 8443
POP3 Backend (if Proxy configured) 7110
POP3S Backend (if Proxy configured) 7995
IMAP Backend (if Proxy configured) 7143
IMAPS Backend (if Proxy configured) 7993

Deployment Strategies, Server Specifications, Impact To Other Non-Proxy Hosts

Source: http://wiki.zimbra.com/wiki/Zimbra_Proxy_Guide#Deployment_strategies.2C_server_specifications.2C_impact_to_other_non-proxy_hosts

* Needs more details, incomplete right now.


The deployment strategy Zimbra actively suggests is to use Zimbra Proxy on the edge with mailbox servers behind the firewall. Please note that the Zimbra Proxy package does not act as a firewall and needs to be behind the firewall in customer deployments.

Installing , Configuring, Disabling the Zimbra Proxy

Things To Review First

Prerequisite Variables To Check First

zimbraPublicServiceHostname zimbraPublicServiceProtocol and zimbraPublicServicePort

* Needs more details, incomplete right now.

In order for the change password link, calendar launching in separate window, and other various functionality to work correctly - meaning, to use the proxy instead of mailbox server, the following LDAP attributes have to be set to the proxy values:

  • zimbraPublicServiceHostname(Name to be used in public API such as REST or SOAP proxy) - proxy hostname
  • zimbraPublicServiceProtocol(Protocol to be used in public API such as REST or SOAP proxy) - proxy protocol (http or https)
  • zimbraPublicServicePort(Port to be used in public API such as REST or SOAP proxy) - proxy port

zimbraVirtualHostname

* Needs more details, incomplete right now.

zimbra_auth_always_send_refer

The zmlocalconfig key zimbra_auth_always_send_refer is now obsolete. Its been replaced by LDAP attribute zimbraMailReferMode. Now with a full-fledged reverse proxy, users do not need to be redirected. The LDAP attribute zimbraMailReferMode is used directly by the Nginx reverse proxy.

zmtlsctl

* Needs more details, incomplete right now.

zmtlsctl sets the zimbraMailMode , this is different than the zimbraReverseProxyMailMode .

zmtlsctl help
Usage: /opt/zimbra/bin/zmtlsctl [mixed|both|http|https|redirect]
$ zmprov desc -a zimbraMailMode                
zimbraMailMode
    whether to run HTTP or HTTPS or both/mixed mode or redirect mode. See
    also related attributes zimbraMailPort and zimbraMailSSLPort

               type : enum
              value : http,https,both,mixed,redirect
           callback : LocalBind
          immutable : false
        cardinality : single
         requiredIn : 
         optionalIn : globalConfig,server
              flags : serverInherited
           defaults : 
                min : 
                max : 
                 id : 308
    requiresRestart : 
              since : 
    deprecatedSince : 
$ zmprov desc -a zimbraReverseProxyMailMode
zimbraReverseProxyMailMode
    whether to run proxy in HTTP, HTTPS, both, mixed, or redirect mode.
    See also related attributes zimbraMailProxyPort and
    zimbraMailSSLProxyPort

               type : enum
              value : http,https,both,mixed,redirect
           callback : 
          immutable : false
        cardinality : single
         requiredIn : 
         optionalIn : globalConfig,server
              flags : serverInherited
           defaults : 
                min : 
                max : 
                 id : 685
    requiresRestart : nginxproxy
              since : 5.0.7
    deprecatedSince : 

From a single ZCS 8.5 server install:

$ zmprov gs `zmhostname` | grep MailMode
zimbraMailMode: https
zimbraReverseProxyMailMode: https

The New WebApp Services In ZCS 8.5 and zm_auth_token

Source: From admin guide draft under 'Configur Zimbra HTTP Proxy'

* Needs more details, incomplete right now.

Note - <<need to add split node feature info in this section. Affects zm_auth_token>> [From Admin Guide Draft]

New ZCS Deployment

Single ZCS Server Environment

* Needs more details, incomplete right now.


Multi-Server ZCS Environment

* Needs more details, incomplete right now.

Adding Zimbra Proxy Services To Existing Non-Proxy Environments via ZCS Installer [Recommended Method]

Using New Servers

Source: http://wiki.zimbra.com/wiki/Enabling_Zimbra_Proxy#Using_new_servers

Here you are installing the proxy on a brand new server and having all your existing mailbox servers being accessed through the proxy on this new server. Simply use the installer script (install.sh) and select the proxy and memcached packages ('Y' by default with ZCS 8.5+, just need to hit enter). This will ask you for LDAP hostname/password, Bind password for nginx ldap user which you need to provide (do 'zmlocalconfig -s ldap_nginx_password' on the host running ldap to get this) and then the Zimbra Proxy configuration menu would be displayed which would look like this.

Proxy configuration

  1) Status:                                  Enabled                       
  2) Enable POP/IMAP Proxy:                   TRUE                          
  3) IMAP server port:                        7143                          
  4) IMAP server SSL port:                    7993                          
  5) IMAP proxy port:                         143                           
  6) IMAP SSL proxy port:                     993                           
  7) POP server port:                         7110                          
  8) POP server SSL port:                     7995                          
  9) POP proxy port:                          110                           
 10) POP SSL proxy port:                      995                           
 11) Bind password for nginx ldap user:       set                           
 12) Enable HTTP[S] Proxy:                    TRUE                          
 13) Web server HTTP port:                    8080                          
 14) Web server HTTPS port:                   8443                          
 15) HTTP proxy port:                         80                            
 16) HTTPS proxy port:                        443                           
 17) Proxy server mode:                       https          

If you need to change any of these intentionally, you can do that now by selecting the corresponding config item from the menu (say for eg. to disable POP/IMAP proxy, select '2' from the above menu). Otherwise, just proceed with all the defaults and you would have the proxy+memcached installed on this new server. Now, to have all the mailbox servers use the proxy, simply set the zimbraMailReferMode to reverse-proxied on each mailbox server and restart mailboxd to have all the traffic go through the proxy.

Using Existing Servers

* Needs more details, incomplete right now.

* /opt/zimbra/libexec/zmsetup.pl vs ./install.sh --Adam

Adding Zimbra Proxy Services To Existing Non-Proxy Environments via CLI [Advanced Method]

Using New Servers

Using Existing Servers

Source: http://wiki.zimbra.com/wiki/Enabling_Zimbra_Proxy#Using_existing_servers

Assuming you are running a 8.0 or earlier version ZCS with no proxy/memcached, zimbraMailMode as https and now want to upgrade to 8.5+ along with adding proxy & memcached, you need to follow the following steps

Start 8.5+ installer (install.sh script)

Note - Recommend /opt/zimbra/libexec/zmsetup.pl vs ./install.sh [Adam Cody]

Do you wish to upgrade? [Y] y

Install zimbra-memcached [N] y

Install zimbra-proxy [N] y

After install is done, enable web/mail proxy, and set the proxy mode and ports:

  • If localconfig key 'zimbra_require_interprocess_security' is set, Only "https" and "both" are valid modes
/opt/zimbra/libexec/zmproxyconfig -e -w -o -a 8080:80:8443:443 -x <https/both>  -H `zmhostname`
  • Else if 'zimbra_require_interprocess_security' is unset, Only "http" and "both" are valid modes
/opt/zimbra/libexec/zmproxyconfig -e -w -o -a 8080:80:8443:443 -x <http/both>  -H `zmhostname`
  • Set the mail proxy ports
/opt/zimbra/libexec/zmproxyconfig -e -m -o -i 7143:143:7993:993 -p 7110:110:7995:995 -H `zmhostname`

Now, to have all the mailbox servers use the proxy, simply set the zimbraMailReferMode to reverse-proxied on each mailbox server and restart mailboxd to have all the traffic go through the proxy. Do a 'zmcontrol restart' on this node and you should be up and running.

Manually Modifying Zimbra Proxy Services And Related Variables via CLI

Source: http://wiki.zimbra.com/wiki/Enabling_Zimbra_Proxy#Manually_Modifying_Proxy_.26_related_Variables_via_CLI

Simple Command With Defaults

Source: http://wiki.zimbra.com/wiki/Enabling_Zimbra_Proxy#Simple_Command_With_Defaults

The zmproxyconfig command can be run with limited arguments if the command defaults are acceptable. Run /opt/zimbra/libexec/zmproxyconfig to view all the argument options and the usage

Protocol Requirements Including HTTPS Redirect

Source: http://wiki.zimbra.com/wiki/Enabling_Zimbra_Proxy#Protocol_Requirements_Including_HTTPS_Redirect

HTTP proxy can support protocol modes for HTTP or HTTPS only, both HTTP and HTTPS, mixed HTTP and HTTPS or HTTPS redirect from HTTP. Redirect is a popular configuration. This configuration must be made to the proxy servers.

  • HTTPS redirect from HTTP
zmprov ms proxy.server.name zimbraReverseProxyMailMode redirect
  • HTTP and HTTPS (support both)
zmprov ms proxy.server.name zimbraReverseProxyMailMode both
  • HTTPS only
zmprov ms proxy.server.name zimbraReverseProxyMailMode https
  • HTTP only
zmprov ms proxy.server.name zimbraReverseProxyMailMode http
  • "mixed" will cause only authentication to be sent over HTTPS
zmprov ms proxy.server.name zimbraReverseProxyMailMode mixed


Documents & Sharing - The zimbraPublicService variables

Source: http://wiki.zimbra.com/wiki/Enabling_Zimbra_Proxy#Documents_.26_Sharing

It is important to consider access to documents (Briefcase) and shares when setting up HTTP proxy. A publicly reachable address must be configured to be used for the REST and SOAP proxy interfaces otherwise components requiring access to these interfaces will fail. Calendar sharing is an example of one component. Set zimbraPublicServiceHostname, zimbraPublicServiceProtocol, and zimbraPublicServicePort when applicable. These values are usually not required without proxy since the REST and SOAP proxy interfaces take the value of the Zimbra mailbox service hostname by default. These attributes can be set globally to be inherited by all domains or per domain.

Set zimbraPublicServiceHostname to the value of the host that will be used in the URL for access to the HTTP proxy.

  • This command sets mail.domain.com as the public hostname to be used for access to all domains in the Zimbra directory:
zmprov mcf zimbraPublicServiceHostname mail.domain.com
  • This command sets mail.domaina.com as the public hostname to be used for access to domaina.com domain:
zmprov md domaina.com zimbraPublicServiceHostname mail.domaina.com
  • Set zimbraPublicServiceProtocol to http or https depending on the protocol requirements for HTTP proxy:
zmprov md domaina.com zimbraPublicServiceProtocol https
  • Set zimbraPublicServicePort to the value that corresponds to the HTTP proxy port used in the URL (optional if standard ports 80 or 443 are used for proxy listeners):
zmprov md domaina.com zimbraPublicServicePort 443

Disabling Zimbra Proxy

Completely Disable Proxy In Single ZCS Server Environment

Note - Recommend /opt/zimbra/libexec/zmsetup.pl as default method to do this? [Adam Cody]

Completely Disable Proxy In Multi-Server ZCS Environment

Note - Recommend /opt/zimbra/libexec/zmsetup.pl as default method to do this? [Adam Cody]

Disable POP/IMAP Proxy In Single ZCS Server Environment

Source: http://wiki.zimbra.com/wiki/Ajcody-Proxy-Notes#Need_To_Disable_Pop.2FImap_Proxy_And_Use_POP.2FIMAP_Normally


Note - Recommend /opt/zimbra/libexec/zmsetup.pl as default method to do this instead ? [Adam Cody]


Sometimes, people install/setup proxy services on their single ZCS server and they don't need them. Here's how you would disable the proxy stuff and get imap/pop working over the default ports.

do a zmprov -l gs `zmhostname` | grep -i port

get the ports, then set variables to port 0:

zmprov ms `zmhostname` zimbraImapProxyBindPort 0
zmprov ms `zmhostname` zimbraImapSSLProxyBindPort 0
zmprov ms `zmhostname` zimbraPop3ProxyBindPort 0
zmprov ms `zmhostname` zimbraPop3SSLProxyBindPort 0

then, set the non "Proxy" ports to the desired standard ports

zmprov ms `zmhostname` zimbraImapBindPort 143
zmprov ms `zmhostname` zimbraImapSSLBindPort 993
zmprov ms `zmhostname` zimbraPop3BindPort 110
zmprov ms `zmhostname` zimbraPop3SSLBindPort 995

once complete:

zmprov ms `zmhostname` -zimbraServiceEnabled memcached
zmprov ms `zmhostname` -zimbraServiceEnabled imapproxy

zmproxyctl stop
zmmemcachedctl stop
zmmailboxdctl stop
zmmailboxdctl start 

Troubleshooting

Source: http://wiki.zimbra.com/wiki/Zimbra_Proxy_Guide#Troubleshooting

Proxy Related Log Files

Source: http://wiki.zimbra.com/wiki/Zimbra_Proxy_Guide#Troubleshooting

Proxy related errors can be found in the following logs on each proxy server:

  • /opt/zimbra/log/nginx.log
  • /opt/zimbra/log/nginx.access.log

Slow Proxy Logins - Route Lookup Handles (garpu)

Source:

A common nginx misconfiguration is to have incorrectly designated non-mailbox servers as routing/zmlookup handlers. Only mailbox servers can perform route handler functions. To view the zmlookup lookup handlers, review the zm_lookup_handlers parameter in /opt/zimbra/conf/nginx/includes/nginx.conf.zmlookup

grep zm_lookup_handlers /opt/zimbra/conf/nginx/includes/nginx.conf.zmlookup

If a non-mailbox server is listed, set the zimbraReverseProxyLookupTarget server configuration attribute to FALSE for that server.

zmprov ms `zmhostname` zimbraReverseProxyLookupTarget FALSE

Additionally, zimbraReverseProxyLookupTarget is a server inherited attribute from the global configuration, so check if zimbraReverseProxyLookupTarget has been incorrectly designated in global config.

zmprov gcf zimbraReverseProxyLookupTarget

No Route To Host Errors

Source:

If your getting "No route to host" errors in /opt/zimbra/log/nginx.log files on the proxy servers, you should check:

  1. If the resolution [DNS] to the host from the proxy servers is correct and working. This oversight might when you've deployed a new ZCS server in your environment.
  2. Following step one, you also have to confirm there is port level access between proxy servers and the server it's trying to reach. memcache port is 11211 .
  3. The server/s [mailbox/webapp servers] are too busy to serve the request.
  4. You have a server in the lookup pool list that shouldn't be there or you should remove the trouble server from the pool to avoid any more customer issues to deescalate the situation.

5xx Errors

Source:

  • 500 Internal Server Error:
    • A generic error message, given when an unexpected condition was encountered and no more specific message is suitable. This might be due to several reasons due to failure on one of the upstream mailstore servers.
  • 502 Bad Gateway:
    • The server was acting as a gateway or proxy and received an invalid response from the upstream server.
  • 503 Service Unavailable:
    • The most common reason for this is that the jetty mailboxd process is down on the mailstore server and hence is unable to process the request. Check using 'netstat -an | grep <port>' to see if the IMAP/POP/HTTP ports used by mailboxd are up and listening for incoming connections. Also check using 'ps -eaf | grep mailboxd'
  • 504 Gateway Timeout:
    • The most common reason for this is that the upstream mailstore server took longer than the configured timeout value and hence the client closed the upstream connection. Try changing the timeout values to a higher value if its expected to take longer for specific kind of requests. If this is not the case, then we should look for reasons why it took longer for a request to get processed.

These proxy configuration changes can improve proxy behavior related to timeouts:

  • This will configure proxy to immediately reconnect on any failure
    • $ zmprov mcf zimbraMailProxyReconnectTimeout 0 
    • If necessary, on each proxy
      • $ zmprov ms `zmhostname` zimbraMailProxyReconnectTimeout 0 
  • This will configure proxy to ignore failures in regards to disconnects
    • $ zmprov mcf zimbraMailProxyMaxFails 0 
    • If necessary, on each proxy
      • $ zmprov ms `zmhostname` zimbraMailProxyMaxFails 0 
  • Then, restart the processes on all proxies:
  •  $ zmproxyctl restart 

See the following bug for more details on this recommendation:

Third Party Issues - Load Balancers, DNS, Etc.

Bad/Invalid command when proxying to external POP/IMAP servers

Source : http://wiki.zimbra.com/index.php?title=NGINX_Configuration_Tips#Bad.2FInvalid_command_when_proxying_to_external_POP.2FIMAP_servers


Nginx issues the XOIP command to the upstream POP3 server, and the ID command to the upstream IMAP server, before logging in to upstream. This is for auditing purposes so that the client's IP address is known to the upstream server. The global config attributes zimbraReverseProxySendPop3Xoip and zimbraReverseProxySendImapId control this aspect.

However, some external IMAP servers may not implement the ID command, and some external POP3 servers may not implement the XOIP command.

To turn off sending the XOIP command, set zimbraReverseProxySendPop3Xoip to false. To turn off sending the IMAP command, set zimbraReverseProxySendImapId to false.

zmprov mcf zimbraReverseProxySendPop3Xoip FALSE
zmprov mcf zimbraReverseProxySendImapId FALSE

LDAP/Nginx Won't Start And Asks For A Password

Source: http://wiki.zimbra.com/wiki/LDAP/Nginx_won%27t_start_and_asks_for_a_password

This tends to happen because Zimbra can't read the key or the certificate of the LDAP/IMAP proxy service on startup. This issue is specific to commercial certificates.

Here is what you can do:

  • You want to check to make sure the private key is not encrypted.
  • Manually link the cert to the services OR simply redeploy the certificate to relink the cert to the services.
cp /opt/zimbra/ssl/zimbra/commercial/commercial.key /opt/zimbra/conf/slapd.key
cp /opt/zimbra/ssl/zimbra/commercial/commercial.crt /opt/zimbra/conf/slapd.crt
cp /opt/zimbra/ssl/zimbra/commercial/commercial.key /opt/zimbra/conf/nginx.key
cp /opt/zimbra/ssl/zimbra/commercial/commercial.crt /opt/zimbra/conf/nginx.crt
  • Restart the Zimbra services.

Advance Topics

Miscellaneous Topics

Set Up Proxy to use Clear Text for Upstream Connections

Source : Admin Guide Draft, 'Set Up Proxy to use Clear Text for Upstream Connections'

When setting up the proxy to use clear text for upstream connections, set zimbraReverseProxySSLToUpstreamEnabled to FALSE.

This attribute defaults to TRUE. In an "out of the box" proxy set up, the upstream communication defaults to SSL.

REST URL Generation

Source : Admin Guide Draft, 'REST URL Generation'

For REST URL, you set the host name, service protocol, and services port globally or for a specific domain from the following attributes.

  • zimbraPublicServiceHostname
  • zimbraPublicServiceProtocol
  • zimbraPublicServicePort

When generating REST URL’s:

  • If domain.zimbraPublicServiceHostname is set, use zimbraPublicServiceProtocol + zimbraPublicServiceHostname + zimbraPublicServicePort
  • Otherwise it falls back to the server (account's home server) attributes:
    • protocol is computed from server.zimbraMailMode
    • hostname is server.zimbraServiceHostname
    • port is computed from the protocol.

Note: Why use zimbraMailReferMode - In earlier versions, a local config variable called zimbra_auth_always_send_refer determined which action the back-end server took when a user’s mailbox did not reside on the server that the user logged in to. The default value of FALSE redirected the user if the user was logging in on the wrong backend host.

On a multiserver ZCS, if a load balanced name was needed to create a friendly landing page, a user would always have to be redirected. In that case, zimbra_auth_always_send_refer was set to TRUE.

Now with a full-fledged reverse proxy, users do not need to be redirected. The localconfig variable zimbraMailReferMode is used with nginx reverse proxy.

Set Proxy Trusted IP Addresses

Source : Admin Guide Draft, 'Set Proxy Trusted IP Addresses'

When a proxy is configured with ZCS, each proxy server’s IP address must be configured in LDAP attribute zimbraMailTrustedIP to identify the proxy addresses as trusted when users log in through the proxy. The proxy IP address is added to the X-Forwarded-For header information. The X-Forwarded-For header is automatically added to the localconfig zimbra_http_originating_ip_header attribute. When a user logs in, this IP address and the user’s address are verified in the Zimbra mailbox log.

Set each proxy IP address in the attribute. For example, if you have two proxy servers: zmprov mcf +zimbraMailTrustedIP {IP of nginx-1} +zimbraMailTrustedIP {IP of nginx-2}

Note: To verify that X-Forwarded-For was correctly added to the localconfig, type zmlocalconfig | grep -i http. You should see zimbra_http originating_ip_header = X-Forwarded-For.

Zimbra Proxy Performance Tuning

See the following unresolved RFE:

Ldap Attributes For Proxy

* Created Zimbra_Proxy_Attributes-Detailed .


Source :

Pre 8.5 Attributes

To check the description, type, default value etc for any of the above attributes, simply do 'zmprov desc -a <attr>' . Full details of attributes can be found at Zimbra_Proxy_Attributes-Detailed .

eg. zmprov desc -a zimbraReverseProxyWorkerConnections

   zimbraReverseProxyWorkerConnections
   Maximum number of connections that an NGINX Proxy worker process is
   allowed to handle
              type : integer
             value : 
          callback : 
         immutable : false
       cardinality : single
        requiredIn : 
        optionalIn : server,globalConfig
             flags : serverInherited
          defaults : 10240
               min : 1
               max : 40960
                id : 725
   requiresRestart : nginxproxy
             since : 5.0.10
   deprecatedSince :


zimbraReverseProxyAdminEnabled

zimbraReverseProxyAdminIPAddress

zimbraReverseProxyAdminPortAttribute

zimbraReverseProxyAuthWaitInterval

zimbraReverseProxyAvailableLookupTargets

zimbraReverseProxyCacheEntryTTL

zimbraReverseProxyCacheFetchTimeout

zimbraReverseProxyCacheReconnectInterval

zimbraReverseProxyClientCertCA

zimbraReverseProxyClientCertMode

zimbraReverseProxyConnectTimeout

zimbraReverseProxyDefaultRealm

zimbraReverseProxyDnsLookupInServerEnabled

zimbraReverseProxyDomainNameAttribute

zimbraReverseProxyDomainNameQuery

zimbraReverseProxyDomainNameSearchBase

zimbraReverseProxyErrorHandlerURL

zimbraReverseProxyExternalRouteIncludeOriginalAuthusername

zimbraReverseProxyGenConfigPerVirtualHostname

zimbraReverseProxyHttpEnabled

zimbraReverseProxyHttpPortAttribute

zimbraReverseProxyHttpSSLPortAttribute

zimbraReverseProxyIPLoginLimit

zimbraReverseProxyIPLoginLimitTime

zimbraReverseProxyImapEnabledCapability

zimbraReverseProxyImapExposeVersionOnBanner

zimbraReverseProxyImapPortAttribute

zimbraReverseProxyImapSSLPortAttribute

zimbraReverseProxyImapSaslGssapiEnabled

zimbraReverseProxyImapSaslPlainEnabled

zimbraReverseProxyImapStartTlsMode

zimbraReverseProxyInactivityTimeout

zimbraReverseProxyIpThrottleMsg

zimbraReverseProxyLogLevel

zimbraReverseProxyLookupTarget

zimbraReverseProxyMailEnabled

zimbraReverseProxyMailHostAttribute

zimbraReverseProxyMailHostQuery

zimbraReverseProxyMailHostSearchBase

zimbraReverseProxyMailImapEnabled

zimbraReverseProxyMailImapsEnabled

zimbraReverseProxyMailMode

zimbraReverseProxyMailPop3Enabled

zimbraReverseProxyMailPop3sEnabled

zimbraReverseProxyPassErrors

zimbraReverseProxyPop3EnabledCapability

zimbraReverseProxyPop3ExposeVersionOnBanner

zimbraReverseProxyPop3PortAttribute

zimbraReverseProxyPop3SSLPortAttribute

zimbraReverseProxyPop3SaslGssapiEnabled

zimbraReverseProxyPop3SaslPlainEnabled

zimbraReverseProxyPop3StartTlsMode

zimbraReverseProxyPortQuery

zimbraReverseProxyPortSearchBase

zimbraReverseProxyRouteLookupTimeout

zimbraReverseProxyRouteLookupTimeoutCache

zimbraReverseProxySSLCiphers

zimbraReverseProxySSLToUpstreamEnabled

zimbraReverseProxySendImapId

zimbraReverseProxySendPop3Xoip

zimbraReverseProxyUpstreamConnectTimeout

zimbraReverseProxyUpstreamPollingTimeout

zimbraReverseProxyUpstreamReadTimeout

zimbraReverseProxyUpstreamSendTimeout

zimbraReverseProxyUpstreamServers

zimbraReverseProxyUseExternalRoute

zimbraReverseProxyUseExternalRouteIfAccountNotExist

zimbraReverseProxyUserLoginLimit

zimbraReverseProxyUserLoginLimitTime

zimbraReverseProxyUserNameAttribute

zimbraReverseProxyUserThrottleMsg

zimbraReverseProxyWorkerConnections

zimbraReverseProxyWorkerProcesses

New 8.5 Attributes

zimbraReverseProxyAcceptMutex

zimbraReverseProxyUpstreamEwsServers

zimbraReverseProxyUpstreamLoginServers

zimbraReverseProxyExactServerVersionCheck

Proxy Configuration And Template Files

Understanding The Zimbra Proxy Config Rewrite (zmproxyconfgen) Process (zmproxyconfig And zmproxyctl)

Source:

One changes the proxy environment variables with either zmproxyconfig or zmprov . After you make your changes, zmproxyctl restart is ran, which will restart the proxy service and invoke zmproxyconfgen in the background. The zmproxyconfgen script reads in the proxy configuration template files and generates the NGINX config files after performing keyword substitution on the template files with values from the LDAP configuration. zmproxyconfgen is usually never invoked directly -- it is invoked automatically by zmproxyctl.

To change multiple LDAP attributes related to the proxy at once (i.e any of the zimbraReverseProxy* attrs) instead of using zmprov command to change each attribute individually, /opt/zimbra/libexec/zmproxyconfig can be used. The usage for this script see Ajcody-Proxy-Guide-Rewrite-Project#zmproxyconfig

You will have to restart the proxy after modifying your proxy server with zmproxyconfig by running 'zmproxyctl restart' .

  • Note: zmproxyconfig is no longer required to enable the proxy if you are doing a fresh installation of ZCS with proxy. The installer already runs this and the proxy is enabled for imap/pop/https access by default. Although it is needed while trying to deploy proxy in existing non-proxy environments when using the existing servers. Kindly refer http://wiki.zimbra.com/wiki/Enabling_Zimbra_Proxy#Using_existing_servers

Config Files And Config Templates

Source: http://wiki.zimbra.com/wiki/Zimbra_Proxy_Guide#Config_Files_and_Config_Templates

To simplify configuration, the Nginx configuration files have been split up into different config files based on functionality. The main, top-level configuration file is /opt/zimbra/conf/nginx.conf, and this file includes the main config, memcache config, mail config, and web config files. The mail config in turn includes the configuration for imap, imaps, pop3 and pop3s. The web config includes the configuration for http and https. The template files follow exactly the same inclusion hierarchy, and each configuration file has a corresponding template file from which it is generated. Each template file resides in /opt/zimbra/conf/nginx/templates/ Each corresponding config file resides in /opt/zimbra/conf/nginx/includes/ (excluding top-level config file which is /opt/zimbra/conf/nginx/nginx.conf)

Config File Hierarchy

Source: http://wiki.zimbra.com/wiki/Zimbra_Proxy_Guide#Config_File_Hierarchy

The symbol |_ indicates that a file is included by the one above it. Increasing levels of indentation indicate lower levels of config files

   /opt/zimbra/conf/nginx.conf
    |_ /opt/zimbra/conf/nginx/includes/nginx.conf.main
    |_ /opt/zimbra/conf/nginx/includes/nginx.conf.memcache
    |_ /opt/zimbra/conf/nginx/includes/nginx.conf.zmlookup
    |_ /opt/zimbra/conf/nginx/includes/nginx.conf.mail
       |_ /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imap.default
       |_ /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imaps.default
       |_ /opt/zimbra/conf/nginx/includes/nginx.conf.mail.pop3.default
       |_ /opt/zimbra/conf/nginx/includes/nginx.conf.mail.pop3s.default
    |_ /opt/zimbra/conf/nginx/includes/nginx.conf.web
       |_ /opt/zimbra/conf/nginx/includes/nginx.conf.web.http.default
       |_ /opt/zimbra/conf/nginx/includes/nginx.conf.web.https.default
       |_ /opt/zimbra/conf/nginx/includes/nginx.conf.web.admin.default  
       |_ /opt/zimbra/conf/nginx/includes/nginx.conf.web.sso.default

Description of Config Files

Source:

   * /opt/zimbra/conf/nginx.conf
     Description:
       Core Nginx configuration file read by Nginx Proxy
     Template:
       /opt/zimbra/conf/nginx/templates/nginx.conf.template
     Includes:
       /opt/zimbra/conf/nginx/includes/nginx.conf.main
       /opt/zimbra/conf/nginx/includes/nginx.conf.memcache
       /opt/zimbra/conf/nginx/includes/nginx.conf.zmlookup
       /opt/zimbra/conf/nginx/includes/nginx.conf.mail
       /opt/zimbra/conf/nginx/includes/nginx.conf.web
   * /opt/zimbra/conf/nginx/includes/nginx.conf.main
     Description:
       Defines global parameters for all NGINX worker processes
     Template:
       /opt/zimbra/conf/nginx/templates/nginx.conf.main.template
     Includes:
       None
   * /opt/zimbra/conf/nginx/includes/nginx.conf.memcache
     Description:
       Defines memcache configuration, common for mail and web
     Template:
       /opt/zimbra/conf/nginx/templates/nginx.conf.memcache.template
     Includes:
       None
   * /opt/zimbra/conf/nginx/includes/nginx.conf.mail
     Description:
       Defines the common mail configuration common to IMAP and POP3
     Template:
       /opt/zimbra/conf/nginx/templates/nginx.conf.mail.template
     Includes:
       /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imap.default
       /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imaps.default
       /opt/zimbra/conf/nginx/includes/nginx.conf.mail.pop3.default
       /opt/zimbra/conf/nginx/includes/nginx.conf.mail.pop3s.default
   * /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imap.default
     Description:
       Defines the server block for IMAP 
     Template:
       /opt/zimbra/conf/nginx/templates/nginx.conf.mail.imap.default.template
     Includes:
       None
   * /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imaps.default
     Description:
       Defines the server block for IMAPS
     Template:
       /opt/zimbra/conf/nginx/templates/nginx.conf.mail.imaps.default.template
     Includes:
       None
   * /opt/zimbra/conf/nginx/includes/nginx.conf.mail.pop3.default
     Description:
       Defines the server block for POP3
     Template:
       /opt/zimbra/conf/nginx/templates/nginx.conf.mail.pop3.default.template
     Includes:
       None
   * /opt/zimbra/conf/nginx/includes/nginx.conf.mail.pop3s.default
     Description:
       Defines the server block for POP3S
     Template:
       /opt/zimbra/conf/nginx/templates/nginx.conf.mail.pop3s.default.template
     Includes:
       None
   * /opt/zimbra/conf/nginx/includes/nginx.conf.web
     Description:
       Defines the common web configuration common to HTTP and HTTPS
     Template:
       /opt/zimbra/conf/nginx/templates/nginx.conf.web.template
     Includes:
       /opt/zimbra/conf/nginx/includes/nginx.conf.web.http.default
       /opt/zimbra/conf/nginx/includes/nginx.conf.web.https.default
       /opt/zimbra/conf/nginx/includes/nginx.conf.web.sso.default
       /opt/zimbra/conf/nginx/includes/nginx.conf.web.admin.default
   * /opt/zimbra/conf/nginx/includes/nginx.conf.web.http.default
     Description:
       Defines the server block for HTTP
     Template:
       /opt/zimbra/conf/nginx/templates/nginx.conf.web.http.default.template
     Includes:
       None
   * /opt/zimbra/conf/nginx/includes/nginx.conf.web.https.default
     Description:
       Defines the server block for HTTPS
     Template:
       /opt/zimbra/conf/nginx/templates/nginx.conf.web.https.default.template
     Includes:
       None
   * /opt/zimbra/conf/nginx/includes/nginx.conf.web.sso.default
     Description:
       Defines the server block for Single Sign on (SSO)
     Template:
       /opt/zimbra/conf/nginx/templates/nginx.conf.web.sso.default.template
     Includes:
       None
   * /opt/zimbra/conf/nginx/includes/nginx.conf.web.admin.default
     Description:
       Defines the server block for admin user
     Template:
       /opt/zimbra/conf/nginx/templates/nginx.conf.web.admin.default.template
     Includes:
       None

Zimbra Proxy Related CLI Commands

zmproxyconfig


Source: http://wiki.zimbra.com/wiki/Zimbra_Proxy_Guide#Proxy_config_rewrite.28zmproxyconfgen.29.2C_zmproxyconfig_and_zmproxyctl

Syntax And Usage

/opt/zimbra/libexec/zmproxyconfig help

Usage: /opt/zimbra/libexec/zmproxyconfig [-h] [-o] [-m] [-w] [-d [-r] [-s] [-a w1:w2:w3:w4] [-c [-n n1:n2]] [-i p1:p2:p3:p4] [-p p1:p2:p3:p4] [-x mailmode]] [-e [-a w1:w2:w3:w4] -C] [-n n1:n2 [-i p1:p2:p3:p4] [-p p1:p2:p3:p4] [-u|-U] [-x mailmode]] [-f] -H hostname

Description

-h: display this help message

-H: Hostname of server on which enable/disable proxy functionality.

-a: Colon separated list of Web ports to use. Format: HTTP-STORE:HTTP-PROXY:HTTPS-STORE:HTTPS-PROXY (Ex: 8080:80:8443:443)

-d: disable proxy

-e: enable proxy

-f: Full reset on memcached port and search queries and POP/IMAP throttling.

-i: Colon separated list of IMAP ports to use. Format: IMAP-STORE:IMAP-PROXY:IMAPS-STORE:IMAPS-PROXY (Ex: 7143:143:7993:993)

-m: Toggle mail proxy portions

-o: Override enabled checks

-p: Colon separated list of POP ports to use. Format: POP-STORE:POP-PROXY:POPS-STORE:POPS-PROXY (Ex: 7110:110:7995:995)

-r: Run against a remote host. Note that this requires the server to be properly configured in the LDAP master.

-s: Set cleartext to FALSE (secure mode) on disable

-t: Disable reverse proxy lookup target for store server. Only valid with -d. Be sure that you intend for all proxy function for the server to be disabled

-w: Toggle Web proxy portions

-c: Disable Admin Console proxy portions.

-C: Enable Admin Console proxy portions.

-n: Colon separated list of Admin Console ports to use. Format: ADMIN-CONSOLE-STORE:ADMIN-CONSOLE-PROXY (Ex: 7071:9071)

-x: the proxy mail mode when enable proxy, or the store mail mode when disable proxy (Both default: http).

-u: disable SSL connection from proxy to mail store.

-U: enable SSL connection from proxy to mail store.

hostname is the value of the zimbra_server_hostname LC key for the server being modified.

Required options are -f by itself, or -f with -d or -e.

Note that -d or -e require one or both of -m and -w.

Note that -i or -p require -m.

Note that -a requires -w.

Note that -c/-C requires -w, and -n requires -c/-C. When disabling web proxy, admin console proxy will be automatically disabled.

Note that -u or -U are only available when proxy is enabled by -e.

Note that -x requires -w and -d for store.

Note that -x requires -w for proxy.

Note that no matter what mail mode is set by -x and no matter proxy is enabled or disabled, admin console's mode is always https.

The following are the defaults for -a, -i, -p, and -x if they are not supplied as options.

-a default on enable: 8080:80:8443:443

-a default on disable: 80:0:443:0

-i default on enable: 7143:143:7993:993

-i default on disable: 143:7143:993:7993

-p default on enable: 7110:110:7995:995

-p default on disable: 110:7110:995:7995

-n default on enable: 7071:9071

-n default on disable: 7071:9071

-x default on store disable: http

-x default on proxy enable/disable: http, but -x default on proxy enable when upstream ssl connection is enabled: https

eg. To change the value of zimbraReverseProxySSLToUpstreamEnabled from TRUE(default) to FALSE (i.e use http for the proxy<->mailstore connections instead of https), execute this on the server running the proxy

/opt/zimbra/libexec/zmproxyconfig -e -m -w -H <hostname> -u

You will have to restart the proxy after this by running 'zmproxyctl restart'

  • Note: zmproxyconfig is no longer required to enable the proxy if you are doing a fresh installation of ZCS with proxy. The installer already runs this and the proxy is enabled for imap/pop/https access by default. Although it is needed while trying to deploy proxy in existing non-proxy environments when using the existing servers. Kindly refer http://wiki.zimbra.com/wiki/Enabling_Zimbra_Proxy#Using_existing_servers

zmproxyctl


Syntax

zmproxyctl help

Usage : /opt/zimbra/bin/zmproxyctl start|stop|restart|reload|status

Description

zmprov


Syntax

zmprov [cmd]

Description

Long Name Short Name Description
--getAllReverseProxyURLs -garpu Used to list all the upstream mailstore servers (NLEs) that should be used for reverse proxy lookup by the proxy
--getAllReverseProxyBackends -garpb Used to list all the upstream mailstore servers that are reverse-proxied by the proxy
--getAllMtaAuthURLs -gamau Used to publish into saslauthd.conf the servers that should be used for saslauthd.conf MTA auth
--getAllMemcachedServers -gamcs Used to list memcached servers (for Zimbra Proxy use)

zmproxyconfgen


Syntax

Description

zmnginxconf


Syntax

Description

Will be functional again with Bug 95169 being fixed.

zmproxypurge


Syntax

Description

zmproxyinit [ deprecated since ???]


Description

This command is no longer used on any supported versions of ZCS. See zmproxyconfig .

Details On Some Of The Zimbra Customization To The Proxy Components

Source : [Internal Eng. Wiki /index.php/Zimbra/Admin/Proxy/Overview#Motivation_behind_Expanding_Nginx ]

Motivation behind Expanding Nginx

Although official Nginx is powerful, it can't meed all the needs of Zimbra.

AUTH_HTTP

Nginx's AUTH_HTTP protocol (described in http://wiki.nginx.org/NginxMailCoreModule#Authentication) is borrowed to implement the lookup with NLE. However, official Nginx's mail AUTH_HTTP config can only specify one NLE URL (auth_http directive). We wish to extend this support so that NZ can use more than one NLE. When a great load of user logins come, NZ can dispatch the load to multiple NLE running in different zimbra servers. This will end up not overloading a single NLE. In a production set-up, it is likely that there will be many different NZ identically configured, and so the relationship between the NZ set and the server set running NLE is a many-to-many relationship.

Besides, the lookup is also necessary for zimbra web accessing, which is not supported by official Nginx. Therefore, this lookup and upstream choose function must also be implemented as an nginx http module.

Finally, because Zimbra doesn't use AUTH_HTTP protocol for real authentication, it might add some custom HTTP headers (such as "Auth-ID") and avoid some useless content transmit (such as password via "Auth-Pass").

All of these above have to be extended by Zimbra.

Memcache Module

It is likely that a single user may log in many times via NZ in a short time, or one http request will generate a great many of sub requests, all of which contains the same login information and thus will get the same route lookup result. In such cases, NZ will have to contact the NLE many times. It is not likely that the upstream server information for a particular client will change frequently. (A change in the upstream server usually corresponds to a user's mailbox being migrated to a different server). And at the same time NLE is relative much slower than Nginx, becoming the bottle neck.

As such, it will be beneficial if NZ is able to cache such information so that repeated trips to the NLE servers are avoided. Also, this cached information must be made available to possibly multiple NZ instances. Hence, "memcached" (http://www.danga.com/memcached/) was introduced for this purpose. Memcached is a high-performance, distributed memory object caching system.

Although official Nginx has a build-in "ngx_http_memcache" module, it can only be used for "URL->web page" cache. Therefore, a global "ngx_memcache" module is implemented in NZ which could be accessed by both web and mail modules. Memcached is used to cache 5 kinds of information as follows. The 4th and 5th are used in mail throttle control, which will be described in [Zimbra/Admin/Proxy/Throttle Control]:

  • alias-->account name
  • account name->route information
  • account id-->route information
  • login IP address->login count (IP throttle control)
  • account name->login count (user throttle control)

Enhanced Mail Protocol Support

Compared to the web proxy part, official nginx is relative weak in mail proxy, such as:

  • Official nginx cannot completely support IMAP ID command, which is required for the logging of upstream zimbra servers
  • Official nginx can't echo the [ALERT] msg before authenticating to the upstream server
  • Official nginx does not support GSSAPI authentications.
  • Official nginx does not support upstream SSL connection, that is, using POP3S and IMAPS protocol connecting nginx and upstream server.

These features are critical for a mail system of enterprise level. ZCS' mail server support these, so NZ has to support them as well.

Unlike web proxy, during which nginx simply proxies all the requests, in mail proxy nginx has to handle some mail protocol commands rather than proxy them to upstream before login is given. The reason is before getting route from NLE, NZ doesn't know to use which upstream server. These commands include "ID", "NOOP", "LOGIN", "CAPABILITY", "AUTHENTICATE", "LOGOUT", "STARTTLS" for IMAP, and "USER", "CAPA", "PASS", "NOOP", "STLS", "QUIT" for POP3. Although official nginx support most of them, but it has several bugs (like IMAP tag/untag responses), which are fixed by NZ.

SMTP proxy os nginx is not used in Zimbra so this part is not touched. Besides, some authentication methods supported by official nginx like SASL LOGIN, SALS CRAM-MD5 and APOP won't work because NLE can't support them.

Throttle Control

To reject DoS like attack, NZ will count the number of logins per IP/username in periods and deny the login if it's beyond a certain configured threshold value. This functionality is called Throttle Control. There are 2 kinds of them: IP Throttle Control and User Throttle Control. IP Throttle Control counts the login by client's IP address. But sometimes, this throttle control is not suitable for some cases (for example, many clients who locate in an internal network and thus have the same outside IP address of the gateway). Therefore, User Throttle Control is also introduced, which counts the login by client's user name. All these counters are stored in memcached so it's not persistent. And its algorithm is not quite accurate, but simple and feasible in practice. It's beneficial to implement Throttle Control in NZ because it can block the attacks before they touch real upstream mail servers.

Currently Throttle Control is only implemented for mail proxy. For web proxy, users can manually configure nginx with "deny" & "allow" directive. The following LDAP attributes have to be set to use this feature which get written using the corresponding nginx directives added for this module in /opt/zimbra/conf/nginx/templates/nginx.conf.mail.template configuration file.

zimbraReverseProxyIPLoginLimit    ->   mail_login_ip_max               ${mail.ipmax};
zimbraReverseProxyIPLoginLimitTime    ->   mail_login_ip_ttl               ${mail.ipttl};
zimbraReverseProxyIpThrottleMsg    ->   mail_login_ip_rejectmsg         "${mail.iprej}";
zimbraReverseProxyUserLoginLimit    ->   mail_login_user_max             ${mail.usermax};
zimbraReverseProxyUserLoginLimitTime    ->   mail_login_user_ttl             ${mail.userttl};
zimbraReverseProxyUserThrottleMsg    ->   mail_login_user_rejectmsg       "${mail.userrej}";

Cert Per Virtual Host Name

Many ZCS host thousands of domains, each of which has several virtual host names. Each domain may own its own SSL certificate/private key. If nginx is deployed, nginx can't proxy the SSL handshake to the upstream. So NZ has to support this "cert per domain" feature. Actually, the upstream Jetty server doesn't implement this feature. Therefore, NZ is critical in this use case. See the following for SSL Certificates Per Domain Set Up.

Advanced Proxy Configuration Examples via CLI

Configure Zimbra Proxy For POP[S] And IMAP[S] Only

Configure Zimbra Proxy For POP[S] And IMAP[S] After A HTTP[S] Configuration

Source: Admin Guide

If you need to set up IMAP/POP proxy after you have already installed Zimbra HTTP proxy, do the following below. Assumption is a Zimbra mailbox server/s and the proxy node/s are already setup and running.

Note - This section doesn't make sense - this needs to be technically proofed.

  1. The first to setup headers don't make sense. Is the first one when your setting up a NEW proxy host but have other ones already deployed that will continue to do just HTTP ? The second header, is this reconfiguring a proxy host? The assumptions to this exercise is that HTTP proxy is already deployed some where in the environment.
  2. In the first section, why is it - "On each Zimbra mailbox server" ? There's no configuration being ran on the proxy nodes at all in this example.
Set Up IMAP/POP Proxy with Separate Proxy Node

If your configuration includes a separate proxy server, you must do the following.

1. 	On each Zimbra mailbox server that you want to proxy with, enable the proxy for IMAP/POP proxy. 
/opt/zimbra/libexec/zmproxyconfig -e -m -H mailbox.node.service.hostname
This configures the following:
• 	zimbraImapBindPort to 7143
• 	zimbraImapProxyBindPort to 143
• 	zimbraImapSSLBindPort to 7993
• 	zimbraImapSSLProxyBindPort to 993
• 	zimbraPop3BindPort to 7110
• 	zimbraPop3ProxyBindPort to110
• 	zimbraPop3SSLBindPort to 7995
• 	zimbraPop3SSLProxyBindPort to 995
• 	zimbralmapCleartextLoginEnabled to TRUE
• 	zimbraReverseProxyLookupTarget to TRUE
• 	zimbraPop3CleartextLoginEnabled to TRUE
2.   	Restart services on the proxy and mailbox servers.
zmcontrol restart


Set Up Proxy Node

1. 	On each proxy node that has the proxy service installed, enable the proxy for the web. 
/opt/zimbra/libexec/zmproxyconfig -e -m -H proxy.node.service.hostname
This configures the following:
• 	zimbraImapBindPort to 7143
• 	zimbraImapProxyBindPort to 143
• 	zimbraImapSSLBindPort to 7993
• 	zimbraImapSSLProxyBindPort to 993
• 	zimbraPop3BindPort to 7110
• 	zimbraPop3ProxyBindPort to110
• 	zimbraPop3SSLBindPort to 7995
• 	zimbraPop3SSLProxyBindPort to 995
• 	zimbraReverseProxyMailEnabled to TRUE


Set Up a Single Node
If Zimbra proxy is installed with Zimbra Collaboration on the same server, do the following. 

1. 	Enable the proxy for the web. 
/opt/zimbra/libexec/zmproxyconfig -e -m -H mailbox.node.service.hostname
This configures the following:
• 	zimbraImapBindPort to 7143
• 	zimbraImapProxyBindPort to 143
• 	zimbraImapSSLBindPort to 7993
• 	zimbraImapSSLProxyBindPort to 993
• 	zimbraPop3BindPort to 7110
• 	zimbraPop3ProxyBindPort to110
• 	zimbraPop3SSLBindPort to 7995
• 	zimbraPop3SSLProxyBindPort to 995
• 	zimbraImapCleartextLoginEnabled to TRUE
• 	zimbraReverseProxyLookupTarget to TRUE
• 	zimbraPop3CleartextLoginEnabled to TRUE
• 	zimbraReverseProxyMailEnabled to TRUE
2.   	Restart services on the proxy and mailbox servers.
zmcontrol restart

Disabling Zimbra Proxy For POP[S] And IMAP[S] Only

Configure Zimbra Proxy For POP[S] Only

Disabling Zimbra Proxy For POP[S] Only

Configure Zimbra Proxy For POP[S] And HTTP[S] Only

Disabling Zimbra Proxy For POP[S] And HTTP[S] Only

Configure Zimbra Proxy For IMAP[S] Only

Disabling Zimbra Proxy For IMAP[S] Only

Configure Zimbra Proxy For IMAP[S] And HTTP[S] Only

Disabling Zimbra Proxy For IMAP[S] And HTTP[S] Only

Configure Zimbra Proxy For HTTP[S] Only

Source: Admin Guide Draft 'Setting Up HTTP Proxy'

To set up HTTP proxy, Zimbra Proxy must be installed on the identified nodes.
Note: You can run the command as /opt/zimbra/libexec/zmproxyconfig -r, to run against a 
remote host. Note that this requires the server to be properly configured in the LDAP master.

Set Up HTTP Proxy as a Separate Proxy Node

When your configuration includes a separate proxy server follow these steps.

1. 	On each Zimbra mailbox server that you want to proxy with, enable the proxy for the web. 
/opt/zimbra/libexec/zmproxyconfig -e -w -H mailbox.node.service.hostname
This configures the following:
• 	zimbraMailReferMode to reverse-proxied. See Note below.
• 	zimbraMailPort to 8080, to avoid port conflicts. 
• 	zimbraMailSSLPort to 8443, to avoid port conflicts.
• 	zimbraReverseProxyLookupTarget to TRUE
• 	zimbraMailMode to http. This is the only supported mode. 

2.   	Restart services on the proxy and mailbox servers.
zmcontrol restart

3.   	Configure each domain with the public service host name to be used for REST URLs, email, 
and Briefcase folders. 
zmprov modifyDomain <domain.com> zimbraPublicServiceHostname <hostname.domain.com>

Set Up Proxy Node

1. 	On each proxy node that has the proxy service installed, enable the proxy for the web. 
/opt/zimbra/libexec/zmproxyconfig -e -w -H proxy.node.service.hostname
This configures the following:
• 	zimbraMailReferMode to reverse-proxied. See Note below.
• 	zimbraMailProxyPort to 80, to avoid port conflicts. 
• 	zimbraMailSSLProxyPort to 443, to avoid port conflicts.
• 	zimbraReverseProxyHttpEnabled to TRUE to indicate that Web proxy is enabled.
• 	zimbraReverseProxyMailMode defaults to HTTP.
To set the proxy server mail mode, add the -x option to the command with the specific mode: http, https, 
both, redirect, mixed.

Set Up a Single Node for HTTP Proxy

If Zimbra proxy is installed along with ZCS on the same server, follow this step.

1. 	On each zimbra mailbox server that you want to proxy with, enable the proxy for the web. 
/opt/zimbra/libexec/zmproxyconfig -e -w -H mailbox.node.service.hostname
This configures the following:
• 	zimbraMailReferMode to reverse-proxied. See Note below.
• 	zimbraMailPort to 8080, to avoid port conflicts. 
• 	zimbraMailSSLPort to 8443, to avoid port conflicts.
• 	zimbraReverseProxyLookupTarget to TRUE
• 	zimbraMailMode to http. This is the only supported mode. 
• 	zimbraMailProxyPort to 80, to avoid port conflicts. 
• 	zimbraMailSSLProxyPort to 443, to avoid port conflicts.
• 	zimbraReverseProxyHttpEnabled to TRUE to indicate that Web proxy is enabled.
• 	zimbraReverseProxyMailMode defaults to HTTP.
To set the proxy server mail mode, add the -x option to the command with the specific mode: http, https, 
both, redirect, mixed.

2.   	Restart services on the proxy and mailbox servers.
zmcontrol restart

Configure each domain with the public service host name to be used for REST URLs, email and Briefcase 
folders. 
zmprov modifyDomain <domain.com> zimbraPublicServiceHostname <hostname.domain.com>

Disabling Zimbra Proxy For HTTP[S] Only

Configure Or Customize The Zimbra Proxy For The Admin Console

Disabling Or Customize The Zimbra Proxy For The Admin Console

Configure Zimbra Proxy For Kerberos Authentication

Source: Admin Guide Draft, 'Configure Zimbra Proxy for Kerbose Authentication'

If you use the Kerberos5 authenticating mechanism, you can configure it for the IMAP and POP proxy.

Note: Make sure that your Kerberos5 authentication mechanism is correctly configured. See Chapter 5, Zimbra LDAP Service.

  • On each proxy node, set the zimbraReverseProxyDefaultRealm server attribute to the realm name corresponding to the proxy server. For example:
zmprov ms [DNS name.isp.net] zimbraReverseProxyDefaultRealm [ISP.NET]
  • Each proxy IP address where email clients connect must be configured for GSSAPI authentication by the mail server. On each proxy node for each of the proxy IP addresses:
zmprov mcf +zimbraReverseProxyAdminIPAddress [IP address]
  • On each proxy server:
zmprov ms [proxyexample.net] zimbraReverseProxyImapSaslGssapiEnabled TRUE
zmprov ms proxyl.isp.net zimbraReverseProxyPop3SaslGssapiEnabled TRUE
  • Restart the proxy server
zmproxyctl restart


Disabling Zimbra Proxy For Kerberos Authentication

Configure Zimbra Proxy For AUTH GSSAPI

Source: http://wiki.zimbra.com/index.php?title=NGINX_Configuration_Tips#AUTH_GSSAPI

Nginx supports the SASL GSSAPI Authentication Mechanism for POP and IMAP through the zimbraReverseProxyPop3SaslGssapiEnabled and zimbraReverseProxyImapSaslGssapiEnabled attributes respectively.

Set them to true to enable GSSAPI Authentication for Nginx

zmprov ms <server> zimbraReverseProxyPop3SaslGssapiEnabled TRUE
zmprov ms <server> zimbraReverseProxyImapSaslGssapiEnabled TRUE

Set these attributes to FALSE to disable GSSAPI Authentication.

Disabling Zimbra Proxy For AUTH GSSAPI

Configure Zimbra Proxy For AUTH PLAIN

Source: http://wiki.zimbra.com/index.php?title=NGINX_Configuration_Tips#AUTH_PLAIN

Nginx supports enablement of the SASL PLAIN Authentication Mechanism (RFC 4616) for POP and IMAP through the zimbraReverseProxyPop3SaslPlainEnabled and zimbraReverseProxyImapSaslPlainEnabled attributes respectively.

Set them to true to enable PLAIN Authentication for Nginx

zmprov ms <server> zimbraReverseProxyPop3SaslPlainEnabled TRUE
zmprov ms <server> zimbraReverseProxyImapSaslPlainEnabled TRUE

Set them to false to disable PLAIN Authentication for Nginx

zmprov ms <server> zimbraReverseProxyPop3SaslPlainEnabled FALSE
zmprov ms <server> zimbraReverseProxyImapSaslPlainEnabled FALSE


Disabling Zimbra Proxy For AUTH PLAIN

Configure Zimbra Proxy For Clear-Text Logins

Source: http://wiki.zimbra.com/index.php?title=NGINX_Configuration_Tips#Allow.2FDisallow_Clear-Text_Logins

To configure Nginx to allow/disallow cleartext logins on non SSL/TLS connections, use the zimbraReverseProxyPop3StartTlsMode and zimbraReverseProxyImapStartTlsMode attributes.

To allow clear-text logins for POP and IMAP (respectively) over non-SSL/TLS connections, run these commands

zmprov ms <server> zimbraReverseProxyPop3StartTlsMode on
zmprov ms <server> zimbraReverseProxyImapStartTlsMode on

To disallow clear-text logins for POP and IMAP (respectively) over non-SSL/TLS connections, run these commands

zmprov ms <server> zimbraReverseProxyPop3StartTlsMode only
zmprov ms <server> zimbraReverseProxyImapStartTlsMode only

Disabling Zimbra Proxy For Clear-Text Logins

To disallow clear-text logins for POP and IMAP (respectively) over non-SSL/TLS connections, run these commands

zmprov ms <server> zimbraReverseProxyPop3StartTlsMode only
zmprov ms <server> zimbraReverseProxyImapStartTlsMode only

SSL Certificates Per Domain Set Up

Source:

This document explains how to add per domain cert on a ZCS running 7.x version.

Until ZCS 6.x, per domain ssl certificate or multiple ssl certificates on a single ZCS was not supported. RFE #8128. From ZCS 7.x, the feature is been added.

In this example, I am adding a new domain called example.com and deploying a new certificate for example.com.

Prerequisites

  • Zimbra proxy service must be installed and enabled on the server. In multi server environment, do these steps on the proxy node.
  • You should have a signed certificate + matching key pair and the trusted chain certs from your CA (Certificate Authority) .
  • You will need to add ipv4 addresses per domain which will pair to the respective domain name. For example:
 1.1.1.1 => example.com
 2.2.2.2 => otherdomain.com
 3.3.3.3 => yetanotherdomain.com

Configure Zimbra Proxy Server

1. Make sure Zimbra proxy service is configured correctly and serving https. If not, configure proxy now, run following as zimbra. It will set "zimbraReverseProxyMailMode" to both.

 su - zimbra
 /opt/zimbra/libexec/zmproxyconfig -m -w -e -x both -H `zmhostname`

2. Restart proxy service

 zmproxyctl restart
  • If its already configured - skip to next section.

Configuring IP address and domain

1. Add a new ipv4 address to the server which will pair to name example.com (via the example virtual hostname mail.example.com). You can do it using IP address aliasing. For example, the new address can be assigned to eth0:1 device. Lets consider the new ipv4 address is 1.2.3.4 which should be an A record for mail.example.com. The IP address could be public (if server is on Internet) or internal (if the server is behind firewall/NAT'ed).

2. Add the new domain example.com. Set zimbraVirtualHostName to mail.example.com and zimbraVirtualIPAddress to 1.2.3.4. Make sure the zimbraVirtualHostName is set to the name which will be used to access the domain (URL) and the SSL certificate is signed for same name.

 zmprov cd example.com zimbraVirtualHostName "mail.example.com" zimbraVirtualIPAddress "1.2.3.4"

NOTE: If the server is behind firewall and NAT'ed with external address, make sure the external requests for "mail.example.com" hits the aliased IP address and not the actual local IP of server.

Verifying and Preparing the Certificates

We have three files received from the CA. The server (domain) certificate, two chain certs. And we have existing key file (which was used to generate the csr)

1. Save the example.com certificate, key and chain files to a directory /tmp/example.com. You can receive single or multiple chain certs from your CA. Here we have two chain certs from the CA. i.e. example.com.root.crt and example.com.intermediate.crt.

 ls /tmp/example.com
 example.com.key
 example.com.crt
 example.com.root.crt
 example.com.intermediate.crt

2. Add the chain certs to a single file called example.com_ca.crt

 cat example.com.root.crt example.com.intermediate.crt >> example.com_ca.crt

3. Confirm if the key and certificate matches and chain certs completes the trust.

 /opt/zimbra/bin/zmcertmgr verifycrt comm /tmp/example.com/example.com.key /tmp/example.com/example.com.crt /tmp/example.com/example.com_ca.crt
  • Check the output, it should say something like this. If not, make sure you have correct key and chain cert files.
 ** Verifying example.com.crt against example.com.key
 Certificate (example.com.crt) and private key (example.com.key) match.
 Valid Certificate: example.com.crt: OK

Deploying the Certificate on domain

1. Add the domain certificate and chain files to a single file called example.com.bundle

 cat example.com.crt example.com_ca.crt >> example.com.bundle

2. Run following to save the certificates and key in ldap database.

 /opt/zimbra/libexec/zmdomaincertmgr savecrt example.com example.com.bundle example.com.key
  • The syntax is:
 /opt/zimbra/libexec/zmdomaincertmgr savecrt <domainname> <certificate with chain certs> <keyfile>

3. Run following to deploy the domain certificate. This will save the certificate and key as /opt/zimbra/conf/domaincerts/example.com

 /opt/zimbra/libexec/zmdomaincertmgr deploycrts

4. Make sure the example.com is resolving to its local IP address from Zimbra host. Or make an similar entry in /etc/hosts file.

 1.2.3.4      example.com

5. Restart proxy service to take the changes in effect.

 zmproxyctl restart

6. Once the restart is successfull, try to access the domain using the URL which is set in "zimbraVirtualHostName" over https. And check the certificate loaded in the browser. In this case the URL will be https://example.com

Testing

Run this command locally on zimbra server to check if the correct domain cert is offered while accessing the domain with "zimbraVirtualHostName" or "zimbraVirtualIPAddress"

 openssl s_client -connect example.com:443
 openssl s_client -connect 1.2.3.4:443

Troubleshooting

  • If you do not see domain cert by accessing the domain with its zimbraVirtualHostName (example.com). Make sure the https connection from Internet/intranet is going to server's local IP address which is defined in zimbraVirtualIPAddress.
  • If the proxy startup gives following error, try to change the order of certificates in /opt/zimbra/conf/domaincerts/example.com.crt file and restart proxy.
 Starting nginx...nginx: [emerg] SSL_CTX_use_PrivateKey_file("/opt/zimbra/conf/domaincerts/example.com.key") failed (SSL: error:0B080074:x509 certificate 
 routines:X509_check_private_key:key values mismatch)
  • If you are using multiple proxy servers or adding new proxy servers, make sure you copy all the contents of /opt/zimbra/conf/domaincerts/ among all proxy servers. Otherwise proxy service will fail to start.
Jump to: navigation, search