Ajcody-Proxy-Guide-Rewrite-Project

Overview And Planning For Zimbra Proxy

Moved to:

Installing , Configuring, Disabling the Zimbra Proxy

Moved To:

Troubleshooting Zimbra Proxy

Source: http://wiki.zimbra.com/wiki/Zimbra_Proxy_Guide#Troubleshooting

Proxy Related Log Files

Source: http://wiki.zimbra.com/wiki/Zimbra_Proxy_Guide#Troubleshooting

Proxy related errors can be found in the following logs on each proxy server:

  • /opt/zimbra/log/nginx.log
  • /opt/zimbra/log/nginx.access.log

Slow Proxy Logins - Route Lookup Handles (garpu)

Source:

A common nginx misconfiguration is to have incorrectly designated non-mailbox servers as routing/zmlookup handlers. Only mailbox servers can perform route handler functions. To view the zmlookup lookup handlers, review the zm_lookup_handlers parameter in /opt/zimbra/conf/nginx/includes/nginx.conf.zmlookup 

grep zm_lookup_handlers /opt/zimbra/conf/nginx/includes/nginx.conf.zmlookup

If a non-mailbox server is listed, set the zimbraReverseProxyLookupTarget server configuration attribute to FALSE for that server. 

zmprov ms `zmhostname` zimbraReverseProxyLookupTarget FALSE

Additionally, zimbraReverseProxyLookupTarget is a server inherited attribute from the global configuration, so check if zimbraReverseProxyLookupTarget has been incorrectly designated in global config. 

zmprov gcf zimbraReverseProxyLookupTarget

No Route To Host Errors

Source:

If your getting "No route to host" errors in /opt/zimbra/log/nginx.log files on the proxy servers, you should check:

  1. If the resolution [DNS] to the host from the proxy servers is correct and working. This oversight might when you've deployed a new ZCS server in your environment.
  2. Following step one, you also have to confirm there is port level access between proxy servers and the server it's trying to reach. memcache port is 11211 .
  3. The server/s [mailbox/webapp servers] are too busy to serve the request.
  4. You have a server in the lookup pool list that shouldn't be there or you should remove the trouble server from the pool to avoid any more customer issues to deescalate the situation.

5xx Errors

Source:

  • 500 Internal Server Error:
    • A generic error message, given when an unexpected condition was encountered and no more specific message is suitable. This might be due to several reasons due to failure on one of the upstream mailstore servers.
  • 502 Bad Gateway:
    • The server was acting as a gateway or proxy and received an invalid response from the upstream server.
  • 503 Service Unavailable:
    • The most common reason for this is that the jetty mailboxd process is down on the mailstore server and hence is unable to process the request. Check using 'netstat -an | grep <port>' to see if the IMAP/POP/HTTP ports used by mailboxd are up and listening for incoming connections. Also check using 'ps -eaf | grep mailboxd'
  • 504 Gateway Timeout:
    • The most common reason for this is that the upstream mailstore server took longer than the configured timeout value and hence the client closed the upstream connection. Try changing the timeout values to a higher value if its expected to take longer for specific kind of requests. If this is not the case, then we should look for reasons why it took longer for a request to get processed.

These proxy configuration changes can improve proxy behavior related to timeouts:

  • This will configure proxy to immediately reconnect on any failure 
    • $ zmprov mcf zimbraMailProxyReconnectTimeout 0
    • If necessary, on each proxy 
      • $ zmprov ms `zmhostname` zimbraMailProxyReconnectTimeout 0
  • This will configure proxy to ignore failures in regards to disconnects 
    • $ zmprov mcf zimbraMailProxyMaxFails 0
    • If necessary, on each proxy 
      • $ zmprov ms `zmhostname` zimbraMailProxyMaxFails 0
  • Then, restart the processes on all proxies:
  •  $ zmproxyctl restart

See the following bug for more details on this recommendation:

Third Party Issues - Load Balancers, DNS, Etc.

Bad/Invalid command when proxying to external POP/IMAP servers

Source : http://wiki.zimbra.com/index.php?title=NGINX_Configuration_Tips#Bad.2FInvalid_command_when_proxying_to_external_POP.2FIMAP_servers


Nginx issues the XOIP command to the upstream POP3 server, and the ID command to the upstream IMAP server, before logging in to upstream. This is for auditing purposes so that the client's IP address is known to the upstream server. The global config attributes zimbraReverseProxySendPop3Xoip and zimbraReverseProxySendImapId control this aspect.

However, some external IMAP servers may not implement the ID command, and some external POP3 servers may not implement the XOIP command.

To turn off sending the XOIP command, set zimbraReverseProxySendPop3Xoip to false. To turn off sending the IMAP command, set zimbraReverseProxySendImapId to false.

zmprov mcf zimbraReverseProxySendPop3Xoip FALSE
zmprov mcf zimbraReverseProxySendImapId FALSE

LDAP/Nginx Won't Start And Asks For A Password

Source: http://wiki.zimbra.com/wiki/LDAP/Nginx_won%27t_start_and_asks_for_a_password

This tends to happen because Zimbra can't read the key or the certificate of the LDAP/IMAP proxy service on startup. This issue is specific to commercial certificates.

Here is what you can do:

  • You want to check to make sure the private key is not encrypted.
  • Manually link the cert to the services OR simply redeploy the certificate to relink the cert to the services.
cp /opt/zimbra/ssl/zimbra/commercial/commercial.key /opt/zimbra/conf/slapd.key

cp /opt/zimbra/ssl/zimbra/commercial/commercial.crt /opt/zimbra/conf/slapd.crt

cp /opt/zimbra/ssl/zimbra/commercial/commercial.key /opt/zimbra/conf/nginx.key

cp /opt/zimbra/ssl/zimbra/commercial/commercial.crt /opt/zimbra/conf/nginx.crt
  • Restart the Zimbra services.

Advance Topics For Zimbra Proxy - Miscellaneous Topics

Miscellaneous Topics

Set Up Proxy to use Clear Text for Upstream Connections

Source : Admin Guide Draft, 'Set Up Proxy to use Clear Text for Upstream Connections'

When setting up the proxy to use clear text for upstream connections, set zimbraReverseProxySSLToUpstreamEnabled to FALSE.

This attribute defaults to TRUE. In an "out of the box" proxy set up, the upstream communication defaults to SSL.

REST URL Generation

Source : Admin Guide Draft, 'REST URL Generation'

For REST URL, you set the host name, service protocol, and services port globally or for a specific domain from the following attributes.

  • zimbraPublicServiceHostname
  • zimbraPublicServiceProtocol
  • zimbraPublicServicePort

When generating REST URL’s:

  • If domain.zimbraPublicServiceHostname is set, use zimbraPublicServiceProtocol + zimbraPublicServiceHostname + zimbraPublicServicePort
  • Otherwise it falls back to the server (account's home server) attributes:
    • protocol is computed from server.zimbraMailMode
    • hostname is server.zimbraServiceHostname
    • port is computed from the protocol.

Note: Why use zimbraMailReferMode - In earlier versions, a local config variable called zimbra_auth_always_send_refer determined which action the back-end server took when a user’s mailbox did not reside on the server that the user logged in to. The default value of FALSE redirected the user if the user was logging in on the wrong backend host.

On a multiserver ZCS, if a load balanced name was needed to create a friendly landing page, a user would always have to be redirected. In that case, zimbra_auth_always_send_refer was set to TRUE.

Now with a full-fledged reverse proxy, users do not need to be redirected. The localconfig variable zimbraMailReferMode is used with nginx reverse proxy.

Set Proxy Trusted IP Addresses

Source : Admin Guide Draft, 'Set Proxy Trusted IP Addresses'

When a proxy is configured with ZCS, each proxy server’s IP address must be configured in LDAP attribute zimbraMailTrustedIP to identify the proxy addresses as trusted when users log in through the proxy. The proxy IP address is added to the X-Forwarded-For header information. The X-Forwarded-For header is automatically added to the localconfig zimbra_http_originating_ip_header attribute. When a user logs in, this IP address and the user’s address are verified in the Zimbra mailbox log.

Set each proxy IP address in the attribute. For example, if you have two proxy servers: zmprov mcf +zimbraMailTrustedIP {IP of nginx-1} +zimbraMailTrustedIP {IP of nginx-2}

Note: To verify that X-Forwarded-For was correctly added to the localconfig, type zmlocalconfig | grep -i http. You should see zimbra_http originating_ip_header = X-Forwarded-For.

Zimbra Proxy Performance Tuning

See the following unresolved RFE:

Zimbra Proxy Related CLI Commands

zmproxyconfig

Source: http://wiki.zimbra.com/wiki/Zimbra_Proxy_Guide#Proxy_config_rewrite.28zmproxyconfgen.29.2C_zmproxyconfig_and_zmproxyctl

Syntax And Usage

/opt/zimbra/libexec/zmproxyconfig help

Usage: /opt/zimbra/libexec/zmproxyconfig [-h] [-o] [-m] [-w] [-d [-r] [-s] [-a w1:w2:w3:w4] [-c [-n n1:n2]] [-i p1:p2:p3:p4] [-p p1:p2:p3:p4] [-x mailmode]] [-e [-a w1:w2:w3:w4] -C] [-n n1:n2 [-i p1:p2:p3:p4] [-p p1:p2:p3:p4] [-u|-U] [-x mailmode]] [-f] -H hostname

Description

-h: display this help message

-H: Hostname of server on which enable/disable proxy functionality.

-a: Colon separated list of Web ports to use. Format: HTTP-STORE:HTTP-PROXY:HTTPS-STORE:HTTPS-PROXY (Ex: 8080:80:8443:443)

-d: disable proxy

-e: enable proxy

-f: Full reset on memcached port and search queries and POP/IMAP throttling.

-i: Colon separated list of IMAP ports to use. Format: IMAP-STORE:IMAP-PROXY:IMAPS-STORE:IMAPS-PROXY (Ex: 7143:143:7993:993)

-m: Toggle mail proxy portions

-o: Override enabled checks

-p: Colon separated list of POP ports to use. Format: POP-STORE:POP-PROXY:POPS-STORE:POPS-PROXY (Ex: 7110:110:7995:995)

-r: Run against a remote host. Note that this requires the server to be properly configured in the LDAP master.

-s: Set cleartext to FALSE (secure mode) on disable

-t: Disable reverse proxy lookup target for store server. Only valid with -d. Be sure that you intend for all proxy function for the server to be disabled

-w: Toggle Web proxy portions

-c: Disable Admin Console proxy portions.

-C: Enable Admin Console proxy portions.

-n: Colon separated list of Admin Console ports to use. Format: ADMIN-CONSOLE-STORE:ADMIN-CONSOLE-PROXY (Ex: 7071:9071)

-x: the proxy mail mode when enable proxy, or the store mail mode when disable proxy (Both default: http).

-u: disable SSL connection from proxy to mail store.

-U: enable SSL connection from proxy to mail store.

hostname is the value of the zimbra_server_hostname LC key for the server being modified.

Required options are -f by itself, or -f with -d or -e.

Note that -d or -e require one or both of -m and -w.

Note that -i or -p require -m.

Note that -a requires -w.

Note that -c/-C requires -w, and -n requires -c/-C. When disabling web proxy, admin console proxy will be automatically disabled.

Note that -u or -U are only available when proxy is enabled by -e.

Note that -x requires -w and -d for store.

Note that -x requires -w for proxy.

Note that no matter what mail mode is set by -x and no matter proxy is enabled or disabled, admin console's mode is always https.

The following are the defaults for -a, -i, -p, and -x if they are not supplied as options.

-a default on enable: 8080:80:8443:443

-a default on disable: 80:0:443:0

-i default on enable: 7143:143:7993:993

-i default on disable: 143:7143:993:7993

-p default on enable: 7110:110:7995:995

-p default on disable: 110:7110:995:7995

-n default on enable: 7071:9071

-n default on disable: 7071:9071

-x default on store disable: http

-x default on proxy enable/disable: http, but -x default on proxy enable when upstream ssl connection is enabled: https

eg. To change the value of zimbraReverseProxySSLToUpstreamEnabled from TRUE(default) to FALSE (i.e use http for the proxy<->mailstore connections instead of https), execute this on the server running the proxy

/opt/zimbra/libexec/zmproxyconfig -e -m -w -H <hostname> -u

You will have to restart the proxy after this by running 'zmproxyctl restart'

  • Note: zmproxyconfig is no longer required to enable the proxy if you are doing a fresh installation of ZCS with proxy. The installer already runs this and the proxy is enabled for imap/pop/https access by default. Although it is needed while trying to deploy proxy in existing non-proxy environments when using the existing servers. Kindly refer http://wiki.zimbra.com/wiki/Enabling_Zimbra_Proxy#Using_existing_servers

zmproxyctl

Syntax

zmproxyctl help

Usage : /opt/zimbra/bin/zmproxyctl start|stop|restart|reload|status

Description

zmprov

Syntax

zmprov [cmd]

Description

Long Name Short Name Description
--getAllReverseProxyURLs -garpu Used to list all the upstream mailstore servers (NLEs) that should be used for reverse proxy lookup by the proxy
--getAllReverseProxyBackends -garpb Used to list all the upstream mailstore servers that are reverse-proxied by the proxy
--getAllMtaAuthURLs -gamau Used to publish into saslauthd.conf the servers that should be used for saslauthd.conf MTA auth
--getAllMemcachedServers -gamcs Used to list memcached servers (for Zimbra Proxy use)

zmproxyconfgen

Syntax

Description

zmnginxconf

Syntax

Description

Will be functional again with Bug 95169 being fixed.

zmproxypurge

Syntax

Description

zmproxyinit [ deprecated since ???]

Description

This command is no longer used on any supported versions of ZCS. See zmproxyconfig .

Details On Some Of The Zimbra Customization To The Proxy Components

Source : [Internal Eng. Wiki /index.php/Zimbra/Admin/Proxy/Overview#Motivation_behind_Expanding_Nginx ]

Motivation behind Expanding Nginx

Although official Nginx is powerful, it can't meed all the needs of Zimbra.

AUTH_HTTP

Nginx's AUTH_HTTP protocol (described in http://wiki.nginx.org/NginxMailCoreModule#Authentication) is borrowed to implement the lookup with NLE. However, official Nginx's mail AUTH_HTTP config can only specify one NLE URL (auth_http directive). We wish to extend this support so that NZ can use more than one NLE. When a great load of user logins come, NZ can dispatch the load to multiple NLE running in different zimbra servers. This will end up not overloading a single NLE. In a production set-up, it is likely that there will be many different NZ identically configured, and so the relationship between the NZ set and the server set running NLE is a many-to-many relationship.

Besides, the lookup is also necessary for zimbra web accessing, which is not supported by official Nginx. Therefore, this lookup and upstream choose function must also be implemented as an nginx http module.

Finally, because Zimbra doesn't use AUTH_HTTP protocol for real authentication, it might add some custom HTTP headers (such as "Auth-ID") and avoid some useless content transmit (such as password via "Auth-Pass").

All of these above have to be extended by Zimbra.

Memcache Module

It is likely that a single user may log in many times via NZ in a short time, or one http request will generate a great many of sub requests, all of which contains the same login information and thus will get the same route lookup result. In such cases, NZ will have to contact the NLE many times. It is not likely that the upstream server information for a particular client will change frequently. (A change in the upstream server usually corresponds to a user's mailbox being migrated to a different server). And at the same time NLE is relative much slower than Nginx, becoming the bottle neck.

As such, it will be beneficial if NZ is able to cache such information so that repeated trips to the NLE servers are avoided. Also, this cached information must be made available to possibly multiple NZ instances. Hence, "memcached" (http://www.danga.com/memcached/) was introduced for this purpose. Memcached is a high-performance, distributed memory object caching system.

Although official Nginx has a build-in "ngx_http_memcache" module, it can only be used for "URL->web page" cache. Therefore, a global "ngx_memcache" module is implemented in NZ which could be accessed by both web and mail modules. Memcached is used to cache 5 kinds of information as follows. The 4th and 5th are used in mail throttle control, which will be described in [Zimbra/Admin/Proxy/Throttle Control]:

  • alias-->account name
  • account name->route information
  • account id-->route information
  • login IP address->login count (IP throttle control)
  • account name->login count (user throttle control)

Enhanced Mail Protocol Support

Compared to the web proxy part, official nginx is relative weak in mail proxy, such as:

  • Official nginx cannot completely support IMAP ID command, which is required for the logging of upstream zimbra servers
  • Official nginx can't echo the [ALERT] msg before authenticating to the upstream server
  • Official nginx does not support GSSAPI authentications.
  • Official nginx does not support upstream SSL connection, that is, using POP3S and IMAPS protocol connecting nginx and upstream server.

These features are critical for a mail system of enterprise level. ZCS' mail server support these, so NZ has to support them as well.

Unlike web proxy, during which nginx simply proxies all the requests, in mail proxy nginx has to handle some mail protocol commands rather than proxy them to upstream before login is given. The reason is before getting route from NLE, NZ doesn't know to use which upstream server. These commands include "ID", "NOOP", "LOGIN", "CAPABILITY", "AUTHENTICATE", "LOGOUT", "STARTTLS" for IMAP, and "USER", "CAPA", "PASS", "NOOP", "STLS", "QUIT" for POP3. Although official nginx support most of them, but it has several bugs (like IMAP tag/untag responses), which are fixed by NZ.

SMTP proxy os nginx is not used in Zimbra so this part is not touched. Besides, some authentication methods supported by official nginx like SASL LOGIN, SALS CRAM-MD5 and APOP won't work because NLE can't support them.

Throttle Control

To reject DoS like attack, NZ will count the number of logins per IP/username in periods and deny the login if it's beyond a certain configured threshold value. This functionality is called Throttle Control. There are 2 kinds of them: IP Throttle Control and User Throttle Control. IP Throttle Control counts the login by client's IP address. But sometimes, this throttle control is not suitable for some cases (for example, many clients who locate in an internal network and thus have the same outside IP address of the gateway). Therefore, User Throttle Control is also introduced, which counts the login by client's user name. All these counters are stored in memcached so it's not persistent. And its algorithm is not quite accurate, but simple and feasible in practice. It's beneficial to implement Throttle Control in NZ because it can block the attacks before they touch real upstream mail servers.

Currently Throttle Control is only implemented for mail proxy. For web proxy, users can manually configure nginx with "deny" & "allow" directive. The following LDAP attributes have to be set to use this feature which get written using the corresponding nginx directives added for this module in /opt/zimbra/conf/nginx/templates/nginx.conf.mail.template configuration file. 

zimbraReverseProxyIPLoginLimit    ->   mail_login_ip_max               ${mail.ipmax};
zimbraReverseProxyIPLoginLimitTime    ->   mail_login_ip_ttl               ${mail.ipttl};
zimbraReverseProxyIpThrottleMsg    ->   mail_login_ip_rejectmsg         "${mail.iprej}";
zimbraReverseProxyUserLoginLimit    ->   mail_login_user_max             ${mail.usermax};
zimbraReverseProxyUserLoginLimitTime    ->   mail_login_user_ttl             ${mail.userttl};
zimbraReverseProxyUserThrottleMsg    ->   mail_login_user_rejectmsg       "${mail.userrej}";

Cert Per Virtual Host Name

Many ZCS host thousands of domains, each of which has several virtual host names. Each domain may own its own SSL certificate/private key. If nginx is deployed, nginx can't proxy the SSL handshake to the upstream. So NZ has to support this "cert per domain" feature. Actually, the upstream Jetty server doesn't implement this feature. Therefore, NZ is critical in this use case. See the following for SSL Certificates Per Domain Set Up.

Advance Topics For Zimbra Proxy - Configuration And Template Files And Proxy Related Variables

https://wiki.zimbra.com/wiki/Advance_Topics_For_Zimbra_Proxy_-_Configuration_And_Template_Files_And_Proxy_Related_Variables

Merged and Updated the following pages below and then set a REDIRECT to the main page above:


Advance Topics For Zimbra Proxy - Advanced Proxy Configuration Examples via CLI

Created :

Retrieved from "http://wiki.zimbra.com/index.php?title=Ajcody-Proxy-Guide-Rewrite-Project&oldid=55788"
Jump to: navigation, search