Ajcody-Proxy-Notes

Revision as of 22:29, 17 December 2014 by Kunal Pariani (talk | contribs) (Doing changes suggested as per https://bugzilla.zimbra.com/show_bug.cgi?id=80135#c32)

Attention.png - This article is NOT official Zimbra documentation. It is a user contribution and may include unsupported customizations, references, suggestions, or information.

A Proxy On You!

Actual Proxy Notes Homepage

Please see Ajcody-Proxy-Notes

Very Rough Draft For Proxy Trouble Shooting Script - More of steam of consciousness really

Do the following on one of your ZCS servers:

$ su - zimbra
$ touch /tmp/proxy_support.sh
$ chmod +x /tmp/proxy_support.sh

Now just vi /tmp/proxy_support.sh again and cut and paste the contents below and save the file. Make sure your not getting odd line wrapping. If you do, try clicking on the view source option at the top of the wiki page and see if that allows your paste into your shell/vi to work better. Note - I removed the leading spaces one would normally use to make the script an easier read since it causes a paste to be a horrible format.

To Do Items: Have a table formatted output, like "Variable Name | Global | ServerA | ServerB | Server C | Server D | Server E | Description of Var | Recommendations" and to include a * next to a variable if we know it should be set to something else when compared to other vars in the environment.

#!/bin/bash

printf "\nProxy Trouble Shooting Script For ZCS 8+ Servers\n\n"
printf "\n\n Reference: https://wiki.zimbra.com/wiki/Ajcody-Proxy-Notes#Very_Rough_Draft_For_Proxy_Trouble_Shooting_Script_-_More_of_steam_of_consciousness_really.\n\n"
### Global Configurations

printf "\n\n## Getting Some Global Variables ##\n\n"
printf "## GARPU [getAllReverseProxyURLs] - Used to publish into nginx.conf what servers should be used for reverse proxy lookup.\n"
printf "## Output from zmprov garpu ##\n\n `zmprov garpu` \n\n"
printf "## GARPB [getAllReverseProxyBackends]  - no description at this time.\n"
printf "## Output from zmprov garpb ##\n\n `zmprov garpb` \n\n"
printf "## GARPD [getAllReverseProxyDomains] - no description at this time.\n"
printf "Output from zmprov -l garpd ##\n\n `zmprov -l garpd` \n\n"
printf "## GAMCS [getAllMemcachedServers] - Used to list memcached servers (for nginix use).\n"
printf "Ouput from zmprov gamcs ##\n\n `zmprov gamcs` \n\n" ;
printf "## Some Global Configs from the following command: ##\n"
printf "## [zimbra@]$ zmprov gcf | egrep 'zimbraWebClientLogoutURL|zimbraWebClientLoginURL|zimbraMailURL|zimbraMailReferMode|zimbraReverseProxySSLToUpstreamEnabled|zimbraMailProxyReconnectTimeout|zimbraMailProxyMaxFails' \n\n"
zmprov gacf | egrep 'zimbraWebClientLogoutURL|zimbraWebClientLoginURL|zimbraMailURL|zimbraMailReferMode|zimbraReverseProxySSLToUpstreamEnabled|zimbraMailProxyReconnectTimeout|zimbraMailProxyMaxFails'
printf "## [For ZCS8+] If zimbraReverseProxySSLToUpstreamEnabled is set to TRUE then zimbraMailMode on the server should NOT be set to http. It should be either https or both. \n\n"
printf "## For details on zimbraMailProxyReconnectTimeout & zimbraMailProxyMaxFails , see https://bugzilla.zimbra.com/show_bug.cgi?id=80135 . Recommendation [ZCS8+] is to set to 0 , when encountering 502/504 timeout errors. \n\n"
printf "\n\n Continuing To The Specific Server Variables...\n\n"
zmlicense -p

### Server Configuration

for server_hostname in `zmprov gas`
do printf "## Need IP Information For Server $server_hostname . Using /etc/resolv.conf on local server to resolve.##\n\n"
printf "`host $server_hostname` \n\n"
printf "## Checking Enabled Services For Server $server_hostname ##\n\n"
zmprov gs $server_hostname | grep ServiceEnabled
printf "\n## Checking Running Services For $server_hostname ##\n\n"
zmcontrol -H $server_hostname status
printf "\n\n## Checking for Some Proxy Variables On $server_hostname ##\n\n"
zmprov gs $server_hostname | egrep 'zimbraReverseProxyHttpEnabled|zimbraReverseProxyMailEnabled|zimbraMailReferMode|zimbraMailMode|zimbraReverseProxySSLToUpstreamEnabled|zimbraMailProxyReconnectTimeout|zimbraMailProxyMaxFails'
printf "## [For ZCS8+] If zimbraReverseProxySSLToUpstreamEnabled is set to TRUE then zimbraMailMode on the server should NOT be set to http. It should be either https or both. \n\n"
printf "## For details on zimbraMailProxyReconnectTimeout & zimbraMailProxyMaxFails , see https://bugzilla.zimbra.com/show_bug.cgi?id=80135 . Recommendation [ZCS8+] is to set to 0 , when encountering 502/504 timeout errors. \n\n"
printf "\n\n## Checking for Some Proxy Port Variables On $server_hostname ##\n\n"
zmprov gs $server_hostname | grep Port
printf "\n\n## Checking for zimbraReverseProxyLookupTarget ; if TRUE then you should be able to telnet to port 7072 on that server. ##\n"
printf "## telnet $server_hostname:7072 \n"
printf "## If $server_hostname shouldn't be a part of the lookup pool. You can remove it by doing:\n"
printf "## [zimbra@]$ zmprov ms $server_hostname zimbraReverseProxyLookupTarget FALSE \n"
printf "## Then check the output of zmprov garpu:\n"
printf "## [zimbra@]$ zmprov garpu \n"
printf "## Then rebuild the configs on all the PROXIES and restart (one at a time is fine):\n"
printf "## [zimbra@]$ zmnginxctl restart \n"
printf "## After doing this, check that $server_hostname is no longer listed in any proxy configs on all proxies:\n"
printf "## [zimbra@]$ grep $server_hostname /opt/zimbra/conf/nginx/includes/*\n"
printf "## $server_hostname Currently has zimbraReverseProxyLookupTarget set to:\n\n"
zmprov gs $server_hostname | grep zimbraReverseProxyLookupTarget
printf "\n\n Finished Getting Variables for $server_hostname.\n\n"
done

### Domain Configuration

for domain in `zmprov gad`
do printf "## Getting Some Domain Information.##\n\n"
printf "Getting Information for $domain \n\n"
printf "## Getting zimbraVirtualHostName And zimbraVirtualIPAddress Information for $domain ##\n\n"
zmprov gd $domain | grep zimbraVirtual
printf "## Getting zimbraPublicServiceHostname, zimbraPublicServicePort And zimbraPublicServiceProtocol for $domain ##\n\n"
zmprov gd $domain | grep zimbraPublicService
printf "\n\n Finished Getting Variables for $domain.\n\n"
done
printf "\n\n Script Complete.\n\n"

Not Finished Part

Not Finished - Need to now adjust how to handle non-zcs commands to be executed on remote hosts . I would like to be able to add to this script but can't with a default multi-server ZCS environment.

Notes on limitations:

  • The below isn't effective for what we need because of the limitation noted after it.
    • ssh -i .ssh/zimbra_identity -o strictHostKeyChecking=no zimbra@server.domain.com -p 22
  • cat /opt/zimbra/.ssh/authorized_keys will give you something like:
    • command="/opt/zimbra/libexec/zmrcd" ssh-dss AAAAB3NzaC1kc3MAA [cut] A== zcs806.us.zimbralab.com
  • zmrcd restricts what commands can be ran, cat /opt/zimbra/libexec/zmrcd to see.
## cut - ifconfig output instead of using host $server_hostname
for server_hostname in `zmprov gas`
do printf "## Need IP Information For Server $server_hostname ##\n\n"
printf "`ifconfig -a` \n\n"
## cut
printf "\n\n## Checking for Some Proxy Port Variables On $server_hostname ##\n\n"
zmprov gs $server_hostname | grep Port
printf "\n\n## Confirmig Ports Are Available ## \n\n"
ZCSPXYPORTS=\'`zmprov gs $server_hostname | grep Port | cut -d':' -f2 | tr '\n' '|' | tr -d ' ' | xargs echo`\'
netstat -plnt | egrep `echo $ZCSPXYPORTS`
## cut

UPDATE PLEASE SEE THESE FIRST

No Route To Host

[working draft]

If your getting "No route to host" errors in /opt/zimbra/log/nginx.log files on the proxy servers, you should check:

  1. The resolution [DNS] to the host from the proxy servers is correct and working. This might happen more when you've deployed a new ZCS server in your environment.
  2. Following step one, you also have to confirm there is port level access to between proxy servers and the server it's trying to reach. memcache port is 11211 .
  3. The server/s are too busy to serve the request.
  4. You have a server in the list that shouldn't be in the lookup pool or you should remove the trouble server from the pool to avoid any more customer issues to deescalate the situation.

50X Errors

  • 500 Internal Server Error
    • A generic error message, given when an unexpected condition was encountered and no more specific message is suitable.
  • 502 Bad Gateway
    • The server was acting as a gateway or proxy and received an invalid response from the upstream server.
  • 503 Service Unavailable
    • The server is currently unavailable (because it is overloaded or down for maintenance). Generally, this is a temporary state. Sometimes, this can be permanent as well on test servers.
  • 504 Gateway Timeout
    • The server was acting as a gateway or proxy and did not receive a timely response from the upstream server.

The following configuration changes on mailbox servers can improve proxy behavior related to timeouts:

  • This will configure proxy to immediately reconnect to all upstream mailbox servers on any failure
    • $ zmprov mcf zimbraMailProxyReconnectTimeout 0 
    • If necessary, on each mailbox server
      • $ zmprov ms `zmhostname` zimbraMailProxyReconnectTimeout 0 
  • This will configure proxy to ignore failures in regards to disconnects for all upstream mailbox servers
    • $ zmprov mcf zimbraMailProxyMaxFails 0 
    • If necessary, on each mailbox server
      • $ zmprov ms `zmhostname` zimbraMailProxyMaxFails 0 
  • Then, restart mailboxd on all servers:
  •  $ zmmailboxdctl restart 

See the following bug for more details on this recommendation:

Admin Guide Update

Our Administration Guide now has everything you should need to setup your proxy server.

Please see:

How To Enable Proxy

And Also Another More Current How-To Enable Proxy here:

My Own Proxy How-To For Setup

Note: We are working on improving the official documentation. I haven't seen the document yet for the 5.0.11 release, but I've been told it will have many improvements.

Here's my attempt at writing something better in the meantime.

High Level Overview

One Package

First, there's only ONE package or rpm in the installer for "proxy".

Where Does It Get Installed?

It can be installed with any of your Zimbra servers (a complete ZCS install, ZCS MTA, ZCS LDAP, ZCS Mailstore) or on its own server. ( Reference )

Other proxy documentation might give the impression it needs to go onto a zimbra server that is running some other function - mta, ldap, or mailstore.

Three Components To The Package

This package, can be broken down into THREE components.

Nginx

Nginx is a high performance IMAP/POP3 proxy server which handles all incoming POP/IMAP requests. In addition to this, it is also able to reverse proxy HTTP requests to the right backend (mailstore) server. This helps to hide the hostnames of the backend mailbox servers from end users. The steps involved in this lookup are described in more detail under the Flow_Process_For_Proxy_Target_Lookups below.

Memcached

Memcached is a high performance, distributed memory object caching system (an open source distributed in-memory hashtable). Route information is cached for further use in order to increase performance.

Memcache does not have authentication and security features so the servers should have a firewall set up appropriately. The default port is 11211 and is controlled by zimbraMemcacheBindPort conf setting in zimbraserver.

Note: Memcached will be split into its own service in the future. Meaning, for example, that you'll be able to configure a server to solely perform this function of providing the cache results. This will give more flexibility in managing performance issues, as related to the proxy environment. Rather than the same server having to share cpu/memory resources with nginx. It will also allow you to configure your different servers running nginx to quiry against a "shared" cached table of memcached. Currently, each nginx will have unique results in their memcached tables.

What is being cached, are the prior results of pop and imap lookups of users and their corresponding mailbox server they should be routed to. The process is described below in Flow_Process_For_Proxy_Target_Lookups.

Zimbra Proxy Route Lookup Handler

Zimbra Proxy Route Lookup Handler is a servlet located on the ZCS mailbox servers. This servlet handles queries for the user account route information (the server and port number where the user account resides). This is more about a difference in functionality when the proxy package is being used rather than a "new" software package being installed. The functionality difference is coming from the "software" related to nginx and memcached. The process is described in more detail in Flow_Process_For_Proxy_Target_Lookups.

NOTE - Again, don't let the reference to "mailbox server" confuse you. The mailbox servers don't need the proxy package installed on them directly to see this effect. The proxy server (whatever one you install it on, will make configuration changes global to handle this function. The referenced servlet is already on a base mailbox server install. This is referring to how the proxy package components will or can interact with that pre-existing servlet on the mailbox server.
Flow Process For Proxy Target Lookups

Here's the flow of the process for pop and imap lookups:

  1. Nginx will query the LOCAL memcached service first to see if the lookup result has a cached result. [FYI - Memcache runs over port 11211]
    • Note: when memcached is allowed to run on another server besides the one where nginx is at, you'll need to account for this port 11211 access between the servers. For now, it's all occurring locally to the server. If you can't access this memcache process, confirm you don't have any firewall/port level blocking occur for the local machine communicating over port 11211.
  2. If the initial lookups by nginx against the LOCAL memcached service have no results, nginx will then proceed to query against the "Zimbra Proxy Route Lookup Handler" services.
    • By default, all mailbox servers participate in this lookup process via the "Zimbra Proxy Route Lookup Handler".
      • This "where" to lookup is determined by the zimbraReverseProxyLookupTarget server attribute on servers running the mailbox service.
    • The lookups are performed in a round-robin fashion across configured mailbox servers.
    • Now that there is a successful lookup, the results are then cached in LOCAL memcached service and future lookups will bypass the "Zimbra Proxy Route Lookup Handler" steps for that specific lookup.
      • Meaning, the lookups will solely operate between nginx and LOCAL memcached service and no longer involve the "Zimbra Proxy Route Lookup Handler"/mailbox stores.

Here's the flow of the process for reverse proxy http lookups:

  1. Nginx will query against the "Zimbra Proxy Route Lookup Handler" services.
    • By default, all mailbox servers participate in this lookup process via the "Zimbra Proxy Route Lookup Handler".
      • This "where" to lookup is determined by the zimbraReverseProxyLookupTarget server attribute on servers running the mailbox service.
    • The lookups are performed in a round-robin fashion across configured mailbox servers.
    • Ngninx will perform this lookup for every http request, it does not involve the memcached server for cache results.
Three Functions That Can Be Done With Proxy Package

There are THREE functions or options within Zimbra that relate to "proxy".

IMAP - POP Proxy

IMAP/POP Proxy is about the use of an IMAP/POP proxy server routes users of these services to the Zimbra mailbox server on which their mailbox resides.

For example, proxying allows users to enter imap.example.com as their IMAP server. The proxy running on imap.example.com inspects their IMAP traffic, does a lookup to determine which backend mailbox server a user’s mailbox lives on (mbs1.example.com, for example), and transparently proxies the connection from user’s IMAP client to the correct mailbox server.

Flow Of The Login Process: When an IMAP or POP3 client logs in through the proxy, the following takes place:

  1. The proxy analyzes the login sequence
  2. Extracts the user name of the user trying to login
  3. Does a HTTP lookup on a mailbox server to find out which server the mailbox of the user attempting to login lives on
    • This lookup service runs on mailbox servers on port 7072, and this port on mailbox servers should be available from all proxy servers.
      • Which mailbox servers participate in this lookup is determined by the zimbraReverseProxyLookupTarget server attribute on servers running the mailbox service.
        • By default all mailbox servers participate in this lookup.
      • Lookup is performed round-robin across configured mailbox servers.
      • The result of the login name to mailbox server lookup are cached in memcached (an open source distributed in-memory hashtable).
        • The memcached process is run alongside all IMAP/POP proxy services (nginx). Meaning wherever the proxy package was install and is configured to run.
  4. Also see Flow_Process_For_Proxy_Target_Lookups, if you have not.

Port Changes For IMAP/POP Proxying: When the proxy server is configured, the service ports on backend Zimbra mailbox server are changed to alternate ports. The proxy now services the standard ports for these protocols. This change is applied even if the proxy services are run on their own independent hosts, in order to distinguish and avoid confusion between the services.

  1. Here's the changes that will be made on the mailbox servers and what ports will now be handled by the proxy server(s).
    • Note: If you have any other services running on these ports, turn them off.
      • Standard Ports now server or handled by Proxy server(s)
        • IMAP proxy port 143
        • IMAP SSL proxy port 993
        • POP proxy port 110
        • POP proxy port 995
      • Mailbox Servers will now work on these alternate ports
        • IMAP port 7072
        • IMAP SSL port 7143
        • POP port 7110
        • POP SSL port 7995
        • Route Lookup Handler (new port) 7072
        • This was described above in regards to the "Flow Of The Login Process".
Reverse HTTP Proxy
  1. Reverse HTTP Proxy
    • Using an nginx-based reverse proxy for HTTP helps to hide names of backend mailbox servers from end users. For example, users can always use their web browser to visit the proxy server at http://mail.example.com. The connection from users whose mailbox lives on mbs1. example.com will be proxied to mbs1.example.com by the proxy running on the mail.example.com server. In addition to the ZCS web interface, clients such as REST and CalDAV clients, Zimbra Connector for Outlook and Zimbra Mobile Sync devices are also supported by the proxy.
      • HTTP reverse proxy routes requests as follows:
        1. If the request has an auth token cookie (ZM_AUTH_TOKEN), the request is routed to the backend mailbox server of the authenticated user.
        2. If the requesting URL can be examined to determine the user name, then the request is routed to the backend mailbox server of the user in the URL. REST, Ca lDAV, and Zimbra Mobile Sync are supported through this mechanism.
        3. If the above methods do not work, the IP hash method is used to load balance the requests across the backend mailbox servers which are able to handle the request or do any necessary internal proxying.
Virtual Hosting

Please see Ajcody-Server-Misc-Topics#Proxy_.5BNGINX.5D_Virtual_Hosting

Proxy References

NGINX Open-Source Project Homepage

Proxy documentation BUG/RFE's:

From Install Guide':

From Wiki:

zmproxyinit Help Output [Oct 16, 2008 - From 5.0.9 I believe]:

[zimbra@mail2 libexec]$ ./zmproxyinit 
Usage: ./zmproxyinit [-h] [-o] [-m] [-w] [-d [-r] [-s] [-a w1:w2:w3:w4] [-i p1:p2:p3:p4] [-p p1:p2:p3:p4] [-x mailmode]] 
[-e [-a w1:w2:w3:w4] [-i p1:p2:p3:p4] [-p p1:p2:p3:p4] [-x mailmode]] [-f] -H hostname
        -h: display this help message
        -H: Hostname of server on which enable/disable proxy functionality.
        -a: Colon separated list of Web ports to use. Format: HTTP-STORE:HTTP-PROXY:HTTPS-STORE:HTTPS-PROXY (Ex: 8080:80:
8443:443)
        -d: disable proxy
        -e: enable proxy
        -f: Full reset on memcached port and search queries and POP/IMAP throttling.
        -i: Colon separated list of IMAP ports to use. Format: IMAP-STORE:IMAP-PROXY:IMAPS-STORE:IMAPS-PROXY (Ex: 7143:14
3:7993:993)
        -m: Toggle mail proxy portions
        -o: Override enabled checks
        -p: Colon separated list of POP ports to use. Format: POP-STORE:POP-PROXY:POPS-STORE:POPS-PROXY (Ex: 7110:110:799
5:995)
        -r: Disable reverse proxy lookup target for store server.  Only valid with -d.  Be sure that you intend for all p
roxy function for the server to be disabled
        -s: Set cleartext to FALSE (secure mode) on disable
        -w: Toggle Web proxy portions
        -x: zimbraMailMode to use on disable (Default: http)
hostname is the value of the zimbra_server_hostname LC key for the server being modified.
Required options are -f by itself, or -f with -d or -e.
Note that -d or -e require one or both of -m and -w.
Note that -i or -p require -m.
Note that -a requires -w.
Note that -x requires -w and -d for store.
Note that -x requires -w for proxy.
The following are the defaults for -a, -i, -p, and -x if they are not supplied as options.
-a default on enable: 8080:80:8443:443
-a default on disable: 80:0:443:0
-i default on enable: 7143:143:7993:993
-i default on disable: 143:7143:993:7993
-p default on enable: 7110:110:7995:995
-p default on disable: 110:7110:995:7995
-x default on store disable: http
-x default on proxy enable/disable: http

What's Wrong?

Did you check in /opt/zimbra/log/nginx.log ?

Notable Proxy Bugs

Unresolved At This Time (Sept 25, 2008)

Unsorted

  • "ZCS Mailbox Server Proxy"
    • See: http://bugzilla.zimbra.com/show_bug.cgi?id=9469
    • From release notes, "HTTP proxy is supported (Beta). This includes REST sharing, Caldav, and Zimbra mobile. For Zimbra Mobile, ZCS redirect/forwards to the correct host. (Bug 9469)"
      • Please note, that ZCS HTTP Proxy (http reverse proxy) is Beta, first released in 5.0.6.
    • Though, as of 5.0.8,""For activesync connection to go thru the proxy, they'll have to configure reversed proxy. However, there're already a few known bugs about sharing broken with reversed proxy enabled. They're better off waiting for 5.0.9 than to try to configure reversed proxy now."
  • "need to support multi-server (proxy) for mobile clients"

Resolved In 5.0.9

Those That Are Sort Of Client Side Impacting
Configuration / Initial Setup Issues

Resolved in 5.0.7

Resolved in 5.0.6

Public Service Hostname

Resolved in 5.0.5

  • Nginx: http proxy does not work with REST
    • Note, this isn't "completely" resolved, as you'll notice by other targeted bugs with 5.0.5+ releases.
    • See: Nginx: http proxy does not work with REST

Single ZCS Server

Need To Disable Pop/Imap Proxy And Use POP/IMAP Normally

Sometimes, people install/setup proxy services on their single ZCS server and they don't need them. Here's how you would disable the proxy stuff and get imap/pop working over the default ports.

do a zmprov -l gs `zmhostname` | grep -i port

get the ports, then set variables to port 0:

zmprov ms `zmhostname` zimbraImapProxyBindPort 0
zmprov ms `zmhostname` zimbraImapSSLProxyBindPort 0
zmprov ms `zmhostname` zimbraPop3ProxyBindPort 0
zmprov ms `zmhostname` zimbraPop3SSLProxyBindPort 0

then, set the non "Proxy" ports to the desired standard ports

zmprov ms `zmhostname` zimbraImapBindPort 143
zmprov ms `zmhostname` zimbraImapSSLBindPort 993
zmprov ms `zmhostname` zimbraPop3BindPort 110
zmprov ms `zmhostname` zimbraPop3SSLBindPort 995

once complete:

zmprov ms `zmhostname` -zimbraServiceEnabled memcached
zmprov ms `zmhostname` -zimbraServiceEnabled imapproxy

zmproxyctl stop
zmmemcachedctl stop
zmmailboxdctl stop
zmmailboxdctl start 

Proxy Guide Rewrite Scratch Pad

Intro

Moved to Ajcody-Proxy-Guide-Rewrite-Project

Jump to: navigation, search