DoSFilter

DoSFilter Throttling Mechanism

   KB 20398        Last updated on 2015-10-6  




0.00
(0 votes)

The denial-of-service filter or DoSFilter was added to the mailbox server in ZCS 8.0 to throttle clients sending a large number of requests over a very short period of time. The DoSFilter is applied to all requests for service, mailbox and admin. This feature was added with the completion of bug 66921.

DoS filtering is enabled by default once ZCS 8 is installed. It may be necessary to adjust the configuration to accommodate specific environmental needs. Disabling DoSFilter is not recommended.

Identifying False Positives

It is possible for clients such as Zimbra Connector for Outlook (ZCO), mobile ActiveSync, zmprov, etc. to trigger the DoSFilter. To the client, the Zimbra mailbox service is unavailable. DoSFilter can be identified on the server in the following ways:

/opt/zimbra/log/sync.log

2013-01-15 15:52:20,426 WARN [qtp1635701107-91:https://10.10.0.54:443/Microsoft-Server-ActiveSync?User=zsupport2&DeviceId=Appl5K0113UN3NR&DeviceType=iPhone&Cmd=FolderSync][name=zsupport2@domain.com;mid=64;ip=71.194.89.54;Cmd=FolderSync;DeviceID=Appl5K0113UN3NR;Version=12.1;] sync - Service exception
com.zimbra.common.service.ServiceException: error while proxying request to target server: HTTP/1.1 503 Service Unavailable
ExceptionId:qtp1635701107-91:https://10.10.0.54:443/Microsoft-Server-ActiveSync?User=zsupport2&DeviceId=Appl5K0113UN3NR&DeviceType=iPhone&Cmd=FolderSync:1358286740426:c5ca7f36bb0a038f Code:service.PROXY_ERROR Arg:(url, STR,"http://mail.domain.com:80/service/soap/SyncRequest")

/opt/zimbra/log/zmmailboxd.out

2013-01-15 15:57:32.537:WARN:oejs.DoSFilter:DOS ALERT:ip=127.0.1.1,session=null,user=null

Configuration

The configuration attributes zimbraHttpDosFilterDelayMillis, zimbraHttpDosFilterMaxRequestsPerSec and zimbraHttpThrottleSafeIPs. Each attribute is server inherited from global or is configurable at the sever level. Our recommendation is to preserve the default configuration whenever possible.

DoSFilter Delay (milliseconds) - zimbraHttpDosFilterDelayMillis

Delay imposed on all requests over the rate limit, before they are considered at all. -1 = Reject request, 0 = No delay, any other value = Delay in ms. The default is -1.

To modify in the global configuration; e.g. set the delay to 20ms:

zmprov mcf zimbraHttpDosFilterDelayMillis 20

DoSFilter Maximum Requests Per Second - zimbraHttpDosFilterMaxRequestsPerSec

Maximum number of requests from a connection per second. Requests in excess of this are throttled. The default is 30 and the minimum is 1.

To set the maximum number for requests in the global configuration:

zmprov mcf zimbraHttpDosFilterMaxRequestsPerSec 100

DoSFilter IP Addresses Whitelist - zimbraHttpThrottleSafeIPs

Warning: Zimbra Collaboration 8.5 or 8.6 doesn't support CIDR, so you must add the IPs individually.

IP addresses to ignore when applying Jetty DosFilter. This attribute does not have a default value, however these loopback IPs are whitelisted by default:

  • 127.0.0.1
  • ::1
  • 0:0:0:0:0:0:0:1
  • All mailboxd servers

You can check if these hosts have been correctly whitelisted by the log entry in /opt/zimbra/log/mailbox.log. This log entry should contain all of the default whitelisted hosts as well as any IPs added to zimbraHttpThrottleSafeIPs:

2014-09-09 10:33:47,772 INFO  [main] [] misc - DoSFilter: Configured whitelist IPs = 192.168.234.130,127.0.0.1,::1,0:0:0:0:0:0:0:1

Note: proxy nodes should not need to be whitelisted, as long as the Originating-IP feature is correctly configured in ZCS: https://wiki.zimbra.com/wiki/Log_Files#Logging_the_Originating_IP

IP addresses should be supplied in the multi-valued zimbraHttpThrottleSafeIPs attribute. CIDR notation can be used in ZCS 8.0.x. Please note in 8.5 and 8.6 you must use IP by IP. To modify:

zmprov mcf zimbraHttpThrottleSafeIPs 10.1.2.3 zimbraHttpThrottleSafeIPs 192.168.4.5 zimbraHttpThrottleSafeIPs 192.168.1.0/24

or to append to an existing list of multi-valued zimbraHttpThrottleSafeIPs

zmprov mcf +zimbraHttpThrottleSafeIPs 10.1.2.3
zmprov mcf +zimbraHttpThrottleSafeIPs 192.168.4.5
zmprov mcf +zimbraHttpThrottleSafeIPs 192.168.1.0/24

Mailbox server restart

A mailbox server restart is required when modifying these attributes.

zmmailboxdctl restart

Using the DoSFilter To Block IPs on Repeated Failed Login - ZCS 8.5+ Only

Starting in ZCS 8.5, you can block IPs for a period of time after a number of failed login attempts. Note that this honors zimbraHttpThrottleSafeIPs, so if set, it will not block the IPs whitelisted there.

You will be looking at these values:

zimbraInvalidLoginFilterDelayInMinBetwnReqBeforeReinstating: 15
zimbraInvalidLoginFilterMaxFailedLogin: 10
zimbraInvalidLoginFilterReinstateIpTaskIntervalInMin: 5

zimbraInvalidLoginFilterDelayInMinBetwnRegBeforeReinstating sets how long an IP is blocked.

zimbraInvalidLoginFilterMaxFailedLogin sets the number of failed logins before an IP is blocked.

zimbraInvalidLoginFilterReinstateIpTaskIntervalInMin sets how long between running the process to unblock IPs.

Examples

To set the DoS filter to block an IP after 5 failed login attempts for 25 minutes, you would do this:

zmprov mcf zimbraInvalidLoginFilterDelayInMinBetwnRegBeforeReinstating 25
zmprov mcf zimbraInvalidLoginFilterMaxFailedLogin 5
zmmailboxdctl restart

Tuning Considerations - 8.0.3 and later

ZCS Member Servers

ZCS servers under the control of a single master LDAP server are automatically whitelisted by IP address. These hosts are discovered using a GetAllServersRequest call; i.e., zmprov gas.

External Provisioning Hosts/SOAP API

External provisioning hosts may be added to the IP whitelist to ensure DoSFilter does not block some requests. For example, a mailbox reindex may make several calls per second that can trigger DoSFilter.

ZCS 8.0.0 - 8.0.2

See this link on the Zimbra forums for information on configuring DoSFilter for ZCS 8 versions prior to 8.0.3.

Verified Against: ZCS 8.0.x Date Created: 03/20/2013
Article ID: https://wiki.zimbra.com/index.php?title=DoSFilter Date Modified: 2015-10-06



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search