- 1 DoSFilter Throttling Mechanism
- 1.1 Identifying False Positives
- 1.2 Configuration
- 1.2.1 DoSFilter Delay (milliseconds) - zimbraHttpDosFilterDelayMillis
- 1.2.2 DoSFilter Maximum Requests Per Second - zimbraHttpDosFilterMaxRequestsPerSec
- 1.2.3 DoSFilter IP Addresses Whitelist - zimbraHttpThrottleSafeIPs
- 1.2.4 Mailbox server restart
- 1.2.5 Using the DoSFilter To Block IPs on Repeated Failed Login - ZCS 8.5+ Only
- 1.3 Tuning Considerations - 8.0.3 and later
- 1.4 ZCS 8.0.0 - 8.0.2
DoSFilter Throttling Mechanism
The denial-of-service filter or DoSFilter was added to the mailbox server in ZCS 8.0 to throttle clients sending a large number of requests over a very short period of time. The DoSFilter is applied to all requests for service, mailbox and admin. This feature was added with the completion of bug 66921.
DoS filtering is enabled by default once ZCS 8 is installed. It may be necessary to adjust the configuration to accommodate specific environmental needs. Disabling DoSFilter is not recommended.
Identifying False Positives
It is possible for clients such as Zimbra Connector for Outlook (ZCO), mobile ActiveSync, zmprov, etc. to trigger the DoSFilter. To the client, the Zimbra mailbox service is unavailable. DoSFilter can be identified on the server in the following ways:
2013-01-15 15:52:20,426 WARN [qtp1635701107-91:https://10.10.0.54:443/Microsoft-Server-ActiveSync?User=zsupport2&DeviceId=Appl5K0113UN3NR&DeviceType=iPhone&Cmd=FolderSync][firstname.lastname@example.org;mid=64;ip=18.104.22.168;Cmd=FolderSync;DeviceID=Appl5K0113UN3NR;Version=12.1;] sync - Service exception com.zimbra.common.service.ServiceException: error while proxying request to target server: HTTP/1.1 503 Service Unavailable ExceptionId:qtp1635701107-91:https://10.10.0.54:443/Microsoft-Server-ActiveSync?User=zsupport2&DeviceId=Appl5K0113UN3NR&DeviceType=iPhone&Cmd=FolderSync:1358286740426:c5ca7f36bb0a038f Code:service.PROXY_ERROR Arg:(url, STR,"http://mail.domain.com:80/service/soap/SyncRequest")
2013-01-15 15:57:32.537:WARN:oejs.DoSFilter:DOS ALERT:ip=127.0.1.1,session=null,user=null
The configuration attributes zimbraHttpDosFilterDelayMillis, zimbraHttpDosFilterMaxRequestsPerSec and zimbraHttpThrottleSafeIPs. Each attribute is server inherited from global or is configurable at the sever level. Our recommendation is to preserve the default configuration whenever possible.
DoSFilter Delay (milliseconds) - zimbraHttpDosFilterDelayMillis
Delay imposed on all requests over the rate limit, before they are considered at all. -1 = Reject request, 0 = No delay, any other value = Delay in ms. The default is -1.
To modify in the global configuration; e.g. set the delay to 20ms:
zmprov mcf zimbraHttpDosFilterDelayMillis 20
DoSFilter Maximum Requests Per Second - zimbraHttpDosFilterMaxRequestsPerSec
Maximum number of requests from a connection per second. Requests in excess of this are throttled. The default is 30 and the minimum is 1.
To set the maximum number for requests in the global configuration:
zmprov mcf zimbraHttpDosFilterMaxRequestsPerSec 100
DoSFilter IP Addresses Whitelist - zimbraHttpThrottleSafeIPs
Warning: Zimbra Collaboration 8.5 or 8.6 doesn't support CIDR, so you must add the IPs individually.
IP addresses to ignore when applying Jetty DosFilter. This attribute does not have a default value, however these loopback IPs are whitelisted by default:
- All mailboxd servers
You can check if these hosts have been correctly whitelisted by the log entry in /opt/zimbra/log/mailbox.log. This log entry should contain all of the default whitelisted hosts as well as any IPs added to zimbraHttpThrottleSafeIPs:
2014-09-09 10:33:47,772 INFO [main]  misc - DoSFilter: Configured whitelist IPs = 192.168.234.130,127.0.0.1,::1,0:0:0:0:0:0:0:1
Note: proxy nodes should not need to be whitelisted, as long as the Originating-IP feature is correctly configured in ZCS: https://wiki.zimbra.com/wiki/Log_Files#Logging_the_Originating_IP
IP addresses should be supplied in the multi-valued zimbraHttpThrottleSafeIPs attribute. CIDR notation can be used in ZCS 8.0.x. Please note in 8.5 and 8.6 you must use IP by IP. To modify:
zmprov mcf zimbraHttpThrottleSafeIPs 10.1.2.3 zimbraHttpThrottleSafeIPs 192.168.4.5 zimbraHttpThrottleSafeIPs 192.168.1.0/24
or to append to an existing list of multi-valued zimbraHttpThrottleSafeIPs
zmprov mcf +zimbraHttpThrottleSafeIPs 10.1.2.3 zmprov mcf +zimbraHttpThrottleSafeIPs 192.168.4.5 zmprov mcf +zimbraHttpThrottleSafeIPs 192.168.1.0/24
Mailbox server restart
A mailbox server restart is required when modifying these attributes.
Using the DoSFilter To Block IPs on Repeated Failed Login - ZCS 8.5+ Only
Starting in ZCS 8.5, you can block IPs for a period of time after a number of failed login attempts. Note that this honors zimbraHttpThrottleSafeIPs, so if set, it will not block the IPs whitelisted there.
You will be looking at these values:
zimbraInvalidLoginFilterDelayInMinBetwnReqBeforeReinstating: 15 zimbraInvalidLoginFilterMaxFailedLogin: 10 zimbraInvalidLoginFilterReinstateIpTaskIntervalInMin: 5
zimbraInvalidLoginFilterDelayInMinBetwnRegBeforeReinstating sets how long an IP is blocked.
zimbraInvalidLoginFilterMaxFailedLogin sets the number of failed logins before an IP is blocked.
zimbraInvalidLoginFilterReinstateIpTaskIntervalInMin sets how long between running the process to unblock IPs.
To set the DoS filter to block an IP after 5 failed login attempts for 25 minutes, you would do this:
zmprov mcf zimbraInvalidLoginFilterDelayInMinBetwnRegBeforeReinstating 25 zmprov mcf zimbraInvalidLoginFilterMaxFailedLogin 5 zmmailboxdctl restart
Tuning Considerations - 8.0.3 and later
ZCS Member Servers
ZCS servers under the control of a single master LDAP server are automatically whitelisted by IP address. These hosts are discovered using a GetAllServersRequest call; i.e., zmprov gas.
External Provisioning Hosts/SOAP API
External provisioning hosts may be added to the IP whitelist to ensure DoSFilter does not block some requests. For example, a mailbox reindex may make several calls per second that can trigger DoSFilter.
ZCS 8.0.0 - 8.0.2
See this link on the Zimbra forums for information on configuring DoSFilter for ZCS 8 versions prior to 8.0.3.