Anti-spam Strategies: Difference between revisions

No edit summary
(32 intermediate revisions by 5 users not shown)
Line 1: Line 1:
{{Article Infobox|{{admin}}|{{ZCS 8.0}}{{ZCS 7.0}}|||}}
{{BC|Community Sandbox}}
== For SpamAssassin and Anti-spam Updates ==
__FORCETOC__
 
<div class="col-md-12 ibox-content">
= Anti-spam Strategies =
{{KB|{{Unsupported}}|{{ZCS 8.8}}|{{ZCS 8.7}}|{{ZCS 8.0}}|}}
{{WIP}}
== Anti-spam Strategies  ==
=== Customizing SpamAssassin ===
=== Customizing SpamAssassin ===


==== ZCS 8.5 and later ====
==== ZCS 8.5 and later ====
For ZCS 8.5, Spamassassin layout has been corrected as per the SpamAssassin developers.  '''sauser.cf''' is migrated to the '''/opt/zimbra/data/spamassassin/localrules''' directory.  This is the supported location for doing customizations of SpamAssassin for ZCS 8.5 and later.
For ZCS 8.5, SpamAssassin layout has been corrected as per the SpamAssassin developers.  '''sauser.cf''' is migrated to the '''/opt/zimbra/data/spamassassin/localrules''' directory.  This is the supported location for doing customizations of SpamAssassin for ZCS 8.5 and later.


==== ZCS 8 ====
==== ZCS 8 ====
Line 11: Line 15:
in alphabetical order.  If you create a '''sauser.cf''' file, it will be  
in alphabetical order.  If you create a '''sauser.cf''' file, it will be  
loaded after salocal.cf is loaded.  This is the supported method for doing  
loaded after salocal.cf is loaded.  This is the supported method for doing  
customizations of SpamAssassin for ZCS8.0
customizations of SpamAssassin for ZCS 8.  Note that '''only''' the sauser.cf file will be migrated when upgrading to later releases.


In 8.0.5, two options for added to the product to enable SpamAssassin rule updates via sa-update [reference: see [https://bugzilla.zimbra.com/show_bug.cgi?id=82201]]:
In 8.0.5, two options were added to the product to enable SpamAssassin rule updates via sa-update (reference: see [https://bugzilla.zimbra.com/show_bug.cgi?id=82201 82201]):


* antispam_enable_rule_updates
* '''antispam_enable_rule_updates'''
* antispam_enable_restarts
* '''antispam_enable_restarts'''


Check that these are set to true, and if not, set them to true and restart amavisd and the MTA:
Check that these are set to true, and if not, set them to true and restart amavisd and the MTA:


  $ zmlocalconfig -e antispam_enable_rule_updates
  $ zmlocalconfig antispam_enable_rule_updates
  antispam_enable_rule_updates = false
  antispam_enable_rule_updates = false
  $ zmlocalconfig antispam_enable_restarts
  $ zmlocalconfig antispam_enable_restarts
Line 31: Line 35:
  $ zmmtactl restart
  $ zmmtactl restart


==== ZCS 6 and ZCS7 ====
==== ZCS 6 and ZCS 7 ====
For ZCS 6 and ZCS7, SpamAssassin customizations go in '''/opt/zimbra/conf/sauser.cf'''. When upgrading to ZCS8 the file will be reloacted to '''/opt/zimbra/conf/sa'''
For ZCS 6 and ZCS 7, SpamAssassin customizations go in '''/opt/zimbra/conf/sauser.cf'''. When upgrading to ZCS 8 the file will be relocated to '''/opt/zimbra/conf/sa'''


==== Automatic rule updates ====
==== Automatic rule updates ====
Line 45: Line 49:
* '''antispam_enable_rule_compilation''' controls whether or not to automatically compile new rules that are automatically updated.  Defaults to '''false'''.
* '''antispam_enable_rule_compilation''' controls whether or not to automatically compile new rules that are automatically updated.  Defaults to '''false'''.


=== Customizing Postfix ===
= Customizing Postfix =
In ZCS 7 and ZCS 8, customizing Postfix is a mix of zmlocalconfig and zmprov settings.  In ZCS 8.5, virutally all settings are done via zmprov (zmlocalconfig settings will be migrated on upgrade if they do not match the default value).
In ZCS 7 and ZCS 8, customizing Postfix is a mix of zmlocalconfig and zmprov settings.  In ZCS 8.5, virtually all settings are done via zmprov (zmlocalconfig settings will be migrated on upgrade if they do not match the default value).


zmprov/zmlocalconfig are both permissible and the recommended way to perform Postfix customizations for supported keys.
zmprov/zmlocalconfig are both permissible and the recommended way to perform Postfix customizations for supported keys.
Line 52: Line 56:
For example:
For example:


'''zmprov ms <server> +zimbraMtaRestriction reject_unknown_reverse_client_hostname'''
zmprov ms <server> +zimbraMtaRestriction reject_unknown_reverse_client_hostname


=== Specific Suggested Tweaks ===
= Specific Suggested Tweaks =
''Last update 24 October 2014 by L. Mark Stone, Reliable Networks
''Last update 24 October 2014 by L. Mark Stone, Reliable Networks''


Our client base is very nervous about spam-delivered malware but even more concerned about "false-positives" i.e. legitimate email incorrectly identified as spam. Consequently, we've had to develop tweaks to improve Zimbra's default SpamAssassin configurations. The results have been that users with very public email addresses who typically receive several hundred to more than a thousand emails per day will see no more than ~3 spam emails per day in their Inbox.  In our experience, anything less than that and you are likely to wind up with false positives.
Our client base is very nervous about spam-delivered malware but even more concerned about "false-positives" i.e. legitimate email incorrectly identified as spam. Consequently, we've had to develop tweaks to improve Zimbra's default SpamAssassin configurations. The results have been that users with very public email addresses who typically receive several hundred to more than a thousand emails per day will see no more than ~3 spam emails per day in their Inbox.  In our experience, anything less than that and you are likely to wind up with false positives.
Line 68: Line 72:
One configuration nuance to DNS is the use of forwarders in your BIND9 configuration. We have seen many Zimbra systems use their ISP's, or Google's public DNS servers as forwarders. The problem is that many of the RBL services embedded in SpamAssassin and configurable within Zimbra limit the number/rate of queries they accept from a particular DNS server.  Since almost all RBL queries will never be cached, the queries get done by the forwarders. And since the forwarders are doing the same queries for lots of other folks, those queries are often blocked.
One configuration nuance to DNS is the use of forwarders in your BIND9 configuration. We have seen many Zimbra systems use their ISP's, or Google's public DNS servers as forwarders. The problem is that many of the RBL services embedded in SpamAssassin and configurable within Zimbra limit the number/rate of queries they accept from a particular DNS server.  Since almost all RBL queries will never be cached, the queries get done by the forwarders. And since the forwarders are doing the same queries for lots of other folks, those queries are often blocked.


We therefore recommend that when using a local cacheing DNS server that you ensure the configuration has current hints for the root servers and that the forwarders section in the BIND9 config file be set to empty.
We therefore recommend that when using a local caching DNS server that you ensure the configuration has current hints for the root servers and that the forwarders section in the BIND9 config file be set to empty.


==== Postfix Tweaks ====
==== Postfix Tweaks ====
At the Postfix level we use just a few complementary and conservative RBLs, one DNS check and one Protocol check.  All of these can be configured via the Admin Console: (Global Settings > MTA).  A list of RBLs can be found on wikipedia: http://en.wikipedia.org/wiki/Comparison_of_DNS_blacklists
 
===== RBLs =====
At the Postfix level we use just a few complementary and conservative RBLs, one DNS check and one Protocol check.  All of these can be configured via the Admin Console: (Global Settings > MTA).  A list of RBLs can be found at https://en.wikipedia.org/wiki/Comparison_of_DNS_blacklists


The RBLs we use are:
The RBLs we use are:
* zen.spamhaus.org
zen.spamhaus.org psbl.surriel.com b.barracudacentral.org
* psbl.surriel.com
* b.barracudacentral.org


Additional RBLs used by zimbra are:
Additional RBLs used by zimbra are:
* bl.spamcop.net
bl.spamcop.net
* dnsbl.sorbs.net
 
Untested RBLs:
* cbl.abuseat.org (Ignore this one if using zen.spamhaus.org, as it is already included in those results)
* new.spam.dnsbl.sorbs.net
* all.s5h.net
* bl.blocklist.de
* dnsbl.inps.de
* ubl.unsubscore.com (note: Rejects mail from places such as Yahoo!, Facebook, etc.)
* virbl.dnsbl.bit.nl
* mail-abuse.blacklist.jippg.org
* dnsbl.njabl.org


The Client RHSBLs we use are (updated June 2, 2014):
The Client RHSBLs we use are (updated June 2, 2014):
* dbl.spamhaus.org
dbl.spamhaus.org multi.uribl.com multi.surbl.org
* multi.uribl.com
* multi.surbl.org


Additional Client RHSBLs used by Zimbra:
Additional Client RHSBLs used by Zimbra:
* rhsbl.sorbs.net
rhsbl.sorbs.net


Sender RHSBLs used by Zimbra:
Sender RHSBLs used by Zimbra:
* multi.uribl.com  
multi.uribl.com multi.surbl.org rhsbl.sorbs.net dbl.spamhaus.org
* multi.surbl.org
* rhsbl.sorbs.net
* dbl.spamhaus.org


Reverse Client RHSBLs used by Zimbra:
Reverse Client RHSBLs used by Zimbra:
* dbl.spamhaus.org
dbl.spamhaus.org


Adding RBL and RHSBLs checks in postfix can also be done via the command line.
Adding RBL and RHSBLs checks in postfix can also be done via the command line.


For RBLs:
For RBLs:
<nowiki>
  zmprov mcf +zimbraMtaRestriction "reject_rbl_client zen.spamhaus.org"
  zmprov mcf +zimbraMtaRestriction reject_rbl_client zen.spamhaus.org
</nowiki>


For RHSBL clients:
For RHSBL clients:
<nowiki>
  zmprov mcf +zimbraMtaRestriction "reject_rhsbl_client dbl.spamhaus.org"
  zmprov mcf +zimbraMtaRestriction reject_rhsbl_client dbl.spamhaus.org
</nowiki>


On the same Admin Console page we also enable (and leave the remaining Protocol and DNS checks disabled):
On the same Admin Console page we also enable (and leave the remaining Protocol and DNS checks disabled):
* reject_non_fqdn_sender
* '''reject_non_fqdn_sender'''
* reject_unknown_sender_domain (Note this setting will be updated in 8.0.5)
* '''reject_unknown_sender_domain''' (Note this setting will be updated in 8.0.5)


On that same page we also make sure disable "Add X-Originating-IP to messages" as this can block email from remote users with fat email clients like Outlook and Thunderbird on home and public networks like Internet cafes (ZWC clients are unaffected by this.)
On that same page we also make sure disable "Add X-Originating-IP to messages" as this can block email from remote users with fat email clients like Outlook and Thunderbird on home and public networks like Internet cafes (ZWC clients are unaffected by this.)
===== fqrdns.pcre from GitHub =====
Hardware Freak.com maintains a PCRE listing of bad IP ranges to be rejected.  This generally rejects larges amounts of bot traffic where the bots are sending out email directly rather than an authenticated user going through the ISP outgoing SMTP servers.  Support for using this PCRE method is built into ZCS 8.7 and later.
cd /opt/zimbra/conf
wget https://raw.githubusercontent.com/stevejenkins/hardwarefreak.com-fqrdns.pcre/master/fqrdns.pcre
zmprov mcf +zimbraMtaRestriction 'check_reverse_client_hostname_access pcre:/opt/zimbra/conf/fqrdns.pcre'
===== Postscreen =====
Postscreen is a pre-screening process at the MTA level that can be used to reject spammers by doing additive scoring from a variety of sites.  Support for postscreen has been added for ZCS 8.7.  Full configuration details will be added to this wiki prior to release.


==== SpamAssassin Tweaks via the Commandline ====
==== SpamAssassin Tweaks via the Commandline ====
Line 143: Line 138:
If you are on an earlier release, this can be achieved by editing /opt/zimbra/conf/amavisd.conf.in.  You will need to change the file's permissions to be writable, edit the file, then change the permissions back. Probably a good idea to make a backup copy of the file first...  The final edit should should look like this:
If you are on an earlier release, this can be achieved by editing /opt/zimbra/conf/amavisd.conf.in.  You will need to change the file's permissions to be writable, edit the file, then change the permissions back. Probably a good idea to make a backup copy of the file first...  The final edit should should look like this:


: $log_level = 2;              # verbosity 0..5 - 1 is the minimum for msg tracing
$log_level = 2;              # verbosity 0..5 - 1 is the minimum for msg tracing


Restart amavis for the change to take effect (zmavavisdctl restart).  If you are on ZCS 8.0.5 or later, zmconfigd will automatically restart Amavis for you if you change the loglevel.
Restart amavis for the change to take effect (zmavavisdctl restart).  If you are on ZCS 8.0.5 or later, zmconfigd will automatically restart Amavis for you if you change the loglevel.


Now when an email is marked as spam and an end user asks you "Why?", you can grep /opt/zimbra/log and find out exactly why. Note the sender and recipient email addresses in the actual log file snippet below have been altered for privacy:
Now when an email is marked as spam and an end user asks you "Why?", you can grep /var/log/zimbra.log and find out exactly why. Note the sender and recipient email addresses in the actual log file snippet below have been altered for privacy (lines wrapped for readability):


  Nov 26 13:55:02 mail2 amavis[19107]: (19107-13) SPAM, <comsumer_health@spamsender.com> -> <masked_recipient@example.com>, Yes, score=17.071 tag=-10 tag2=3.8 kill=16 tests=[BAYES_99=4, HTML_MESSAGE=0.001, RAZOR2_CF_RANGE_51_100=0.5, RAZOR2_CF_RANGE_E8_51_100=1.886, RAZOR2_CHECK=2.75, RDNS_NONE=3.5, SPF_PASS=-0.001, T_HK_NAME_DR=0.01, URIBL_BLACK=2.725, URIBL_DBL_SPAM=1.7] autolearn=spam
  Nov 26 13:55:02 mail2 amavis[19107]: (19107-13) SPAM, <comsumer_health@spamsender.com> -> <masked_recipient@example.com>,
 
  Yes, score=17.071 tag=-10 tag2=3.8 kill=16 tests=[BAYES_99=4, HTML_MESSAGE=0.001, RAZOR2_CF_RANGE_51_100=0.5,
ZCS8 logs:
  RAZOR2_CF_RANGE_E8_51_100=1.886, RAZOR2_CHECK=2.75, RDNS_NONE=3.5, SPF_PASS=-0.001, T_HK_NAME_DR=0.01, URIBL_BLACK=2.725,
Apr 21 13:55:54 edge01 amavis[32619]: (32619-05) spam-tag, <DrOz@spamsender.us> -> <masked_recipient@example.com>, Yes, score=9.014 tagged_above=-10 required=3 tests=[BAYES_40=-0.001, DIGEST_MULTIPLE=0.293, DKIM_SIGNED=0.1, HTML_IMAGE_ONLY_32=0.001, HTML_IMAGE_RATIO_06=0.001, HTML_MESSAGE=0.001, MIME_HTML_ONLY=0.723, PYZOR_CHECK=2.75, RAZOR2_CF_RANGE_51_100=0.5, RAZOR2_CF_RANGE_E8_51_100=1.886, RAZOR2_CHECK=2.75, T_DKIM_INVALID=0.01] autolearn=no autolearn_force=no
  URIBL_DBL_SPAM=1.7] autolearn=spam


ZCS 8 logs (lines wrapped for readability):
Apr 21 13:55:54 edge01 amavis[32619]: (32619-05) spam-tag, <DrOz@spamsender.us> -> <masked_recipient@example.com>,
  Yes, score=9.014 tagged_above=-10 required=3 tests=[BAYES_40=-0.001, DIGEST_MULTIPLE=0.293, DKIM_SIGNED=0.1,
  HTML_IMAGE_ONLY_32=0.001, HTML_IMAGE_RATIO_06=0.001, HTML_MESSAGE=0.001, MIME_HTML_ONLY=0.723, PYZOR_CHECK=2.75,
  RAZOR2_CF_RANGE_51_100=0.5, RAZOR2_CF_RANGE_E8_51_100=1.886, RAZOR2_CHECK=2.75, T_DKIM_INVALID=0.01]
  autolearn=no autolearn_force=no


In the above example you can see that the sending server has no PTR (Reverse DNS record) and has already been reported to Razor.
In the above example you can see that the sending server has no PTR (Reverse DNS record) and has already been reported to Razor.
Line 164: Line 165:
An /etc/fstab entry for a 1GB RAM disks on the server therefore looks like:
An /etc/fstab entry for a 1GB RAM disks on the server therefore looks like:


: zimbra@host:~$ cat /etc/fstab | grep amavis
$ grep amavis /etc/fstab
: tmpfs        /opt/zimbra/data/amavisd/tmp              tmpfs  defaults,noexec,nodev,nosuid,size=1024m,mode=750,uid=zimbra,gid=zimbra 0 0
tmpfs        /opt/zimbra/data/amavisd/tmp              tmpfs  defaults,noexec,nodev,nosuid,size=1024m,mode=750,uid=zimbra,gid=zimbra 0 0
: zimbra@host:~$


===== 3. Tweak Selected SpamAssasin Scores After Installing Pyzor and Razor2 =====
===== 3. Tweak Selected SpamAssasin Scores After Installing Pyzor and Razor2 =====
====== Installing Razor and Pyzor on Ubuntu ======
* aptitude install razor pyzor


====== Installing Razor and Pyzor on RHEL6/CentOS6 ======  
= How to install Razor and Pyzor =
* Create /etc/yum.repos.d/epel.repo
== Installing Razor and Pyzor on Ubuntu ==
<code>
aptitude install razor pyzor
 
==Installing Razor and Pyzor on RHEL6/CentOS6 ==
Create /etc/yum.repos.d/epel.repo
  [epel]
  [epel]
  name=EPEL repository
  name=EPEL repository
Line 180: Line 181:
  enabled=1
  enabled=1
  gpgcheck=0
  gpgcheck=0
</code>
* yum update
* yum install pyzor perl-Razor-Agent


====== Configuring Pyzor ======
yum update
yum install pyzor perl-Razor-Agent
 
== Configuring Pyzor ==
As the zimbra user
As the zimbra user
<code>
  pyzor --homedir /opt/zimbra/data/amavisd/.pyzor discover
  pyzor --homedir /opt/zimbra/data/amavisd/.pyzor discover
</code>


Update /opt/zimbra/conf/sa/sauser.cf
Update /opt/zimbra/conf/sa/sauser.cf
<code>
 
  # pyzor
  # pyzor
  use_pyzor 1
  use_pyzor 1
Line 198: Line 197:
  # may slow down email delivery
  # may slow down email delivery
  pyzor_timeout 20
  pyzor_timeout 20
</code>


====== Configuring Razor ======
== Configuring Razor ==
As the zimbra user
As the zimbra user
<code>
 
  razor-admin -home=/opt/zimbra/data/amavisd/.razor -create
  razor-admin -home=/opt/zimbra/data/amavisd/.razor -create
  razor-admin -home=/opt/zimbra/data/amavisd/.razor -discover
  razor-admin -home=/opt/zimbra/data/amavisd/.razor -discover
  razor-admin -home=/opt/zimbra/data/amavisd/.razor -register -user postmaster@yourdomain.com
  razor-admin -home=/opt/zimbra/data/amavisd/.razor -register -user postmaster@yourdomain.com
</code>


Update /opt/zimbra/conf/sa/sauser.cf
Update /opt/zimbra/conf/sa/sauser.cf
<code>
 
  # razor
  # razor
  use_razor2 1
  use_razor2 1
</code>


====== Update SpamAssassin scoring ======
== Update SpamAssassin scoring ==
After installing Pyzor and Razor2 and restarting Zimbra's Amavis to make sure these modules are loaded by SpamAssassin, Reliable Networks adds custom (higher) scoring for certain SpamAssassin tests to the appropriate custom SpamAssassin configuration file, which on ZCS 8 should be /opt/zimbra/conf/sa/sauser.cf.  Our complete sauser.cf now looks like this (as of September 3, 2014):
After installing Pyzor and Razor2 and restarting Zimbra's Amavis to make sure these modules are loaded by SpamAssassin, Reliable Networks adds custom (higher) scoring for certain SpamAssassin tests to the appropriate custom SpamAssassin configuration file, which on ZCS 8 should be /opt/zimbra/conf/sa/sauser.cf.  Our complete sauser.cf now looks like this (as of September 3, 2014):


<code>
  pyzor_timeout 10
  pyzor_timeout 10
  use_razor2 1
  use_razor2 1
Line 229: Line 224:
  score BAYES_00 -0.500
  score BAYES_00 -0.500
  score RP_MATCHES_RCVD -0.000
  score RP_MATCHES_RCVD -0.000
</code>


Then run as the zimbra user "zmantispamctl restart ; zmmtactl restart" to restart and load the new scores. The RP_MATCHES_RCVD score is normally -1.713, but we have found that many spammers using cloud servers have DNS and mail forwarding set to RFC standards, and that their emails then get a bump in good reputation from the default score on this test specifically.
Then as the '''zimbra''' user, run "zmantispamctl restart ; zmmtactl restart" to restart and load the new scores. The RP_MATCHES_RCVD score is normally -1.713, but we have found that many spammers using cloud servers have DNS and mail forwarding set to RFC standards, and that their emails then get a bump in good reputation from the default score on this test specifically.
 
We have found that increasing the scores of the above selected SpamAssassin scores blocks a lot of spam that would otherwise get through.
 
= 4. Add custom rules from Kevin McGrail to your scores =
As zimbra user:


===== 4. Add custom rules from Kevin McGrail to your scores (UNTESTED) =====
* 8.0 and previous:
* As zimbra user: cd /opt/zimbra/conf/sa
<code>
wget http://www.pccc.com/downloads/SpamAssassin/contrib/KAM.cf -O sakam.cf
</code>


There are reports of high false positives using these rules, so user beware
cd /opt/zimbra/conf/sa
wget -N https://www.pccc.com/downloads/SpamAssassin/contrib/KAM.cf -O sakam.cf
zmamavisdctl restart


===== 5. Add the SOUGHT ruleset to your scores (UNTESTED) =====
* 8.5 and later:


TODO: Document configuring http://taint.org/2007/08/15/004348a.html for Zimbra
cd /opt/zimbra/data/spamassassin/localrules
wget -N https://www.pccc.com/downloads/SpamAssassin/contrib/KAM.cf -O sakam.cf
zmamavisdctl restart


===== 6. Enable DCC =====
= 5. Enable DCC =
The source for DCC can be obtained from http://www.rhyolite.com/dcc/.  Please read the restrictions and limitations carefully.
The source for DCC can be obtained from https://www.dcc-servers.net/dcc/.  Please read the restrictions and limitations carefully.  In particular, it is important to keep in mind that DCC just marks whether something is bulk mail or not, and will tag completely legitimate bulk mailings.


After downloading and extracting the source, as the '''zimbra''' user, you will need to build it.  It will take several tools (gcc, make, wget, etc).
After downloading and extracting the source, as the '''zimbra''' user, you will need to build it.  It will take several tools (gcc, make, wget, etc).
Line 252: Line 251:
There is some setup to be done as '''root''' initially.  This is assuming using version 1.3.154 of dcc, adjust as necessary:
There is some setup to be done as '''root''' initially.  This is assuming using version 1.3.154 of dcc, adjust as necessary:


<nowiki>
  # mkdir -p /opt/zimbra/dcc-1.3.154
  # mkdir -p /opt/zimbra/dcc-1.3.154
  # chown zimbra:zimbra /opt/zimbra/dcc-1.3.154
  # chown zimbra:zimbra /opt/zimbra/dcc-1.3.154
  # cd /opt/zimbra;ln -s dcc-1.3.154 dcc
  # cd /opt/zimbra;ln -s dcc-1.3.154 dcc
</nowiki>


Now, as '''zimbra''' we need to build the software.  Here's an example of downloading, extracting, and building:
Now, as '''zimbra''' we need to build the software.  Here's an example of downloading, extracting, and building:


<nowiki>
  [zimbra@host]$ cd /tmp
  [zimbra@host ~]$ cd /tmp
  [zimbra@host]$ mkdir dcc
  [zimbra@host tmp]$ mkdir dcc
  [zimbra@host]$ wget https://www.dcc-servers.net/dcc/source/dcc.tar.Z
  [zimbra@host dcc]$ wget http://www.rhyolite.com/dcc/source/dcc.tar.Z
  [zimbra@host]$ tar xfz dcc.tar.Z
  [zimbra@host dcc]$ tar xfz dcc.tar.Z
  [zimbra@host]$ cd dcc-1.3.154
  [zimbra@host dcc]$ cd dcc-1.3.154
  [zimbra@host]$ ./configure --homedir=/opt/zimbra/dcc-1.3.154 \
  [zimbra@host dcc-1.3.154]$ ./configure --homedir=/opt/zimbra/dcc-1.3.154 \
  --disable-sys-inst --with-uid=zimbra --disable-server \
--disable-sys-inst --with-uid=zimbra --disable-server \
  --disable-dccifd --disable-dccm \
--disable-dccifd --disable-dccm \
  --with-updatedcc_pfile=/opt/zimbra/data/dcc \
--with-updatedcc_pfile=/opt/zimbra/data/dcc \
  --with-rundir=/opt/zimbra/data/dcc/run \
--with-rundir=/opt/zimbra/data/dcc/run \
  --bindir=/opt/zimbra/dcc-1.3.154/bin
--bindir=/opt/zimbra/dcc-1.3.154/bin
  [zimbra@host]$ make
  [zimbra@host dcc-1.3.154]$ make
  [zimbra@host]$ make install
  [zimbra@host dcc-1.3.154]$ make install
  [zimbra@host]$ cd /opt/zimbra/data
  [zimbra@host dcc-1.3.154]$ cd /opt/zimbra/data
  [zimbra@host data]$ mkdir -p dcc/run
  [zimbra@host data]$ mkdir -p dcc/run
</nowiki>


As the '''zimbra''' user, update '''sauser.cf''' as appropriate for your Zimbra version:
As the '''zimbra''' user, update '''sauser.cf''' as appropriate for your Zimbra version:
<nowiki>
 
  use_dcc 1
  use_dcc 1
  dcc_path /opt/zimbra/dcc/bin/dccproc
  dcc_path /opt/zimbra/dcc/bin/dccproc
</nowiki>


For ZCS releases 8.0 and earlier, you will need to enable the dcc module by modifying the '''v310.pre''' file from SpamAssassin.
For ZCS 8.0 and earlier, you will need to enable the dcc module by modifying the '''v310.pre''' file from SpamAssassin.
Find the line that looks like:
Find the line that looks like:
  #loadplugin Mail::SpamAssassin::Plugin::DCC
  #loadplugin Mail::SpamAssassin::Plugin::DCC
Line 291: Line 285:


Last, but not least, restart amavis to pick up the changes:
Last, but not least, restart amavis to pick up the changes:
<nowiki>
[zimbra@host ~]$ zmamavisdctl restart
</nowiki>


=== DNSWL registration ===
[zimbra@host]$ zmamavisdctl restart
Register your MTAs with DNSWL: http://dnswl.org/request.pl


=== Other notes ===
= DNSWL registration =
We have found that increasing the scores of the above selected SpamAssassin scores blocks a lot of spam that would otherwise get through.
Register your MTAs with DNSWL: https://www.dnswl.org/selfservice/
 
= Other notes =


As we make updates to our own configurations, we will endeavor to keep this page updated as well.
As we make updates to our own configurations, we will endeavor to keep this page updated as well.
{{Article Footer|Zimbra Collaboration 8.0, 7.0|04/16/2014}}

Revision as of 12:33, 4 April 2018

Anti-spam Strategies

   KB 20549        Last updated on 2018-04-4  




0.00
(0 votes)

Anti-spam Strategies

Customizing SpamAssassin

ZCS 8.5 and later

For ZCS 8.5, SpamAssassin layout has been corrected as per the SpamAssassin developers. sauser.cf is migrated to the /opt/zimbra/data/spamassassin/localrules directory. This is the supported location for doing customizations of SpamAssassin for ZCS 8.5 and later.

ZCS 8

For ZCS 8.0, SpamAssassin scans for all *.cf files in /opt/zimbra/conf/sa and loads them in alphabetical order. If you create a sauser.cf file, it will be loaded after salocal.cf is loaded. This is the supported method for doing customizations of SpamAssassin for ZCS 8. Note that only the sauser.cf file will be migrated when upgrading to later releases.

In 8.0.5, two options were added to the product to enable SpamAssassin rule updates via sa-update (reference: see 82201):

  • antispam_enable_rule_updates
  • antispam_enable_restarts

Check that these are set to true, and if not, set them to true and restart amavisd and the MTA:

$ zmlocalconfig antispam_enable_rule_updates
antispam_enable_rule_updates = false
$ zmlocalconfig antispam_enable_restarts
antispam_enable_restarts = false
$ zmlocalconfig -e antispam_enable_rule_updates=true
$ zmlocalconfig -e antispam_enable_restarts=true
$ zmamavisdctl restart
$ zmmtactl restart

ZCS 6 and ZCS 7

For ZCS 6 and ZCS 7, SpamAssassin customizations go in /opt/zimbra/conf/sauser.cf. When upgrading to ZCS 8 the file will be relocated to /opt/zimbra/conf/sa

Automatic rule updates

With ZCS 8 and later, it is possible to enable automatic rule updates for SpamAssassin to help improve scoring. There are two localconfig keys that control the automatic update behavior.

  • antispam_enable_rule_updates controls whether or not to enable automatic rule updates. Defaults to false.
  • antispam_enable_restarts controls whether or not Amavisd will be automatically restarted after a rule update if they are enabled. Defaults to false.

Automatic rule compilation

With ZCS 8.5 and later, it is possible to enable automatic rule compilation when automatic updates are enabled. Compiling the SA rules helps decrease the amount of time it takes to score email. This is controlled via a localconfig key.

  • antispam_enable_rule_compilation controls whether or not to automatically compile new rules that are automatically updated. Defaults to false.

Customizing Postfix

In ZCS 7 and ZCS 8, customizing Postfix is a mix of zmlocalconfig and zmprov settings. In ZCS 8.5, virtually all settings are done via zmprov (zmlocalconfig settings will be migrated on upgrade if they do not match the default value).

zmprov/zmlocalconfig are both permissible and the recommended way to perform Postfix customizations for supported keys.

For example:

zmprov ms <server> +zimbraMtaRestriction reject_unknown_reverse_client_hostname

Specific Suggested Tweaks

Last update 24 October 2014 by L. Mark Stone, Reliable Networks

Our client base is very nervous about spam-delivered malware but even more concerned about "false-positives" i.e. legitimate email incorrectly identified as spam. Consequently, we've had to develop tweaks to improve Zimbra's default SpamAssassin configurations. The results have been that users with very public email addresses who typically receive several hundred to more than a thousand emails per day will see no more than ~3 spam emails per day in their Inbox. In our experience, anything less than that and you are likely to wind up with false positives.

If your end-user base is more tolerant of false positives, then you can tighten things up.

Keep in mind that Zimbra's Postfix takes a cut at filtering the email stream before Zimbra's SpamAssassin, and that SpamAssassin's processing of emails is much more resource intensive than Postfix's. Consequently, any filtering that you can do at the Postfix level to block emails outright will be helpful in both blocking spam and lowering resource utilization on your Zimbra server. Just be careful of inducing false positives!

DNS Tweaks

Zimbra recommends using a caching DNS server locally, and we like BIND9 but DNSMasq is fine as well. (As we understand it, Zimbra may start shipping a DNS server bundled with Zimbra in a later release.)

One configuration nuance to DNS is the use of forwarders in your BIND9 configuration. We have seen many Zimbra systems use their ISP's, or Google's public DNS servers as forwarders. The problem is that many of the RBL services embedded in SpamAssassin and configurable within Zimbra limit the number/rate of queries they accept from a particular DNS server. Since almost all RBL queries will never be cached, the queries get done by the forwarders. And since the forwarders are doing the same queries for lots of other folks, those queries are often blocked.

We therefore recommend that when using a local caching DNS server that you ensure the configuration has current hints for the root servers and that the forwarders section in the BIND9 config file be set to empty.

Postfix Tweaks

RBLs

At the Postfix level we use just a few complementary and conservative RBLs, one DNS check and one Protocol check. All of these can be configured via the Admin Console: (Global Settings > MTA). A list of RBLs can be found at https://en.wikipedia.org/wiki/Comparison_of_DNS_blacklists

The RBLs we use are:

zen.spamhaus.org	psbl.surriel.com	b.barracudacentral.org

Additional RBLs used by zimbra are:

bl.spamcop.net

The Client RHSBLs we use are (updated June 2, 2014):

dbl.spamhaus.org	multi.uribl.com		multi.surbl.org

Additional Client RHSBLs used by Zimbra:

rhsbl.sorbs.net

Sender RHSBLs used by Zimbra:

multi.uribl.com 	multi.surbl.org		rhsbl.sorbs.net		dbl.spamhaus.org

Reverse Client RHSBLs used by Zimbra:

dbl.spamhaus.org

Adding RBL and RHSBLs checks in postfix can also be done via the command line.

For RBLs:

zmprov mcf +zimbraMtaRestriction "reject_rbl_client zen.spamhaus.org"

For RHSBL clients:

zmprov mcf +zimbraMtaRestriction "reject_rhsbl_client dbl.spamhaus.org"

On the same Admin Console page we also enable (and leave the remaining Protocol and DNS checks disabled):

  • reject_non_fqdn_sender
  • reject_unknown_sender_domain (Note this setting will be updated in 8.0.5)

On that same page we also make sure disable "Add X-Originating-IP to messages" as this can block email from remote users with fat email clients like Outlook and Thunderbird on home and public networks like Internet cafes (ZWC clients are unaffected by this.)

fqrdns.pcre from GitHub

Hardware Freak.com maintains a PCRE listing of bad IP ranges to be rejected. This generally rejects larges amounts of bot traffic where the bots are sending out email directly rather than an authenticated user going through the ISP outgoing SMTP servers. Support for using this PCRE method is built into ZCS 8.7 and later.

cd /opt/zimbra/conf
wget https://raw.githubusercontent.com/stevejenkins/hardwarefreak.com-fqrdns.pcre/master/fqrdns.pcre
zmprov mcf +zimbraMtaRestriction 'check_reverse_client_hostname_access pcre:/opt/zimbra/conf/fqrdns.pcre'
Postscreen

Postscreen is a pre-screening process at the MTA level that can be used to reject spammers by doing additive scoring from a variety of sites. Support for postscreen has been added for ZCS 8.7. Full configuration details will be added to this wiki prior to release.

SpamAssassin Tweaks via the Commandline

Our current recommended SpamAssassin customizations comprise three complementary methods:

  1. Increase the log level reported by Amavis to get clarity from SpamAssassin on why/how spam is being blocked and getting through.
  2. Put Amavis's temporary directory on a RAM disk to speed up processing.
  3. Tweak the scores for a few selected individual SpamAssassin tests after installing Pyzor and Razor2.
1. Increase Amavis's Log Level

We found that increasing the log level from 1 to 2 puts in /var/log/zimbra.log the specific SpamAssassin tests which each email has triggered.

Customizing the Amavis Loglevel is supported in ZCS 8.0.5 and later:

zmprov mcf zimbraAmavisLogLevel 2

If you are on an earlier release, this can be achieved by editing /opt/zimbra/conf/amavisd.conf.in. You will need to change the file's permissions to be writable, edit the file, then change the permissions back. Probably a good idea to make a backup copy of the file first... The final edit should should look like this:

$log_level = 2;              # verbosity 0..5 - 1 is the minimum for msg tracing

Restart amavis for the change to take effect (zmavavisdctl restart). If you are on ZCS 8.0.5 or later, zmconfigd will automatically restart Amavis for you if you change the loglevel.

Now when an email is marked as spam and an end user asks you "Why?", you can grep /var/log/zimbra.log and find out exactly why. Note the sender and recipient email addresses in the actual log file snippet below have been altered for privacy (lines wrapped for readability):

Nov 26 13:55:02 mail2 amavis[19107]: (19107-13) SPAM, <comsumer_health@spamsender.com> -> <masked_recipient@example.com>,
 Yes, score=17.071 tag=-10 tag2=3.8 kill=16 tests=[BAYES_99=4, HTML_MESSAGE=0.001, RAZOR2_CF_RANGE_51_100=0.5,
 RAZOR2_CF_RANGE_E8_51_100=1.886, RAZOR2_CHECK=2.75, RDNS_NONE=3.5, SPF_PASS=-0.001, T_HK_NAME_DR=0.01, URIBL_BLACK=2.725,
 URIBL_DBL_SPAM=1.7] autolearn=spam

ZCS 8 logs (lines wrapped for readability):

Apr 21 13:55:54 edge01 amavis[32619]: (32619-05) spam-tag, <DrOz@spamsender.us> -> <masked_recipient@example.com>,
 Yes, score=9.014 tagged_above=-10 required=3 tests=[BAYES_40=-0.001, DIGEST_MULTIPLE=0.293, DKIM_SIGNED=0.1,
 HTML_IMAGE_ONLY_32=0.001, HTML_IMAGE_RATIO_06=0.001, HTML_MESSAGE=0.001, MIME_HTML_ONLY=0.723, PYZOR_CHECK=2.75,
 RAZOR2_CF_RANGE_51_100=0.5, RAZOR2_CF_RANGE_E8_51_100=1.886, RAZOR2_CHECK=2.75, T_DKIM_INVALID=0.01]
 autolearn=no autolearn_force=no

In the above example you can see that the sending server has no PTR (Reverse DNS record) and has already been reported to Razor.

2. Put Amavis's Temp Dir on a RAM Disk

We have seen even with fast RAID10 arrays that Amavis's processing an email with large attachments through SpamAssassin can take as long as 10-20 seconds. Putting Amavis'd temp directory on a RAM disk cuts this down to 1-2 seconds. Ralf Hildebrandt's book on Postfix has a section describing how to size the RAM disk, and why this is entirely safe for mail flow even in the event of a server crash. After you've done the homework for sizing, all you need to do is:

  1. Stop amavis, mount the RAM disk, start amavis and then edit /etc/fstab to make the change permanent.

An /etc/fstab entry for a 1GB RAM disks on the server therefore looks like:

$ grep amavis /etc/fstab
tmpfs        /opt/zimbra/data/amavisd/tmp              tmpfs   defaults,noexec,nodev,nosuid,size=1024m,mode=750,uid=zimbra,gid=zimbra 0 0
3. Tweak Selected SpamAssasin Scores After Installing Pyzor and Razor2

How to install Razor and Pyzor

Installing Razor and Pyzor on Ubuntu

aptitude install razor pyzor

Installing Razor and Pyzor on RHEL6/CentOS6

Create /etc/yum.repos.d/epel.repo

[epel]
name=EPEL repository
baseurl=http://mirrors.kernel.org/fedora-epel/6/x86_64
enabled=1
gpgcheck=0
yum update
yum install pyzor perl-Razor-Agent

Configuring Pyzor

As the zimbra user

pyzor --homedir /opt/zimbra/data/amavisd/.pyzor discover

Update /opt/zimbra/conf/sa/sauser.cf

# pyzor
use_pyzor 1
pyzor_path /usr/bin/pyzor
# DNS lookups for pyzor can time out easily.  Set the following line IF you want to give pyzor up to 20 seconds to respond
# may slow down email delivery
pyzor_timeout 20

Configuring Razor

As the zimbra user

razor-admin -home=/opt/zimbra/data/amavisd/.razor -create
razor-admin -home=/opt/zimbra/data/amavisd/.razor -discover
razor-admin -home=/opt/zimbra/data/amavisd/.razor -register -user postmaster@yourdomain.com

Update /opt/zimbra/conf/sa/sauser.cf

# razor
use_razor2 1

Update SpamAssassin scoring

After installing Pyzor and Razor2 and restarting Zimbra's Amavis to make sure these modules are loaded by SpamAssassin, Reliable Networks adds custom (higher) scoring for certain SpamAssassin tests to the appropriate custom SpamAssassin configuration file, which on ZCS 8 should be /opt/zimbra/conf/sa/sauser.cf. Our complete sauser.cf now looks like this (as of September 3, 2014):

pyzor_timeout 10
use_razor2 1
use_pyzor 1
score URIBL_BLACK 3.250
score RAZOR2_CHECK 3.250
score PYZOR_CHECK 3.250
score BAYES_99 4.000
score BAYES_60 2.250
score BAYES_50 1.500
score BAYES_00 -0.500
score RP_MATCHES_RCVD -0.000

Then as the zimbra user, run "zmantispamctl restart ; zmmtactl restart" to restart and load the new scores. The RP_MATCHES_RCVD score is normally -1.713, but we have found that many spammers using cloud servers have DNS and mail forwarding set to RFC standards, and that their emails then get a bump in good reputation from the default score on this test specifically.

We have found that increasing the scores of the above selected SpamAssassin scores blocks a lot of spam that would otherwise get through.

4. Add custom rules from Kevin McGrail to your scores

As zimbra user:

  • 8.0 and previous:
cd /opt/zimbra/conf/sa
wget -N https://www.pccc.com/downloads/SpamAssassin/contrib/KAM.cf -O sakam.cf
zmamavisdctl restart
  • 8.5 and later:
cd /opt/zimbra/data/spamassassin/localrules
wget -N https://www.pccc.com/downloads/SpamAssassin/contrib/KAM.cf -O sakam.cf
zmamavisdctl restart

5. Enable DCC

The source for DCC can be obtained from https://www.dcc-servers.net/dcc/. Please read the restrictions and limitations carefully. In particular, it is important to keep in mind that DCC just marks whether something is bulk mail or not, and will tag completely legitimate bulk mailings.

After downloading and extracting the source, as the zimbra user, you will need to build it. It will take several tools (gcc, make, wget, etc).

There is some setup to be done as root initially. This is assuming using version 1.3.154 of dcc, adjust as necessary:

# mkdir -p /opt/zimbra/dcc-1.3.154
# chown zimbra:zimbra /opt/zimbra/dcc-1.3.154
# cd /opt/zimbra;ln -s dcc-1.3.154 dcc

Now, as zimbra we need to build the software. Here's an example of downloading, extracting, and building:

[zimbra@host]$ cd /tmp
[zimbra@host]$ mkdir dcc
[zimbra@host]$ wget https://www.dcc-servers.net/dcc/source/dcc.tar.Z
[zimbra@host]$ tar xfz dcc.tar.Z
[zimbra@host]$ cd dcc-1.3.154
[zimbra@host]$ ./configure --homedir=/opt/zimbra/dcc-1.3.154 \
 --disable-sys-inst --with-uid=zimbra --disable-server \
 --disable-dccifd --disable-dccm \
 --with-updatedcc_pfile=/opt/zimbra/data/dcc \
 --with-rundir=/opt/zimbra/data/dcc/run \
 --bindir=/opt/zimbra/dcc-1.3.154/bin
[zimbra@host]$ make
[zimbra@host]$ make install
[zimbra@host]$ cd /opt/zimbra/data
[zimbra@host data]$ mkdir -p dcc/run

As the zimbra user, update sauser.cf as appropriate for your Zimbra version:

use_dcc 1
dcc_path /opt/zimbra/dcc/bin/dccproc

For ZCS 8.0 and earlier, you will need to enable the dcc module by modifying the v310.pre file from SpamAssassin. Find the line that looks like:

#loadplugin Mail::SpamAssassin::Plugin::DCC

and uncomment it (remove the # sign)

Last, but not least, restart amavis to pick up the changes:

[zimbra@host]$ zmamavisdctl restart

DNSWL registration

Register your MTAs with DNSWL: https://www.dnswl.org/selfservice/

Other notes

As we make updates to our own configurations, we will endeavor to keep this page updated as well.

Verified Against: Zimbra Collaboration 8.0, 7.0 Date Created: 04/16/2014
Article ID: https://wiki.zimbra.com/index.php?title=Anti-spam_Strategies Date Modified: 2018-04-04



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »


Jump to: navigation, search