JDunphy-Letsencrypt
Letsencrypt - Using acme.sh to Generate Certs
A simplified version of this wiki can be found here: https://wiki.zimbra.com/wiki/JDunphy-LE-Easy
Introduction
Letsencrypt is a free, automated, and open Certificate Authority to generate all your PKI certificates. Instead of installing a development environment like some other Letsencrypt methods, this article describes a single bash script called acme.sh and can be installed and operated without being root. Here is how to get Zimbra up and running with your Letsencrypt certificate. We are describing a DNS challenge method in this document. For further background, see: https://letsencrypt.org/how-it-works/ and this link for rate-limits. Full Documentation for acme.sh is here Letsencrypt has 100's of programs that can generate LE certificates for you. They all work the same way and use the same acme protocol so that certificates can be issued in real-time. Another popular one for Zimbra is: certbot. Choose the one that works best for you.
There are 2 methods described below. The first method is manual and can be followed step by step. The second method is fully automatic and puts the manual steps into a "deploy script" that acme.sh can invoke and do all this automatically for you. It will also handle certificate renewals and restart zimbra automatically every 60 days. The manual method will re-issue certificates every 60 days but you would be responsible for automating the installation of the certificate into zimbra and restarting zimbra.
acme.sh version v3.0.6 and above has switched the default certificate type to ec-256 from rsa certificates. zmcertmgr will have a problem with installing ec-256 certificates without a patch as I write this with 8.8.15P36 and 9.0.0.P28 being the latest versions. Fortunately, the zimbra supported RSA certificates can still be issued by adding --keylength 2048 when issuing your certificate the very first time. Ref: https://forums.zimbra.org/viewtopic.php?p=308088#p308088
Install acme.sh
The user that you run the following command will be where the acme.sh script will be installed.
To install acme.sh, do the following:
% curl https://get.acme.sh | sh Or: % wget -O - https://get.acme.sh | sh
At this point, if you ran the command as root, you can expect to see a .acme.sh in your home directory. If you ran this as user zimbra, you have already found out that it failed. The reason is that /opt/zimbra is owned by root and the zimbra user does not have write permission to create the /opt/zimbra/.acme.sh directory. Here is one solution for that problem:
# mkdir /opt/zimbra/.acme.sh # chown zimbra:zimbra /opt/zimbra/.acme.sh # su - zimbra % wget -O - https://get.acme.sh | sh
The installation of acme.sh has done three things.
- created a directory .acme.sh in the home directory of the user that installed it
- updated your .cshrc and .bashrc so that script is in your path
- created a cron entry for the user for automatic certificate re-issue
Configure acme.sh
Next we have to choose the provider that manages our DNS entries.
% cd ~/.acme.sh % ls dnsapi
A few of the ones that many people use would be dns_gd (godaddy) or dns_cf (cloudflare). Here we will choose cloudflare which is free. Add the following variables to ~/.acme.sh/account.conf
CF_Key="sdfsdfsdfljlbjkljlkjsdfoiwje" CF_Email="xxxx@sss.com" or CF_Token="xxxx" CF_Account_ID="xxxx" CF_Zone_ID="xxxx"
Because acme.sh supports more than letsencrypt signed certificates, we need to do change the defaults for future certificate issue with zimbra.
% cd; cd .acme.sh % ./acme.sh --set-default-ca --server letsencrypt % ./acme.sh --set-default-chain --preferred-chain ISRG --server letsencrypt
Issue Certificate
acme.sh --issue --keylength 2048 --dns dns_cf -d mail.example.com
If we have multiple domains associated with your Zimbra server, then it works like this:
acme.sh --issue --keylength 2048 --dns dns_cf -d mail.example.com -d mail.example.net -d mail.example.org
Wild card certs are supported with ACME v2 protocol
acme.sh --issue --keylength 2048 --dns dns_cf -d example.com -d '*.example.com'
Your certificates can be found at: ~/.acme.sh/mail.example.com ... It uses the first '-d' name to create a directory to store your certificates. If you don't want to use cloudflare, look inside the dnsapi directory for 100's of scripts from various DNS hosting providers. Here is the documentation for many of those scripts. https://github.com/acmesh-official/acme.sh/wiki/dnsapi
Method 1: Manual
This will describe how to manually install the certificate previously issued with zimbra. Regardless of which challenge method you used with the acme.sh bash script, the following commands will install it. Note: I have also created a script to perform these steps automatically at https://github.com/JimDunphy/deploy-zimbra-letsencrypt.sh and the forums have a thread on this method https://forums.zimbra.org/viewtopic.php?f=15&t=60781 for additional background information. For this article we walk through those steps.
Step 1: Append ISRG Root X1
cd ~/.acme.sh/mail.example.com echo '-----BEGIN CERTIFICATE----- MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4 WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+ 0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ 3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5 ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq 4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc= -----END CERTIFICATE-----' >> fullchain.cer
Because zmcertmgr will chdir during install which can abort when permissions are incorrect in some circumstances, we do the following. Make sure the zimbra user has read permission of these files.
% cd ~.acme.sh/mail.example.com % cp mail.example.com.key mail.example.com.cer fullchain.cer /tmp
Step 2 Verify your certificate
% su - zimbra % cd /tmp % /opt/zimbra/bin/zmcertmgr verifycrt comm mail.example.com.key mail.example.com.cer fullchain.cer
If there were no errors, you can install the certificate
Step 3 Install your certificate
% su - zimbra % cd /tmp % cp mail.example.key /opt/zimbra/ssl/zimbra/commercial/commercial.key % /opt/zimbra/bin/zmcertmgr deploycrt comm mail.example.com.cer fullchain.cer
If there were no errors, proceed to restart zimbra
Step 4 Restart Zimbra
% su - zimbra % zmcontrol restart
Note: If you go this route, you are still responsible for renewals as there was no automatic verification, installation, and restart of zimbra. You might have a new certificate every 60 days if you followed along and used the dns challenge method but it will not be installed. The next method uses a deploy scripts that does all the above steps and will also handle automatic renewals every 60 days. Letsencrypt certificates will expire every 90 days.
Method 2: Automatic
With this method, we will install and configure acme.sh as the zimbra user. That is important as the deploy script needs to run as zimbra. We have already issued our certificate in the previous steps. It needs to stated again. This method requires that you have installed acme.sh as the zimbra user and issued the certificates as the zimbra user.
Install the following deploy script for acme.sh:
% wget 'https://raw.githubusercontent.com/JimDunphy/acme.sh/master/deploy/zimbra.sh' -O /opt/zimbra/.acme.sh/deploy/zimbra.sh
Now we can use the --deploy option and have acme.sh perform all the steps in the manual steps above for our zimbra installation. The above script looks like this:
#!/bin/bash
# Zimbra Assumptions:
# 1) acme.sh is installed as Zimbra
# 2) see: https://wiki.zimbra.com/wiki/JDunphy-Letsencrypt
# 3) --preferred-chain "ISRG" or are using this chain
######## Public functions #####################
#domain keyfile certfile cafile fullchain
zimbra_deploy() {
_cdomain="$1"
_ckey="$2"
_ccert="$3"
_cca="$4"
_cfullchain="$5"
_debug _cdomain "$_cdomain"
_debug _ckey "$_ckey"
_debug _ccert "$_ccert"
_debug _cca "$_cca"
_debug _cfullchain "$_cfullchain"
# Zimbra's still needs CA pem to verify on some versions
ISG_X1="$(dirname "$_cca")/../ISG_X1.pem"
_debug ISG_X1 "$ISG_X1"
# grab root pem if we don't have it
if [ ! -f "$ISG_X1" ]; then
_debug No "$ISG_X1"
wget -q "https://letsencrypt.org/certs/isrgrootx1.pem.txt" -O "$ISG_X1" || return 1
fi
# append root pem so verifycrt can walk the chain
cat "$_cfullchain" "$(dirname "$_cca")/../ISG_X1.pem" > "${_cca}.real"
/opt/zimbra/bin/zmcertmgr verifycrt comm "$_ckey" "$_ccert" "${_cca}.real" || return 1
#if it verifies we can deploy it
/bin/logger -p local2.info NETWORK "Certificate has been Renewed for $_cdomain"
cp -f "$_ckey" /opt/zimbra/ssl/zimbra/commercial/commercial.key
/opt/zimbra/bin/zmcertmgr deploycrt comm "$_ccert" "${_cca}.real" || return 1
#/opt/zimbra/bin/ldap restart
#/opt/zimbra/bin/zmmailboxdctl reload
#/opt/zimbra/bin/zmproxyctl reload
#/opt/zimbra/bin/zmmtactl reload
/opt/zimbra/bin/zmcontrol restart
return 0
}
Step 1 - install the certificate
The first -d argument is the directory that contains our certificate. It will be the first name we used when we issued our certificate. So if we did something like:
% ./acme.sh --issue --keylength 2048 --dns dns_cf -d mail.example.com -d mail.anotherdomain.com -d yet.anothername.com
We would do this:
% ./acme.sh --deploy --deploy-hook zimbra -d mail.example.com
Subsequent renewals are performed automatically every 60 days via a cron entry for acme.sh like this. Note: If it isn't time, the script will tell you to use the --force option if you run it from the command line.
% "/opt/zimbra/.acme.sh"/acme.sh --cron --home "/opt/zimbra/.acme.sh" or % "/opt/zimbra/.acme.sh"/acme.sh --force --cron --home "/opt/zimbra/.acme.sh"
We could also do it manually like this:
% su - zimbra % cd .acme.sh % ./acme.sh --issue --keylength 2048 --dns dns_cf -d mail.example.com -d mail.example.net -d mail.example.org % ./acme.sh --deploy --deploy-hook zimbra -d mail.example.com
When we use the--cron option, it will do the above 2 steps if there are not any errors. It is up to you if you want to use the --cron method or issue the 2 steps manually. This also means that you can copy certifcates around to different machines by copying the mail.example.com direcctory and issue the --deploy option on the remote machine. You do not have to re-issue certificates if SAN's cover all your hosts in the certificate.
Errors
If the certificate fails to renew, the --cron option will not deploy it. If the --deploy script fails to validate the certificate and therefore will not work with zimbra, it will not attempt to install it or change a running zimbra instance. Re-issue your certifcate and try again.
Certificate Tricks
look into the --challenge-alias option with the automatic DNS method to further isolate/secure your zone updates with letsencrypt. You only require a CNAME entry for your trusted zimbra domains for the domains above. In other words, each letsencrypt secured zimbra domain would have this in their zone file. Same entry for every one. If your zimbra instance is on private address space (RFC1918), this method would work for those scenarios.
_acme-challenge IN CNAME _acme-challenge.adifferentCFzone.com.
where adifferentCFzone.com is a completely different and managed zone from a DNS provider that has an API such as cloudflare (CF) and not a zimbra domain. It can be any of the supported automatic DNS providers including BIND directly.
Here is how this would look using the CNAME alias where example.com, example.net, and example.org are not managed by CF (cloudflare) but we want to secure for zimbra:
% su - zimbra % cd .acme.sh % ./acme.sh --issue --keylength 2048 --dns dns_cf --challenge-alias adifferentCFzone.com -d mail.example.com -d mail.example.net -d mail.example.org % ./acme.sh --deploy --deploy-hook zimbra --d mail.example.com
Script to Notify of Pending Renewal
Because this happens automatically, I use a script to send me an email 24 hours in advance that I have a pending certificate coming up for renewal and reminds me to watch for it. The source can be found at: https://raw.githubusercontent.com/JimDunphy/ZimbraScripts/master/src/zmcertNotice.sh
Notes
Use Certificate Transparency Monitoring to spot malicious certificates. If your DNS provider offers this service (many do), enable that option. For example one can opt into Cloudflare Monitoring (works even with free accounts), they will send you an email whenever a certificate is issued for one of your domains by crawling the public logs to find new issued certificates with domains under your control. Ref: https://blog.cloudflare.com/introducing-certificate-transparency-monitoring/
Where are the Certs Installed?
Zimbra has 4 major daemons that require certificates. nginx, ldap, postfix, and mailboxd... Below is where zmcertmgr installs the certificate. Because mailboxd is java based, it uses a keystore. Note: /opt/zimbra/ssl contains your certificates. The other locations are copies from here. Further: nginx, ldap, and postfix can reload those new certificates hot without shutting down the services so in theory we are performing a restart because mailboxd and taking an outage during certificate renewal.
% ls -lt /opt/zimbra/conf/slapd.* -rw-r----- 1 zimbra zimbra 7213 Aug 4 10:46 slapd.crt -rw-r----- 1 zimbra zimbra 1679 Aug 4 10:46 slapd.key % ls -lt /opt/zimbra/ssl/zimbra/commercial -rw-r----- 1 zimbra zimbra 5030 Aug 4 10:46 commercial_ca.crt -rw-r----- 1 zimbra zimbra 7213 Aug 4 10:46 commercial.crt -rw-r----- 1 zimbra zimbra 1679 Aug 4 10:46 commercial.key % ls -lt /opt/zimbra/conf/nginx.??? -rw-r----- 1 zimbra zimbra 7213 Aug 4 10:46 /opt/zimbra/conf/nginx.crt -rw-r----- 1 zimbra zimbra 1679 Aug 4 10:46 /opt/zimbra/conf/nginx.key % -l /opt/zimbra/conf/smtpd.??? -rw-r----- 1 zimbra zimbra 7213 Aug 4 10:46 /opt/zimbra/conf/smtpd.crt -rw-r----- 1 zimbra zimbra 1679 Aug 4 10:46 /opt/zimbra/conf/smtpd.key % ls -l /opt/zimbra/mailboxd/etc/keystore -rw-r----- 1 zimbra zimbra 4965 Aug 4 10:46 /opt/zimbra/mailboxd/etc/keystore % ls -l /opt/zimbra/ssl/zimbra/jetty.pkcs12 -rw-r----- 1 zimbra zimbra 6952 Aug 4 10:46 /opt/zimbra/ssl/zimbra/jetty.pkcs12
Bad Certificate Recovery
Should you receive an error with your new certificates because they were not validated correctly you can recover by re-issuing your certificate and then re-install that certificate to zimbra. This will work even if ldap is down and nothing has started.
# su - zimbra % ./acme.sh --issue --keylength 2048 --dns dns_cf --challenge-alias adifferentCFzone.com -d mail.example.com -d mail.example.net -d mail.example.org % ./acme.sh --deploy --deploy-hook zimbra --d mail.example.com
Misc Zimbra Commands
Confirm that your SSL certs are all valid and not-expired
% /opt/zimbra/bin/zmcertmgr viewdeployedcrt all - ldap: /opt/zimbra/conf/slapd.crt notBefore=Oct 27 18:10:32 2018 GMT notAfter=Jan 25 18:10:32 2019 GMT subject= /CN=mail.example.com issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3 SubjectAltName=mail.example.com, mail.example.net, tmail.example.com - mailboxd: /opt/zimbra/mailboxd/etc/mailboxd.pem notBefore=Oct 27 18:10:32 2018 GMT notAfter=Jan 25 18:10:32 2019 GMT subject= /CN=mail.example.com issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3 SubjectAltName=mail.example.com, mail.example.net, tmail.example.com - mta: /opt/zimbra/conf/smtpd.crt notBefore=Oct 27 18:10:32 2018 GMT notAfter=Jan 25 18:10:32 2019 GMT subject= /CN=mail.example.com issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3 SubjectAltName=mail.example.com, mail.example.net, tmail.example.com - proxy: /opt/zimbra/conf/nginx.crt notBefore=Oct 27 18:10:32 2018 GMT notAfter=Jan 25 18:10:32 2019 GMT subject= /CN=mail.example.com issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3 SubjectAltName=mail.example.com, mail.example.net, tmail.example.com
acme.sh --list
acme.sh can also tell you when renewal would occur if you have this automated via the supplied crontab entry.
./acme.sh --list Main_Domain KeyLength SAN_Domains CA Created Renew example.com "" www.example.com LetsEncrypt.org Mon Sep 6 16:36:38 UTC 2021 Fri Nov 5 16:36:38 UTC 2021 example.us "" www.example.us,www2.example.com LetsEncrypt.org Wed Sep 8 17:11:56 UTC 2021 Sun Nov 7 17:11:56 UTC 2021 example.net "" www.example.net,www.example.net,db.example.net LetsEncrypt.org Fri Sep 10 07:05:27 UTC 2021 Tue Nov 9 07:05:27 UTC 2021
Upgrade acme.sh to latest version
% cd ~/.acme.sh % ./acme.sh --upgrade
Remove acme.sh
% cd ~/.acme.sh % ./acme.sh --uninstall
Verify .cshrc and .bashrc has no alias and /bin/rm -rf ~/.acme.sh if that has not been removed.
More articles written by me, https://wiki.zimbra.com/wiki/JDunphy-Notes
