JDunphy-Letsencrypt
Letsencrypt - Another Method Using acme.sh to Generate Certs
Introduction
Letsencrypt is a free, automated, and open Certificate Authority to generate all your PKI certificates. Instead of installing a development environment like other Letsencrypt methods, this article describes a single bash script and can be installed and operated without being root. Here is how to get Zimbra up and running with your Letsencrypt certificate. For further background, see: https://letsencrypt.org/how-it-works/ and this link for rate-limits: https://letsencrypt.org/docs/rate-limits/
Update: 2023
acme.sh version v3.0.6 and above has switched the default certificate type to ec-256 from rsa certificates. zmcertmgr will have a problem with installing ec-256 certificates without a patch as I write this with 8.8.15P36 and 9.0.0.P28 being the latest versions.
Add --keylength 2048 to use RSA (previous default)
Ref: https://forums.zimbra.org/viewtopic.php?p=308088#p308088
Requirements (1 time only)
- Install acme.sh bash script in your home directory. Ref: https://github.com/Neilpang/acme.sh
- If you try this as the zimbra user, you will not have write permission to create the .acme.sh in the zimbra home directory. See section below labelled "all in one method" for the workaround or do something like: mkdir /opt/zimbra/.acme.sh; chown zimbra:zimbra /opt/zimbra/.acme.sh as the root user before attempting to install acme.sh as the zimbra user from the /opt/zimbra/.acme.sh directory.
% curl https://get.acme.sh | sh Or: % wget -O - https://get.acme.sh | sh then upgrade to the latest version if not already there: % cd ~/.acme.sh % ./acme.sh --upgrade % ./acme.sh --version v3.0.1 % ./acme.sh --set-default-ca --server letsencrypt % ./acme.sh --set-default-chain --preferred-chain ISRG --server letsencrypt
Note: This will do three things.
- create a directory ~/.acme.sh
- update your .cshrc and .bashrc so that script is in your path
- create a cron job for the local user for automatic renewal
The acme.sh client supports different letsencrypt preferred chains and sslzero free certificates. https://github.com/acmesh-official/acme.sh/wiki/Preferred-Chain and https://zerossl.com/features/acme/
sslzero is now the default certificate for acme.sh as of July 2021 and version 3.0.0 ... These instructions are for the letsencrypt certs so the options are to use a command line option the first time you request a certificate or modify the acme.sh program and change the default ca:
DEFAULT_CA=$CA_LETSENCRYPT_V2 #DEFAULT_CA=$CA_ZEROSSL
or you can use the command line option.
--server --preferred-chain "ISRG"
or you can tell acme.sh to use letsencrypt all the time with version 3.0.0 and above
./acme.sh --set-default-ca --server letsencrypt ./acme.sh --set-default-chain --preferred-chain ISRG --server letsencrypt
After a certificate is issued, you can verify the type of certificate using the command and you should see LetsEncrypt.org under the CA collumn.
./acme.sh --list
Issue Your Certificate
Letsencrypt needs to verify you have control of your domains before they will sign your certificate. To do that, we complete a challenge and prove we have control of the domains using their acme protocol. The acme.sh script supports all challenge methods but for this article we will focus on the Automatic DNS challenge. See https://github.com/Neilpang/acme.sh for other methods or my own documentation https://github.com/JimDunphy/deploy-zimbra-letsencrypt.sh/tree/master/Recipies/SingleServer which lists 3 different type of DNS methods. All challenge methods that acme.sh supports work with this article including --standalone/--tls if you prefer an alternative to the DNS method described here. It is safe to skip to the all in one method in section 6 if you don't care to understand how all this works.
When using the Automatic DNS Method for the first time, you will need to update ~/.acme.sh/account.conf to contain your DNS provider api key. A list of supported DNS providers can be found at ~/.acme.sh/dnsapi. In this article we will use CloudFlare. Login to your CloudFlare account to get your API key before proceeding and then add these 2 lines to your ~/.acme.sh/account.conf file
- SAVED_CF_Key= '......Your API key..........'
- CF_EMAIL='XXXX@example.com'
From now on, anytime we need a certificate or renew a certificate we can do the following:
acme.sh --issue --dns dns_cf -d mail.example.com
If we have multiple domains associated with our Zimbra server, then it works like this:
acme.sh --issue --dns dns_cf -d mail.example.com -d mail.example.net -d mail.example.org
Wild card certs are supported with ACME v2 protocol
acme.sh --issue --dns dns_cf -d example.com -d '*.example.com'
Your certificates can be found at: ~/.acme.sh/mail.example.com ... It uses the first '-d' name to create a directory to store your certificates. If you don't want to use cloudflare, look inside the dnsapi directory for 100's of scripts from various DNS hosting providers. Here is the documentation for many of those scripts. https://github.com/acmesh-official/acme.sh/wiki/dnsapi
Install Certificate With Zimbra
Regardless of which challenge method you used with the acme.sh bash script, the following commands will install it. Note: I have also created a script to perform these steps automatically at https://github.com/JimDunphy/deploy-zimbra-letsencrypt.sh and the forums have a thread on this method https://forums.zimbra.org/viewtopic.php?f=15&t=60781 for additional background information. For this article we walk through those steps.
Step 1 (Append ISRG Root X1 to fullchain)
cd ~/.acme.sh/mail.example.com echo '-----BEGIN CERTIFICATE----- MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4 WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+ 0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ 3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5 ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq 4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc= -----END CERTIFICATE-----' >> fullchain.cer
Because zmcertmgr will chdir during install which can abort when permissions are incorrect in some circumstances, we do the following.
% cd ~.acme.sh/mail.example.com % cp mail.example.com.key mail.example.com.cer fullchain.cer /tmp
Note: For version 8.7 and above, zmcertmgr runs as zimbra. For all earlier versions you will run zmcertmgr as root. Example below is for 8.7 and 8.8+ versions.
Step 2 Verify your certificate
% su - zimbra % cd /tmp % /opt/zimbra/bin/zmcertmgr verifycrt comm mail.example.com.key mail.example.com.cer fullchain.cer
If there were no errors, you can install the certificate
Step 3 Install your certificate
% su - zimbra % cd /tmp % cp mail.example.key /opt/zimbra/ssl/zimbra/commercial/commercial.key % /opt/zimbra/bin/zmcertmgr deploycrt comm mail.example.com.cer fullchain.cer
If there were no errors, proceed to restart zimbra
Step 4 Restart Zimbra
% su - zimbra % zmcontrol restart
All in One Method (Simplest)
Once you understand how to issue your certificates and install acme.sh, you can use the --deploy and --deploy-hook options and have acme.sh perform the zimbra installation for you. This method requires you install and run the acme.sh bash script as the zimbra user and will also handle the intermediate certificate for you during your certificate installation to zimbra. Note: If you leave the crontab entry, all subsequent renewals including the loading of the certificate with zimbra will happen automatically for future unattended renewals approximately every 60 days. Versions prior to 8.7, need to modify the hook script below so that the two zmcertmgr commands are run as root.
- Copy the hook script below to /opt/zimbra/.acme.sh/deploy/zimbra.sh. You can also get it from here: https://raw.githubusercontent.com/JimDunphy/acme.sh/master/deploy/zimbra.sh
#!/bin/bash
# Zimbra Assumptions:
# 1) acme.sh is installed as Zimbra
# 2) see: https://wiki.zimbra.com/wiki/JDunphy-Letsencrypt
# 3) --preferred-chain "ISRG" or are using this chain
######## Public functions #####################
#domain keyfile certfile cafile fullchain
zimbra_deploy() {
_cdomain="$1"
_ckey="$2"
_ccert="$3"
_cca="$4"
_cfullchain="$5"
_debug _cdomain "$_cdomain"
_debug _ckey "$_ckey"
_debug _ccert "$_ccert"
_debug _cca "$_cca"
_debug _cfullchain "$_cfullchain"
# Zimbra's still needs CA pem to verify on some versions
ISG_X1="$(dirname "$_cca")/../ISG_X1.pem"
_debug ISG_X1 "$ISG_X1"
# grab root pem if we don't have it
if [ ! -f "$ISG_X1" ]; then
_debug No "$ISG_X1"
wget -q "https://letsencrypt.org/certs/isrgrootx1.pem.txt" -O "$ISG_X1" || return 1
fi
# append root pem so verifycrt can walk the chain
cat "$_cfullchain" "$(dirname "$_cca")/../ISG_X1.pem" > "${_cca}.real"
/opt/zimbra/bin/zmcertmgr verifycrt comm "$_ckey" "$_ccert" "${_cca}.real" || return 1
#if it verifies we can deploy it
/bin/logger -p local2.info NETWORK "Certificate has been Renewed for $_cdomain"
cp -f "$_ckey" /opt/zimbra/ssl/zimbra/commercial/commercial.key
/opt/zimbra/bin/zmcertmgr deploycrt comm "$_ccert" "${_cca}.real" || return 1
#/opt/zimbra/bin/ldap restart
#/opt/zimbra/bin/zmmailboxdctl reload
#/opt/zimbra/bin/zmproxyctl reload
#/opt/zimbra/bin/zmmtactl reload
/opt/zimbra/bin/zmcontrol restart
return 0
}
Complete example including installation of acme.sh bash script.
$ su - # mkdir /opt/zimbra/.acme.sh; chown zimbra:zimbra /opt/zimbra/.acme.sh # su - zimbra % cd /opt/zimbra/.acme.sh % wget -O - https://get.acme.sh | sh
issue the certificates
% acme.sh --issue --dns dns_cf -d mail.example.com -d mail.example.net -d mail.example.org
install the certificates
% ./acme.sh --issue --deploy --deploy-hook zimbra --dns dns_cf -d mail.example.com -d mail.example.net -d mail.example.org or % ./acme.sh --deploy --deploy-hook zimbra -d mail.example.com
Note: if you get an error attempting to install acme.sh as the zimbra user, do this as /opt/zimbra is owned by root. Switch to root before switching back to zimbra. Here is an example:
% su - # cd /opt/zimbra/ # mkdir .acme.sh # chown zimbra:zimbra .acme.sh # su - zimbra % cd /opt/zimbra/.acme.sh % wget -O - https://get.acme.sh | sh
Subsequent renewals are performed automatically every 60 days via a cron entry for acme.sh like this. Note: If it isn't time, the script will tell you to use the --force option.
% "/opt/zimbra/.acme.sh"/acme.sh --cron --home "/opt/zimbra/.acme.sh" or % "/opt/zimbra/.acme.sh"/acme.sh --force --cron --home "/opt/zimbra/.acme.sh"
It can also be performed manually like this (issue the cert and if successful then deploy the certificate):
% su - zimbra % cd .acme.sh % ./acme.sh --issue --dns dns_cf -d mail.example.com -d mail.example.net -d mail.example.org % ./acme.sh --deploy --deploy-hook zimbra -d mail.example.com
Note: You don't need to worry about the intermediate certificate described above as the deploy-hook handles this automatically including its fetch. The hook will be called on your successful certificate verification and restart/reload zimbra. If it fails to renew the certificate, the hook will not be called. While the automatic dns method is shown above, any of the challenge methods that acme.sh supports can be used.
- CAVEAT *****: If you are running an end of life operating system like centos 6 later than Sep 30, 2021... the wget will fail when retrieving the cert... add --no-check-certificate to the wget in zimbra.sh and remove ISG_X1.pem and try again to work around this problem. You could also copy the certificate and put it in ISG_X1.pem manually.
Pro Tip: look into the --challenge-alias option with the automatic DNS method to further isolate/secure your zone updates with letsencrypt. You only require a CNAME entry for your trusted zimbra domains for the domains above. In other words, each letsencrypt secured zimbra domain would have this in their zone file. Same entry for every one.
_acme-challenge IN CNAME _acme-challenge.adifferentCFzone.com.
where adifferentCFzone.com is a completely different and managed zone from a DNS provider that has an API such as cloudflare (CF) and not a zimbra domain. It can be any of the supported automatic DNS providers including BIND directly.
Here is how this would look using the CNAME alias where example.com, example.net, and example.org are not managed by CF (cloudflare) but we want to secure for zimbra:
% su - zimbra % cd .acme.sh % ./acme.sh --issue --dns dns_cf --challenge-alias adifferentCFzone.com -d mail.example.com -d mail.example.net -d mail.example.org % ./acme.sh --issue --deploy --deploy-hook zimbra --d mail.example.com
Script to Notify of Pending Renewal
Because this just happens automatically, I use a script to send me an email 24 hours in advance that I have a pending certificate coming up for renewal. The source can be found at:
* https://raw.githubusercontent.com/JimDunphy/ZimbraScripts/master/src/zmcertNotice.sh
Due to bugs in zmcontrol restart, it is wise to verify that all sub-systems came back up. Zimbra doesn't account for pid wrap so the logic can fail if you are unlucky to hit that condition. ZBUG798 Use my fix for the postfix : https://forums.zimbra.org/viewtopic.php?f=15&t=65332
Notes
Use Certificate Transparency Monitoring to spot malicious certificates. If your DNS provider offers this service (many do), enable that option. For example one can opt into Cloudflare Monitoring (works even with free accounts), they will send you an email whenever a certificate is issued for one of your domains by crawling the public logs to find any new certificates with domains under your control. This is a great way to watch PKI that is only as secure as the weakest link and there are 100's if not 1000's of certificate authorities out there.
https://blog.cloudflare.com/introducing-certificate-transparency-monitoring/
Zimbra has 4 major daemons that require certificates. nginx, ldap, postfix, and mailboxd... Below is where zmcertmgr installs the certificate. Because mailboxd is java based, it uses a keystore. Note: /opt/zimbra/ssl contains your certificates. The other locations are copies from here. Further: nginx, ldap, and postfix can reload those new certificates hot without shutting down the services so in theory we are performing a restart because mailboxd and taking an outage during certificate renewal.
% ls -lt /opt/zimbra/conf/slapd.* -rw-r----- 1 zimbra zimbra 7213 Aug 4 10:46 slapd.crt -rw-r----- 1 zimbra zimbra 1679 Aug 4 10:46 slapd.key % ls -lt /opt/zimbra/ssl/zimbra/commercial -rw-r----- 1 zimbra zimbra 5030 Aug 4 10:46 commercial_ca.crt -rw-r----- 1 zimbra zimbra 7213 Aug 4 10:46 commercial.crt -rw-r----- 1 zimbra zimbra 1679 Aug 4 10:46 commercial.key % ls -lt /opt/zimbra/conf/nginx.??? -rw-r----- 1 zimbra zimbra 7213 Aug 4 10:46 /opt/zimbra/conf/nginx.crt -rw-r----- 1 zimbra zimbra 1679 Aug 4 10:46 /opt/zimbra/conf/nginx.key % -l /opt/zimbra/conf/smtpd.??? -rw-r----- 1 zimbra zimbra 7213 Aug 4 10:46 /opt/zimbra/conf/smtpd.crt -rw-r----- 1 zimbra zimbra 1679 Aug 4 10:46 /opt/zimbra/conf/smtpd.key % ls -l /opt/zimbra/mailboxd/etc/keystore -rw-r----- 1 zimbra zimbra 4965 Aug 4 10:46 /opt/zimbra/mailboxd/etc/keystore % ls -l /opt/zimbra/ssl/zimbra/jetty.pkcs12 -rw-r----- 1 zimbra zimbra 6952 Aug 4 10:46 /opt/zimbra/ssl/zimbra/jetty.pkcs12
Bad Certificate Recovery
Should you receive an error with your new certificates because they were not validated correctly you can recover by re-issuing your certificate and then re-install to zimbra. Should zimbra not allow you to re-install the corrected certificates, issue a self signed as a quick workaround before proceeding to re-install your corrected letsencrypt certs. Ref: https://wiki.zimbra.com/wiki/Administration_Console_and_CLI_Certificate_Tools#Single-Node_Self-Signed_Certificate Ref: https://forums.zimbra.org/viewtopic.php?f=15&t=64882&p=285958&hilit=zmcertmgr+deploycrt#p285958
Confirm that your SSL certs are all valid and not-expired
% /opt/zimbra/bin/zmcertmgr viewdeployedcrt all - ldap: /opt/zimbra/conf/slapd.crt notBefore=Oct 27 18:10:32 2018 GMT notAfter=Jan 25 18:10:32 2019 GMT subject= /CN=mail.example.com issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3 SubjectAltName=mail.example.com, mail.example.net, tmail.example.com - mailboxd: /opt/zimbra/mailboxd/etc/mailboxd.pem notBefore=Oct 27 18:10:32 2018 GMT notAfter=Jan 25 18:10:32 2019 GMT subject= /CN=mail.example.com issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3 SubjectAltName=mail.example.com, mail.example.net, tmail.example.com - mta: /opt/zimbra/conf/smtpd.crt notBefore=Oct 27 18:10:32 2018 GMT notAfter=Jan 25 18:10:32 2019 GMT subject= /CN=mail.example.com issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3 SubjectAltName=mail.example.com, mail.example.net, tmail.example.com - proxy: /opt/zimbra/conf/nginx.crt notBefore=Oct 27 18:10:32 2018 GMT notAfter=Jan 25 18:10:32 2019 GMT subject= /CN=mail.example.com issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3 SubjectAltName=mail.example.com, mail.example.net, tmail.example.com
acme.sh can also tell you when renewal would occur if you have this automated via the supplied crontab entry.
./acme.sh --list Main_Domain KeyLength SAN_Domains CA Created Renew example.com "" www.example.com LetsEncrypt.org Mon Sep 6 16:36:38 UTC 2021 Fri Nov 5 16:36:38 UTC 2021 example.us "" www.example.us,www2.example.com LetsEncrypt.org Wed Sep 8 17:11:56 UTC 2021 Sun Nov 7 17:11:56 UTC 2021 example.net "" www.example.net,www.example.net,db.example.net LetsEncrypt.org Fri Sep 10 07:05:27 UTC 2021 Tue Nov 9 07:05:27 UTC 2021
More articles written by me, https://wiki.zimbra.com/wiki/JDunphy-Notes
Upgrade acme.sh to latest version
As of July 2021, acme.sh now defaults to zerossl. Run this command to put the defaults back
./acme.sh --set-default-ca --preferred-chain "ISRG" --server letsencrypt
For older versions, this can be run to put back the defaults to letsencrypt. Note: The cert would now default to ISRG Root X1 with either method. I prefer the --set-default-ca method.
% cd ~/.acme.sh % ./acme.sh --upgrade % cat switch_default_acme.sh #!/bin/bash # # acme.sh changed the defaults from letsencrypt to zerossl. # switch back # 10/4/2021 - jad # # Performs this action: # DEFAULT_CA=$CA_LETSENCRYPT_V2 # #JAD#DEFAULT_CA=$CA_ZEROSSL # acme_sh_in=~/.acme.sh/acme.sh grep -q JAD $acme_sh_in if [ $? -eq 1 ]; then cp $acme_sh_in $acme_sh_in.bak echo "Appying patch to $acme_sh_in" sed -i -e 's/DEFAULT_CA=$CA_ZEROSSL/#JAD#DEFAULT_CA=$CA_ZEROSSL\nDEFAULT_CA=$CA_LETSENCRYPT_V2/g' $acme_sh_in else echo "nothing to patch" fi % ./switch_default_acme.sh Appying patch to /opt/zimbra/.acme.sh/acme.sh % ./switch_default_acme.sh nothing to patch
