JDunphy-Letsencrypt

Letsencrypt - Using acme.sh to Generate Certs

A simplified version of this wiki can be found here: https://wiki.zimbra.com/wiki/JDunphy-LE-Easy

Introduction

Letsencrypt is a free, automated, and open Certificate Authority to generate all your PKI certificates. Instead of installing a development environment like some other Letsencrypt methods, this article describes a single bash script called acme.sh and can be installed and operated without being root. Here is how to get Zimbra up and running with your Letsencrypt certificate. We are describing a DNS challenge method in this document. For further background, see: https://letsencrypt.org/how-it-works/ and this link for rate-limits. Full Documentation for acme.sh is here Letsencrypt has 100's of programs that can generate LE certificates for you. They all work the same way and use the same acme protocol so that certificates can be issued in real-time. Another popular one for Zimbra is: certbot. Choose the one that works best for you.

There are 2 methods described below. The first method is manual and can be followed step by step. The second method is fully automatic and puts the manual steps into a "deploy script" that acme.sh can invoke and do all this automatically for you. It will also handle certificate renewals and restart zimbra automatically every 60 days. The manual method will re-issue certificates every 60 days but you would be responsible for automating the installation of the certificate into zimbra and restarting zimbra.

Install acme.sh

The user that you run the following command will be where the acme.sh script will be installed.

To install acme.sh, do the following:

% curl https://get.acme.sh | sh
Or:
% wget -O -  https://get.acme.sh | sh

At this point, if you ran the command as root, you can expect to see a .acme.sh in your home directory. If you ran this as user zimbra, you have already found out that it failed. The reason is that /opt/zimbra is owned by root and the zimbra user does not have write permission to create the /opt/zimbra/.acme.sh directory. Here is one solution for that problem:

# mkdir /opt/zimbra/.acme.sh
# chown zimbra:zimbra /opt/zimbra/.acme.sh
# su - zimbra
% wget -O -  https://get.acme.sh | sh

The installation of acme.sh has done three things.

1. created a directory .acme.sh in the home directory of the user that installed it
2. updated your .cshrc and .bashrc so that script is in your path
3. created a cron entry for the user for automatic certificate re-issue

Configure acme.sh

Next we have to choose the provider that manages our DNS entries.

% cd ~/.acme.sh
% ls dnsapi

A few of the ones that many people use would be dns_gd (godaddy) or dns_cf (cloudflare). Here we will choose cloudflare which is free. Add the following variables to ~/.acme.sh/account.conf

CF_Key="sdfsdfsdfljlbjkljlkjsdfoiwje"
CF_Email="xxxx@sss.com"
or

CF_Token="xxxx"
CF_Account_ID="xxxx"
CF_Zone_ID="xxxx"

Because acme.sh supports more than letsencrypt signed certificates, we need to do change the defaults for future certificate issue with zimbra.

% cd; cd .acme.sh
% ./acme.sh --set-default-ca  --server letsencrypt
% ./acme.sh  --set-default-chain  --preferred-chain  ISRG  --server letsencrypt

Issue Certificate

acme.sh --issue --keylength 2048 --dns dns_cf -d mail.example.com

If we have multiple domains associated with your Zimbra server, then it works like this:

acme.sh --issue --keylength 2048 --dns dns_cf -d mail.example.com -d mail.example.net -d mail.example.org 

Wild card certs are supported with ACME v2 protocol

acme.sh --issue --keylength 2048 --dns dns_cf -d example.com -d '*.example.com'

Your certificates can be found at: ~/.acme.sh/mail.example.com ... It uses the first '-d' name to create a directory to store your certificates. If you don't want to use cloudflare, look inside the dnsapi directory for 100's of scripts from various DNS hosting providers. Here is the documentation for many of those scripts. https://github.com/acmesh-official/acme.sh/wiki/dnsapi

Method 1: Manual

This will describe how to manually install the certificate previously issued with zimbra. Regardless of which challenge method you used with the acme.sh bash script, the following commands will install it. Note: I have also created a script to perform these steps automatically at https://github.com/JimDunphy/deploy-zimbra-letsencrypt.sh and the forums have a thread on this method https://forums.zimbra.org/viewtopic.php?f=15&t=60781 for additional background information. For this article we walk through those steps.

Step 1: Append ISRG Root X1

cd ~/.acme.sh/mail.example.com 
echo '-----BEGIN CERTIFICATE-----
MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4
WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu
ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY
MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc
h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+
0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U
A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW
T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH
B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC
B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv
KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn
OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn
jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw
qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI
rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV
HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq
hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL
ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ
3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK
NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5
ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur
TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC
jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc
oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq
4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA
mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d
emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=
-----END CERTIFICATE-----' >> fullchain.cer

Because zmcertmgr will chdir during install which can abort when permissions are incorrect in some circumstances, we do the following. Make sure the zimbra user has read permission of these files.

% cd ~.acme.sh/mail.example.com
% cp mail.example.com.key mail.example.com.cer fullchain.cer /tmp

Step 2 Verify your certificate

% su - zimbra
% cd /tmp
% /opt/zimbra/bin/zmcertmgr verifycrt comm mail.example.com.key mail.example.com.cer fullchain.cer

If there were no errors, you can install the certificate

Step 3 Install your certificate

% su - zimbra
% cd /tmp
% cp mail.example.key /opt/zimbra/ssl/zimbra/commercial/commercial.key
% /opt/zimbra/bin/zmcertmgr deploycrt comm mail.example.com.cer fullchain.cer

If there were no errors, proceed to restart zimbra

Step 4 Restart Zimbra

% su - zimbra
% zmcontrol restart

Note: If you go this route, you are still responsible for renewals as there was no automatic verification, installation, and restart of zimbra. You might have a new certificate every 60 days if you followed along and used the dns challenge method but it will not be installed. The next method uses a deploy scripts that does all the above steps and will also handle automatic renewals every 60 days. Letsencrypt certificates will expire every 90 days.

Method 2: Automatic

With this method, we will install and configure acme.sh as the zimbra user. That is important as the deploy script needs to run as zimbra. We have already issued our certificate in the previous steps. It needs to stated again. This method requires that you have installed acme.sh as the zimbra user and issued the certificates as the zimbra user.

Install the following deploy script for acme.sh:

% wget 'https://raw.githubusercontent.com/JimDunphy/acme.sh/master/deploy/zimbra.sh' -O /opt/zimbra/.acme.sh/deploy/zimbra.sh

Now we can use the --deploy option and have acme.sh perform all the steps in the manual steps above for our zimbra installation. The above script looks like this:

#!/bin/bash

# Zimbra Assumptions:
#    1) acme.sh is installed as Zimbra
#    2) see: https://wiki.zimbra.com/wiki/JDunphy-Letsencrypt
#    3) --preferred-chain "ISRG" or are using this chain

########  Public functions #####################

#domain keyfile certfile cafile fullchain
zimbra_deploy() {
  _cdomain="$1"
  _ckey="$2"
  _ccert="$3"
  _cca="$4"
  _cfullchain="$5"

  _debug _cdomain "$_cdomain"
  _debug _ckey "$_ckey"
  _debug _ccert "$_ccert"
  _debug _cca "$_cca"
  _debug _cfullchain "$_cfullchain"

   # Zimbra's still needs CA pem to verify on some versions
   ISG_X1="$(dirname "$_cca")/../ISG_X1.pem"
   _debug ISG_X1 "$ISG_X1"

   # grab root pem if we don't have it
   if [ ! -f "$ISG_X1" ]; then
      _debug No "$ISG_X1"
      wget -q "https://letsencrypt.org/certs/isrgrootx1.pem.txt" -O "$ISG_X1" || return 1
   fi

   # append root pem so verifycrt can walk the chain
   cat "$_cfullchain" "$(dirname "$_cca")/../ISG_X1.pem" > "${_cca}.real"
   /opt/zimbra/bin/zmcertmgr verifycrt comm "$_ckey" "$_ccert" "${_cca}.real" || return 1

   #if it verifies we can deploy it
   /bin/logger -p local2.info NETWORK "Certificate has been Renewed for $_cdomain"
   cp -f "$_ckey" /opt/zimbra/ssl/zimbra/commercial/commercial.key
   /opt/zimbra/bin/zmcertmgr deploycrt comm "$_ccert" "${_cca}.real" || return 1
   #/opt/zimbra/bin/ldap restart
   #/opt/zimbra/bin/zmmailboxdctl reload
   #/opt/zimbra/bin/zmproxyctl reload
   #/opt/zimbra/bin/zmmtactl reload
   /opt/zimbra/bin/zmcontrol restart
   return 0
}

Step 1 - install the certificate

The first -d argument is the directory that contains our certificate. It will be the first name we used when we issued our certificate. So if we did something like:

% ./acme.sh --issue --keylength 2048 --dns dns_cf -d mail.example.com -d mail.anotherdomain.com -d yet.anothername.com

We would do this:

% ./acme.sh --deploy --deploy-hook zimbra -d mail.example.com

Subsequent renewals are performed automatically every 60 days via a cron entry for acme.sh like this. Note: If it isn't time, the script will tell you to use the --force option if you run it from the command line.

% "/opt/zimbra/.acme.sh"/acme.sh --cron --home "/opt/zimbra/.acme.sh" 
or
% "/opt/zimbra/.acme.sh"/acme.sh --force  --cron --home "/opt/zimbra/.acme.sh" 

We could also do it manually like this:

% su - zimbra
% cd .acme.sh
% ./acme.sh --issue --keylength 2048 --dns dns_cf -d mail.example.com -d mail.example.net -d mail.example.org 
% ./acme.sh --deploy --deploy-hook zimbra -d mail.example.com 

When we use the--cron option, it will do the above 2 steps if there are not any errors. It is up to you if you want to use the --cron method or issue the 2 steps manually. This also means that you can copy certifcates around to different machines by copying the mail.example.com direcctory and issue the --deploy option on the remote machine. You do not have to re-issue certificates if SAN's cover all your hosts in the certificate.

Errors

If the certificate fails to renew, the --cron option will not deploy it. If the --deploy script fails to validate the certificate and therefore will not work with zimbra, it will not attempt to install it or change a running zimbra instance. Re-issue your certifcate and try again.

Certificate Tricks

look into the --challenge-alias option with the automatic DNS method to further isolate/secure your zone updates with letsencrypt. You only require a CNAME entry for your trusted zimbra domains for the domains above. In other words, each letsencrypt secured zimbra domain would have this in their zone file. Same entry for every one. If your zimbra instance is on private address space (RFC1918), this method would work for those scenarios.

_acme-challenge           IN CNAME _acme-challenge.adifferentCFzone.com.

where adifferentCFzone.com is a completely different and managed zone from a DNS provider that has an API such as cloudflare (CF) and not a zimbra domain. It can be any of the supported automatic DNS providers including BIND directly.

Here is how this would look using the CNAME alias where example.com, example.net, and example.org are not managed by CF (cloudflare) but we want to secure for zimbra:

% su - zimbra
% cd .acme.sh
% ./acme.sh --issue --keylength 2048 --dns dns_cf --challenge-alias  adifferentCFzone.com -d mail.example.com -d mail.example.net -d mail.example.org 
% ./acme.sh --deploy --deploy-hook zimbra --d mail.example.com 

Script to Notify of Pending Renewal

Because this happens automatically, I use a script to send me an email 24 hours in advance that I have a pending certificate coming up for renewal and reminds me to watch for it. The source can be found at: https://raw.githubusercontent.com/JimDunphy/ZimbraScripts/master/src/zmcertNotice.sh

Notes

Use Certificate Transparency Monitoring to spot malicious certificates. If your DNS provider offers this service (many do), enable that option. For example one can opt into Cloudflare Monitoring (works even with free accounts), they will send you an email whenever a certificate is issued for one of your domains by crawling the public logs to find new issued certificates with domains under your control. Ref: https://blog.cloudflare.com/introducing-certificate-transparency-monitoring/

Where are the Certs Installed?

Zimbra has 4 major daemons that require certificates. nginx, ldap, postfix, and mailboxd... Below is where zmcertmgr installs the certificate. Because mailboxd is java based, it uses a keystore. Note: /opt/zimbra/ssl contains your certificates. The other locations are copies from here. Further: nginx, ldap, and postfix can reload those new certificates hot without shutting down the services so in theory we are performing a restart because mailboxd and taking an outage during certificate renewal.

% ls -lt /opt/zimbra/conf/slapd.*
-rw-r----- 1 zimbra zimbra 7213 Aug  4 10:46 slapd.crt
-rw-r----- 1 zimbra zimbra 1679 Aug  4 10:46 slapd.key
% ls -lt /opt/zimbra/ssl/zimbra/commercial
-rw-r----- 1 zimbra zimbra 5030 Aug  4 10:46 commercial_ca.crt
-rw-r----- 1 zimbra zimbra 7213 Aug  4 10:46 commercial.crt
-rw-r----- 1 zimbra zimbra 1679 Aug  4 10:46 commercial.key
% ls -lt /opt/zimbra/conf/nginx.???
-rw-r----- 1 zimbra zimbra 7213 Aug  4 10:46 /opt/zimbra/conf/nginx.crt
-rw-r----- 1 zimbra zimbra 1679 Aug  4 10:46 /opt/zimbra/conf/nginx.key
%  -l /opt/zimbra/conf/smtpd.???
-rw-r----- 1 zimbra zimbra 7213 Aug  4 10:46 /opt/zimbra/conf/smtpd.crt
-rw-r----- 1 zimbra zimbra 1679 Aug  4 10:46 /opt/zimbra/conf/smtpd.key
% ls -l /opt/zimbra/mailboxd/etc/keystore
-rw-r----- 1 zimbra zimbra 4965 Aug  4 10:46 /opt/zimbra/mailboxd/etc/keystore
% ls -l /opt/zimbra/ssl/zimbra/jetty.pkcs12
-rw-r----- 1 zimbra zimbra 6952 Aug  4 10:46 /opt/zimbra/ssl/zimbra/jetty.pkcs12

Bad Certificate Recovery

Should you receive an error with your new certificates because they were not validated correctly you can recover by re-issuing your certificate and then re-install that certificate to zimbra. This will work even if ldap is down and nothing has started.

# su - zimbra
% ./acme.sh --issue --keylength 2048 --dns dns_cf --challenge-alias  adifferentCFzone.com -d mail.example.com -d mail.example.net -d mail.example.org 
% ./acme.sh --deploy --deploy-hook zimbra --d mail.example.com 

Misc Zimbra Commands

Confirm that your SSL certs are all valid and not-expired

% /opt/zimbra/bin/zmcertmgr viewdeployedcrt all

- ldap: /opt/zimbra/conf/slapd.crt
notBefore=Oct 27 18:10:32 2018 GMT
notAfter=Jan 25 18:10:32 2019 GMT
subject= /CN=mail.example.com
issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
SubjectAltName=mail.example.com, mail.example.net, tmail.example.com
- mailboxd: /opt/zimbra/mailboxd/etc/mailboxd.pem
notBefore=Oct 27 18:10:32 2018 GMT
notAfter=Jan 25 18:10:32 2019 GMT
subject= /CN=mail.example.com
issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
SubjectAltName=mail.example.com, mail.example.net, tmail.example.com
- mta: /opt/zimbra/conf/smtpd.crt
notBefore=Oct 27 18:10:32 2018 GMT
notAfter=Jan 25 18:10:32 2019 GMT
subject= /CN=mail.example.com
issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
SubjectAltName=mail.example.com, mail.example.net, tmail.example.com
- proxy: /opt/zimbra/conf/nginx.crt
notBefore=Oct 27 18:10:32 2018 GMT
notAfter=Jan 25 18:10:32 2019 GMT
subject= /CN=mail.example.com
issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
SubjectAltName=mail.example.com, mail.example.net, tmail.example.com

acme.sh --list

acme.sh can also tell you when renewal would occur if you have this automated via the supplied crontab entry.

./acme.sh --list
Main_Domain   KeyLength  SAN_Domains                                       CA               Created                       Renew
example.com     ""         www.example.com                                 LetsEncrypt.org  Mon Sep  6 16:36:38 UTC 2021  Fri Nov  5 16:36:38 UTC 2021
example.us      ""         www.example.us,www2.example.com                 LetsEncrypt.org  Wed Sep  8 17:11:56 UTC 2021  Sun Nov  7 17:11:56 UTC 2021
example.net     ""         www.example.net,www.example.net,db.example.net  LetsEncrypt.org  Fri Sep 10 07:05:27 UTC 2021  Tue Nov  9 07:05:27 UTC 2021

Upgrade acme.sh to latest version

% cd ~/.acme.sh
% ./acme.sh --upgrade

Remove acme.sh

% cd ~/.acme.sh
% ./acme.sh --uninstall

Verify .cshrc and .bashrc has no alias and /bin/rm -rf ~/.acme.sh if that has not been removed.

More articles written by me, https://wiki.zimbra.com/wiki/JDunphy-Notes