Letsencrypt - Certificate and Signing
A certificate contains a public key and other metadata information like issuer, what the certificate is to be used for, valid dates, etc. Clients such as Browsers, operating system libraries and tools have trust stores with these certificates from Certificate Authorities (CA's). If you are issued a certificate that was signed by an entity that isn't in your trust store than the user will see a warning that the site isn't safe or the browser will block acess to the site. This will become important later as zimbra (zmcertmgr) will attempt to validate the certificate chains before you will be allowed to install it.
Letsencrypt came about in 2015 with its ISRG Root X1 certificate from its major sponsors such as google, amazon, etc to help get more sites encrypted given known threats such as DNS rebinding attacks. This ISRG Root X1 certificate will expire in 2035 as root certificates generally run for 20+ years. Because not all client software had this certificate in its trust store in 2015, they had another and more established CA (IdenTrust) sign their issued certificates. That other certificate was DST Root CA X3 (Formally IdenTrust) and came about in 2000 but expired this year on Sept 30, 2021. Prior to May 2021, the certificate signing looked like this:
<customer certificate> <letsencrypt R3 intermediate><letsencrypt ISRG Root X1> and (default chain) <customer certificate> <letsencrypt R3 intermediate><DST Root CA X3>
Letsencrypt came up with 2 solutions for navigating past Sept 30, 2021. Beginning around May of 2021, they changed the way they sign certificates. They now looked like this:
<customer certificate> <letsencrypt R3 intermediate>< letsencrypt ISRG Root X1> and the default chain <customer certificate> <letsencrypt R3 intermediate>< letsencrypt ISRG Root X1><DST Root CA X3>
When you are issued a certificate from a letsetencrypt acme client such as acme.sh or certbot, the fullchain contains one of these chains depending on the options you used. Note: The last root chain certificate (ISRG Root X1 or DST Root CA X3) is not present as it is assumed to be in your client trust store. For zimbra, that means we will append that before we can verify the chain. That is why you may see 2 or 3 certificates in fullchain.cer depending on the chain you asked for. With Zimbra, zmcertmgr uses openssl to validate and walk these chains so manually adding the ISRG Root X1 or DST Root CA X3 (formally IdenTrust CA) is required before attempting to validate the issued certificate with zmcertmgr. Some web based certificate chain validating tools will add these root certs for you as part of their output (whatsmychaincert.com) if you ask for the root certficate.
Current Default Chain (DST Root CA X3 Signed)
If you don't specify the chain than Letsencrypt will use this chain with the expired Root CA X3. They do this on purpose because most clients have the ISRG Root CA already in their trust store. For older sites like Android version 6 or older that is no longer under support and therefore can not update their trust stores. This chain is preferred if you are running really old clients and for modern clients, it will work without issues as the client will stop at the ISGR Root X1 cert. This is accomplished because they have their ISRG X1 root certificate signed by DST Root CA X3. If your trust store has the ISRG Root X1 certificate (anything modern does) than it works as before. This special ISRG Root X1 Certificate in this chain has an expiration of Sept 30, 2024. It is important to understand this distinction. DST Root CA X3 expired sept 30, 2021 but the ISRG Root X1 signed intermediate Certificate returned in this default chain will expire on Sept 30, 2024. This also means depending on the chain you choose, you will end up with 2 different versions of the ISRG Root X1 certificate. Your certificate that is signed by the default chain is technically signed by 2 trusted root certificates so clients like all browsers, openssl 1.1+ and older android 6 can accept your certificate as trusted.
The alternative chain used by Letsencrypt chain is shorter and expires in 2035 and should be used if your clients connecting to your zimbra installation have been updated in recent years (meaning since 2015). For most users that should be the case given the security implications of not keeping network clients like browsers or MUA updated on a regular basis with today's threat arena. To use this chain, you must specify at the time of certificate issue with your letsencrypt acme client to receive a certificate signed by this chain. For example with certbot you would use this:
For acme.sh users, the syntax is a little different because that client also supports commercial and other free CA's.
acme.sh --set-default-chain --preferred-chain ISRG --server letsencrypt
Problems with the Default Chain
Because the chain is technically signed by an expired root certificate, how can this work? It works because older clients such as Android < 6.0 ignored the expiration dates in trust anchor certificates but would have the DST Root CA X3 in their old trust store and therefore trust it. For modern clients, they will see the ISRG Root X1 certificate which they also have in their trust store and stop validating there given the trusted 1st flag bit is set in that certificate.
Unfortunately, older operating systems running openssl 1.0.2 for examaple need another method because they do validate the dates in trust anchors and this chain will cause them problems depending on the version. A few workarounds have been provide such as having the client remove the DST Root CA X3 CA from their trust store. In that case, validation by the older openssl clients will stop at the valid ISG Root X1 cert provided the trust store has been updated. With newer openssl 1.1.X, the library will take the alternative path and validate from the users certificate to the R3 to the ISG Root X1 and stop and not follow the DST Root CA X3 signed certficate. Unfortunately, for openssl 1.0.2, it can take the other path and not stop at the ISRG Root X1 certificate and eventually fail when it validates the dates in the DST Root CA X3 CA trust anchor certicate.
Problems for those running Zimbra end of life Versions
Zimbra with 8.8.15 Patch 20 and version 9 Patch 13 moved to openssl 1.1.X. If you are running the older openssl, then you might have to update the trust store for you operating system (centos 6, ubuntu 14, etc) or zmcertmgr will fail. An alternative approach is to use acme.sh and use the zeroSSL free certifcates which doesn't have these problems currently or switch to commercial certificates. See the video in the reference section to determine the best work around for openssl 1.02 and earlier versions.
Which letsencrypt chain to run
For most, choosing the ISRG chain will be the best option but knowing the clients that connect to your zimbra installation will dicate the correct chain. In addition to getting a shorter chain with ISRG, you will be good until 2035.
Which Chain signed your Certificate?
With some acme clients, this can be done with the following by looking at the fullchain that is returned during a successful issue from letsencrypt. The acme client (acme.sh, certbot, etc) does not build this file as this is returned by letsencrypt themselves.
$ grep -A 1 BEGIN fullchain.cer -----BEGIN CERTIFICATE----- MIIFTTCCBDWgAwIBAgISBHspU+YismyjljcNO0976JmNMA0GCSqGSIb3DQEBCwUA -----BEGIN CERTIFICATE----- MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
Where the certificate starting with MIIFTTCC is your certificate and the next certificate beginning with MIIFFjCC is the R3 signed certificate by ISRG Root X1. Because we do not have a 3rd BEGIN (indicating another certificate is present) in our fullchain.cer file, we can conclude this certificate is the alternative chain from Letsencrypt and the acme client specified the ISRG option when issuing the certificate. The root CA certificate is never included in the fullchain as the client would have that in their trust store. We add this manually when using zmcertmgr because we are using openssl from the command line and are acting as a client during the verify process.
Production Chain Changes
Chain of Trust
DST Root CA X3 Certicate Expiration Problems and Fix
More articles written by me, https://wiki.zimbra.com/wiki/JDunphy-Notes