We get a lot of cert issues, and a lot of the people aren't familiar with them. So we've written the following wikis:
Below is a brief general+Zimbra-specific explanation of how they work. We've assumed you know what 'keys' and 'certificates' are. If you don't, go through 'The basics' first, else skip it.
http://www.tldp.org/HOWTO/SSL-Certificates-HOWTO/x64.html (Only 1.2 for now, but I recommend reading the entire thing whenever you're done with this.)
Commercial and Intermediate Certificates
Alice wants a cake, but she doesn't know if she can trust the new baker in town, Carol. So she asks her foodie friend Bob for opinions, and he just happens to be Carol's good friend as well.
Alice has never met Carol, but she knows that if Bob vouches for her, the cakes must be fantastic.
So the trust works in this way:
The common link between Alice and Carol is Bob.
Similarly, a client PC trusts a root Certificate Authority (CA). It has certificates stored signed by the CA, in a place called, well, 'store'. The store stores the certificates of all commercial root CAs, which are implicitly trusted. The certificate is a written agreement of the CA's trust. These are commonly called 'root certificates'. Browsers, OSes, servers (even Zimbra), etc, have their own store, with all the commercial CA root certs already installed in them. Zimbra servers also have their own store.
As long as a remote server has a certificate that is signed by one of these CAs, the client will trust it.
The client doesn't know if the remote server is a trust worthy server, but because the CA vouches for it, it will blindly trust the server.
The server sends the client a certificate that would look something like this:
openssl x509 -text -in /tmp/server.crt | egrep 'Issuer:|Subject:' Issuer: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root CA Subject: C=IN, ST=MAH, L=Pune, O=Example, Inc., CN=server.example.com
'openssl x509 -text -in /tmp/server.crt' is a command that converts the PEM format cert into plain text, and thus human-readable.
We've grepped out only the Subject and Issuer from the entire certificate. The 'Issuer' is the entity issuing the certificate and the 'Subject' is the entity for whom the certificate was issued. So in this case, the issuer is the CA, 'DigiCert Global Root CA' and the subject is the server's hostname (server.example.com)
The hostname can be specific, like mail.example.com, or a wildcard - *.example.com. Wildcard certs can be used on multiple servers, but are more expensive.
Now the client needs to trust the issuer, 'C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root CA', and thus have the CA certificate in its store.
So on the client, we check and it does. You can see the same certificate in Firefox's store.
Go to Preferences > Advanced > Certificates. Over here, click on 'View Certificates', and then 'Authorities'. Scroll down to 'DigiCert' certificates, and you'll see one of them as 'DigiCert Global Root CA'.
This is what it looks like:
openssl x509 -text -in DigiCertGlobalRootCA.crt | egrep 'Issuer:|Subject:' Issuer: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root CA Subject: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root CA
Since this is the certificate we are blindly trusting, the issuer and subject is the same entity, i.e, the root CA.
The negotiation goes like this:
1) Client requests a secure connection (say https)
2) Server responds, and gives a certificate to show it can be trusted. This contains the server's name (the 'Subject' field), as well as proof that it has been signed by the root CA (the 'Issuer' field)
3) The client checks it's store, sees that the CA is trusted, and accepts this certificate.
But what if Bob doesn't know Carol directly? Well he has a friend Jason, who knows Carol. So now we have a 'chain of trust', that looks like:
Bob -> Jason -> Carol
And our pyramid becomes:
Similarly, most server certificates aren't signed directly by the CA. They are signed by entities called 'intermediate authorities', and these trusted by the CA.
Now the client doesn't know, or trusts, these intermediate guys. It only trusts the root CA.
So we have a 'chain of trust:
1) Server has a certificate signed by intermediate authority.
2) The intermediate authority has a certificate that is signed by the CA.
3) The client trusts the CA, and hence trusts the server.
Now in this chain of trust, the chain is important.
Alice trusts Bob, and Bob trusts Jason who trusts Carol. If we remove any one person, or rearrange the order, the chain breaks and the trust breaks.
The client needs to see the chain of trust to accept the connection. If there's any intermediate cert missing, or of course if the root cert vouching for the intermediate certificate is missing, we *will* get an error.
So for example, here's the certificate of mail.zimbra.com:
openssl x509 -text -in /tmp/zimbra_cert | egrep 'Issuer:|Subject:' Issuer: C=US, O=DigiCert Inc, CN=DigiCert Secure Server CA Subject: C=US, ST=Texas, L=Frisco, O=Zimbra, Inc., CN=*.zimbra.com
We do not know who 'DigiCert Secure Server CA' is, and we do not trust it. So mail.zimbra.com also has to provide a certificate that shows who trusts this intermediate authority.
It does, and the chain of trust ends up looking like the following. Note that the issuer of one certificate is always the subject of the next.:
Issuer: C=US, O=DigiCert Inc, CN=DigiCert Secure Server CA Subject: C=US, ST=Texas, L=Frisco, O=Zimbra, Inc., CN=*.zimbra.com -- Issuer: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root CA Subject: C=US, O=DigiCert Inc, CN=DigiCert Secure Server CA -- Issuer: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root CA Subject: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root CA
*.zimbra.com's cert > 'DigiCert Secure Server CA' > 'DigiCert Global Root CA'.
This chain of trust HAS to be there.
And this is where nearly all errors while deploying a commercial certificate come up. The customer sends a CSR, and gets a bundle from the issuing authority. Ideally, this bundle should contain the intermediate certificate, the CA root, and nothing else.
Usually, it contains a whole bunch of intermediate certificates, and sometimes doesn't contain the one needed, or misses out the root CA's cert. To make things worse, one intermediate authority can assign another intermediate authority to generate certs. This leads to a trust chain of two, three, even four levels.
To fix this, follow the chain. Get the server's certificate, and then get its issuer's certificate. Then get that issuer's certificate, and so on till you get the root CA's self-signed cert. The chain may contain more than one intermediate certificate.
Generating and deploying:
When the customer doesn't buy a commercial certificate, we generate a CA and certificate ourselves. These are called self-signed certificates.
The only difference between these an commercial CAs is that these are not inherently trusted by the browser, or any other application. Commercial CAs ensure that their root cert is stored in all browsers, OSes, etc beforehand. This is essentially why we pay them. If we generate our own CA, it'll of course be brand new, and thus not installed in any store. It will be untrusted by all clients, and this is why we get that familiar 'This connection is untrusted!' when trying to connect.
Generating and deploying these is pretty straight forward, and is explained quite well in our wiki -
When we generate the certs, we just create a CA, and use that CA to sign our server certificates.
In a multi-server environment, ALWAYS create the CA and certificate on the ldap master first. Start the server, and ensure it is working perfectly on the master.
After this, use ' /opt/zimbra/bin/zmcertmgr deploycrt self -allserver' to deploy to the other servers. If this fails, or if you're getting cert errors, check if the CA on the ldap master is exactly the same on the other servers. If it isn't, copy the LDAP master's CA to the other servers, and use that to generate new certificates.
Exercise: Install a self-signed Zimbra CA cert in Firefox's store, to avoid getting the 'The connection is untrusted' error.
What happens when you run the commands creating and deploying commands?
Creating CA and creating certs:
When you create a CA and certificate (createca/createcrt), they are saved in the "/opt/zimbra/ssl/zimbra/" directory. If another new certificate is generated, the old one will be automatically backed up in "/opt/zimbra/ssl/zimbra.<date>". This is very useful if someone accidentally creates a new commercial cert request.
The ca is stored under the 'ca/' directory, while the certificate is stored under 'server/'. A jetty.pkcs12 file is also created, and this is just a combination of the certificate key and cert.
With self-signed certs, on deploying the CA, the certificate and key are stored in the global config attributes 'zimbraCertAuthorityCertSelfSigned' and 'zimbraCertAuthorityKeySelfSigned', and the files are copied to "/opt/zimbra/conf/ca/". A symlink is also created with the hash of the certificate -
ln -s ca.crt `openssl x509 -hash -noout -in ca.crt`
With commercial certs there's no need to deploy the CA, as it'll already be present in Zimbra's store
Deploying the server certificate edits the server config attributes zimbraSSLCertificate and zimbraSSLPrivateKey files, as well as installs the certificate for the MTA, proxy. The jetty.pkcs12 and keystore are also generated.
This cert is copied to the following locations:
Now what's 'jetty.pkcs12' and what's a keystore?
keystore: Exactly why the name says - a java keystone stores keys and certificates, and that's about it. When you deploy certificates, java stores them in this file. To see certs here, you can run:
For ZCS version 8.8 above
/opt/zimbra/common/bin/keytool -list -v -keystore /opt/zimbra/mailboxd/etc/keystore
For ZCS version 8.8 below
/opt/zimbra/java/bin/keytool -list -v -keystore /opt/zimbra/mailboxd/etc/keystore
jetty.pkcs12 : As previously mentioned, this too stores key and certificates. This too is a keystore. To see certs here, run:
openssl pkcs12 -info -in /opt/zimbra/ssl/zimbra/jetty.pkcs12
So if you get an error with any of these two, it means for some reason they weren't able to store the certificate and key there. Just redeploying should work, and if it doesn't, check why it isn't able to write there. (Permissions, etc)