Ajcody-Notes-SSLCerts: Difference between revisions

Latest revision as of 03:41, 21 June 2016

SSL Certificate Issues

KB 2505	Last updated on 2016-06-21	Last updated by Ajcody	0.00 (0 votes)	Verified in: ZCS 8.5 ZCS 8.0 ZCS 7.0
- This is Zeta Alliance Certified Documentation. The content has been tested by the Community.

KB 2505	Last updated on 2016-06-21
0.00 (0 votes)
- This is Zeta Alliance Certified Documentation. The content has been tested by the Community.

- This article is a Work in Progress, and may be unfinished or missing sections.

Actual SSL Certificate Issues Homepage

Please see Ajcody-Notes-SSLCerts

Resources For SSL Certificates

Bug & RFE's Related To SSL

Multiple SSL Certificates Aren't Supported On One Server

"multiple SSL certificates on one server"
- http://bugzilla.zimbra.com/show_bug.cgi?id=8128
  - Allot of private comment details unfortunately.

The -subjectAltNames option is the way to do, if your certificate provider doesn't support then you should inquire into another provider who does.

Recreating Self-Signed SSL Certificates

Other references:

I believe this wiki page has the best instructions. Sorry for the confusion about the state of the wiki pages on ssl certs. We are attempting to get them cleaned up.
- Problem_with_Certificate_can_cause_MTA_Failure#For_Multi-Server:_Run_this_on_all_other_systems_in_the_multi-server_setup
  - Zmcertinstall#Single-Node_Self-Signed_Certificate
    - Has "Multi-Node Self-Signed Certificate" instructions as well.
  - Recreating_a_Self-Signed_SSL_Certificate

Steps I've used for a single ZCS 6 server that WAS NOT using commercial certificates. Also, this is for recreating self-signed certificates and not changing them. This documentation was done specifically for the issue when the self-signed certificates expired and caused upgrade issues.

** have zimbra running - ldap at least - need to double check this
** run as root
** Prep work
mv /opt/zimbra/ssl/zimbra /opt/zimbra/ssl/zimbra_old
** I had used the below, but think the above step is *better*
* mv /opt/zimbra/ssl /root/ssl_old
* mkdir /opt/zimbra/ssl
* chown zimbra:zimbra /opt/zimbra/ssl
** Creating new certs
/opt/zimbra/bin/zmcertmgr createca -new 
/opt/zimbra/bin/zmcertmgr deployca -localonly 
** others have just used: /opt/zimbra/bin/zmcertmgr deployca 
/opt/zimbra/bin/zmcertmgr createcrt self -new 
** others have just used: /opt/zimbra/bin/zmcertmgr createcrt -new
/opt/zimbra/bin/zmcertmgr verifycrt self 
/opt/zimbra/bin/zmcertmgr deploycrt self
** run as zimbra
zmcontrol stop
zmcontrol start

Keystore Password Errors - Server Move Stuff

Seen this with user attempting the 32 > 64 server move wiki page, mailboxd wouldn't start. Errors like:

keytool error: java.io.IOException: Keystore was tampered with, or password was incorrect
*
Zimbra server reserving server socket port=995 bindaddr=null ssl=true
Fatal error: exception while binding to ports
java.net.SocketException: Unbound server sockets not implemented
*
Exception in thread "main" java.io.IOException: Keystore was tampered with, or password was incorrect

And when I attempted to recreate the self-sign cert using the steps above, it ended with this error:

# /opt/zimbra/bin/zmcertmgr deploycrt self
** Saving server config key zimbraSSLCertificate...failed.
** Saving server config key zimbraSSLPrivateKey...failed.
** Installing mta certificate and key...done.
** Installing slapd certificate and key...done.
** Installing proxy certificate and key...done.
** Creating pkcs12 file /opt/zimbra/ssl/zimbra/jetty.pkcs12...done.
** Creating keystore file /opt/zimbra/mailboxd/etc/keystore...failed.

Exception in thread "main" java.io.IOException: Keystore was tampered with, or password was incorrect
	at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:771)
	at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:38)
	at java.security.KeyStore.load(KeyStore.java:1185)
	at com.zimbra.cert.MyPKCS12Import.main(MyPKCS12Import.java:98)
Caused by: java.security.UnrecoverableKeyException: Password verification failed
	at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:769)
	... 3 more

** Installing CA to /opt/zimbra/conf/ca...done.

And the above recreating of self-cert fails because of issues about keystore password.

Fix was to :

[as root]  mv /opt/zimbra/mailboxd/etc/keystore /root/keystore.old

And then run the recreating self-certs above.

Note: this also appears to happen on 8.6, with self-signed certs after rsync migration. Moving the old keystore and redeploying ca/certs corrects the issue.

Contrib from Baylink

(Here's an actual script you can copy to a file and run, by Baylink 16:08, 25 March 2010 (UTC))

echo "Backing up old certs..."
mv /opt/zimbra/ssl/zimbra /opt/zimbra/ssl/zimbra.newcert-`timestamp`

echo "Creating new CA..."
/opt/zimbra/bin/zmcertmgr createca -new

echo "Deploying new CA..."
/opt/zimbra/bin/zmcertmgr deployca -localonly

# added 12 Apr 2010 per tonyp@zimbra.com
echo "Creating new CSR..."
/opt/zimbra/bin/zmcertmgr createcsr self -new -subject "/C=US/ST=N\/A/L=N\/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration Suite/CN=server.company.com" -subjectAltNames "server.company.com,altname.company.com,othername.company.com"

echo "Creating new cert..."
/opt/zimbra/bin/zmcertmgr createcrt -new -subject "/C=US/ST=N\/A/L=N\/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration Suite/CN=server.company.com" -subjectAltNames "server.company.com,altname.company.com,othername.company.com"

echo "Verifying new cert..."
/opt/zimbra/bin/zmcertmgr verifycrt self

echo "Deploying new cert..."
/opt/zimbra/bin/zmcertmgr deploycrt self

echo "Restarting Zimbra server"
su - zimbra -c 'zmcontrol stop; zmcontrol start'

Note that if you're running Exchange ActiveSync with iPhones, the iPhones require that the "server.company.com" name (the primary name) must

be the primary name on the SSL cert (I've tested this) and
resolve to the same IP both inside and outside your firewall which
- requires a firewall that can do "hairpin" inbound-NAT

So, if your mailbox server has a "real" name, then that, and any "role" name you use for ZWC user access must be secondary names, and the Active Sync name (I chose "async") must be primary.

That script calls my 'timestamp' script, which is just:

date +%Y%m%d-%H%M%S

(end contrib)

Ldap And-Or MTA Doesn't Start After Cert Changes Or Upgrade

Brief summary of issues:

The ca directory contained extra links to different certificates. This seemed to not bother 5.0.9 but under 5.0.10 postfix has a fit if there are more then 3 files in that directory.

Example of LDAP error:

[zimbra@server-01 ~]$ zmcontrol start
Host server-01.DOMAIN.com
        Starting ldap...Done.
FAILED
Failed to start slapd.  Attempting debug start to determine error.
TLS: error:0906D06C:PEM routines:PEM_read_bio:no start line pem_lib.c:647
TLS: error:0906D06C:PEM routines:PEM_read_bio:no start line pem_lib.c:647
TLS: error:0906D06C:PEM routines:PEM_read_bio:no start line pem_lib.c:647
TLS: error:0906D06C:PEM routines:PEM_read_bio:no start line pem_lib.c:647
TLS: error:0906D06C:PEM routines:PEM_read_bio:no start line pem_lib.c:647
TLS: error:0906D06C:PEM routines:PEM_read_bio:no start line pem_lib.c:647
TLS: error:0906D06C:PEM routines:PEM_read_bio:no start line pem_lib.c:647
TLS: error:0906D06C:PEM routines:PEM_read_bio:no start line pem_lib.c:647
TLS: error:02001002:system library:fopen:No such file or directory bss_file.c:352
TLS: error:20074002:BIO routines:FILE_CTRL:system lib bss_file.c:354
main: TLS init def ctx failed: -1

The fix is described in:

http://www.zimbra.com/forums/administrators/23104-5-0-10-upgrade-problem.html

Details as related to MTA/Postfix - Error_(MTA):_Unable_to_set_STARTTLS

More details as shared by a customer:

I moved the "ca" directory to "ca.BAK" to back it up, made a new ca directory, and then went through the steps detailed in that post.
Once done, zimbra started up without a problem:

1) Clear all the contents of the /opt/zimbra/conf/ca directory by backing them up
somewhere on disk.
2) Copy the /opt/zimbra/ssl/zimbra/commercial/commercial.key /opt/zimbra/conf/ca/ca.key
3) Copy /opt/zimbra/ssl/zimbra/commercial/commercial.crt /opt/zimbra/conf/ca/ca.pem
4) Create the hash value
ln -f -s ca.pem /opt/zimbra/conf/ca/`openssl x509 -hash -noout -in
/opt/zimbra/conf/ca/ca.pem`.0
5) Chmod 644 /opt/zimbra/conf/ca/*
6) Restart the zmcontrol


A sample of what the /opt/zimbra/conf/ca directory looked like:

lrwxrwxrwx  1 root root   20 Apr  9 20:29 0bb21872.0 -> commercial_ca_26.pem
lrwxrwxrwx  1 root root   20 Apr  9 20:29 0c364b2d.0 -> commercial_ca_14.pem
lrwxrwxrwx  1 root root   20 Apr  9 20:29 0e82f83a.0 -> commercial_ca_36.pem
lrwxrwxrwx  1 root root   20 Apr  9 20:29 11f154d6.0 -> commercial_ca_49.pem
lrwxrwxrwx  1 root root   19 Apr  9 20:29 128b9c8d.0 -> commercial_ca_9.pem
lrwxrwxrwx  1 root root   19 Apr  9 20:29 1a147d5b.0 -> commercial_ca_5.pem
lrwxrwxrwx  1 root root   20 Apr  9 20:29 1bb6c7e0.0 -> commercial_ca_24.pem
lrwxrwxrwx  1 root root   20 Apr  9 20:29 1c647a6d.0 -> commercial_ca_21.pem
lrwxrwxrwx  1 root root   20 Apr  9 20:29 256fd83b.0 -> commercial_ca_33.pem
lrwxrwxrwx  1 root root   20 Apr  9 20:29 25f0cbee.0 -> commercial_ca_23.pem
lrwxrwxrwx  1 root root   20 Apr  9 20:29 2d047263.0 -> commercial_ca_28.pem

There we many more of the above entries, all of which pointed to .pem files that didn't exist.

There was also:
lrwxrwxrwx  1 root root    6 Apr 13 11:43 555ebb99.0 -> ca.pem
lrwxrwxrwx  1 root root   17 Apr  9 20:29 8e6e2991.0 -> commercial_ca.pem
lrwxrwxrwx  1 root root   17 Apr 13 11:43 c33a80d4.0 -> commercial_ca.pem
lrwxrwxrwx  1 root root   20 Apr  9 20:29 c527e4ab.0 -> commercial_ca_57.pem
-rw-r--r--  1 root root  887 Apr 13 11:43 ca.key
-rw-r--r--  1 root root  989 Apr 13 11:43 ca.pem
-rw-r--r--  1 root root 1155 Apr 13 11:43 commercial_ca_1.pem
-rw-r--r--  1 root root 1156 Apr 13 11:43 commercial_ca.pem

zmcertmgr output:

./zmcertmgr deploycrt comm /opt/zimbra/certs/server-01.DOMAIN.com.crt /opt/zimbra/certs/ca_chain-server-01.DOMAIN.com.crt
** Verifying /opt/zimbra/certs/server-01.DOMAIN.com.crt against /opt/zimbra/ssl/zimbra/commercial/commercial.key
Certificate (/opt/zimbra/certs/server-01.DOMAIN.com.crt) and private key (/opt/zimbra/ssl/zimbra/commercial/commercial.key) match.
Valid Certificate: /opt/zimbra/certs/server-01.DOMAIN.com.crt: OK
** Copying /opt/zimbra/certs/server-01.DOMAIN.com.crt to /opt/zimbra/ssl/zimbra/commercial/commercial.crt
** Appending ca chain /opt/zimbra/certs/ca_chain-server-01.DOMAIN.com.crt to /opt/zimbra/ssl/zimbra/commercial/commercial.crt
** Saving server config key zimbraSSLCertificate...done.
** Saving server config key zimbraSSLPrivateKey...done.
** Installing mta certificate and key...done.
** Installing slapd certificate and key...done.
** Installing proxy certificate and key...done.
** Creating pkcs12 file /opt/zimbra/ssl/zimbra/jetty.pkcs12...done.
** Creating keystore file /opt/zimbra/mailboxd/etc/keystore...done.
** Installing CA to /opt/zimbra/conf/ca...done.

Moving Your Certificates To New Or Another Server

Please see Transfer_SSL_certificates_between_servers

Commercial Cert Error - Subject Does Not Start With /

As reported by a customer to me:

When creating a commercial cert for a server the zmcertmgr will fail if you don't supply a subjectAltName ---- Took me awhile to figure this out since the error isn't correctly describing the problem. It says that "Subject does not start with '/'." Which is incorrect. Subject does start with "/" , it's the subjectAltName that was needed. After I supplied this name, it generated the csr. Here are my commands for your own reference.

I modified the Some* entries below.

/opt/zimbra/bin/zmcertmgr createcsr comm -new -subject "/C=US/ST=SomeState/L=SomeCity/O=Some Community College/OU=ITS/CN=zimbra.somecommunitycollege.edu" -subjectAltNames zimbra.somecommunitycollege.edu

then I went to thawte and applied for a ssl cert.

How To Setup Certs With CACert.org - Free Certs

CaCert.Org References

Free Certs with http://www.cacert.org/

How-To (tested on 5.0.2)

Note, the following :

su - root ; cd /opt/zimbra/ssl/zimbra/commercial/

included in all steps in case someone is skipping through instructions.

Clean up and start fresh

su - root
cd /opt/zimbra/ssl/zimbra/commercial/
tar -czvf /tmp/ssl.commercial.backup.tar.gz *
rm -rf *

Generate new csr

[ su - root ; cd /opt/zimbra/ssl/zimbra/commercial/ ]
/opt/zimbra/bin/zmcertmgr createcsr comm -new

This uses the defaults, note the items to change.

/opt/zimbra/bin/zmcertmgr createcsr comm -new "/C=US/ST=TX/L=Somewhere/O=Company, Inc./OU=ITDepartment/CN=mail.CHANGEME.com"

Confirm

[ su - root ; cd /opt/zimbra/ssl/zimbra/commercial/ ]
ls -la

There should only be two files and time/date should match - commercial.csr & commercial.key

cat /opt/zimbra/ssl/zimbra/commercial/commercial.csr
 -----BEGIN CERTIFICATE REQUEST-----
[delete]CCAWwCAQAwgZkxCzAJBgNVBAYTAlVTMQwwCgYDVQQIEwNOL0ExDDAKBgNV
[delete]4vQTEjMCEGA1UEChMaWmltYnJhIENvbGxhYm9yYXRpb24gU3VpdGUxIzAh
[delete]AsTGlppbWJyYSBDb2xsYWJvcmF0aW9uIFN1aXRlMSQwIgYDVQQDExttYWls
[delete]nRlcm5hbC5ob21ldW5peC5jb20wgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJ
[delete]AOri9/m6RtM1vASBROPgLvkUYybwf2WDI2xTdKUuAMI0rTpMH1IzjPRP/J+m
[delete]RQTiJe1mRX3rJCy3qVooVzsLe2yJ1+rs3FzLSfQhazK6PqMD8GhpqHO0Y75
[delete]LEA/qdOCrTFjosO9C3j3WPCW8lutTxf/QsoKGkIVs5tjAgMBAAGgKTAnBgkq
[delete]0BCQ4xGjAYMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgXgMA0GCSqGSIb3DQEB
[delete]A4GBAKMLVFilRjI9xvU/vZmP69yReVZyxa5YVpF/cEvwFwbOU6E4USkdONGT
[delete]DRj1XxfzYD+CDf8TVuTY4tapaLvKPRUtdd/mM1PidY5t126QAObyKjHBRzy
[delete]RJFQeP+0ktxcYJ99+sfiescwR/qzPJM58i6daqmMamQBZi
  -----END CERTIFICATE REQUEST-----

Sign up for cacert.org use

Goto http://www.cacert.org/

Sign up - https://www.cacert.org/index.php?id=1

Verify the email the confirmation email.

Add a domain , it will send an email to some "admin" account to the domain. Make sure you can get it before you do this.

Get New Server Certificate

Now do a New Server Certificate from your administration page at http://www.cacert.org/

You'll copy in the /opt/zimbra/ssl/zimbra/commercial/commercial.csr contents in window [all of it].

It'll generate your cert. on the webpage.

Copy this onto the server [paste in cert details]:

vi /opt/zimbra/ssl/zimbra/commercial/commercial.crt
-----BEGIN CERTIFICATE-----
[delete]TCCAl2gAwIBAgIDBPRRMA0GCSqGSIb3DQEBBQUAMHkxEDAOBgNVBAoTB1Jv
[delete]0ExHjAcBgNVBAsTFWh0dHA6Ly93d3cuY2FjZXJ0Lm9yZzEiMCAGA1UEAxMZ
[delete]2VydCBTaWduaW5nIEF1dGhvcml0eTEhMB8GCSqGSIb3DQEJARYSc3VwcG9y
[delete]NlcnQub3JnMB4XDTA4MDQxMTIwMjQzNloXDTA4MTAwODIwMjQzNlowJjEk
[delete]1UEAxMbbWFpbDMuaW50ZXJuYWwuaG9tZXVuaXguY29tMIGfMA0GCSqGSIb3
[delete]AQUAA4GNADCBiQKBgQDq4vf5ukbTNbwEgUTj4C75FGMm8H9lgyNsU3SlLgDC
[delete]B9SM4z0T/yfpoZc3yUUE4iXtZkV96yQst6laKFc7C3tsidfq7Nxcy0n0IWs
[delete]BoaahztGO+ZgsfCxAP6nTgq0xY6LDvQt491jwlvJbrU8X/0LKChpCFbOb
[delete]QABo4HcMIHZMAwGA1UdEwEB/wQCMAAwNAYDVR0lBC0wKwYIKwYBBQUHAwIG
[delete]QUFBwMBBglghkgBhvhCBAEGCisGAQQBgjcKAwMwCwYDVR0PBAQDAgWgMDMG
[delete]QUFBwEBBCcwJTAjBggrBgEFBQcwAYYXaHR0cDovL29jc3AuY2FjZXJ0Lm9y
[delete]QYDVR0RBEowSIIbbWFpbDMuaW50ZXJuYWwuaG9tZXVuaXguY29toCkGCCsG
[delete]wgFoB0MG21haWwzLmludGVybmFsLmhvbWV1bml4LmNvbTANBgkqhkiG9w0B
[delete]AAOCAgEANzr/jRcEd5BF2QqF+X8deq4Xxp1tN9lFgji38C5ixNJ+Busq9Sk5
[delete]O7YYJQbSf5K14eZyC1jaNEOEwqgzFiM1HRWL1HCca3EM7TXUoH8sMXS1Ng
[delete]M5oyfQcFWZYa22CtKQANQEX5l7EYNkr0yvD/YnP02l3hk1jZr+3pszCW6Iw
[delete]vabHMYcAXus+iOGgws788QsMaqzoZwla1AaacZ98s0lFAR0xdRiuXCHUFz
[delete]meS5sK+med95/z+Mb6ShJzC7KAi1nfZk9CoNHUHVxMis5Cr+GT7MoIvhQ
[delete]8fkiANQQoEgam37lyHezPKyc6iLxW4ag2PWKrZa2+3pyTg/6aHKxZR325z
[delete]kcdwKYo/eUGaN1tNmsY638N4hCz01FHHKr97W0m4u5wtwKBo4/5Gy9e5nG6
[delete]khOyjfOz6VYvZHNqDaqGJwsxitxSGGDc8bA+9d73RCOFuztwVrKYg5OJ
[delete]Ei5C9gWzee7AmoGpgxOrYjgBrx4nuBw71EFzgKSOZqxUxSNiLuGAx+oVd2
[delete]Z4EAPsa90ZNb0mLGagAuTAdccekOqPVnyZrqiINelY7fpAAUvO9rgTSB9A
[delete]RxUydTgY1jyBtoXjp59HMVbCkAtOtX43NqIhPYJNPeSoyw/5SU=
-----END CERTIFICATE-----

Get Root CA's

Root CA certs are found here : https://www.cacert.org/index.php?id=3

Do the following on the server:

[ su - root ; cd /opt/zimbra/ssl/zimbra/commercial/ ]
wget http://www.cacert.org/certs/root.crt
mv root.crt commercial_ca.crt

Verify

Let's verify all is good.

[ su - root ; cd /opt/zimbra/ssl/zimbra/commercial/ ]
/opt/zimbra/bin/zmcertmgr verifycrt comm commercial.key commercial.crt

Giving something like this:

** Verifying commercial.crt against commercial.key
Certificate (commercial.crt) and private key (commercial.key) match.
Valid Certificate: commercial.crt: OK

Errors - Double check time

Date [is it wrong?]
- Install ntpd if it's not
- Stop ntpd :
  - ```
  /etc/init.d/ntpd stop
```
- Set time with :
  - ```
  ntpdate us.pool.ntp.org
```
- Confirm time change :
  - ```
  date
```
- Confirm hardware clock time :
  - ```
  hwclock
```
- Sync hardware clock time :
  - ```
  hwclock --systohc
```
- Confirm hardware clock time :
  - ```
  hwclock
```
- Start ntpd now :
  - ```
  /etc/init.d/ntpd start
```

Deploy CA

[ su - root ; cd /opt/zimbra/ssl/zimbra/commercial/ ]

/opt/zimbra/bin/zmcertmgr deploycrt comm commercial.crt commercial_ca.crt

Restart the webserver

su - zimbra
zmmailboxdctl restart