RemoteManager exception
Remote Manager Exception
Problem
While accessing the email queue or monitoring graphs and other services at times server gives the exception saying "Server error encountered" with the system failure exception during auth. mailbox.log would have similar exception. Keyword to look for is RemoteManager
com.zimbra.common.service.ServiceException: system failure: exception during auth {RemoteManager: mail.domain.com->zimbra@mail.domain.com:22} ExceptionId:qtp1068934215-357:https:https ://mail.domain.com:7071/service/admin/soap/GetMailQueueRequest: Code:service.FAILURE at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:286) at com.zimbra.cs.rmgmt.RemoteManager.getSession(RemoteManager.java:209) at com.zimbra.cs.rmgmt.RemoteManager.execute(RemoteManager.java:139) at com.zimbra.cert.GetCert.addCertsOnServer(GetCert.java:112) at com.zimbra.cert.GetCert.handle(GetCert.java:75)
Caused by: java.io.IOException: There was a problem while connecting to mail.domain.com:22 at ch.ethz.ssh2.Connection.connect(Connection.java:699) at ch.ethz.ssh2.Connection.connect(Connection.java:490) at com.zimbra.cs.rmgmt.RemoteManager.getSession(RemoteManager.java:200) ... 59 more
Solution
Regenerate the SSH key and make sure auth keys get updated in all the servers.
Regenerating Keys
To regenerate the ssh keys, on all hosts (as the zimbra user):
zmsshkeygen
To deploy the keys, on all hosts (as the zimbra user):
zmupdateauthkeys
Verifying sshd configuration
The authentication method assumes that sshd on the mta is running on port 22, and that RSA Authentication is enabled. You can test the ssh command with:
ssh -i .ssh/zimbra_identity -o strictHostKeyChecking=no zimbra@MAIL.DOMAIN.COM
(Swap MAIL.DOMAIN.COM for your hostname, as it appears in the error).
You should NOT be prompted for a password; if you are, recreate the ssh keys and retry the test.
If you're not running sshd on port 22, modify the zimbraRemoteManagementPort attribute on the server:
zmprov ms MAIL.DOMAIN.COM zimbraRemoteManagementPort 2222
Verify in /etc/sshd_config (or /etc/ssh/sshd_config) that the zimbra user is an allow user
AllowUsers admin zimbra
Note: applying this change resulted in not being to ssh as root. Should we add root to the list of AllowUsers!
As soon this fix you should be able to see the following logging in mailbox.log while accessing the queue in the monitoring tab of admin console.
yyyy-mm-dd 11:20:32,654 INFO [{RemoteManager: mail.domain.com->zimbra@mail.domain.com:22}-zmqstat deferred] [] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/data/tmp/432bf1dc-9e76-43c6-b508-49e7cbeee6bc-deferred yyyy-mm-dd 11:20:33,285 INFO [{RemoteManager: mail.domain.com->zimbra@mail.domain.com:22}-zmqstat deferred] [] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/data/tmp/432bf1dc-9e76-43c6-b508-49e7cbeee6bc-deferred yyyy-mm-dd 11:20:35,225 INFO [qtp998351292-10080:https:https ://mail.domain.com:7071/service/admin/soap/GetMailQueueRequest] [name=admin@mail.domain.com;mid=1;ip=10.15.33.238;port=57206;ua=ZimbraWebClient - FF59 (Linux);soapId=b7e88ac;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/data/tmp/432bf1dc-9e76-43c6-b508-49e7cbeee6bc-deferred yyyy-mm-dd 11:20:35,374 INFO [qtp998351292-10080:https:https ://mail.domain.com:7071/service/admin/soap/GetMailQueueRequest] [name=admin@mail.domain.com;mid=1;ip=10.15.33.238;port=57206;ua=ZimbraWebClient - FF59 (Linux);soapId=b7e88ac;] soap - GetMailQueueRequest elapsed=3426 yyyy-mm-dd 11:20:39,614 INFO [qtp998351292-10075:https:https ://mail.domain.com:7071/service/admin/soap/GetMailQueueRequest] [name=admin@mail.domain.com;mid=1;ip=10.15.33.238;port=57206;ua=ZimbraWebClient - FF59 (Linux);soapId=b7e88ad;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/data/tmp/432bf1dc-9e76-43c6-b508-49e7cbeee6bc-deferred yyyy-mm-dd 11:20:39,619 INFO [qtp998351292-10075:https:https ://mail.domain.com:7071/service/admin/soap/GetMailQueueRequest] [name=admin@mail.domain.com;mid=1;ip=10.15.33.238;port=57206;ua=ZimbraWebClient - FF59 (Linux);soapId=b7e88ad;] soap - GetMailQueueRequest elapsed=3009 yyyy-mm-dd 11:20:39,741 INFO [qtp998351292-10078:https:https ://mail.domain.com:7071/service/admin/soap/GetMailQueueInfoRequest] [name=admin@mail.domain.com;mid=1;ip=10.15.33.238;port=57208;ua=ZimbraWebClient - FF59 (Linux);soapId=b7e88ab;] soap - GetMailQueueInfoRequest elapsed=7822 yyyy-mm-dd 11:20:40,560 INFO [qtp998351292-10076:https:https ://mail.domain.com:7071/service/admin/soap/GetCertRequest] [name=admin@mail.domain.com;mid=1;ip=10.15.33.238;port=57204;ua=ZimbraWebClient - FF59 (Linux);soapId=b7e889e;] soap - GetCertRequest elapsed=45637 yyyy-mm-dd 11:20:40,852 INFO [qtp998351292-10077:https:https ://mail.domain.com:7071/service/admin/soap/GetMailQueueRequest] [name=admin@mail.domain.com;mid=1;ip=10.15.33.238;port=57208;ua=ZimbraWebClient - FF59 (Linux);soapId=b7e88ae;] index - OpenLuceneIndex impl=NIOFSDirectory,dir=/opt/zimbra/data/tmp/432bf1dc-9e76-43c6-b508-49e7cbeee6bc-deferred yyyy-mm-dd 11:20:40,855 INFO [qtp998351292-10077:https:https ://mail.domain.com:7071/service/admin/soap/GetMailQueueRequest] [name=admin@mail.domain.com;mid=1;ip=10.15.33.238;port=57208;ua=ZimbraWebClient - FF59 (Linux);soapId=b7e88ae;] soap - GetMailQueueRequest elapsed=3