Network service error
Problem
- Not able to login to admin and webclient after changing value of any attribute which exceeding the max allowed size.
- Such as zimbraFileUploadMaxSize, zimbraMailContentMaxSize, zimbraMtaMaxMessageSize etc
Such changes usually get neglected and users start facing login issues as follows.
While investigating, the admin may notice the following log entries which are constantly appearing in /opt/zimbra/log/mailbox.log
022-04-27 20:53:19,281 WARN [qtp195615004-782:https://xxx.xxx.xxx.xxx/] [] webclient - system failure: ,: Service Unavailable com.zimbra.common.service.ServiceException: system failure: error while proxying request to target server: Service Unavailable ExceptionId:qtp195615004-782:https://xxx.xxx.xxx.xxx/:1651072999281:2a0061c1a4fb9afe Code:service.FAILURE at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:288)
And, following entries from /opt/zimbra/log/nginx.log
2022/04/27 21:37:58 [error] 211623#0: *5 zm lookup: lookup handler xxx.xxx.xxx.xxx:7072 sent error response: 503 Service Unavailable while SSL handshaking to lookup handler, client: aaa.aaa.aaa.aaa, server: mail.example.tld, request: "POST /service/soap/NoOpRequest HTTP/1.1", host: " xxx.xxx.xxx.xxx", referrer: "https:// xxx.xxx.xxx.xxx/#8" 2022/04/27 21:37:58 [warn] 211623#0: *5 zmauth: an error occurs during zm lookup: , fall back to IPHASH to get the upstream route while SSL handshaking to lookup handler, client: aaa.aaa.aaa.aaa, server: mail.exampl.tld, request: "POST /service/soap/NoOpRequest HTTP/1.1", host: " xxx.xxx.xxx.xxx", referrer: "https:// xxx.xxx.xxx.xxx/#8"
Analysis
- If the issue is unknown, start thinking of all the changes done on the server recently and revert all the changes one-by-one and check which change was affecting the server.(if the changes were done from the CLI, checking bash history will help quickly.)
- If various changes are done from the admin panel but not sure which were they, execute the below command and check all Size related attributes and check whether the size of any attribute is showing exceptionally high.
su - zimbra zmprov -l gacf | grep -i size zmprov -l gs `zmhostname` | grep -i size
- Also, search for the below kind of error in the /opt/zimbra/log/mailbox.log, there will be only one entry of this hence it is possible it may get overlooked while analysing the log file. (instead of zimbraFileUploadMaxSize there could be any attribute hence don't search using the keyword zimbraFileUploadMaxSize)
com.zimbra.cs.account.AccountServiceException: zimbraFileUploadMaxSize must be a valid long: at com.zimbra.cs.account.AccountServiceException.INVALID_ATTR_VALUE(AccountServiceException.java:212)
Once find the root cause do the following
Solution
- Verify the existing value on zimbraFileUploadMaxSize.
zmprov -l gcf zimbraFileUploadMaxSize
- Now change the value to the allowed max limit as follows (Example: let's set to 21MB) and restart mailbox services.
zmprov -l mcf zimbraFileUploadMaxSize 2147483647 zmmailboxdctl restart
Next
- If the above doesn't resolve the issue, NE customers can raise a case with Zimbra support and provide the following details while submitting the case.
- Log files from below location (if having multi-server environment, provide the logs from the respective servers)
- -/opt/zimbra/log/mailbox.log
- -/opt/zimbra/log/ngnx.log
- An output of the following commands.
su - zimbra zmprov -l gacf zmprov -l gs `zmhostname` zmlocalconfig zmcontrol -v
These are quite basic details, that are required to start an investigation on the initial level at least, further, the support engineer may ask for more details If needed.
| Submitted by: Amol Mistry |
