Zimbra webclient and Admin Console not accessible due to jetty asm

 - This article is a Work in Progress, and may be unfinished or missing sections.

Zimbra webclient and Admin Console not accessible due to jetty asm-7.0 jar files

Issue

This issue is occuring after upgrade to the 8.8.15.p40 patch/9.0.0.p33 patch. This issue is due to asm*7.0* jar files which are part of the jetty-distribution-9.4.18.v20190429. asm*7.0* jar files should be removed as part of jetty upgrade (current version jetty-distribution-9.4.46.v20220331-2), somehow asm*7.0* jar files not removed from the server and this is not case with all. It is depends on the history of the server for ex : https://forums.zimbra.org/viewtopic.php?p=309899#p309899.

All services are shown to be running but when a user try's to login a message saying "A network service error has occurred" appears, also the admin page shows up blank when we try to access it through the admin url. Inspecting the page shows 503 (service Unavailable) errors.

Zimbra webclient and Admin Console not accessible and getting below logs in /opt/zimbra/log/mailbox.log

2023-05-31 22:47:55,838 ERROR [qtp439928219-42:https://mail.example.com/] [] webclient - Unable to get domain config
com.zimbra.common.service.ServiceException: error while proxying request to target server: Service Unavailable

2023-05-31 18:53:27,660 WARN  [qtp439928219-23:https://mail.example.com/] [] webclient - system failure: error while proxying request to target server: Service Unavailable

com.zimbra.common.service.ServiceException: system failure: error while proxying request to target server: Service Unavailable

exceptions in /opt/zimbra/log/zmmailboxd.out

Caused by:
java.lang.RuntimeException: Error scanning entry org/aopalliance/aop/Advice.class from jar file:///opt/zimbra/jetty_base/webapps/service/WEB-INF/lib/spring-aop-6.0.8.jar
at org.eclipse.jetty.annotations.AnnotationParser.lambda$parseJar$2(AnnotationParser.java:880)
at java.base/java.util.TreeMap$ValueSpliterator.forEachRemaining(TreeMap.java:3215)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
at org.eclipse.jetty.annotations.AnnotationParser.parseJar(AnnotationParser.java:872)
at org.eclipse.jetty.annotations.AnnotationParser.parse(AnnotationParser.java:835)
at org.eclipse.jetty.annotations.AnnotationConfiguration$ParserTask.call(AnnotationConfiguration.java:159)
at org.eclipse.jetty.annotations.AnnotationConfiguration$1.run(AnnotationConfiguration.java:455)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
 
Caused by:
java.lang.IllegalArgumentException: Unsupported class file major version 61
at org.objectweb.asm.ClassReader.<init>(ClassReader.java:184)
at org.objectweb.asm.ClassReader.<init>(ClassReader.java:166)
at org.objectweb.asm.ClassReader.<init>(ClassReader.java:152)
at org.objectweb.asm.ClassReader.<init>(ClassReader.java:273)
at org.eclipse.jetty.annotations.AnnotationParser.scanClass(AnnotationParser.java:932)
at org.eclipse.jetty.annotations.AnnotationParser.parseJarEntry(AnnotationParser.java:917)
at org.eclipse.jetty.annotations.AnnotationParser.lambda$parseJar$2(AnnotationParser.java:876)
at java.base/java.util.TreeMap$ValueSpliterator.forEachRemaining(TreeMap.java:3215)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
at org.eclipse.jetty.annotations.AnnotationParser.parseJar(AnnotationParser.java:872)
at org.eclipse.jetty.annotations.AnnotationP

Solution:

Delete below files

/opt/zimbra/common/jetty_home/lib/annotations/asm-7.0.jar
/opt/zimbra/common/jetty_home/lib/annotations/asm-analysis-7.0.jar
/opt/zimbra/common/jetty_home/lib/annotations/asm-commons-7.0.jar
/opt/zimbra/common/jetty_home/lib/annotations/asm-tree-7.0.jar
/opt/zimbra/common/jetty_home/lib/annotations/javax.annotation-api-1.3.jar
/opt/zimbra/common/lib/jvm/openjdk-13.0.1-zimbra/

As zimbra user run below commands

su - zimbra
zmcontrol restart