Testing: Selenium: ZimbraSelenium CodeCoverage
General Information
The Zimbra Selenium Harness contains code to measure code coverage using JSCoverage.
Code coverage statistics are gathered for the javascript clients:
- ajax client
- the admin console.
JSCoverage
JSCoverage install
Code Coverage depends on having jscoverage installed on the server in /usr/local/bin/jscoverage, which corresponds to the coverage.tool property in ZimbraSelenium/conf/config.properties.
Don't use the apt-get install jscoverage, that doesn't seem to work correctly. Instead, get the source zip and compile. Then, copy/link /usr/local/bin/jscoverage to the built files.
JSCoverage instrumentation
To manually instrument the server, the command is (as root):
/usr/local/bin/jscoverage --no-instrument=help/ /opt/zimbra/jetty/webapps/zimbra /opt/zimbra/jetty/webapps/instrumented
Design
The harness uses jscoverage-server mode, where jscoverage.html is not required. Therefore, the code coverage does not require the zimbra application to run inside a containing window. See JSCoverage documentation for more details.
STAF must be enabled with level 5 permissions on both the client and server for coverage to be activated. STAF is used to remotely execute instrumentation commands.
Configuration
The harness config.properties file has several settings that affect coverage reporting.
coverage.enabled=false coverage.tool=/usr/local/bin/jscoverage coverage.reportsource=false coverage.query=dev=1&debug=0 coverage.maxpageload.msec=90000 coverage.instrument=true
- coverage.enabled: Set to "true" or "false" to enable or disable, respectively, coverage reporting
- coverage.tool: Set to the path of jscoverage tool on the server
- coverage.reportsource: Set to "true" or "false" to enable or disable, respectively, source code line counts in the coverage report. Enabling this option greatly increases harness memory usage, since all the source code is loaded into memory.
- coverage.query: the URL query options to instruct the application to not zip the JS files. (JScoverage requires the source code to be in JS format. The tool will not work with zip format files.)
- coverage.maxpageload.msec: The Zimbra application takes much longer to load when running in code coverage mode. This timeout setting will override the default harness page load timeout, so that the harness will not report failures when the application takes too long to load.
- coverage.instrument: Set to "true" or "false" to enable or disable, respectively, server instrumentation at the beginning and end of the tests
The configuration settings can be used to control the behavior of the code coverage mechanism. For instance, when developing new scripts with code coverage enabled, you may choose to manually instrument the code, rather than the harness instrumenting the code at run-time (which takes an additional 2-3 minutes). In this case, you would likely choose these settings
coverage.enabled=true # Enable coverage coverage.instrument=false # The harness should not instrument, since it was manually done
Coverage reporting
After executing the harness, the output will be written to ZimbraSelenium/test-output/.../coverage/jscoverage.html
There are multiple files in the coverage folder, however, jscoverage.html is the main html page and jscoverage.json is the JSON file which contains the coverage metrics. See JSCoverage documentation for more details.
TIP: The report uses quite a lot of memory and processing when loaded. The Chrome browser seems to work best when viewing the report.
Logging
Using default settings, code coverage is logged to the log4j object (log4j.logger.com.zimbra.qa.selenium.framework.utils.CodeCoverage). Per default conf/log4j.properties settings, you can find the output in the test-output/.../debug.txt file. Here are some examples:
Startup instrumentation:
20938 INFO com.zimbra.qa.selenium.framework.util.CodeCoverage - instrumentServer() 20938 INFO com.zimbra.qa.selenium.framework.util.CodeCoverage - instrumentServerCheck()
Point calculation of code coverage after each test case:
140909 INFO com.zimbra.qa.selenium.framework.util.CodeCoverage - calculateCoverage() 142034 INFO com.zimbra.qa.selenium.framework.util.CodeCoverage - CodeCoverage: files(568) lines(33201) duplicates(0) 142034 INFO com.zimbra.qa.selenium.framework.core.ExecuteHarnessMain$MethodListener - MethodListener: FINISH: com.zimbra.qa.selenium.projects.ajax.tests.addressbook.contactgroups.ContextMenuContactGroup.NewEmail ... 144691 INFO com.zimbra.qa.selenium.framework.util.CodeCoverage - calculateCoverage() 145784 INFO com.zimbra.qa.selenium.framework.util.CodeCoverage - CodeCoverage: files(0) lines(69) duplicates(302) 146206 INFO com.zimbra.qa.selenium.framework.core.ExecuteHarnessMain$MethodListener - MethodListener: FINISH: com.zimbra.qa.selenium.projects.ajax.tests.addressbook.contactgroups.ContextMenuContactGroup.ShowContextMenu ... 185019 INFO com.zimbra.qa.selenium.framework.util.CodeCoverage - calculateCoverage() 186003 INFO com.zimbra.qa.selenium.framework.util.CodeCoverage - CodeCoverage: files(0) lines(149) duplicates(73) 186409 INFO com.zimbra.qa.selenium.framework.core.ExecuteHarnessMain$MethodListener - MethodListener: FINISH: com.zimbra.qa.selenium.projects.ajax.tests.addressbook.contactgroups.CreateContactGroup.CreateContactGroup_01 ...
files is the number of new files that were analyzed. lines is the number of new lines that were exercised. duplicates is the number of lines that were exercised that were already previously exercised.
Shutdown un-instrumentation:
8439284 INFO com.zimbra.qa.selenium.framework.util.CodeCoverage - instrumentServerUndo()
Writing the coverage report webpage files:
8466097 INFO com.zimbra.qa.selenium.framework.util.CodeCoverage - writeCoverage() 8477409 INFO com.zimbra.qa.selenium.framework.util.CodeCoverage - CodeCoverage: took an additional 191 seconds of processing time
The additional X seconds of processing time refers to the amount of time used to instrument the server, calculate coverage, uninstrument the server, and write the report files. It does not include slowness due to loading instrumented files in the browser (i.e. page load slowness).