Difference between revisions of "Testing: Selenium: ZimbraSelenium CodeCoverage"

Line 54: Line 54:
  
 
===Coverage reporting===
 
===Coverage reporting===
 +
 +
[[Image:ZimbraSeleniumCoverageSample.jpeg|thumb|right|400px|Sample Report]]
  
 
After executing the harness, the output will be written to ZimbraSelenium/test-output/.../coverage/jscoverage.html
 
After executing the harness, the output will be written to ZimbraSelenium/test-output/.../coverage/jscoverage.html
Line 61: Line 63:
 
TIP: The report uses quite a lot of memory and processing when loaded.  The [http://www.google.com/chrome/ Chrome browser] seems to work best when viewing the report.
 
TIP: The report uses quite a lot of memory and processing when loaded.  The [http://www.google.com/chrome/ Chrome browser] seems to work best when viewing the report.
  
Here is an example report:
 
  
[[Image:ZimbraSeleniumCoverageSample.jpeg|thumb|right|200px|Selenium Structure]]
+
===Logging===
 +
 
 +
Using default settings, code coverage is logged to the log4j object (log4j.logger.com.zimbra.qa.selenium.framework.utils.CodeCoverage).  Per default conf/log4j.properties settings, you can find the output in the test-output/.../debug.txt file.  Here are some examples:
 +
 
 +
Startup instrumentation:
 +
 
 +
20938 INFO  com.zimbra.qa.selenium.framework.util.CodeCoverage  - instrumentServer()
 +
20938 INFO  com.zimbra.qa.selenium.framework.util.CodeCoverage  - instrumentServerCheck()
 +
 
 +
Point calculation of code coverage after each test case:
 +
 
 +
140909 INFO  com.zimbra.qa.selenium.framework.util.CodeCoverage  - calculateCoverage()
 +
142034 INFO  com.zimbra.qa.selenium.framework.util.CodeCoverage  - CodeCoverage: files(568) lines(33201) duplicates(0)
 +
142034 INFO  com.zimbra.qa.selenium.framework.core.ExecuteHarnessMain$MethodListener  - MethodListener: FINISH: com.zimbra.qa.selenium.projects.ajax.tests.addressbook.contactgroups.ContextMenuContactGroup.NewEmail
 +
...
 +
144691 INFO  com.zimbra.qa.selenium.framework.util.CodeCoverage  - calculateCoverage()
 +
145784 INFO  com.zimbra.qa.selenium.framework.util.CodeCoverage  - CodeCoverage: files(0) lines(69) duplicates(302)
 +
146206 INFO  com.zimbra.qa.selenium.framework.core.ExecuteHarnessMain$MethodListener  - MethodListener: FINISH: com.zimbra.qa.selenium.projects.ajax.tests.addressbook.contactgroups.ContextMenuContactGroup.ShowContextMenu
 +
...
 +
185019 INFO  com.zimbra.qa.selenium.framework.util.CodeCoverage  - calculateCoverage()
 +
186003 INFO  com.zimbra.qa.selenium.framework.util.CodeCoverage  - CodeCoverage: files(0) lines(149) duplicates(73)
 +
186409 INFO  com.zimbra.qa.selenium.framework.core.ExecuteHarnessMain$MethodListener  - MethodListener: FINISH: com.zimbra.qa.selenium.projects.ajax.tests.addressbook.contactgroups.CreateContactGroup.CreateContactGroup_01
 +
...
  
 +
'''files''' is the number of new files that were analyzed.  '''lines''' is the number of new lines that were exercised.  '''duplicates''' is the number of lines that were exercised that were already previously exercised.
  
===Logging===
+
Shutdown un-instrumentation:
  
 +
8439284 INFO  com.zimbra.qa.selenium.framework.util.CodeCoverage  - instrumentServerUndo()
  
 +
Writing the coverage report webpage files:
  
 +
8466097 INFO  com.zimbra.qa.selenium.framework.util.CodeCoverage  - writeCoverage()
 +
8477409 INFO  com.zimbra.qa.selenium.framework.util.CodeCoverage  - CodeCoverage: took an additional 191 seconds of processing time
  
 +
The '''additional X seconds of processing time''' refers to the amount of time used to instrument the server, calculate coverage, uninstrument the server, and write the report files.  It does not include slowness due to loading instrumented files in the browser (i.e. page load slowness).
  
  

Revision as of 18:41, 26 May 2011

General Information

The Zimbra Selenium Harness contains code to measure code coverage using JSCoverage.

Code coverage statistics are gathered for the javascript clients:

  • ajax client
  • the admin console.


JSCoverage

JSCoverage install

Code Coverage depends on having jscoverage installed on the server in /usr/local/bin/jscoverage, which corresponds to the coverage.tool property in ZimbraSelenium/conf/config.properties.

Don't use the apt-get install jscoverage, that doesn't seem to work correctly. Instead, get the source zip and compile. Then, copy/link /usr/local/bin/jscoverage to the built files.

JSCoverage instrumentation

To manually instrument the server, the command is (as root):

/usr/local/bin/jscoverage --no-instrument=help/ /opt/zimbra/jetty/webapps/zimbra /opt/zimbra/jetty/webapps/instrumented


Design

The harness uses jscoverage-server mode, where jscoverage.html is not required. Therefore, the code coverage does not require the zimbra application to run inside a containing window. See JSCoverage documentation for more details.

STAF must be enabled with level 5 permissions on both the client and server for coverage to be activated. STAF is used to remotely execute instrumentation commands.

Configuration

The harness config.properties file has several settings that affect coverage reporting.

coverage.enabled=false
coverage.tool=/usr/local/bin/jscoverage
coverage.reportsource=false
coverage.query=dev=1&debug=0
coverage.maxpageload.msec=90000
coverage.instrument=true
  • coverage.enabled: Set to "true" or "false" to enable or disable, respectively, coverage reporting
  • coverage.tool: Set to the path of jscoverage tool on the server
  • coverage.reportsource: Set to "true" or "false" to enable or disable, respectively, source code line counts in the coverage report. Enabling this option greatly increases harness memory usage, since all the source code is loaded into memory.
  • coverage.query: the URL query options to instruct the application to not zip the JS files. (JScoverage requires the source code to be in JS format. The tool will not work with zip format files.)
  • coverage.maxpageload.msec: The Zimbra application takes much longer to load when running in code coverage mode. This timeout setting will override the default harness page load timeout, so that the harness will not report failures when the application takes too long to load.
  • coverage.instrument: Set to "true" or "false" to enable or disable, respectively, server instrumentation at the beginning and end of the tests

The configuration settings can be used to control the behavior of the code coverage mechanism. For instance, when developing new scripts with code coverage enabled, you may choose to manually instrument the code, rather than the harness instrumenting the code at run-time (which takes an additional 2-3 minutes). In this case, you would likely choose these settings

coverage.enabled=true       # Enable coverage
coverage.instrument=false   # The harness should not instrument, since it was manually done

Coverage reporting

Sample Report

After executing the harness, the output will be written to ZimbraSelenium/test-output/.../coverage/jscoverage.html

There are multiple files in the coverage folder, however, jscoverage.html is the main html page and jscoverage.json is the JSON file which contains the coverage metrics. See JSCoverage documentation for more details.

TIP: The report uses quite a lot of memory and processing when loaded. The Chrome browser seems to work best when viewing the report.


Logging

Using default settings, code coverage is logged to the log4j object (log4j.logger.com.zimbra.qa.selenium.framework.utils.CodeCoverage). Per default conf/log4j.properties settings, you can find the output in the test-output/.../debug.txt file. Here are some examples:

Startup instrumentation:

20938 INFO  com.zimbra.qa.selenium.framework.util.CodeCoverage  - instrumentServer()
20938 INFO  com.zimbra.qa.selenium.framework.util.CodeCoverage  - instrumentServerCheck()

Point calculation of code coverage after each test case:

140909 INFO  com.zimbra.qa.selenium.framework.util.CodeCoverage  - calculateCoverage()
142034 INFO  com.zimbra.qa.selenium.framework.util.CodeCoverage  - CodeCoverage: files(568) lines(33201) duplicates(0)
142034 INFO  com.zimbra.qa.selenium.framework.core.ExecuteHarnessMain$MethodListener  - MethodListener: FINISH: com.zimbra.qa.selenium.projects.ajax.tests.addressbook.contactgroups.ContextMenuContactGroup.NewEmail
...
144691 INFO  com.zimbra.qa.selenium.framework.util.CodeCoverage  - calculateCoverage()
145784 INFO  com.zimbra.qa.selenium.framework.util.CodeCoverage  - CodeCoverage: files(0) lines(69) duplicates(302)
146206 INFO  com.zimbra.qa.selenium.framework.core.ExecuteHarnessMain$MethodListener  - MethodListener: FINISH: com.zimbra.qa.selenium.projects.ajax.tests.addressbook.contactgroups.ContextMenuContactGroup.ShowContextMenu
...
185019 INFO  com.zimbra.qa.selenium.framework.util.CodeCoverage  - calculateCoverage()
186003 INFO  com.zimbra.qa.selenium.framework.util.CodeCoverage  - CodeCoverage: files(0) lines(149) duplicates(73)
186409 INFO  com.zimbra.qa.selenium.framework.core.ExecuteHarnessMain$MethodListener  - MethodListener: FINISH: com.zimbra.qa.selenium.projects.ajax.tests.addressbook.contactgroups.CreateContactGroup.CreateContactGroup_01
...

files is the number of new files that were analyzed. lines is the number of new lines that were exercised. duplicates is the number of lines that were exercised that were already previously exercised.

Shutdown un-instrumentation:

8439284 INFO  com.zimbra.qa.selenium.framework.util.CodeCoverage  - instrumentServerUndo()

Writing the coverage report webpage files:

8466097 INFO  com.zimbra.qa.selenium.framework.util.CodeCoverage  - writeCoverage()
8477409 INFO  com.zimbra.qa.selenium.framework.util.CodeCoverage  - CodeCoverage: took an additional 191 seconds of processing time

The additional X seconds of processing time refers to the amount of time used to instrument the server, calculate coverage, uninstrument the server, and write the report files. It does not include slowness due to loading instrumented files in the browser (i.e. page load slowness).


Verified Against: unknown Date Created: 11/7/2007
Article ID: https://wiki.zimbra.com/index.php?title=Testing:_Selenium:_ZimbraSelenium_CodeCoverage Date Modified: 2011-05-26



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search