Difference between revisions of "Testing: Selenium: ZimbraSelenium CodeCoverage"

(Created page with '==General Information== The '''Zimbra Selenium Harness''' contains code to measure [http://en.wikipedia.org/wiki/Code_coverage code coverage] using [http://siliconforks.com/jsco…')
 
Line 3: Line 3:
 
The '''Zimbra Selenium Harness''' contains code to measure [http://en.wikipedia.org/wiki/Code_coverage code coverage] using [http://siliconforks.com/jscoverage/ JSCoverage].
 
The '''Zimbra Selenium Harness''' contains code to measure [http://en.wikipedia.org/wiki/Code_coverage code coverage] using [http://siliconforks.com/jscoverage/ JSCoverage].
  
Code coverage statistics are gathered for the javascript clients, the ajax client and the admin console.
+
Code coverage statistics are gathered for the javascript clients:
  
==Design==
+
* ajax client
 +
* the admin console.
  
The harness uses jscoverage-server mode, where jscoverage.html is not required.  Therefore, the code coverage does not require the zimbra application to run inside a containing window.  See [http://siliconforks.com/jscoverage/manual.html JSCoverage documentation] for more details.
 
  
 
==JSCoverage==
 
==JSCoverage==
Line 22: Line 22:
  
 
  /usr/local/bin/jscoverage --no-instrument=help/ /opt/zimbra/jetty/webapps/zimbra /opt/zimbra/jetty/webapps/instrumented
 
  /usr/local/bin/jscoverage --no-instrument=help/ /opt/zimbra/jetty/webapps/zimbra /opt/zimbra/jetty/webapps/instrumented
 +
  
 
==Design==
 
==Design==
 +
 +
The harness uses jscoverage-server mode, where jscoverage.html is not required.  Therefore, the code coverage does not require the zimbra application to run inside a containing window.  See [http://siliconforks.com/jscoverage/manual.html JSCoverage documentation] for more details.
 +
 +
[http://staf.sourceforge.net/ STAF] must be enabled with level 5 permissions on both the client and server for coverage to be activated.  STAF is used to remotely execute instrumentation commands.
  
 
===Configuration===
 
===Configuration===
  
===Logging===
+
The harness config.properties file has several settings that affect coverage reporting.
 +
 
 +
coverage.enabled=false
 +
coverage.tool=/usr/local/bin/jscoverage
 +
coverage.reportsource=false
 +
coverage.query=dev=1&debug=0
 +
coverage.maxpageload.msec=90000
 +
coverage.instrument=true
 +
 
 +
* '''coverage.enabled''': Set to "true" or "false" to enable or disable, respectively, coverage reporting
 +
* '''coverage.tool''': Set to the path of jscoverage tool on the server
 +
* '''coverage.reportsource''': Set to "true" or "false" to enable or disable, respectively, source code line counts in the coverage report.  Enabling this option greatly increases harness memory usage, since all the source code is loaded into memory.
 +
* '''coverage.query''': the URL query options to instruct the application to not zip the JS files.  (JScoverage requires the source code to be in JS format.  The tool will not work with zip format files.)
 +
* '''coverage.maxpageload.msec''': The Zimbra application takes much longer to load when running in code coverage mode.  This timeout setting will override the default harness page load timeout, so that the harness will not report failures when the application takes too long to load.
 +
* '''coverage.instrument''': Set to "true" or "false" to enable or disable, respectively, server instrumentation at the beginning and end of the tests
 +
 
 +
The configuration settings can be used to control the behavior of the code coverage mechanism.  For instance, when developing new scripts with code coverage enabled, you may choose to manually instrument the code, rather than the harness instrumenting the code at run-time (which takes an additional 2-3 minutes).  In this case, you would likely choose these settings
 +
 
 +
coverage.enabled=true      # Enable coverage
 +
coverage.instrument=false  # The harness should not instrument, since it was manually done
  
 
===Coverage reporting===
 
===Coverage reporting===
  
 
After executing the harness, the output will be written to ZimbraSelenium/test-output/.../coverage/jscoverage.html
 
After executing the harness, the output will be written to ZimbraSelenium/test-output/.../coverage/jscoverage.html
 +
 +
There are multiple files in the coverage folder, however, jscoverage.html is the main html page and jscoverage.json is the JSON file which contains the coverage metrics.  See [http://siliconforks.com/jscoverage/manual.html JSCoverage documentation] for more details.
 +
 +
TIP: The report uses quite a lot of memory and processing when loaded.  The [http://www.google.com/chrome/ Chrome browser] seems to work best when viewing the report.
 +
 +
Here is an example report:
 +
 +
[[Image:ZimbraSeleniumCoverageSample.jpeg|thumb|right|200px|Selenium Structure]]
 +
 +
 +
===Logging===
 +
 +
  
  
Line 37: Line 74:
  
 
{{Article Footer|unknown|11/7/2007}}
 
{{Article Footer|unknown|11/7/2007}}
 
  
 
[[Category: Testing]]
 
[[Category: Testing]]
 
[[Category: Selenium]]
 
[[Category: Selenium]]

Revision as of 18:19, 26 May 2011

General Information

The Zimbra Selenium Harness contains code to measure code coverage using JSCoverage.

Code coverage statistics are gathered for the javascript clients:

  • ajax client
  • the admin console.


JSCoverage

JSCoverage install

Code Coverage depends on having jscoverage installed on the server in /usr/local/bin/jscoverage, which corresponds to the coverage.tool property in ZimbraSelenium/conf/config.properties.

Don't use the apt-get install jscoverage, that doesn't seem to work correctly. Instead, get the source zip and compile. Then, copy/link /usr/local/bin/jscoverage to the built files.

JSCoverage instrumentation

To manually instrument the server, the command is (as root):

/usr/local/bin/jscoverage --no-instrument=help/ /opt/zimbra/jetty/webapps/zimbra /opt/zimbra/jetty/webapps/instrumented


Design

The harness uses jscoverage-server mode, where jscoverage.html is not required. Therefore, the code coverage does not require the zimbra application to run inside a containing window. See JSCoverage documentation for more details.

STAF must be enabled with level 5 permissions on both the client and server for coverage to be activated. STAF is used to remotely execute instrumentation commands.

Configuration

The harness config.properties file has several settings that affect coverage reporting.

coverage.enabled=false
coverage.tool=/usr/local/bin/jscoverage
coverage.reportsource=false
coverage.query=dev=1&debug=0
coverage.maxpageload.msec=90000
coverage.instrument=true
  • coverage.enabled: Set to "true" or "false" to enable or disable, respectively, coverage reporting
  • coverage.tool: Set to the path of jscoverage tool on the server
  • coverage.reportsource: Set to "true" or "false" to enable or disable, respectively, source code line counts in the coverage report. Enabling this option greatly increases harness memory usage, since all the source code is loaded into memory.
  • coverage.query: the URL query options to instruct the application to not zip the JS files. (JScoverage requires the source code to be in JS format. The tool will not work with zip format files.)
  • coverage.maxpageload.msec: The Zimbra application takes much longer to load when running in code coverage mode. This timeout setting will override the default harness page load timeout, so that the harness will not report failures when the application takes too long to load.
  • coverage.instrument: Set to "true" or "false" to enable or disable, respectively, server instrumentation at the beginning and end of the tests

The configuration settings can be used to control the behavior of the code coverage mechanism. For instance, when developing new scripts with code coverage enabled, you may choose to manually instrument the code, rather than the harness instrumenting the code at run-time (which takes an additional 2-3 minutes). In this case, you would likely choose these settings

coverage.enabled=true       # Enable coverage
coverage.instrument=false   # The harness should not instrument, since it was manually done

Coverage reporting

After executing the harness, the output will be written to ZimbraSelenium/test-output/.../coverage/jscoverage.html

There are multiple files in the coverage folder, however, jscoverage.html is the main html page and jscoverage.json is the JSON file which contains the coverage metrics. See JSCoverage documentation for more details.

TIP: The report uses quite a lot of memory and processing when loaded. The Chrome browser seems to work best when viewing the report.

Here is an example report:

Selenium Structure


Logging

Verified Against: unknown Date Created: 11/7/2007
Article ID: https://wiki.zimbra.com/index.php?title=Testing:_Selenium:_ZimbraSelenium_CodeCoverage Date Modified: 2011-05-26



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search