Testing: Selenium: ZimbraSelenium CodeCoverage: Difference between revisions
(Created page with '==General Information== The '''Zimbra Selenium Harness''' contains code to measure [http://en.wikipedia.org/wiki/Code_coverage code coverage] using [http://siliconforks.com/jsco…') |
No edit summary |
||
Line 3: | Line 3: | ||
The '''Zimbra Selenium Harness''' contains code to measure [http://en.wikipedia.org/wiki/Code_coverage code coverage] using [http://siliconforks.com/jscoverage/ JSCoverage]. | The '''Zimbra Selenium Harness''' contains code to measure [http://en.wikipedia.org/wiki/Code_coverage code coverage] using [http://siliconforks.com/jscoverage/ JSCoverage]. | ||
Code coverage statistics are gathered for the javascript clients | Code coverage statistics are gathered for the javascript clients: | ||
* ajax client | |||
* the admin console. | |||
==JSCoverage== | ==JSCoverage== | ||
Line 22: | Line 22: | ||
/usr/local/bin/jscoverage --no-instrument=help/ /opt/zimbra/jetty/webapps/zimbra /opt/zimbra/jetty/webapps/instrumented | /usr/local/bin/jscoverage --no-instrument=help/ /opt/zimbra/jetty/webapps/zimbra /opt/zimbra/jetty/webapps/instrumented | ||
==Design== | ==Design== | ||
The harness uses jscoverage-server mode, where jscoverage.html is not required. Therefore, the code coverage does not require the zimbra application to run inside a containing window. See [http://siliconforks.com/jscoverage/manual.html JSCoverage documentation] for more details. | |||
[http://staf.sourceforge.net/ STAF] must be enabled with level 5 permissions on both the client and server for coverage to be activated. STAF is used to remotely execute instrumentation commands. | |||
===Configuration=== | ===Configuration=== | ||
=== | The harness config.properties file has several settings that affect coverage reporting. | ||
coverage.enabled=false | |||
coverage.tool=/usr/local/bin/jscoverage | |||
coverage.reportsource=false | |||
coverage.query=dev=1&debug=0 | |||
coverage.maxpageload.msec=90000 | |||
coverage.instrument=true | |||
* '''coverage.enabled''': Set to "true" or "false" to enable or disable, respectively, coverage reporting | |||
* '''coverage.tool''': Set to the path of jscoverage tool on the server | |||
* '''coverage.reportsource''': Set to "true" or "false" to enable or disable, respectively, source code line counts in the coverage report. Enabling this option greatly increases harness memory usage, since all the source code is loaded into memory. | |||
* '''coverage.query''': the URL query options to instruct the application to not zip the JS files. (JScoverage requires the source code to be in JS format. The tool will not work with zip format files.) | |||
* '''coverage.maxpageload.msec''': The Zimbra application takes much longer to load when running in code coverage mode. This timeout setting will override the default harness page load timeout, so that the harness will not report failures when the application takes too long to load. | |||
* '''coverage.instrument''': Set to "true" or "false" to enable or disable, respectively, server instrumentation at the beginning and end of the tests | |||
The configuration settings can be used to control the behavior of the code coverage mechanism. For instance, when developing new scripts with code coverage enabled, you may choose to manually instrument the code, rather than the harness instrumenting the code at run-time (which takes an additional 2-3 minutes). In this case, you would likely choose these settings | |||
coverage.enabled=true # Enable coverage | |||
coverage.instrument=false # The harness should not instrument, since it was manually done | |||
===Coverage reporting=== | ===Coverage reporting=== | ||
After executing the harness, the output will be written to ZimbraSelenium/test-output/.../coverage/jscoverage.html | After executing the harness, the output will be written to ZimbraSelenium/test-output/.../coverage/jscoverage.html | ||
There are multiple files in the coverage folder, however, jscoverage.html is the main html page and jscoverage.json is the JSON file which contains the coverage metrics. See [http://siliconforks.com/jscoverage/manual.html JSCoverage documentation] for more details. | |||
TIP: The report uses quite a lot of memory and processing when loaded. The [http://www.google.com/chrome/ Chrome browser] seems to work best when viewing the report. | |||
Here is an example report: | |||
[[Image:ZimbraSeleniumCoverageSample.jpeg|thumb|right|200px|Selenium Structure]] | |||
===Logging=== | |||
Line 37: | Line 74: | ||
{{Article Footer|unknown|11/7/2007}} | {{Article Footer|unknown|11/7/2007}} | ||
[[Category: Testing]] | [[Category: Testing]] | ||
[[Category: Selenium]] | [[Category: Selenium]] |
Revision as of 18:19, 26 May 2011
General Information
The Zimbra Selenium Harness contains code to measure code coverage using JSCoverage.
Code coverage statistics are gathered for the javascript clients:
- ajax client
- the admin console.
JSCoverage
JSCoverage install
Code Coverage depends on having jscoverage installed on the server in /usr/local/bin/jscoverage, which corresponds to the coverage.tool property in ZimbraSelenium/conf/config.properties.
Don't use the apt-get install jscoverage, that doesn't seem to work correctly. Instead, get the source zip and compile. Then, copy/link /usr/local/bin/jscoverage to the built files.
JSCoverage instrumentation
To manually instrument the server, the command is (as root):
/usr/local/bin/jscoverage --no-instrument=help/ /opt/zimbra/jetty/webapps/zimbra /opt/zimbra/jetty/webapps/instrumented
Design
The harness uses jscoverage-server mode, where jscoverage.html is not required. Therefore, the code coverage does not require the zimbra application to run inside a containing window. See JSCoverage documentation for more details.
STAF must be enabled with level 5 permissions on both the client and server for coverage to be activated. STAF is used to remotely execute instrumentation commands.
Configuration
The harness config.properties file has several settings that affect coverage reporting.
coverage.enabled=false coverage.tool=/usr/local/bin/jscoverage coverage.reportsource=false coverage.query=dev=1&debug=0 coverage.maxpageload.msec=90000 coverage.instrument=true
- coverage.enabled: Set to "true" or "false" to enable or disable, respectively, coverage reporting
- coverage.tool: Set to the path of jscoverage tool on the server
- coverage.reportsource: Set to "true" or "false" to enable or disable, respectively, source code line counts in the coverage report. Enabling this option greatly increases harness memory usage, since all the source code is loaded into memory.
- coverage.query: the URL query options to instruct the application to not zip the JS files. (JScoverage requires the source code to be in JS format. The tool will not work with zip format files.)
- coverage.maxpageload.msec: The Zimbra application takes much longer to load when running in code coverage mode. This timeout setting will override the default harness page load timeout, so that the harness will not report failures when the application takes too long to load.
- coverage.instrument: Set to "true" or "false" to enable or disable, respectively, server instrumentation at the beginning and end of the tests
The configuration settings can be used to control the behavior of the code coverage mechanism. For instance, when developing new scripts with code coverage enabled, you may choose to manually instrument the code, rather than the harness instrumenting the code at run-time (which takes an additional 2-3 minutes). In this case, you would likely choose these settings
coverage.enabled=true # Enable coverage coverage.instrument=false # The harness should not instrument, since it was manually done
Coverage reporting
After executing the harness, the output will be written to ZimbraSelenium/test-output/.../coverage/jscoverage.html
There are multiple files in the coverage folder, however, jscoverage.html is the main html page and jscoverage.json is the JSON file which contains the coverage metrics. See JSCoverage documentation for more details.
TIP: The report uses quite a lot of memory and processing when loaded. The Chrome browser seems to work best when viewing the report.
Here is an example report:
Logging