Testing: Selenium: Running ZimbraSelenium: Difference between revisions

No edit summary
 
(19 intermediate revisions by 3 users not shown)
Line 1: Line 1:
==General Information==
==General Information==


ZimbraSelenium is the automation test harness for verifying the browser based Zimbra clients.
zm-selenium is the automation test harness for verifying the browser based Zimbra clients.


==Prerequisites==
==Prerequisites==


===Downloading and Building ZimbraSelenium===
#Create .zcs-deps directory at your %UserProfile%
#Install ant, ivy, Java1.8+ and set respective environment variables.
#Copy following jar at %UserProfile%/.zcs-deps:


Please follow the steps in [[Testing:_Selenium:_Building_ZimbraSelenium]].


Once the project built and no dependency errors were found, you are almost ready to run your tests. Just configure the tests to work properly in your environment as explained in the next section and you'll be ready to test.
    ant-contrib-1.0b1.jar
 
==Downloading the Dependencies==
 
The zm-selenium project depends on the following Zimbra projects:


===Software===
#zimbra-package-stub
#zm-zcs
#zm-mailbox
#zm-web-client
#zm-ajax
#zm-zimlets


==== STAF ====
Clone these projects from GitHub in C:/zimbra/ directory as an example:
The Zimbra Selenium harness depends on [http://staf.sourceforge.net/ STAF] being installed on both the test client and zimbra server.  See the design overview for more details.


Install STAF on both the client and server. And configure security level 5 for communication from the client to the server. For example, if the client IP address is 192.168.1.1 and the server IP address is 192.168.1.2, run this command to set security level 5:
<pre>
cd C:/zimbra/
git clone https://github.com/Zimbra/zimbra-package-stub.git
git clone https://github.com/Zimbra/zm-zcs.git
git clone https://github.com/Zimbra/zm-mailbox.git
git clone https://github.com/Zimbra/zm-web-client.git
git clone https://github.com/Zimbra/zm-ajax.git
git clone https://github.com/Zimbra/zm-zimlets.git
</pre>


Build zm-native and zm-common jar using zm-mailbox repo.
<pre>
<pre>
Go to zm-mailbox and build using following command.
      ant publish-local-all -Dzimbra.buildinfo.version=8.8.0_GA
It will create zm-common.jar and zm-native.jar file
</pre>


root@192.168.1.1:~# ssh root@192.168.1.2
==Downloading zm-selenium==
root@192.168.1.2's password: *****
root@192.168.1.2:~# staf local trust set machine 192.168.1.1 level 5
Response
--------


root@192.168.1.2:~#
Clone zm-selenium from github in same folder:


<pre>
git clone https://github.com/Zimbra/zm-selenium.git
</pre>
</pre>


==Building zm-selenium using Eclipse==
Download and install latest [http://www.eclipse.org/downloads/ eclipse IDE] and [https://ant.apache.org/ivy/ivyde/download.html Ivy plugin].
====Build the Code====
Use the following steps to build the code:
# Create new eclipse workspace
## Open eclipse
## Create new workspace in C:/zimbra/
# Add zm-selenium project to eclipse
## Select File->New->Project->Java Project
## For "Project Name", enter zm-selenium
# Build zm-selenium
## Expand the zm-selenium project
## Select build.xml, right click, select "Run As"->"Ant Build"
- it will create 3 jars at ..\zm-selenium\build\dist\lib:
coverage.jar, resources.jar and zimbraselenium.jar


The output in the console will look like:
<pre>
...
compile:
    [echo] Compiling framework...
    [javac] Compiling 1728 source files to C:\zimbra\zm-selenium\build\classes
jar:
      [jar] Building jar: C:\zimbra\zm-selenium\build\dist\lib\zimbraselenium.jar
BUILD SUCCESSFUL
Total time: 27 seconds
</pre>
Once the project built and no dependency errors were found, you are almost ready to run your tests. Just configure the tests to work properly in your environment as explained in the next section and you'll be ready to test.


=== Tests Configuration===
==== Tests Configuration====


The file located in ZimbraSelenium/conf/conf.properties is the master configuration file. Every parameter change required to run the tests properly is configurable using this properties file.  
The file located in zm-selenium/conf/conf.properties is the master configuration file. Every parameter change required to run the tests properly is configurable using this properties file.  
Configure the test harness with the basic parameters required to run properly against your desired version of Zimbra by changing the following variables in the file:
Configure the test harness with the basic parameters required to run properly against your desired version of Zimbra by changing the following variables in the file:


* server: This is the URL in which the Zimbra install to be tested is running. Please notice that the same must be reachable from the location in which the Selenium RC server is running (localhost, [http://saucelabs.com Sauce OnDemand], a remote machine).
* server.host: This is the name of server in which the zimbra is installed. Please notice that the same must be reachable from the client machine in which test cases will be executed.
* serverMachineName: The machine in which the Selenium RC server will be running (localhost, sauceondemand or a reachable ip/domain).
* browser: The browser in which the tests should run. This is the standard Selenium browser string.
* browser: The browser in which the tests should run. This is the standard Selenium browser string.
* locale: The locale in which the installed version of Zimbra is running (en_US, ch, es_Es, etc).


=== Screen Resolution ===
===== Screen Resolution =====


Be sure to set client resolution to 1024 X 768 or higher, otherwise the zimbra client will default to the HTML client (See also https://bugzilla.zimbra.com/show_bug.cgi?id=27521).
Be sure to set client resolution to 1024 X 768 or higher, otherwise the zimbra client will default to the HTML client (See also https://bugzilla.zimbra.com/show_bug.cgi?id=27521).


==Running ZimbraSelenium==
==Running zm-selenium==
 
====Running from Eclipse====


===Running from Eclipse===
=====Java Target=====
The ZimbraSelenium source tree includes a build.xml file that has targets to execute the test suite.


# Open ZimbraSelenium/build.xml
You can also run selenium tests using Java target for e.g. create java run using:
 
<pre>
Project: zm-selenium
Main class: com.zimbra.qa.selenium.framework.core.ExecuteHarnessMain
Program arguments: -j C:\opt\qa\JUDASPRIEST\zm-selenium\build\dist\lib\zimbraselenium.jar -p com.zimbra.qa.selenium.projects.ajax.tests -g always,sanity -l conf/log4j.properties
</pre>
 
The zm-selenium source tree includes a build.xml file that has targets to execute the test suite.
 
# Open zm-selenium/build.xml
# Open the "outline view", if not already open
# Open the "outline view", if not already open
## Window -> Show View -> Outline
## Window -> Show View -> Outline
Line 58: Line 121:
# Right-click on "Run-ExecuteHarnessMain" in the Outline, and select "Run"
# Right-click on "Run-ExecuteHarnessMain" in the Outline, and select "Run"


Here is an example build.xml "Run-ExecuteHarnessMain" configuration, which runs the "sanity" and "smoke" address book test cases for the Zimbra Mobile Client suite.
Here is an example build.xml "Run-ExecuteHarnessMain" configuration, which runs the "sanity" and "smoke" mail compose test cases for Zimbra Ajax client suite.


<pre>
<pre>
<target name="Run-ExecuteHarnessMain" depends="jar" description="Run all tests per specified arguments">
<target name="Run-ExecuteHarnessMain" depends="jar" description="Run all tests per specified arguments">
<property name="jarfile" value="${dist.lib.dir}/zimbraselenium.jar"/>
<property name="jarfile" value="${dist.lib.dir}/zimbraselenium.jar"/>
<property name="pattern" value="projects.mobile.tests.addressbook"/>
<property name="pattern" value="ajax.test.mail.compose"/>
<property name="groups" value="always,sanity,smoke"/>
<property name="groups" value="always,sanity,smoke"/>
<echo>Executing ...</echo>
<echo>Executing ...</echo>
Line 77: Line 140:
* "zimbraselenium.jar" must be specified in as the "jarfile" property
* "zimbraselenium.jar" must be specified in as the "jarfile" property


===Running from Eclipse (Dev Environment)===
====Running from Eclipse (Dev Environment)====
 
The zm-selenium source tree is designed to run in the '''Zimbra Development Environment'''.  The Dev Environment is used to develop and build the Zimbra product.  When developing web clients or zimlets, the Zimbra Selenium test suite can be used to [http://en.wikipedia.org/wiki/Sanity_testing#Software_development Sanity Test] code changes.
 
Some test suite setup needs to be completed to execute the suite.


The ZimbraSelenium source tree is designed to run in the '''Zimbra Development Environment'''. The Dev Environment is used to develop and build the Zimbra product.  When developing web clients or zimlets, the Zimbra Selenium test suite can be used to [http://en.wikipedia.org/wiki/Sanity_testing#Software_development Sanity Test] code changes.
===== Modify zm-selenium Configuration File =====


To set up the Dev Environment, use the instructions found in the Zimbra source depot under ZimbraServer/docs/INSTALL-*.txt, for example ZimbraServer/docs/INSTALL-win.txt.  Once you have the environment up and running, follow the steps in this section to run the Zimbra Selenium suite.
Modify the zm-selenium/conf/config.properties


Some test suite setup needs to be completed to execute the suite.
*Zimbra Server -- Define the Zimbra server against which you want to run your tests
<pre>
server.scheme=http
# server.host value should be same as the default domain name; hence it can't be an IP address
server.host=host.local
server.port=7070
</pre>
 
<pre>
# store.host would be same as server.host
store.host=host.local
admin.port=7071
</pre>
 
*Browser
<pre>
# Standard Selenium browser
# Note: Running Selenium tests on firefox are not fully supported (see https://bugzilla.mozilla.org/show_bug.cgi?id=1303234).
browser=*googlechrome
#browser=*firefox
OS=OSX
</pre>
 
*disable staf
<pre>
postqueue.use.staf=false
</pre>
 
*non-staf sleep duration
<pre>
postqueue.sleep.nonstaf.msec=10000
</pre>
 
===== Create global admin user =====
 
The harness uses global admin user for some initial SOAP requests, so you must create an global admin account named 'globaladmin' with password 'test123':
<pre>
zmprov ca globaladmin@host.local test123 zimbraIsAdminAccount TRUE
</pre>
 
=====ANT Targets=====
* Run all the test via command line. (Developers will never really have a need for running all tests)
<pre>
ant Run-ExecuteHarnessMain
</pre>
 
You can check the output by loading the output index.html file -- zm-selenium/test-output/VERSION/AJAX/BROWSER/en_US/TestNG


# Open ZimbraSelenium/conf/config.properties
* Target a specific pattern, e.g. all mail compose tests.
# Modify the server settings for the client URL (i.e. <nowiki>http://localhost:7070</nowiki>)
<pre>
## Set server.scheme to http or https (by default, the dev env uses http)
ant Run-ExecuteHarnessMain -Dpattern ajax.tests.mail.compose
## Set server.host to localhost (by default, the dev env uses localhost)
</pre>
## Set server.port to the web server port number (by default, the dev env uses 7070)
# Modify the admin name to use the PC name
## Set adminName to the admin email address (by default, the dev env uses admin@hostname, where hostname is the computer name.  You can check this value by logging into the admin console, usually <nowiki>https://localhost:7071</nowiki> with username/password of admin/test123, then searching for the admin account, which will show the email address as something like "admin@mypcname")
# Open ZimbraSelenium/build.xml
# Locate the "Run-ExecuteHarnessMain (dev)" target
## This target is pre-configured to run the sanity suite
# Right-click on "Run-ExecuteHarnessMain (dev)" in the Outline, and select "Run"
<br>


Here is an example of the config.properties changes:
* You may also need to modify the group.  The following will run the smoke tests for quick reply.
<pre>
<pre>
...
ant Run-ExecuteHarnessMain -Dpattern ajax.tests.conversation.quickreply -Dgroups always,smoke
</pre>
 
Currently, there are three groups defined as of now:
 
# "sanity" - create basic items
# "smoke" - take basic actions on basic items
# "functional" - everything else
 
Additionally, these are four other related groups:
 
# "L0" - smoke test
# "L1" - critical path
# "L2" - functional/negative path
# "L3" - edge case


# Define the Zimbra server
<pre>
server.scheme=http
Sanity is very basic and probably doesn't cover much code other than making sure navigation to the different tabs works and create an item for all apps.
server.host=localhost
Smoke probably gets the most coverage in the least amount of tests. It covers internal operations like delete, modify, rename etc. for apps.
server.port=7070
Functional can be used when the change was very specific and the tests can be filtered to a pattern. It covers all narrowed down functionality of zimbra.
</pre>


(Note - When specifying any of the above three groups, "always" group, as shown in above example, must also be explicitly specified)


ZimbraLogRoot=test-output
*Another example for navigating away from preferences
adminName=admin@mypcname
<pre>
adminPwd=test123
ant Run-ExecuteHarnessMain -Dpattern ajax.tests.preferences.gui.NavigateAway -Dgroups always,functional
isAppliance=false
</pre>


...
*To get a count/preview of the tests that are going to be executed, use Sum-ExecuteHarnessMain:
<pre>
C:\> ant Sum-ExecuteHarnessMain -Dpattern ajax.tests.conversation.quickreply -Dgroups always,smoke
*****
1: Quick Reply (Forward) a conversation (1 message, 1 recipient)
2: Quick Reply to a conversation (1 message, 1 recipient)
3: Quick Reply to a conversation (1 message, 1 recipient)
Number of matching test cases: 3
</pre>
</pre>
<br>


After selecting "Run", the results will be displayed in the Eclipse console output. Here is an example:
Remember - On an average, each test takes about a minute to run. You should also not use other applications on your mac while the tests are running due to selenium keyboard input simulation that happens during the run. So basically, after you have made your change, select the tests that you want to run, count them to get an estimate of how long it would take for them to run, and then leave your mac and take a break while the tests are running.
 
To see all available tests you can use your IDE and browse the source tree or navigate via the command line:
<pre>
<pre>
    ...
csb-2-eng-dhcp258:main testuser$ cd zm-selenium/src/java/com/zimbra/qa/selenium/projects/ajax/tests
    [java] 100500 [main] INFO  org.openqa.jetty.util.Container  - Stopped org.openqa.jetty.jetty.servlet.ServletHandler@10ffb38
csb-2-eng-dhcp258:tests testuser$ cd mail
    [java] 100547 [main] INFO  org.openqa.jetty.util.Container  - Stopped HttpContext[/wd,/wd]
csb-2-eng-dhcp258:mail testuser$ ls
    [java] 100547 [main] INFO  org.openqa.jetty.util.Container  - Stopped org.openqa.jetty.jetty.Server@1e328e0
assistant feeds mountpoints readreceipt
    [java] 101500 [main] WARN  framework.core.SeleniumService  - Selenium server is stopped
attachments folders newwindow sort
    [java] 101500 [main] INFO  framework.core.ExecuteHarnessMain  - Total Tests:   7
bugs gui performance tags
    [java] Total Passed:  35
compose mail quickcommands undo
    [java] Total Failed:  0
csb-2-eng-dhcp258:mail testuser$
    [java] Total Skipped: 0
</pre>
    [java] Duration: 215 seconds
 
BUILD SUCCESSFUL
===Debugging test failures===
Total time: 4 minutes 14 seconds
 
A test failure could be due to a bug in your code, a bug in the test case (less likely), or an intermittent timing issue with the test run.
 
The first thing to try would be to rerun the failing tests and see if they still occur on a rerun. It may help to run the 'clean' step first:


<pre>
ant clean Run-ExecuteHarnessMain -Dpattern ajax.tests.conversation.quickreply -Dgroups always,smoke
</pre>
</pre>


You can usually find the error (typically an exception) by scrolling back up through the output. Beyond that you can look at the debug logs (and screenshots too!) for each test under zm-selenium/test-output/VERSION/AJAX/BROWSER/en_US/debug directory. If there are multiple test failures you may want to copy the list of failed tests somewhere before re-running tests individually, because each ant test run overrides the previous test report.
===== If you have Ubuntu Dev setup and tests are failing from Mac: =====
a. If you are getting this error while running tests on Mac system: File not found /opt/zimbra/common/lib/jvm/java/jre/lib/security/cacerts
* Find your current JDK :
<pre>% printenv JAVA_HOME</pre>
* Create the directory where ZCS looks for certificates:
<pre>mkdir -p /opt/zimbra/common/lib/jvm/java/jre/lib/security</pre>
* Copy system cacerts to Zimbra:
<pre>cp -f <Path-to-JDK>/Contents/Home/jre/lib/security/cacerts /opt/zimbra/common/lib/jvm/java/jre/lib/security/</pre>
b. If you are getting this exception : javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException
* Export Zimbra certificate from Ubuntu VM:
<pre>cd /opt/zimbra
keytool -exportcert -keystore jetty/etc/keystore -file cert -alias jetty
</pre>
Password : zimbra
* Copy /opt/zimbra/cert to Mac environment.
* Import Certificate in Mac:
<pre>keytool -importcert -keystore cacerts -file <path to cert file in mac></pre>
Password: changeit


===Running from STAF===
===Running from STAF===
The ZimbraSelenium test suite can be configured using [http://staf.sourceforge.net/ STAF].
The zm-selenium test suite can be configured using [http://staf.sourceforge.net/ STAF].
Zimbra Selenium harness depends on [http://staf.sourceforge.net/ STAF] being installed on both the test client and zimbra server.  See the design overview for more details.
 
Install STAF on both the client and server.  And configure security level 5 for communication from the client to the server.  For example, if the client IP address is 192.168.1.1 and the server IP address is 192.168.1.2, run this command to set security level 5:
 
<pre>
 
root@192.168.1.1:~# ssh root@192.168.1.2
root@192.168.1.2's password: *****
root@192.168.1.2:~# staf local trust set machine 192.168.1.1 level 5
Response
--------


====Build the ZimbraSelenium STAF jar====
root@192.168.1.2:~#


# Open ZimbraSelenium/build.xml
</pre>
 
====Build the zm-selenium STAF jar====
 
# Open zm-selenium/build.xml
# Open the "outline view", if not already open
# Open the "outline view", if not already open
## Window -> Show View -> Outline
## Window -> Show View -> Outline
# Locate the "staf-jar" target
# Locate the "staf-jar" target
# Right-click on "staf-jar" in the Outline, and select "Run"
# Right-click on "staf-jar" in the Outline, and select "Run"
# The jar file should be built (ZimbraSelenium/build/dist/zimbra-6.0.0/lib/zimbraselngstaf.jar)
# The jar file should be built (zm-selenium/build/dist/lib/zimbrastaf.jar)


<pre>
<pre>
Line 151: Line 324:
jar:
jar:
jar-staf-selenium:
jar-staf-selenium:
     [copy] Copying 697 files to C:\P4\main\ZimbraSelenium\build\staf\selenium\STAF-INF\classes
     [copy] Copying 697 files to C:\zimbra\zm-selenium\build\staf\selenium\STAF-INF\classes
     [copy] Copied 171 empty directories to 1 empty directory under C:\P4\main\ZimbraSelenium\build\staf\selenium\STAF-INF\classes
     [copy] Copied 171 empty directories to 1 empty directory under C:\zimbra\zm-selenium\build\staf\selenium\STAF-INF\classes
     [copy] Copying 23 files to C:\P4\main\ZimbraSelenium\build\staf\selenium\STAF-INF\jars
     [copy] Copying 23 files to C:\zimbra\zm-selenium\build\staf\selenium\STAF-INF\jars
       [jar] Building jar: C:\P4\main\ZimbraSelenium\build\dist\zimbra-6.0.0\lib\zimbraselngstaf.jar
       [jar] Building jar: C:\zimbra\zm-selenium\build\dist\lib\zimbrastaf.jar
jar-staf:
jar-staf:
BUILD SUCCESSFUL
BUILD SUCCESSFUL
Line 163: Line 336:
==== Configure the STAF jar ====
==== Configure the STAF jar ====


Configure zimbraselngstaf.jar with STAF
Configure zimbrastaf.jar with STAF


# Open C:\STAF\bin\STAF.cfg using a text editor
# Open C:\STAF\bin\STAF.cfg using a text editor
# Configure the service by adding "service seleniumm library JSTAF execute /P4/main/ZimbraSelenium/build/dist/zimbra-6.0.0/lib/zimbraselngstaf.jar"
# Configure the service by adding "service seleniumm library JSTAF execute /zimbra/zm-selenium/build/dist/lib/zimbrastaf.jar"
# Stop STAF
# Stop STAF
## C:\STAF> staf local shutdown shutdown
## C:\STAF> staf local shutdown shutdown
Line 188: Line 361:
# Add default service loader
# Add default service loader
serviceloader library STAFDSLS
serviceloader library STAFDSLS
</pre>


 
===== Add JSTAF Service =====
service selenium library JSTAF execute /P4/main/ZimbraSelenium/build/dist/zimbra-6.0.0/lib/zimbraselngstaf.jar
STAF <client-machine> SERVICE ADD SERVICE SELENIUM LIBRARY JSTAF EXECUTE c:/zimbra/zm-selenium/build/dist/SelNG/jars/zimbrastaf.jar
 
</pre>


==== Execute the tests ====
==== Execute the tests ====
Line 201: Line 373:
## C:\STAF> STAF local SELENIUM HELP
## C:\STAF> STAF local SELENIUM HELP
# Run a suite
# Run a suite
## C:\STAF> STAF local SELENIUM EXECUTE SERVER server.com ROOT /p4/main/ZimbraSelenium JARFILE /p4/main/ZimbraSelenium/build/dist/zimbra-6.0.0/lib/zimbraselenium.jar PATTERN projects.mobile.tests.addressbook GROUP always GROUP sanity LOG /tmp
## C:\STAF> STAF <client-machine> SELENIUM EXECUTE ROOT c:/zimbra/zm-selenium/build/dist/SelNG JARFILE c:/zimbra/zm-selenium/build/dist/SelNG/jars/zimbraselenium.jar PATTERN projects.ajax.tests GROUP always GROUP sanity CONFIG c:/zimbra/zm-selemium/conf/config.properties HOST <client-machine> LOG c:/zimbra/zm-selemium/test-output/ LOG4j c:/zimbra/zm-selemium/conf/log4j.properties


Example help:
Example help:


<pre>
<pre>
C:\STAF>
C:\STAF>
C:\STAF>STAF local SELENIUM HELP


Response
Response
Line 215: Line 383:
StafTest Service Help
StafTest Service Help


EXECUTE SERVER <servername|IP address> ROOT <ZimbraSelenium path> JARFILE <path>  
EXECUTE SERVER <servername|IP address> ROOT <zm-selenium path> JARFILE <path>  
PATTERN <projects.zcs.tests> [ GROUP <always|sanity|smoke|full> ]*  
PATTERN <projects.zcs.tests> [ GROUP <always|sanity|smoke|full> ]*  
[ LOG <folder> ] [ LOG4J <properties file> ]
[ LOG <folder> ] [ LOG4J <properties file> ]
Line 224: Line 392:


HELP
HELP
C:\STAF>
</pre>
</pre>


Example execute:
Example execute:
Line 235: Line 398:
<pre >  
<pre >  


C:\STAF>
C:\STAF> STAF local SELENIUM EXECUTE ROOT c:/zimbra/zm-selenium/build/dist/SelNG JARFILE c:/zimbra/zm-selenium/build/dist/SelNG/jars/zimbraselenium.jar PATTERN projects.ajax.tests GROUP always GROUP sanity CONFIG c:/zimbra/zm-selemium/conf/config.properties HOST local LOG c:/zimbra/zm-selemium/test-output/ LOG4j c:/zimbra/zm-selemium/conf/log4j.properties
C:\STAF>
C:\STAF> STAF local SELENIUM EXECUTE SERVER server.com ROOT /p4/main/ZimbraSelenium JARFILE  
/p4/main/ZimbraSelenium/build/dist/zimbra-6.0.0/lib/zimbraselenium.jar PATTERN  
projects.mobile.tests.addressbook GROPU always GROUP sanity LOG /tmp
Response
Response
---------------
---------------
Line 247: Line 406:
Total Skipped: 0
Total Skipped: 0


C:\Documents and Settings\zimbra>
C:\STAF>
</pre>
</pre>


Line 282: Line 436:
   Apache Ant version 1.7.0 compiled on December 13 2006
   Apache Ant version 1.7.0 compiled on December 13 2006


===== Running the tests =====
==== Running the tests ====


# cd to ZimbraSelenium dir
# cd to zm-selenium dir
# run ant targets:  
# run ant targets:  
  ant "Run tests"
Runs both ajax and html suite


   ant "Run ajax tests"  
   ant "Run ajax tests"  
Copies i18n and test data, compiles the code, creates the jar file and runs ajax suite
Copies i18n and test data, compiles the code, creates the jar file and runs ajax suite


  ant "Run html tests"
===Note/FAQs===
Copies i18n and test data, compiles the code, creates the jar file and runs html suite
 
* Why QA uses STAF and is this recommended for Dev environment?
 
<pre>
Yes. QA recommends STAF for dev environment too for better reliability and avoiding fixed wait for related testcases. QA uses STAF mainly for inter-communication between two machines for e.g. selenium test sends email and now you want to check whether remote server has successfully delivered email or you. Selenium test will check postqueue by sending STAF command internally as:
 
STAF <server> PROCESS START SHELL COMMAND "su - zimbra -c 'postqueue -p'" RETURNSTDOUT RETURNSTDERR WAIT 30000
 
If above command returns "Mail queue is empty" then queue is empty otherwise mail sending might be in progress.
 
</pre>
 
* Should browser window remain active when executing selenium tests?
 
<pre>
Yes. Many tests emulates keyboard related actions and types like user is typing using keyboard so selenium opened browser must remain active when running selenium tests and reliability perspective.
</pre>
 
* Mail inject fails using lmtpInject method. How can we resolve this issue?
 
<pre>
 
For lmtpInject, you need java cacerts inside mentioned directory for security perspective for e.g.
 
In Windows:
 
C:\opt\zimbra\common\lib\jvm\java\jre\lib\security
 
In Mac:
 
/opt/zimbra/common/lib/jvm/java/jre/lib/security
 
Create directories like above path. Search cacerts from your Java installed directory, copy it and put in security folder to resolve it.
</pre>
 
* My keyboard/mouse focus moves to Java foreground process window in between of the run and test fails (for e.g. when selenium/java tries to type into body using keyboard and looses focus using Robot class). How can we resolve this issue?
 
<pre>
 
Please add following line in your bash_profile to resolve it:
export JAVA_TOOL_OPTIONS="-Dapple.awt.UIElement=true"
 
</pre>


{{Article Footer|unknown|11/7/2007}}
{{Article Footer|unknown|11/7/2007}}

Latest revision as of 14:02, 3 November 2017

General Information

zm-selenium is the automation test harness for verifying the browser based Zimbra clients.

Prerequisites

  1. Create .zcs-deps directory at your %UserProfile%
  2. Install ant, ivy, Java1.8+ and set respective environment variables.
  3. Copy following jar at %UserProfile%/.zcs-deps:


   ant-contrib-1.0b1.jar

Downloading the Dependencies

The zm-selenium project depends on the following Zimbra projects:

  1. zimbra-package-stub
  2. zm-zcs
  3. zm-mailbox
  4. zm-web-client
  5. zm-ajax
  6. zm-zimlets

Clone these projects from GitHub in C:/zimbra/ directory as an example:

cd C:/zimbra/
git clone https://github.com/Zimbra/zimbra-package-stub.git
git clone https://github.com/Zimbra/zm-zcs.git
git clone https://github.com/Zimbra/zm-mailbox.git
git clone https://github.com/Zimbra/zm-web-client.git
git clone https://github.com/Zimbra/zm-ajax.git
git clone https://github.com/Zimbra/zm-zimlets.git

Build zm-native and zm-common jar using zm-mailbox repo.

Go to zm-mailbox and build using following command.
      ant publish-local-all -Dzimbra.buildinfo.version=8.8.0_GA

It will create zm-common.jar and zm-native.jar file

Downloading zm-selenium

Clone zm-selenium from github in same folder:

git clone https://github.com/Zimbra/zm-selenium.git

Building zm-selenium using Eclipse

Download and install latest eclipse IDE and Ivy plugin.

Build the Code

Use the following steps to build the code:

  1. Create new eclipse workspace
    1. Open eclipse
    2. Create new workspace in C:/zimbra/
  2. Add zm-selenium project to eclipse
    1. Select File->New->Project->Java Project
    2. For "Project Name", enter zm-selenium
  3. Build zm-selenium
    1. Expand the zm-selenium project
    2. Select build.xml, right click, select "Run As"->"Ant Build"

- it will create 3 jars at ..\zm-selenium\build\dist\lib: coverage.jar, resources.jar and zimbraselenium.jar

The output in the console will look like:

...
compile:
     [echo] Compiling framework...
    [javac] Compiling 1728 source files to C:\zimbra\zm-selenium\build\classes
jar:
      [jar] Building jar: C:\zimbra\zm-selenium\build\dist\lib\zimbraselenium.jar
BUILD SUCCESSFUL
Total time: 27 seconds

Once the project built and no dependency errors were found, you are almost ready to run your tests. Just configure the tests to work properly in your environment as explained in the next section and you'll be ready to test.

Tests Configuration

The file located in zm-selenium/conf/conf.properties is the master configuration file. Every parameter change required to run the tests properly is configurable using this properties file. Configure the test harness with the basic parameters required to run properly against your desired version of Zimbra by changing the following variables in the file:

  • server.host: This is the name of server in which the zimbra is installed. Please notice that the same must be reachable from the client machine in which test cases will be executed.
  • browser: The browser in which the tests should run. This is the standard Selenium browser string.
Screen Resolution

Be sure to set client resolution to 1024 X 768 or higher, otherwise the zimbra client will default to the HTML client (See also https://bugzilla.zimbra.com/show_bug.cgi?id=27521).

Running zm-selenium

Running from Eclipse

Java Target

You can also run selenium tests using Java target for e.g. create java run using:

Project: zm-selenium
Main class: com.zimbra.qa.selenium.framework.core.ExecuteHarnessMain
Program arguments: -j C:\opt\qa\JUDASPRIEST\zm-selenium\build\dist\lib\zimbraselenium.jar -p com.zimbra.qa.selenium.projects.ajax.tests -g always,sanity -l conf/log4j.properties

The zm-selenium source tree includes a build.xml file that has targets to execute the test suite.

  1. Open zm-selenium/build.xml
  2. Open the "outline view", if not already open
    1. Window -> Show View -> Outline
  3. Locate the "Run-ExecuteHarnessMain" target
  4. Set the jarfile, pattern, and groups desired to be executed
  5. Right-click on "Run-ExecuteHarnessMain" in the Outline, and select "Run"

Here is an example build.xml "Run-ExecuteHarnessMain" configuration, which runs the "sanity" and "smoke" mail compose test cases for Zimbra Ajax client suite.

<target name="Run-ExecuteHarnessMain" depends="jar" description="Run all tests per specified arguments">
	<property name="jarfile" value="${dist.lib.dir}/zimbraselenium.jar"/>
	<property name="pattern" value="ajax.test.mail.compose"/>
	<property name="groups" value="always,sanity,smoke"/>
	<echo>Executing ...</echo>
	<java classname="framework.core.ExecuteHarnessMain" classpathref="class.path" fork="true" failonerror="true">
		<arg line="-j '${jarfile}' -p ${pattern} -g ${groups} -l conf/log4j.properties"/>				
	</java>
</target>

Notes:

  • "always" must be included in the "groups" property
  • "zimbraselenium.jar" must be specified in as the "jarfile" property

Running from Eclipse (Dev Environment)

The zm-selenium source tree is designed to run in the Zimbra Development Environment. The Dev Environment is used to develop and build the Zimbra product. When developing web clients or zimlets, the Zimbra Selenium test suite can be used to Sanity Test code changes.

Some test suite setup needs to be completed to execute the suite.

Modify zm-selenium Configuration File

Modify the zm-selenium/conf/config.properties

  • Zimbra Server -- Define the Zimbra server against which you want to run your tests
server.scheme=http 
# server.host value should be same as the default domain name; hence it can't be an IP address
server.host=host.local 
server.port=7070
# store.host would be same as server.host
store.host=host.local
admin.port=7071
  • Browser
# Standard Selenium browser
# Note: Running Selenium tests on firefox are not fully supported (see https://bugzilla.mozilla.org/show_bug.cgi?id=1303234).
browser=*googlechrome
#browser=*firefox
OS=OSX
  • disable staf
postqueue.use.staf=false
  • non-staf sleep duration
postqueue.sleep.nonstaf.msec=10000
Create global admin user

The harness uses global admin user for some initial SOAP requests, so you must create an global admin account named 'globaladmin' with password 'test123':

zmprov ca globaladmin@host.local test123 zimbraIsAdminAccount TRUE
ANT Targets
  • Run all the test via command line. (Developers will never really have a need for running all tests)
ant Run-ExecuteHarnessMain

You can check the output by loading the output index.html file -- zm-selenium/test-output/VERSION/AJAX/BROWSER/en_US/TestNG

  • Target a specific pattern, e.g. all mail compose tests.
ant Run-ExecuteHarnessMain -Dpattern ajax.tests.mail.compose
  • You may also need to modify the group. The following will run the smoke tests for quick reply.
ant Run-ExecuteHarnessMain -Dpattern ajax.tests.conversation.quickreply -Dgroups always,smoke

Currently, there are three groups defined as of now:

  1. "sanity" - create basic items
  2. "smoke" - take basic actions on basic items
  3. "functional" - everything else

Additionally, these are four other related groups:

  1. "L0" - smoke test
  2. "L1" - critical path
  3. "L2" - functional/negative path
  4. "L3" - edge case
Sanity is very basic and probably doesn't cover much code other than making sure navigation to the different tabs works and create an item for all apps.
Smoke probably gets the most coverage in the least amount of tests. It covers internal operations like delete, modify, rename etc. for apps.
Functional can be used when the change was very specific and the tests can be filtered to a pattern. It covers all narrowed down functionality of zimbra.

(Note - When specifying any of the above three groups, "always" group, as shown in above example, must also be explicitly specified)

  • Another example for navigating away from preferences
ant Run-ExecuteHarnessMain -Dpattern ajax.tests.preferences.gui.NavigateAway -Dgroups always,functional
  • To get a count/preview of the tests that are going to be executed, use Sum-ExecuteHarnessMain:
C:\> ant Sum-ExecuteHarnessMain -Dpattern ajax.tests.conversation.quickreply -Dgroups always,smoke
*****
1: Quick Reply (Forward) a conversation (1 message, 1 recipient)
2: Quick Reply to a conversation (1 message, 1 recipient)
3: Quick Reply to a conversation (1 message, 1 recipient)
Number of matching test cases: 3

Remember - On an average, each test takes about a minute to run. You should also not use other applications on your mac while the tests are running due to selenium keyboard input simulation that happens during the run. So basically, after you have made your change, select the tests that you want to run, count them to get an estimate of how long it would take for them to run, and then leave your mac and take a break while the tests are running.

To see all available tests you can use your IDE and browse the source tree or navigate via the command line:

csb-2-eng-dhcp258:main testuser$ cd zm-selenium/src/java/com/zimbra/qa/selenium/projects/ajax/tests
csb-2-eng-dhcp258:tests testuser$ cd mail
csb-2-eng-dhcp258:mail testuser$ ls
assistant	feeds		mountpoints	readreceipt
attachments	folders		newwindow	sort
bugs		gui		performance	tags
compose		mail		quickcommands	undo
csb-2-eng-dhcp258:mail testuser$ 

Debugging test failures

A test failure could be due to a bug in your code, a bug in the test case (less likely), or an intermittent timing issue with the test run.

The first thing to try would be to rerun the failing tests and see if they still occur on a rerun. It may help to run the 'clean' step first:

ant clean Run-ExecuteHarnessMain -Dpattern ajax.tests.conversation.quickreply -Dgroups always,smoke

You can usually find the error (typically an exception) by scrolling back up through the output. Beyond that you can look at the debug logs (and screenshots too!) for each test under zm-selenium/test-output/VERSION/AJAX/BROWSER/en_US/debug directory. If there are multiple test failures you may want to copy the list of failed tests somewhere before re-running tests individually, because each ant test run overrides the previous test report.

If you have Ubuntu Dev setup and tests are failing from Mac:

a. If you are getting this error while running tests on Mac system: File not found /opt/zimbra/common/lib/jvm/java/jre/lib/security/cacerts

  • Find your current JDK :
% printenv JAVA_HOME
  • Create the directory where ZCS looks for certificates:
mkdir -p /opt/zimbra/common/lib/jvm/java/jre/lib/security
  • Copy system cacerts to Zimbra:
cp -f <Path-to-JDK>/Contents/Home/jre/lib/security/cacerts /opt/zimbra/common/lib/jvm/java/jre/lib/security/

b. If you are getting this exception : javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException

  • Export Zimbra certificate from Ubuntu VM:
cd /opt/zimbra
keytool -exportcert -keystore jetty/etc/keystore -file cert -alias jetty

Password : zimbra

  • Copy /opt/zimbra/cert to Mac environment.
  • Import Certificate in Mac:
keytool -importcert -keystore cacerts -file <path to cert file in mac>

Password: changeit

Running from STAF

The zm-selenium test suite can be configured using STAF. Zimbra Selenium harness depends on STAF being installed on both the test client and zimbra server. See the design overview for more details.

Install STAF on both the client and server. And configure security level 5 for communication from the client to the server. For example, if the client IP address is 192.168.1.1 and the server IP address is 192.168.1.2, run this command to set security level 5:


root@192.168.1.1:~# ssh root@192.168.1.2
root@192.168.1.2's password: *****
root@192.168.1.2:~# staf local trust set machine 192.168.1.1 level 5
Response
--------

root@192.168.1.2:~#

Build the zm-selenium STAF jar

  1. Open zm-selenium/build.xml
  2. Open the "outline view", if not already open
    1. Window -> Show View -> Outline
  3. Locate the "staf-jar" target
  4. Right-click on "staf-jar" in the Outline, and select "Run"
  5. The jar file should be built (zm-selenium/build/dist/lib/zimbrastaf.jar)
...
jar:
jar-staf-selenium:
     [copy] Copying 697 files to C:\zimbra\zm-selenium\build\staf\selenium\STAF-INF\classes
     [copy] Copied 171 empty directories to 1 empty directory under C:\zimbra\zm-selenium\build\staf\selenium\STAF-INF\classes
     [copy] Copying 23 files to C:\zimbra\zm-selenium\build\staf\selenium\STAF-INF\jars
      [jar] Building jar: C:\zimbra\zm-selenium\build\dist\lib\zimbrastaf.jar
jar-staf:
BUILD SUCCESSFUL
Total time: 33 seconds


Configure the STAF jar

Configure zimbrastaf.jar with STAF

  1. Open C:\STAF\bin\STAF.cfg using a text editor
  2. Configure the service by adding "service seleniumm library JSTAF execute /zimbra/zm-selenium/build/dist/lib/zimbrastaf.jar"
  3. Stop STAF
    1. C:\STAF> staf local shutdown shutdown
  4. Start STAF
    1. C:\STAF> stafproc

Staf.cfg example:

# Turn on tracing of internal errors and deprecated options
trace enable tracepoints "error deprecated"

# Enable TCP/IP connections
interface ssl library STAFTCP option Secure=Yes option Port=6550


# Set default local trust
trust level 2 default
trust machine local://local level 5

# Add default service loader
serviceloader library STAFDSLS
Add JSTAF Service

STAF <client-machine> SERVICE ADD SERVICE SELENIUM LIBRARY JSTAF EXECUTE c:/zimbra/zm-selenium/build/dist/SelNG/jars/zimbrastaf.jar

Execute the tests

Use STAF commands to run tests.

  1. Get the service help
    1. C:\STAF> STAF local SELENIUM HELP
  2. Run a suite
    1. C:\STAF> STAF <client-machine> SELENIUM EXECUTE ROOT c:/zimbra/zm-selenium/build/dist/SelNG JARFILE c:/zimbra/zm-selenium/build/dist/SelNG/jars/zimbraselenium.jar PATTERN projects.ajax.tests GROUP always GROUP sanity CONFIG c:/zimbra/zm-selemium/conf/config.properties HOST <client-machine> LOG c:/zimbra/zm-selemium/test-output/ LOG4j c:/zimbra/zm-selemium/conf/log4j.properties

Example help:


Response
--------
StafTest Service Help

EXECUTE SERVER <servername|IP address> ROOT <zm-selenium path> JARFILE <path> 
PATTERN <projects.zcs.tests> [ GROUP <always|sanity|smoke|full> ]* 
[ LOG <folder> ] [ LOG4J <properties file> ]

QUERY -- TBD: should return statistics on active jobs

HALT <TBD> -- TBD: should stop any executing tests

HELP

Example execute:

 

C:\STAF> STAF local SELENIUM EXECUTE ROOT c:/zimbra/zm-selenium/build/dist/SelNG JARFILE c:/zimbra/zm-selenium/build/dist/SelNG/jars/zimbraselenium.jar PATTERN projects.ajax.tests GROUP always GROUP sanity CONFIG c:/zimbra/zm-selemium/conf/config.properties HOST local LOG c:/zimbra/zm-selemium/test-output/ LOG4j c:/zimbra/zm-selemium/conf/log4j.properties
Response
---------------
Total Tests: 2
Total Passed: 2
Total Failed: 0
Total Skipped: 0

Running from the command line

Running with ant commands

Installing Ant

You need Ant 1.7+ to be installed on your system.

Notice: If you are running Mac OS X 10.5 or newer you can safely skip this step — the system comes with Ant 1.7.0 pre-installed.

If you are running an older version on Mac OS X you still need to install Ant 1.7: the pre-installed Ant version is too much behind. Follow the instructions described in the "Other Systems" section.

Ubuntu

If you are running Ubuntu 8.04 "Hardy Heron", the easier way to install Ant 1.7 is to run:

 sudo apt-get install ant ant-optional

Other Systems

Install Ant 1.7.x and make sure it is in your PATH:

Download the binary distribution.
Unpack the binary distribution in the directory of your choice.
Add <where you unpacked the zip file>/apache-ant-1.7.0/bin to your PATH environment variable.
Check that your Ant installation is OK:

 ant -version
 Apache Ant version 1.7.0 compiled on December 13 2006

Running the tests

  1. cd to zm-selenium dir
  2. run ant targets:
 ant "Run ajax tests" 

Copies i18n and test data, compiles the code, creates the jar file and runs ajax suite

Note/FAQs

  • Why QA uses STAF and is this recommended for Dev environment?
Yes. QA recommends STAF for dev environment too for better reliability and avoiding fixed wait for related testcases. QA uses STAF mainly for inter-communication between two machines for e.g. selenium test sends email and now you want to check whether remote server has successfully delivered email or you. Selenium test will check postqueue by sending STAF command internally as:

STAF <server> PROCESS START SHELL COMMAND "su - zimbra -c 'postqueue -p'" RETURNSTDOUT RETURNSTDERR WAIT 30000

If above command returns "Mail queue is empty" then queue is empty otherwise mail sending might be in progress.

  • Should browser window remain active when executing selenium tests?
Yes. Many tests emulates keyboard related actions and types like user is typing using keyboard so selenium opened browser must remain active when running selenium tests and reliability perspective.
  • Mail inject fails using lmtpInject method. How can we resolve this issue?

For lmtpInject, you need java cacerts inside mentioned directory for security perspective for e.g.

In Windows:

C:\opt\zimbra\common\lib\jvm\java\jre\lib\security

In Mac:

/opt/zimbra/common/lib/jvm/java/jre/lib/security

Create directories like above path. Search cacerts from your Java installed directory, copy it and put in security folder to resolve it.
  • My keyboard/mouse focus moves to Java foreground process window in between of the run and test fails (for e.g. when selenium/java tries to type into body using keyboard and looses focus using Robot class). How can we resolve this issue?

Please add following line in your bash_profile to resolve it:
export JAVA_TOOL_OPTIONS="-Dapple.awt.UIElement=true"

Verified Against: unknown Date Created: 11/7/2007
Article ID: https://wiki.zimbra.com/index.php?title=Testing:_Selenium:_Running_ZimbraSelenium Date Modified: 2017-11-03



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search