Testing: Selenium: Running ZimbraSelenium

General Information

zm-selenium is the automation test harness for verifying the browser based Zimbra clients.

Prerequisites

  1. Create .zcs-deps directory at your %UserProfile%
  2. Install ant, ivy, Java1.8+ and set respective environment variables.
  3. Copy following jar at %UserProfile%/.zcs-deps:


   ant-contrib-1.0b1.jar

Downloading the Dependencies

The zm-selenium project depends on the following Zimbra projects:

  1. zimbra-package-stub
  2. zm-zcs
  3. zm-mailbox
  4. zm-web-client
  5. zm-ajax
  6. zm-zimlets

Clone these projects from github in C:/zimbra/ directory as an example:

cd C:/zimbra/
git clone https://github.com/Zimbra/zimbra-package-stub.git
git clone https://github.com/Zimbra/zm-zcs.git
git clone https://github.com/Zimbra/zm-mailbox.git
git clone https://github.com/Zimbra/zm-web-client.git
git clone https://github.com/Zimbra/zm-ajax.git
git clone https://github.com/Zimbra/zm-zimlets.git

Build zm-native and zm-common jar using zm-mailbox repo.

Go to zm-mailbox and build using following command.
      ant publish-local-all -Dzimbra.buildinfo.version=8.7.6_GA

It will create zm-common.jar and zm-native.jar file

Downloading zm-selenium

Clone zm-selenium from github in same folder:

git clone https://github.com/Zimbra/zm-selenium.git

Building zm-selenium using Eclipse

Download and install latest eclipse IDE and Ivy plugin.

Build the Code

Use the following steps to build the code:

  1. Create new eclipse workspace
    1. Open eclipse
    2. Create new workspace in C:/zimbra/
  2. Add zm-selenium project to eclipse
    1. Select File->New->Project->Java Project
    2. For "Project Name", enter zm-selenium
  3. Build zm-selenium
    1. Expand the zm-selenium project
    2. Select build.xml, right click, select "Run As"->"Ant Build"

- it will create 3 jars at ..\zm-selenium\build\dist\lib: coverage.jar, resources.jar and zimbraselenium.jar

The output in the console will look like:

...
compile:
     [echo] Compiling framework...
    [javac] Compiling 1728 source files to C:\zimbra\zm-selenium\build\classes
jar:
      [jar] Building jar: C:\zimbra\zm-selenium\build\dist\lib\zimbraselenium.jar
BUILD SUCCESSFUL
Total time: 27 seconds

Once the project built and no dependency errors were found, you are almost ready to run your tests. Just configure the tests to work properly in your environment as explained in the next section and you'll be ready to test.

Tests Configuration

The file located in zm-selenium/conf/conf.properties is the master configuration file. Every parameter change required to run the tests properly is configurable using this properties file. Configure the test harness with the basic parameters required to run properly against your desired version of Zimbra by changing the following variables in the file:

  • server.host: This is the name of server in which the zimbra is installed. Please notice that the same must be reachable from the client machine in which test cases will be executed.
  • browser: The browser in which the tests should run. This is the standard Selenium browser string.
Screen Resolution

Be sure to set client resolution to 1024 X 768 or higher, otherwise the zimbra client will default to the HTML client (See also https://bugzilla.zimbra.com/show_bug.cgi?id=27521).

Running zm-selenium

Running from Eclipse

Java Target

You can also run selenium tests using Java target for e.g. create java run using:

Project: zm-selenium
Main class: com.zimbra.qa.selenium.framework.core.ExecuteHarnessMain
Program arguments: -j C:\opt\qa\JUDASPRIEST\zm-selenium\build\dist\lib\zimbraselenium.jar -p com.zimbra.qa.selenium.projects.ajax.tests -g always,sanity -l conf/log4j.properties

The zm-selenium source tree includes a build.xml file that has targets to execute the test suite.

  1. Open zm-selenium/build.xml
  2. Open the "outline view", if not already open
    1. Window -> Show View -> Outline
  3. Locate the "Run-ExecuteHarnessMain" target
  4. Set the jarfile, pattern, and groups desired to be executed
  5. Right-click on "Run-ExecuteHarnessMain" in the Outline, and select "Run"

Here is an example build.xml "Run-ExecuteHarnessMain" configuration, which runs the "sanity" and "smoke" mail compose test cases for Zimbra Ajax client suite.

<target name="Run-ExecuteHarnessMain" depends="jar" description="Run all tests per specified arguments">
	<property name="jarfile" value="${dist.lib.dir}/zimbraselenium.jar"/>
	<property name="pattern" value="ajax.test.mail.compose"/>
	<property name="groups" value="always,sanity,smoke"/>
	<echo>Executing ...</echo>
	<java classname="framework.core.ExecuteHarnessMain" classpathref="class.path" fork="true" failonerror="true">
		<arg line="-j '${jarfile}' -p ${pattern} -g ${groups} -l conf/log4j.properties"/>				
	</java>
</target>

Notes:

  • "always" must be included in the "groups" property
  • "zimbraselenium.jar" must be specified in as the "jarfile" property

Running from Eclipse (Dev Environment)

The zm-selenium source tree is designed to run in the Zimbra Development Environment. The Dev Environment is used to develop and build the Zimbra product. When developing web clients or zimlets, the Zimbra Selenium test suite can be used to Sanity Test code changes.

Some test suite setup needs to be completed to execute the suite.

Modify zm-selenium Configuration File

Modify the zm-selenium/conf/config.properties

  • Zimbra Server -- Define the Zimbra server against which you want to run your tests
server.scheme=http 
# server.host value should be same as the default domain name; hence it can't be an IP address
server.host=host.local 
server.port=7070
# store.host would be same as server.host
store.host=host.local
admin.port=7071
  • Browser
# Standard Selenium browser
# Note: Running Selenium tests on firefox are not fully supported (see https://bugzilla.mozilla.org/show_bug.cgi?id=1303234).
browser=*googlechrome
#browser=*firefox
OS=OSX
  • disable staf
postqueue.use.staf=false
  • non-staf sleep duration
postqueue.sleep.nonstaf.msec=10000
Create global admin user

The harness uses global admin user for some initial SOAP requests, so you must create an global admin account named 'globaladmin' with password 'test123':

zmprov ca globaladmin@host.local test123 zimbraIsAdminAccount TRUE
ANT Targets
  • Run all the test via command line. (Developers will never really have a need for running all tests)
ant Run-ExecuteHarnessMain

You can check the output by loading the output index.html file -- zm-selenium/test-output/VERSION/AJAX/BROWSER/en_US/TestNG

  • Target a specific pattern, e.g. all mail compose tests.
ant Run-ExecuteHarnessMain -Dpattern ajax.tests.mail.compose
  • You may also need to modify the group. The following will run the smoke tests for quick reply.
ant Run-ExecuteHarnessMain -Dpattern ajax.tests.conversation.quickreply -Dgroups always,smoke

Currently, there are three groups defined as of now:

  1. "sanity" - create basic items
  2. "smoke" - take basic actions on basic items
  3. "functional" - everything else

Additionally, these are four other related groups:

  1. "L0" - smoke test
  2. "L1" - critical path
  3. "L2" - functional/negative path
  4. "L3" - edge case
Sanity is very basic and probably doesn't cover much code other than making sure navigation to the different tabs works and create an item for all apps.
Smoke probably gets the most coverage in the least amount of tests. It covers internal operations like delete, modify, rename etc. for apps.
Functional can be used when the change was very specific and the tests can be filtered to a pattern. It covers all narrowed down functionality of zimbra.

(Note - When specifying any of the above three groups, "always" group, as shown in above example, must also be explicitly specified)

  • Another example for navigating away from preferences
ant Run-ExecuteHarnessMain -Dpattern ajax.tests.preferences.gui.NavigateAway -Dgroups always,functional
  • To get a count/preview of the tests that are going to be executed, use Sum-ExecuteHarnessMain:
C:\> ant Sum-ExecuteHarnessMain -Dpattern ajax.tests.conversation.quickreply -Dgroups always,smoke
*****
1: Quick Reply (Forward) a conversation (1 message, 1 recipient)
2: Quick Reply to a conversation (1 message, 1 recipient)
3: Quick Reply to a conversation (1 message, 1 recipient)
Number of matching test cases: 3

Remember - On an average, each test takes about a minute to run. You should also not use other applications on your mac while the tests are running due to selenium keyboard input simulation that happens during the run. So basically, after you have made your change, select the tests that you want to run, count them to get an estimate of how long it would take for them to run, and then leave your mac and take a break while the tests are running.

To see all available tests you can use your IDE and browse the source tree or navigate via the command line:

csb-2-eng-dhcp258:main testuser$ cd zm-selenium/src/java/com/zimbra/qa/selenium/projects/ajax/tests
csb-2-eng-dhcp258:tests testuser$ cd mail
csb-2-eng-dhcp258:mail testuser$ ls
assistant	feeds		mountpoints	readreceipt
attachments	folders		newwindow	sort
bugs		gui		performance	tags
compose		mail		quickcommands	undo
csb-2-eng-dhcp258:mail testuser$ 

Debugging test failures

A test failure could be due to a bug in your code, a bug in the test case (less likely), or an intermittent timing issue with the test run.

The first thing to try would be to rerun the failing tests and see if they still occur on a rerun. It may help to run the 'clean' step first:

ant clean Run-ExecuteHarnessMain -Dpattern ajax.tests.conversation.quickreply -Dgroups always,smoke

You can usually find the error (typically an exception) by scrolling back up through the output. Beyond that you can look at the debug logs (and screenshots too!) for each test under zm-selenium/test-output/VERSION/AJAX/BROWSER/en_US/debug directory. If there are multiple test failures you may want to copy the list of failed tests somewhere before re-running tests individually, because each ant test run overrides the previous test report.

If you have Ubuntu Dev setup and tests are failing from Mac:

a. If you are getting this error while running tests on Mac system: File not found /opt/zimbra/common/lib/jvm/java/jre/lib/security/cacerts

  • Find your current JDK :
% printenv JAVA_HOME
  • Create the directory where ZCS looks for certificates:
mkdir -p /opt/zimbra/common/lib/jvm/java/jre/lib/security
  • Copy system cacerts to Zimbra:
cp -f <Path-to-JDK>/Contents/Home/jre/lib/security/cacerts /opt/zimbra/common/lib/jvm/java/jre/lib/security/

b. If you are getting this exception : javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException

  • Export Zimbra certificate from Ubuntu VM:
cd /opt/zimbra
keytool -exportcert -keystore jetty/etc/keystore -file cert -alias jetty

Password : zimbra

  • Copy /opt/zimbra/cert to Mac environment.
  • Import Certificate in Mac:
keytool -importcert -keystore cacerts -file <path to cert file in mac>

Password: changeit

Running from STAF

The zm-selenium test suite can be configured using STAF. Zimbra Selenium harness depends on STAF being installed on both the test client and zimbra server. See the design overview for more details.

Install STAF on both the client and server. And configure security level 5 for communication from the client to the server. For example, if the client IP address is 192.168.1.1 and the server IP address is 192.168.1.2, run this command to set security level 5:


root@192.168.1.1:~# ssh root@192.168.1.2
root@192.168.1.2's password: *****
root@192.168.1.2:~# staf local trust set machine 192.168.1.1 level 5
Response
--------

root@192.168.1.2:~#

Build the zm-selenium STAF jar

  1. Open zm-selenium/build.xml
  2. Open the "outline view", if not already open
    1. Window -> Show View -> Outline
  3. Locate the "staf-jar" target
  4. Right-click on "staf-jar" in the Outline, and select "Run"
  5. The jar file should be built (zm-selenium/build/dist/lib/zimbrastaf.jar)
...
jar:
jar-staf-selenium:
     [copy] Copying 697 files to C:\zimbra\zm-selenium\build\staf\selenium\STAF-INF\classes
     [copy] Copied 171 empty directories to 1 empty directory under C:\zimbra\zm-selenium\build\staf\selenium\STAF-INF\classes
     [copy] Copying 23 files to C:\zimbra\zm-selenium\build\staf\selenium\STAF-INF\jars
      [jar] Building jar: C:\zimbra\zm-selenium\build\dist\lib\zimbrastaf.jar
jar-staf:
BUILD SUCCESSFUL
Total time: 33 seconds


Configure the STAF jar

Configure zimbrastaf.jar with STAF

  1. Open C:\STAF\bin\STAF.cfg using a text editor
  2. Configure the service by adding "service seleniumm library JSTAF execute /zimbra/zm-selenium/build/dist/lib/zimbrastaf.jar"
  3. Stop STAF
    1. C:\STAF> staf local shutdown shutdown
  4. Start STAF
    1. C:\STAF> stafproc

Staf.cfg example:

# Turn on tracing of internal errors and deprecated options
trace enable tracepoints "error deprecated"

# Enable TCP/IP connections
interface ssl library STAFTCP option Secure=Yes option Port=6550


# Set default local trust
trust level 2 default
trust machine local://local level 5

# Add default service loader
serviceloader library STAFDSLS
Add JSTAF Service

STAF <client-machine> SERVICE ADD SERVICE SELENIUM LIBRARY JSTAF EXECUTE c:/zimbra/zm-selenium/build/dist/SelNG/jars/zimbrastaf.jar

Execute the tests

Use STAF commands to run tests.

  1. Get the service help
    1. C:\STAF> STAF local SELENIUM HELP
  2. Run a suite
    1. C:\STAF> STAF <client-machine> SELENIUM EXECUTE ROOT c:/zimbra/zm-selenium/build/dist/SelNG JARFILE c:/zimbra/zm-selenium/build/dist/SelNG/jars/zimbraselenium.jar PATTERN projects.ajax.tests GROUP always GROUP sanity CONFIG c:/zimbra/zm-selemium/conf/config.properties HOST <client-machine> LOG c:/zimbra/zm-selemium/test-output/ LOG4j c:/zimbra/zm-selemium/conf/log4j.properties

Example help:


Response
--------
StafTest Service Help

EXECUTE SERVER <servername|IP address> ROOT <zm-selenium path> JARFILE <path> 
PATTERN <projects.zcs.tests> [ GROUP <always|sanity|smoke|full> ]* 
[ LOG <folder> ] [ LOG4J <properties file> ]

QUERY -- TBD: should return statistics on active jobs

HALT <TBD> -- TBD: should stop any executing tests

HELP

Example execute:

 

C:\STAF> STAF local SELENIUM EXECUTE ROOT c:/zimbra/zm-selenium/build/dist/SelNG JARFILE c:/zimbra/zm-selenium/build/dist/SelNG/jars/zimbraselenium.jar PATTERN projects.ajax.tests GROUP always GROUP sanity CONFIG c:/zimbra/zm-selemium/conf/config.properties HOST local LOG c:/zimbra/zm-selemium/test-output/ LOG4j c:/zimbra/zm-selemium/conf/log4j.properties
Response
---------------
Total Tests: 2
Total Passed: 2
Total Failed: 0
Total Skipped: 0

Running from the command line

Running with ant commands

Installing Ant

You need Ant 1.7+ to be installed on your system.

Notice: If you are running Mac OS X 10.5 or newer you can safely skip this step — the system comes with Ant 1.7.0 pre-installed.

If you are running an older version on Mac OS X you still need to install Ant 1.7: the pre-installed Ant version is too much behind. Follow the instructions described in the "Other Systems" section.

Ubuntu

If you are running Ubuntu 8.04 "Hardy Heron", the easier way to install Ant 1.7 is to run:

 sudo apt-get install ant ant-optional

Other Systems

Install Ant 1.7.x and make sure it is in your PATH:

Download the binary distribution.
Unpack the binary distribution in the directory of your choice.
Add <where you unpacked the zip file>/apache-ant-1.7.0/bin to your PATH environment variable.
Check that your Ant installation is OK:

 ant -version
 Apache Ant version 1.7.0 compiled on December 13 2006

Running the tests

  1. cd to zm-selenium dir
  2. run ant targets:
 ant "Run ajax tests" 

Copies i18n and test data, compiles the code, creates the jar file and runs ajax suite

Note/FAQs

  • Why QA uses STAF and is this recommended for Dev environment?
Yes. QA recommends STAF for dev environment too for better reliability and avoiding fixed wait for related testcases. QA uses STAF mainly for inter-communication between two machines for e.g. selenium test sends email and now you want to check whether remote server has successfully delivered email or you. Selenium test will check postqueue by sending STAF command internally as:

STAF <server> PROCESS START SHELL COMMAND "su - zimbra -c 'postqueue -p'" RETURNSTDOUT RETURNSTDERR WAIT 30000

If above command returns "Mail queue is empty" then queue is empty otherwise mail sending might be in progress.

  • Should browser window remain active when executing selenium tests?
Yes. Many tests emulates keyboard related actions and types like user is typing using keyboard so selenium opened browser must remain active when running selenium tests and reliability perspective.
  • Mail inject fails using lmtpInject method. How can we resolve this issue?

For lmtpInject, you need java cacerts inside mentioned directory for security perspective for e.g.

In Windows:

C:\opt\zimbra\common\lib\jvm\java\jre\lib\security

In Mac:

/opt/zimbra/common/lib/jvm/java/jre/lib/security

Create directories like above path. Search cacerts from your Java installed directory, copy it and put in security folder to resolve it.
  • My keyboard/mouse focus moves to Java foreground process window in between of the run and test fails (for e.g. when selenium/java tries to type into body using keyboard and looses focus using Robot class). How can we resolve this issue?

Please add following line in your bash_profile to resolve it:
export JAVA_TOOL_OPTIONS="-Dapple.awt.UIElement=true"

Verified Against: unknown Date Created: 11/7/2007
Article ID: https://wiki.zimbra.com/index.php?title=Testing:_Selenium:_Running_ZimbraSelenium Date Modified: 06/9/2017



Try Zimbra

Try Zimbra Collaboration with a 60-day free trial.
Get it now »

Want to get involved?

You can contribute in the Community, Wiki, Code, or development of Zimlets.
Find out more. »

Looking for a Video?

Visit our YouTube channel to get the latest webinars, technology news, product overviews, and so much more.
Go to the YouTube channel »

Jump to: navigation, search