Testing: Selenium: ZimbraSelenium Overview

 - This is archive documentation, which means it is not supported or valid for recent versions of Zimbra Collaboration.

General Information

The Zimbra Selenium Harness is a system testing test harness for the Zimbra Web Clients. The Selenium harness uses Java, Selenium, and TestNG technologies to drive web client usage against the Zimbra server.

These Zimbra clients are tested by the Selenium Harness:

• Ajax Client
• HTML Client
• Mobile Client
• Admin Console
• Desktop

Source Code

See Building ZimbraSelenium

Javadoc

Javadoc can be generated from the source code tree. After building the software, run the "javadocs" target from ZimbraSelenium/build.xml. A javadoc tree will be created in ZimbraSelenium/build/generated/javadocs. Open index.html to view the javadocs.

Project Layout

Project Structure

The following folders are located in the ZimbraSelenium java project.

Selenium Structure

• conf: contains configuration files for how the harness executes
• data: contains test data files, such as MIME files, ICS files.
• jars: contains third party external jar files
• src/bin: contains test scripts, such as perl scripts and shell scripts
• src/java: contains java source code
• build.xml: project ant build file

Java Structure

The following major packages are located in the ZimbraSelenium java project.

• framework: basic java classes that apply to all testable clients
• projects: java classes specific to the tested clients
• staf: java classes that apply to integration with [STAF]

Framework

The framework package contains basic classes that apply to all testable clients.

"framework.core" contains the main execution method and Zimbra selenium classes.

"framework.items" contains classes that define basic Zimbra objects, such as Mail, Appointments, Contacts, Tasks, Tags, and Folders.

"framework.ui" contains abstract classes that define basic GUI objects, such as an Client Application, Application Page, Displayed Object, Form Object, Folder Tree Object.

"framework.util" contains utilities classes, such as injecting a MIME or creating a Zimbra Account.

Projects

Each web application is contained in a package in projects.

• projects.admin: java classes specific to the Admin Console
• projects.ajax: java classes specific to the Ajax client
• projects.html: java classes specific to the HTML client
• projects.mobile: java classes specific to the Mobile client
• projects.desktop: java classes specific to the Desktop client

Each project contains 3 sub-packages:

• projects.name.tests: all TestNG test classes are placed in "tests". Classes are further separated by functional areas within the package.
• projects.name.ui: all abstract classes from "framework.ui" are placed in "ui".
• projects.name.core: all non-test classes and non-ui classes are placed in the core package

Coding Conventions

Java coding style standard

Java coding standard@vmware

Zimbra coding style convention

To aid in understanding the purpose for classes, methods, and variables, the following coding conventions are used.

Variables start with a lower case letter as follows:

• "s" - re-use of "Selenium" methods. For example, AbsSeleniumObject.sIsElementPresent() simply points to DefaultSelenium.isElementPresent()
• "z" - applies to Zimbra specific use-case or functionality. For example, PageMail.zToolbarPressButton()
• "g" - applies to a "GUI" specific data. For example, MailItem.gIsFlagged
• "d" - applies to general "Data" for the item. This data could come be a general property or come from SOAP. For example, MailItem.dIsFlagged

Class names for UI objects should start with the abstract class type. For example, PageMail begins with "Page" because is extends from AbsPage.

Groups

Test cases are separated into the following groups: always, sanity, smoke, functional, deprecated, unstable

Always

The always group is used to specify test methods that always execute. Mainly, this group is used for @Before* and @After* annotated methods.

Sanity

The sanity group is used to annotate a select group of test methods that can be used as a sanity suite. The sanity suite should complete quickly (in a few minutes) and provide a broad verification of basic functionality.

Currently, only test cases that create basic objects in the GUI are marked as sanity. For example:

• Create a message
• Create a folder
• Create a contact
• Create an addressbook
• Create an appointment
• Create a calendar
• Create a task
• Create a task list
• Create a briefcase document
• Upload a briefcase file
• Create a briefcase folder
• Create a tag
• Create a saved search

Smoke

The smoke group is used to annotate a select group of test methods that can be used as a smoke suite. The smoke suite should complete relatively quickly (in a few hours) and provide a detailed verification of basic functionality.

Currently, test cases that apply all basic actions to all basic objects in the GUI are marked as smoke. For example:

• Basic objects
  • Messages
  • Contacts
  • Appointments/Meeting Requests
  • Tasks
  • Briefcase Files/Documents
  • Folders, Addressbooks, Calendars, Task List, Briefcase folders, Mountpoints
  • Tags
  • Saved Searches
• Basic actions
  • Create
  • Edit
  • Delete
  • Move/Drag and Drop
  • Undo
  • Share
  • Flag/Unflag
  • Tag/Untag
  • Mark read/unread

A cross matrix of objects and actions defines the smoke suite. For instance, the test case that verifies a message (basic object) can be flagged (basic action) is in the smoke suite.

Functional

The functional group is used to annotate a group of test methods that can be used as a full regression suite. The functional suite includes all remaining active test cases.

Deprecated

The deprecated group is used to mark individual test cases that are no longer active, due to removed or changed functionality.

Unstable

The unstable group is used to mark individual test cases that are unstable. Some test cases may block the harness from executing, and these test methods must be removed from the active test suites to reduce false negative results.

Harness Design

Business Logic

The Selenium Harness is designed with a Business Logic layer that abstracts the GUI implementation from the Test Case methods. With the abstraction layer, test case maintenance is reduced when the GUI implementation is changed. For example, to compose a new mail, the test method code may look like:

//// This is an example of a test method to compose a new mail


// Create the application object
Application app = new ZimbraAjaxClientApplication();

// Click the new button, get the compose page
ComposePage compose = app.mailPage.toolbar(Button.NEW);

// Fill out the compose page
compose.to = "foo@example.com";
compose.subject = "itinerary";
compose.body = "Meet me at the Zimbra Offices at 10:00 AM";

// Send the message
compose.send();

The test case methods use general end-user steps - the 'business logic' that does not change in an email application. The test case methods are not required to use any reference of the GUI implementation or selenium data (such as locators). The Application, ZimbraAjaxClientApplication, and ComposePage classes are defined in the test harness, and those classes contain the logic to drive the specified user actions.

For most basic test cases, it is suggested to use the test harness building blocks within the test methods. However, a test method may be verifying a feature that is not defined in the test harness, and in those cases, the test methods should invoke lower level Selenium requests.

Test Method Structure

Most GUI test cases fall into 2 categories:

1. GUI presentation tests
2. GUI action tests

GUI Presentation Tests

GUI presentation tests use the GUI to verify how Zimbra mailbox data is presented to the end user. Test methods normally use 2 steps:

1. Data setup
2. GUI verification

In Data setup, test case preconditions are set up. When a mailbox requires data to be created, it is preferred to use SOAP, REST, LMTP, etc. to set up such data which is much faster and more stable than using the GUI. Setup actions may include injecting a message into the mailbox (LMTP), sending a appointment invitation from a separate account (SOAP), posting a file to the briefcase (REST), setting the account preferences (SOAP), etc.

In GUI verification, the displayed data in the GUI is measured and verified. The Zimbra Selenium Harness has methods to get displayed information as objects and simple strings.

For example, a test case objective may be to "Verify the initial search string is used":

//// ** Data setup

// Change account setting
String initialSearchPref = "is:unread";
account.modifyPref("zimbraPrefMailInitialSearch", initialSearchPref);


//// ** GUI or Data verification

// Verify the message list in the GUI contains the new message
app.refresh();
String displayedSearch = app.SearchPage.getSearchString();
ZAssert.assertEquals(initialSearchPref, displayedSearech", "Verify the initial search string is used");

GUI Action Tests

GUI action tests use the GUI under test to modify Zimbra mailbox data. Test methods normally use 3 steps:

1. Data setup
2. GUI action
3. GUI or Data verification

Data setup, is the same as GUI presentation tests.

In GUI action, the test case actions are executed. Actions may include navigation to a separate app such as Contacts, refreshing the calendar by clicking on the Refresh button, sending a new message, etc.

In GUI or Data verification, the test verification points are measured. The Zimbra Selenium Harness uses a ZAssert class to verify. Where possible, it is preferred to use SOAP, REST, etc. to verify data (but, obviously these interfaces will not work when verifying a GUI requirement).

For example, a test case objective may be to "Verify an incoming message is placed in the inbox list."

//// ** Data setup

// Inject the message using LMTP
LMTPUtil.inject(testaccount, "C:\data\mimesample.txt");
String subject = "mimesample";

//// ** GUI action

// Refresh the inbox list
app.MailPage.toolbarClick(Button.GetMail);

//// ** GUI or Data verification

// Verify the message list in the GUI contains the new message
List<Message> messages app.MailPage.getMessageList();
ZAssert.assertContains(messages, new Message(subject), "Verify the new message appears in the inbox list");

Migrating From Selenium 1 RC to Selenium 2 WebDriver

Basic Info

Getting Started

In order to use WebDriver libraries in the existing Selenium test harness several
common classes have been modified across the framework:
-com.zimbra.qa.selenium.framework.core.ClientSession
-com.zimbra.qa.selenium.framework.core.ExecuteHarnessMain
-com.zimbra.qa.selenium.framework.ui.AbsSeleniumObject
-com.zimbra.qa.selenium.framework.util.ZimbraSeleniumProperties
-com.zimbra.qa.selenium.projects.ajax.core.AjaxCommonTest

To turn on WebDriver mode in the test execution it is required to uncomment the following line 
in the ZimbraSelenium/conf/config.properties:
seleniumDriver=WebDriver 

Common Problems

-Element cannot be scrolled into view:[object HTMLInputElement]issue:3075
-Unable to click drop down link (Sub menu) issue:3082
-By.cssSelector with space works incorrectly in IE issue:2949
-css locator: the Descendant selector does not work with nth-child issue:762
-CSS option selectors are mutually exclusive in IE7/8 issue: 1518
-Firefox driver is inconsistent when it has to scroll down to find an element to interact with issue:1861
-WebDriver is not deleting the profile directory after test exits issue: 1934
-Chrome Driver can’t find elements after switching to new frame issue:1969
-xpath determines differently in Selenium 1 and Selenium 2 issue:2001
-Inconsistent driver behavior when switching to a frame by index issue:2162
-Selenium2: selenium is hanging when doing "click" in one window, which will trigger alert in another window issue:2274
-Selenium 2 /WebDriver Migration Issue. WebDriver Does not select correct element issue:2354
-Click on element in iframe does nothing in IE driver issue: 2387
-RenderedWebElementTest#testMoveRelativeToBody is incredibly flaky in Chrome issue:2559
-Cannot locate the element with exact matching id value (only for IE) issue:2753
-WebDriver hangs on frame change to AJAX long polling/server push frame issue:2767
-getWindowHandles() not working while using ChromeDriver issue:2833
-Unable to find elements in some pages, but can find in others [Windows XP, Internet Explorer 8, Selenium-java] issue:2973
-Get Window Handle Hangs issue:3615
-click() requires multiple calls to fire event with IEDriver issue:3623

Lessons Learned

Selenium Tips

Selenium Knowledge