Testing: Selenium: ZimbraSelenium Overview
The Zimbra Selenium Harness is a system testing test harness for the Zimbra Web Clients. The Selenium harness uses Java, Selenium, and TestNG technologies to drive web client usage against the Zimbra server.
These Zimbra clients are tested by the Selenium Harness:
- Ajax Client
- HTML Client
- Mobile Client
- Admin Console
Javadoc can be generated from the source code tree. After building the software, run the "javadocs" target from ZimbraSelenium/build.xml. A javadoc tree will be created in ZimbraSelenium/build/generated/javadocs. Open index.html to view the javadocs.
Project StructureThe following folders are located in the ZimbraSelenium java project.
- conf: contains configuration files for how the harness executes
- data: contains test data files, such as MIME files, ICS files.
- jars: contains third party external jar files
- src/bin: contains test scripts, such as perl scripts and shell scripts
- src/java: contains java source code
- build.xml: project ant build file
The following major packages are located in the ZimbraSelenium java project.
- framework: basic java classes that apply to all testable clients
- projects: java classes specific to the tested clients
- staf: java classes that apply to integration with [STAF]
The framework package contains basic classes that apply to all testable clients.
"framework.core" contains the main execution method and Zimbra selenium classes.
"framework.items" contains classes that define basic Zimbra objects, such as Mail, Appointments, Contacts, Tasks, Tags, and Folders.
"framework.ui" contains abstract classes that define basic GUI objects, such as an Client Application, Application Page, Displayed Object, Form Object, Folder Tree Object.
"framework.util" contains utilities classes, such as injecting a MIME or creating a Zimbra Account.
Each web application is contained in a package in projects.
- projects.admin: java classes specific to the Admin Console
- projects.ajax: java classes specific to the Ajax client
- projects.html: java classes specific to the HTML client
- projects.mobile: java classes specific to the Mobile client
- projects.desktop: java classes specific to the Desktop client
Each project contains 3 sub-packages:
- projects.name.tests: all TestNG test classes are placed in "tests". Classes are further separated by functional areas within the package.
- projects.name.ui: all abstract classes from "framework.ui" are placed in "ui".
- projects.name.core: all non-test classes and non-ui classes are placed in the core package
Variables start with a lower case letter as follows:
- "s" - re-use of "Selenium" methods. For example, AbsSeleniumObject.sIsElementPresent() simply points to DefaultSelenium.isElementPresent()
- "z" - applies to Zimbra specific use-case or functionality. For example, PageMail.zToolbarPressButton()
- "g" - applies to a "GUI" specific data. For example, MailItem.gIsFlagged
- "d" - applies to general "Data" for the item. This data could come be a general property or come from SOAP. For example, MailItem.dIsFlagged
Class names for UI objects should start with the abstract class type. For example, PageMail begins with "Page" because is extends from AbsPage.
Test cases are separated into the following groups: always, sanity, smoke, functional, deprecated, unstable
The always group is used to specify test methods that always execute. Mainly, this group is used for @Before* and @After* annotated methods.
The sanity group is used to annotate a select group of test methods that can be used as a sanity suite. The sanity suite should complete quickly (in a few minutes) and provide a broad verification of basic functionality.
Currently, only test cases that create basic objects in the GUI are marked as sanity. For example:
- Create a message
- Create a folder
- Create a contact
- Create an addressbook
- Create an appointment
- Create a calendar
- Create a task
- Create a task list
- Create a briefcase document
- Upload a briefcase file
- Create a briefcase folder
- Create a tag
- Create a saved search
The smoke group is used to annotate a select group of test methods that can be used as a smoke suite. The smoke suite should complete relatively quickly (in a few hours) and provide a detailed verification of basic functionality.
Currently, test cases that apply all basic actions to all basic objects in the GUI are marked as smoke. For example:
- Basic objects
- Appointments/Meeting Requests
- Briefcase Files/Documents
- Folders, Addressbooks, Calendars, Task List, Briefcase folders, Mountpoints
- Saved Searches
- Basic actions
- Move/Drag and Drop
- Mark read/unread
A cross matrix of objects and actions defines the smoke suite. For instance, the test case that verifies a message (basic object) can be flagged (basic action) is in the smoke suite.
The functional group is used to annotate a group of test methods that can be used as a full regression suite. The functional suite includes all remaining active test cases.
The deprecated group is used to mark individual test cases that are no longer active, due to removed or changed functionality.
The unstable group is used to mark individual test cases that are unstable. Some test cases may block the harness from executing, and these test methods must be removed from the active test suites to reduce false negative results.
The Selenium Harness is designed with a Business Logic layer that abstracts the GUI implementation from the Test Case methods. With the abstraction layer, test case maintenance is reduced when the GUI implementation is changed. For example, to compose a new mail, the test method code may look like:
//// This is an example of a test method to compose a new mail // Create the application object Application app = new ZimbraAjaxClientApplication(); // Click the new button, get the compose page ComposePage compose = app.mailPage.toolbar(Button.NEW); // Fill out the compose page compose.to = "firstname.lastname@example.org"; compose.subject = "itinerary"; compose.body = "Meet me at the Zimbra Offices at 10:00 AM"; // Send the message compose.send();
The test case methods use general end-user steps - the 'business logic' that does not change in an email application. The test case methods are not required to use any reference of the GUI implementation or selenium data (such as locators). The Application, ZimbraAjaxClientApplication, and ComposePage classes are defined in the test harness, and those classes contain the logic to drive the specified user actions.
For most basic test cases, it is suggested to use the test harness building blocks within the test methods. However, a test method may be verifying a feature that is not defined in the test harness, and in those cases, the test methods should invoke lower level Selenium requests.
Test Method Structure
Most GUI test cases fall into 2 categories:
- GUI presentation tests
- GUI action tests
GUI Presentation Tests
GUI presentation tests use the GUI to verify how Zimbra mailbox data is presented to the end user. Test methods normally use 2 steps:
- Data setup
- GUI verification
In Data setup, test case preconditions are set up. When a mailbox requires data to be created, it is preferred to use SOAP, REST, LMTP, etc. to set up such data which is much faster and more stable than using the GUI. Setup actions may include injecting a message into the mailbox (LMTP), sending a appointment invitation from a separate account (SOAP), posting a file to the briefcase (REST), setting the account preferences (SOAP), etc.
In GUI verification, the displayed data in the GUI is measured and verified. The Zimbra Selenium Harness has methods to get displayed information as objects and simple strings.
For example, a test case objective may be to "Verify the initial search string is used":
//// ** Data setup // Change account setting String initialSearchPref = "is:unread"; account.modifyPref("zimbraPrefMailInitialSearch", initialSearchPref); //// ** GUI or Data verification // Verify the message list in the GUI contains the new message app.refresh(); String displayedSearch = app.SearchPage.getSearchString(); ZAssert.assertEquals(initialSearchPref, displayedSearech", "Verify the initial search string is used");
GUI Action Tests
GUI action tests use the GUI under test to modify Zimbra mailbox data. Test methods normally use 3 steps:
- Data setup
- GUI action
- GUI or Data verification
Data setup, is the same as GUI presentation tests.
In GUI action, the test case actions are executed. Actions may include navigation to a separate app such as Contacts, refreshing the calendar by clicking on the Refresh button, sending a new message, etc.
In GUI or Data verification, the test verification points are measured. The Zimbra Selenium Harness uses a ZAssert class to verify. Where possible, it is preferred to use SOAP, REST, etc. to verify data (but, obviously these interfaces will not work when verifying a GUI requirement).
For example, a test case objective may be to "Verify an incoming message is placed in the inbox list."
//// ** Data setup // Inject the message using LMTP LMTPUtil.inject(testaccount, "C:\data\mimesample.txt"); String subject = "mimesample"; //// ** GUI action // Refresh the inbox list app.MailPage.toolbarClick(Button.GetMail); //// ** GUI or Data verification // Verify the message list in the GUI contains the new message List<Message> messages app.MailPage.getMessageList(); ZAssert.assertContains(messages, new Message(subject), "Verify the new message appears in the inbox list");
Migrating From Selenium 1 RC to Selenium 2 WebDriver
In order to use WebDriver libraries in the existing Selenium test harness several common classes have been modified across framework: com.zimbra.qa.selenium.framework.core.ClientSession com.zimbra.qa.selenium.framework.core.ExecuteHarnessMain com.zimbra.qa.selenium.framework.ui.AbsSeleniumObject com.zimbra.qa.selenium.framework.util.ZimbraSeleniumProperties com.zimbra.qa.selenium.projects.ajax.core.AjaxCommonTest To turn on WebDriver mode in the test execution it is required to uncomment the following line in the ZimbraSelenium/conf/config.properties: seleniumDriver=WebDriver
Element cannot be scrolled into view:[object HTMLInputElement]issue:3075 Unable to click drop down link (Sub menu) issue:3082 By.cssSelector with space works incorrectly in IE issue:2949 css locator: the Descendant selector does not work with nth-child issue:762 CSS option selectors are mutually exclusive in IE7/8 issue: 1518 Firefox driver is inconsistent when it has to scroll down to find an element to interact with issue:1861 WebDriver is not deleting the profile directory after test exits issue: 1934 Chrome Driver can’t find elements after switching to new frame issue:1969 xpath determines differently in Selenium 1 and Selenium 2 issue:2001 Inconsistent driver behavior when switching to a frame by index issue:2162 Selenium2: selenium is hanging when doing "click" in one window, which will trigger alert in another window issue:2274 Selenium 2 /WebDriver Migration Issue. WebDriver Does not select correct element issue:2354 Click on element in iframe does nothing in IE driver issue: 2387 RenderedWebElementTest#testMoveRelativeToBody is incredibly flaky in Chrome issue:2559 Cannot locate the element with exact matching id value (only for IE) issue:2753 WebDriver hangs on frame change to AJAX long polling/server push frame issue:2767 getWindowHandles() not working while using ChromeDriver issue:2833 Unable to find elements in some pages, but can find in others [Windows XP, Internet Explorer 8, Selenium-java] issue:2973