FitNesse versus Robot Framework

In our last instalment, I announced a comparison of automatic web UI test tools using Selenium, a popular browser automation tool. It can be hard to define high-level scenarios in Selenium, so we decided to define scenarios in a high-level tool, which in turn drives Selenium. Doing a bit of online research, we found two popular high-level automatic test tools to compare: FitNesse and Robot Framework.

Both tools need extra libraries for Selenium integration. For Robot Framework, the Robot Framework developers maintain a Selenium integration library. For FitNesse, a third-party Selenium integration library named Selenesse is developed, but it only supports Selenium RC, the previous version of Selenium. After more online research, we uncovered Fitnium and Xebium and settled on using Xebium. The rationale behind this is discussed below.

Both high-level tools allow scenarios to be composed of other scenarios and low-level steps, such as button presses and assertions. This allows tests to have several abstraction layers before mapping them to low-level steps.

Fitnesse and Xebium

FitNesse is high-level testtool to be configured through a wiki. Steps are mapped to low-level actions using fixtures, of which Xebium is one for Selenium integration. For example, these low-level actions include starting a browser or pressing a button. FitNesse has several fixtures out-of-the-box: One for database access, for calling webservices, for Java Swing GUIs and a command line fixture to run tests against a command line.
As described above, Xebium is a Selenium fixture. It supports almost all of Selenium’s commands, including pressing buttons, typing text in textboxes, measuring response times and reading and writing variables. It comes with a Selenium IDE formatting script, allowing test scripts to be copied from Selenium IDE and pasted directly into FitNesse. These scripts will still require effort to be abstracted into high-level tests, but it is a user-friendly way to design a new test. Xebium is still in development, but not at a high pace or with a large team.

An advantage of these tools is FitNesse is a wiki, allowing multiple users to work simultaneously on testcases and scenarios. A disadvantage is its limited support in doing this: Naming of wikipages is limited to camel-cased names with at least two capital letters, which are used to split the name to words. It also does not feature help in composing testcases, such as suggesting scenario names or navigating to a scenario by clicking on it. There is a limited refactortool, allowing Selenium commands to be replaced by high-level scenarios.

Robot Framework

Robot Framework is a high-level testtool with a separately developed Windows UI named RIDE. Both are written in Python. It supports both the old and new version of Selenium: Selenium RC and Selenium WebDriver. This is done through a connecting library, respectively SeleniumLibrary and Selenium2Library). Robot Framework supports more backends than just Selenium.

Because Robot Framework uses a Windows UI, it can only be used by one user at a time. Cooperation can be achieved by bringing testcases and scenarios under source control, but Robot Framework has no further support for this source control.

Comparison

Eventually, we decided to use FitNesse and Xebium. FitNesse seems to have a larger community, which will speed up development and increase chances the project will continue to receive development  and support. FitNesse is developed in Java, which is more familiar to Info Support than Robot Framework´s Python. Also, FitNesse is a wiki with integrated webserver, which allows concurrent online access and making cooperation between testers easier. Finally, FitNesse has better readability of tests, writing user stories in natural language means they are better readable to stakeholders. These conclusions are summarised in the following table.

The next two paragraphs briefly discuss Selenesse and Fitnium and why they were not selected over Xebium.

Selenesse

Development on this FitNesse-Selenium fixture began using the name WebTest Fixture.  It is mentioned on the FitNesse website as one of the web UI fixtures for FitNesse. This fixture was dropped from selection because Selenesse does not support Selenium WebDriver. Support for Selenium RC will be dropped in the future, using Selenium WebDriver has preference.

Fitnium

This is also a FitNesse-Selenium fixture. Development seems to have stopped and there is little documentation available, which is why it was not selected for use.

Concluding

Next, we will discuss installing FitNesse, Xebium and Selenium, as well as best practices and lessons we learned using FitNesse and Xebium. Stay tuned!