Google Analytics

Search

To search for specific articles you can use advanced Google features. Go to www.google.com and enter "site:darrellgrainger.blogspot.com" before your search terms, e.g.

site:darrellgrainger.blogspot.com CSS selectors

will search for "CSS selectors" but only on my site.


Saturday, February 6, 2010

Watij versus Selenium-RC [Java]

If you want to use open source test automation frameworks for testing web applications there are two categories. Some of the frameworks have their own engines (CSS, Javascript, HTML, etc.) and others use a real web browser.

One advantage of the frameworks with their own engines is they have more control and you don't have to deal with the real life implementations of a web browser. You are also not dealing with TCP/IP, network delays, etc.

The disadvantage, on the other hand, you are not dealing with real life implementations of a web browser. You are also not dealing with TCP/IP, network delays, etc. Basically, if you use a framework with its own engine and your application passes, it does not guarantee it will work on a real web browser like Firefox or Internet Explorer.

Two frameworks which drive a real web browser are Watij (Web Application Testing In Java) and Selenium (pronounced sa-LEE-nee-am).

Watij is a Java test framework. You will need to know Java and jUnit to use Watij. The documentation is VERY lacking. The people developing Watij are developers. They are not creating a commercial product. If I was programming a calculator, adding comments to the source code would not be important. Instead, I would want the source code to be self-documenting. The method names should tell me what they do. The class names should imply what methods exist in it (i.e. I'd know what to expect in the file even before I opened it). Additionally, I would write unit test cases to show how the code was expected to be use.

As something built on the jUnit framework, I expected better documentation. I quickly realized that it was best to have the source code jars added to my project and occasionally needed to step into the Watij code to figure out how things worked.

Everything was nicely structured however. You have HtmlElement which was the base class for all other elements. You have things like Button, Table, Link, etc. which inherit HtmlElement. These are all interfaces and you have implementations like IEButton. I believe the idea was that some day you might have IEButton, FirefoxButton, ChromeButton, OperaButton, etc.

In Java I might have:

public List doSomething() {
List text = new ArrayList();
// more code here
return text;
}

Similarly, in Watij all the methods return an interface rather than a specific implementation. So the signature for a method returning an IELink is:

public Link getLink(String locator);

Nothing special for a Java test framework. Additionally, a Table extends a TableBody, a TableBody (and subsequently a Table) can tell you how many rows and columns it has. It can also return a specific TableRow or TableCell. Back to the fact there is no JavaDoc for the methods but it does not made because the code is self-documenting.

When a method need to return a collection of elements, e.g.

Links links = ie.links();

The implementation of Links is typical Java. It assumes we have Collections, Lists, Maps, etc. So the Links implementation is Iterable<link>. So we can have:

for(Link link : links) {
// do something with each link in the collection
}

Essentially, writing automation is no hard for a Java programmer than writing a simple application.

Selenium on the other hand is totally different from Watij. First, it is not a Java framework. Selenium comes in multiple parts and one of them allows you to use Java. The starting point for Selenium is Selenium IDE. You get an AddOn for Firefox. When you open the Selenium IDE it turns on recording right away. Everything you do in Firefox is then recorded into an HTML table. The table has three columns. The first column is the command, the second column is the target and the third column is an optional value. For example, if I wanted to click a button the command would be 'click' and the target would be some way of identifying the button. This could be CSS, an xpath, the id of the button, the name of the button, etc. Another example using all three columns would be entering text in a text field. The command would be 'type', the target would be the text field and the third field would be the text you want to put in the text field.

This is great for doing some quick and dirty automation but lacks the ability to build into a test suite. Additionally, if every test case started with logging into the application you would have duplicate code. If the way you log in changed, you would have to edit all the test cases. This is possibility the number one reason test automation fails. It is not maintainable.

The next part of Selenium is Selenium RC (remote control). After you record the test case using the IDE, you can save it in a variety of different languages. If you are a Java programmer, you can save it as Java code.

To run the Java code you'll need the Selenium RC jars and a Selenium Server. The way it works is the Selenium RC jar is a client. Your Java code sends commands to the server via the client. The server then uses Selenium IDE to control the web browser (via Javascript).

The nice thing about this is you can now record a snippet using the Selenium IDE, save it as Java and incorporate it into your Java test suite.

This is the theory. The reality is that how Selenium IDE recognizes the elements in the browser is not always the best choice. For example, I might have a set of SPAN elements inside the TABLE to make the cells format nicely. Along comes Internet Explorer 9 and my table looks horrible. So I have to add a DIV around each SPAN. From a customers point of view everything looks the same, but from a Selenium locator point of view my test cases all break. It can no longer find the cell. So you will need to look at the locator Selenium IDE has selected and decide if you want to change it.

Additionally, AJAX and timing are a huge issue. For example, you have a form with two select lists. The first list is a list of countries. The second list is empty. You select your country and an AJAX call populates the second list with states, provinces, territories, etc. Selenium IDE will record you waited 1 second before selecting your province because you selected Canada for the country. If you select United States it takes 2 seconds to populate the second list. A bad automator will just add a few seconds to the wait. A good automator will look for something in the DOM which signals the second list is populated. For example, the second list might be hidden. So you want to wait for the second list to become visible. Knowing to code this is one thing you need to have when using Selenium IDE and knowing what to code is even harder.

So the record feature is not that good. It will most often encourage less experienced automators to create code duplication and use sleep/wait statements to handle things like AJAX/Web 2.0 features.

Another issue with Selenium is the Java support. When you look at the Java support it is fairly simplistic. There is no hierarchy to the data structures and all the methods return arrays. You'll also find things like, I can find out how many elements match an xpath but I have no way of iterating over the list of elements. In Watij you can do the following:

List result = new ArrayList();
String myXPath = "//A[contains(text(), 'Phoebe')]";
Links phoebeLinks = ie.links(xpath(myXPath));
for(Link currentPhoebeLink : phoebeLinks) {
result.add(currentPhoebeLink.text());
}

To do the same thing in Selenium RC using Java, in theory, you need to do:

String[] allLinks = selenium.getAllLinks();

then iterate over all the strings to figure out which ones are the ones you are looking for. However, when I use this method it actually returns back an array of one element and the one string it does contain is "". In other words, it doesn't seem to work.

My workaround for this problem was to use the getHtml() method. This returns all the source code for the current page. I can then load it into something like jTidy and use the jTidy interface to traverse the page and find all the links. To actually iterate over all the links and do something with them I'd have to create a unique list of the links using jTidy then go back to Selenium and interact with them.

Essentially, I found that writing a test case in pure Java was simple with Watij and difficult if not sometimes impossible with Selenium RC. Ultimately, the maintenance on Selenium will be the potential downfall.

On the other hand, when I actually run the test cases I find that Internet Explorer has problems. It hangs and locks up occasionally. For an automated test suite this is unacceptable. With Watij I am stuck. With Selenium I can run the tests in Firefox, Opera, Safari or Internet Explorer. So automated nightly build and test is possible with Selenium but not really viable with Watij.

As I see it, Watij is a fantastic house built on a bad foundation. Selenium is a good starter home which can easily be moved to a variety of different foundations.

Neither is a great solution. I spent a year trying to work around the bugs in Internet Explorer with Watij but in the end just gave up. For now, I'm looking at Selenium. Maybe it is time to start doing some Selenium RC programming, i.e. give back to the community.

9 comments:

Jake Dempsey said...

Can you help us understand what "exactly"is lacking in the documentation of watij? Do you not find the multitude of unit tests as documentation or the documentation on the watij.com site good enough? Are you just referring to the fact that your IDE doesnt give you documentation on the fly with intellisense?

Also its a misnomer to say watij is built on top of jUnit. Its not. You can use watij with any test runner you like. We user junit for writing our unit tests for watij, but you can use anything you like as there is no coupling to a test runner.

I do however agree with you that IE is buggy, flakey, etc and can cause watij to hang because the actual browser is hanging as well. I have found though that the hanging can be fixed depending on the application.

Darrell said...

Jake, the comment about lack of documentation was relating to the lack of JavaDocs. I was talking from the point of view of someone who typically uses commerical packages like SQABasic (IBM/Rational Robot) or TSL (WinRunner). Most testers will expect documentation rather than solid programming and unit tests to explain how the code works.

I just re-read the blog entry and realize I left out one important paragraph.

"Watij is well documented from a programmer's perspective. The source code is self-documenting, the method names clearly tell me what they do, the classes imply what methods exist within it and the unit test cases show how the API should be used."

I should add that the consistency of Watij API makes this sort of documentation far less important than Selenium. I also love being able to give Watij an xpath which matches multiple cells, rows, links, whatever then get back an Iterable List of those elements.

My only major complaint about Watij is the fact it uses IE.

I recently read about the possibility to have Watij supporting other web browsers. If this happens I will be thrilled. If there is anything I can do to help please let me know.

I tried to figure out what the application was doing to make IE hang. I eliminated all the things people said would cause the problem. It reduce the hanging but never eliminated.

I tried running a thread in the background that monitored the status bar. If it did not change within one minute I assumed the browser had hung and sent a refresh. This helped a little more but not enough.

Jake Dempsey said...

Do you feel that the documentation at: http://watij.com/usage-guide/ is not adequate for testers? My goal was to provide "how" you would use watij rather than provide method level documentation in javadoc because of the inherent problem of those becoming stale.

Yeah I was looking at JxBrowser from TeamDev to make watij multi platform, multi browser, but it doesnt look like it can fit the bill. I was thinking of also looking into webdriver and some other technologies. Its a tough one :)

Darrell said...

Jake, the documentation at http://watij.com/usage-guide/ was a huge help when I first started using Watij and definitely something every Watij user should have readily available.

I agree that JavaDocs can become quickly stale. I guess documentation which does not go stale is part of what you are paying for when you buy a commercial product.
That is, if Watij cost $5000/year you could afford to hire technical writers to maintain the documentation. :)

Jake Dempsey said...

lol, yeah I could just do it myself then :)

Anonymous said...

Hi Darrell,

We already have a decent framework running with Selenium + mozilla. now, my application is also going to be supported on IE. Selenium and IE dont go very well together. My selenium gets hung up or cant find elements with the same x-path as on mozilla. I am also having problems finding equivalent tools like xpath checker, firebug on IE. Do you know any equivalent tools/addons/plugins etc? I was thinking of watij to be used on IE. But after your blog, i am not so sure about watij. I dont know what to do since selenium doesnt work.

Darrell said...

Hi Anonymous,

If you search Google for "dev toolbar for ie7" you should find links to a toolbar similar to Firebug. One of the results will be a Microsoft website (in case you are worried about a trojan). At the time of writing, you can download the toolbar from http://www.microsoft.com/downloads/en/details.aspx?familyid=95E06CBE-4940-4218-B75D-B8856FCED535&displaylang=en.

If you are using IE8 then just press F12. The development toolbar is built into IE8.

In some cases the xpath will be different because Internet Explorer isn't complete compliant with HTML standards. In other cases, the application server will actually send different web pages to IE. A third reason for xpaths on IE being different from Firefox can be javascript. Often web pages now have javascript which detects the browser model and version then modifies the page on the client side. If you are using IIS as the application server, it will often send nonstandard HTML when it detects the browser is IE.

You might want to go to the Watij web site. They have a new version fo Watij called WebSpec. It supports more than IE but is fairly different from the original Watij. I have not had a good look at WebSpec but it is a possibility.

Anonymous said...

Thank you Darrell for the tip on F12. Its much convenient to get an x-path. Though now, selenium says element not found when i can see the element on the develop toolbar. Missing x-path checker a lot! :(

I am hoping , I can make my framework work with selenium + IE. Using watij is all effort again.

So, now, I am trying to find out if the X-path in IE has any special requirements since it refuses to be found!

Darrell said...

One trick I do is using the old school method of 'print'. Just before you are attempting to locate something, get the HTML source of the page using the test automation tool.

Using Java and Selenium RC, if I was trying to find a button to click, just above the button click I would save the contents of selenium.getHtmlSource() into a variable. I'd then set a breakpoint on the line to click the button. When it reaches the breakpoint, I can examine the variable holding the results of selenium.getHtmlSource(). Run it once with Firefox then again with Internet Explorer. The getHtmlSource() method will show me what Selenium is seeing at that moment. This can often be different from what the browser will show you with 'View Source'.