Mobile web testing

I've been working on a project recently which required testing on a mobile device. The project started in April this year and was focused solely on iOS.

When I looked into what was available for mobile testing I found a number of different tools:

• KIF
• Appium
• Frank
• Calabash
• EarlGray
• UI Automation

What I found next was a bother. I need a tool which would be used for testing the IPA we would ship from an app store. Tools like Frank and Calabash are great at automating tests but they required you to build a special version of the app. This would not be the same app you deployed to an app store.

This made it easy to eliminate those two great tools from my list of potential test automation tools.

I then looked into KIF and EarlGray. They had great reviews and looked really promising until I noticed Apple made significant changes to the UI Automation framework and broke KIF and EarlGray. So if I wanted to test against iOS 9.3 developed with XCode 7 and Swift I was probably not going to want to use KIF or EarlGray.

So the obvious choice was Appium. However, even Appium seem to be affected by the changes Apple announced at the July 2015 Developer Conference. :(

Since our iterations were one week and waiting to see who would 'fix' issues with their framework wasn't really an option, we branched Frank and started developing the app using Frank for UI testing. In the meantime we looked at UI Automation (our app was iOS only, so we didn't need to worry about Android support).

Initial use of UI Automation seemed good. So we started automating UI tests with it but continued to keep the Frank tests running in parallel. However after a few iterations we started to see maintaining the UI Automation tests was becoming increasingly difficult. Since I got into UI automation in 1998 I have found that failure to maintain a test automation framework is one of the common reasons for UI automation to fail. We didn't have a team of 20 QA Automation experts to keep the UI Automation framework going. :(

So I had a second look at the previously discarded frameworks. To my surprise and delight I found that support for them had been re-established and I took a second look at using Appium. 

Appium is definitely not fast and I'm looking for ways I can reduce the execution time of the Appium test suite (currently 15 minutes when run on hardware; I'd like to get it down to 5 minutes plus add more tests; maybe run tests in parallel on four or more phones).

Bottom line, Appium seems to be working well for us. I've created a page object model framework. In my next article I'll talk about using the Appium Inspector on a Mac laptop and things I found blocked me or slowed me down.

The difference between NotFoundException, NoSuchElementException and StateElementReferenceException

Been a while since I posted something here...

Recently, someone on the WebDriver Google Group asked what the difference between NotFoundException, NoSuchElementException and StateElementReferenceException are. Here is the answer I posted:

The NotFoundException is a super class which includes the subclass NoSuchElementException. The known direct subclasses of NotFoundException are: NoAlertPresentException, NoSuchContextException, NoSuchElementException, NoSuchFrameException and NoSuchWindowException. So if I want one catch statement to catch all five exception and they will all be handled the same way then I can just handle NotFoundException. But if I want to handle any of these five exceptions differently, I can catch the more specific subclass.

The NoSuchElementException is thrown when the element you are attempting to find is not in the DOM. This can happen for three reasons. 

The first is because the element does not exist and never will. To fix this, change your findElement to be correct.

The second is that you need to do something on the page to make the element appear. For example, the user selects Country and javascript populates a City field. If you attempt to look for a city before you select a country, the city you are looking for does not exist and you get a NoSuchElementException. To fix this you have to make sure the steps in your test are correct.

The third is that the element is generated by javascript but WebDriver attempts to find the element before the javascript has created it. The fix for this is to use WebDriverWait to wait for the element to appear (visibility and/or clickable).

StaleElementReferenceException is when you find an element, the DOM gets modified then you reference the WebElement. For example,

WebElement we = driver.findElement(By.cssSelector("#valid"));

// you do something which alters the page or a javascript event alters the page

we.click();

A classic example if this might be:

List<WebElement> listOfAnchors = driver.findElements(By.tag("a"));

for(WebElement anchor : listOfAnchors) {

anchor.click();

System.out.println(driver.getTitle());

driver.navigate.back();

}

This code will get all the anchor elements into a list. Lets say there are 5 anchors on the page. The list now has 5 WebElement references. We get the first reference and click it. This take us to a new page. This is a new DOM. We print the title of the new page. Then we use back() to go back to the original page. The DOM looks just like the same DOM but it is a different DOM. So now all the references in the list a stale. On the second iteration, it gets the second reference and clicks it. This will throw a StaleElementReferenceException.

More difficult to debug is:

WebElement we = driver.findElement(...);

// javascript event gets fired by the website

we.click();

Sometimes this will throw a StaleElementReferenceException but sometimes the timing will be different and the click will work. I've seen many people have this intermittent problem. They add more code which doesn't fix the problem. The extra code just changes the timing and hides the problem. The problem comes back a few days later. So they add more random code. It looks like they fixed the problem but they just changed the timing. So if you get a StaleElementReferenceException and it is not clear why, it is probably this problem and you need to figure out how to make the findElement and click atomic.

Difference between using current node (.) and text() function in XPath for Selenium locators

Recently someone asked about using the partial link text to find an anchor with an IMG tag in the middle of the text. The HTML snippet was:

<a href="something.html">
&nbsp;
<img src="filename.gif">
&nbsp;
partial link text
</a>

The initial attempt for a locator was:

"//a[contains(text(),'partial link text')]"

Normally I would expect this to work. However, the text() function does not seem to find it. Peter Jeffery Gale (thanks Peter) noticed that the following locator did work:

"//a[contains(.,'partial link text')]"

The . notation is the current node in the DOM. This is going to be an object of type Node. I'm guessing that the Node is getting cast to a string. Something similar to:

"//a[contains(string(.),'partial link text')]"

The end result seems to be that getting the entire Node, convert it to a string and scanning the string for a substring always works. Using the XPath function text() to get the text for an element only gets the text up to the first inner element. If the text you are looking for is after the inner element you must use the current node to search for the string and not the XPath text() function.