Mobile web testing

I've been working on a project recently which required testing on a mobile device. The project started in April this year and was focused solely on iOS.

When I looked into what was available for mobile testing I found a number of different tools:

• KIF
• Appium
• Frank
• Calabash
• EarlGray
• UI Automation

What I found next was a bother. I need a tool which would be used for testing the IPA we would ship from an app store. Tools like Frank and Calabash are great at automating tests but they required you to build a special version of the app. This would not be the same app you deployed to an app store.

This made it easy to eliminate those two great tools from my list of potential test automation tools.

I then looked into KIF and EarlGray. They had great reviews and looked really promising until I noticed Apple made significant changes to the UI Automation framework and broke KIF and EarlGray. So if I wanted to test against iOS 9.3 developed with XCode 7 and Swift I was probably not going to want to use KIF or EarlGray.

So the obvious choice was Appium. However, even Appium seem to be affected by the changes Apple announced at the July 2015 Developer Conference. :(

Since our iterations were one week and waiting to see who would 'fix' issues with their framework wasn't really an option, we branched Frank and started developing the app using Frank for UI testing. In the meantime we looked at UI Automation (our app was iOS only, so we didn't need to worry about Android support).

Initial use of UI Automation seemed good. So we started automating UI tests with it but continued to keep the Frank tests running in parallel. However after a few iterations we started to see maintaining the UI Automation tests was becoming increasingly difficult. Since I got into UI automation in 1998 I have found that failure to maintain a test automation framework is one of the common reasons for UI automation to fail. We didn't have a team of 20 QA Automation experts to keep the UI Automation framework going. :(

So I had a second look at the previously discarded frameworks. To my surprise and delight I found that support for them had been re-established and I took a second look at using Appium. 

Appium is definitely not fast and I'm looking for ways I can reduce the execution time of the Appium test suite (currently 15 minutes when run on hardware; I'd like to get it down to 5 minutes plus add more tests; maybe run tests in parallel on four or more phones).

Bottom line, Appium seems to be working well for us. I've created a page object model framework. In my next article I'll talk about using the Appium Inspector on a Mac laptop and things I found blocked me or slowed me down.

The difference between NotFoundException, NoSuchElementException and StateElementReferenceException

Been a while since I posted something here...

Recently, someone on the WebDriver Google Group asked what the difference between NotFoundException, NoSuchElementException and StateElementReferenceException are. Here is the answer I posted:

The NotFoundException is a super class which includes the subclass NoSuchElementException. The known direct subclasses of NotFoundException are: NoAlertPresentException, NoSuchContextException, NoSuchElementException, NoSuchFrameException and NoSuchWindowException. So if I want one catch statement to catch all five exception and they will all be handled the same way then I can just handle NotFoundException. But if I want to handle any of these five exceptions differently, I can catch the more specific subclass.

The NoSuchElementException is thrown when the element you are attempting to find is not in the DOM. This can happen for three reasons. 

The first is because the element does not exist and never will. To fix this, change your findElement to be correct.

The second is that you need to do something on the page to make the element appear. For example, the user selects Country and javascript populates a City field. If you attempt to look for a city before you select a country, the city you are looking for does not exist and you get a NoSuchElementException. To fix this you have to make sure the steps in your test are correct.

The third is that the element is generated by javascript but WebDriver attempts to find the element before the javascript has created it. The fix for this is to use WebDriverWait to wait for the element to appear (visibility and/or clickable).

StaleElementReferenceException is when you find an element, the DOM gets modified then you reference the WebElement. For example,

WebElement we = driver.findElement(By.cssSelector("#valid"));

// you do something which alters the page or a javascript event alters the page

we.click();

A classic example if this might be:

List<WebElement> listOfAnchors = driver.findElements(By.tag("a"));

for(WebElement anchor : listOfAnchors) {

anchor.click();

System.out.println(driver.getTitle());

driver.navigate.back();

}

This code will get all the anchor elements into a list. Lets say there are 5 anchors on the page. The list now has 5 WebElement references. We get the first reference and click it. This take us to a new page. This is a new DOM. We print the title of the new page. Then we use back() to go back to the original page. The DOM looks just like the same DOM but it is a different DOM. So now all the references in the list a stale. On the second iteration, it gets the second reference and clicks it. This will throw a StaleElementReferenceException.

More difficult to debug is:

WebElement we = driver.findElement(...);

// javascript event gets fired by the website

we.click();

Sometimes this will throw a StaleElementReferenceException but sometimes the timing will be different and the click will work. I've seen many people have this intermittent problem. They add more code which doesn't fix the problem. The extra code just changes the timing and hides the problem. The problem comes back a few days later. So they add more random code. It looks like they fixed the problem but they just changed the timing. So if you get a StaleElementReferenceException and it is not clear why, it is probably this problem and you need to figure out how to make the findElement and click atomic.

Difference between using current node (.) and text() function in XPath for Selenium locators

Recently someone asked about using the partial link text to find an anchor with an IMG tag in the middle of the text. The HTML snippet was:

<a href="something.html">
&nbsp;
<img src="filename.gif">
&nbsp;
partial link text
</a>

The initial attempt for a locator was:

"//a[contains(text(),'partial link text')]"

Normally I would expect this to work. However, the text() function does not seem to find it. Peter Jeffery Gale (thanks Peter) noticed that the following locator did work:

"//a[contains(.,'partial link text')]"

The . notation is the current node in the DOM. This is going to be an object of type Node. I'm guessing that the Node is getting cast to a string. Something similar to:

"//a[contains(string(.),'partial link text')]"

The end result seems to be that getting the entire Node, convert it to a string and scanning the string for a substring always works. Using the XPath function text() to get the text for an element only gets the text up to the first inner element. If the text you are looking for is after the inner element you must use the current node to search for the string and not the XPath text() function.

Desktop automation tools

Talking to one of my colleagues today. He asked me about testing desktop applications. As someone who predates the Internet, testing desktop applications was not foreign to me. However I realized that many testers today have only worked on web or mobile device applications.

Testing things like Microsoft Word, Notepad, Calendar or Eclipse isn't something many testers have done.

You wouldn't use something like Selenium, cucumber or Watir to test a desktop application.

So where would you go to find desktop application test tools? The first place I look is http://en.wikipedia.org/wiki/List_of_GUI_testing_tools. At this time this page notes:

• AutoIt: http://www.autoitscript.com/site/autoit/ (free)
• IBM/Rational Functional Tester: http://www-03.ibm.com/software/products/en/functional (commercial)
• Quality First Software: http://www.qfs.de/en/index.html (commercial)
• Sikuli: http://www.sikuli.org/ (free)
• SilkTest: http://www.borland.com/products/silktest/ (commercial)
• Test Automation FX: http://www.testautomationfx.com/ (commercial)
• Telerik TestStudio: http://www.telerik.com/teststudio (commercial)

In addition to these links I would also check out:

• http://www.ministryoftesting.com/resources/software-testing-tools/
• http://sourceforge.net/directory/development/development-testing/freshness:recently-updated/

The first site has a wide range of links and it currently actively maintained. Unfortunately, it has more links for web and mobile testing but there are some desktop application tools listed.

The second link is to sourceforge. It will take you to the recently updated testing links. It includes open source applications for all sorts of testing. You can remove the "recently-updated" filter and see more but I usually restrict myself to recently updated tools. 

If the tool is for desktop testing and it has not been recently updated, there is a strong chance it does not work with modern applications or a current operating system. The older the tool the higher the risk it will not work on Windows 7 or 8, Mac OS X 10.8 or 10.9.

If you enter in a search term, like "testing" then a whole set of menus will appear below the search term and you can narrow things down even further. For example, after entering "testing" into the search text box I can select OS to be "Mac" and category "Quality Assurance".

I can also clear all the filters and search for things which might help narrow it down in different ways. For example, if I know how to program Java and I think I might want to add to the open source tool then I can filter for things with Programming Language "Java". Additionally, if the desktop application is written using Swing and Java I could search for Programming Language "Java", Category "Testing" then enter "swing". From the results I find jrobot (http://jayrobot.sourceforge.net/) and gtt (GUI Test Tool; http://gtt.sourceforge.net/). 

It is also worth just poking around sourceforge and keeping a mental note about things you find. This is because you'll be surprised what you stumble. For example, JFCUnit is a good tool for testing Java Swing applications but it didn't come with JFCUnit (http://jfcunit.sourceforge.net/).

Creating good locators

I am constantly seeing people look for tools which will create locators. I think a lot of us starting by using Firebug to examine the DOM and select locators. With Chrome I can now right-click on an element and select Inspect Element.

Now of this really helps figure out what is a good locator. So people are still looking for a tool which will tell them the best locator for an element.

If I go to http://www.google.com and inspect the Google Search button I will see:

<button class="gbqfba" aria-label="Google Search" id="gbqfba" name="btnK">
<span id="gbqfsa">Google Search</span>
</button>

We might think that the Selenium locator By.cssSelector("#gbqfba") might be a good locator. One definition of a good locator is that it needs to be consistent and unique. For years the Google Search button has had the id attribute of gbqfba. So it is consistent. It is an id and according to the HTML standard there can be only one element for each id value. So it will be unique.

But is it a good locator? I will argue that it is not a good locator. If I had the following Selenium code:

driver.findElement(By.cssSelector("#gbqfq")).sendKeys("Darrell Grainger");
driver.findElement(By.cssSelector("#gbqfba")).click();
assertThat(driver.findElement(By.xpath("//a[contains(text(),'Testing')]")).getText(), startsWith("QA"));

it will run consistently over years. But what happens if something changes and I need to refactor this code? The first line is sending my name to something with id='gbqfq', it is clicking something with id='gbqfba'. What if we changed the HTML to have:

<button class="gbqfba" aria-label="Google Search" id="gbqfba" data-qa='search button'>
<span id="gbqfsa" data-qa='text for search button'>Google Search</span>
</button>

then the Selenium code could read:

driver.findElement(By.cssSelector("[data-qa='search input box']")).sendKeys("Darrell Grainger");
driver.findElement(By.cssSelector("[data-qa='search button']")).click();
assertThat(driver.findElement(By.cssSelector("[data-qa='first search result']")).getText(), startsWith("QA"));

Now if I'm looking at the code and have no knowledge of the website being automated, it should be easy for me to figure out that the first element is the search input box, the second element is the search button and the last element is the first search result.

Essentially, by adding the data-qa attribute the elements we are trying to locate:

• easier to locate the element
• easier to read the code when running/refactoring it
• make the code less brittle

A possible complaint would be that adding these elements could interfere with other elements on the page. The whole 'data-' syntax is a standard for HTML5. If there is a change someone is going to use data-qa then you can use data-foo where foo is the name of your application or your company name or your name.

Another possibility is that it is not unique. It only has to be unique to the given page. If the value is incredibly descriptive then it should be unique. You make the value of data-qa reflect the information necessary for someone testing the application manually then it should be unique. If it is not unique then how would a manual tester find THE element if its description was not unique?

I also considered adding more data to the page would increase what the CSS and XPath engines had to index more information. So I tried loading up page with these attributes added and could not measure any noticeable impact on the performance.

The only other concern I can image is if I add too much data or too verbose descriptions on the data-qa values then I could increase the amount of data transmitted. More bytes transmitted might impact performance of the web page. Wasn't able to see any significant impact in this area but your mileage might vary. So this is something to watch out for.

Opening two windows for one WebDriver

I'm currently testing a project with two applications. The first is an administrator tool for updating/modifying the customer site and the second is the actual customer site.

There are three ways to test a project like this. The first is creating one instance of WebDriver and access each application one at a time, e.g.

WebDriver driver = new ChromeDriver();
driver.get(adminToolURL);
// do some admin stuff
driver.get(customerSiteURL);
// do some site stuff

The second would be to create two instances of WebDriver. One to access the admin tool and the other to access the customer site, e.g.

WebDriver adminDriver = new ChromeDriver();
adminDriver.get(adminToolURL);
WebDriver driver = new InternetExplorerDriver();
driver.get(customerSiteURL);
// do some admin stuff with adminDriver
// do some site stuff with driver

The nice thing about this scenario is that you might have different browser requirements. The admin tool only has to work with one browser and the customer site has to work with a variety of browsers. So I can hard code the adminDriver to one browser and configure the test framework to change the browser. Run 1, InternetExplorerDriver; run 2, ChromeDriver; run 3, SafariDriver; etc.

The third way is to create one instance of WebDriver and open two windows. If you use the driver.get() method, it will open the site in the current window. To open a second window you'll need to use something in Selenium to force a second window. The trick is to use JavascriptExecutor to run javascript which opens a second window, e.g.

WebDriver driver = new ChromeDriver();
driver.get(adminToolURL);
Set<String> windows = driver.getWindowHandles();
String adminToolHandle = driver.getWindowHandle();
((JavascriptExecutor)driver).executeScript("window.open();");
Set<String> customerWindow = driver.getWindowHandles();
customerWindow.removeAll(windows);
String customerSiteHandle = ((String)customerWindow.toArray()[0]);
driver.switchTo().window(customerSiteHandle);
driver.get(customerSiteURL);
driver.switchTo().window(adminToolHandle);

The first two lines are straight forward and open a window with the admin tool. The next few lines does the following: get a list of the currently open windows, save the window handle for the admin tool window, open a new window using executeScript, get a second list of the currently open window (this will be the same as the first list PLUS the new window). Next remove all the original windows handles from the second list. This should leave the second list (customerWindow) with only one window handle, e.g. the new window. The last three lines show you how to switch between the customer site window and the admin tool window.

WebDriverWait

In an older article I wrote about a method I created called waitForElement. After looking through some of the WebDriver code I found WebDriverWait. You should read the original article before you read this one.

The WebDriverWait class extends FluentWait<WebDriver> which is a special version of FluentWait which has WebDriver instances.

In the article for waitForElement I was talking about how Selenium cannot detect that Javascript has finished altering the DOM. So if you try to interact with an element with Selenium and Javascript is still altering it, you will have undefined results. The waitForElement method would wait for the element to be full rendered, i.e. for the Javascript to be finished. The WebDriverWait serves the same purpose.

To define a WebDriverWait would be the following:

WebDriverWait wdw = new WebDriverWait(driver, timeoutInSeconds, pollingTimeInMs);

The first parameter is an already existing WebDriver element. The second parameter is how long we want to wait for the element to be present. If we set it to say 30 it will wait for 30 seconds before it throws an error. The third parameter is how often you want to check for the element. If you set it really low, e.g. 10, it will check every 10 milliseconds. However this could put a load on the test machine. If you know it typically takes 55 milliseconds then maybe waiting for 60 would be best. You would not want to wait for 1000 milliseconds as it would make your test too slow.

This just creates the WebDriverWait object. It doesn't actually do the waiting. Let's say you click the ACCEPT checkbox. This causes Javascript to make the Next button visible. If you just click the ACCEPT checkbox then click the Next button it might fail because the Next button isn't visible as quickly as you can click it. So you need to click the ACCEPT checkbox, wait for the Next button to be visible then click the Next button. Here is some example code:

long timeoutInSeconds = 30;
long pollingTimeInMs = 250;
WebDriverWait wdw = new WebDriverWait(driver, timeoutInSeconds, pollingTimeInMs);
wdw.until(visibilityOfElementLocated(By.id("next"))).click();

This will check every 250 milliseconds to see if the element located by id='next' is visible. If the element does not become visible in 30 seconds it will fail the step and throw an error.

The list of conditions you can wait for are:

presenceOfElementLocated(by);
visibilityOf(driver.findElement(by));
alertIsPresent();
elementSelectionStateToBe(by, true);
elementSelectionStateToBe(we, true);
elementToBeClickable(by);
elementToBeSelected(by);
frameToBeAvailableAndSwitchToIt(frameLocator);
invisibilityOfElementLocated(by);
invisibilityOfElementWithText(by,text);
presenceOfAllElementsLocatedBy(by);
textToBePresentInElement(by, text);
textToBePresentInElementValue(by, text);
titleContains(title);
titleIs(title);
visibilityOf(we);
visibilityOfElementLocated(by);

The WebDriverWait also lets you alter the polling time, timeout, message displayed when it times out and exceptions it should ignore while waiting:

wdw.pollingEvery(delayBetweenPolling, TimeUnit.MILLISECONDS)
.ignoring(NoSuchElementException.class)
.withTimeout(timeoutInSeconds, TimeUnit.SECONDS)
.withMessage(message)
.until(visibilityOfElementLocated(By.id("next"))).click();

Because you can set the unit of measure for polling time and timeout, you can refine these after you instantiate the WebDriverWait object.

.

Windows Server and WebDriver

From time to time I see people trying to use Windows Server (2003 or 2008 R2) to run WebDriver scripts on Internet Explorer. They are posting to the WebDriver Google Group because it is not working and they are looking for answers on how to make it work.

The problem is they are asking the wrong question. They should not be asking, "How do I use WebDriver tests on Windows Server?" They should be asking, "Why don't my tests, which run on Windows XP or Windows 7, run on Windows Server 2008 R2?"

The answer is that Windows Server is not meant for people to use for surfing the web. They provide Internet Explorer if you need to access a local web application. You can explicitly add in trusted sites. But you cannot use it the way you would Internet Explorer on a Windows Workstation (like Windows XP, Windows 7 or Windows 8).

For more on this read the posting in Microsoft Developer Network: Enhanced Security Configuration for Windows Internet Explorer. The most important thing to note from this posting is the first two sentences:

As a best security practice, a server administrator should not browse Internet Web sites from the server. The administrator should only browse the Internet from a limited user account on a client work station to reduce the possibility of an attack on the server by a malicious Web site.

If you are really bent on using Windows Server there are ways to alter things but you will have to read the documentation on each version of Internet Explorer to understand how to do it.

The instructions for making Internet Explorer 7 on Windows Server 2003 work with Selenium will be different from getting Internet Explorer 8 on Windows Server 2008 R2 working with Selenium. Even getting Internet Explorer 9 on Windows Server 2008 R2 will be different as well. Microsoft is always changing what things are locked down and how you unlock them.

You should also realize that no respectable system administrator is going to use Windows Server to access a web application. They will use Virtual PC or a real Windows Workstation to access web applications. Therefore testing the application works okay with Internet Explorer on a Windows Server is not a configuration which should be supported. It does not test that the application runs on a real supported configuration, i.e. Internet Explorer on a Windows Workstation.

Dealing with HTTP Basic Authentication

If you go to a website and it has HTTP Basic Authentication turned on it will pop up a dialog asking you for username and password.

You can pass this information in the URL so the dialog does not pop up. This will allow you to go to these websites with automation tools like Selenium.

For example, if you go to httpwatch and click on the Display Image button it will pop open a dialog asking you for a username and password. For this site you can enter httpwatch for the username and any string for the password. So if I go to:

http://httpwatch:password@www.httpwatch.com/httpgallery/authentication/

then click the Display Image button, it will not pop open a dialog. This is because I have already been authenticated on the website.

HOWEVER, this will not work with current web browsers.

The reason for this is because web browser manufacturers recognized the fact that people can use this URL format to create fake websites. For example, I could have a website at 10.23.56.234, turn on basic authentication and make the username www.microsoft.com. I then send out an email with the URL:

http://www.microsoft.com:80@10.23.56.234/survey.html

This URL will go to my fake website with the username:password of www.microsoft.com:80. Some people might be tricked into thinking they are going to www.microsoft.com.

Fortunately, you can turn this feature back on.

For Internet Explorer you can make it accept the above URLs. It does open you up to someone spoofing you. So do this to a test machine, used only for automation, is okay. Using this on the computer you use to surf the web is a back idea.

If you go to http://support.microsoft.com/kb/834489/EN-US it will explain all the above. At the bottom it will talk about how to enable using username:password in the URL. In a nutshell, do the following:

• Open regedit.exe
• Find the key HKEY_LOCAL_MACHINE\Software\Microsoft\Internet Explorer\Main\FeatureControl\FEATURE_HTTP_USERNAME_PASSWORD_DISABLE
• Create the DWORD iexplore.exe=0

By setting iexplore.exe to 0 you are disabling the disabling of username:password, i.e. double negative.

If you are running 32-bit Windows this works as stated. If you are running 64-bit Internet Explorer on 64-bit Windows this works as stated. However, if you are running 32-bit Internet Explorer on 64-bit Windows this does not work. By default, 64-bit Windows 7 will run 32-bit Internet Explorer.

When you look in the registry, all the settings for 32-bit programs on a 64-bit computer will be in the WOW6432Node. So rather than HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft you want to go to HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft.

For Chrome & Firefox, they should support basic authentication through the URL.

For Safari, go into the Preferences, go to the Security tab and disable Warn when visiting a fraudulent website.

In summary, if I have the Selenium Java code:

    driver.get("http://www.httpwatch.com/httpgallery/authentication/");

    driver.findElement(By.cssSelector("#displayImage")).click();

    WebElement img = driver.findElement(By.cssSelector("#downloadImg"));

It will never make it to the third line because the .click() will bring up an HTTP basic authentication dialog which will block Selenium. Instead, I would use:

    driver.get("http://httpwatch:password@www.httpwatch.com/httpgallery/authentication/");

    driver.findElement(By.cssSelector("#displayImage")).click();

    WebElement img = driver.findElement(By.cssSelector("#downloadImg"));

.

automatic log in, automatic run

When writing test scripts for testing an installer, on Windows, one of the challenges is when the installer requires a reboot. To handle this I use automatic log in and the run once features of Windows.

To create automatic log in:

1. Run regedit.exe
2. Go to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon
3. Set DefaultUserName to the name of the person you want to automatically log in. If the key does not exist add it as REG_SZ.
4. Set DefaultPassword to the user's password. If the key does not exist add it as REG_SZ. Note: anyone who can read the registry can see the user's name and password. Create a local user with no permission on your network.
5. Set AutoAdminLogon to 1. If the key does not exist add it as REG_SZ.
6. Set ForceAutoLogon to 1. If the key does not exist add it as REG_SZ.

And that is all you need to have the computer automatically log in after a reboot.

The second part is to create a script which will start up the automation at the right place. What usually happens is the installer will put something in place so that after the reboot it will continue with the install the moment the user logs in. If this is the case for your installer, write a script which continues the automation and add it to the RunOnce key. Assuming you have a batch file which starts part 2 of your automation:

1. Run regedit.exe
2. Go to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\RunOnce
3. Add a string value
4. The Name of the should be something like zzz_Automation
5. The Data value will be the full path to the batch file. It is important to note that the current directory will not be the location of the batch file. Do not assume it will be in your script. Either use full paths for everything in the script or change to the directory you think you should be in then continue with the script.

Enjoy.

Creating a screen capture on every action

Someone recently commented on an article I wrote about generating a screen capture when an exception is thrown (see Generating a screen capture on exception thrown with Selenium 2).

Performing an action when an exception is thrown is built into the Selenium framework. You just need to create the action to generate a screen capture and hook it into the framework.

The Selenium framework comes with a WebDriverEventListener interface. In my article above I created an implementation of the WebDriverEventListener interface. To do this properly, you need to implement the following methods:

beforeNavigateTo(String url, WebDriver driver);
afterNavigateTo(String url, WebDriver driver);
beforeNavigateBack(WebDriver driver);
afterNavigateBack(WebDriver driver);
beforeNavigateForward(WebDriver driver);
afterNavigateForward(WebDriver driver);
beforeFindBy(By by, WebElement element, WebDriver driver);
afterFindBy(By by, WebElement element, WebDriver driver);
beforeClickOn(WebElement element, WebDriver driver);
afterClickOn(WebElement element, WebDriver driver);
beforeChangeValueOf(WebElement element, WebDriver driver);
afterChangeValueOf(WebElement element, WebDriver driver);
beforeScript(String script, WebDriver driver);
afterScript(String script, WebDriver driver);
onException(Throwable throwable, WebDriver driver);

In my screen capture when an exception is thrown, I put the necessary code to generate a screen capture in the onException method. You can see from the list above that you can have Selenium perform an action when other events occur. For example, if you wanted to do a screen capture after each click() action, you could write:

public void afterClickOn(WebElement element, WebDriver driver) {
    String filename = generateRandomFilenameFromWebElementAndDriver(element, driver);
    createScreenCaptureJPEG(filename);
}

I'll leave it to the reader to figure out how to create the generateRandomFilenameFromWebElementAndDriver. You might do something like use getCurrentUrl(), convert things like colon, slash, etc. into valid filename characters then append the web element getTagName() to the end of the filename.

Or you could do something tricky like have the beforeClickOn generate the filename and the afterClickOn use that filename to generate the screen capture.

If the event you want to trigger a screen capture is not listed above, there is no easy answer for how you would do it.

Initially, you might considering changing the WebDriverEventListener interface. But this would require a change to how Selenium works. Every time there is a new release of Selenium, you would have to merge your changes back in.

I would recommend submitting the changes as a new feature to the Selenium project to see if they'll incorporate it or wrap the Selenium methods with your own methods. For example, you would call your sendKeys() which could do a driver.sendKeys then do a screen capture.

Frames and WebDriver

When dealing with iframes and WebDriver things can quickly get confusing. Especially if you add popup windows to the mix.

When you have an iframe, it is a separate DOM. You can look at it as a separate web page inside the current web page. Lets take an example diagram:

If we look at the source code for this it might be something like:

<html>
    <body>
    <iframe src="frame1.html" style="border: red;">
        <html>
        ...
        </html>
    </iframe>
    <iframe id="2" src="frame2.html" style="border: green;">
        <iframe id="2-1" src="frame2-1.html">...</iframe>
        <iframe id="2-2" src="frame2-2.html">...</iframe>

        <iframe id="2-3" src="frame2-3.html">...</iframe>

        <iframe id="2-4" src="frame2-4.html">....</iframe>

        <iframe id="2-5" src="frame2-5.html">....</iframe>

        <iframe id="2-6" src="frame2-6.html">...</iframe>

        <iframe id="2-7" src="frame2-7.html">...</iframe>

        <iframe id="2-8" src="frame2-8.html">...</iframe>

        <iframe id="2-9" src="frame2-9.html">...</iframe>

    ...
    </iframe>
    <iframe id="3" src="frame3.html" style="border: brown;">

        <iframe id="3-1" src="frame3-1.html">...</iframe>

        <iframe id="3-2" src="frame3-2.html">...</iframe>

    ...
    </iframe>
    <iframe id="4" src="frame4.html" style="border: blue;">

        <iframe id="4-1" src="frame4-1.html">...</iframe>

        <iframe id="4-2" src="frame4-2.html">...</iframe>

        <iframe id="4-3" src="frame4-3.html">...</iframe>

        <iframe id="4-4" src="frame4-4.html">...</iframe>

        <iframe id="4-5" src="frame4-5.html">...</iframe>

        <iframe id="4-6" src="frame4-6.html">...</iframe>

        <iframe id="4-7" src="frame4-7.html">...</iframe>

        <iframe id="4-8" src="frame4-8.html">...</iframe>

        <iframe id="4-9" src="frame4-9.html">...</iframe>

    ...
    </iframe>
    </body>
</html>

All the rectangles in the diagram are iframes. The iframe with the red border would be the first iframe in the HTML code. Inside each iframe will be a full HTML page. It will have the <html></html> and everything which goes inside an HTML page.

So in WebDriver you have a switchTo method. The switchTo method returns a WebDriver.TargetLocator interface. If we look at the WebDriver.TargetLocator interface we see the following methods:

• frame(int index)
• frame(String nameOrId)
• frame(WebElement frameElement)
• defaultContent()

We can use the index to find the frame. If you have one main page and it contains a set of iframes, this is fine. However, if you have frames within frames it can get a little difficult to follow. Even if you can figure it out today, you will have to go through the whole exercise again if they change the layout by moving, adding or deleting a frame.

The best way to find a frame is with the id attribute or find the frame element using findElement() then use the third version listed above.

Now here is the most important thing to remember: you cannot jump in two or more frames. So if you are at the main content page and want an element in frame3-1.html you have to switch to frame3.html then to frame3.1.html. Assuming we are at the main page, the code for this might look like:


    driver.switchTo().frame("3");
    driver.switchTo().frame("3-1");


Additionally, if I have switched to frame4-7.html and I want to go to frame2-1.html, I have to go back to the top, then to frame2.html, then to frame2-1.html.

So how do you get to the top? If you are dealing with iframes then the defaultContent() method will take you to the main page, above all the iframes. If you are dealing with frames, defaultContent() will take you to the first frame.

So you can either leave focus were every you last left it then every action in a frame assumes you are at some unknown focus, call driver.switchTo().defaultContent() to get to the top, then go down to the frame you want OR you can start at the top, go to the frame you want then back to the top when you are done. The first way you go to the top as needed. The second way you ensure you are always at the top. Both ways work. It is just a matter of convention.

Sometimes drawing the layout of the page is a little harder then the diagram above. Additionally, if the layout changes, it can be difficult to alter the picture, depending on how you drew it. What I like to do is draw the relationships like a tree. For example, the diagram above might be draw as:

From this tree the rules are that you can go down a branch (switchTo().frame() from the parent) or you can get to the root of the tree (defaultContent() for iframes). You cannot jump across nodes or up levels.

Help for selenium-server-standalone.jar

 
One thing you might not realize if you have been trying to get command line help from the Selenium Server jar file is that there are two different help outputs.

If you run the server with no inputs it just runs with no help at all.

As an old UNIX/Linux guy I have gotten used to the standard of a single dash and a single letter (e.g. -h for help) or two dashes and a word (e.g. --help) I was a little thrown by the Selenium Server jar file.

The reason is that -h will give you help for the Standalone Selenium Server but --help will launch the Standalone Selenium Server. The Grid Selenium Server is in there but getting help for it is not obvious.

The help switch for Grid Selenium Server is actually -help (not the same as -h but not quite the Linux convention). Actually, this will give you the help message for the Standalone Server and the Grid Server.

For running the Grid, you need to specify hub or node. If you want to run it as a hub you would use:

java -jar selenium-server-standalone.jar -role hub

Looking at the rest of the help you will see some switches have (hub), some have (node) and some have (hub & node). If the switch has a (hub) then that switch applies only when using -role hub. Conversely, if it has (node) then it only applies to -role node.

Generally, you will set up one hub and multiple nodes. For example, the application I am currently testing needs to be tested on:

• Windows 7 with IE9
• Windows XP with IE6
• Mac OS X 10.8 with Safari 10

So I would run Grid Selenium Server with (omit the text after the # symbol):

• -role hub # (defaults to localhost and 4444)
• -role node -hubHost hubmachine -hubPort 4444 -browser browserName=iexplore,version=9,platform=VISTA # (Windows 7, IE9)
• -role node -hubHost  hubmachine -hubPort 4444 -browser browserName=iexplore,version=6,platform=XP # (Windows XP, IE6)
• -role node -hubHost  hubmachine -hubPort 4444 -browser browserName=safariproxy,version=10,platform=MAC # (Mac OS X, Safari 10)

The -hubHost and -hubPort just need to match the host and port for the hub. If you specify a -port for the hub then you need to change the settings for the nodes as well. In this example, I have the hub running on the computer with name hubmachine.

The -browser option is a little tricker. It mirrors the Selenium code. 

If you look at the Selenium Client source code for src/org/openqa/selenium/remote/BrowserType.java you will find a list of the strings which can be used with the browserName key. At this time for following values are permitted:

• firefox
• firefoxproxy
• safari
• opera
• iexplore
• iexploreproxy
• safariproxy
• chrome
• mock
• iehta

The platform key comes from src/org/openqa/selenium/Platform.java and valid values at this time are:

• WINDOWS
• XP
• VISTA
• MAC
• UNIX
• LINUX
• ANDROID
• ANY

When you look at the Platform.java file you will see that Windows 7 'feels like' Windows Vista. So we use the VISTA platform when we want Windows 7.

At this point, if you go to http://localhost:4444/ you'll have a link to the Grid2 Wiki and a link to the console. If you go to the console you should see an entry for each of the nodes you are running. All the nodes will be running on port 5555 but different machines.

You can also go to the Grid2 Wiki from the hub home page. You might want to take the time to have a look at that as well. There are additional parameters for the node. For example, I can add maxInstances=3 to the browser string and it will let me run up to 3 instances of that browser on the computer. By default the maxInstances will be 5.

 

My environment for Selenium 2.0 (WebDriver)

One thing I believe to be very important for any sort of software development (and test automation with Selenium is software development) is having a good environment. So here is how I set up my environment to start with Selenium 2.0 and Java.

What you need:

1. Java (go to  http://www.oracle.com/technetwork/java/index.html)
1. I typically go with Java SE 1.6.0. If you don't need the latest, go with something more mature.
2. Download the full SDK and docs.
3. Install them in C:\jdk1.6.0_30 (assuming you downloaded build 30)
4. I like to put it in the root of the C drive with no spaces in case other things don't like spaces.
5. Unpack the docs into the same directory so they are easy to find. 
2. Eclipse (go to  http://www.eclipse.org/downloads/)
1. I typically go with Eclipse IDE for Java Developers because it has everything for Selenium and it is small.
3. SVN (go to  http://subversion.apache.org/)
1. You need some sort of source control.
2. I usually go with Git but more people are still using Subversion.
3. There are more tutorials on SVN and better support.
4. Selenium (go to  http://code.google.com/p/selenium/downloads/list)
1. You'll need to download:
1. selenium-java-2.19.0.zip (or whatever the latest version is)
2. selenium-server-standalone-2.19.0.jar (or whatever the latest version is)
2. Unpack the zip file to some location
1. I'll typically unzip them into the workspace for the project
2. I'll also put the jar file in the same location
3. This way I can add the jar files to subversion
5. Eclipse support for SVN (go to  http://subclipse.tigris.org/ )
1. Normally I would suggest using Marketplace from the Help menu but not for the latest SVN client.
2. I like subclipse for an SVN client.
3. Once on the subclipse site go to download and install.
4. Copy the link for Eclipse Update Site URL for 1.8.x
5. Go to Eclipse Help->Install New Software...
6. Click Add...
1. Name: subclipse 1.8.x
2. Location: the update site URL you copied above.
7. Select all
8. Unselect Mylyn 3.x if you don't want it.
9. Follow the install wizard
6. Adding the JDK to Eclipse
1. Go to Window->Preferences in Eclipse
2. Expand Java
3. Select Installed JREs
4. Click Add...
5. Select Standard VM
6. Set JRE home to C:\jdk1.6.0_30\jre
7. Set JRE name to JDK 1.6.0
8. Everything else can be left as default.
7. Creating a project
1. Select File->New->Java Project
1. Project name: something relating to the application you are testing
2. I usually go with the defaults.
3. Maybe change the JRE.
4. Click Next >
5. On the Libraries tab
6. Add JARs... (add External JARs... if you didn't put them in the project workspace)
7. Go to the location you unzip the Selenium files.
8. Select the selenium-java-2.19.0.jar and selenium-java-2.19.0-src.jar files and add them.
9. Add JARs... (add External JARs... if you didn't put them in the project workspace)
10. Go to the location you unzip the Selenium files.
11. Go to the libs folder and add all the jar files in there.
12. Add JARs... (add External JARs... if you didn't put them in the project workspace)
13. Go to the location you downloaded the selenium-server-standalone-2.19.0.jar file
14. Add this file to the libraries.
2. Finish the rest of the project creation using defaults.
3. You can now start adding JUnit Test Cases to the project
1. I select JUnit 4
2. I select a package which follows Java conventions
3. For example, com.mycompany.selenium.application where application is the application we are testing.
4. Add a setUp() and tearDown()
5. Finish
4. Have a look at  http://seleniumhq.org/docs/03_webdriver.html for what to put in the setUp(), tearDown() and test cases.

The most important thing about using Selenium with Java is that you really need to know Java, Source Control (Subversion), an IDE (Eclipse) and a test framework (JUnit) before you even start using Selenium. So search the web for basic Java tutorials, play with Eclipse, check out http://www.junit.org/ and search for tutorials on JUnit 4 then look at the docs on Selenium HQ.

If you need help with Selenium and WebDriver, post a message to http://groups.google.com/group/webdriver. I usually respond to people there on a regular basis.

Update to using WebDriver / Selenum 2.0 with Safari

Around this time last year I wrote an article about using Safari with WebDriver. You can find this article at http://darrellgrainger.blogspot.com/2011/02/using-selenium-20-with-webdriver-and.html.

It has been a year and there is still no SafariDriver which extends WebDriver. Unfortunately, Safari 5.0 and 5.1 have hit the market and the Selenium 1.0 "*safari" driver is no longer working. As the Selenium look at dropping support for older web browsers they are also looking into supporting things like Safari 5.x.

Ideally, I would like to see:

    WebDriver driver = new SafariDriver();


or

    URL hub = new URL("http://localhost:4444/wd/hub");
    DesiredCapabilities dc = DesiredCapabilities.safari();
    WebDriver driver = new RemoteWebDriver(hub, dc);


Until this happens, we'll have to rely on using Selenium 1.0 to create an instance of Safari. But if "*safari" is no longer working and there are no plans to update it, what do we do? The solution appears to be use "*safariproxy" instead.

Looking at the code for SeleneseCommandExecutor() there are two constructors. The first takes as input a Selenium instance. The second takes as input a CommandProcessor. The second constructor actually calls the first constructor. So either method will work. So here it is:

    String baseURL = "http://www.google.com";
    Selenium sel = new DefaultSelenium("localhost", 4444, "*safariproxy", baseURL);
    CommandExecutor executor = new SeleneseCommandExecutor(sel);
    DesiredCapabilities dc = new DesiredCapabilities();
    WebDriver browser = new RemoteWebDriver(executor, dc);


or

    String baseURL = "http://www.google.com";
    CommandProcessor cp = new HttpCommandProcessor("localhost", 4444, "*safariproxy", baseURL);
    CommandExecutor executor = new SeleneseCommandExecutor(sel);
    DesiredCapabilities dc = new DesiredCapabilities();
    WebDriver browser = new RemoteWebDriver(executor, dc);


Additionally, if the browser cannot be found, i.e. it is not in the default locations, then you can specific the path to the Safari executable using:


    "*safariproxy /Application/Safari.app/Contents/MacOS/Safari"


This code is untested but hopefully better then the previous example.

Happy automating.

.

Automated GUI Testing

From time to time I will see an automation tool with image compare. The idea is to get a screen capture, save it as a baseline then when the automation runs it gets a screen capture and compares it to the baseline.

Originally this always failed because there would be small differences. For example, the screen displayed the time and date, this would always be different and the compare would always fail.

The solution was to mask out parts of the screen capture we know will not be the same or to capture a portion of the screen and see if any part of the baseline matches the portion of screen capture.

This was a better solution but the video drivers from one machine to the next would create slightly different screen captures. So if you developed the baseline on one computer and ran it on another there was a high probability the compare would fail. Even comparing the screen capture to a baseline created on the same machine has a high probability of failing.

So what is the alternative?

We could go back to manual testing but that would be a step in the wrong direction.

We could automate the tests but hire someone to sit and watch them. If they see something wrong, do a screen capture and file a bug. Problems with this are, you still have one person tied up for the day watching the automation. Additionally, getting a screen shot and filing a bug while the automation is running could be a problem. Ideally, you'd want to be able to pause the automation, get the screen capture, file the bug and resume the automation.

However, there is a third alternative. When I create the test cases they will usually have points in the testing when you want to confirm the display looks okay. In some cases we just assume the user will be constantly looking. With automation, you want to explicitly decide when we need to confirm the display is okay. At those points in the automation, do a screen capture. The name of the file and the directory you store it in will have enough information for you to know which test case created the screen capture.

Once you have a directory of screen captures, you can quickly scan them and see that everything looks okay.

How does this save time? Most the time, GUI Testing is 80% getting the to place you want to confirm good display and 20% actually confirming the display. So if you are just reviewing the screen captures the time to get to the point to create the screen capture does not involve a tester and you save, on average, 80% of the time.

Classic examples of this being even better are confirming different configurations on the same test cases. For example, I tested a Java library for creating graphics. I would run the tests on 5 different operating systems with 3 different versions of Java.. So the test runner would essentially run:

for os in (5 different operating systems):
    for jvm in (3 different versions of Java):
        run all tests

This would result in 15 different test runs. I would then display all 15 screen captures for test case #1. If they all looked the same it would be a pass. If any one of the screen captures looked wrong I'd know which operating system, which JVM and which test case. So I could file a bug against that configuration.

You could do a similar thing for web testing. Run the same tests against the different operating systems and different web browsers. Then compare the screen captures to ensure all the configurations displayed correctly.

If you wanted to you could video capture the display as the tests ran. With a good video play back you could fast forward through the parts you know won't change.

Selecting WebDriver locators

When you are using WebDriver for test automation a test case really boils down to:

1. Find an element
2. Perform an action
3. Confirm the expected result

Finding an element and confirming the expected result requires locators. In WebDriver a locator is a way of uniquely identifying an element on the web page, i.e. in the Document Object Model (DOM). The By class is used in WebDriver to locator elements. You have:

• By.className(className);
• By.id(id);
• By.linkText(linkText);
• By.name(name);
• By.partialLinkText(linkText);
• By.tagName(name);
• By.cssSelector(selector);
• By.xpath(xpathExpression);

The most powerful locators are CSS and XPath. All the other selectors can actually be done using CSS or XPath. For example:

Original	CSS	XPath
By.className("foo");	By.cssSelector(".foo");	By.xpath("//*[@class='foo']");
By.id("bar");	By.cssSelector("#bar");	By.xpath("//*[@id='bar']");
By.linkText("Click Me");	N/A	By.xpath("//a[text()='Click Me']");
By.name("fee");	By.cssSelector("[name='fee']");	By.xpath("//*[@name='fee']");
By.partialLinkText("some");	N/A	By.xpath("//a[contains(text(),'some')]");
By.tagname("div");	By.cssSelector("div");	By.xpath("//div");

In addition to the simple locators, CSS and XPath can selector more complex elements. Rather than saying I want all the DIV tags, I can say I want all the DIV tags whose parent is a SPAN. In CSS this would be "span>div" and in XPath this would be "//span/div".

The combinations are endless. For each tag in the XPath or CSS I can add multiple identifiers. So I could have locators for things as complex as "all DIV tags, with name containing 'foo' and class equal 'bar' whose parent is a TD but only in the TABLE with id='summary' and the class equal 'humho'"

The first thing to understand is that CSS will be noticeably faster than XPath when testing against Internet Explorer. Your tests could run as much as 10 times slower (something which runs on a day on Firefox could take a week on Internet Explorer) when using XPath.

So the first thing to remember is CSS is better than XPath. However, some things are easier to express as XPath. So occasionally you might need to use XPath.

If you have a selector like "html>body>table>tbody>tr[2]>td[3]>a" it might work but if the developer finds it does not format nicely on Chrome, they need to throw in a DIV. So the selector changes to "html>body>div>table>tbody>tr[2]>td[3]>a". Later a new version of Internet Explorer comes out and the developer finds they need to add a SPAN to make it look proper on the new Internet Explorer and still look okay on older versions. So the locator becomes "html>body>div>table>tbody>tr[2]>td[3]>span>a".

If we spend all our time maintaining the locators, it could end up that the cost of maintaining the automation is greater than running the tests manually. In which case the automation is deemed a failure.

So you have to start looking for patterns. Is there something I could use on the first version of the application which also works on the second and third version? Can I predict a locator which will work on the fourth and subsequent versions?

Often the underlying technology changes but it continues to look the same to the user. So is there something visual I can use which will not change? In this example, the text for the anchor probably never changed. So I'd use By.linkText("whatever"); locator or By.xpath("//a[text()='whatever']);.

What if I find myself changing locators because sometimes the text is "  whatever", sometimes it is "whatever" and other times it is "whatever  "? Then I'm going to use By.partialLinkText("whatever"); or By.xpath("//a[contains(text(), 'whatever')]");.

The danger is that there might be two links which contain the substring "whatever". I need to make sure I am selecting the correct link. So the locator might need to be more complex. It might need to be partial text and parent information. For example, if the text appears in two different tables and I want the text from table 2. Table 2 has the id='foo2' then the locator might be:

• "table.foo2 a"
• "//table[@id='foo2']/tbody/tr/td/a[contains(text(),'whatever')]"

The first locator assume there is only 1 anchor in the second table. This might not be true in all cases. The second locator finds the second table but it searches all rows (TR) and all columns (TD) for an anchor (A) whose text contains the substring "whatever". This can be extremely slow, especially for large tables.

Finding the balance between locators which are too long and too short can be an art. The trick is to pick something. If it requires maintenance, pick a new locator which works on the previous versions and the new version. As you continue to maintain the locators you will see a pattern. You will start to see that chunks of HTML code never change Outside these chunks change (so keep the locator short enough to stay inside the chunk that does not change). Within the chunk there might be multiple matches if you make the locator too short. So figure out, within that chunk, what makes the element you want different from all the other matches.

So how do I look at the DOM? I need to see what the DOM looks like to be able to see all the possible locators which would work.

If you are using Internet Explorer 8 or higher you can press F12 to open the developer tools. If you are using Firefox you need to install Firebug then F12 will open Firebug. If you are using Chrome then CTRL-SHIFT-I will open the developer tools.

Beyond that, the only tool I use is my brain and the W3 standards.

Reading the W3 standards (or any standards documentation, ISO, ANSI, IEEE, etc.) can be difficult at first. Especially if you have been learning from books like "Web Design in 21 Days" or "Software Testing for Dummies." However, the more you read and understand standards documentation, the easier it gets to read other standards documents. If generating XPath was easy enough for a piece of software then why would they pay you to do the work? There are probably a dozen XPath locators for any given element on a page. Some will work once and need to be revised on the next release of the application. Some will work within the design pattern of the application and might never need updating. There is no way for a piece of software to spot the design pattern and know which locator will work best. This is what they pay you to do.

Excessively long XPath is brittle and will need a great deal of revising from release to release. Extremely short XPath will sometimes find the wrong element between releases. This leads to a test which fails unpredictably and can be difficult to debug. Not something you want in an automation suite. Finding the right balance is your job. The first time you select a locator it might need revising for the next release. You need to look at why you selected the first locator when selecting the revised locator. The second locator should work for the first release and the second release. When the locator fails, you need to select a new locator which would have worked on the first release and all subsequent releases, including the next release. After a while you should start to see the pattern. The pattern is usually derived from some design pattern the application is being developed with. Learn about Design Patterns, it will be extremely helpful in generating good test automation. If the developers change the tools, libraries, design patterns, etc. you should expect the locators to fail. At this point, selecting a locator which works with the next release but does not work with the previous release makes sense. Major change in development usually implies major change in test automation. It would be difficult for a tool to realize when it needs to abandon old locators.

Essentially, automation is all about finding elements (locators), performing actions on them, confirming the expected results (usually involves more locators). Two thirds of the work is about the locators. Learning XPath, CSS and DOM will make your job that much easier.

• XPath 1.0: http://www.w3.org/TR/xpath/
• XPath 2.0: http://www.w3.org/TR/xpath20/
• XPath functions: http://www.w3.org/TR/xpath-functions/
• CSS 1.0: http://www.w3.org/TR/2008/REC-CSS1-20080411/
• CSS 2.0: http://www.w3.org/TR/CSS2/
• CSS 3.0: http://www.w3.org/TR/selectors/
• DOM: http://www.w3.org/TR/DOM-Level-2-HTML/html.html

When possible, use CSS selectors as they are faster. Some things are easier to locate using XPath and XQuery (XPath functions). It is better to have a test run slow and be easy to maintain. So if CSS selectors are complex and unintuitive you might want to use XPath functions instead.

This is essentially how I decide on locators.

.

Your automation must not dictate your test plan

One of the things I see people getting into automation doing is selecting what to automate or how to test an application based on what the automation tool will let them do. This is a dangerous approach to automation.

The test cases I create for an application are based on what is important to the customer. I want to make sure that the customer experience is a good experience. If I create a set of test cases or use stories which reflect real customer usage of the application then I am most likely to find issues which will affect the customer.

I remember working on a project for 4 years. After 4 years of testing and improving the application we were at a point that over 90% of the features important to the customer were tested and bug free. Of the remaining 10% we knew of most the defects and had a work-around. We were at a point where we were testing extreme edge cases. At this point I found a major defect. The developer looked at the fix and realized the defect had been there since the beginning. In 4 years not one single customer reported this defect. The defect was easy to automate but really added zero value to the application. This is NOT a test case you want to start with when automating.

On another project someone found a defect in a desktop application. The steps to reproduce were:

1. Run an application not involved in the test case at all (Outlook for example)
2. Hand edit a project file using notepad or something other than the application it was intended for
3. Make a very specific change to the file
4. Run the application under test
5. Open the corrupted file
6. Select a feature which relies on the corrupted file
7. A modal dialog appears telling you the file is corrupt, do you wish to repair it.
8. Ignore the dialog
9. Use CTRL-TAB to switch to a different application not involved in the test case at all
10. Click on the application under test in a very specific location on the MDI client window

At this point the modal dialog is hidden behind the window with focus and the window with focus appears to be frozen. It is really waiting for you to respond to the modal dialog. This was a design flaw in the operating system. It was virtually impossible to fix in the application under test without a major re-design. It was highly unlikely a customer would be able to reproduce this defect. When the tester showed me the 'locked' state it only took me a few minutes to figure out what was going on. Our customer was typically a software developer with 7+ years of experience.

This was a useless test case. In both this and the previous test case it was a bad test case regardless of creating it manually or automating it. My point is, the test case came first. Even before we attempted to automate it, we decided whether or not it was a good test case.

Test automation NEVER proceeds test case creation or test planning.

Once you know what you want to test and the steps to testing it, you automate those steps.

This is the second mistake I see people getting into automation making. They decide WHAT they want to test but when they start automating it, the steps they generate with the automation are not the same steps as they would do manually. In this case you have taken the time to create a good set of test cases and thrown them out the door when you start automating. This is not a good idea.

Rather than changing the test case to something which is easy to automate, you need to figure out how to automate the test steps. This is what separates good automation from bad automation.

Many times I have seen a test case automated. It gets run and passes. We ship the application to the customer. He uses the same feature and it fails horribly. Why? Because the steps he used to get to the defect where not the steps we automated. We had a good test case. If an experienced tester had executed the test case manually, they would have found the defect. The person automating the test case just found it easier to automate something close to but not equivalent to the test case.

I am currently using Selenium 2.x with WebDriver. One of the changes from Selenium 1.x to 2.x is that you cannot interact with invisible elements. For example, a common trick on a website is to have an Accept checkbox on a download page. If you accept the terms the Download button becomes visible. In Selenium 1.x I could click on the Download button without clicking the Accept checkbox. The REAL test case was:

1. Go to download page
2. Click Accept checkbox
3. Click Download button
4. Confirm download

What someone would automate with Selenium 1.x was:

1. Go to download page
2. Click Download button
3. Confirm download

The idea is that it saves a step. One less step means quicker to code, runs quicker, one less thing to maintain. You do this a thousand times and it adds up. HOWEVER, the customer would never click on the invisible Download button.

In Selenium 2.x you would get a failure with the shortened test case. People occasionally complain that Selenium 2.x has removed an important feature. They want to know how they can click on the invisible Download button. They come up with these tricky Javascript snippets which will allow Selenium 2.x to 'see' and click the Download button. Is a customer going to create a Javascript snippet, inject it into the page, run it just so they can click the Download button? Is a manually tester going to do this? If the answer is no, then why is our automation doing this? If the manual test case calls for clicking the Accept checkbox then our automation should as well. If clicking the Accept checkbox does not enable the Download button, file a bug and move on to something else.

Finally, automation is all about finding elements on a page, interacting with them (clicking, right clicking, typing, etc.) and checking what happened. As a manual tester you are going to use your eyes and hands to do everything. The test case might have a step like, "Locate the folder called 'My Documents' and double click it." This is really two steps. The automation should locate the folder called 'My Documents', this is step 1. It should double click the element, this is step 2. As a manual tester I find the element by looking for the text 'My Documents'. If this is a web page and the HTML is:

<div id='lsk499s'><a href="...">My Documents</a></div>

I am not going to use the div id to find the element. I'm going to use the text. As a manual tester I used the text to find the element. There is no reason to do anything differently with the automation tool.

What if the web page is made to look like Window Explorer. On the left is a tree view with one of the nodes being 'My Documents' and on the right is a thumbnail view with a 'My Documents' folder. In the manual test case, does it specify which 'My Documents' to double click? If yes, follow the test case. If no, how would you as a tester decide? Do you always go with the thumbnail view? Do you pick randomly? Do you change ever test run? If you are a good manual tester, we want that experience captured by the automation. If I would normally change every test run but I never test the same feature twice in one day, it might be sufficient to say, if the day of the year is even, double click thumbnail else double click tree view. If the automation gets run daily, it will pick a different way each day.

The important thing in all this is that I am a good tester. I write good test plans and test cases. When I decide to automation my good test cases, I should not compromise the quality of my testing just because I am testing it with an automation tool rather than manually.

Happy testing!

.

Using the right tool for the job

From time to time I see people asking questions about how to use an automation tool to do something the tool was never meant to do. For example, how do I use Selenium to get the web page for a site without loading the javascript or CSS?

Selenium is designed to simulate a user browsing a website. When I open a web page with a browser, the website sends me javascript and CSS files. The browser just naturally processes those. If I don't want that, I shouldn't use a browser. If I am not using a browser, why would I use Selenium to send the HTTP request?

That is all the get() method in Selenium does. It opens a connection to the website and sends an HTTP request using the web browser. The website sends back an HTTP response and the browser processes it.

If all I want to do is send the request and get the response back, unprocessed. I don't need a web browser.

So how can I send an HTTP request and get the HTTP response back? There are a number of tools to do this.

Fiddler2: http://www.fiddler2.com/fiddler2/

The way Fiddler works is you add a proxy to your web browser (actually Fiddler does it automatically). Now when you use the web browser, if Fiddler is running, the web browser sends the HTTP request to Fiddler and Fiddler records the request and passes it on to the intended website. The website sends the response back to Fiddler and Fiddler passes it back to the web browser.

You can save the request/response pair and play them back. Before you play the request back you can get it. You can edit the website address, you can edit the context root of the URL and if there is POST data you can get the data as well.

Charles: http://www.charlesproxy.com/

Charles is much like Fiddler2 but there are two main differences. The first is that Charles is not free. You can get an evaluation copy of Charles but ultimately, you need to pay for it. So why would you use Charles? With purchase comes support. If there are things not working (SSL decrypting for example) you can get help with that. Additionally, Fiddler is only available on Windows. Charles works on Mac OS X and Linux as well.

curl: http://curl.haxx.se/

Fiddler and Charles are GUI applications with menus and dialogs. They are intended for interacting with humans. If you are more of a script writer or want something you can add to an automated test, you want something you can run from the command line. That would be curl. Because it is lightweight and command line driven, I can run curl commands over and over again. I can even use it crude for load testing.

The most common place to find curl is checking the contents of a web page or that a website is up and running. There are many command line options (-d to pass POST data, -k to ignore certificate errors, etc.) but the general use is curl -o output.txt http://your.website.com/some/context/root. This will send the HTTP request for /some/context/root to the website your.website.com. A more real example would be:

curl -o output.txt http://www.google.ca/search?q=curl

I could then use another command line tool to parse the output.txt file. Or I could use piping to pipe the output to another program.

wget: http://www.gnu.org/software/wget/

Another nice command line tool is wget. The wget command, like curl, will let you send an HTTP request. The nice thing about wget is that you can use it to crawl an entire website. One of my favourite wget commands is:

wget -t 1 -nc -S --ignore-case -x -r -l 999 -k -p http://your.website.com

The -t sets the number of tries. I always figure if they don't send it to me on the first try they probably won't send it to me ever. The -nc is for 'no clobber'. If there are two files sent with the same name, it will write the first file using the full name and the second file with a .1 on the end. You might wonder, how could it have the same file twice in the same directory? The answer is UNIX versus Windows. On a UNIX system there might be index.html and INDEX.html. To UNIX these are different files but downloading it to Windows I need to treat these as the same file. The -S prints the server reponse header to stderr. It doesn't get saved to the files but lets me see that things are still going and something is being sent back. The --ignore-case option is because Windows ignores case so we should as well. The -x option forces the creation of directories. This will create a directory structure similar to the original website. This is important because two different directories on the server might have the same file name and we want to preserve that. The -r option is for recursive. Keep going down into subdirectories. The -l option is for the number of levels to recurse. If you don't specify it, the default is 5. The -k option is for converting links. If there are links in the pages being downloaded, they get converted. Relative links like src="../../index.html" will be fine. But if they hard coded something like src="http://your.website.com/foo.html" we want to convert this to a file:// rather than go back to the original website. Finally, the -p option says to get entire pages. If the HTML page we retrieve needs other things like CSS files, javascript, images, etc. then the -p option will retrieve them as well.

These are just some of the tools I use when Selenium is not the right tool for the job.

Interface versus implementation

I've seen a few people posting Java code for Selenium automation where they have things like:

FirefoxDriver driver = new FirefoxDriver();

If you look at the code for Selenium you will see they have an interface called WebDriver. FirefoxDriver, InternetExplorerDriver, HtmlUnitDriver, RemoteWebDriver all implement WebDriver.

If I'm writing a test suite, I ultimately want to run the suite on Firefox, Internet Explorer, Chrome, etc. but if I implement my framework using FirefoxDriver, I have to edit the code to make it work on InternetExplorerDriver. The proper solution is to write your code using:

WebDriver driver = new FirefoxDriver();

You might argue that I need to change my code still. Okay, so you would implement it using:

WebDriver driver;
    if(browserType.equals("firefox")) {
        driver = new FirefoxDriver();
    } else if(browserType.equals("ie")) {
        driver = new InternetExplorerDriver();
    } else {
        driver = new RemoteWebDriver();
    }

Now all subsequent code will use driver as if it was merely a WebDriver. However, if you look at RemoteWebDriver you will see that it also implements JavascriptExecutor. So if I wanted to use:

driver.executeScript(script, args);

I'll get an error because WebDriver does not support executeScript. My driver is REALLY a RemoteWebDriver and does support executeScript. So how do I access that functionality? Quite simple:

((RemoteWebDriver)driver).executeScript(script, args);

Even better would be:

if(driver instanceof RemoteWebDriver) {
        ((RemoteWebDriver)driver).executeScript(script, args);
    } else if(driver instanceof FirefoxDriver) {
        ((FirefoxDriver)driver).executeScript(script, args);
    } else if(driver instanceof InternetExplorerDriver) {
        ((InternetExplorerDriver)driver).executeScript(script, args);
    }

Now if you run the test suite against a browser that does not support a particular feature, it will skip that part of the code automatically.

And that is proper use if polymorphism.