Browser statistics

A lot of discussion goes into browser statistics. Whenever a company is creating a new web application they need to decide which browsers will be supported and their order of importance.

Years ago it was a given that Internet Explorer was the #1 browser. Just the shear volume of Windows users and lack of serious competition in that area made Internet Explorer the dominant browser.

Then along came Firefox and later Chrome. Additionally, Mac Desktop is becoming a large enough market that Safari is worth considering.

So how do we decide which browsers we should support and what order we should rank them in?

There are many sites which give statistics on browser usage:

W3 SchoolsStatCounterW3 CounterClickyNetMarketShareWikiMedia

and probably a whole lot more to check. If we look at the different sites we can see some radically different numbers. Some claim Chrome is close to 60% of the market. Others have Chrome around 30% of the market.

Why the big difference? Different sites will gather statistics on different people. The W3 Schools site is very popular among programmers and software testers. I tend to like Chrome because it has the built-in Inspect feature and other helpful development tools. Firefox is okay when you install Firebug or other tools but often we want to test with a clean browser and don't have these tools available.

Starting with Internet Explorer 8 it came with development tools. The tools in Internet Explorer 9 are better but nothing as good as the tools in Firefox or Chrome. So Chrome and then Firefox are going to be the browser of choice for programmers and software testers.

Not surprisingly, W3 Schools is the site which claims Chrome is almost 60% of the market. But it will be biased towards power users, programmers and testers. My mother doesn't use a computer. If she did it would be a Windows 7 with Internet Explorer 11 or a Mac with Safari 6.1.

So which statistics site you select depends on your target audience. You don't want to use the statistics from W3 Schools if your target audience is my mom.

What if you have a website and you are going to be updating it or adding another website with the same target audience? Then you have been hopefully gathering statistics on your existing website. Statistics from your existing website will tell you exactly who your users are. If you statistics say that 97% of your users are using Internet Explorer 6 then you need to support Internet Explorer 6. You could make it painless to upgrade your existing user base to a new version of the same browser but odds are it is safer to just support what your users are using.

The only time I would say you might not want to completely follow the statistics on an existing website would be if you are attempting to grow your market and know the new customer base is going to be a different type of user. Let's say that my user base is traditionally Internet Explorer users but I want to start targeting Mac users. I want to make sure that I don't sacrifice existing users for the new user base. So I will continue to value the Internet Explorer users but I want to start looking at statistics for when the operating system is Mac OS X. If I drill down to just Mac OS X users, which is the dominate browser?

In addition to which browser there is also the resolution for the display. This tends to be a little easier to decide if we are talking desktop. You pick the small display which is widely supported. Originally, I would design a website so it looked good on 640x480. Then the average display was 800x600. Later we would start seeing wide displays like 1440x900. I found it easiest to set a minimum size that would be supported. So you don't want so many things on a menu that the menu is longer than say 600 pixels high if you supported 800x600. We can use tools to resize the browser to the different sizes and make sure things still look good. This is easy to do on pretty much any desktop today.

Back in the days of EGA and VGA the number of colours you would support was an issue but today it isn't really something people worry about any more.

This is how I generally recommend clients decide on which desktop browsers to support. It does not really address the idea of mobile browsers. This is an area which is still in flux. I see a lot of different mobile browsers. My own personal experience has been updating my browser can make it more unstable. So if I have a browser which runs on my device I'll stick with it. However, if I get a new device I tend to find the newer browsers come with it and run better than old browsers. So you should be seeing a wider range of browsers on mobile devices than on desktops. Additionally, mobile devices include smart phones and tablets. There are full size tablets, mini tablets, large smart phones, medium smart phones and small smart phones. The different resolutions can vary much more than desktop devices.

Additionally, I cannot take a large device and resize the browser to small sizes. The idea of a window on a desktop doesn't exist on many mobile devices. So testing on the different resolutions is a major concern. Fortunately, with emulators you can run a software emulated version of a device and check that the website displays properly on it. Or you can use features of our existing browser to pretend to be a specific mobile device. This will change the size and User-Agent information to make your web application display as if it was getting rendered on a mobile device. Again, Chrome Inspect has some built-in features in this area which are quite helpful. They should not replace emulators or even better actual devices when doing system testing.

Ideally, in a test lab I will have all the machine with the largest display we want to support. I will then use a virtual computer player like Virtual PC, Parallels, VMware, etc. to emulate all the other operating systems. A Mac works really well because you can use it to run Mac browsers, a virtual PC like Parallels or Fusion to run Windows, Linux and Solaris x86, XCode to emulate iOS devices and Eclipse or IDEA to emulate Android devices.

If the client runs mostly Microsoft products and Mac isn't a concern then you can use Virtual PC on a Windows 7 device to run Windows XP and Windows 7 virtual machines for free.

For virtualizing all the Windows machines right now you can go to http://www.modern.ie/en-us/virtualization-tools#downloads to download virtual machines for Windows XP, Windows 7 and Windows 8. Everthing from Internet Explorer 6 to Internet Explorer 11.

WebDriverWait

In an older article I wrote about a method I created called waitForElement. After looking through some of the WebDriver code I found WebDriverWait. You should read the original article before you read this one.

The WebDriverWait class extends FluentWait<WebDriver> which is a special version of FluentWait which has WebDriver instances.

In the article for waitForElement I was talking about how Selenium cannot detect that Javascript has finished altering the DOM. So if you try to interact with an element with Selenium and Javascript is still altering it, you will have undefined results. The waitForElement method would wait for the element to be full rendered, i.e. for the Javascript to be finished. The WebDriverWait serves the same purpose.

To define a WebDriverWait would be the following:

WebDriverWait wdw = new WebDriverWait(driver, timeoutInSeconds, pollingTimeInMs);

The first parameter is an already existing WebDriver element. The second parameter is how long we want to wait for the element to be present. If we set it to say 30 it will wait for 30 seconds before it throws an error. The third parameter is how often you want to check for the element. If you set it really low, e.g. 10, it will check every 10 milliseconds. However this could put a load on the test machine. If you know it typically takes 55 milliseconds then maybe waiting for 60 would be best. You would not want to wait for 1000 milliseconds as it would make your test too slow.

This just creates the WebDriverWait object. It doesn't actually do the waiting. Let's say you click the ACCEPT checkbox. This causes Javascript to make the Next button visible. If you just click the ACCEPT checkbox then click the Next button it might fail because the Next button isn't visible as quickly as you can click it. So you need to click the ACCEPT checkbox, wait for the Next button to be visible then click the Next button. Here is some example code:

long timeoutInSeconds = 30;
long pollingTimeInMs = 250;
WebDriverWait wdw = new WebDriverWait(driver, timeoutInSeconds, pollingTimeInMs);
wdw.until(visibilityOfElementLocated(By.id("next"))).click();

This will check every 250 milliseconds to see if the element located by id='next' is visible. If the element does not become visible in 30 seconds it will fail the step and throw an error.

The list of conditions you can wait for are:

presenceOfElementLocated(by);
visibilityOf(driver.findElement(by));
alertIsPresent();
elementSelectionStateToBe(by, true);
elementSelectionStateToBe(we, true);
elementToBeClickable(by);
elementToBeSelected(by);
frameToBeAvailableAndSwitchToIt(frameLocator);
invisibilityOfElementLocated(by);
invisibilityOfElementWithText(by,text);
presenceOfAllElementsLocatedBy(by);
textToBePresentInElement(by, text);
textToBePresentInElementValue(by, text);
titleContains(title);
titleIs(title);
visibilityOf(we);
visibilityOfElementLocated(by);

The WebDriverWait also lets you alter the polling time, timeout, message displayed when it times out and exceptions it should ignore while waiting:

wdw.pollingEvery(delayBetweenPolling, TimeUnit.MILLISECONDS)
.ignoring(NoSuchElementException.class)
.withTimeout(timeoutInSeconds, TimeUnit.SECONDS)
.withMessage(message)
.until(visibilityOfElementLocated(By.id("next"))).click();

Because you can set the unit of measure for polling time and timeout, you can refine these after you instantiate the WebDriverWait object.

.

Windows Server and WebDriver

From time to time I see people trying to use Windows Server (2003 or 2008 R2) to run WebDriver scripts on Internet Explorer. They are posting to the WebDriver Google Group because it is not working and they are looking for answers on how to make it work.

The problem is they are asking the wrong question. They should not be asking, "How do I use WebDriver tests on Windows Server?" They should be asking, "Why don't my tests, which run on Windows XP or Windows 7, run on Windows Server 2008 R2?"

The answer is that Windows Server is not meant for people to use for surfing the web. They provide Internet Explorer if you need to access a local web application. You can explicitly add in trusted sites. But you cannot use it the way you would Internet Explorer on a Windows Workstation (like Windows XP, Windows 7 or Windows 8).

For more on this read the posting in Microsoft Developer Network: Enhanced Security Configuration for Windows Internet Explorer. The most important thing to note from this posting is the first two sentences:

As a best security practice, a server administrator should not browse Internet Web sites from the server. The administrator should only browse the Internet from a limited user account on a client work station to reduce the possibility of an attack on the server by a malicious Web site.

If you are really bent on using Windows Server there are ways to alter things but you will have to read the documentation on each version of Internet Explorer to understand how to do it.

The instructions for making Internet Explorer 7 on Windows Server 2003 work with Selenium will be different from getting Internet Explorer 8 on Windows Server 2008 R2 working with Selenium. Even getting Internet Explorer 9 on Windows Server 2008 R2 will be different as well. Microsoft is always changing what things are locked down and how you unlock them.

You should also realize that no respectable system administrator is going to use Windows Server to access a web application. They will use Virtual PC or a real Windows Workstation to access web applications. Therefore testing the application works okay with Internet Explorer on a Windows Server is not a configuration which should be supported. It does not test that the application runs on a real supported configuration, i.e. Internet Explorer on a Windows Workstation.

Creating a screen capture on every action

Someone recently commented on an article I wrote about generating a screen capture when an exception is thrown (see Generating a screen capture on exception thrown with Selenium 2).

Performing an action when an exception is thrown is built into the Selenium framework. You just need to create the action to generate a screen capture and hook it into the framework.

The Selenium framework comes with a WebDriverEventListener interface. In my article above I created an implementation of the WebDriverEventListener interface. To do this properly, you need to implement the following methods:

beforeNavigateTo(String url, WebDriver driver);
afterNavigateTo(String url, WebDriver driver);
beforeNavigateBack(WebDriver driver);
afterNavigateBack(WebDriver driver);
beforeNavigateForward(WebDriver driver);
afterNavigateForward(WebDriver driver);
beforeFindBy(By by, WebElement element, WebDriver driver);
afterFindBy(By by, WebElement element, WebDriver driver);
beforeClickOn(WebElement element, WebDriver driver);
afterClickOn(WebElement element, WebDriver driver);
beforeChangeValueOf(WebElement element, WebDriver driver);
afterChangeValueOf(WebElement element, WebDriver driver);
beforeScript(String script, WebDriver driver);
afterScript(String script, WebDriver driver);
onException(Throwable throwable, WebDriver driver);

In my screen capture when an exception is thrown, I put the necessary code to generate a screen capture in the onException method. You can see from the list above that you can have Selenium perform an action when other events occur. For example, if you wanted to do a screen capture after each click() action, you could write:

public void afterClickOn(WebElement element, WebDriver driver) {
    String filename = generateRandomFilenameFromWebElementAndDriver(element, driver);
    createScreenCaptureJPEG(filename);
}

I'll leave it to the reader to figure out how to create the generateRandomFilenameFromWebElementAndDriver. You might do something like use getCurrentUrl(), convert things like colon, slash, etc. into valid filename characters then append the web element getTagName() to the end of the filename.

Or you could do something tricky like have the beforeClickOn generate the filename and the afterClickOn use that filename to generate the screen capture.

If the event you want to trigger a screen capture is not listed above, there is no easy answer for how you would do it.

Initially, you might considering changing the WebDriverEventListener interface. But this would require a change to how Selenium works. Every time there is a new release of Selenium, you would have to merge your changes back in.

I would recommend submitting the changes as a new feature to the Selenium project to see if they'll incorporate it or wrap the Selenium methods with your own methods. For example, you would call your sendKeys() which could do a driver.sendKeys then do a screen capture.

Selecting WebDriver locators

When you are using WebDriver for test automation a test case really boils down to:

1. Find an element
2. Perform an action
3. Confirm the expected result

Finding an element and confirming the expected result requires locators. In WebDriver a locator is a way of uniquely identifying an element on the web page, i.e. in the Document Object Model (DOM). The By class is used in WebDriver to locator elements. You have:

• By.className(className);
• By.id(id);
• By.linkText(linkText);
• By.name(name);
• By.partialLinkText(linkText);
• By.tagName(name);
• By.cssSelector(selector);
• By.xpath(xpathExpression);

The most powerful locators are CSS and XPath. All the other selectors can actually be done using CSS or XPath. For example:

Original	CSS	XPath
By.className("foo");	By.cssSelector(".foo");	By.xpath("//*[@class='foo']");
By.id("bar");	By.cssSelector("#bar");	By.xpath("//*[@id='bar']");
By.linkText("Click Me");	N/A	By.xpath("//a[text()='Click Me']");
By.name("fee");	By.cssSelector("[name='fee']");	By.xpath("//*[@name='fee']");
By.partialLinkText("some");	N/A	By.xpath("//a[contains(text(),'some')]");
By.tagname("div");	By.cssSelector("div");	By.xpath("//div");

In addition to the simple locators, CSS and XPath can selector more complex elements. Rather than saying I want all the DIV tags, I can say I want all the DIV tags whose parent is a SPAN. In CSS this would be "span>div" and in XPath this would be "//span/div".

The combinations are endless. For each tag in the XPath or CSS I can add multiple identifiers. So I could have locators for things as complex as "all DIV tags, with name containing 'foo' and class equal 'bar' whose parent is a TD but only in the TABLE with id='summary' and the class equal 'humho'"

The first thing to understand is that CSS will be noticeably faster than XPath when testing against Internet Explorer. Your tests could run as much as 10 times slower (something which runs on a day on Firefox could take a week on Internet Explorer) when using XPath.

So the first thing to remember is CSS is better than XPath. However, some things are easier to express as XPath. So occasionally you might need to use XPath.

If you have a selector like "html>body>table>tbody>tr[2]>td[3]>a" it might work but if the developer finds it does not format nicely on Chrome, they need to throw in a DIV. So the selector changes to "html>body>div>table>tbody>tr[2]>td[3]>a". Later a new version of Internet Explorer comes out and the developer finds they need to add a SPAN to make it look proper on the new Internet Explorer and still look okay on older versions. So the locator becomes "html>body>div>table>tbody>tr[2]>td[3]>span>a".

If we spend all our time maintaining the locators, it could end up that the cost of maintaining the automation is greater than running the tests manually. In which case the automation is deemed a failure.

So you have to start looking for patterns. Is there something I could use on the first version of the application which also works on the second and third version? Can I predict a locator which will work on the fourth and subsequent versions?

Often the underlying technology changes but it continues to look the same to the user. So is there something visual I can use which will not change? In this example, the text for the anchor probably never changed. So I'd use By.linkText("whatever"); locator or By.xpath("//a[text()='whatever']);.

What if I find myself changing locators because sometimes the text is "  whatever", sometimes it is "whatever" and other times it is "whatever  "? Then I'm going to use By.partialLinkText("whatever"); or By.xpath("//a[contains(text(), 'whatever')]");.

The danger is that there might be two links which contain the substring "whatever". I need to make sure I am selecting the correct link. So the locator might need to be more complex. It might need to be partial text and parent information. For example, if the text appears in two different tables and I want the text from table 2. Table 2 has the id='foo2' then the locator might be:

• "table.foo2 a"
• "//table[@id='foo2']/tbody/tr/td/a[contains(text(),'whatever')]"

The first locator assume there is only 1 anchor in the second table. This might not be true in all cases. The second locator finds the second table but it searches all rows (TR) and all columns (TD) for an anchor (A) whose text contains the substring "whatever". This can be extremely slow, especially for large tables.

Finding the balance between locators which are too long and too short can be an art. The trick is to pick something. If it requires maintenance, pick a new locator which works on the previous versions and the new version. As you continue to maintain the locators you will see a pattern. You will start to see that chunks of HTML code never change Outside these chunks change (so keep the locator short enough to stay inside the chunk that does not change). Within the chunk there might be multiple matches if you make the locator too short. So figure out, within that chunk, what makes the element you want different from all the other matches.

So how do I look at the DOM? I need to see what the DOM looks like to be able to see all the possible locators which would work.

If you are using Internet Explorer 8 or higher you can press F12 to open the developer tools. If you are using Firefox you need to install Firebug then F12 will open Firebug. If you are using Chrome then CTRL-SHIFT-I will open the developer tools.

Beyond that, the only tool I use is my brain and the W3 standards.

Reading the W3 standards (or any standards documentation, ISO, ANSI, IEEE, etc.) can be difficult at first. Especially if you have been learning from books like "Web Design in 21 Days" or "Software Testing for Dummies." However, the more you read and understand standards documentation, the easier it gets to read other standards documents. If generating XPath was easy enough for a piece of software then why would they pay you to do the work? There are probably a dozen XPath locators for any given element on a page. Some will work once and need to be revised on the next release of the application. Some will work within the design pattern of the application and might never need updating. There is no way for a piece of software to spot the design pattern and know which locator will work best. This is what they pay you to do.

Excessively long XPath is brittle and will need a great deal of revising from release to release. Extremely short XPath will sometimes find the wrong element between releases. This leads to a test which fails unpredictably and can be difficult to debug. Not something you want in an automation suite. Finding the right balance is your job. The first time you select a locator it might need revising for the next release. You need to look at why you selected the first locator when selecting the revised locator. The second locator should work for the first release and the second release. When the locator fails, you need to select a new locator which would have worked on the first release and all subsequent releases, including the next release. After a while you should start to see the pattern. The pattern is usually derived from some design pattern the application is being developed with. Learn about Design Patterns, it will be extremely helpful in generating good test automation. If the developers change the tools, libraries, design patterns, etc. you should expect the locators to fail. At this point, selecting a locator which works with the next release but does not work with the previous release makes sense. Major change in development usually implies major change in test automation. It would be difficult for a tool to realize when it needs to abandon old locators.

Essentially, automation is all about finding elements (locators), performing actions on them, confirming the expected results (usually involves more locators). Two thirds of the work is about the locators. Learning XPath, CSS and DOM will make your job that much easier.

• XPath 1.0: http://www.w3.org/TR/xpath/
• XPath 2.0: http://www.w3.org/TR/xpath20/
• XPath functions: http://www.w3.org/TR/xpath-functions/
• CSS 1.0: http://www.w3.org/TR/2008/REC-CSS1-20080411/
• CSS 2.0: http://www.w3.org/TR/CSS2/
• CSS 3.0: http://www.w3.org/TR/selectors/
• DOM: http://www.w3.org/TR/DOM-Level-2-HTML/html.html

When possible, use CSS selectors as they are faster. Some things are easier to locate using XPath and XQuery (XPath functions). It is better to have a test run slow and be easy to maintain. So if CSS selectors are complex and unintuitive you might want to use XPath functions instead.

This is essentially how I decide on locators.

.

Tools to determine how to use Selenium

Being able to view the source code to a page is helpful when trying to use Selenium. Many of the methods in Selenium require you to locate the element via something related to the source code. For example, if I wanted to click on an element I need to look at the source code. From the user's point of view they might see:

Click Me

But the underlying source code could be:

<span>

    <div>

        <table>

            <tbody>

                <tr><td><strong onclick="javascript:clickme();">Click Me</strong></td></tr>

            </tbody>

        </table>

    </div>

<span>

I might assume there is an anchor and I want to find the <a> element with the text 'Click Me'.  By looking at the source code I can see that I actually want to find a <strong> element and click on it.

Just viewing the source code isn't always sufficient however. AJAX and Web 2.0 means that the source code gets changed in memory after it is loaded. So what is actually available in the DOM (and Selenium) might be different than what you see when viewing the source code.

So how do you see what Selenium sees? A simple solution is to write your Selenium code to get to a point in the program then print out the source code using Selenium. Selenium 1.0 and 2.0 both have methods to return the HTML source code as a string, which you can print.

The problem with this is that you might have to write a program to print the source. Then with the source, find the code you want to click. Then run the program so it clicks the element then prints the new source. Then run the program to click the first element, click the second element and print the source. For n steps this will take O(n^2) run time. Not very efficient.

So how can you view the dynamically changing source? The answer is in browser tools.

If you are using IE7, search for "microsoft dev toolbar". When you install it, it gives you development tools which can view the page in real time. If you are using IE8, the tool is built in. Just press F12 and it will display the development tools.

If you are using Firefox, install the Firebug extension.

If you are using Google Chrome, right click an element and select Inspect to bring up the development tools.

Using Selenium 2.0 with WebDriver and Safari

I've been looking at Selenium 2.0 and writing test cases using WebDriver. Looking at the APIs I see there is a class for Android, iPhone, InternetExplorer, Firefox and Chrome which extends the RemoteWebDriver.

So how do I use Safari? The code to set the search string for Google, using WebDriver, would be:

WebDriver browser = new FirefoxDriver();

browser.get("http://www.google.com");
WebElement input = browser.findElement(By.name("q"));
input.sendKeys("Selenium");

If I wanted to do the same thing using Safari web browser, I could use:

Selenium browser = new DefaultSelenium("localhost", 4444, "*safari", "http://www.google.com");

browser.type(name("q"), "Selenium");

The problem with this is I need to do things differently for Safari. I have to use Selenium 1.0 commands to test Safari and Selenium 2.0 for everything else. So how can I use a browser which was supported in Selenium 1.0 with all the Selenium 2.0 APIs?

The solution is:

Selenium sel = new DefaultSelenium("localhost", 4444, "*safari", baseURL);
CommandExecutor executor = new SeleneseCommandExecutor(sel);
DesiredCapabilities dc = new DesiredCapabilities();
WebDriver browser = new RemoteWebDriver(executor, dc);

browser.get("http://www.google.com");
WebElement input = browser.findElement(By.name("q"));
input.sendKeys("Selenium");

If you look at this block of code, the last 3 lines are the same as the last three lines of the first block of code. Essentially, I add the first three lines then change the WebDriver declaration to a new RemoteWebDriver.

The one thing I found however, on my Mac OS X, if I started the SeleniumServer with setSingleWindow(false) it would fail to work. I run my SeleniumServer inside my Java code using:

private static SeleniumServer ss;
private static RemoteControlConfiguration rcc;

@BeforeClass
public static void setUpBeforeClass() throws Exception {
rcc = new RemoteControlConfiguration();
rcc.setInteractive(true);
rcc.setSingleWindow(true);
rcc.setTimeoutInSeconds(10);
ss = new SeleniumServer(rcc);
ss.start();
}

If you are running the SeleniumServer from the command line, you'll have to look at the command line options to ensure you are running in Single Window mode.

New Job

It certainly has been a while since I posted to my blog. For anyone who is curious, I started a job as QA Manager at Certicom Corporation.

Certicom is in the business of cryptography. We create libraries or toolkits for C and Java which are used by customer applications. At the lowest level is Crypto. Crypto is cryptographic routines and algorithms used as part of a security solution. Customers like XM Radio use our Security Builder® Crypto™.

The next level up is PKI or Public Key Infrastructure. Security Builder® PKI™ enables you to add robust, standards-based digital certificates and key management to applications and devices, ensuring trust and non-repudiation. Some customers will develop their own Crypto solution for use with our PKI or they will use our Crypto solution with our PKI.

The security most people in the public know about is SSL or Secure Sockets Layer. Our Security Builder® SSL™ product can be used for people wishing to implement SSL. Either in a client or server. For example, with our product you can develop a mod_ssl for use with Apache Web Server.

Another term you may be familar with is VPN or Virtual Private Network. To create a VPN requires IPSec or Internet Protocol Security. This is acheived using Certicom's Security Builder® IPSec™.

In addition to all the publicly available products we create custom solutions for various industries and companies.

The company was recently acquired by Research In Motion. Most people know this company as BlackBerry, which is just a product the company produces.

As QA Manager I have the challenging task of testing all the different implementations of our products. Some are Windows, AIX, HP-UX, Linux, Solaris or Mac OS based but others are build on embedded devices. Our Asset Management System, used in chip manufacturing plants, utilizes Web Services, AJAX, JavaEE and other web technologies as a front end to a complex cryptography solution.

At this time I am actually looking to hire testers for testing these products. It is quite challenging to find people who can do the work. Ideally, they need to know or be able to learn:

• C, C++ and Java programming
• Defect tracking systems
• Test reporting
• Knowledge of Windows, Linux or UNIX
• Experience testing embedded devices or mobile devices, e.g. BlackBerry
• Ability to create test plans
• Working knowledge of test automation
• Unit testing, system testing, integration testing
• Experience with source control
• Knowledge of cryptography
• Development experience or experience testing toolkits and libraries

Basically, someone who is a junior programmer, an intermediate tester and some IT or support knowledge.

If you know anyone who fits this description or you believe you are up for the challenge, you can apply to the position at https://rim.taleo.net/careersection/professional/jobdetail.ftl?lang=en&job=188752. If this link does not work, try the following:

1. Go to http://www.rim.com/
2. Go to the Careers section
3. Go to Americas, this should bring you to a Job Search page
4. In the keywords field enter: Certicom

This should give you a list of all the jobs current available. Anything relating to Test would be my department.

One advantage of test automation

One big advantage of test automation is testing different configurations. For example, when I was working at Quest Software I tested web applications for multiple configurations. There were different operating systems with different web browsers. Here are some of the combinations:

1. Windows 2000, Internet Explorer 6.x
2. Windows 2000, Internet Explorer 7.x
3. Windows 2000, Firefox 3.x
4. Windows 2000, Safari 3.x
5. Windows 2000, Safari 4.x
6. Windows XP, Internet Explorer 6.x
7. Windows XP, Internet Explorer 7.x
8. Windows XP, Firefox 3.x
9. Windows XP, Safari 3.x
10. Windows XP, Safari 4.x
11. Windows Vista, Internet Explorer 7.x
12. Windows Vista, Firefox 3.x
13. Windows Vista, Safari 4.x
14. Solaris 10, Firefox 3.x
15. Redhat Linux, Firefox 3.x
16. SuSE Linux, Firefox 3.x

Without automation, we would look at equivalence classes. Using our experience and knowledge of the different operating systems, we would estimate that Internet Explorer 6.x on Windows 2000 is going to behave the same as Internet Explorer 6.x on Windows XP and Vista. We might also assume Firefox 3.x on Redhat Linux and SuSE Linux is going to be equivalent. Additionally, we would look at customer base and see if we can reduce the combinations because customers are unlikely to be using some of them. For example, we found that most Windows customers who were using Firefox 3.x were also on Windows XP. So the list was trimmed down to:

1. Windows 2000, Internet Explorer 6.x
2. Windows XP, Firefox 3.x
3. Windows XP, Safari 3.x
4. Windows Vista, Internet Explorer 7.x
5. Windows Vista, Safari 4.x
6. Solaris 10, Firefox 3.x
7. SuSE Linux, Firefox 3.x

We selected SuSE Linux because we had encountered issues unique to SuSE Linux. All issues found in Firefox 3.x on Redhat Linux were similar to Firefox 3.x on Solaris 10.

Looking at the list, there are still 7 configurations. If manually testing each configuration takes two weeks and a test cycle is two weeks, we need 7 testers working on this full time. This also assumes everything goes fine and there aren't additional issues to content with. The reality was that we had to add a couple more configurations. The different desktop managers for Linux actually made a different on how things rendered. It was really exposing problems in the web browser but from a customer's point of view, most websites look fine but our application does not, therefore the problem is in our application not the browser. Non-technical customers don't want to hear the explanation why our application doesn't work with the Redhat Linux 4, Gnome desktop and Firefox 3.x.

So we had to add in another configuration:

1. Windows 2000, Internet Explorer 6.x
2. Windows XP, Firefox 3.x
3. Windows XP, Safari 3.x
4. Windows Vista, Internet Explorer 7.x
5. Windows Vista, Safari 4.x
6. Solaris 10, Firefox 3.x
7. SuSE Linux, KDE Desktop, Firefox 3.x
8. Redhat Linux 4, Gnome Desktop, Firefox 3.x

This adds one more tester to the test cycle. If we assume testers will be productive 6 hours a day (checking mail in the morning, lunch, meetings, etc.) then we are looking at 6 hours * 5 days * 2 weeks * 8 configurations for a total of 480 hours.

But what if I had an automation tool that ran on all the platforms and worked with all the different web browsers? I could write one test suite and run it with different configurations. One such tool is Selenium. It is written using JavaScript and Java and therefore runs on all these platforms.

The test automation can be run after hours. This means 24 hours a day and 7 days a week. We can also run the test suite in parallel. Even if the test suite ran as slow as a manual tester, I just need to add another computer to run the test suite. A computer costs a lot less then a manual tester.

This is a little naive however. You will need at least one tester to automate the test suite. Since automation is development, it should be someone who knows how to create test strategies, test plans, test cases and knows how to do software development. This one employee is probably going to cost more than a manual tester would. However, they are not going to cost 8 times as much.

The reality is that creating a maintainable test automation framework takes longer than 2 weeks. Additionally, as you find defects time will be lost filing the defect and following up on it. Other issues which occur with this is that the implementation of Selenium on the different platforms will present their own set of problems. Basically, the same reason we find issues with our web application on the different platforms is going to cause issues with Selenium as well.

So be forewarned, test automation is not as great as some people would lead you to believe. If done correctly, there will be some cost saving. The best thing is once you have a good test framework in place, the maintenance of it does yield significant cost saving. The problem is, most people don't put enough effort into the design and implementation of the framework to reap the benefits later.

One huge things which can derail test automation will also add cost to development. One project I worked on looked the same from the customer's point of view regardless of which configuration they were using but if you looked at source control they actually had different codes for the different configurations. The web pages being sent down from the server were slightly different depending on the configuration. The code was littered with lots of "if IE6 then do one thing else if IE7 do another thing else if FF3 do something TOTALLY different". Patches to the web browsers and operating systems often broke the application.

From an automation point of view, if the web page served for IE6 is different from the web page served for FF3 then it is as if we are testing two different web pages. The worst part is that the pages are often, initially, close enough that an automator will try to code once for both pages. The effort to create a single automation script that works for both configurations will be more time consuming then normally anticipated and the maintenance for this script will be greater than expected.

But if coded well, the application can be automated for one configuration. A few tweaks for different configurations and then run continuously on all configurations.

what goes where with Selenium RC?

When using Selenium RC. You have five machines, conceptually:

1. The Application Server (and the application you are testing)
2. The Web Server
3. The web browser (client)
4. The Selenium Server
5. The Selenium Client

The reality is that the web browser and the Selenium server are the same machine. Additionally, there is no reason you cannot be running everything on one machine. Let's say we have unlimited machine and we want to test on a number of different platforms with a variety of different browsers. Here is a theoretical setup:

1. Application Server
2. Web Server
3. Windows XP, Internet Explorer 6, Selenium Server #1
4. Windows Vista, Internet Explorer 7, Selenium Server #2
5. Windows 7, Internet Explorer 8, Selenium Server #3
6. Linux, Firefox 3.5, Selenium Server #4
7. Mac OS X, Safari 4.0, Selenium Server #5
8. Build Machine with Test Suite
9. Database
10. Subversion repository with application and test suite source code

Here is how communication will flow:

1. Machine #8 will check out the code from machine #10
2. Machine #8 will compile the application
3. Machine #8 will tell machine #1 to stop the application server
4. Machine #8 will deploy the application to the application server
5. Machine #8 will make any necessary changes to the database
6. Machine #8 will tell machine #1 to start the application server
7. Machine #8 will compile the test suite code
8. Machine #8 will start the Selenium Server on machines 3, 4, 5, 6 and 7
9. Machine #8 will run, in parallel, five copies of the test suite, each one configured to point to a different test machine
10. Assuming the test runner used on step 9 creates test results, machine #8 will save the test results.
11. If may also send an email to interested parties

During step 9 there will be the following call in the Test Suite:

    Selenium selenium = new DefaultSelenium(seleniumServer, seleniumPort, browser, baseURL);

where seleniumServer/seleniumPort is the IP address and network port of the machine running the test. If we have this parameter configurable, we can configure the parameter to the selenium server on machine 3 then run it, configure the parameter to machine 4 then run it again, and so on.

The browser parameter would be "*iexplore" for machines 3, 4, 5, "*firefox" for machine 6 and "*safari" for machine 7. The baseURL is going to be the URL for machine #2, which redirects HTTP requests to our application on machine #1.

If  your test code is written in Java, everything can be done so long as some version of Java is installed on all the machines. You could even test your application on Firefox and Solaris 10.

Step by Step Selenium Java

This is a document on how to write a Selenium Test Case in Java.

1. Go to Selenium HQ.
2. Go to the Download section and download Selenium IDE plus Selenium RC
3. Install the Selenium IDE into Firefox (I assume you have Firefox installed)
4. Go to Google.
5. From the Tools menu in Firefox select Selenium IDE.
6. From the Add-On Preferences, change Selenium to save in Java.
7. On the Google page, right click and select Open /ig
8. Type Selenium in the Google search text field.
9. Press TAB to move focus away from the input field.
10. Right click on the search text field and select verifyValue q Selenium
1. Note that verifyValue is the action, q is the locator and Selenium is the optional third value
11. Click the I'm Feeling Lucky button.
12. Go to the Selenium IDE and you should see everything we just did.
13. Click the red dot in the upper right to stop the recording.
14. From the File menu, save the file as LearningSelenium.java.
15. Open Eclipse
16. Create a Java Project
17. Add the jar selenium-java-client-driver.jar to the project.
18. Add the library junit 3.
19. Right click on the project name in the Package Explorer and add new package com.example.tests.
20. Import from File System, the LearningSelenium.java file we saved earlier.
21. When you finish there should be an error because the class is called Untitled by default.
22. Edit the file and change the class name to LearningSelenium
23. In the setUp() change the http://change-this-to-the-site-you-are-testing/ to http://www.google.ca/
24. If using Mac OS, change the browser string from *chrome to *safari.
25. Add a public void tearDown() throws Exception which just calls super.tearDown();
26. Run Firefox from the command line using firefox -P (or firefox-bin -P on UNIX/Linux machines)
27. Create a new profile, call it selenium and save it in the Eclipse workspace
28. Go to the command line and start the selenium server using:
1. java -jar selenium-server.jar -firefoxProfileTemplate /directory/where/you/saved/firefox/profile/selenium
2. But change the /directory/where/you/saved/firefox/profile/selenium to the place you saved the profile in step 25.
29. Back in Eclipse, right click the class name and select Run As, JUnit Test

You are done.

Selenium 101

So you want to learn about Selenium for web testing.

Selenium comes in 3 parts.

1. Selenium-IDE
2. Selenium-Core
3. Selenium-RC

The IDE is a Firefox plugin. You open the IDE and it starts recording what you do inside the browser. The information is stored in an HTML table with one action per row and three columns per action. The first column is the action, the second column is the locator and the third column is optional parameter.

For example, if I click a link the first column will be 'click', the second column will be some way of uniquely identifying the link and the third column will be blank. Another example, if I enter text into an INPUT the first column will be 'type', the second column will be some way of uniquely identifying the text field and the third column will be the text I entered into the field.

You could actually write Selenium test cases using an HTML edit and creating a TABLE but it would be hard for you to know all the actions and you'd have to have some way of determining locators.

When Selenium IDE is running, you can right click on elements and Selenium actions will be available on the context menu. Things like verifyTitle. When you select these, they are added to the Selenium IDE.

From the Selenium IDE you can save what you have recorded. You could actually record everything you do, save it and then load it and play it back later. By default, it will save an HTML TABLE but you can change the settings for Selenium IDE and save it in different languages. I use Java so I save my recording as jUnit.

The whole idea behind Selenium Core is to use javascript, in the Selenium window, to drive the window displaying your web application. So when you load a script into Selenium IDE and run it, the IDE translates the commands in the table to actions using the Selenium Core. If you install the IDE you automatically have the Core.

Selenium RC (Remote Control) allows you to play the scripts from a remote machine. It doesn't have to be a remote machine. It can be the same machine and usually is for development. The Selenium RC comes in two parts: the client and the server.

If you saved the recording as HTML, you pass the HTML script in on the command line directly to the server. To run the server you use Java. The server is an executable jar. To run the server I would use:

java -jar selenium-server.jar -help

This will display a help message. If I run it with:

java -jar selenium-server.jar -htmlSuite myscript.html

It will run the Selenium IDE recording (assuming the file myscript.html contains a file I saved from Selenium IDE). If I save the file as Java jUnit I will have to create a class to run the file. I would go into my favourite editor (today it is Eclipse), create a project, add the Java file created from Selenium IDE as a jUnit 3 file. To allow the project to talk to the Selenium Server I would add the Selenium Java Client to my Java project. Now when I run the jUnit it will create an instance of Selenium and send the appropriate commands to the Selenium Server. The server needs to be running before you run the test case. To run the server I would just use:

java -jar selenium-server.jar

If I need to change the port or host there are command line switches I can add but the -help will show you those. If you don't give a port or host it will default to port 4444 and host localhost (so the server and client will be on the same machine).

If you don't want to use Java and you want to use say Perl, you can save the recording from the IDE in Perl the use the Selenium Perl Client to talk to the Selenium Server.

That is essentially Selenium in a nutshell.