Forcing a remote computer to restart

Occasionally I find I need to reboot a remote computer. In my current situation the remote computer is Windows XP and accessed via VNC. However the VNC server will occasionally stop responding.

If the machine's IP address was 1.2.3.4 and I try:

shutdown /m 1.2.3.4 /r /t 1

it does not reboot the computer. Assuming there is nothing to save on the computer and I just want it to reboot and restart the services, I am happy to just kill processes until I hit something critical and it reboots.

However, if you hit the RPC process, you will be unable to send the command to kill more processes. So you have to be careful about which process you are killing. What I have found is killing the lsass.exe process will cause the computer to reboot. So issuing:

taskkill /im "lsass.exe" /s 1.2.3.4 /f

Will kill the lsass.exe proccess on the remote machine and force a reboot of that computer. The reboot will not be immediate. A 30 second countdown will happen as the system warns you a critical process was killed. However it will reboot the computer after the countdown is complete.

Dealing with things that disappear from the DOM when you look at them

From time to time I encounter a javascript feature which does not let me examine it. For example, auto-complete features on input boxes.

We have all seen this feature if you go to http://www.google.ca. You start typing something and Google gives you suggestions on what you are going to type. For example, I enter "WebD" and Google suggests "WebDriver".

The problem automating some of these is finding the list of suggestions. For example, go to http://www.asp.net/ajaxlibrary/ajaxcontroltoolkitsamplesite/autocomplete/autocomplete.aspx with Chrome, Inspect the auto-complete input box (right click in the input box and select "Inspect") and type something into the input box.

You will see a list of suggestions appear as you type but looking at the DOM in the Inspect window there is no list of suggestions attached to the input element.

They have to exist somewhere in the DOM but you have to find them. So while the suggestion list is open I click on the Inspect window and press CTRL-F to find one of the suggestions.

The problem is the moment I click in the Inspect window, the list of suggestions disappears. So the CTRL-F in the Inspect window finds nothing.

If I type a word in the input box then press CTRL-F (don't change focus to the Inspect window) the auto-complete list disappears. Even if it didn't disappear, it would find the suggestions in the browser window, not in the Inspect window.

So here is how I find the suggestions in the DOM...

WebDriver driver = new ChromeDriver();
driver.get("http://www.asp.net/ajaxlibrary/ajaxcontroltoolkitsamplesite/autocomplete/autocomplete.aspx");
driver.findElements(By.cssSelector("input[name$='myTextBox']")).sendKeys("test");
String src = driver.getPageSource();
driver.quit();

then I set a breakpoint on the driver.quit(); and run in Debug mode. When it stops on the breakpoint I start using the Watch Window of my IDE.

If I enter "test" and one of the suggestions is "testASG" then I'm trying to find the list with "testASG" as one of the suggestions. So I will add:

src.indexOf("testASG");

to the Watch Window. Let's say it returns 34923. So now I know the text it at 34923 and that the element wrapping the suggestion is BEFORE 34923. So I can try:

src.substring(34500);

and examine the results. From this I might see "<li>testASG</li>" and some other stuff before the <li>. This tells me that the list of suggestion is a <ul> element (<ol> would be numbered). So I can start using:

src.indexOf("<ul");

I know it will start with <ul but there would be different, unknown attributes after the ul. When I did this it returned something like 5723. So I looked at:

src.substring(5720);

I see it is finding a "null". Not what I was looking for. So I might start looking for ul but using the index of "testASG". So I might try:

src.substring(33000).indexOf("<ul");

If that returns a number higher than 34923 (after the li) then I need a smaller number. If it gives me a number between 33000 and 34923 then I can look at the substring and see if that was the ul I was looking for.

By narrowing down (binary search) where the ul is for the list of suggestions I can finally see what the DOM looks like by examining the src variable. At some point I will figure out the locators for the ul, li, etc. I can even trying the following locator in the Watch Window:

driver.findElements(By.cssSelector("ul#AutoCompleteEx_completionListElem>li");

To get a list of all the suggestions.

Determining which version of Selenium goes with which browsers

Sometimes it is hard to know which version of Selenium you should be using. There is no hard and fast rule of which Selenium works with which browser but I have always found that you want a Selenium which was released after the browser but as close to the browser release date as possible.

It has been my experience that the latest version of Selenium and the latest version of Chrome go together. I have never found that the latest version of Selenium does not work with the latest version of Chrome.

On the other hand, I have seen Mozilla release a new version of Firefox and it takes a few weeks for a new version of Selenium to appear. During this period I find the latest version of Selenium does not work with the latest version of Firefox.

For example, Firefox 18.0 was released on 08-Jan-2013. At the time the latest version of Selenium was 2.28.0, released on 11-Dec-2012. If you updated to Firefox 18.0 the moment it was available, you might find that some of your Selenium tests started failing unexpectedly. Selenium 2.29.0 was released on 17-Jan-2013. I always disabled the ability for Firefox to automatically upgrade. When Selenium 2.29.0 was released, I upgraded my Selenium to 2.29.0 and my Firefox to 18.0.

You can see the release dates of Firefox (and other Mozilla browsers) on https://wiki.mozilla.org/Releases. What I do is create a list of Selenium release dates from the Selenium download area (https://code.google.com/p/selenium/downloads/list). I would then look at the Mozilla wiki and determine which Firefox had been released on an older date. From my example above:

11-Dec-2012    Selenium 2.28.0
30-Nov-2012    Firefox 17.0.1

or

17-Jan-2013    Selenium 2.29.0
08-Jan-2013    Firefox 18.0

Or in other words, if the project I am on has set a specific version of Firefox to support then I look for the Selenium which was released AFTER the Firefox release date. That is the combination I would test with.

For Internet Explorer I can find the release dates by searching for "Internet Explorer Release Dates". This currently takes me to IE Downloads. I also find the Wikipedia has an Internet Explorer page which gives release dates as well.

For the most part however, I let the version of Firefox drive which version of Selenium I need. Because I tend to have clients request IE7, IE8 and IE9 support, I find it best to go with the latest version of Selenium to support the latest version of IE.

Selenium does have limited support for Safari but this is typically not an issue. Much like IE, I let the version of Firefox drive which version of Selenium I need regardless of which version of Safari I need to support.

It should be noted that this is a general idea of how I determine which version of Selenium I want. If I use this rule and start finding that the version of Chrome matters, I would start trying to match the release date of Chrome to the release date of Selenium.

List of Selenium/WebDriver blogs.

I was recently pointed to a nice list of Selenium/WebDriver blogs. If you are looking for more information about Selenium/WebDriver you should check out http://it-kosmopolit.de/Selenium/blog/selenium-blogs/selenium_blogs.php.

I haven't checked out all the links but there appears to be a good start to the list of Selenium/WebDriver blogs available out there.

Generating a file of a specific size

Every once in a while someone is looking for a file of a specific size. Occasionally, it must be real data. If you are transmitting the file and the data will be compressed then the type of data will make a difference.

However, if you just need a file to fill some space or there will be no compression then Windows has a neat little utility called FSUTIL.

The FSUTIL file can be used for a number of things but the nicest feature is creating a new file filled with zero bytes. First, you need to know how many bytes. If you want a file which is 38 gigabytes then you need to figure out how many bytes that is. Technically, it is 1024*1024*1024*38. If you want  a rough idea you can just use 38,000,000,000 but a 38 gigabyte file is really 40,802,189,312. Next is which file you want to hold the data. Let's say you want to create C:\DELETEME.TXT then the full FSUTIL command is:

fsutil file createnew C:\DELETEME.TXT 40802189312

This will create a 38G file is a matter of seconds.

For UNIX, Mac OS X or Linux you can use DD. The DD command is for converting and copying files. If we wanted to create the 38G file with DD the command would be:

dd of=deleteme.txt oseek=79691776

The oseek is the magic. It will seek n*512 bytes. The standard size of a block is 512. So we take 79691776*512, which is 38G. Even easier would be:

dd of=deleteme.txt oseek=38m obs=1024

This will generate a file of 38M * 1024 or 38G. Much easier to figure things on working with values like these. That is, no need for a calculator.

IMPORTANT: after you press enter on the DD command it will take the standard input as its input. So you need to enter CONTROL-D.

The other option for UNIX/Linux/Mac OS X is to copy a file of a set size. The nice thing about this is you can take a file with real data and copy it enough times to make a single file of the correct size. For example, if I have a text file with 512 bytes of real data I can set the if= to that file and make multiple copies of that one file into the output file.

Removing logout button

When I set up a Selenium Server it only works when someone is logged in. In a previous article I wrote how to make a user automatically log in. For this article I'll write how to prevent that user from logging out.

1. Run regedit.exe
2. Go to HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Policies
3. Create a new Key called Explorer
4. Go to Explorer folder
5. Create the DWORD key StartMenuLogoff.
6. Set it to 1.
7. Go to the Start Menu and use Shutdown or Reboot (so Logoff is not the last option used).

When you log in as that user now, they will not have a Logoff button available to them. This has been tested on Windows 7.

StaleElementException

Most automation tools depend on the concept of the page has finished loading. With AJAX and Web 2.0 this has become a grey area. META tags can refresh the page and Javascript can update the DOM at regular intervals.

For Selenium this means that StaleElementException can occur. StaleElementException occurs if I find an element, the DOM gets updated then I try to interact with the element.

Actions like:

driver.findElement(By.id("foo")).click();

are not atomic. Just because it was all entered on one line, the code generated is no different than:

By fooID = By.id("foo");
WebElement foo = driver.findElement(fooID);
foo.click();

If Javascript updates the page between the findElement call and the click call then I'll get a StaleElementException. It is not uncommon for this to occur on modern web pages. It will not happen consistently however. The timing has to be just right for this bug to occur.

Generally speaking, if you know the page has Javascript which automatically updates the DOM, you should assume a StaleElementException will occur. It might not occur when you are writing the test or running it on your local machine but it will happen. Often it will happen after you have 5000 test cases and haven't touched this code for over a year. Like most developers, if it worked yesterday and stopped working today you'll look at what you changed recently and never find this bug.

So how do I handle it? I use the following click method:

public boolean retryingFindClick(By by) {
        boolean result = false;
        int attempts = 0;
        while(attempts < 2) {
            try {
                driver.findElement(by).click();
                result = true;
                break;
            } catch(StaleElementException e) {
            }
            attempts++;
        }
        return result;
}

This will attempt to find and click the element. If the DOM changes between the find and click, it will try again. The idea is that if it failed and I try again immediately the second attempt will succeed. If the DOM changes are very rapid then this will not work. At that point you need to get development to slow down the DOM change so this works or you need to make a custom solution for that particular project.

The method takes as input a locator for the element you want to click. If it is successful it will return true. Otherwise it returns false. If it makes it past the click call, it will return true. All other failures will return false.

Personally, I would argue this should always work. If the developers are refreshing the page too quickly then it will be overloading the browser on the client machine.

When trial and error is a bad thing

I'm a hacker. Not the breaking into systems and doing damage hacker. To me a hacker is someone who learns things by trial and error. I will systematically poke away at something which for all intents and purpose is a black box. I hack to learn. I hack things I own. I'll create an instance of something and hack away at it. People I work with will create development, test or staging environments which I will hack.

I do not hack sites I don't own or have permission to hack. This is what differentiates good hackers and bad hackers.

What I do is poke at something. Maybe I'll try changing an input or altering the environment slightly and see how that changes things. I'll keep doing this until a pattern merges. From trying different things I start forming a hypothesis of what is happening inside the black box. If I try something and the result does not fit in my hypothesis, I form a new hypothesis. At some point I usually get a clear understanding of what is happening in the black box without ever seeing what is in the black box.

I essentially look at the symptoms and narrow down what the cause would be.

This is a good use of trial and error. The goal is not to find the input which gives me the desired output. If I stopped the moment I got the desired output, I might think I have the solution but I don't. Case in point, I input 2 and 2 and get 4. My hypothesis is that the black box does addition. At this point my hypothesis is correct. However, if I poke further I might find that inputing 2 and 3 gives me 6. Now I see that it is not addition. New hypothesis is that it is multiplication.

Hacking is really empirical. Unless I try every possible input, I cannot be certain my hypothesis is correct. For example, I might input 1 to a function and it returns 43, I input 2 and get 47, I input 3 and get 53. After inputting the numbers from 1 to 20 I notice all the numbers are prime! My hypothesis is that the function is a prime number generator. However, if I input 41 I get 1763. This is not prime (43 * 41 = 1763). Turns out the function is Euler's formula for finding prime numbers, i.e. n^2 + n + 41. This has been proven to only produce prime numbers when n is less than 40.

Still hacking can be a good thing. Trail and error to find THE answer is never a good thing.

I see a lot of people solving problems as follows:

1. Program or computer not functioning correctly.
2. Change something.
3. If program or computer not functioning correctly go to step 2.
4. Problem solved.

Now maybe they did find the right solution but most often they don't. Later the problem will come back with different symptoms. If I purchased a program from you and you used this method to solve the problem, here is how I see this as a consumer of software:

My car is running slower than normal. I bring it to my mechanic and he does the following:

1. He changes the spark plugs and charges me for that.
2. Car is still running slow.
3. He adjusts the valve on the carburetor and charges me for that.
4. Car is still running slow.
5. He rotates the tires and charges me for that.
6. Car is still running slow.
7. He changes all the fluids and charges me for that.
8. Car is still running slow.
9. Cars today have a lot of electronics, so he disconnects the battery for a week.
10. All my programming, bluetooth, radio stations, clock, GPS, etc. are gone.
11. The car is no longer running slow.
12. Three months later the car is running slow again.
13. My mechanic disconnects the battery for a week.
14. All my programming, bluetooth, radio stations, clock, GPS, etc. are gone.
15. My car is still running slow.

Would you pay for all the work the mechanic did? I think it is safe to say that NO ONE would put up with this. Some people might put up with it until step 12 then find a new mechanic. Others would put up with this until just step 2 or 4. Most of us would not pay for anything after step 2.

I've worked in industries where EVERYONE programs like this. There might be 4 or 5 different vendors and you really don't have any other choice. However, it just takes one guy to write quality software and everyone switches to that other guy. Trying to win back those customers means you have to make up for all the poor software issues PLUS give them some incentive to switch away from the guy who has always given them good software.

My environment for Selenium 2.0 (WebDriver)

One thing I believe to be very important for any sort of software development (and test automation with Selenium is software development) is having a good environment. So here is how I set up my environment to start with Selenium 2.0 and Java.

What you need:

1. Java (go to  http://www.oracle.com/technetwork/java/index.html)
1. I typically go with Java SE 1.6.0. If you don't need the latest, go with something more mature.
2. Download the full SDK and docs.
3. Install them in C:\jdk1.6.0_30 (assuming you downloaded build 30)
4. I like to put it in the root of the C drive with no spaces in case other things don't like spaces.
5. Unpack the docs into the same directory so they are easy to find. 
2. Eclipse (go to  http://www.eclipse.org/downloads/)
1. I typically go with Eclipse IDE for Java Developers because it has everything for Selenium and it is small.
3. SVN (go to  http://subversion.apache.org/)
1. You need some sort of source control.
2. I usually go with Git but more people are still using Subversion.
3. There are more tutorials on SVN and better support.
4. Selenium (go to  http://code.google.com/p/selenium/downloads/list)
1. You'll need to download:
1. selenium-java-2.19.0.zip (or whatever the latest version is)
2. selenium-server-standalone-2.19.0.jar (or whatever the latest version is)
2. Unpack the zip file to some location
1. I'll typically unzip them into the workspace for the project
2. I'll also put the jar file in the same location
3. This way I can add the jar files to subversion
5. Eclipse support for SVN (go to  http://subclipse.tigris.org/ )
1. Normally I would suggest using Marketplace from the Help menu but not for the latest SVN client.
2. I like subclipse for an SVN client.
3. Once on the subclipse site go to download and install.
4. Copy the link for Eclipse Update Site URL for 1.8.x
5. Go to Eclipse Help->Install New Software...
6. Click Add...
1. Name: subclipse 1.8.x
2. Location: the update site URL you copied above.
7. Select all
8. Unselect Mylyn 3.x if you don't want it.
9. Follow the install wizard
6. Adding the JDK to Eclipse
1. Go to Window->Preferences in Eclipse
2. Expand Java
3. Select Installed JREs
4. Click Add...
5. Select Standard VM
6. Set JRE home to C:\jdk1.6.0_30\jre
7. Set JRE name to JDK 1.6.0
8. Everything else can be left as default.
7. Creating a project
1. Select File->New->Java Project
1. Project name: something relating to the application you are testing
2. I usually go with the defaults.
3. Maybe change the JRE.
4. Click Next >
5. On the Libraries tab
6. Add JARs... (add External JARs... if you didn't put them in the project workspace)
7. Go to the location you unzip the Selenium files.
8. Select the selenium-java-2.19.0.jar and selenium-java-2.19.0-src.jar files and add them.
9. Add JARs... (add External JARs... if you didn't put them in the project workspace)
10. Go to the location you unzip the Selenium files.
11. Go to the libs folder and add all the jar files in there.
12. Add JARs... (add External JARs... if you didn't put them in the project workspace)
13. Go to the location you downloaded the selenium-server-standalone-2.19.0.jar file
14. Add this file to the libraries.
2. Finish the rest of the project creation using defaults.
3. You can now start adding JUnit Test Cases to the project
1. I select JUnit 4
2. I select a package which follows Java conventions
3. For example, com.mycompany.selenium.application where application is the application we are testing.
4. Add a setUp() and tearDown()
5. Finish
4. Have a look at  http://seleniumhq.org/docs/03_webdriver.html for what to put in the setUp(), tearDown() and test cases.

The most important thing about using Selenium with Java is that you really need to know Java, Source Control (Subversion), an IDE (Eclipse) and a test framework (JUnit) before you even start using Selenium. So search the web for basic Java tutorials, play with Eclipse, check out http://www.junit.org/ and search for tutorials on JUnit 4 then look at the docs on Selenium HQ.

If you need help with Selenium and WebDriver, post a message to http://groups.google.com/group/webdriver. I usually respond to people there on a regular basis.

Automated GUI Testing

From time to time I will see an automation tool with image compare. The idea is to get a screen capture, save it as a baseline then when the automation runs it gets a screen capture and compares it to the baseline.

Originally this always failed because there would be small differences. For example, the screen displayed the time and date, this would always be different and the compare would always fail.

The solution was to mask out parts of the screen capture we know will not be the same or to capture a portion of the screen and see if any part of the baseline matches the portion of screen capture.

This was a better solution but the video drivers from one machine to the next would create slightly different screen captures. So if you developed the baseline on one computer and ran it on another there was a high probability the compare would fail. Even comparing the screen capture to a baseline created on the same machine has a high probability of failing.

So what is the alternative?

We could go back to manual testing but that would be a step in the wrong direction.

We could automate the tests but hire someone to sit and watch them. If they see something wrong, do a screen capture and file a bug. Problems with this are, you still have one person tied up for the day watching the automation. Additionally, getting a screen shot and filing a bug while the automation is running could be a problem. Ideally, you'd want to be able to pause the automation, get the screen capture, file the bug and resume the automation.

However, there is a third alternative. When I create the test cases they will usually have points in the testing when you want to confirm the display looks okay. In some cases we just assume the user will be constantly looking. With automation, you want to explicitly decide when we need to confirm the display is okay. At those points in the automation, do a screen capture. The name of the file and the directory you store it in will have enough information for you to know which test case created the screen capture.

Once you have a directory of screen captures, you can quickly scan them and see that everything looks okay.

How does this save time? Most the time, GUI Testing is 80% getting the to place you want to confirm good display and 20% actually confirming the display. So if you are just reviewing the screen captures the time to get to the point to create the screen capture does not involve a tester and you save, on average, 80% of the time.

Classic examples of this being even better are confirming different configurations on the same test cases. For example, I tested a Java library for creating graphics. I would run the tests on 5 different operating systems with 3 different versions of Java.. So the test runner would essentially run:

for os in (5 different operating systems):
    for jvm in (3 different versions of Java):
        run all tests

This would result in 15 different test runs. I would then display all 15 screen captures for test case #1. If they all looked the same it would be a pass. If any one of the screen captures looked wrong I'd know which operating system, which JVM and which test case. So I could file a bug against that configuration.

You could do a similar thing for web testing. Run the same tests against the different operating systems and different web browsers. Then compare the screen captures to ensure all the configurations displayed correctly.

If you wanted to you could video capture the display as the tests ran. With a good video play back you could fast forward through the parts you know won't change.

waitForElement

If you have Ajax calls or javascript which updates the page after the page has finished loading, you may have to wait for that code to finish before you can interact with the page. Essentially, waiting for page load is not sufficient.

The best way to deal with this is watch for something in the DOM which signals the script has finished executing. However, this is not always the easiest solution. If you are not the developer or familiar with javascript, it could take you days to figure out what event or change you need to wait for.

A better solution is to wait for whatever you want to interact with to be done. If you are waiting for a SELECT tag to be loaded with OPTION tags via Ajax there is no need to wait for the entire page to finish loading. You can just wait for the OPTION you wait to appear on the SELECT.

The best wait to do this is:

1. wait for a short period of time (250ms)
2. check for element
3. if element does not exist go to 1

There is one problem with this algorithm. If the element never appears, this becomes an infinite loop. So we add a time out:

1. maxTime = 5 seconds
2. timeSlice = 250 ms
3. elapsedTime = 0
4. wait for timeSlice
5. elapsedTime += timeSlice
6. check for element
7. if element does not exist AND elapsedTime < maxTime go to 4

Finally, this method must return true or false, where false means we did not find the element:

1. maxTime = 5 seconds
2. timeSlice = 250 ms
3. elapsedTime = 0
4. wait for timeSlice
5. elapsedTime += timeSlice
6. check for element
7. if element does not exist AND elapsedTime < maxTime go to 4
8. if elapsedTime < maxTime return true // found it
9. else return false // timed out

In Selenium and Java this might look like:

public WebElement waitForElement(By by) {
    WebElement result = null;
    long maxTime = 5 * 1000; // time in milliseconds
    long timeSlice = 250;
    long elapsedTime = 0;

    do {
        try{
            Thread.sleep(timeSlice);
            elapsedTime += timeSlice;
            result = driver.findElement(by);
        } catch(Exception e) {
        }
    } while(result == null && elapsedTime < maxTime);

    return result;
}

.

Your automation must not dictate your test plan

One of the things I see people getting into automation doing is selecting what to automate or how to test an application based on what the automation tool will let them do. This is a dangerous approach to automation.

The test cases I create for an application are based on what is important to the customer. I want to make sure that the customer experience is a good experience. If I create a set of test cases or use stories which reflect real customer usage of the application then I am most likely to find issues which will affect the customer.

I remember working on a project for 4 years. After 4 years of testing and improving the application we were at a point that over 90% of the features important to the customer were tested and bug free. Of the remaining 10% we knew of most the defects and had a work-around. We were at a point where we were testing extreme edge cases. At this point I found a major defect. The developer looked at the fix and realized the defect had been there since the beginning. In 4 years not one single customer reported this defect. The defect was easy to automate but really added zero value to the application. This is NOT a test case you want to start with when automating.

On another project someone found a defect in a desktop application. The steps to reproduce were:

1. Run an application not involved in the test case at all (Outlook for example)
2. Hand edit a project file using notepad or something other than the application it was intended for
3. Make a very specific change to the file
4. Run the application under test
5. Open the corrupted file
6. Select a feature which relies on the corrupted file
7. A modal dialog appears telling you the file is corrupt, do you wish to repair it.
8. Ignore the dialog
9. Use CTRL-TAB to switch to a different application not involved in the test case at all
10. Click on the application under test in a very specific location on the MDI client window

At this point the modal dialog is hidden behind the window with focus and the window with focus appears to be frozen. It is really waiting for you to respond to the modal dialog. This was a design flaw in the operating system. It was virtually impossible to fix in the application under test without a major re-design. It was highly unlikely a customer would be able to reproduce this defect. When the tester showed me the 'locked' state it only took me a few minutes to figure out what was going on. Our customer was typically a software developer with 7+ years of experience.

This was a useless test case. In both this and the previous test case it was a bad test case regardless of creating it manually or automating it. My point is, the test case came first. Even before we attempted to automate it, we decided whether or not it was a good test case.

Test automation NEVER proceeds test case creation or test planning.

Once you know what you want to test and the steps to testing it, you automate those steps.

This is the second mistake I see people getting into automation making. They decide WHAT they want to test but when they start automating it, the steps they generate with the automation are not the same steps as they would do manually. In this case you have taken the time to create a good set of test cases and thrown them out the door when you start automating. This is not a good idea.

Rather than changing the test case to something which is easy to automate, you need to figure out how to automate the test steps. This is what separates good automation from bad automation.

Many times I have seen a test case automated. It gets run and passes. We ship the application to the customer. He uses the same feature and it fails horribly. Why? Because the steps he used to get to the defect where not the steps we automated. We had a good test case. If an experienced tester had executed the test case manually, they would have found the defect. The person automating the test case just found it easier to automate something close to but not equivalent to the test case.

I am currently using Selenium 2.x with WebDriver. One of the changes from Selenium 1.x to 2.x is that you cannot interact with invisible elements. For example, a common trick on a website is to have an Accept checkbox on a download page. If you accept the terms the Download button becomes visible. In Selenium 1.x I could click on the Download button without clicking the Accept checkbox. The REAL test case was:

1. Go to download page
2. Click Accept checkbox
3. Click Download button
4. Confirm download

What someone would automate with Selenium 1.x was:

1. Go to download page
2. Click Download button
3. Confirm download

The idea is that it saves a step. One less step means quicker to code, runs quicker, one less thing to maintain. You do this a thousand times and it adds up. HOWEVER, the customer would never click on the invisible Download button.

In Selenium 2.x you would get a failure with the shortened test case. People occasionally complain that Selenium 2.x has removed an important feature. They want to know how they can click on the invisible Download button. They come up with these tricky Javascript snippets which will allow Selenium 2.x to 'see' and click the Download button. Is a customer going to create a Javascript snippet, inject it into the page, run it just so they can click the Download button? Is a manually tester going to do this? If the answer is no, then why is our automation doing this? If the manual test case calls for clicking the Accept checkbox then our automation should as well. If clicking the Accept checkbox does not enable the Download button, file a bug and move on to something else.

Finally, automation is all about finding elements on a page, interacting with them (clicking, right clicking, typing, etc.) and checking what happened. As a manual tester you are going to use your eyes and hands to do everything. The test case might have a step like, "Locate the folder called 'My Documents' and double click it." This is really two steps. The automation should locate the folder called 'My Documents', this is step 1. It should double click the element, this is step 2. As a manual tester I find the element by looking for the text 'My Documents'. If this is a web page and the HTML is:

<div id='lsk499s'><a href="...">My Documents</a></div>

I am not going to use the div id to find the element. I'm going to use the text. As a manual tester I used the text to find the element. There is no reason to do anything differently with the automation tool.

What if the web page is made to look like Window Explorer. On the left is a tree view with one of the nodes being 'My Documents' and on the right is a thumbnail view with a 'My Documents' folder. In the manual test case, does it specify which 'My Documents' to double click? If yes, follow the test case. If no, how would you as a tester decide? Do you always go with the thumbnail view? Do you pick randomly? Do you change ever test run? If you are a good manual tester, we want that experience captured by the automation. If I would normally change every test run but I never test the same feature twice in one day, it might be sufficient to say, if the day of the year is even, double click thumbnail else double click tree view. If the automation gets run daily, it will pick a different way each day.

The important thing in all this is that I am a good tester. I write good test plans and test cases. When I decide to automation my good test cases, I should not compromise the quality of my testing just because I am testing it with an automation tool rather than manually.

Happy testing!

.

Using the right tool for the job

From time to time I see people asking questions about how to use an automation tool to do something the tool was never meant to do. For example, how do I use Selenium to get the web page for a site without loading the javascript or CSS?

Selenium is designed to simulate a user browsing a website. When I open a web page with a browser, the website sends me javascript and CSS files. The browser just naturally processes those. If I don't want that, I shouldn't use a browser. If I am not using a browser, why would I use Selenium to send the HTTP request?

That is all the get() method in Selenium does. It opens a connection to the website and sends an HTTP request using the web browser. The website sends back an HTTP response and the browser processes it.

If all I want to do is send the request and get the response back, unprocessed. I don't need a web browser.

So how can I send an HTTP request and get the HTTP response back? There are a number of tools to do this.

Fiddler2: http://www.fiddler2.com/fiddler2/

The way Fiddler works is you add a proxy to your web browser (actually Fiddler does it automatically). Now when you use the web browser, if Fiddler is running, the web browser sends the HTTP request to Fiddler and Fiddler records the request and passes it on to the intended website. The website sends the response back to Fiddler and Fiddler passes it back to the web browser.

You can save the request/response pair and play them back. Before you play the request back you can get it. You can edit the website address, you can edit the context root of the URL and if there is POST data you can get the data as well.

Charles: http://www.charlesproxy.com/

Charles is much like Fiddler2 but there are two main differences. The first is that Charles is not free. You can get an evaluation copy of Charles but ultimately, you need to pay for it. So why would you use Charles? With purchase comes support. If there are things not working (SSL decrypting for example) you can get help with that. Additionally, Fiddler is only available on Windows. Charles works on Mac OS X and Linux as well.

curl: http://curl.haxx.se/

Fiddler and Charles are GUI applications with menus and dialogs. They are intended for interacting with humans. If you are more of a script writer or want something you can add to an automated test, you want something you can run from the command line. That would be curl. Because it is lightweight and command line driven, I can run curl commands over and over again. I can even use it crude for load testing.

The most common place to find curl is checking the contents of a web page or that a website is up and running. There are many command line options (-d to pass POST data, -k to ignore certificate errors, etc.) but the general use is curl -o output.txt http://your.website.com/some/context/root. This will send the HTTP request for /some/context/root to the website your.website.com. A more real example would be:

curl -o output.txt http://www.google.ca/search?q=curl

I could then use another command line tool to parse the output.txt file. Or I could use piping to pipe the output to another program.

wget: http://www.gnu.org/software/wget/

Another nice command line tool is wget. The wget command, like curl, will let you send an HTTP request. The nice thing about wget is that you can use it to crawl an entire website. One of my favourite wget commands is:

wget -t 1 -nc -S --ignore-case -x -r -l 999 -k -p http://your.website.com

The -t sets the number of tries. I always figure if they don't send it to me on the first try they probably won't send it to me ever. The -nc is for 'no clobber'. If there are two files sent with the same name, it will write the first file using the full name and the second file with a .1 on the end. You might wonder, how could it have the same file twice in the same directory? The answer is UNIX versus Windows. On a UNIX system there might be index.html and INDEX.html. To UNIX these are different files but downloading it to Windows I need to treat these as the same file. The -S prints the server reponse header to stderr. It doesn't get saved to the files but lets me see that things are still going and something is being sent back. The --ignore-case option is because Windows ignores case so we should as well. The -x option forces the creation of directories. This will create a directory structure similar to the original website. This is important because two different directories on the server might have the same file name and we want to preserve that. The -r option is for recursive. Keep going down into subdirectories. The -l option is for the number of levels to recurse. If you don't specify it, the default is 5. The -k option is for converting links. If there are links in the pages being downloaded, they get converted. Relative links like src="../../index.html" will be fine. But if they hard coded something like src="http://your.website.com/foo.html" we want to convert this to a file:// rather than go back to the original website. Finally, the -p option says to get entire pages. If the HTML page we retrieve needs other things like CSS files, javascript, images, etc. then the -p option will retrieve them as well.

These are just some of the tools I use when Selenium is not the right tool for the job.

How to do focused testing

After years of software testing I have found one constant... there is never enough time to test everything.

So how do I pick what I'm going to test? Here are a few tricks I use to focus what I'm going to test.

First, test what changed. I like to get a snapshot of the last release (source code) and a snapshot of the current release candidate then compare them. Often the source code is readable enough for any tester to understand what has changed. If you are not a programmer, not familiar with the particular language or the code requires a little indepth knowledge only the programmers have then you might want to talk with some of the developers about what you are looking at.

For example, if I am testing a shopping cart for an online web store and I see changes to a class called TaxRules then I'm going to look at the requirements for tax rules and see if there is anything new. Even if there aren't requirements for tax rules, I can talk to the person who wanted this feature (or improvement) and I can look at the code. Comments will be for humans and I'm human (no really, I am). Good code should also be self documenting. In other words, the variables and method names should almost read like English.

The next trick is to get someone to add a code coverage tool to the web site (or whatever it is you are testing). As you test the application (automated or manually) the coverage tool will record which parts of the code you executed. Look to see of the code which has changed was executed. Set the coverage tool so it records actions within the methods and not just method level data. This way you can see which branch conditions were called. If there is a loop which never gets entered or a portion of an if/elsif/else which does not executed, you know you need another test case. You could even ask the developer to relate the statements missed to user actions. Essentially, get them to tell you what test case you missed. :)

Finally, look at the inputs to the methods. Can you see a way to affect the inputs? Does the method handle situations where the data is bad? Developers think in terms of "make the program do what it is supposed to do." They always make it do the right thing when given the right data. As a tester you need to think in terms of "make sure the program doesn't do what it is not supposed to do." In other words, if you input bad data, it should report a nice, helpful message to the user or just restrict the user's ability to input the bad data.

As you do code coverage, you can keep notes with your test cases. Just a comment about how test case XXX should be run with class YYY is altered.

Tools to determine how to use Selenium

Being able to view the source code to a page is helpful when trying to use Selenium. Many of the methods in Selenium require you to locate the element via something related to the source code. For example, if I wanted to click on an element I need to look at the source code. From the user's point of view they might see:

Click Me

But the underlying source code could be:

<span>

    <div>

        <table>

            <tbody>

                <tr><td><strong onclick="javascript:clickme();">Click Me</strong></td></tr>

            </tbody>

        </table>

    </div>

<span>

I might assume there is an anchor and I want to find the <a> element with the text 'Click Me'.  By looking at the source code I can see that I actually want to find a <strong> element and click on it.

Just viewing the source code isn't always sufficient however. AJAX and Web 2.0 means that the source code gets changed in memory after it is loaded. So what is actually available in the DOM (and Selenium) might be different than what you see when viewing the source code.

So how do you see what Selenium sees? A simple solution is to write your Selenium code to get to a point in the program then print out the source code using Selenium. Selenium 1.0 and 2.0 both have methods to return the HTML source code as a string, which you can print.

The problem with this is that you might have to write a program to print the source. Then with the source, find the code you want to click. Then run the program so it clicks the element then prints the new source. Then run the program to click the first element, click the second element and print the source. For n steps this will take O(n^2) run time. Not very efficient.

So how can you view the dynamically changing source? The answer is in browser tools.

If you are using IE7, search for "microsoft dev toolbar". When you install it, it gives you development tools which can view the page in real time. If you are using IE8, the tool is built in. Just press F12 and it will display the development tools.

If you are using Firefox, install the Firebug extension.

If you are using Google Chrome, right click an element and select Inspect to bring up the development tools.

Knowledge sharing

Joining a new company is always difficult at first. You spend a lot of time learning what you don't know about the company and the culture. Here are some ideas to make this normally unproductive time more productive.

Most companies have a wiki or central web site like Sharepoint. Create an employee page with pictures of the employees, there name, contact information (office/location, email, phone, etc.).

Next create a Subject Matter Expert (SME) table. Put the areas of expertise and who is the main and secondary contact. You could even work this from the other direction. As a manager, think about the needs of your department, create a table with one row for each subject your department needs an expert in. In the second column put the name of the person who is the SME for that subject. In the third column put the name of who would be your second choice for SME. Any row which does not have a second name means you need to create a backup for that area. Otherwise, when your one and only SME goes on vacation you could be in trouble. Or worse, they leave the company. Any row which has no SME at all means you need to either get someone to become that SME or you have a case for hiring someone.

Whenever someone new joins the company, take their picture, add an entry to the employee page and a link to the SME table. If they are an SME for a subject you have no one, make them the SME. If they are the SME for a subject you have one SME, make them the backup.

The next step in the process is for the SME to document what they know. Some people will believe documenting what they know will allow the company to get rid of them. The truth is that an SME's practical experience can never really be captured in a document. Most often this knowledge comes into play because the SME has left the company, is away on vacation or ill. If they don't document their knowledge, they will be critical to the company. Going on vacation will be discouraged. They will most likely be put under a great deal of stress. Sooner or later they will fall ill or quit.

The ultimate goal is to create a backup for the SME. If the SME is promoted or put on other activities, the company will require him to be available to help out the backup/replacement. If someone is replacing the SME, the SME will most likely become the backup until a suitable backup can be found. By the time the replacement is fully up to speed and a suitable backup has been found, the SME will probably be an SME for something new or they can at least make sure they are becoming the SME for something new and critical to the company.

Bottom line, if they are trying to phase you out it would be obvious. So don't worry and help your company share that knowledge.

Creating Time

I've started a new job and I'm working with my staff to see how things are done. One of the problems I see in many organizations is 'not enough time'. There never seems to be enough time in the day/week/month to get everything done.

This will always be true because there is always a pressure to get to market before the competition.

However, I have found ways of getting more done compared to someone else in the same time period. The trick is to look for lost moments.

Yesterday I was talking to a staff member while he set up an appliance. The process took 5 minutes of running commands and answering questions and around 15 minutes of the software getting installed. I asked him a question and he immediate paused to answer me. He was literally one key press away from the point the setup no longer required his attention. I stopped him and told him to press ENTER first.

When you look at what he was going to do:

• Enter 99% of the interactive portion of setup (4 minutes, 59 seconds)
• Answer my question (10 minutes)
• Press ENTER to finish the interactive portion of setup (1 second)
• Wait for the batch portion of setup to finish (15 minutes)
• Total running time = 30 minutes

The way I would do it would be:

• Emter 99% of the interactive portion of setup (4 minutes, 59 seconds)
• Ask me to wait for a moment (3 seconds)
• Press ENTER to finish the interactive portion ofsetup (1 second)
• Answer my question (10 minutes)
• Wait for the batch portion of setup to finish (5 minutes)
• Total running time = 20 minutes, 3 seconds

I just saved 9 minutes and 57 seconds. I realized this saving from my class in Microprocessor Design. In this class we looked at what happens when the CPU reads in an opcode. Internal to the CPU there are various registers, arithmetic units (ALU), etc. If an opcode takes 12 cycles to execute, there are 12 steps inside the CPU where data is flowing from one area to another. What a designer will do is see if there are two steps I can do at the same time. If yes, I can reduce a cycle.

With microprocessors, you save a cycle here, a cycle there. It does not look like a lot but if you look at the ratio. I have 1 cycle saved for 12 cycles spent. This means 1/12 or over 8%. In real time, saving 8% means an extra 40 minutes a day. Over the course of a week I have an extra 3 hours and 20 minutes.

This is how you create extra time. I will also do things like using tools like expect to automate a process. Rather than typing in all the prompts to a Bourne shell script I will do the following:

•  Run the script using sh -x <script-name>
• This will output everything which is happening
• Take the output and determine what all the prompts are
• Use a tool like expect to script the response to the Bourne shell script
• Use the expect script and while it is running do something else.

If something like this is only going to save me 5 minutes I might do something like check my email, stretch, etc. These are all things I will do in the day anyways. I might as well do them while I'm waiting for something else to finish.

Important to note however is that if you have 5 or 10 minutes while you are waiting for something, don't switch to a task that requires you to change what you are thinking about. Getting back into the right mindset for the thing you are waiting for could cost you 5 or 10 minutes. In that case there is no real savings.

So think about the times you are waiting for something and what can you do while you are waiting. Thing about what tools you can use to create situations you will be waiting and therefore able to do something else.

One last parting example, if you are using automation tools that take over your computer (they create mouse and keyboard activity), set up a Vmware image and run the tool inside the Vmware. While the automation is running, minimize the Vmware window and do something else on your desktop.

Organizing your automation

It has been a while since I have posted something to the blog. Life has been keeping me busy with things other than software testing.

I've been to a number of interviews in the past few weeks and I've been asked a number of questions. One of those questions was regarding traceability. If I have a number of automated tests and a set of requirements, how do I connect the two and how do I determine what has been run and what has not?

The first part to this is how are the requirements organized. If there is a tracking system and some unique identifier for the requirements, I want to use that identifier in my test automation. For example, a previous job the defect tracking system was used to manage requirements. In addition to categories like 'defect', 'enhancement', etc. there was a 'new feature' category. The business analyst would file a 'new feature' request into the system. Using mock ups and text descriptions, they would flesh out the requirements.

Each defect report has a unique tracking number. I would need to incorporate this number into the test automation. There might be one number for the requirement but a number of different test cases. Additionally, the number might be just a number, e.g. 28364. In the automation environment this might not stand out as a requirement. For this reason I would develop a convention where all requirements references would start with REQ-. Thus REQ-28364 could get added to the test case automation.

Ideally, you want the automation to be self-explanatory. If a test case fails, it would be helpful to know what failed without having to look up the requirements documentation. With automation like Selenium RC or Watij (both using Java) I can name the test case class the same as the requirement number, e.g. REQ-28364 but if I am looking at the test suite or test results it might not be obvious what this is. So I would create an annotation @REQ and put the requirement information in the comments of the source code.

The name of the class can then be used to indicate what the new feature is. The name of each test case would be a description of what is being tested. For example, if I'm adding Print support to an editor I might have the class name "PrintSupport" or "NewFeaturePrintSupport". The test cases might be:

• canPrintAllPages
• canPrintAllEvenPages
• canPrintAllOddPages
• canPrintRangeOfPages
• canCancelPrintDialog

When when I look at the results for the test run I would see:

• PrintSupport->canPrintAllPages: Pass
• PrintSupport->canPrintAllEvenPages: Pass
• PrintSupport->canPrintAllOddPages: Pass
• PrintSupport->canPrintRangeOfPages: Pass
• PrintSupport->canCancelPrintDialog: Fail

Very easy to see what is not working and what is.

The most important thing for tying requirements and automation together is creating a convention and sticking to it. To help you stick to it, edit the templates for the IDE. Whenever I create a new Test Case class, the template will have an @REQ field in the comments at the top of the class. I can even go one step further and have source control check for the @REQ field. If the field does not exist or it is blank, the check in of my automation will fail with an error telling me to add a requirement reference to the source code.

Ways to facilitate communication

One of the most important things on a team is communication. I have worked on teams were everything was put into a document. Unfortunately, as the project progressed, time became more and more precious. People needed to be more efficient and far too often, the documentation suffered. Decisions would be made between two individuals and the rest of the team would miss out.

So how do we make sure everything is well communicated? The simplest answer is to talk to each other.

In scrum you have sprints and planning meetings. What I would do is at the end of the week look at the back log (features and defects in the tracking system) and decide what I could get done be the end of next week. I might add a few notes to the defect report and assign it to myself.

On Monday morning we would have the planning meeting for the week. Everyone would decide what features and bug fixes should and could be completed that week. If there was anything which would take more than one week to complete, it need to be broken down into multiple, smaller tasks. No task should take more than one week. This means implementing and testing the feature/bug fix should happen the same week.

Everyone would pick the defects they were going to work on and give estimates for how long each would take. Every day we would meet and go over the status of the team. Have things changed? Did sales land a new contract but promise a feature we need to complete this week? Or is that feature multiple tasks and we want to start on one of those tasks this week? If yes, we reassess the plan and change priorities.

The key to all of this is that developers, business analysts, testers, project stakeholders all meet and decide what will be done each week and continue to track progress on a daily basis. Most people who do scrum understand this.

What about the time between each scrum? What if you need clarification on the feature you are working on? Whether you are a developer or a tester you might need further clarification. If everyone is in the same room you could just shout to the people involved. This could be disruptive to the other people in the room.

So how do you deal with it? We could create teams and partition the rooms. Each room would have a tester, a developer and a business analyst. What about the project manager? There will usually be one project manager and multiple developers. We cannot have one manager to each developer. Additionally, I have never worked on a team were the developer to tester ratio was 1:1. So this idea is not going to work.

What has worked for me in the past is everyone is on MSN Messenger or Google Talk. These tools let you talk with one or multiple people at once. You can set up aliases for the different groups you need to talk with. This is really helpful if you are working in an open concept office or a distributed team. In the case of an open concept office, shouting to all interested parties might work for you but it is disruptive to the people in the room who are not involved in the conversation. By using chat software I can ask quick questions of just the people who need to be involved. The same is true for people at remote locations. I can chat with everyone involved without having to book a conference room and have everyone call in.

An added benefit is the ability to opt out of a conversation. If I am working on something which requires my full concentration I can log out of chat. This send a signal to everyone that I cannot be disturbed. The danger of this is forgetting to log back into chat but hopefully, you will just get into the habit of remembering. Everyone checks email after they have been working hard on a feature which required their full attention. If someone is not logging back into chat and you need to talk with them, send them an email. When they are done it will be a gentle reminder to log back into chat.

If you find that chat is becoming too tedious and requires too much typing, switch to a meeting. Book a conference room and invite everyone to a 30 minute or 1 hour meeting. If the team is distributed you can use Skype or VOIP to get everyone calling in.

Selenium RC [Java] waitForCondition example

I've seen a few people post questions about using the Selenium waitForCondition method.

If you read the documentation for the waitForCondition method (in either Selenium or DefaultSelenium) you will see a mention of selenium.browserbot.getCurrentWindow(). This is key to the wait for condition.

When you start a test case there are two windows which open up. The first window has the Selenium Command History and ability to display the log. I call this the Selenium window. The next window which opens is the window for the application under test or AUT window.

The waitForCondition method has two parameters. The second parameter is the easiest to understand. We don't want the test to wait for ever. If we don't get a response by a certain time we can assume the test failed. The second parameter is the timeout period in milliseconds. So if the condition should occur within 20 seconds, the second parameter is "20000".

The second parameter is the key to the waiting. It is the condition we are waiting for. There are two windows open, Selenium and AUT. The first parameter to waitForCondition is a javascript expression. It will normally be tested on the Selenium window. If you want to evaluate some javascript in the AUT window you need to use: selenium.browserbot.getCurrentWindow(). For example, Let's say this is the AUT window:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Insert title here</title>
</head>
<body>
<script type="text/javascript">

// this variable is the key. When it is false the loading is done
var loading=true;

function loadingDone() {

// hidden the GIF
 document.getElementById("loading").style.visibility="hidden";
 // set the flag to indicate we are done loading

loading=false;
}

// this sets a timer for ten seconds to simulate something

// loading for ten seconds. After ten seconds this will call

// the loadingDone() function.
var seconds=10;
var t=setTimeout("loadingDone()", seconds * 1000);
loading=true;

</script>

<!-- the moment the page loads this will create -->

<!-- a spinning GIF to signal something loading -->
<img id="loading" src="http://www.oshawa.ca/images/loading.gif" style="visibility:visible"/>
</body>
</html>

What this page does is simulate something loading. After 10 seconds it will hide the spinning GIF and set the loading variable to false. So the condition we are waiting for is when loading == false.

This means from waitForCondition we want to look at the variable "selenium.browserbot.getCurrentWindow().loading == false". In other words, the full statement is going to be:

selenium.waitForCondition("selenium.browserbot.getCurrentWindow().loading == false", "20000");

It is important to note that the selenium in selenium.waitForCondition is a Java object in your test code and the selenium in selenium.browserbot is a javascript object running in the web browser.

Let's say we take this HTML and save it to the file /Users/darrell/workspace/LearningSelenium/waitForCondition.html then we can create the following Selenium RC test case:

package com.example.tests;

import org.openqa.selenium.server.RemoteControlConfiguration;
import org.openqa.selenium.server.SeleniumServer;
import com.thoughtworks.selenium.DefaultSelenium;
import com.thoughtworks.selenium.SeleneseTestCase;

public class TestingSelenium extends SeleneseTestCase {

 SeleniumServer ss;
 RemoteControlConfiguration rcc = new RemoteControlConfiguration();

 public void setUp() throws Exception {
  int ssPort = 4444;
  rcc.setPort(ssPort);
  rcc.setTimeoutInSeconds(1200);
  ss = new SeleniumServer(rcc);
  ss.start();
  selenium = new DefaultSelenium("localhost", ssPort, "*safari", "file://");
  selenium.start();
 }

 public void testStub() throws Exception {
  long start, stop;
  selenium.open("/Users/darrell/workspace/LearningSelenium/waitForCondition.html");
  start = System.currentTimeMillis();
  selenium.waitForCondition("selenium.browserbot.getCurrentWindow().loading == false", "300000");
  stop = System.currentTimeMillis();
  System.err.println("Elapsed time: " + ((stop - start) / 1000.0) + " seconds");
 }

 public void tearDown() throws Exception {
  super.tearDown();
  ss.stop();
 }
}

and that is one example of waitForCondition. You will have to know what condition to wait for in your application. If you are using AJAX you might find there is an AJAX object on your application. In it will be a activeRequestCount. When the AJAX.activeRequestCount goes to zero, all the AJAX calls are done. So you can wait for an AJAX call to complete with:



selenium.waitForCondition("selenium.browserbot.getCurrentWindow().AJAX.activeRequestCount == 0", "30000");