Google Analytics


To search for specific articles you can use advanced Google features. Go to and enter "" before your search terms, e.g. CSS selectors

will search for "CSS selectors" but only on my site.

Sunday, February 28, 2010

Measuring code complexity as a way to determine number of test cases

There are tools which will measure code complexity (cyclomatic complexity) for a software project. Cyclomatic complexity is the number of linearly independent paths through a program's source code. For example, given the following code:
if(expr1) {
    // option1
} else if(expr2) {
    // option2
} else {
    // option3

if(expr1) {
    // option4
} else {
    // option5
Assuming expr1 != expr2, the following paths through the code exist:
  1. option1, option4 (expr1 == true)
  2. option2, option5 (expr2 == true)
  3. option3, option5 (expr1 == false, expr2 == false)
If you look carefully at this list, when testing this snippet of code, for full path coverage, I want a set of tests which will cause each line of code to be executed at least once. I want each if/elseif/else to get evaluated and the body of each to get executed. To make this happen I need:
  1. expr1 == true, which implies expr2 == false
  2. expr2 == true, which implies expr1 == false
  3. expr1 == false and expr2 == false
The number of paths through the code is the number of test cases! If you are looking at branch coverage the number of paths required for branch coverage is going to be similar. You can give an example where the branch coverage requires less than the code complexity. For example:
if(expr1) {
    // option1
} else {
    // option2

if(expr2) {
    // option3
} else {
    // option4
For this example, I need the following:

  1. expr1 == true, expr2 == true
  2. expr1 == false, expr2 == false
or I could use:
  1. expr1 == true, expr2 == false
  2. expr1 == false, expr2 == true
Both examples will cause all branches to be taken at least once. Generally speaking the rule is:
branch coverage <= code complexity <= path coverage
So if you have the code complexity, you know the number of paths through the program has to be at least the code complexity of the application. This also assumes that all paths are reachable. A code complexity tool should also tell you if certain paths are not reachable. Actually, many development environments will warn you if a path is not reachable.

So next time you are asked to estimate how many test cases you will require to adequately test an application, use a code complexity tool to give you a rough idea.

Additionally, the complex of the code can give you some idea of the number of defects which should be found. The more complex the code, the greater the number of defects should be suspected. This is not always true but can be a good rule of thumb.

See an article from Enerjy for an example of this phenomenon.

Tuesday, February 23, 2010

Are most binary searches broken?

I was poking around the web and found an article by Joshua Bloch (you can see it here) about a software bug in the Java binary search back in 2006. Joshua found a number of implementations of binary search with the bug.

With binary search there is a line where you take the low point plus the high point then divide the sum by two. In code this would be:
int mid = (low + high) / 2;
Can you see the bug? There is no check for overflow. If low + high is greater than the maximum value for an int it will wrap around and become negative. I used to teach this concept as an odometer on a car or a clock.

Imagine you have a clock:

A regular clock has 60 increments (seconds). With something like int in Java the clock would have 4 billion increments. With this clock our number system goes from -5 to +6. Here is how it works, addition is clockwise and subtraction is counterclockwise. So if I have 2 + 1, I start at the number 2 and move 1 increment clockwise. This lands me at 3.

If I have 1 + 5, I start at 1 on the clock and move 5 increments clockwise to land at 6.

For subtraction, let's try 5 - 3. We start at 5 and move counterclockwise (subtraction) 3 increments. We land at 2. If you take something like 4 - 6 you get, start at 4 and move counterclockwise for 6 increments. This lands you at -2.

Now what happens if I take 4 + 5? The rule is start at 4 and go 5 increments clockwise. We land at -3. So in my  number system 4 + 5 = -3. This is obviously wrong and this is what can happen with the binary search. If low + high is greater than Integer.MAX_VALUE (2^31-1) the value will become negative. A negative number divided by two is still a negative number.

If the size of the array is greater than 2^30-1 then binary search will fail.

The funny thing is, I was teaching about the dangers of integer overflow ten years before Joshua Bloch found this bug and I never suspected something like this in modern programming language libraries.

The text for the course which tests about integer overflow does not deal with things like binary search. By the time my university teaches integer overflow we aren't dealing with anything as simple as binary search. Binary search is taught in first year, first course. Makes me wonder how many other places have fallen prey to the integer overflow bug.

By the way, one fix for the binary search algorithm is:
int mid = low + (high - low) / 2;
Since high is always greater than or equal to low, (high - low) will always result in a positive number or zero.

Sunday, February 21, 2010

How to reduce code duplication

In a previous blog entry I talked about starting and ending test cases in the same place. One of the problems with this is that there will be duplication in test cases. For example, I might have the following two test cases:

- Log into web site
- Find a product to purchase
- Add to shopping cart

- Log into web site
- Find a product to purchase
- Add to shopping cart
- Check out

If the way you log in, find a product or add to shopping cart changes there will be two locations to update. If 99% of the test cases require you to log in, there could be hundreds if not thousands of instances. A change in the log in process could take days to update.

The solution is to create libraries. You can organize them by feature or by page. I typically organize them by page. You could write detail libraries that do atomic actions (click button, fill in form, go to page, etc.) and other libraries which do higher level actions. The higher level actions could be calls to a set of the atomic actions.

Your test case would then be much shorter and simpler to write. As you build up the library of actions, writing test cases will be faster and faster. In other words, the time spend up front will save you time in the long run.

The key to doing this well is organization. You need to look at how other, successful, libraries are organized and try to make your libraries reflect the same consistent structure. Use descriptive names. Both for the grouping (classes in Java, modules in Perl, etc.) and for the method calls.

A test case should almost read like your native tongue and the high level methods should read almost like your native tongue as well. The atomic methods should be simple and straight forward.

Finally, all atomic methods should make sure the input parameters are valid (fail if they are not with a good error message), the state of the web page is what you expected (again fail if it is not) and you get the results you expect at the end of the method. That is, from Computer Science 101, make sure the pre and post conditions are met!

A good error message does not assume what went wrong. Instead it will tell you the state of the test case and let a human determine what was wrong.

Using source control within Eclipse

In my last blog entry I talked about setting up a Subversion (SVN) repository, getting the subclipse plugin for Eclipse and adding a project to source control.

So why do we need source control? There are many reasons to use source control. Here are some of them:

  1. It creates a backup of your work
  2. It allows you to undo changes you made to the project
  3. It lets you work as a team with others on the same project
  4. You can create branches of the project for different reasons
The first point is pretty obvious. If something happened to your working copy of the project you still have a backup of the project in source control. If the repository is on a remote machine it is even better. If the repository is on the same machine but a different hard drive, that is a little better. Obviously not a huge advantage of having the repository on the same hard drive as the working copy.

The ability to undo changes is huge. Let's say you have a project using Selenium and it looks fantastic. You have 80% of the application automated. Then you add one more feature. You run the automation and there are failures all over. You check one of them and find there is nothing wrong with the application under test. It was a change you made to the automation. You have been working on the new automation for 3 hours.

Let's say you didn't have source control. You realize you probably should have tested some of the changes rather than working solid for 3 hours. It is too late to fix that now. You try to remember what you changed in the last few hours. Was it something you changed at the beginning? Was it something you just changed? You have no idea. You step through the automation and try to figure out what is wrong. It has been a long day and you are tired. You try changing a few things, you believe, back to the way they were. After another hour of changing things and just making it worse you decide to go to sleep and look at it tomorrow.

The next day you have a look at the code and immediately realize you messed up one of the locators. You thought the automation was failing at step 37 but there was actually a failure in the setup. You fix the locator and check the setup. It is working fine now. What about the hour of changes you made yesterday? After around 3 hours of work you think you got it back to were it was yesterday.

One minor mistake, 1 hour franticly trying to fix it, 3 hours undoing the frantic work you did yesterday. Total time wasted 4 hours. Maybe things aren't this bad in a situation you might encounter but I can guarantee you will waste an hour or more with situations like this.

Now let's look at the same situation with source control. You realize you should have tested your automation a few times in the last few hours. No problem. You have a few options at this point. You can compare your working copy with the last thing you checked in. In Eclipse,
  1. Right click on the project
  2. Go down to Compare With and select Latest from Repository
It will give you a window with a tree view in the top. The tree will contain a list of all the files which have changed. If you select one of the files, by double clicking it, it will open a compare view. On the left will be your working copy with the changes and on the right will be the last thing you checked into the repository. Using this tool you can review all your changes and see if one of them caused the failure. Hopefully you spot the change in the setup. Worst case, set a breakpoint at all the places you made a change, run the automation in debug mode and confirm everything is good at each change. You should definitely see that the setup is broken.

Typically, you don't want to check in broken code but if you are on a private branch (more about this later) or working alone it is okay. You could check in the code, compare the last checked in with the previous version. You'll see the same comparison tree but this time you can undo all the changes and bring them back one at a time. The whole procedure would be:
  1. Check in broken code
  2. Check out the previous, working version
  3. Compare this with the checked in, broken version
  4. Re-introduce one change at a time
  5. Run the automation and confirm the change didn't break anything
  6. Keep repeating steps 4 and 5 until you find the change that break the automation
The third advantage of source control is working as a team. If the repository is in a location two or more people can access it, each person can check out a working copy and make changes. This is one of the biggest advantages of source control. Without source control, if two people wanted to make a change on one file they'd have to one at a time. Imagine we have version 1 of a file. I get a working copy, you get a working copy. I make a change to line 17. You make a change to line 43. Without source control, I save the file back to the shared folder. You save the file back to the shared folder. My change to line 17 is lost. Not good.

With source control, it is smart enough to merge my changes and your changes. If for some reason it could not figure out how to merge the changes, the last person to attempt checking in would be told there was a conflict. To fix a conflict in SVN, you update your working copy to the latest from the repository. It will keep your change and grab a copy of the conflicting change. You then compare the changes, fix them by hand and check the file back in. At this point there should be no conflict.

The last advantage I mentioned was creating branches. A branch is a way of creating copies of the entire project. An example of a branch would be version v1.0 and trunk. When I start my project, everything is put in trunk (some source control software calls this head or main). At first things aren't very stable. I would never give a copy of trunk to a customer. At some point I get trunk stable and running well. If I add a new feature, trunk will become unstable again. What I can do is make a branch. Now the repository has trunk and v1.0. Any changes I make on v1.0 will not affect trunk and vice versa. Now I can add the new feature to trunk and destabilize it. At the same time, I can do safe, simple bug fixes on v1.0. I then ship the build from v1.0 to the customer (automation for v1.0 is also updated to match the product I shipped to the customer and nothing more). 

A few months later the customer calls up and says he found a bug. Development has made a LOT of changes to trunk by now. Test automation has been updated to match the changes in trunk. If you tried to run the test automation on v1.0 it would fail to run properly. We don't want to give a copy of trunk to the customer. Partly because it is not stable but also because it has new features the customer hasn't paid for.

So a developer will check out a copy of v1.0, fix the bug and build it as version v1.0.1. While the developer is working on the bug fix, the test automator will check out a copy of the automation from branch v1.0, update the automation to catch the bug, test it to make sure it does find the bug in v1.0 of the product. The tester will now run their automation against version v1.0.1 and confirm the fix. At this point the source control administrator will tag the code (development and automation) for v1.0.1.

Hold on, we missed something. The bug still exists in trunk. Whenever you work on a branch, you have to be sure to merge bug fixes from the branch down into trunk. The change to test automation has to be merged down into trunk as well. Sometimes it is a simple procedure but in some cases, the code in trunk has changed so much they (a) the bug no longer exists or (b) the fix is totally different. In case (b) the developer will have to devise a new fix and the test automator needs to change the test to find the bug in trunk.

There are just some of the things to know about using source control. I'll just leave you with a few recommendations:
  1. Test your code often. Don't wait 3 hours.
  2. Only check in code that works.
  3. After you test code and confirm it works, check it in.
  4. Put good comments so you know what you checked in and why.
Good luck and happy coding.

Saturday, February 20, 2010

Setting up source control at home

If you are learning how to do development (or test automation) you need source control. I first spoke about this in So you want to create automated tests. Automated testing is software development and part of good software development is source control.

There are a number of open source projects you can use. Originally I would have been suggesting Concurrent Versions System (CVS). When you install something like Eclipse you will see it has the CVS plugin already installed. CVS is well over 20 years old and quite stable. However, Subversion (SVN) is another choice and one I'm going to talk about today.

SVN has been around for over 10 years. As of 3 days ago Subversion become Apache Subversion. The old official site was here. Now the official site is here. If you go to the old site you will find it directs you to the new site.

First thing we need to do is go to the new official site and download a copy of the binaries. SVN is supported on a wide variety of platforms including Linux, Mac OS X and Windows. You will need to download a server/client package. The SVN server will be installed on the machine hosting your files and the SVN client will be installed on the machine you want to do your development on (probably the same computer but you could work with a friend and have them host the server or put the server on a school computer).

Once you have downloaded and installed SVN you should be able to go to a command prompt and enter:
svn help
On a Mac OS X computer, go to Spotlight (upper right corner) and enter Terminal then select Terminal the application to get a command prompt. On a Windows computer, go to the Start->Run menu and enter cmd to get a command prompt.

Now on to creating your SVN repository. But first, a little bit about source control or Software Configuration Management (SCM). Files will exist in at least two locations. There is a repository and there is a working copy. The idea is that you create the files and copy the files to the repository (check in). You can then delete the files you created (working copy) because you have them backed up in the repository. You can make multiple copies of the files (check out) or you and a friend can each have your own copy of the files. First thing we need is a place to store the files or the repository.

We have to make a few decisions before we set up the repository. At some point you will probably have multiple projects. Do you want one repository for each project? Or do you want all the projects in one repository?

At first it will not matter but as you start using advance features it might. For example, at companies they might have a policy that no code change can be checked in to SVN unless you add a comment with the defect tracking system number. Thus to add a new feature to the project a developer must create an entry in the defect tracking system, create the feature and refer to the defect tracking number when they check in the work. SVN has triggers which allow us to remind the developer to do this. How they work is, when a developer tries to check in the code, SVN will scan the comments for a defect tracking number. If there is no number it will reject the check in and ask the developer to include the number in the comments. Do you want this for all the projects you set up? Or only some? If you have one repository, rules like this apply to all projects. If you have multiple repositories then each can have its own set of rules.

For now, lets assume you are just working on one project but leave room for more projects in the same repository. If you change your mind, you can create another repository later.

Select a computer to host the repository (localhost probably) and a location on the file system (on Mac OS X, Linux, UNIX you can use maybe /var/svn/repos; on Windows you can use C:\var\svn\repos). I'll refer to this location at the SVN repository root.

When selecting a computer to host the files on you have to consider things like who will be accessing it and how. If you were setting up a repository for a company and had to deal with firewalls, proxies, user security, etc. it would be an entire chapter in a book. However, this is just a personal repository for you to start learning at home.

Now that we know where the files are going to live we need to decide if we want to store the files in the file system or a Berkeley database. Using a Berkeley DB is more reliable for day to day operations but more work to setup and administer. There have been some bugs relating to using the file system but for our purposes I think it will be fine. Additionally, using the file system is the default since SVN 1.2. So the file system it is.

So, assuming the repository is going into /svn/repositories, lets create a repository for Selenium testing and we'll call it the SELTEST project. From the command prompt:
svnadmin create /var/svn/repos/SELTEST
And that is it. If you look in the directory /var/svn/repos/SELTEST you should find something like:
-rw-r--r-- 1 darrell darrell 229 20 Feb 10:06 README.txt drwxr-xr-x 5 darrell darrell 170 20 Feb 10:06 conf drwxr-xr-x 2 darrell darrell 68 20 Feb 10:06 dav drwxr-sr-x 10 darrell darrell 340 20 Feb 10:06 db -r--r--r-- 1 darrell darrell 2 20 Feb 10:06 format drwxr-xr-x 11 darrell darrell 374 20 Feb 10:06 hooks drwxr-xr-x 4 darrell darrell 136 20 Feb 10:06 locks
but you want to access these files using the svn command. On UNIX/Linux/Mac OS X you can use the file system directory permissions to control access. Create groups and use the chmod, chgrp commands.

At this point you are ready to start using SVN. If at a later date you decide you want things setup differently, no problem. There are instructions on migrating, updating, backing up, renaming, etc. your SVN repositories. For now, let's start checking in some code.

Let's assume we are using Eclipse and want to access our SVN repository. Eclipse comes with a plugin for CVS but you'll have to add one for SVN. I recommend using subclipse. There are other SVN client plugins for Eclipse but I have been extremely happy with subclipse. Assuming you already have Eclipse installed and running:

  • Go to the subclipse web site
  • Go to the download and install section
  • Find the update link for your version of Eclipse
  • Copy the link and go to Eclipse
  • Select the Install Software... menu option (usually in the Help menu)
  • When this dialog open you can add the subclipse link
  • There should be 3 parts to subclipse
  • Select all of them and click Next
  • Follow the wizard and accept the license agreement
  • You may get a warning about unsigned content
  • I accept the installation anyway
  • Once it is done you'll have to restart Eclipse
  • Now you can switch to the SVN perspective
  • Go to the Window menu and open a perspective
  • You'll have to select Other... and find the SVN Repository Exploring perspective
  • In this perspective there will be a SVN Repositories tab (upper left corner)
  • Right click in there and select New->Repository Location...
Now assuming the files are on the local file system, the SVN URL will start with file: but if you are accessing the files from a remote computer you'll need to use ssh. We assume that ssh is set up on the remote computer and you have a login. For a remote computer you want svn+ssh: protocol.

Let's assume the files are located, the repository root is at /u/dgrainge/svn/repos/SELTEST and I can log in with dgrainge. The URL is going to be:

When I enter this into Eclipse it will ask me for my username and password. These are the username and password I use when I ssh to

If I am going to be hosting the repository locally in /var/svn/repos/SELTEST then I would use:
Because it is local there is no need for a username or password.

Once the repository has been added to Eclipse I can switch back to the Java perspective. Assuming I already have a Java project I want to add to SVN:

  • Right click on the Project Name in Package Explorer
  • Go down to Team and select Share Project
  • Select SVN for the type of repository to use then click Next
  • Select the existing repository we just created and click Next
  • Click Next to accept the default URL
  • Click Finish to start copying the project to the repository
  • It will ask you for an 'author' name
  • This just identified that you want to check the project into SVN
  • Now we have to pick which files to commit (copy)
We do not want to add things like .project, .classpath or .setting. In the Synchronize view:
  • Right click on .project and select Add to svn:ignore
  • Right click on .classpath and select Add to svn:ignore
  • Right click on the root of the project and select Commit
  • A list of files to copy to SVN will be listed
  • If any of the files are generated (.class files) you don't want to add them
  • Any log files or output from running the project should not be added
  • Uncheck everything you don't want to add to the repository
  • Put the comment in "Initial check in"
  • Click OK
If you check in too much you can always remove it later. If you fail to check something in then delete the local copy, it is gone forever.

Next blog entry will be about using the repository and all the benefits it gives you.

Thursday, February 18, 2010


Joined Twitter today. Twitter: darrellgrainger. We'll see how that goes. Trying out three Twitter apps for my BlackBerry. One seems very powerful (UberTwitter) but is a little slow. One seems pretty plain and simple (bbtweet). One seems somewhere in the middle (OpenBeak). Not really sure if anyone wants to hear me babble but I'll give it a try for a bit and see what happens.

Changing ports in JBoss

If you are running JBoss and find some of the ports it requires (and it requires a few) are in use or if you wish to run multiple copies of JBoss, there is a quick and easy way to change the ports.

There are three sets of ports defined for JBoss. They are referred to as 'ports-default', 'ports-01' and 'ports-02'. The ports-default are the ones we all know and love, e.g. 8080 for the HTTP, 1098 for RMI, etc. The ports-01 are the same numbers but add 100. So the HTTP port becomes 8180, 1198 for RMI, etc. The ports-02 are the same numbers but add 200 to the default ports. So HTTP becomes 8280, 1298 for RMI, etc.

If you go to the JBOSS_HOME (the directory you unzipped the JBoss files into) you will find a server/ directory. In the server/ directory is usually an all/, default/ and minimal/ directory. This works for all of them but lets look at the default/ directory. In the default/ directory is a conf/ directory. So the directory we are looking in would be $JBOSS_HOME/server/default/conf/ (or %JBOSS_HOME%\server\default\conf\ on Windows).

In this directory is a jboss-service.xml file. Open this file, with a text editor, and search for Service Binding. When you find it you will see something like:
<!-- ==================================================================== -->
   <!-- Service Binding                                                      -->
   <!-- ==================================================================== -->

   <!-- Automatically activated when generatting the clustering environment -->

      | Binding service manager for port/host mapping. This is a sample
      | config that demonstrates a JBoss instances with a server name 'ports-01'
      | loading its bindings from an XML file using the ServicesStoreFactory
      | implementation returned by the XMLServicesStoreFactory.
      | ServerName: The unique name assigned to a JBoss server instance for
      | lookup purposes. This allows a single ServicesStore to handle mulitiple
      | JBoss servers.
      | StoreURL: The URL string passed to
      | during initialization that specifies how to connect to the bindings store.
      | StoreFactory: The interface
      | implementation to create to obtain the ServicesStore instance.

   <mbean code=""
     <attribute name="ServerName">ports-01</attribute>
     <attribute name="StoreURL">${jboss.home.url}/docs/examples/binding-manager/sample-bindings.xml</attribute>
     <attribute name="StoreFactoryClassName">

You will notice at the bottom is -->. This means the <mbean> tag is inside a comment block. If you move the --> to above the start of the <mbean> tag, so you have:
<!-- ==================================================================== -->
   <!-- Service Binding                                                      -->
   <!-- ==================================================================== -->

   <!-- Automatically activated when generatting the clustering environment -->

      | Binding service manager for port/host mapping. This is a sample
      | config that demonstrates a JBoss instances with a server name 'ports-01'
      | loading its bindings from an XML file using the ServicesStoreFactory
      | implementation returned by the XMLServicesStoreFactory.
      | ServerName: The unique name assigned to a JBoss server instance for
      | lookup purposes. This allows a single ServicesStore to handle mulitiple
      | JBoss servers.
      | StoreURL: The URL string passed to
      | during initialization that specifies how to connect to the bindings store.
      | StoreFactory: The interface
      | implementation to create to obtain the ServicesStore instance.

   <mbean code=""
     <attribute name="ServerName">ports-01</attribute>
     <attribute name="StoreURL">${jboss.home.url}/docs/examples/binding-manager/sample-bindings.xml</attribute>
     <attribute name="StoreFactoryClassName">

The <mbean> will no longer be commented out. Notice the ports-01 in the <mbean>. This will cause JBoss to start up with the different ports. You can also change this to ports-02 and run a third copy of JBoss. Good if you want to create a cluster.

Personally, I copy the default/ directory and save it as 01/ and another copy as 02/. I then edit the jboss-service.xml in 01/ to use ports-01 and the jboss-service.xml in 02/ to use ports-02. So now I have default, 01 and 02 to match with ports-default, ports-01 and ports-02.

Wednesday, February 17, 2010

How to handling timing in Selenium

I see a lot of people struggling with timing issues in Selenium. After you click an element you will need to wait for something. There are four possible outcomes of the click:

  1. It takes you to another page
  2. It creates a pop up
  3. It opens a frame
  4. It updates the DOM
The most basic of outcomes is #1. You have an anchor with an href attribute. You need to wait for the new page to load. For this you need to use:


If clicking the element causes a pop up use:

selenium.waitForPopUp(windowID, timeout);

If clicking the element causes a new frame to appear use:

selenium.waitForFrameToLoad(frameAddress, timeout);

If clicking the element causes a change to the DOM it gets a little harder. The method to call is fairly straight forward:

selenium.waitForCondition(script, timeout);

The script is a piece of javascript typically. Finding the value for script is the hard part. If you need to look for something on the DOM of the application you need to get the 'document' for that window. So the script needs at least selenium.browserbot.getCurrentWindow(). For example, if I click a link and it updates the price of a shopping cart by modifying the DOM, at the start of the function it might set updating = 1; then at the end of the function, when it has finished updating, it will set updating = 0;. So the condition I am wait for is:

String script = "selenium.browserbot.getCurrentWindow().updating == 0";
The hardest part you will have is determining what condition to wait for. If possible, talk to the person developing the code and see what condition they recommend you wait for.

multiple versions of Java for Eclipse

If you are using Eclipse you can use different version of Java to run your applications. When you create a new Java project, one of the settings is to select a JRE. There is also a link to configure the JREs.

First thing you will need is multiple JREs. If you go to you should be able to find different versions of Java for your computer. You are mostly interested in the Java Platform Standard Edition (Java SE) and Java 2 Platform Enterprise Edition (J2EE). Download and install a few. I usually install them in C:\ on Windows so they are easy to find and unaffected by Windows Update.

In Eclipse, if you go to the link to configure JREs you will come to a dialog with a list of JREs (probably only one maybe two entries) and an Add... button. Click the Add... button, select Standard VM, then browse to the home of the JRE you installed. If you installed the JRE into C:\jre142_5 then this is the JRE home you are looking for. If you installed a JDK it is probably something like C:\jdk142_5.

Once you have added all the different JREs and JDKs you have installed, you can use them in new and existing projects.

For a new project, during the setup wizard you can select a project specific JRE from the list of JREs you have identified for the workspace.

For an existing project, right click on the Project name in the Package Explorer. From the menu select Build Path->Configure Build Path... option. In the dialog which appears you should see a list of Libraries for the project. One of the entries will be the JRE or JDK. Select it then click the Edit... button. On the dialog which appears you can select one of the JRE/JDKs you just added to the system.

what goes where with Selenium RC?

When using Selenium RC. You have five machines, conceptually:
  1. The Application Server (and the application you are testing)
  2. The Web Server
  3. The web browser (client)
  4. The Selenium Server
  5. The Selenium Client
The reality is that the web browser and the Selenium server are the same machine. Additionally, there is no reason you cannot be running everything on one machine. Let's say we have unlimited machine and we want to test on a number of different platforms with a variety of different browsers. Here is a theoretical setup:

  1. Application Server
  2. Web Server
  3. Windows XP, Internet Explorer 6, Selenium Server #1
  4. Windows Vista, Internet Explorer 7, Selenium Server #2
  5. Windows 7, Internet Explorer 8, Selenium Server #3
  6. Linux, Firefox 3.5, Selenium Server #4
  7. Mac OS X, Safari 4.0, Selenium Server #5
  8. Build Machine with Test Suite
  9. Database
  10. Subversion repository with application and test suite source code
Here is how communication will flow:
  1. Machine #8 will check out the code from machine #10
  2. Machine #8 will compile the application
  3. Machine #8 will tell machine #1 to stop the application server
  4. Machine #8 will deploy the application to the application server
  5. Machine #8 will make any necessary changes to the database
  6. Machine #8 will tell machine #1 to start the application server
  7. Machine #8 will compile the test suite code
  8. Machine #8 will start the Selenium Server on machines 3, 4, 5, 6 and 7
  9. Machine #8 will run, in parallel, five copies of the test suite, each one configured to point to a different test machine
  10. Assuming the test runner used on step 9 creates test results, machine #8 will save the test results.
  11. If may also send an email to interested parties
During step 9 there will be the following call in the Test Suite:

    Selenium selenium = new DefaultSelenium(seleniumServer, seleniumPort, browser, baseURL);

where seleniumServer/seleniumPort is the IP address and network port of the machine running the test. If we have this parameter configurable, we can configure the parameter to the selenium server on machine 3 then run it, configure the parameter to machine 4 then run it again, and so on.

The browser parameter would be "*iexplore" for machines 3, 4, 5, "*firefox" for machine 6 and "*safari" for machine 7. The baseURL is going to be the URL for machine #2, which redirects HTTP requests to our application on machine #1.

If  your test code is written in Java, everything can be done so long as some version of Java is installed on all the machines. You could even test your application on Firefox and Solaris 10.

Monday, February 15, 2010

Why each test case should start and end at the same state

I have seen a number of test automators struggling with creating a test suite. Their problem is that they are using xUnit style test cases but they want to change the way they were intended to be used.

For JUnit, the execution is:
  • Run Before, open web browser
  • Run Test Case #1
  • Run After, close web browser
  • Run Before, open web browser
  • Run Test Case #2
  • Run After, close web browser
  • Run Before, open web browser
  • Run Test Case #3
  • Run After, close web browser
What test automators want is:
  • Run Before, open web browser
  • Run Test Case #1
  • Run Test Case #2
  • Run Test Case #3
  • Run After, close web browser
The problem is, this is not how things work in JUnit. So they have been using static methods and helper classes to create the web browser. It looks like:
  • Run Before, if browser == null [true], browser = open web browser
  • Run Test Case #1
  • Run After, if last test case [false], close web browser
  • Run Before, if browser == null [false], browser = open web browser
  • Run Test Case #2
  • Run After, if last test case [false], close web browser
    • Run Before, if browser == null [false], browser = open web browser
      • Run Test Case #3, flag last test case
        • Run After, if last test case [true], close web browser

        This is essentially what test automators want because the After/Before calls between test cases do nothing and leave the web browser open.

        Writing test cases which follow each other, i.e. the final state of test case n is the setup state for test case n+1, is a bad idea.

        If you start with something simple and current it is not a bad idea.

        Imagine the test suite growing and growing. A few months from now you have 5000 'test cases'. Everything is going great. Then they make a change to the application. Your test cases start failing. You investigate the first failure which happens to be test case 3476 of 5000. You need to run all the test cases from 1 to 3476 before you can get to the point it APPEARS to fail.

        A few hours later you find out that things have changed in the application and it is a false negative, i.e. the problem is in the test automation and not the application. You fix it and run the test suite again. A few hours later you find out your fix didn't work. So you tweak it and run the test suite again. A few hours later you find the fix still isn't right. You try one more time. While the test suite is running for the third time that day, 5pm hits. Do you go home and check the results in the morning? Do you work late?

        Let's say you work late and find test case 3476 is working again. But wait, test case 3788 fails now. You set a break point just before the failure point and run the test suite again. Even if you can fix the problems in one attempt, if there are multiple test cases needing maintenance you will still take days to just FIX the old test cases. Where do you find time to add new test cases for the next features? They will take just as long if you add them to the end of this chain.

        Another thing to consider is, hopefully, you will get to a point that your test suite takes MANY hours to run. I've had test suites which took over 8 hours to run. What do you do? You could ask the Project Manager for more time. As a Project Manager he is going to look at the problem just like any other problem he deals with. If Tim has a task to do and it is going to take him 12 days to complete but we need it done in 4 days, break the task into 3 sub-tasks and bring in two more people to help Tim. For safety, bring in three more people. With four people (Tim plus the three new people) working on it, each person should take 3 days to do their part. Working in parallel, the whole task will be done in 3 days.

        How does this apply to test automation? If one machine is taking 40 hours to run the test suite but we want it run daily. Out of a 24 hour day we need time for maintenance, building the application, backing up the system, etc. So let's say you have 8 hours to run the test suite. If it takes 40 hours to run the whole thing, 40 / 8 = 5 so get 5 computers and break the test suite into 5 parts. Heck, computers are a lot cheaper than people; get 10 computers and break the test suite into 10 parts. Now it takes 4 hours to run the test suite.

        But how do you break apart the test suite? The way I write a test suite is to have the Before call setup for the test, I run the test case then the After call returns the system to the exact same state as before the test was run. This means all test cases start from the same point and end at the same point. I can run the tests in any order I want. I can run one, some or all the tests without worrying about how one test will affect another test. For me, if I have 5000 tests and I want to run them on 10 machines then I run tests 1 to 499 on machine 1, tests 500 to 999 runs on machine 2, tests 1000 to 1499 runs on machine 3, ..., tests 4500 to 5000 runs on machine 10.

        If I try this and find that machine 3 is taking 7 hours and machines 5 and 9 are taking 3 hours and 2 hours, I can start moving some from machine 3 to machine 9, keep doing this until machine 9 is taking 4 hours and machine 3 is taking 5 hours. Then I can move some from machine 3 to machine 5 until they are both taking 4 hours.

        Which test cases I move to which machine does not matter. It should take me seconds to make the change.

        Other the other hand, if your test cases all depend on the previous test case, you will need to figure out where to make the first break, i.e. test case 1 to what? Once you figure that out, you'll have to figure out how to get the next test case into the correct state before it starts. You will have to do this 10 times and each time it could take you over a day to get things set up. In other words, it could take weeks just to get back to nightly build and test.

        Bottom line, it feels like you are saving time by having the tests run one after the other without closing but in the long run it will cost you so much that you might have to abandon test automation or seriously jeopardize the project.

        It is important to understand that if this happens a project manager has to deal with a lot more than testing, a LOT MORE. He will not want to hear why you need two EXTRA weeks. He will just expect you to find a solution and bring the project in on time and under budget. If fingers start pointing you can be sure that the project manager will throw the test automators 'under the bus'.

        Creating a Selenium Test Case in Java from scratch

        I you want to create a Selenium test case from scratch, i.e. not using the record method listed earlier in my blog, here is how you do it:

        1. I assume you have Eclipse installed
        2. I assume you have downloaded the Selenium RC
          1. The files selenium-server.jar and selenium-java-client-driver.jar will be needed
        3. In Eclipse create a new Java Project
          1. Pick a name for the project, set anything else you think you'll need on the first page
          2. On the next page, go to the Libraries tab and add the two selenium jar files to the project
          3. Finish
        4. Right click on the src folder and create a new JUnit Test Case
          1. Select JUnit 3, pick a package if you want (can be changed later), pick a name, tick setUp and tearDown, click Finish
          2. It should ask you if you want to add the JUnit 3 library to the project. Answer yes.
        5. You should now be looking at the JUnit test case class in the editor
        6. Change the class so it 'extends SeleneseTestCase
        7. You should get a warning on the SeleneseTestCase
        8. Hover over it and you should get the option to import com.thoughtworks.selenium.SeleneseTestCase which you should do
        9. In the setUp() method, the body should be:
          1. super.setUp(url, browser); where url is the URL of you web site being tested and browser is something like "*firefox", "*iehta" or "*safari"
          2. For example, super.setUp("", "*safari");
          3. The setUp method should now look like:

            public void setUp() throws Exception {
                    super.setUp("", "*safari");
        10. In the tearDown() method the body should be:
          1. super.tearDown();
          2. The tearDown method should now look like:

            public void tearDown() throws Exception {
        11. Now we can add a test case. It might look like:
        12. public void testAGoodDescriptionOfWhatWeAreTesting() throws Exception {
        13. To run this, you'll need to start the Selenium Server.
        14. Go to a command line and enter:
        15. java -jar selenium-server.jar
        16. You might need some more command line switches like -firefoxProfileTemplate or -trustAllSSLCertificates. To see help on the server use:
        17. java -jar selenium-server.jar -help
        18. Once you have the server running, in Eclipse you want to run the test case as a JUnit Test
        NOTE: the SeleneseTestCase is a JUnit 3 TestCase. It assumes the names of the methods are fixed and does not use annotations. You have to use setUp, tearDown and all test cases need to start with 'test'. If you want to create the same thing as JUnit 4 you can use:

        import org.junit.After;
        import org.junit.Before;
        import org.junit.Test;
        import com.thoughtworks.selenium.DefaultSelenium;
        import com.thoughtworks.selenium.Selenium;
        public class Whatever {
         Selenium selenium;
         public void setUp() throws Exception {
          selenium = new DefaultSelenium("localhost", 4444, "*firefox", "");
         public void whatever() throws Exception {
         public void tearDown() throws Exception {
        And there is a simple test case create in Eclipse.

        So you want to create automated tests

        Are you doing manual testing and want to break into automated testing? Are you doing something completely different and want to get a job in automated testing?

        One of the problems of getting into a new field is finding someone to give you the time to learn. If you are working on a team of manual testers they often hire someone else to write automated tests. As they write the automated tests there are less and less things you need to test but how does this get you closer to being an automated tester?

        The answer is that as the automated test suite grows they will need more people on that team to help maintain and create automated tests. But what if they decide to hire someone new and let you go? Why wouldn't they do that? The truth of the matter is, if you are a good employee they won't WANT to let you go unless necessary. You need to make it unnecessary. You need to learn to become an automated tester.

        Another scenario is that everyone is manually testing the project and there is absolutely no plans to do test automation. Maybe the manager tried test automation and it was a huge failure. What would happen if you could automate things and it was a big success? How do you think your manager would feel about you? Pretty good I suspect.

        So how do you do it? Set something up at home and start playing. The most important thing is understanding WHY. If you do something and it works but you don't know WHY then when if fails you won't be able to fix it because you never knew why it worked before and doesn't now.

        What will you need?

        1. An automated test language
        2. A development environment
        3. Source control
        4. Something to test
        5. Knowledge of what to test

        I'll cover everything but the last point here. What test cases you need, how to create them is a whole set of blog entries.

        First, you need to know what automated test language you are going to use. You could use SQA Basic... if you had a few thousand dollars. This is the language used for IBM/Rational Test Studio. If you can afford this then you have the development environment as well.

        I suspect you are looking for something FREE. Best thing to do is use open source tools. A good starting point for this is There are hundreds if not thousands of projects here. Some haven't been active for years and some are used by companies today. You can search the job ads to see what tools they are hiring for. If they are hiring for say Selenium then there is a good chance that your company could use Selenium as well.

        Different open source test automation tools will use different languages. It is usually best to pick a language that is already a programming language. For example, Selenium can be used with Java, Ruby, Perl, C#, etc. or Watir is only Ruby and Watij is only Java.

        If you search for "automation" you'll get 720 projects. If you search for "test automation" you'll get everything with test, automation and test automation. It actually gives you thousands of projects. You can also try to browse the projects using the "Find Software" link. From there go to Development, Testing but you'll still get 1782 projects. Finally, use the Filter button to narrow things down. You can keep adding filters. So for me, I'd want Language 'Java', Operating System 'OS X'. This gets me down to 17 projects. From there I'd scan them by hand. You might also want to filter for Development Status. Go for Production/Stable or Mature. You don't want to be dealing with bugs in the test tool while trying to find bugs in your application.

        If you are going into automated testing for a web site I can tell you that Watij, WatiR, Selenium are very popular choices right now.

        Once you have done some research and know which automated test tool you want to learn you'll need a development environment. If you select C# you are probably looking at buying Microsoft Visual Studio however there are cut down, FREE, versions on the Microsoft website. You just need to hunt for them.

        If you select a tool which uses Java, Ruby or Perl you had a wide variety of development environments to select from. I'm currently using Java. I've always like NetBeans but more companies want people with Eclipse knowledge, so I'l using Eclipse.

        A good development environment is essentially an editor, project management and a debugger. A project manager to organize all the files, an editor to create the files and a debugger to maintain the files.

        Next is source control. What is source control? If you have done any programming you have probably done this:

        • Create a small program
        • Add some features to it
        • Find a problem
        • Try to fix it
        • Try something else
        • Try something else
        • Try something else
        • Realize you are missing a semicolon, or quote or something simple
        Now you are trying to remember all the "try something else" things you did so you can get back to the original code before you made it worse. Ideally, you want to get back to before all the 'try something else' then fix the 'something simple'.

        Next time, you write a small program, run it and it works. Let's say the file is called Before you try to add some more features you copy the file. So now you have and a backup called After a few days you have everything from to

        You add another feature and it totally breaks the program. You do a lot of "try something" only to realize it was a simple fix. Now you can throw away, rename to and do the simple fix.

        You have created a very primitive source control system. There are much better server/client systems like CVS or Subversion. The way they work is you create a repository (a folder on your computer somewhere or on another computer). When you create a file, like, you add it to the repository. Now you have a back up in the repository. Every time you make a change you can add the change to the repository. At anytime, you can get a list of all the versions in the repository, you can add notes when you save the file and later look up the notes. The best part is you can tag a point in time. If you create 500 files and complete version 1.0 today, file 1 might be version 173, file 2 might be version 37, file 3 might version 1, file 4 might be version 28838, etc. you 'tag' all the files as VER1.0. Two months from now someone wants to add a patch to version 1.0. You have made changes to many files. You can ask source control for all versions tagged with VER1.0. It will restore the system to just like it is today. This way I can have one project for current and one project for past projects. Current project is new features, past project is bug fixing.

        If you are using a Mac OS X computer you have CVS built in. You can also download Subversion (SVN) and install it. These source control systems are open source so you don't have to pay for them. CVS was very good but SVN is newer and used by more new projects.

        So go out and find Subversion. Download it and create a repository. There are tutorials on it out there so I'll leave that as an exercise for you to do on your own.

        The final thing is something to test. Pick a simple web site and start automating it. If you have access to your test environment from work, you could start testing it right now. Otherwise, start automating something like

        Saturday, February 13, 2010

        Baking & Testing?

        Are there other testers out there who like to bake? I always liked baking even as a small child. I remember trying all kinds of weird experiments as a kid. At the same time I was experimenting with baking I was also playing with a chemistry kit. I added a few things from under the sink and got some really cool results, like a thermic reaction that melted a pyrex test tube. But I digress...

        I was never good at cooking. The whole trying something, add a pinch of this, a dash of that and keep tweaking it until you created the dish you want. Nope, never worked for me. I liked the certainty of 5g salt, 12g baking soda, 7g ginger, 355g all-purpose flour, etc. A pound cake is a pound of sugar, a pound of flour and a pound of butter. There are a few other things in there but the whole concept is that you have equal portions of sugar, flour and butter. Little tricks like creaming the butter with the sugar before adding the flour are good to know.

        Does this have anything to do with being a good tester? I know that certain things in a baking recipe are there to counterbalance other things in the recipe. For example, the amount of flour and the amount of butter matters. Too much or not enough of either will change the baked item. On the other hand, if you have a recipe for a cookie which calls for raisins, you can usually leave the raisins out and it will be fine.

        I will play with things like that. I learn which things are notably acidic or alkaline. Those things are usually there for a reason. In chemistry you have ionic bonding and you have mechanically mixes elements. In baking, if two ingredients (chemical elements) will produce an ionic bond during the baking process you cannot mess with the ratio. Just like NaCl is Sodium (highly explosive oxidizing metal) and Chlorine (a toxic gas). If you have a ratio of 1:1, when bonded they form table salt; totally harmless (unless we consider high blood pressure).

        How does this relate to testing? First thing which pops to mind is writing a defect report. Some details are important and other things can be put in or left out.

        How do I know what is important and what is not? Patterns. After years of baking I start to notice pairs of ingredients which always go together. I start seeing them in terms of a ratio. I realize they are important. During testing, I do analysis on the defects reported (by me and others). I understand why the bug occurred. I start to see patterns. These two things go together.

        I also look for differences. What is different from a pound cake and a victoria sandwich? What difference does it make? During testing, if I try something one way then another does it seem different? If testing a web page and I go to page X then to page Y. Later I go from page Q to page Y. It takes three times longer for page Y to load. Why? Was it because other things happened and there is a memory leak on the server? Or is page Q doing something to make page Y load slower? Look at the POST from page X, look at the POST from page Q. If they look the same then maybe it is the server. Also, when I go back to page X then to page Y. Is it slower now that it was when I started testing?

        Patterns. Look for patterns.

        Baking also reminds me of programming. When you write a recipe you have the ingredients (data) and the instructions on how to use the ingredients (program). The order you put the ingredients together (execution order through the program) can make a huge different. When baking, you want to make sure you have all the right ingredients. You also have to plan out how you are going to use them. Sometimes you have to prepare two parts at once. They both need to be ready at the same time so you can combine them together later.

        So are there any other bakers out there? Do you se any parallels between baking and testing? What about baking and programming?

        Friday, February 12, 2010

        Step by Step Selenium Java

        This is a document on how to write a Selenium Test Case in Java.

        1. Go to Selenium HQ.
        2. Go to the Download section and download Selenium IDE plus Selenium RC
        3. Install the Selenium IDE into Firefox (I assume you have Firefox installed)
        4. Go to Google.
        5. From the Tools menu in Firefox select Selenium IDE.
        6. From the Add-On Preferences, change Selenium to save in Java.
        7. On the Google page, right click and select Open /ig
        8. Type Selenium in the Google search text field.
        9. Press TAB to move focus away from the input field.
        10. Right click on the search text field and select verifyValue q Selenium
          1. Note that verifyValue is the action, q is the locator and Selenium is the optional third value
        11. Click the I'm Feeling Lucky button.
        12. Go to the Selenium IDE and you should see everything we just did.
        13. Click the red dot in the upper right to stop the recording.
        14. From the File menu, save the file as
        15. Open Eclipse
        16. Create a Java Project
        17. Add the jar selenium-java-client-driver.jar to the project.
        18. Add the library junit 3.
        19. Right click on the project name in the Package Explorer and add new package com.example.tests.
        20. Import from File System, the file we saved earlier.
        21. When you finish there should be an error because the class is called Untitled by default.
        22. Edit the file and change the class name to LearningSelenium
        23. In the setUp() change the http://change-this-to-the-site-you-are-testing/ to
        24. If using Mac OS, change the browser string from *chrome to *safari.
        25. Add a public void tearDown() throws Exception which just calls super.tearDown();
        26. Run Firefox from the command line using firefox -P (or firefox-bin -P on UNIX/Linux machines)
        27. Create a new profile, call it selenium and save it in the Eclipse workspace
        28. Go to the command line and start the selenium server using:
          1. java -jar selenium-server.jar -firefoxProfileTemplate /directory/where/you/saved/firefox/profile/selenium
          2. But change the /directory/where/you/saved/firefox/profile/selenium to the place you saved the profile in step 25.
        29. Back in Eclipse, right click the class name and select Run As, JUnit Test
        You are done.

        Thursday, February 11, 2010

        Selenium 101

        So you want to learn about Selenium for web testing.

        Selenium comes in 3 parts.

        1. Selenium-IDE
        2. Selenium-Core
        3. Selenium-RC

        The IDE is a Firefox plugin. You open the IDE and it starts recording what you do inside the browser. The information is stored in an HTML table with one action per row and three columns per action. The first column is the action, the second column is the locator and the third column is optional parameter.

        For example, if I click a link the first column will be 'click', the second column will be some way of uniquely identifying the link and the third column will be blank. Another example, if I enter text into an INPUT the first column will be 'type', the second column will be some way of uniquely identifying the text field and the third column will be the text I entered into the field.

        You could actually write Selenium test cases using an HTML edit and creating a TABLE but it would be hard for you to know all the actions and you'd have to have some way of determining locators.

        When Selenium IDE is running, you can right click on elements and Selenium actions will be available on the context menu. Things like verifyTitle. When you select these, they are added to the Selenium IDE.

        From the Selenium IDE you can save what you have recorded. You could actually record everything you do, save it and then load it and play it back later. By default, it will save an HTML TABLE but you can change the settings for Selenium IDE and save it in different languages. I use Java so I save my recording as jUnit.

        The whole idea behind Selenium Core is to use javascript, in the Selenium window, to drive the window displaying your web application. So when you load a script into Selenium IDE and run it, the IDE translates the commands in the table to actions using the Selenium Core. If you install the IDE you automatically have the Core.

        Selenium RC (Remote Control) allows you to play the scripts from a remote machine. It doesn't have to be a remote machine. It can be the same machine and usually is for development. The Selenium RC comes in two parts: the client and the server.

        If you saved the recording as HTML, you pass the HTML script in on the command line directly to the server. To run the server you use Java. The server is an executable jar. To run the server I would use:

        java -jar selenium-server.jar -help

        This will display a help message. If I run it with:

        java -jar selenium-server.jar -htmlSuite myscript.html

        It will run the Selenium IDE recording (assuming the file myscript.html contains a file I saved from Selenium IDE). If I save the file as Java jUnit I will have to create a class to run the file. I would go into my favourite editor (today it is Eclipse), create a project, add the Java file created from Selenium IDE as a jUnit 3 file. To allow the project to talk to the Selenium Server I would add the Selenium Java Client to my Java project. Now when I run the jUnit it will create an instance of Selenium and send the appropriate commands to the Selenium Server. The server needs to be running before you run the test case. To run the server I would just use:

        java -jar selenium-server.jar

        If I need to change the port or host there are command line switches I can add but the -help will show you those. If you don't give a port or host it will default to port 4444 and host localhost (so the server and client will be on the same machine).

        If you don't want to use Java and you want to use say Perl, you can save the recording from the IDE in Perl the use the Selenium Perl Client to talk to the Selenium Server.

        That is essentially Selenium in a nutshell.

        Tuesday, February 9, 2010

        Do hackers make good testers?

        Someone on LinkedIn was asking about becoming an ethical hacker.

        Back before I got into computers I understood the term hacker to mean someone who learns from trial and error. So a computer hacker was someone who tried things and remembered the results. Now if you started hacking a computer site you probably found things the average user didn't and potentially crashed the site. Because a lot of what was happening on the server was hidden from you, you had no idea what trying different things would do until you tried them. And even then you might not know what actually happened other than you lost the connection.

        If I was hacking a web site or piece of software I would need to narrow things down. If I tried something I might know it will be one of four things that caused the resulting crash. If I tried something else, it might have three things in common with the first hack. If it crashed you might say that one thing that is not common to both hacks is not the crash. You'd be wrong. You're assuming there is one crash. Could be there are two different crashes. :)

        Keeping your mind open like that is what is needed to be a good hacker. I picture things in terms of Venn diagrams. Overlapping areas are not important as it could be one or the other. The things which can only happen in one area are the interesting bits.

        There is something about this way of thinking that makes it easy to find defects. It is almost like detective work. You have some evidence and you need to see if you can find more. You poke and probe to see what falls out. You interrogation the application under test and see what secrets it reveals. You keep track of cause and effect. I do this over a long period of time. You look for patterns.

        Case in point, I was testing an application that had a load feature. There was a defect in the load routine. I filed the defect and the developer fixed it. Once I could load, I could save. A similar defect existed in the save routine. Programmers make common mistakes. Is there more than one place that programmer could make the same mistake? I always take the time to find out what the defect was (from a technical standpoint) and what was really happening.

        After a while I can almost predict what errors will occur in a program.

        The other trait I have and see in other good testers is curiosity. I'm always wondering why things work the way they do. Before computers I'd take apart lawnmowers, watches, bicycles, kitchen stand mixer. When my first hard drive died, took it apart. Hooked it up with the cover off to figure out what was wrong with it. Head was stuck on a scratch on the platter. Smoothed out the scratch and got the hard drive working again. Had some problems with the area I smoothed out but it worked in all other places.

        When it came to understanding the Internet, Telecommunications, etc. I started reading things like RFCs, research papers at MIT, JSRs, ISO/ANSI drafts.

        As I write this I am reminded of many of my students (I used to teach programming). During computer labs there was always someone who would ask me, "What would happen if I did XXX?" to which I always replied, "Try it and see." It always amazed me that most students would ask rather than just try it.

        So if you want to be a good tester. Understand, Try, Hack.

        Sunday, February 7, 2010

        2009 was a bad year?

        You might notice that I've been posting more regularly lately. I used to keep a personal journal. Whenever I moved (I moved a lot) I'd find my old journals and read them. It was interesting to see how I've changed. Today I decided to look at my old postings and see how I've changed.

        I had ONE entry for 2009. What happened? I know that work seemed to be going well. I started a new job in December 2008. A lot of my time was dealing with the new job. One day melts into the next. In September I went away on holiday. I didn't get a lot of time off work so I took all my vacation time for one trip to Europe. I'd sold my home and was unable to find a new home. I was living out of a suitcase. I really needed a vacation... or did I? At the end of September I got ill. Really ill. I'd take a day or two off work. Because I was the entire QA department, I felt I couldn't afford to take too much time off. When I got back from vacation, the person who was supposed to run my test automation couldn't get it to work. There was a bit of panic when I got back.

        Just as I was recovering from one illness another would strike. Things started looking better. I felt better, found a house and the neighbours were great. Then my dog died. I'd had Dante since he was a pup. He was my heart dog. Its been two months and I still cry when I think of him.

        So was 2009 a bad year? Not really. I got to see Europe. I didn't cling to the hope Dante would survive (I wanted to) and we had 3 good months at home rather than 5 really bad months in the hospital. I learned a lot about working on a start up company. I met some really interesting people. I was in a rut up until 2008. I think 2009 was just say, "Hey! Wake up. Get your priorities right." I think 2009 was the kick in the pants I needed.

        Alex Aitken, one the guys I work with, started pointing out articles by some pretty interesting and bright individuals. I knew these guys and had read their work previously but like K&R's book The C Programming Language, you need to go back and read these guys again and again. I've been reading Kent Beck's blog. I highly recommend everyone read his blog.

        Thanks Alex.


        I often see people exhibiting hubris. I look at them and see how obviously wrong they are and how arrogantly they believe themselves to be right.

        It is strange how when I'm being arrogant I just don't see it...

        I used to hate dealing with people. Growing up I lived in a very chaotic environment. To this day I long for an environment with structure. I always feel people were unpredictable. Just when I thought I'd figured out the people around me, as a child, they totally confounded me. Sometimes in a good way but most off in a bad way.

        In my late teens I was introduced to computers. It had limited memory (a few kilobytes). The instruction set was relatively small. It was CISC based with no pipelines, caching, etc. Execution of an application was fairly linear and simple. There were interrupts but the main application would halt when an interrupt occurred. So there was never two things happening at once. This was bliss. I could make this box do things. It was VERY predictable.

        To this day I look at computers and think, it is a finite state machine. We should be able to write bug free software. Why can't people program properly. If I had the time, I could write PERFECT software. I've often dreamed of building a computer, writing the operating system, writing all the applications, doing everything right from the start. No crashing, no updates needed, no patches, no viruses.

        Today I was reading a blog on software design by someone who reminds me a lot about myself but is obviously smarter... Kent Beck. After reading the article I thought, "Kent seems to have a lot more answers and is much more self-aware then I am but he still has questions and things he is trying to figure out about software design. How arrogant of me to even imagine for a moment that time is the only limiting factor in writing the perfect operating system with perfect software."

        Have you spotted your hubris today?

        Saturday, February 6, 2010

        Watij versus Selenium-RC [Java]

        If you want to use open source test automation frameworks for testing web applications there are two categories. Some of the frameworks have their own engines (CSS, Javascript, HTML, etc.) and others use a real web browser.

        One advantage of the frameworks with their own engines is they have more control and you don't have to deal with the real life implementations of a web browser. You are also not dealing with TCP/IP, network delays, etc.

        The disadvantage, on the other hand, you are not dealing with real life implementations of a web browser. You are also not dealing with TCP/IP, network delays, etc. Basically, if you use a framework with its own engine and your application passes, it does not guarantee it will work on a real web browser like Firefox or Internet Explorer.

        Two frameworks which drive a real web browser are Watij (Web Application Testing In Java) and Selenium (pronounced sa-LEE-nee-am).

        Watij is a Java test framework. You will need to know Java and jUnit to use Watij. The documentation is VERY lacking. The people developing Watij are developers. They are not creating a commercial product. If I was programming a calculator, adding comments to the source code would not be important. Instead, I would want the source code to be self-documenting. The method names should tell me what they do. The class names should imply what methods exist in it (i.e. I'd know what to expect in the file even before I opened it). Additionally, I would write unit test cases to show how the code was expected to be use.

        As something built on the jUnit framework, I expected better documentation. I quickly realized that it was best to have the source code jars added to my project and occasionally needed to step into the Watij code to figure out how things worked.

        Everything was nicely structured however. You have HtmlElement which was the base class for all other elements. You have things like Button, Table, Link, etc. which inherit HtmlElement. These are all interfaces and you have implementations like IEButton. I believe the idea was that some day you might have IEButton, FirefoxButton, ChromeButton, OperaButton, etc.

        In Java I might have:

        public List doSomething() {
        List text = new ArrayList();
        // more code here
        return text;

        Similarly, in Watij all the methods return an interface rather than a specific implementation. So the signature for a method returning an IELink is:

        public Link getLink(String locator);

        Nothing special for a Java test framework. Additionally, a Table extends a TableBody, a TableBody (and subsequently a Table) can tell you how many rows and columns it has. It can also return a specific TableRow or TableCell. Back to the fact there is no JavaDoc for the methods but it does not made because the code is self-documenting.

        When a method need to return a collection of elements, e.g.

        Links links = ie.links();

        The implementation of Links is typical Java. It assumes we have Collections, Lists, Maps, etc. So the Links implementation is Iterable<link>. So we can have:

        for(Link link : links) {
        // do something with each link in the collection

        Essentially, writing automation is no hard for a Java programmer than writing a simple application.

        Selenium on the other hand is totally different from Watij. First, it is not a Java framework. Selenium comes in multiple parts and one of them allows you to use Java. The starting point for Selenium is Selenium IDE. You get an AddOn for Firefox. When you open the Selenium IDE it turns on recording right away. Everything you do in Firefox is then recorded into an HTML table. The table has three columns. The first column is the command, the second column is the target and the third column is an optional value. For example, if I wanted to click a button the command would be 'click' and the target would be some way of identifying the button. This could be CSS, an xpath, the id of the button, the name of the button, etc. Another example using all three columns would be entering text in a text field. The command would be 'type', the target would be the text field and the third field would be the text you want to put in the text field.

        This is great for doing some quick and dirty automation but lacks the ability to build into a test suite. Additionally, if every test case started with logging into the application you would have duplicate code. If the way you log in changed, you would have to edit all the test cases. This is possibility the number one reason test automation fails. It is not maintainable.

        The next part of Selenium is Selenium RC (remote control). After you record the test case using the IDE, you can save it in a variety of different languages. If you are a Java programmer, you can save it as Java code.

        To run the Java code you'll need the Selenium RC jars and a Selenium Server. The way it works is the Selenium RC jar is a client. Your Java code sends commands to the server via the client. The server then uses Selenium IDE to control the web browser (via Javascript).

        The nice thing about this is you can now record a snippet using the Selenium IDE, save it as Java and incorporate it into your Java test suite.

        This is the theory. The reality is that how Selenium IDE recognizes the elements in the browser is not always the best choice. For example, I might have a set of SPAN elements inside the TABLE to make the cells format nicely. Along comes Internet Explorer 9 and my table looks horrible. So I have to add a DIV around each SPAN. From a customers point of view everything looks the same, but from a Selenium locator point of view my test cases all break. It can no longer find the cell. So you will need to look at the locator Selenium IDE has selected and decide if you want to change it.

        Additionally, AJAX and timing are a huge issue. For example, you have a form with two select lists. The first list is a list of countries. The second list is empty. You select your country and an AJAX call populates the second list with states, provinces, territories, etc. Selenium IDE will record you waited 1 second before selecting your province because you selected Canada for the country. If you select United States it takes 2 seconds to populate the second list. A bad automator will just add a few seconds to the wait. A good automator will look for something in the DOM which signals the second list is populated. For example, the second list might be hidden. So you want to wait for the second list to become visible. Knowing to code this is one thing you need to have when using Selenium IDE and knowing what to code is even harder.

        So the record feature is not that good. It will most often encourage less experienced automators to create code duplication and use sleep/wait statements to handle things like AJAX/Web 2.0 features.

        Another issue with Selenium is the Java support. When you look at the Java support it is fairly simplistic. There is no hierarchy to the data structures and all the methods return arrays. You'll also find things like, I can find out how many elements match an xpath but I have no way of iterating over the list of elements. In Watij you can do the following:

        List result = new ArrayList();
        String myXPath = "//A[contains(text(), 'Phoebe')]";
        Links phoebeLinks = ie.links(xpath(myXPath));
        for(Link currentPhoebeLink : phoebeLinks) {

        To do the same thing in Selenium RC using Java, in theory, you need to do:

        String[] allLinks = selenium.getAllLinks();

        then iterate over all the strings to figure out which ones are the ones you are looking for. However, when I use this method it actually returns back an array of one element and the one string it does contain is "". In other words, it doesn't seem to work.

        My workaround for this problem was to use the getHtml() method. This returns all the source code for the current page. I can then load it into something like jTidy and use the jTidy interface to traverse the page and find all the links. To actually iterate over all the links and do something with them I'd have to create a unique list of the links using jTidy then go back to Selenium and interact with them.

        Essentially, I found that writing a test case in pure Java was simple with Watij and difficult if not sometimes impossible with Selenium RC. Ultimately, the maintenance on Selenium will be the potential downfall.

        On the other hand, when I actually run the test cases I find that Internet Explorer has problems. It hangs and locks up occasionally. For an automated test suite this is unacceptable. With Watij I am stuck. With Selenium I can run the tests in Firefox, Opera, Safari or Internet Explorer. So automated nightly build and test is possible with Selenium but not really viable with Watij.

        As I see it, Watij is a fantastic house built on a bad foundation. Selenium is a good starter home which can easily be moved to a variety of different foundations.

        Neither is a great solution. I spent a year trying to work around the bugs in Internet Explorer with Watij but in the end just gave up. For now, I'm looking at Selenium. Maybe it is time to start doing some Selenium RC programming, i.e. give back to the community.

        Monday, February 1, 2010

        Testing under a deadline -- the definition of insanity

        Ever have to test something with limited time? Or course you have. When I first started testing the plan for the Software Development Life Cycle (SDLC) was:

        - get the requirements
        - design the application
        - start thinking about how to test it
        - start implementing the design
        - start testing the application
        - fix the defects
        - test it again
        - ship it

        Generally speaking there is a lot more iterating over parts of this and occasionally the application gets redesigned in the middle of everything.

        On many projects, the plan is laid out so that each step gets a set amount of time. They put a buffer in for unanticipated problems. As the project progresses, when it looks like we aren't going to meet the target date, they put more resources on it, reduce the scope or both.

        If things really aren't going well (at some point putting more people on the project just makes things worse and if you cut too much it just isn't worth shipping), they do Albert Einstein's definition of insanity: doing the same thing over and over again and expecting different results. They just do it faster and more frequently. In the end, the Spiral development model becomes the Death Spiral and testing time gets shorter and shorter on each iteration.

        So what do you do as a tester?

        If you started off well intentioned, you asked for say 2 weeks (10 business days) to do testing. After you get the initial design and more detailed information, you reestimate and ask for 3 weeks (15 business days). You create a wonderful test suite with hundreds of necessary tests. You feel you should be able to complete all the testing in 12 days (80% of 15 days is 12 days; keep 20% in reserve for unforeseen problems).

        You ran the entire test suite and filed a few hundred defect reports. Development threw a new version of the product over the wall but you now have 5 days to test it again. Do you run the entire test suite? There are a few possible choices.

        The first is to order the tests by importance. This way, if you don't get through all the tests before the ship date, you have covered the most important ones. But 5 days is only 33% of the expected 15 day test period. Even if we assume nothing goes wrong, 5 days is less than 42% of 12 day test period. This means best case scenario we have 58% of the tests not run. Not good.

        We can test those things which will be hardest or impossible to patch once the product has shipped. Then we continue testing after the product has shipped, create a patch and hope we didn't lose too many customers.

        Another option is to examine source code and determine what areas have definitely not changed. Determine which test cases apply to only those areas which have not changed and drop them from the test suite. If the test failed previously, we know they had to make changes to that code or the defect still exists. So tests which failed must be kept. How much interdependency does the product code have? The poorer the development the harder it will be to test this way. From a QA standpoint, good development comes before good testing.

        If your staff is fairly senior and good at adhoc testing, you might have a better chance at just re-testing the failed tests and using good judgement to test the related features. But now we are off the test plan. How do we record what testing was done and the results? What I like to do is keep a copy of notepad/textpad/vi/etc. open and write point form notes on what I'm doing, as I do it, and what results I see. If the application is very graphical and that matters, use something like Microsoft Word or Apple Pages so you can copy and paste images into the document. The important thing is being able to reproduce the tests if needed.

        I once had the idea of recording the video output onto video tape. Then you could play the recording back and transcribe the video to a test plan, if necessary.

        All of these ideas are accepting the idea that you have to deal with the limited time to test. How about talking to project management (or their management) about doing a post mortem. Figure out what went wrong and put things in place to make sure it does not happen again.

        The most important thing is to not do it again; remember Einstein's definition of insanity. Features will have been dropped because the project was behind schedule. There will be a strong pressure to release a patch or the next version in order to get those dropped features out to the customer. Resist the urge to just dive in and do it all over again.