Should your test environment be dirty or clean?

I'm not talking the state of your desktop. Whether you have pizza boxes, open bags of chips and a bowl of jelly beans is another topic altogether different.

What I want to talk about is what the operating system image should look like. On the one hand, you can setup a Vmware session, install the OS and apply all the patches to it. We'll assume these are bare minimums for any of your customers. Some might skip patches and service packs but the combinations get to difficult to support. It is easier if you just put as system requirements a minimum service pack and tell your customer support to say, "have you applied all the latest patches?"

Let's say we are testing a web application. We want to test with Windows XP SP3, Internet Explorer 7, Internet Explorer 8, Firefox 3.5, Safari 4.0.

We could set up one image with Windows XP SP3, Internet Explorer 7 and Firefox 3.5. On another image we would have Internet Explorer 8 and Safari 4.

The reason I have at least two images is because you cannot have IE7 and IE8 on the same image. Even if you could figure out how to do it, system libraries get updated by IE8 and this affects the way IE7 will run. I could install IE7, FF3.5 and Safari 4.0 on one image then only IE8 on the other image. If I'm doing automation I'd want to distribute the tests evenly so two browsers per image makes more sense than 3 on one image and 1 on the other image.

Ideally, I'd like four images. On web browser for each image. However, you cannot get an image of Windows XP SP3 without IE7 installed. So we have to have at least IE7 and Firefox or IE7 and Safari.

So, what do we install after this? We can install software that a typical customer has on their computer. We could install Microsoft Office. A lot of people use this so it is safe to assume it will be installed.

If I was testing a desktop application, what I installed would start matter a lot more than testing a web application. What files are going to be sent from the server? Maybe we don't want Microsoft Office installed. We want to see what happens if the user doesn't have it installed and we send them a spreadsheet. If we send it with the wrong mime type it could appear as gibberish on the web screen.

The cleaner the environment the more likely we are to catch things like that.

What about plug-ins or development tools? This is a hard call. If you need to call over a developer so they can see the problem on your system, they might need some basic tools to debug what is going on. Especially if they cannot reproduce the problem on there system.

On the other hand, I have seen applications function correctly because development tools were installed. The moment a customer tried to use it, it crashed. Solution, install minimal tools on one of the images, if necessary. If you press F12 while in IE8 you will find they have a good set of development tools native to the browser.

Essentially, there is a fine balance of what you want to install and what you don't. you really have to stop and think about the inner action between what you are installing and the application you are testing.

For example, does installing a plugin to Firefox affect how web pages render? Is it a popular plugin? Most the decisions I make for setting up my environment are based on what would my customer do? and not on what I personally like. If the majority of customers are using a particular plugin, even if I find it useless, I will install that plugin.

The more you know about how the operating systems and applications work, the easier it will be to anticipate unwanted interactions between the environment and the application you are testing.

In summary, I think your test environment should be as clean or dirty as your typical customer.

Top 25 Most Dangerous Programming Errors

Every wonder what the top 25 most danger programming errors were? Well wonder no more, Common Weakness Enumeration has created just such a list for your enjoyment. You can find the list here.

Whether you are new or an old hat at security testing, this is a document you want to read.

How to reduce code duplication, part 2

In a previous entry I blogged about how to reduce code duplication. My solution is using libraries.

Let's say I have a web application for writing blogs. Okay, I'm actually going to use the Blogger software as my example. First thing I'm going to do is break the application down into manageable parts. Across the top is:

• Posting
• Settings
• Layout
• Monetize
• View Blog

Each one will take me to a new page. I typically automate with Selenium using Java. So I might create one class for each of these areas. However, when I select Posting I see it has the following subsections:

• New Post
• Edit Posts
• Edit Pages
• Comment Moderation

So I might want to create the following packages:

• com.google.blogger.posting
• com.google.blogger.settings
• com.google.blogger.layout
• com.google.blogger.monetize
• com.google.blogger.viewblog

Then within the com.google.blogger.posting package I'm going to create the following classes:

• NewPost
• EditPosts
• EditPages
• CommentModeration

Next I'm going to focus on one class at a time. Additionally, I'm going to create Java classes to represent the data on the application. For example, in the Posting section, on the New Post page I have the following:

• Title
• Link
• HTML (the HTML text in a text editor)
• Labels
• Post Options
• Reader comments
• Backlinks
• Post date and time
• Edit HTML Line Breaks
• Compose Settings

All this is data for the page so I'd create:

class NewPostForm {
        String title;
        String link;
        String htmlBody;
        String labels;
        PostOptions postOptions;
    }

    class PostOptions {
        boolean allowReaderComments;
        boolean allowBacklinks;
        boolean enterIsLineBreak;
        boolean htmlShownLiterally;
        boolean automaticPostDateTime;
        String scheduledAtPostDate;
        String scheduledAtPostTime;
    }

I would also add getters, setters and constructors to these classes, e.g.

public void setTitle(String title) {
        this.title = title;
    }

    public String getTitle() {
        return title;
    }

Now, if I create a library function to fill in a new post, I don't have to pass in a dozen or more parameters. I can create an instance of NewPostForm, populated it then pass it in to my library function. So back to the NewPost class. This is the library of routines to be used by my test suite:

class NewPost {
        Selenium selenium;

        public NewPost(Selenium selenium) {
            this.selenium = selenium;
        }

        public void gotoNewPost() {
            // if a link to New Post exists then
                // click the link
                // wait for the new post page to appear
                // this might be a page load so WaitForPageToLoad
                // or it might be AJAX so waitForCondition or
                // whatever the case may be
           // else
                // fail the test case
        }

        public void fillInNewPostForm(NewPostForm npf) {
            assertNotNull(npf);
            assertNewPostForm();
            if(npf.getTitle() != null) {
                // set the title input field to npf.getTitle()
            }
            // more of the same thing for each field in npf
        }

        public void assertNewPostForm() {
            assertTrue(selenium.isElementPresent(titleTextFieldLocator));
            assertTrue(selenium.isElementPresent(linkTextFieldLocator));
            // more of the same for all the inputs in the form
        }

        public void clickPublishPost() {
            // if a link for Publish post exists then
                // click the link
                // wait for the post to get published
            // else
                // fail the test case
        }
    }

I've left out things like the values for titleTextFieldLocator but that is easy enough. It would be something like:

String titleTextFieldLocator = "xpath=//input[@id='postingTitleField']";

As you build things up you will have a library of methods for the different 'pages' of the application. So if I wanted to test all the different ways of posting something I could have an instance of NewPost as a class variable for a test case then initialize the variable in the @Before method, just after the initialization of the selenium instance. For example:

class NewPostTestCases extends SeleneseTestCase {
    NewPost np;

    @Before
    public void setUp() throws Exception {
        super.setUp(); // initializes inherited selenium
        np = new NewPost(selenium);
    }

    @TestCase
    public void someTest() throws Exception {
        np.gotoNewPost();
        NewPostForm npf = new NewPostForm();
        npf.setTitle("How to reduce code duplication, part 2");
        npf.setHtmlBody("this is my new blog body");
        npf.setLabels("selenium, java, testing, development");
        np.fillInNewPostForm(npf);
        np.clickPublishPost();
        // do some sort of assertion to confirm the post went okay
    }

    @After
    public void tearDown() throws Exception {
        super.tearDown();
    }
}

And that is the basic idea. I'd used my IDE to generate all the getters/setters and constructors. I'd also let it suggest when I needed to throw an exception (I'd always throw the exception rather than try/catch and deal with it; junit was made assuming the exception gets thrown up to the TestRunner). And there are things like, put the datatypes in a different package, e.g. com.google.blogger.posting.datatypes for the NewPostForm datatype plus let the IDE tell you when to add the appropriate import statements.

Measuring code complexity as a way to determine number of test cases

There are tools which will measure code complexity (cyclomatic complexity) for a software project. Cyclomatic complexity is the number of linearly independent paths through a program's source code. For example, given the following code:

if(expr1) {
    // option1
} else if(expr2) {
    // option2
} else {
    // option3
}

if(expr1) {
    // option4
} else {
    // option5
}

Assuming expr1 != expr2, the following paths through the code exist:

1. option1, option4 (expr1 == true)
2. option2, option5 (expr2 == true)
3. option3, option5 (expr1 == false, expr2 == false)

If you look carefully at this list, when testing this snippet of code, for full path coverage, I want a set of tests which will cause each line of code to be executed at least once. I want each if/elseif/else to get evaluated and the body of each to get executed. To make this happen I need:

1. expr1 == true, which implies expr2 == false
2. expr2 == true, which implies expr1 == false
3. expr1 == false and expr2 == false

The number of paths through the code is the number of test cases! If you are looking at branch coverage the number of paths required for branch coverage is going to be similar. You can give an example where the branch coverage requires less than the code complexity. For example:

if(expr1) {
    // option1
} else {
    // option2
}

if(expr2) {
    // option3
} else {
    // option4
}

For this example, I need the following:

1. expr1 == true, expr2 == true
2. expr1 == false, expr2 == false

or I could use:

1. expr1 == true, expr2 == false
2. expr1 == false, expr2 == true

Both examples will cause all branches to be taken at least once. Generally speaking the rule is:

branch coverage <= code complexity <= path coverage

So if you have the code complexity, you know the number of paths through the program has to be at least the code complexity of the application. This also assumes that all paths are reachable. A code complexity tool should also tell you if certain paths are not reachable. Actually, many development environments will warn you if a path is not reachable.

So next time you are asked to estimate how many test cases you will require to adequately test an application, use a code complexity tool to give you a rough idea.

Additionally, the complex of the code can give you some idea of the number of defects which should be found. The more complex the code, the greater the number of defects should be suspected. This is not always true but can be a good rule of thumb.

See an article from Enerjy for an example of this phenomenon.

Are most binary searches broken?

I was poking around the web and found an article by Joshua Bloch (you can see it here) about a software bug in the Java binary search back in 2006. Joshua found a number of implementations of binary search with the bug.

With binary search there is a line where you take the low point plus the high point then divide the sum by two. In code this would be:

int mid = (low + high) / 2;

Can you see the bug? There is no check for overflow. If low + high is greater than the maximum value for an int it will wrap around and become negative. I used to teach this concept as an odometer on a car or a clock.

Imagine you have a clock:

A regular clock has 60 increments (seconds). With something like int in Java the clock would have 4 billion increments. With this clock our number system goes from -5 to +6. Here is how it works, addition is clockwise and subtraction is counterclockwise. So if I have 2 + 1, I start at the number 2 and move 1 increment clockwise. This lands me at 3.

If I have 1 + 5, I start at 1 on the clock and move 5 increments clockwise to land at 6.

For subtraction, let's try 5 - 3. We start at 5 and move counterclockwise (subtraction) 3 increments. We land at 2. If you take something like 4 - 6 you get, start at 4 and move counterclockwise for 6 increments. This lands you at -2.

Now what happens if I take 4 + 5? The rule is start at 4 and go 5 increments clockwise. We land at -3. So in my  number system 4 + 5 = -3. This is obviously wrong and this is what can happen with the binary search. If low + high is greater than Integer.MAX_VALUE (2^31-1) the value will become negative. A negative number divided by two is still a negative number.

If the size of the array is greater than 2^30-1 then binary search will fail.

The funny thing is, I was teaching about the dangers of integer overflow ten years before Joshua Bloch found this bug and I never suspected something like this in modern programming language libraries.

The text for the course which tests about integer overflow does not deal with things like binary search. By the time my university teaches integer overflow we aren't dealing with anything as simple as binary search. Binary search is taught in first year, first course. Makes me wonder how many other places have fallen prey to the integer overflow bug.

By the way, one fix for the binary search algorithm is:

int mid = low + (high - low) / 2;

Since high is always greater than or equal to low, (high - low) will always result in a positive number or zero.

How to reduce code duplication

In a previous blog entry I talked about starting and ending test cases in the same place. One of the problems with this is that there will be duplication in test cases. For example, I might have the following two test cases:

Test#1
- Log into web site
- Find a product to purchase
- Add to shopping cart

Test#2
- Log into web site
- Find a product to purchase
- Add to shopping cart
- Check out

If the way you log in, find a product or add to shopping cart changes there will be two locations to update. If 99% of the test cases require you to log in, there could be hundreds if not thousands of instances. A change in the log in process could take days to update.

The solution is to create libraries. You can organize them by feature or by page. I typically organize them by page. You could write detail libraries that do atomic actions (click button, fill in form, go to page, etc.) and other libraries which do higher level actions. The higher level actions could be calls to a set of the atomic actions.

Your test case would then be much shorter and simpler to write. As you build up the library of actions, writing test cases will be faster and faster. In other words, the time spend up front will save you time in the long run.

The key to doing this well is organization. You need to look at how other, successful, libraries are organized and try to make your libraries reflect the same consistent structure. Use descriptive names. Both for the grouping (classes in Java, modules in Perl, etc.) and for the method calls.

A test case should almost read like your native tongue and the high level methods should read almost like your native tongue as well. The atomic methods should be simple and straight forward.

Finally, all atomic methods should make sure the input parameters are valid (fail if they are not with a good error message), the state of the web page is what you expected (again fail if it is not) and you get the results you expect at the end of the method. That is, from Computer Science 101, make sure the pre and post conditions are met!

A good error message does not assume what went wrong. Instead it will tell you the state of the test case and let a human determine what was wrong.

Using source control within Eclipse

In my last blog entry I talked about setting up a Subversion (SVN) repository, getting the subclipse plugin for Eclipse and adding a project to source control.

So why do we need source control? There are many reasons to use source control. Here are some of them:

1. It creates a backup of your work
2. It allows you to undo changes you made to the project
3. It lets you work as a team with others on the same project
4. You can create branches of the project for different reasons

The first point is pretty obvious. If something happened to your working copy of the project you still have a backup of the project in source control. If the repository is on a remote machine it is even better. If the repository is on the same machine but a different hard drive, that is a little better. Obviously not a huge advantage of having the repository on the same hard drive as the working copy.

The ability to undo changes is huge. Let's say you have a project using Selenium and it looks fantastic. You have 80% of the application automated. Then you add one more feature. You run the automation and there are failures all over. You check one of them and find there is nothing wrong with the application under test. It was a change you made to the automation. You have been working on the new automation for 3 hours.

Let's say you didn't have source control. You realize you probably should have tested some of the changes rather than working solid for 3 hours. It is too late to fix that now. You try to remember what you changed in the last few hours. Was it something you changed at the beginning? Was it something you just changed? You have no idea. You step through the automation and try to figure out what is wrong. It has been a long day and you are tired. You try changing a few things, you believe, back to the way they were. After another hour of changing things and just making it worse you decide to go to sleep and look at it tomorrow.

The next day you have a look at the code and immediately realize you messed up one of the locators. You thought the automation was failing at step 37 but there was actually a failure in the setup. You fix the locator and check the setup. It is working fine now. What about the hour of changes you made yesterday? After around 3 hours of work you think you got it back to were it was yesterday.

One minor mistake, 1 hour franticly trying to fix it, 3 hours undoing the frantic work you did yesterday. Total time wasted 4 hours. Maybe things aren't this bad in a situation you might encounter but I can guarantee you will waste an hour or more with situations like this.

Now let's look at the same situation with source control. You realize you should have tested your automation a few times in the last few hours. No problem. You have a few options at this point. You can compare your working copy with the last thing you checked in. In Eclipse,

1. Right click on the project
2. Go down to Compare With and select Latest from Repository

It will give you a window with a tree view in the top. The tree will contain a list of all the files which have changed. If you select one of the files, by double clicking it, it will open a compare view. On the left will be your working copy with the changes and on the right will be the last thing you checked into the repository. Using this tool you can review all your changes and see if one of them caused the failure. Hopefully you spot the change in the setup. Worst case, set a breakpoint at all the places you made a change, run the automation in debug mode and confirm everything is good at each change. You should definitely see that the setup is broken.

Typically, you don't want to check in broken code but if you are on a private branch (more about this later) or working alone it is okay. You could check in the code, compare the last checked in with the previous version. You'll see the same comparison tree but this time you can undo all the changes and bring them back one at a time. The whole procedure would be:

1. Check in broken code
2. Check out the previous, working version
3. Compare this with the checked in, broken version
4. Re-introduce one change at a time
5. Run the automation and confirm the change didn't break anything
6. Keep repeating steps 4 and 5 until you find the change that break the automation

The third advantage of source control is working as a team. If the repository is in a location two or more people can access it, each person can check out a working copy and make changes. This is one of the biggest advantages of source control. Without source control, if two people wanted to make a change on one file they'd have to one at a time. Imagine we have version 1 of a file. I get a working copy, you get a working copy. I make a change to line 17. You make a change to line 43. Without source control, I save the file back to the shared folder. You save the file back to the shared folder. My change to line 17 is lost. Not good.

With source control, it is smart enough to merge my changes and your changes. If for some reason it could not figure out how to merge the changes, the last person to attempt checking in would be told there was a conflict. To fix a conflict in SVN, you update your working copy to the latest from the repository. It will keep your change and grab a copy of the conflicting change. You then compare the changes, fix them by hand and check the file back in. At this point there should be no conflict.

The last advantage I mentioned was creating branches. A branch is a way of creating copies of the entire project. An example of a branch would be version v1.0 and trunk. When I start my project, everything is put in trunk (some source control software calls this head or main). At first things aren't very stable. I would never give a copy of trunk to a customer. At some point I get trunk stable and running well. If I add a new feature, trunk will become unstable again. What I can do is make a branch. Now the repository has trunk and v1.0. Any changes I make on v1.0 will not affect trunk and vice versa. Now I can add the new feature to trunk and destabilize it. At the same time, I can do safe, simple bug fixes on v1.0. I then ship the build from v1.0 to the customer (automation for v1.0 is also updated to match the product I shipped to the customer and nothing more). 

A few months later the customer calls up and says he found a bug. Development has made a LOT of changes to trunk by now. Test automation has been updated to match the changes in trunk. If you tried to run the test automation on v1.0 it would fail to run properly. We don't want to give a copy of trunk to the customer. Partly because it is not stable but also because it has new features the customer hasn't paid for.

So a developer will check out a copy of v1.0, fix the bug and build it as version v1.0.1. While the developer is working on the bug fix, the test automator will check out a copy of the automation from branch v1.0, update the automation to catch the bug, test it to make sure it does find the bug in v1.0 of the product. The tester will now run their automation against version v1.0.1 and confirm the fix. At this point the source control administrator will tag the code (development and automation) for v1.0.1.

Hold on, we missed something. The bug still exists in trunk. Whenever you work on a branch, you have to be sure to merge bug fixes from the branch down into trunk. The change to test automation has to be merged down into trunk as well. Sometimes it is a simple procedure but in some cases, the code in trunk has changed so much they (a) the bug no longer exists or (b) the fix is totally different. In case (b) the developer will have to devise a new fix and the test automator needs to change the test to find the bug in trunk.

There are just some of the things to know about using source control. I'll just leave you with a few recommendations:

1. Test your code often. Don't wait 3 hours.
2. Only check in code that works.
3. After you test code and confirm it works, check it in.
4. Put good comments so you know what you checked in and why.

Good luck and happy coding.