Google Analytics


To search for specific articles you can use advanced Google features. Go to and enter "" before your search terms, e.g. CSS selectors

will search for "CSS selectors" but only on my site.

Friday, March 5, 2010

Should your test environment be dirty or clean?

I'm not talking the state of your desktop. Whether you have pizza boxes, open bags of chips and a bowl of jelly beans is another topic altogether different.

What I want to talk about is what the operating system image should look like. On the one hand, you can setup a Vmware session, install the OS and apply all the patches to it. We'll assume these are bare minimums for any of your customers. Some might skip patches and service packs but the combinations get to difficult to support. It is easier if you just put as system requirements a minimum service pack and tell your customer support to say, "have you applied all the latest patches?"

Let's say we are testing a web application. We want to test with Windows XP SP3, Internet Explorer 7, Internet Explorer 8, Firefox 3.5, Safari 4.0.

We could set up one image with Windows XP SP3, Internet Explorer 7 and Firefox 3.5. On another image we would have Internet Explorer 8 and Safari 4.

The reason I have at least two images is because you cannot have IE7 and IE8 on the same image. Even if you could figure out how to do it, system libraries get updated by IE8 and this affects the way IE7 will run. I could install IE7, FF3.5 and Safari 4.0 on one image then only IE8 on the other image. If I'm doing automation I'd want to distribute the tests evenly so two browsers per image makes more sense than 3 on one image and 1 on the other image.

Ideally, I'd like four images. On web browser for each image. However, you cannot get an image of Windows XP SP3 without IE7 installed. So we have to have at least IE7 and Firefox or IE7 and Safari.

So, what do we install after this? We can install software that a typical customer has on their computer. We could install Microsoft Office. A lot of people use this so it is safe to assume it will be installed.

If I was testing a desktop application, what I installed would start matter a lot more than testing a web application. What files are going to be sent from the server? Maybe we don't want Microsoft Office installed. We want to see what happens if the user doesn't have it installed and we send them a spreadsheet. If we send it with the wrong mime type it could appear as gibberish on the web screen.

The cleaner the environment the more likely we are to catch things like that.

What about plug-ins or development tools? This is a hard call. If you need to call over a developer so they can see the problem on your system, they might need some basic tools to debug what is going on. Especially if they cannot reproduce the problem on there system.

On the other hand, I have seen applications function correctly because development tools were installed. The moment a customer tried to use it, it crashed. Solution, install minimal tools on one of the images, if necessary. If you press F12 while in IE8 you will find they have a good set of development tools native to the browser.

Essentially, there is a fine balance of what you want to install and what you don't. you really have to stop and think about the inner action between what you are installing and the application you are testing.

For example, does installing a plugin to Firefox affect how web pages render? Is it a popular plugin? Most the decisions I make for setting up my environment are based on what would my customer do? and not on what I personally like. If the majority of customers are using a particular plugin, even if I find it useless, I will install that plugin.

The more you know about how the operating systems and applications work, the easier it will be to anticipate unwanted interactions between the environment and the application you are testing.

In summary, I think your test environment should be as clean or dirty as your typical customer.

No comments: