Using Charles from the command line

If you are testing network traffic you are probably familiar with Fiddler. Fiddler is a nice, easy to use tool for monitoring network traffic.

It works very easy. You start up Fiddler and it configures your Internet Settings. Now when you start up a web browser, it automatically routes traffic through Fiddler. As you hit web pages on the browser, the HTTP requests and response show up in Fiddler. It is very easy to read and understand what is going on.

If you are using a macOS computer you will be sad to learn that Fiddler does not exist for macOS. It is a Windows only product. If you check for free options to do the same thing you find Wireshark (formerly Ethereal). But Wireshark's configuration and output assumes you have knowledge of TCP, HTTP, Sockets, packets, etc. You can get the information that you need but it is not as easy as Fiddler.

Additionally, to play back a request with some modifications is a lot harder with Wireshark than with Fiddler.

So what do you do? Charles Proxy. Unfortunately, it is not free but at $50 it is a good investment. If you are working at a company with many people needing it, there are discounts available as well.

Now if you get Charles you will find it automatically starts up and changes the Network Settings on your macOS. So all the browsers and anything which uses Network Settings, will automatically go through Charles. 

What about command line? For example, I have a Docker script which creates a container, deploys a web service and waits for someone to hit it. What if I'm creating automation using Python, Java, bash script, etc.? These do not use the macOS Network Settings. So you will see nothing in Charles.

The solution is to add the necessary information to the shell before you launch your test scripts.

The way Charles works is rather simple. If my machine is using 192.168.0.4 and I want to hit www.google.com (209.52.144.114) it might following the following route:

• 192.168.0.4
• 64.114.101.7
• 209.121.102.146
• 209.52.144.114

The way Charles works is creating a MITM (Man-In-The-Middle). So if I want Charles to be able to observe the traffic the route might be:

• 192.168.0.4
• CharlesProxy
• 64.114.101.7
• 209.121.102.146
• 209.52.144.114

The way it does this is by creating proxy settings in Network Settings. To create proxy settings on the command line you need to set certain variables. For HTTP traffic and HTTPS traffic Charles tells macOS to set it to use IP address 127.0.0.1 and port 8888.

For the command line you want to use:

export http_proxy="http://127.0.0.1:8888"
export https_proxy="http://127.0.0.1:8888"

Additionally, Charles tells the macOS to bypass certain addresses. What I do is go to System Preferences, select Network, select the Advanced... button, to to the Proxies tab.

On this page, assuming you are running Charles, you will see a bunch of addresses in the Bypass proxy settings box. Select all of them, copy them into the clipboard, go back to the command line and enter:

export no_proxy="<paste>"

With these three settings, anything you run from the command line will go through Charles.

However, if you close the shell you lose all the settings. If you want to keep the settings you can add them to your ~/.bash_profile text file. Every time you open a shell it will add the proxy information to the shell. HOWEVER, you don't want this if you are not using Charles. To disable this you need to enter:

unset http_proxy
unset https_proxy
unset no_proxy

So what I do is add the following to my ~/.bash_profile text file:

# Charles shortcuts
function charles_on {
        export http_proxy="http://127.0.0.1:8888"
        export https_proxy="http://127.0.0.1:8888"
        export no_proxy="127.0.0.1:6258, 127.0.0.1:6263, 127.0.0.1:10191, 127.0.0.1:14821, 127.0.0.1:24861, 127.0.0.1:25007, 127.0.0.1:38151, 127.0.0.1:46360, 127.0.0.1:49801, 127.0.0.1:55730, 127.0.0.1:59483"
}
function charles_off {
        unset http_proxy
        unset https_proxy
        unset no_proxy
}

By adding this to my ~/.bash_profile text file I can use:

charles_on

to enable Charles. And I can use:

charles_off

to disable Charles. Whenever Charles is not running I MUST disable Charles on the command line.

Browser statistics

A lot of discussion goes into browser statistics. Whenever a company is creating a new web application they need to decide which browsers will be supported and their order of importance.

Years ago it was a given that Internet Explorer was the #1 browser. Just the shear volume of Windows users and lack of serious competition in that area made Internet Explorer the dominant browser.

Then along came Firefox and later Chrome. Additionally, Mac Desktop is becoming a large enough market that Safari is worth considering.

So how do we decide which browsers we should support and what order we should rank them in?

There are many sites which give statistics on browser usage:

W3 SchoolsStatCounterW3 CounterClickyNetMarketShareWikiMedia

and probably a whole lot more to check. If we look at the different sites we can see some radically different numbers. Some claim Chrome is close to 60% of the market. Others have Chrome around 30% of the market.

Why the big difference? Different sites will gather statistics on different people. The W3 Schools site is very popular among programmers and software testers. I tend to like Chrome because it has the built-in Inspect feature and other helpful development tools. Firefox is okay when you install Firebug or other tools but often we want to test with a clean browser and don't have these tools available.

Starting with Internet Explorer 8 it came with development tools. The tools in Internet Explorer 9 are better but nothing as good as the tools in Firefox or Chrome. So Chrome and then Firefox are going to be the browser of choice for programmers and software testers.

Not surprisingly, W3 Schools is the site which claims Chrome is almost 60% of the market. But it will be biased towards power users, programmers and testers. My mother doesn't use a computer. If she did it would be a Windows 7 with Internet Explorer 11 or a Mac with Safari 6.1.

So which statistics site you select depends on your target audience. You don't want to use the statistics from W3 Schools if your target audience is my mom.

What if you have a website and you are going to be updating it or adding another website with the same target audience? Then you have been hopefully gathering statistics on your existing website. Statistics from your existing website will tell you exactly who your users are. If you statistics say that 97% of your users are using Internet Explorer 6 then you need to support Internet Explorer 6. You could make it painless to upgrade your existing user base to a new version of the same browser but odds are it is safer to just support what your users are using.

The only time I would say you might not want to completely follow the statistics on an existing website would be if you are attempting to grow your market and know the new customer base is going to be a different type of user. Let's say that my user base is traditionally Internet Explorer users but I want to start targeting Mac users. I want to make sure that I don't sacrifice existing users for the new user base. So I will continue to value the Internet Explorer users but I want to start looking at statistics for when the operating system is Mac OS X. If I drill down to just Mac OS X users, which is the dominate browser?

In addition to which browser there is also the resolution for the display. This tends to be a little easier to decide if we are talking desktop. You pick the small display which is widely supported. Originally, I would design a website so it looked good on 640x480. Then the average display was 800x600. Later we would start seeing wide displays like 1440x900. I found it easiest to set a minimum size that would be supported. So you don't want so many things on a menu that the menu is longer than say 600 pixels high if you supported 800x600. We can use tools to resize the browser to the different sizes and make sure things still look good. This is easy to do on pretty much any desktop today.

Back in the days of EGA and VGA the number of colours you would support was an issue but today it isn't really something people worry about any more.

This is how I generally recommend clients decide on which desktop browsers to support. It does not really address the idea of mobile browsers. This is an area which is still in flux. I see a lot of different mobile browsers. My own personal experience has been updating my browser can make it more unstable. So if I have a browser which runs on my device I'll stick with it. However, if I get a new device I tend to find the newer browsers come with it and run better than old browsers. So you should be seeing a wider range of browsers on mobile devices than on desktops. Additionally, mobile devices include smart phones and tablets. There are full size tablets, mini tablets, large smart phones, medium smart phones and small smart phones. The different resolutions can vary much more than desktop devices.

Additionally, I cannot take a large device and resize the browser to small sizes. The idea of a window on a desktop doesn't exist on many mobile devices. So testing on the different resolutions is a major concern. Fortunately, with emulators you can run a software emulated version of a device and check that the website displays properly on it. Or you can use features of our existing browser to pretend to be a specific mobile device. This will change the size and User-Agent information to make your web application display as if it was getting rendered on a mobile device. Again, Chrome Inspect has some built-in features in this area which are quite helpful. They should not replace emulators or even better actual devices when doing system testing.

Ideally, in a test lab I will have all the machine with the largest display we want to support. I will then use a virtual computer player like Virtual PC, Parallels, VMware, etc. to emulate all the other operating systems. A Mac works really well because you can use it to run Mac browsers, a virtual PC like Parallels or Fusion to run Windows, Linux and Solaris x86, XCode to emulate iOS devices and Eclipse or IDEA to emulate Android devices.

If the client runs mostly Microsoft products and Mac isn't a concern then you can use Virtual PC on a Windows 7 device to run Windows XP and Windows 7 virtual machines for free.

For virtualizing all the Windows machines right now you can go to http://www.modern.ie/en-us/virtualization-tools#downloads to download virtual machines for Windows XP, Windows 7 and Windows 8. Everthing from Internet Explorer 6 to Internet Explorer 11.

List of Selenium/WebDriver blogs.

I was recently pointed to a nice list of Selenium/WebDriver blogs. If you are looking for more information about Selenium/WebDriver you should check out http://it-kosmopolit.de/Selenium/blog/selenium-blogs/selenium_blogs.php.

I haven't checked out all the links but there appears to be a good start to the list of Selenium/WebDriver blogs available out there.

Generating a file of a specific size

Every once in a while someone is looking for a file of a specific size. Occasionally, it must be real data. If you are transmitting the file and the data will be compressed then the type of data will make a difference.

However, if you just need a file to fill some space or there will be no compression then Windows has a neat little utility called FSUTIL.

The FSUTIL file can be used for a number of things but the nicest feature is creating a new file filled with zero bytes. First, you need to know how many bytes. If you want a file which is 38 gigabytes then you need to figure out how many bytes that is. Technically, it is 1024*1024*1024*38. If you want  a rough idea you can just use 38,000,000,000 but a 38 gigabyte file is really 40,802,189,312. Next is which file you want to hold the data. Let's say you want to create C:\DELETEME.TXT then the full FSUTIL command is:

fsutil file createnew C:\DELETEME.TXT 40802189312

This will create a 38G file is a matter of seconds.

For UNIX, Mac OS X or Linux you can use DD. The DD command is for converting and copying files. If we wanted to create the 38G file with DD the command would be:

dd of=deleteme.txt oseek=79691776

The oseek is the magic. It will seek n*512 bytes. The standard size of a block is 512. So we take 79691776*512, which is 38G. Even easier would be:

dd of=deleteme.txt oseek=38m obs=1024

This will generate a file of 38M * 1024 or 38G. Much easier to figure things on working with values like these. That is, no need for a calculator.

IMPORTANT: after you press enter on the DD command it will take the standard input as its input. So you need to enter CONTROL-D.

The other option for UNIX/Linux/Mac OS X is to copy a file of a set size. The nice thing about this is you can take a file with real data and copy it enough times to make a single file of the correct size. For example, if I have a text file with 512 bytes of real data I can set the if= to that file and make multiple copies of that one file into the output file.

Generating a screen capture when using RemoteWebDriver

Recently I was asked if all the different implementations of WebDriver allow for screen captures. I had a look at the source code and see that almost all of them "implement TakesScreenshot". If the Java class for the driver implements TakeScreenshot then it allows for screen captures. This means a previous posting (http://darrellgrainger.blogspot.ca/2011/02/generating-screen-capture-on-exception.html) there isn't really any need for using Robot to generate the screen capture.

The nicest thing about using the built-in screen capture is that Robot will capture the visible screen. The built-in screen capture will capture the entire window in the browser, including the parts not visible, i.e. things you have to scroll to see.

Shortly after this someone asked of RemoteWebDriver supported screen capture. This was very important because Robot would not capture even the visible part of a REMOTE screen.

A quick check of RemoteWebDriver.java showed that it did NOT implement TakesScreenshot.

Recently however I found a solution. Assuming you have the code:

DesiredCapabilities dc = DesiredCapabilities.firefox();
URL url = new URL("http://localhost:4444/wd/hub");
WebDriver driver = new RemoteWebDriver(url,dc);

You can change the driver to:

WebDriver driver = new Augmenter().augment(new RemoteWebDriver(url,dc));

Now it has the capability to take a screen shot. To make the call, here are the three different outputs:

File f = ((TakesScreenshot)driver).getScreenshotAs(OutputType.FILE);
String s = ((TakesScreenshot)driver).getScreenshotAs(OutputType.BASE64);
byte[] b = ((TakesScreenshot)driver).getScreenshotAs(OutputType.BYTES);

And that is all there is to taking a screenshot with RemoteWebDriver. I have tried this with Firefox and InternetExplorer. I assume it works with the other browsers but I leave that for you to explore.

When trial and error is a bad thing

I'm a hacker. Not the breaking into systems and doing damage hacker. To me a hacker is someone who learns things by trial and error. I will systematically poke away at something which for all intents and purpose is a black box. I hack to learn. I hack things I own. I'll create an instance of something and hack away at it. People I work with will create development, test or staging environments which I will hack.

I do not hack sites I don't own or have permission to hack. This is what differentiates good hackers and bad hackers.

What I do is poke at something. Maybe I'll try changing an input or altering the environment slightly and see how that changes things. I'll keep doing this until a pattern merges. From trying different things I start forming a hypothesis of what is happening inside the black box. If I try something and the result does not fit in my hypothesis, I form a new hypothesis. At some point I usually get a clear understanding of what is happening in the black box without ever seeing what is in the black box.

I essentially look at the symptoms and narrow down what the cause would be.

This is a good use of trial and error. The goal is not to find the input which gives me the desired output. If I stopped the moment I got the desired output, I might think I have the solution but I don't. Case in point, I input 2 and 2 and get 4. My hypothesis is that the black box does addition. At this point my hypothesis is correct. However, if I poke further I might find that inputing 2 and 3 gives me 6. Now I see that it is not addition. New hypothesis is that it is multiplication.

Hacking is really empirical. Unless I try every possible input, I cannot be certain my hypothesis is correct. For example, I might input 1 to a function and it returns 43, I input 2 and get 47, I input 3 and get 53. After inputting the numbers from 1 to 20 I notice all the numbers are prime! My hypothesis is that the function is a prime number generator. However, if I input 41 I get 1763. This is not prime (43 * 41 = 1763). Turns out the function is Euler's formula for finding prime numbers, i.e. n^2 + n + 41. This has been proven to only produce prime numbers when n is less than 40.

Still hacking can be a good thing. Trail and error to find THE answer is never a good thing.

I see a lot of people solving problems as follows:

1. Program or computer not functioning correctly.
2. Change something.
3. If program or computer not functioning correctly go to step 2.
4. Problem solved.

Now maybe they did find the right solution but most often they don't. Later the problem will come back with different symptoms. If I purchased a program from you and you used this method to solve the problem, here is how I see this as a consumer of software:

My car is running slower than normal. I bring it to my mechanic and he does the following:

1. He changes the spark plugs and charges me for that.
2. Car is still running slow.
3. He adjusts the valve on the carburetor and charges me for that.
4. Car is still running slow.
5. He rotates the tires and charges me for that.
6. Car is still running slow.
7. He changes all the fluids and charges me for that.
8. Car is still running slow.
9. Cars today have a lot of electronics, so he disconnects the battery for a week.
10. All my programming, bluetooth, radio stations, clock, GPS, etc. are gone.
11. The car is no longer running slow.
12. Three months later the car is running slow again.
13. My mechanic disconnects the battery for a week.
14. All my programming, bluetooth, radio stations, clock, GPS, etc. are gone.
15. My car is still running slow.

Would you pay for all the work the mechanic did? I think it is safe to say that NO ONE would put up with this. Some people might put up with it until step 12 then find a new mechanic. Others would put up with this until just step 2 or 4. Most of us would not pay for anything after step 2.

I've worked in industries where EVERYONE programs like this. There might be 4 or 5 different vendors and you really don't have any other choice. However, it just takes one guy to write quality software and everyone switches to that other guy. Trying to win back those customers means you have to make up for all the poor software issues PLUS give them some incentive to switch away from the guy who has always given them good software.

So you want to do unit testing

What is a unit test? Wikipedia describes unit testing as testing individual units of code in isolation. If the code has external dependencies, you simulate the dependencies using mock objects.

For example, if I am testing code which gets data from a database, hopefully access to the database is via something like ODBC or JDBC. In which case, it is possible to use a fake database (file system or memory based) rather than say an Oracle or SQL Server driver.

If my database connection is hard coded to a particular machine or assumes the machine is localhost then my first step is to refactor the code to remove this dependency.

Part of the purpose of having unit test cases is so that we can safely change the code and know we didn't break any existing functionality. So if we need to modify the code to be able to add unit tests we have a bit of a Catch-22 situation. The truth of the matter is, if we have been changing the code without unit tests, changing it one more time in order to add unit tests is actually a step in the right direction and no worse than previous development.

Another important feature of unit tests are speed. If I am adding a new feature and I want to be sure it hasn't broken anything, I want to know as soon as possible. I don't want to write the feature, run the tests and check the results tomorrow. Ideally, I want to know in seconds. Realistically, I might have to live with minutes at first.

Test runs should be automated. If I have to make a change and figure out what tests to run, run them and check the results there is a strong chance I will stop running them. Especially if I'm on a tight timeline.

Ideally, I would check in my code. This will fire a trigger which builds my code (not the entire product, just my code) and run the unit tests against it. Putting such a build system in place is a great deal of work but worth the effort. Every minute it takes to create this build system should be weighed against how much time developers spend testing their code before they check in, how many minutes testers spend finding bugs, how much time developers take understanding the bug and fixing it. Numerous studies have shown fixing bugs is much more expensive than never introducing them in the first place.

So what do we need so far?

First, we need a unit test framework. You wouldn't create your own replacement for JDBC/ODBC. So why create your own unit test framework. There are plenty of them out there.

Second, we need mocking frameworks for the technologies we are utilizing. Which mock object frameworks you require depends on what you are using in your application. If it is a web application, you might need to mock out the web server. If it accesses a database, you will need to mock out the database.

Third, we need a build system to automate the running and reporting of the unit tests. Reporting the results is important too. Most systems will either report back to the source control client or send you an email. If the tests run in, literally, seconds, you can afford to reject the checkin if a unit test fails. If it takes more than say 5 seconds, you might want to send an email when a checkin fails.

Fourth, we need commitment from management and the team. If you don't believe there is benefit to unit testing there will be no benefit to unit testing. Training people on how to create good unit tests and maintain them is critical. If I'm starting a new project and writing tests from the beginning it is easy but the majority of you will be adding unit tests to existing code.

The first three things are relatively easy to obtain. There are plenty of technologies and examples of people using them. The fourth requirement is the biggest reason adopting unit testing fails. If you don’t get buy in from everyone involved it just won’t work. The developers need to understand this will benefit them in the long run. The testers need to understand that less testing will be required and they need to focus on things unit testing will not catch. There will always be plenty of things to test. So there should be no fear unit testing will replace  integration or system testing. Management has to understand if they cut timelines for a project, they will not give developers time to write the unit tests. If you reward the Project Manager for getting the project out on time, he will get the project out on time even if it means giving developers no time for unit test creation. As a Project Manager, if reducing the number of issues AFTER the project has shipped is not a metric I’m evaluated on, I’m happy to ship a product which will make the next project difficult to get out on time.

So, you have the tools and you have buy in from everyone. Now what? If you have 100,000+ lines of code, where do you start writing unit tests? The answer is actually really simple. For example piece of code a developer touches, they should add unit tests. Bug fixing is the best place to start. I would FIRST write a unit test which would have caught the bug. Then I’d fix the bug and see the unit test pass.

By focusing on unit tests for bug fixes it reduces the need for regression testing, it focuses on the features customers are using and the developers are in that code anyways. If we need to refactor the code to support unit testing, might as well happen as we are changing the code. The code was broken when  we started the bug fix. So we’ll have to manually test the fix without unit tests. Hopefully, with a unit test in place, it will be the last time we manually test changes to this code.

If we are modifying the code for feature creation, not bug fixing, we want to write unit tests to confirm the current behaviour. Once we have a test which passes with the current code, we can add the feature and the tests should continue to pass.

At this point we know what we need and where to start. So let’s cover some of the how to write a unit test.

First, a unit test is going to be a function/method which calls our code. We want the name of the unit test to reflect what it is testing. When results are published they will go out to the developer but they will also be seen by the backup developer, project management and various other people as well. If I got an email telling me test17() failed I’m going to have to open the code and read what test17() is testing. You added comments and kept them up to date, right? Or course you didn’t. The comments shouldn’t be necessary. The test name should tell me what it is doing. If the test method was called, callingForgotPasswordWhenNoEmailInUserPreferences() then we all know what is being tested.

Second, what failed? Most unit test frameworks has assert statements. There is the basic fail() call but there are also things like AssertTrue, AssertEquals, AssertNotNull, etc. They can be called with just what you are checking or with a message and what you are checking. You don’t want to code any more than you have to but enough that someone receiving the results will know what failed. If the requirement for my software is “When a user clicks the Forgot Password button but they have not set an email address in their preferences, they should be presented with a message telling them to contact the system administrator.” Then the result message from my example here might be something like, “callingForgotPasswordWhenNoEmailInUserPreferences() failed. Was expecting: ‘No email address was set for your account. Please contact the System Administrator.’ but received: ‘No email address.’”. From this is it pretty clear what was expected and what we received instead. Failing to tell the user how to proceed should be considered a show stopper for the customer. On the other hand, if the result was: “callingForgotPasswordWhenNoEmailInUserPreferences() failed. Was expecting: ‘No email address was set for your account. Please contact the System Administrator.’ but received: ‘No email address was set for your account. Please contact the system administrator.’” the customer might consider this acceptable. We might even update the unit test case to ignore case so the test becomes a pass.

Unit test frameworks are pretty well established now. The general structure of a unit test is:

• set up for the test
• run the test
• assert the test passed
• clean up so the next test starts at the same point

The set up would be things like creating mock objects, initializing the inputs for the test, etc. The running of the test would be a call to the method being testing. Next would be an assert statement confirming that we received the expected results or side effect. Finally, clean up (often called tear down) the environment so it is at the exact same condition it was before the set up occurred.

Often you will group similar tests in one test suite. If I have 12 tests and they all require the same set up I will put them all in one suite. The code will then have one setUp() method that creates the environment for each test, one method for each test (12 methods in total for this example) and one tearDown(). The setUp() method will create any mock objects, initial global variables, etc. The test method will create anything particular to that test, call the method being tested then make an assert call. The tearDown() method will then clean up the environment so it is just like it was before the setUp() method was called. This is important because most unit test frameworks do no guarantee the order the tests will be run. Assuming one test starts where a previous test left off is just bad practice. I have worked on a project with 45,000 unit tests. All test are run as part of the nightly build. Rather than running all the tests on one machine, they are distributed to 238 different machines. If they all ran on one machine they would take 378 hours (over 2 weeks) to run. By distributing them over 238 computers they run in approximately 3 hours. However, if test1932 depends on test1931 and the two tests get sent to different machines, test1932 will not run correctly. Each test must be independent of all other tests. This will not seem important at first but 1 year later you might find yourself needing weeks (possibly months) to refactor all your unit tests. Moments like these often cause management to abandon unit testing.

This is unit testing is a nutshell. I will warn you, ‘the devil is in the details.’ Hiring someone who has gone through the pains of setting up a unit test framework is always a good idea. Either find a good consultant or hire someone full time to work on the framework for you. Some unit test frameworks are jUnit for Java, cppUnit for C++, nUnit for .NET, etc. Gerard Meszaros has written an excellent book called “xUnit Test Patterns: Refactoring Test Code”. In it he talks about “Test Smells”. Essentially, you can sometimes look at a piece of code and say, “This code stinks.” A code or test ‘smell’ is an indicating that the code has problems, i.e. it stinks. I have found reading Gerard Meszaros book I know what to look for before I do it. Originally the book was designed for people who created unit tests, found the tests have issues, i.e. they ‘smell’ and are looking to fix them, i.e. refactor. By reading the book, I avoid creating the bad unit tests in the first place.

Good luck and have fun!

Your automation must not dictate your test plan

One of the things I see people getting into automation doing is selecting what to automate or how to test an application based on what the automation tool will let them do. This is a dangerous approach to automation.

The test cases I create for an application are based on what is important to the customer. I want to make sure that the customer experience is a good experience. If I create a set of test cases or use stories which reflect real customer usage of the application then I am most likely to find issues which will affect the customer.

I remember working on a project for 4 years. After 4 years of testing and improving the application we were at a point that over 90% of the features important to the customer were tested and bug free. Of the remaining 10% we knew of most the defects and had a work-around. We were at a point where we were testing extreme edge cases. At this point I found a major defect. The developer looked at the fix and realized the defect had been there since the beginning. In 4 years not one single customer reported this defect. The defect was easy to automate but really added zero value to the application. This is NOT a test case you want to start with when automating.

On another project someone found a defect in a desktop application. The steps to reproduce were:

1. Run an application not involved in the test case at all (Outlook for example)
2. Hand edit a project file using notepad or something other than the application it was intended for
3. Make a very specific change to the file
4. Run the application under test
5. Open the corrupted file
6. Select a feature which relies on the corrupted file
7. A modal dialog appears telling you the file is corrupt, do you wish to repair it.
8. Ignore the dialog
9. Use CTRL-TAB to switch to a different application not involved in the test case at all
10. Click on the application under test in a very specific location on the MDI client window

At this point the modal dialog is hidden behind the window with focus and the window with focus appears to be frozen. It is really waiting for you to respond to the modal dialog. This was a design flaw in the operating system. It was virtually impossible to fix in the application under test without a major re-design. It was highly unlikely a customer would be able to reproduce this defect. When the tester showed me the 'locked' state it only took me a few minutes to figure out what was going on. Our customer was typically a software developer with 7+ years of experience.

This was a useless test case. In both this and the previous test case it was a bad test case regardless of creating it manually or automating it. My point is, the test case came first. Even before we attempted to automate it, we decided whether or not it was a good test case.

Test automation NEVER proceeds test case creation or test planning.

Once you know what you want to test and the steps to testing it, you automate those steps.

This is the second mistake I see people getting into automation making. They decide WHAT they want to test but when they start automating it, the steps they generate with the automation are not the same steps as they would do manually. In this case you have taken the time to create a good set of test cases and thrown them out the door when you start automating. This is not a good idea.

Rather than changing the test case to something which is easy to automate, you need to figure out how to automate the test steps. This is what separates good automation from bad automation.

Many times I have seen a test case automated. It gets run and passes. We ship the application to the customer. He uses the same feature and it fails horribly. Why? Because the steps he used to get to the defect where not the steps we automated. We had a good test case. If an experienced tester had executed the test case manually, they would have found the defect. The person automating the test case just found it easier to automate something close to but not equivalent to the test case.

I am currently using Selenium 2.x with WebDriver. One of the changes from Selenium 1.x to 2.x is that you cannot interact with invisible elements. For example, a common trick on a website is to have an Accept checkbox on a download page. If you accept the terms the Download button becomes visible. In Selenium 1.x I could click on the Download button without clicking the Accept checkbox. The REAL test case was:

1. Go to download page
2. Click Accept checkbox
3. Click Download button
4. Confirm download

What someone would automate with Selenium 1.x was:

1. Go to download page
2. Click Download button
3. Confirm download

The idea is that it saves a step. One less step means quicker to code, runs quicker, one less thing to maintain. You do this a thousand times and it adds up. HOWEVER, the customer would never click on the invisible Download button.

In Selenium 2.x you would get a failure with the shortened test case. People occasionally complain that Selenium 2.x has removed an important feature. They want to know how they can click on the invisible Download button. They come up with these tricky Javascript snippets which will allow Selenium 2.x to 'see' and click the Download button. Is a customer going to create a Javascript snippet, inject it into the page, run it just so they can click the Download button? Is a manually tester going to do this? If the answer is no, then why is our automation doing this? If the manual test case calls for clicking the Accept checkbox then our automation should as well. If clicking the Accept checkbox does not enable the Download button, file a bug and move on to something else.

Finally, automation is all about finding elements on a page, interacting with them (clicking, right clicking, typing, etc.) and checking what happened. As a manual tester you are going to use your eyes and hands to do everything. The test case might have a step like, "Locate the folder called 'My Documents' and double click it." This is really two steps. The automation should locate the folder called 'My Documents', this is step 1. It should double click the element, this is step 2. As a manual tester I find the element by looking for the text 'My Documents'. If this is a web page and the HTML is:

<div id='lsk499s'><a href="...">My Documents</a></div>

I am not going to use the div id to find the element. I'm going to use the text. As a manual tester I used the text to find the element. There is no reason to do anything differently with the automation tool.

What if the web page is made to look like Window Explorer. On the left is a tree view with one of the nodes being 'My Documents' and on the right is a thumbnail view with a 'My Documents' folder. In the manual test case, does it specify which 'My Documents' to double click? If yes, follow the test case. If no, how would you as a tester decide? Do you always go with the thumbnail view? Do you pick randomly? Do you change ever test run? If you are a good manual tester, we want that experience captured by the automation. If I would normally change every test run but I never test the same feature twice in one day, it might be sufficient to say, if the day of the year is even, double click thumbnail else double click tree view. If the automation gets run daily, it will pick a different way each day.

The important thing in all this is that I am a good tester. I write good test plans and test cases. When I decide to automation my good test cases, I should not compromise the quality of my testing just because I am testing it with an automation tool rather than manually.

Happy testing!

.

Using the right tool for the job

From time to time I see people asking questions about how to use an automation tool to do something the tool was never meant to do. For example, how do I use Selenium to get the web page for a site without loading the javascript or CSS?

Selenium is designed to simulate a user browsing a website. When I open a web page with a browser, the website sends me javascript and CSS files. The browser just naturally processes those. If I don't want that, I shouldn't use a browser. If I am not using a browser, why would I use Selenium to send the HTTP request?

That is all the get() method in Selenium does. It opens a connection to the website and sends an HTTP request using the web browser. The website sends back an HTTP response and the browser processes it.

If all I want to do is send the request and get the response back, unprocessed. I don't need a web browser.

So how can I send an HTTP request and get the HTTP response back? There are a number of tools to do this.

Fiddler2: http://www.fiddler2.com/fiddler2/

The way Fiddler works is you add a proxy to your web browser (actually Fiddler does it automatically). Now when you use the web browser, if Fiddler is running, the web browser sends the HTTP request to Fiddler and Fiddler records the request and passes it on to the intended website. The website sends the response back to Fiddler and Fiddler passes it back to the web browser.

You can save the request/response pair and play them back. Before you play the request back you can get it. You can edit the website address, you can edit the context root of the URL and if there is POST data you can get the data as well.

Charles: http://www.charlesproxy.com/

Charles is much like Fiddler2 but there are two main differences. The first is that Charles is not free. You can get an evaluation copy of Charles but ultimately, you need to pay for it. So why would you use Charles? With purchase comes support. If there are things not working (SSL decrypting for example) you can get help with that. Additionally, Fiddler is only available on Windows. Charles works on Mac OS X and Linux as well.

curl: http://curl.haxx.se/

Fiddler and Charles are GUI applications with menus and dialogs. They are intended for interacting with humans. If you are more of a script writer or want something you can add to an automated test, you want something you can run from the command line. That would be curl. Because it is lightweight and command line driven, I can run curl commands over and over again. I can even use it crude for load testing.

The most common place to find curl is checking the contents of a web page or that a website is up and running. There are many command line options (-d to pass POST data, -k to ignore certificate errors, etc.) but the general use is curl -o output.txt http://your.website.com/some/context/root. This will send the HTTP request for /some/context/root to the website your.website.com. A more real example would be:

curl -o output.txt http://www.google.ca/search?q=curl

I could then use another command line tool to parse the output.txt file. Or I could use piping to pipe the output to another program.

wget: http://www.gnu.org/software/wget/

Another nice command line tool is wget. The wget command, like curl, will let you send an HTTP request. The nice thing about wget is that you can use it to crawl an entire website. One of my favourite wget commands is:

wget -t 1 -nc -S --ignore-case -x -r -l 999 -k -p http://your.website.com

The -t sets the number of tries. I always figure if they don't send it to me on the first try they probably won't send it to me ever. The -nc is for 'no clobber'. If there are two files sent with the same name, it will write the first file using the full name and the second file with a .1 on the end. You might wonder, how could it have the same file twice in the same directory? The answer is UNIX versus Windows. On a UNIX system there might be index.html and INDEX.html. To UNIX these are different files but downloading it to Windows I need to treat these as the same file. The -S prints the server reponse header to stderr. It doesn't get saved to the files but lets me see that things are still going and something is being sent back. The --ignore-case option is because Windows ignores case so we should as well. The -x option forces the creation of directories. This will create a directory structure similar to the original website. This is important because two different directories on the server might have the same file name and we want to preserve that. The -r option is for recursive. Keep going down into subdirectories. The -l option is for the number of levels to recurse. If you don't specify it, the default is 5. The -k option is for converting links. If there are links in the pages being downloaded, they get converted. Relative links like src="../../index.html" will be fine. But if they hard coded something like src="http://your.website.com/foo.html" we want to convert this to a file:// rather than go back to the original website. Finally, the -p option says to get entire pages. If the HTML page we retrieve needs other things like CSS files, javascript, images, etc. then the -p option will retrieve them as well.

These are just some of the tools I use when Selenium is not the right tool for the job.

How to do focused testing

After years of software testing I have found one constant... there is never enough time to test everything.

So how do I pick what I'm going to test? Here are a few tricks I use to focus what I'm going to test.

First, test what changed. I like to get a snapshot of the last release (source code) and a snapshot of the current release candidate then compare them. Often the source code is readable enough for any tester to understand what has changed. If you are not a programmer, not familiar with the particular language or the code requires a little indepth knowledge only the programmers have then you might want to talk with some of the developers about what you are looking at.

For example, if I am testing a shopping cart for an online web store and I see changes to a class called TaxRules then I'm going to look at the requirements for tax rules and see if there is anything new. Even if there aren't requirements for tax rules, I can talk to the person who wanted this feature (or improvement) and I can look at the code. Comments will be for humans and I'm human (no really, I am). Good code should also be self documenting. In other words, the variables and method names should almost read like English.

The next trick is to get someone to add a code coverage tool to the web site (or whatever it is you are testing). As you test the application (automated or manually) the coverage tool will record which parts of the code you executed. Look to see of the code which has changed was executed. Set the coverage tool so it records actions within the methods and not just method level data. This way you can see which branch conditions were called. If there is a loop which never gets entered or a portion of an if/elsif/else which does not executed, you know you need another test case. You could even ask the developer to relate the statements missed to user actions. Essentially, get them to tell you what test case you missed. :)

Finally, look at the inputs to the methods. Can you see a way to affect the inputs? Does the method handle situations where the data is bad? Developers think in terms of "make the program do what it is supposed to do." They always make it do the right thing when given the right data. As a tester you need to think in terms of "make sure the program doesn't do what it is not supposed to do." In other words, if you input bad data, it should report a nice, helpful message to the user or just restrict the user's ability to input the bad data.

As you do code coverage, you can keep notes with your test cases. Just a comment about how test case XXX should be run with class YYY is altered.

floating point numbers and representation error

Floating point numbers can be a great source of error. When we think about pure mathematics it includes the concept of infinity. For example, between the numbers 0 and 1 are infinite fractions.

When you store a floating point number in memory it has a limited number of values. This means that it is impossible to represent all possible floating point numbers. Different languages have different ways to represent numbers. The more accurate the floating point numbers are, the more expensive, computationally, using them becomes.

This means that operations like 1.47 - 1.00 might result in 0.46999999997 because the math library cannot represent 0.47 and 0.46999999997 is the closest match.

If the program you are testing is financial and it only deals with dollar and cents, this representation error can be problematic.

As a programmer there is a simple solution to this problem. Use only integers (long or double long if possible) and treat them as cents. So the above example would become:

long n1 = 147;           // $1.47
long n2 = n1 - 100;      // $1.47 - $1.00
long dollars = n2 / 100; // 0
long cents = n2 % 100;   // 47
System.out.printf("$%d.%02d", dollars, cents);

This means if you have a program which uses dollars and cents, you want to check for representation error (assuming the programmer used floating point variables) but you also want to check for issues which might be integer related. So you want to consider things like integer overflow and underflow. See the article Are most binary searches broken? for a discussion of overflow.

New term... soak test

I was just reviewing some documentation for my new job and came across the term soak test. I had never heard the term before and tried to imagine what it might be.

Turns out the term soak test is essentially a load test conducting for a period of time.

A stress test is putting a load on a system which is beyond acceptable limits. The purpose is to see how the system handles the load. All systems will fail but a proper system will fail gracefully.

A load test is putting a load on a system which is within acceptable limits. The purpose is to confirm the system will handle what is considered an acceptable load. For example, when a university opens the online course selection system, thousands of students will log on and try to select their courses for the year. We know how many students are enrolled and there is the theoretical possibility they will all submit selections at the same time. That would be an acceptable load test.

I've looked at load testing over a period of time before but never knew it was called a soak test. From the descriptions I have read, you would put a load on the system and leave it there for a period of time. In some examples they talk about leaving the load for days or weeks.

For me, I would record metrics on the system and plot a graph. If any of the recorded metrics increased over time but with no change in the load, I would suspect a failure. For example, if I am recording used memory and the application continues to increase the amount of memory required, I can extrapolate (a) the application will run out of memory and (b) the approximate time when this will happen. There was never a need to run the test for days or weeks.

Technically, this was a soak test. I was running a test with a load for a period of time. However, I took the data from the load test and projected where it would go over time. By doing that I could see trends and determine when they would have an impact.

For example, on a system which is performing well I might see the memory usage go to 10,000 megabytes then plateau. On a system with a memory leak I would see it go to 10,000 megabytes then it would start to slowly climb. It might be that it leaks 100 megabytes ah hour. If the system had 200,000 megabytes of memory for the application, I can predict it will take 1,900 hours or approximately 80 days for an out of memory error to occur.

If I ran a soak test for 30 days I might assume the application is fine. I would still need to analyze the data to see that it was leaking memory. It would be easier to see the memory leak after 30 days but not always necessary.

Personally, I would run a soak test if I found no obvious problem using load testing and an analysis of the data. If I found a memory leak or data I/O issue without the soak test then a load test is sufficient to find the error.

Once the load test with analysis passes however, it would be prudent to run a soak test if time and resources allowed. You could even run the soak test after releasing the application. If you  do find an issue, create a patch.

Selenium RC [Java] waitForCondition example

I've seen a few people post questions about using the Selenium waitForCondition method.

If you read the documentation for the waitForCondition method (in either Selenium or DefaultSelenium) you will see a mention of selenium.browserbot.getCurrentWindow(). This is key to the wait for condition.

When you start a test case there are two windows which open up. The first window has the Selenium Command History and ability to display the log. I call this the Selenium window. The next window which opens is the window for the application under test or AUT window.

The waitForCondition method has two parameters. The second parameter is the easiest to understand. We don't want the test to wait for ever. If we don't get a response by a certain time we can assume the test failed. The second parameter is the timeout period in milliseconds. So if the condition should occur within 20 seconds, the second parameter is "20000".

The second parameter is the key to the waiting. It is the condition we are waiting for. There are two windows open, Selenium and AUT. The first parameter to waitForCondition is a javascript expression. It will normally be tested on the Selenium window. If you want to evaluate some javascript in the AUT window you need to use: selenium.browserbot.getCurrentWindow(). For example, Let's say this is the AUT window:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Insert title here</title>
</head>
<body>
<script type="text/javascript">

// this variable is the key. When it is false the loading is done
var loading=true;

function loadingDone() {

// hidden the GIF
 document.getElementById("loading").style.visibility="hidden";
 // set the flag to indicate we are done loading

loading=false;
}

// this sets a timer for ten seconds to simulate something

// loading for ten seconds. After ten seconds this will call

// the loadingDone() function.
var seconds=10;
var t=setTimeout("loadingDone()", seconds * 1000);
loading=true;

</script>

<!-- the moment the page loads this will create -->

<!-- a spinning GIF to signal something loading -->
<img id="loading" src="http://www.oshawa.ca/images/loading.gif" style="visibility:visible"/>
</body>
</html>

What this page does is simulate something loading. After 10 seconds it will hide the spinning GIF and set the loading variable to false. So the condition we are waiting for is when loading == false.

This means from waitForCondition we want to look at the variable "selenium.browserbot.getCurrentWindow().loading == false". In other words, the full statement is going to be:

selenium.waitForCondition("selenium.browserbot.getCurrentWindow().loading == false", "20000");

It is important to note that the selenium in selenium.waitForCondition is a Java object in your test code and the selenium in selenium.browserbot is a javascript object running in the web browser.

Let's say we take this HTML and save it to the file /Users/darrell/workspace/LearningSelenium/waitForCondition.html then we can create the following Selenium RC test case:

package com.example.tests;

import org.openqa.selenium.server.RemoteControlConfiguration;
import org.openqa.selenium.server.SeleniumServer;
import com.thoughtworks.selenium.DefaultSelenium;
import com.thoughtworks.selenium.SeleneseTestCase;

public class TestingSelenium extends SeleneseTestCase {

 SeleniumServer ss;
 RemoteControlConfiguration rcc = new RemoteControlConfiguration();

 public void setUp() throws Exception {
  int ssPort = 4444;
  rcc.setPort(ssPort);
  rcc.setTimeoutInSeconds(1200);
  ss = new SeleniumServer(rcc);
  ss.start();
  selenium = new DefaultSelenium("localhost", ssPort, "*safari", "file://");
  selenium.start();
 }

 public void testStub() throws Exception {
  long start, stop;
  selenium.open("/Users/darrell/workspace/LearningSelenium/waitForCondition.html");
  start = System.currentTimeMillis();
  selenium.waitForCondition("selenium.browserbot.getCurrentWindow().loading == false", "300000");
  stop = System.currentTimeMillis();
  System.err.println("Elapsed time: " + ((stop - start) / 1000.0) + " seconds");
 }

 public void tearDown() throws Exception {
  super.tearDown();
  ss.stop();
 }
}

and that is one example of waitForCondition. You will have to know what condition to wait for in your application. If you are using AJAX you might find there is an AJAX object on your application. In it will be a activeRequestCount. When the AJAX.activeRequestCount goes to zero, all the AJAX calls are done. So you can wait for an AJAX call to complete with:



selenium.waitForCondition("selenium.browserbot.getCurrentWindow().AJAX.activeRequestCount == 0", "30000");

Should we use customer data or static test data when testing an application?

My general approach when testing a legacy system is to use the customer data in the system for testing. I'll often find the application will let me do something that creates bad data in a persistent storage (e.g. database). Before we ship version 8.0, development will fix the application so the user interface will not allow the bad data to get stored.

The idea is to control the scenarios by restricting the user input. Unfortunately, the application has been in use for years and that ability to store bad data has been there for years. There is a good chance that someone stored that bad data in the production database. It wasn't a problem in version 7.7 but that nice new feature in 8.0 breaks on the bad data.

So although version 8.0 will stop any future customers storing the bad data, there is still the old data in the system. What we need to do is get rid of the bad data, ideally. If the user interface had of stopped the customer from entering the bad data in the past, what would the customer have done? If the customer would have skipped entering the data altogether then we can delete the data now. If the customer would have changed the input to be valid data (e.g. don't leave fields blank) or if we now store bad inputs as default good inputs (e.g. blank fields are no longer stored as null but as an empty string or -1 for a numeric value). But what if different customers would have done different things. The business rules could become quite complex. Especially if you have hundreds of customers using the same application (e.g. web based service).

So the solution is to leave the bad data and change version 8.0 so it handles bad data.

How do we test this? There are two camps of thought on this. We can use the existing customer data or we can generate static data.

How do I test this? I like to use both ways. The short term solution is to use customer data. Often during my testing it is hard to imagine what bad data might be hiding in the system. For the long term I will create static data which tests all the scenarios I think are valid. Trying to test all the possible inputs would be impossible. So I'm going to test what the application will let me use. If I didn't use customer data and simply used what version 8.0 was willing to generate, I'd miss the hidden bad data.

On pretty much every project I have worked on, using customer data has resulted in old data causing problems that could never be created with the application as it exists when I joined the project.

So create test cases with known, static data but use legacy data as well.

A good defect report

What makes a good defect report? Let's look at the flip side.

A bad defect report is a defect report with insufficient information. If I file a report and the developer has to come back to me for more information, it is a bad report. If I write down incorrect information or make assumptions and report them as facts.

A bad defect report can also be a report with too much information. You might think I am wrong. How can you have too much information? Well, if the report goes on and on the developer will have to wade through all the information you provided to find the relevant information. The developer's time costs the company money. Also, how long time do you spend gathering and entering in information that is not useful or necessary? Your time costs the company money.

So like many things in life, you need balance. But how do you know what is the right amount of information?

Most defect tracking systems will have a form to fill out. As you are doing your testing, think about the spaces on that form. There will be:

• Which project you are testing.
• The version of the project. 
• A summary of the defect. 

This is where you need to start thinking. When someone is trying to find the defect, they will search the summary. Was there an error message? The title of an error dialog, the message in the dialog, the Exception from the log file (e.g. NullPointerException, IllegalFormatException, etc.). Try to put a keyword the Developer or Project Manager will associate with the problem. Error dialogs are good for the Project Manager and log messages are good for the Developer (and many Project Managers as well).

• The severity.

This should not be confused with the priority. The priority is how important fixing the defect is. On most projects this is decided by the Project Manager. The severity is how harmful this is to the customer. For example, let's say I found the following defect: If you enter "Bob" at the top of the page, scroll down two rows, press the Home key, press the shift key repeatedly for at least 37 times, turn the monitor off then back on again, while an audio CD is playing in the computer, it will short circuit the cause the monitor to burst into flames.

The severity of this is VERY high. Any time a computer program can cause your equipment to catch fire I would classify it as the highest possible priority. Now if the defect only happens with THE exact steps above and no other way then the priority would be extremely low, in my opinion.

• Steps to reproduce.

I see a lot of defect tracking systems that do not have a specific section for this  but it is very important. This also requires balance. I might be recording everything I did all day and after 5 hours of testing I find a bug. It would not be reasonable to list all 5 hours of testing. For me to enter all that information would probably take over 5 hours and for the developer to step through everything would take him at probably 5 hours. So this would triple the test time for each defect found. Additionally, I'd have to do all these steps when the defect is resolved to confirm it was fixed.

Often what I find myself doing is trying to find a short set of steps to reproduce the error. The more I know about the application the more I can guess at what went wrong. Hopefully this will help me determine a shorter test case to produce the problem.

• The environment.

If you are testing a web site, the type of web browser is important. The version might matter as well. When I test a web site I will use different browsers for the same test. If I see it looks fine in one browser but not the other I will mention it. What was the error/problem? What are the contributing factors. Is the amount of memory a factor? What about disk space or file permissions?

When you are trying to determine what to include and what not to, i.e. finding that balance, it is usually best to err on the side of caution. Anything you know is a factor must be included in the defect report. Anything you know does not contribute to the defect should be left out. Is there something which you are unsure about? Take a few seconds (maybe minutes) to try a few things to eliminate or confirm the information should be included. If I click on something in a web page and an error dialog pops up, I'd try a different web browser. If we supported four web browser, I have to test all of them anyways, try the test case with all four browsers. Now I know if it is a defect for all browsers or just some browsers and I can include that in the defect report.

On the other hand, if I think it might have something to do with the amount of memory available I might just put the amount of memory available in the defect report. The effort to free up more memory might be more than it is worth. Additionally, how much memory do I need to free up to determine memory was a contributing factor?

Finally, any files associated with the defect should be attached. Any log files but only if there is something in the log that looks unusual, like am Exception message or gibberish.

Should your test environment be dirty or clean?

I'm not talking the state of your desktop. Whether you have pizza boxes, open bags of chips and a bowl of jelly beans is another topic altogether different.

What I want to talk about is what the operating system image should look like. On the one hand, you can setup a Vmware session, install the OS and apply all the patches to it. We'll assume these are bare minimums for any of your customers. Some might skip patches and service packs but the combinations get to difficult to support. It is easier if you just put as system requirements a minimum service pack and tell your customer support to say, "have you applied all the latest patches?"

Let's say we are testing a web application. We want to test with Windows XP SP3, Internet Explorer 7, Internet Explorer 8, Firefox 3.5, Safari 4.0.

We could set up one image with Windows XP SP3, Internet Explorer 7 and Firefox 3.5. On another image we would have Internet Explorer 8 and Safari 4.

The reason I have at least two images is because you cannot have IE7 and IE8 on the same image. Even if you could figure out how to do it, system libraries get updated by IE8 and this affects the way IE7 will run. I could install IE7, FF3.5 and Safari 4.0 on one image then only IE8 on the other image. If I'm doing automation I'd want to distribute the tests evenly so two browsers per image makes more sense than 3 on one image and 1 on the other image.

Ideally, I'd like four images. On web browser for each image. However, you cannot get an image of Windows XP SP3 without IE7 installed. So we have to have at least IE7 and Firefox or IE7 and Safari.

So, what do we install after this? We can install software that a typical customer has on their computer. We could install Microsoft Office. A lot of people use this so it is safe to assume it will be installed.

If I was testing a desktop application, what I installed would start matter a lot more than testing a web application. What files are going to be sent from the server? Maybe we don't want Microsoft Office installed. We want to see what happens if the user doesn't have it installed and we send them a spreadsheet. If we send it with the wrong mime type it could appear as gibberish on the web screen.

The cleaner the environment the more likely we are to catch things like that.

What about plug-ins or development tools? This is a hard call. If you need to call over a developer so they can see the problem on your system, they might need some basic tools to debug what is going on. Especially if they cannot reproduce the problem on there system.

On the other hand, I have seen applications function correctly because development tools were installed. The moment a customer tried to use it, it crashed. Solution, install minimal tools on one of the images, if necessary. If you press F12 while in IE8 you will find they have a good set of development tools native to the browser.

Essentially, there is a fine balance of what you want to install and what you don't. you really have to stop and think about the inner action between what you are installing and the application you are testing.

For example, does installing a plugin to Firefox affect how web pages render? Is it a popular plugin? Most the decisions I make for setting up my environment are based on what would my customer do? and not on what I personally like. If the majority of customers are using a particular plugin, even if I find it useless, I will install that plugin.

The more you know about how the operating systems and applications work, the easier it will be to anticipate unwanted interactions between the environment and the application you are testing.

In summary, I think your test environment should be as clean or dirty as your typical customer.

How to reduce code duplication, part 2

In a previous entry I blogged about how to reduce code duplication. My solution is using libraries.

Let's say I have a web application for writing blogs. Okay, I'm actually going to use the Blogger software as my example. First thing I'm going to do is break the application down into manageable parts. Across the top is:

• Posting
• Settings
• Layout
• Monetize
• View Blog

Each one will take me to a new page. I typically automate with Selenium using Java. So I might create one class for each of these areas. However, when I select Posting I see it has the following subsections:

• New Post
• Edit Posts
• Edit Pages
• Comment Moderation

So I might want to create the following packages:

• com.google.blogger.posting
• com.google.blogger.settings
• com.google.blogger.layout
• com.google.blogger.monetize
• com.google.blogger.viewblog

Then within the com.google.blogger.posting package I'm going to create the following classes:

• NewPost
• EditPosts
• EditPages
• CommentModeration

Next I'm going to focus on one class at a time. Additionally, I'm going to create Java classes to represent the data on the application. For example, in the Posting section, on the New Post page I have the following:

• Title
• Link
• HTML (the HTML text in a text editor)
• Labels
• Post Options
• Reader comments
• Backlinks
• Post date and time
• Edit HTML Line Breaks
• Compose Settings

All this is data for the page so I'd create:

class NewPostForm {
        String title;
        String link;
        String htmlBody;
        String labels;
        PostOptions postOptions;
    }

    class PostOptions {
        boolean allowReaderComments;
        boolean allowBacklinks;
        boolean enterIsLineBreak;
        boolean htmlShownLiterally;
        boolean automaticPostDateTime;
        String scheduledAtPostDate;
        String scheduledAtPostTime;
    }

I would also add getters, setters and constructors to these classes, e.g.

public void setTitle(String title) {
        this.title = title;
    }

    public String getTitle() {
        return title;
    }

Now, if I create a library function to fill in a new post, I don't have to pass in a dozen or more parameters. I can create an instance of NewPostForm, populated it then pass it in to my library function. So back to the NewPost class. This is the library of routines to be used by my test suite:

class NewPost {
        Selenium selenium;

        public NewPost(Selenium selenium) {
            this.selenium = selenium;
        }

        public void gotoNewPost() {
            // if a link to New Post exists then
                // click the link
                // wait for the new post page to appear
                // this might be a page load so WaitForPageToLoad
                // or it might be AJAX so waitForCondition or
                // whatever the case may be
           // else
                // fail the test case
        }

        public void fillInNewPostForm(NewPostForm npf) {
            assertNotNull(npf);
            assertNewPostForm();
            if(npf.getTitle() != null) {
                // set the title input field to npf.getTitle()
            }
            // more of the same thing for each field in npf
        }

        public void assertNewPostForm() {
            assertTrue(selenium.isElementPresent(titleTextFieldLocator));
            assertTrue(selenium.isElementPresent(linkTextFieldLocator));
            // more of the same for all the inputs in the form
        }

        public void clickPublishPost() {
            // if a link for Publish post exists then
                // click the link
                // wait for the post to get published
            // else
                // fail the test case
        }
    }

I've left out things like the values for titleTextFieldLocator but that is easy enough. It would be something like:

String titleTextFieldLocator = "xpath=//input[@id='postingTitleField']";

As you build things up you will have a library of methods for the different 'pages' of the application. So if I wanted to test all the different ways of posting something I could have an instance of NewPost as a class variable for a test case then initialize the variable in the @Before method, just after the initialization of the selenium instance. For example:

class NewPostTestCases extends SeleneseTestCase {
    NewPost np;

    @Before
    public void setUp() throws Exception {
        super.setUp(); // initializes inherited selenium
        np = new NewPost(selenium);
    }

    @TestCase
    public void someTest() throws Exception {
        np.gotoNewPost();
        NewPostForm npf = new NewPostForm();
        npf.setTitle("How to reduce code duplication, part 2");
        npf.setHtmlBody("this is my new blog body");
        npf.setLabels("selenium, java, testing, development");
        np.fillInNewPostForm(npf);
        np.clickPublishPost();
        // do some sort of assertion to confirm the post went okay
    }

    @After
    public void tearDown() throws Exception {
        super.tearDown();
    }
}

And that is the basic idea. I'd used my IDE to generate all the getters/setters and constructors. I'd also let it suggest when I needed to throw an exception (I'd always throw the exception rather than try/catch and deal with it; junit was made assuming the exception gets thrown up to the TestRunner). And there are things like, put the datatypes in a different package, e.g. com.google.blogger.posting.datatypes for the NewPostForm datatype plus let the IDE tell you when to add the appropriate import statements.

So you want to create automated tests

Are you doing manual testing and want to break into automated testing? Are you doing something completely different and want to get a job in automated testing?

One of the problems of getting into a new field is finding someone to give you the time to learn. If you are working on a team of manual testers they often hire someone else to write automated tests. As they write the automated tests there are less and less things you need to test but how does this get you closer to being an automated tester?

The answer is that as the automated test suite grows they will need more people on that team to help maintain and create automated tests. But what if they decide to hire someone new and let you go? Why wouldn't they do that? The truth of the matter is, if you are a good employee they won't WANT to let you go unless necessary. You need to make it unnecessary. You need to learn to become an automated tester.

Another scenario is that everyone is manually testing the project and there is absolutely no plans to do test automation. Maybe the manager tried test automation and it was a huge failure. What would happen if you could automate things and it was a big success? How do you think your manager would feel about you? Pretty good I suspect.

So how do you do it? Set something up at home and start playing. The most important thing is understanding WHY. If you do something and it works but you don't know WHY then when if fails you won't be able to fix it because you never knew why it worked before and doesn't now.

What will you need?

1. An automated test language
2. A development environment
3. Source control
4. Something to test
5. Knowledge of what to test

I'll cover everything but the last point here. What test cases you need, how to create them is a whole set of blog entries.

First, you need to know what automated test language you are going to use. You could use SQA Basic... if you had a few thousand dollars. This is the language used for IBM/Rational Test Studio. If you can afford this then you have the development environment as well.

I suspect you are looking for something FREE. Best thing to do is use open source tools. A good starting point for this is http://sourceforge.net/. There are hundreds if not thousands of projects here. Some haven't been active for years and some are used by companies today. You can search the job ads to see what tools they are hiring for. If they are hiring for say Selenium then there is a good chance that your company could use Selenium as well.

Different open source test automation tools will use different languages. It is usually best to pick a language that is already a programming language. For example, Selenium can be used with Java, Ruby, Perl, C#, etc. or Watir is only Ruby and Watij is only Java.

If you search for "automation" you'll get 720 projects. If you search for "test automation" you'll get everything with test, automation and test automation. It actually gives you thousands of projects. You can also try to browse the projects using the "Find Software" link. From there go to Development, Testing but you'll still get 1782 projects. Finally, use the Filter button to narrow things down. You can keep adding filters. So for me, I'd want Language 'Java', Operating System 'OS X'. This gets me down to 17 projects. From there I'd scan them by hand. You might also want to filter for Development Status. Go for Production/Stable or Mature. You don't want to be dealing with bugs in the test tool while trying to find bugs in your application.

If you are going into automated testing for a web site I can tell you that Watij, WatiR, Selenium are very popular choices right now.

Once you have done some research and know which automated test tool you want to learn you'll need a development environment. If you select C# you are probably looking at buying Microsoft Visual Studio however there are cut down, FREE, versions on the Microsoft website. You just need to hunt for them.

If you select a tool which uses Java, Ruby or Perl you had a wide variety of development environments to select from. I'm currently using Java. I've always like NetBeans but more companies want people with Eclipse knowledge, so I'l using Eclipse.

A good development environment is essentially an editor, project management and a debugger. A project manager to organize all the files, an editor to create the files and a debugger to maintain the files.

Next is source control. What is source control? If you have done any programming you have probably done this:

• Create a small program
• Add some features to it
• Find a problem
• Try to fix it
• Try something else
• Try something else
• Try something else
• Realize you are missing a semicolon, or quote or something simple

Now you are trying to remember all the "try something else" things you did so you can get back to the original code before you made it worse. Ideally, you want to get back to before all the 'try something else' then fix the 'something simple'.

Next time, you write a small program, run it and it works. Let's say the file is called MyProg.java. Before you try to add some more features you copy the file. So now you have MyProg.java and a backup called MyProg1.java. After a few days you have everything from MyProg1.java to MyProg47.java.

You add another feature and it totally breaks the program. You do a lot of "try something" only to realize it was a simple fix. Now you can throw away MyProg.java, rename MyProg47.java to MyProg.java and do the simple fix.

You have created a very primitive source control system. There are much better server/client systems like CVS or Subversion. The way they work is you create a repository (a folder on your computer somewhere or on another computer). When you create a file, like MyProg.java, you add it to the repository. Now you have a back up in the repository. Every time you make a change you can add the change to the repository. At anytime, you can get a list of all the versions in the repository, you can add notes when you save the file and later look up the notes. The best part is you can tag a point in time. If you create 500 files and complete version 1.0 today, file 1 might be version 173, file 2 might be version 37, file 3 might version 1, file 4 might be version 28838, etc. you 'tag' all the files as VER1.0. Two months from now someone wants to add a patch to version 1.0. You have made changes to many files. You can ask source control for all versions tagged with VER1.0. It will restore the system to just like it is today. This way I can have one project for current and one project for past projects. Current project is new features, past project is bug fixing.

If you are using a Mac OS X computer you have CVS built in. You can also download Subversion (SVN) and install it. These source control systems are open source so you don't have to pay for them. CVS was very good but SVN is newer and used by more new projects.

So go out and find Subversion. Download it and create a repository. There are tutorials on it out there so I'll leave that as an exercise for you to do on your own.

The final thing is something to test. Pick a simple web site and start automating it. If you have access to your test environment from work, you could start testing it right now. Otherwise, start automating something like http://www.google.com/

Do hackers make good testers?

Someone on LinkedIn was asking about becoming an ethical hacker.

Back before I got into computers I understood the term hacker to mean someone who learns from trial and error. So a computer hacker was someone who tried things and remembered the results. Now if you started hacking a computer site you probably found things the average user didn't and potentially crashed the site. Because a lot of what was happening on the server was hidden from you, you had no idea what trying different things would do until you tried them. And even then you might not know what actually happened other than you lost the connection.

If I was hacking a web site or piece of software I would need to narrow things down. If I tried something I might know it will be one of four things that caused the resulting crash. If I tried something else, it might have three things in common with the first hack. If it crashed you might say that one thing that is not common to both hacks is not the crash. You'd be wrong. You're assuming there is one crash. Could be there are two different crashes. :)

Keeping your mind open like that is what is needed to be a good hacker. I picture things in terms of Venn diagrams. Overlapping areas are not important as it could be one or the other. The things which can only happen in one area are the interesting bits.

There is something about this way of thinking that makes it easy to find defects. It is almost like detective work. You have some evidence and you need to see if you can find more. You poke and probe to see what falls out. You interrogation the application under test and see what secrets it reveals. You keep track of cause and effect. I do this over a long period of time. You look for patterns.

Case in point, I was testing an application that had a load feature. There was a defect in the load routine. I filed the defect and the developer fixed it. Once I could load, I could save. A similar defect existed in the save routine. Programmers make common mistakes. Is there more than one place that programmer could make the same mistake? I always take the time to find out what the defect was (from a technical standpoint) and what was really happening.

After a while I can almost predict what errors will occur in a program.

The other trait I have and see in other good testers is curiosity. I'm always wondering why things work the way they do. Before computers I'd take apart lawnmowers, watches, bicycles, kitchen stand mixer. When my first hard drive died, took it apart. Hooked it up with the cover off to figure out what was wrong with it. Head was stuck on a scratch on the platter. Smoothed out the scratch and got the hard drive working again. Had some problems with the area I smoothed out but it worked in all other places.

When it came to understanding the Internet, Telecommunications, etc. I started reading things like RFCs, research papers at MIT, JSRs, ISO/ANSI drafts.

As I write this I am reminded of many of my students (I used to teach programming). During computer labs there was always someone who would ask me, "What would happen if I did XXX?" to which I always replied, "Try it and see." It always amazed me that most students would ask rather than just try it.

So if you want to be a good tester. Understand, Try, Hack.

Test Automation Manifesto

I have been recently reading xUnit Test Patterns by Gerard Meszaros. Excellent book with a lot of things I learned the hard way. I also found an article by Gerard et al titled, "The Test Automation Manifesto". The manifesto is:

Automated tests should be:

1. Concise: As simple as possible and no simpler.
2. Self Checking: Test reports its own results; needs no human interpretation.
3. Repeatable: Test can be run many times in a row without human intervention.
4. Robust: Test produces same result now and forever. Tests are not affected by changes in the external environment.
5. Sufficient: Tests verify all the requirements of the software being tested.
6. Necessary: Everything in each test contributes to the specification of desired behavior.
7. Clear: Every statement is easy to understand
8. Efficient: Tests run in a reasonable amount of time.
9. Specific: Each test failure points to a specific piece of broken functionality; unit test failures provide “defect triangulation”
10. Independent: Each test can be run by itself or in a suite with an arbitrary set of other tests in any order.
11. Maintainable: Tests should be easy to understand and modify and extend.
12. Traceable: To and from the code it tests and to and from the requirements.

In his article he talks about bad code 'smells'. A 'smell' is something which you notice again and again as a problem. Even before you have clearly identified it, you can 'sniff' them out.

The first two code smells he talks about are THE reason record and playback test automation doesn't work.

When you use record and playback to automate, it will (a) hard-code test data and (b) duplicate code. If I'm testing an application which requires me to log in before each test, the recorder will record me logging in for each test case. If I do not refactor the log in to a function that all the test cases use and the developer changes the login, I would have a maintenance nightmare. Additionally, if the username and password change and the test data is hard-coded, I'd have to find all the instances and change them.

By the way, the idea of 'refactoring' is to change the code to be more maintainable without changing the way it runs. For example, if I had the following code:

// test#1
// put "darrell" in username text field
// put "password" in password text field
// click Submit button
//
// test#2
// put "darrell" in username text field
// put "password" in password text field
// click Submit button
//
// test#3
// put "darrell" in username text field
// put "password" in password text field
// click Submit button
//
// test#4
// put "darrell" in username text field
// put "password" in password text field
// click Submit button
//

and it worked as expected, I could refactor it into:

// login(username, password)
// put $username in username text field
// put $password in password text field
// click Submit button

// test#1
// login("darrell", "password")
//
// test#2
// login("darrell", "password")
//
// test#3
// login("darrell", "password")
//
// test#4
// login("darrell", "password")
//

This will run pretty much the same but now if the Submit button gets changed to a Login button, I don't have to change all 4 test cases (what if I have 40,000 test cases). I just change:

// login(username, password)
// put username in username text field
// put password in password text field
// click Login button

and it updates all the test cases. This code still has hard-coded data. How about changing this to put the data in a property file:

# login data
username=darrell
password=password

then change my code to:

// read property file
// username = getProperty("username")
// password = getProperty("password")

// test#1
// login(username, password)
//
// test#2
// login(username, password)
//
// test#3
// login(username, password)
//
// test#4
// login(username, password)
//

Now I can edit the property file if I want to change the username or password.

I picked property file to hold the test data but it could just as easily been a database, spreadsheet, text file, compiled resource, global variables, etc.