So you want to do unit testing

What is a unit test? Wikipedia describes unit testing as testing individual units of code in isolation. If the code has external dependencies, you simulate the dependencies using mock objects.

For example, if I am testing code which gets data from a database, hopefully access to the database is via something like ODBC or JDBC. In which case, it is possible to use a fake database (file system or memory based) rather than say an Oracle or SQL Server driver.

If my database connection is hard coded to a particular machine or assumes the machine is localhost then my first step is to refactor the code to remove this dependency.

Part of the purpose of having unit test cases is so that we can safely change the code and know we didn't break any existing functionality. So if we need to modify the code to be able to add unit tests we have a bit of a Catch-22 situation. The truth of the matter is, if we have been changing the code without unit tests, changing it one more time in order to add unit tests is actually a step in the right direction and no worse than previous development.

Another important feature of unit tests are speed. If I am adding a new feature and I want to be sure it hasn't broken anything, I want to know as soon as possible. I don't want to write the feature, run the tests and check the results tomorrow. Ideally, I want to know in seconds. Realistically, I might have to live with minutes at first.

Test runs should be automated. If I have to make a change and figure out what tests to run, run them and check the results there is a strong chance I will stop running them. Especially if I'm on a tight timeline.

Ideally, I would check in my code. This will fire a trigger which builds my code (not the entire product, just my code) and run the unit tests against it. Putting such a build system in place is a great deal of work but worth the effort. Every minute it takes to create this build system should be weighed against how much time developers spend testing their code before they check in, how many minutes testers spend finding bugs, how much time developers take understanding the bug and fixing it. Numerous studies have shown fixing bugs is much more expensive than never introducing them in the first place.

So what do we need so far?

First, we need a unit test framework. You wouldn't create your own replacement for JDBC/ODBC. So why create your own unit test framework. There are plenty of them out there.

Second, we need mocking frameworks for the technologies we are utilizing. Which mock object frameworks you require depends on what you are using in your application. If it is a web application, you might need to mock out the web server. If it accesses a database, you will need to mock out the database.

Third, we need a build system to automate the running and reporting of the unit tests. Reporting the results is important too. Most systems will either report back to the source control client or send you an email. If the tests run in, literally, seconds, you can afford to reject the checkin if a unit test fails. If it takes more than say 5 seconds, you might want to send an email when a checkin fails.

Fourth, we need commitment from management and the team. If you don't believe there is benefit to unit testing there will be no benefit to unit testing. Training people on how to create good unit tests and maintain them is critical. If I'm starting a new project and writing tests from the beginning it is easy but the majority of you will be adding unit tests to existing code.

The first three things are relatively easy to obtain. There are plenty of technologies and examples of people using them. The fourth requirement is the biggest reason adopting unit testing fails. If you don’t get buy in from everyone involved it just won’t work. The developers need to understand this will benefit them in the long run. The testers need to understand that less testing will be required and they need to focus on things unit testing will not catch. There will always be plenty of things to test. So there should be no fear unit testing will replace  integration or system testing. Management has to understand if they cut timelines for a project, they will not give developers time to write the unit tests. If you reward the Project Manager for getting the project out on time, he will get the project out on time even if it means giving developers no time for unit test creation. As a Project Manager, if reducing the number of issues AFTER the project has shipped is not a metric I’m evaluated on, I’m happy to ship a product which will make the next project difficult to get out on time.

So, you have the tools and you have buy in from everyone. Now what? If you have 100,000+ lines of code, where do you start writing unit tests? The answer is actually really simple. For example piece of code a developer touches, they should add unit tests. Bug fixing is the best place to start. I would FIRST write a unit test which would have caught the bug. Then I’d fix the bug and see the unit test pass.

By focusing on unit tests for bug fixes it reduces the need for regression testing, it focuses on the features customers are using and the developers are in that code anyways. If we need to refactor the code to support unit testing, might as well happen as we are changing the code. The code was broken when  we started the bug fix. So we’ll have to manually test the fix without unit tests. Hopefully, with a unit test in place, it will be the last time we manually test changes to this code.

If we are modifying the code for feature creation, not bug fixing, we want to write unit tests to confirm the current behaviour. Once we have a test which passes with the current code, we can add the feature and the tests should continue to pass.

At this point we know what we need and where to start. So let’s cover some of the how to write a unit test.

First, a unit test is going to be a function/method which calls our code. We want the name of the unit test to reflect what it is testing. When results are published they will go out to the developer but they will also be seen by the backup developer, project management and various other people as well. If I got an email telling me test17() failed I’m going to have to open the code and read what test17() is testing. You added comments and kept them up to date, right? Or course you didn’t. The comments shouldn’t be necessary. The test name should tell me what it is doing. If the test method was called, callingForgotPasswordWhenNoEmailInUserPreferences() then we all know what is being tested.

Second, what failed? Most unit test frameworks has assert statements. There is the basic fail() call but there are also things like AssertTrue, AssertEquals, AssertNotNull, etc. They can be called with just what you are checking or with a message and what you are checking. You don’t want to code any more than you have to but enough that someone receiving the results will know what failed. If the requirement for my software is “When a user clicks the Forgot Password button but they have not set an email address in their preferences, they should be presented with a message telling them to contact the system administrator.” Then the result message from my example here might be something like, “callingForgotPasswordWhenNoEmailInUserPreferences() failed. Was expecting: ‘No email address was set for your account. Please contact the System Administrator.’ but received: ‘No email address.’”. From this is it pretty clear what was expected and what we received instead. Failing to tell the user how to proceed should be considered a show stopper for the customer. On the other hand, if the result was: “callingForgotPasswordWhenNoEmailInUserPreferences() failed. Was expecting: ‘No email address was set for your account. Please contact the System Administrator.’ but received: ‘No email address was set for your account. Please contact the system administrator.’” the customer might consider this acceptable. We might even update the unit test case to ignore case so the test becomes a pass.

Unit test frameworks are pretty well established now. The general structure of a unit test is:

• set up for the test
• run the test
• assert the test passed
• clean up so the next test starts at the same point

The set up would be things like creating mock objects, initializing the inputs for the test, etc. The running of the test would be a call to the method being testing. Next would be an assert statement confirming that we received the expected results or side effect. Finally, clean up (often called tear down) the environment so it is at the exact same condition it was before the set up occurred.

Often you will group similar tests in one test suite. If I have 12 tests and they all require the same set up I will put them all in one suite. The code will then have one setUp() method that creates the environment for each test, one method for each test (12 methods in total for this example) and one tearDown(). The setUp() method will create any mock objects, initial global variables, etc. The test method will create anything particular to that test, call the method being tested then make an assert call. The tearDown() method will then clean up the environment so it is just like it was before the setUp() method was called. This is important because most unit test frameworks do no guarantee the order the tests will be run. Assuming one test starts where a previous test left off is just bad practice. I have worked on a project with 45,000 unit tests. All test are run as part of the nightly build. Rather than running all the tests on one machine, they are distributed to 238 different machines. If they all ran on one machine they would take 378 hours (over 2 weeks) to run. By distributing them over 238 computers they run in approximately 3 hours. However, if test1932 depends on test1931 and the two tests get sent to different machines, test1932 will not run correctly. Each test must be independent of all other tests. This will not seem important at first but 1 year later you might find yourself needing weeks (possibly months) to refactor all your unit tests. Moments like these often cause management to abandon unit testing.

This is unit testing is a nutshell. I will warn you, ‘the devil is in the details.’ Hiring someone who has gone through the pains of setting up a unit test framework is always a good idea. Either find a good consultant or hire someone full time to work on the framework for you. Some unit test frameworks are jUnit for Java, cppUnit for C++, nUnit for .NET, etc. Gerard Meszaros has written an excellent book called “xUnit Test Patterns: Refactoring Test Code”. In it he talks about “Test Smells”. Essentially, you can sometimes look at a piece of code and say, “This code stinks.” A code or test ‘smell’ is an indicating that the code has problems, i.e. it stinks. I have found reading Gerard Meszaros book I know what to look for before I do it. Originally the book was designed for people who created unit tests, found the tests have issues, i.e. they ‘smell’ and are looking to fix them, i.e. refactor. By reading the book, I avoid creating the bad unit tests in the first place.

Good luck and have fun!

Technical Debt

What is technical debt? 

For the past ten years I have been hearing the expression "Technical Debt" in reference to programming. But what is technical debt?

At the same time I noticed the best project managers would have a summary chart for upper management. It broke things down into red light, yellow light or green light. The idea was that upper management have a lot of information to sift through so we needed to keep it simple.

I think the same idea applies to technical debt. You don't need a precise measurement of technical debt but you need to know if it is growing (red light), stable but still there (yellow light) or reducing (green light).

This still leaves the question, what is technical debt? If I'm running a business and my business needs a piece of machinery to finish a project. I cannot get money from the customer until I deliver the product but I need money to buy the machine to finish the project. So I get a small business loan, buy the machine, finish the project and sell it to the customer. At this point I have debt. Is the interest on the debt acceptable? If the interest is equal to the profit I made on the sale then I'll never get ahead (yellow light). If the interest is greater than the profit I made then I'll slowly go deeper and deeper into debt (red light). But if the interest is less than the profit I made, I can pay down or pay off my debt. BEFORE I get the loan, I need to know the profit will be greater than the interest on the loan. I need to know I will still turn a profit and can get out of debt.

Technical debt is similar. It MUST be a conscience decision to do something which is not good from a development point of view but which I'll be able to correct after the product is out the door. For example, I'm going to pick an inexpensive or easy to implement technology which I know will not scale. I know it will exceed the current load. I can predict, if successful, this choice will be a bottleneck in 18 months. In other words, the 'interest' on this 'loan' will eat away all my profit in 18 months. The long I wait to change to a more scalable technology the less profitable my project will be.

Worse would be to not only fail to pay off or at least pay down this technical debt but to incur more technical debt. Most people would like to have a nice home, a good car, annual trips to Europe or the Caribbean, send their kids to the best schools, etc. but would you use a credit card to pay for it all? When the bill collector comes knocking would you apply for another credit card to pay off the first credit card? Hopefully, you answered no. Even if you didn't, credit bureaus like Equifax or TransUnion would quickly make it impossible for you to get more credit.

Unfortunately, in software development there is no Equifax or TransUnion. Creating technical debt is a LOT easier than creating financial debt. This means you need to monitor your technical debt and make sure you are paying it down.

For short periods of time your technical debt might rise but the over all trend should be a reduction.

What is NOT technical debt? 

It is MORE important to understand what is not technical debt. A lot of people will chalk things up to technical debt when it is really lack of planning.

It isn't even poor planning but more a total lack of planning. Many times I have worked on projects where the project manager (or above) have made it very clear that the project must ship by a specific date. No excuses. What are you telling the programmers? You are telling them you don't care what the cost it, it has to ship by a specific date. The truth is that no one REALLY means 'money is no object'.

If you don't look at the cost of a technical compromise and just implement it, it is the equivalent of going to the local loan shark and signing a piece of paper without reading it. No successful business person would do this but I have seen Fortune 500 companies do the technical equivalent.

As a programmer, if my management tells me something has to be completed by a specific date, no matter what then they are telling me technical debt is no object. Take on whatever technical debt is necessary to get the project done on time. As a senior programmer I know they don't REALLY mean this. However, I have worked in companies where I tried to point out the technical debt is too high only to be told they don't want to hear excuses. Another programmer will swoop in and tell them he can get it done. I get demoted, the other programmer gets promoted.

Inevitably, the project ships on time and under budget. The programmer who swooped in and saved the day might even get a bonus. He might even do this again on the next release of the product. But then I noticed a pattern. Sooner or later, the programmer does a lateral shift to another department or worse, leaves the company to go work for the competition. Shortly after his departure from the project, the new developer starts trying to point out how much technical debt the previous programmer created. For example, a 50,000+ line function which uses lots of GOTOs and stack manipulation to make things work. Adding a new feature which should take 2 days to implement takes months and introduces dozens of new bugs in other features.

Reducing technical debt.

So what if you already have technical debt, planned or otherwise? You need to pay it down. I worked at one company where they went to their customers and let them know they had incurred a lot of unplanned technical   debt. They needed to pay it down. This meant the product would not be growing new features but it would be more stable and scalable. Most the customers looked at the market and believed we could still stay ahead of the competition plus the cost of re-training staff on a new product, creating a business relationship with a new company, etc. would be more costly than giving us a chance. Some major clients walked away and we took a big hit. The clients who did give us a second chance wanted to see that we had a plan. They wanted to see how we would make sure it didn't happen again. They wanted to see a road map of how we'd get back to adding new features. We had to regain their trust.

Hopefully you never get to this point. You want to start addressing unplanned technical debt before it gets this bad. But how do you do it?

Try and leave this world a little better than you found it. - Robert Baden-Powell.

This is a quote from the father of scouting. The idea was, when you went camping to leave the camp site in better condition than you found it. If you show up at a camp site and someone has left some garbage, pick up their garbage as well as any garbage you create. If they trampled some saplings, plant new saplings.

I used to be a tradesman. Many homes in North America have aluminium wiring. Aluminium wiring was used because copper was in high demand and electricians could save money using aluminium wiring. However, the aluminium cannot handle as high a load. Homes were 40 to 60 Amp service. Today we are finding homes with 100 to 200 Amp service. The aluminium wiring cannot handle the higher amperage and fires occur.

New home owners buy a used home only to find out that it has aluminium wiring. At first, insurance companies were telling home owners you had to replace all the wiring or no insurance. Mortgage companies would then say, no insurance == no mortgage. So the home owner needed to come up with $50,000 to $70,000 to re-wire their new home.

This would be the equivalent of saving, I have too much technical debt so for the next year I'm going to add no new features and just refactor the code to remove technical debt. After a year the product will look the same from a customer's point of view but you will have spent hundreds of thousands of dollars. No customer is going to pay extra for a product that looks the same from their perspective. So you are out the money. Just like a new home owner saddled with $50,000 of extra debt on day 1 of their new home, this sort of debt can ruin a company. There is no way most companies can afford to take a year to remove technical debt.

The insurance companies soon realized they were asking the impossible and losing a LOT of potential customers. So they told the new home owners that whenever they renovated a room they needed to make removing the aluminium wiring part of the project.

Software companies need to do the same thing. Rather than give the customer nothing and spend 100% of your time reducing technical debt, whenever you adding a feature you should remove the technical debt as part of adding the feature. In other words, try and leave the software a little better than you found it.

Rather than removing all technical debt in a year it might take you 3 to 5 years but you are still adding new features and therefore getting revenue from customers.

How much time do you spend removing technical debt and how much time adding new features? This is a judgement call. You have to add enough new feature to make the customer feel they are getting their money's worth but leave enough time to reduce the technical debt.

Remember, it might have taken you 5 to 10 years to create the technical debt. It is not unreasonable to take a few years to remove it.

Your automation must not dictate your test plan

One of the things I see people getting into automation doing is selecting what to automate or how to test an application based on what the automation tool will let them do. This is a dangerous approach to automation.

The test cases I create for an application are based on what is important to the customer. I want to make sure that the customer experience is a good experience. If I create a set of test cases or use stories which reflect real customer usage of the application then I am most likely to find issues which will affect the customer.

I remember working on a project for 4 years. After 4 years of testing and improving the application we were at a point that over 90% of the features important to the customer were tested and bug free. Of the remaining 10% we knew of most the defects and had a work-around. We were at a point where we were testing extreme edge cases. At this point I found a major defect. The developer looked at the fix and realized the defect had been there since the beginning. In 4 years not one single customer reported this defect. The defect was easy to automate but really added zero value to the application. This is NOT a test case you want to start with when automating.

On another project someone found a defect in a desktop application. The steps to reproduce were:

1. Run an application not involved in the test case at all (Outlook for example)
2. Hand edit a project file using notepad or something other than the application it was intended for
3. Make a very specific change to the file
4. Run the application under test
5. Open the corrupted file
6. Select a feature which relies on the corrupted file
7. A modal dialog appears telling you the file is corrupt, do you wish to repair it.
8. Ignore the dialog
9. Use CTRL-TAB to switch to a different application not involved in the test case at all
10. Click on the application under test in a very specific location on the MDI client window

At this point the modal dialog is hidden behind the window with focus and the window with focus appears to be frozen. It is really waiting for you to respond to the modal dialog. This was a design flaw in the operating system. It was virtually impossible to fix in the application under test without a major re-design. It was highly unlikely a customer would be able to reproduce this defect. When the tester showed me the 'locked' state it only took me a few minutes to figure out what was going on. Our customer was typically a software developer with 7+ years of experience.

This was a useless test case. In both this and the previous test case it was a bad test case regardless of creating it manually or automating it. My point is, the test case came first. Even before we attempted to automate it, we decided whether or not it was a good test case.

Test automation NEVER proceeds test case creation or test planning.

Once you know what you want to test and the steps to testing it, you automate those steps.

This is the second mistake I see people getting into automation making. They decide WHAT they want to test but when they start automating it, the steps they generate with the automation are not the same steps as they would do manually. In this case you have taken the time to create a good set of test cases and thrown them out the door when you start automating. This is not a good idea.

Rather than changing the test case to something which is easy to automate, you need to figure out how to automate the test steps. This is what separates good automation from bad automation.

Many times I have seen a test case automated. It gets run and passes. We ship the application to the customer. He uses the same feature and it fails horribly. Why? Because the steps he used to get to the defect where not the steps we automated. We had a good test case. If an experienced tester had executed the test case manually, they would have found the defect. The person automating the test case just found it easier to automate something close to but not equivalent to the test case.

I am currently using Selenium 2.x with WebDriver. One of the changes from Selenium 1.x to 2.x is that you cannot interact with invisible elements. For example, a common trick on a website is to have an Accept checkbox on a download page. If you accept the terms the Download button becomes visible. In Selenium 1.x I could click on the Download button without clicking the Accept checkbox. The REAL test case was:

1. Go to download page
2. Click Accept checkbox
3. Click Download button
4. Confirm download

What someone would automate with Selenium 1.x was:

1. Go to download page
2. Click Download button
3. Confirm download

The idea is that it saves a step. One less step means quicker to code, runs quicker, one less thing to maintain. You do this a thousand times and it adds up. HOWEVER, the customer would never click on the invisible Download button.

In Selenium 2.x you would get a failure with the shortened test case. People occasionally complain that Selenium 2.x has removed an important feature. They want to know how they can click on the invisible Download button. They come up with these tricky Javascript snippets which will allow Selenium 2.x to 'see' and click the Download button. Is a customer going to create a Javascript snippet, inject it into the page, run it just so they can click the Download button? Is a manually tester going to do this? If the answer is no, then why is our automation doing this? If the manual test case calls for clicking the Accept checkbox then our automation should as well. If clicking the Accept checkbox does not enable the Download button, file a bug and move on to something else.

Finally, automation is all about finding elements on a page, interacting with them (clicking, right clicking, typing, etc.) and checking what happened. As a manual tester you are going to use your eyes and hands to do everything. The test case might have a step like, "Locate the folder called 'My Documents' and double click it." This is really two steps. The automation should locate the folder called 'My Documents', this is step 1. It should double click the element, this is step 2. As a manual tester I find the element by looking for the text 'My Documents'. If this is a web page and the HTML is:

<div id='lsk499s'><a href="...">My Documents</a></div>

I am not going to use the div id to find the element. I'm going to use the text. As a manual tester I used the text to find the element. There is no reason to do anything differently with the automation tool.

What if the web page is made to look like Window Explorer. On the left is a tree view with one of the nodes being 'My Documents' and on the right is a thumbnail view with a 'My Documents' folder. In the manual test case, does it specify which 'My Documents' to double click? If yes, follow the test case. If no, how would you as a tester decide? Do you always go with the thumbnail view? Do you pick randomly? Do you change ever test run? If you are a good manual tester, we want that experience captured by the automation. If I would normally change every test run but I never test the same feature twice in one day, it might be sufficient to say, if the day of the year is even, double click thumbnail else double click tree view. If the automation gets run daily, it will pick a different way each day.

The important thing in all this is that I am a good tester. I write good test plans and test cases. When I decide to automation my good test cases, I should not compromise the quality of my testing just because I am testing it with an automation tool rather than manually.

Happy testing!

.

Organizing your automation

It has been a while since I have posted something to the blog. Life has been keeping me busy with things other than software testing.

I've been to a number of interviews in the past few weeks and I've been asked a number of questions. One of those questions was regarding traceability. If I have a number of automated tests and a set of requirements, how do I connect the two and how do I determine what has been run and what has not?

The first part to this is how are the requirements organized. If there is a tracking system and some unique identifier for the requirements, I want to use that identifier in my test automation. For example, a previous job the defect tracking system was used to manage requirements. In addition to categories like 'defect', 'enhancement', etc. there was a 'new feature' category. The business analyst would file a 'new feature' request into the system. Using mock ups and text descriptions, they would flesh out the requirements.

Each defect report has a unique tracking number. I would need to incorporate this number into the test automation. There might be one number for the requirement but a number of different test cases. Additionally, the number might be just a number, e.g. 28364. In the automation environment this might not stand out as a requirement. For this reason I would develop a convention where all requirements references would start with REQ-. Thus REQ-28364 could get added to the test case automation.

Ideally, you want the automation to be self-explanatory. If a test case fails, it would be helpful to know what failed without having to look up the requirements documentation. With automation like Selenium RC or Watij (both using Java) I can name the test case class the same as the requirement number, e.g. REQ-28364 but if I am looking at the test suite or test results it might not be obvious what this is. So I would create an annotation @REQ and put the requirement information in the comments of the source code.

The name of the class can then be used to indicate what the new feature is. The name of each test case would be a description of what is being tested. For example, if I'm adding Print support to an editor I might have the class name "PrintSupport" or "NewFeaturePrintSupport". The test cases might be:

• canPrintAllPages
• canPrintAllEvenPages
• canPrintAllOddPages
• canPrintRangeOfPages
• canCancelPrintDialog

When when I look at the results for the test run I would see:

• PrintSupport->canPrintAllPages: Pass
• PrintSupport->canPrintAllEvenPages: Pass
• PrintSupport->canPrintAllOddPages: Pass
• PrintSupport->canPrintRangeOfPages: Pass
• PrintSupport->canCancelPrintDialog: Fail

Very easy to see what is not working and what is.

The most important thing for tying requirements and automation together is creating a convention and sticking to it. To help you stick to it, edit the templates for the IDE. Whenever I create a new Test Case class, the template will have an @REQ field in the comments at the top of the class. I can even go one step further and have source control check for the @REQ field. If the field does not exist or it is blank, the check in of my automation will fail with an error telling me to add a requirement reference to the source code.

One advantage of test automation - part 2

In my previous blog One advantage of test automation I talked about the saves of writing once and running many times. Essentially, if I have 8 configurations, it will take me 480 hours to test everything and I will test everything 12 times in a release I am looking at 5,760 hours of testing. Let's say the cost of a manual tester is $30/hour (salary, benefits, computer equipment, office space, vacation pay, support staff, etc.) then we are looking at $172,800 to manually test the application for the current release.

If I can write the automation once and run it 8 times (once for each configuration) I'm going to see significant savings. If automated testers cost twice as much, I'm going to see a cost savings of 4 times. Still looking good. So what I'm looking for is the cost of creating the initial framework but the cost of maintaining that framework needs to be less than 4 times as long as manual testing.

Here are four scenarios I have observed (the amount noted is the total cost from the beginning of the release):

Scenario #1

Iteration #1: manual=X, automated=X
Iteration #2: manual=2X, automated=2X
Iteration #3: manual=3X, automated=3X
Iteration #4: manual=4X, automated=4X
Iteration #5: manual=5X, automated=5X
Iteration #6: manual=6X, automated=6X
Iteration #7: manual=7X, automated=7X
Iteration #8: manual=8X, automated=8X
Iteration #9: manual=9X, automated=9X
Iteration #10: manual=10X, automated=10X
Iteration #11: manual=11X, automated=11X
Iteration #12: manual=12X, automated=12X

What is happening in scenario #1 is record and playback. The idea is that you buy the software, you record one iteration then play it back over and over. The cost of recording is the same as manual testing. You turn on record then manually test the application. Stop recording and save it. For the next iteration you simple play the recorded first iteration. You find there are timing issues. Changes to the application are causing tests to no longer work. Debugging what is wrong takes time. If you are lucky, it takes around the same amount of time as manual testing. But you paid for the software and training your staff. Let's say that is $20,000. You could have saved $20,000 if you just stuck to manual testing.

Scenario #2

Iteration #1: manual=X, automated=2X
Iteration #2: manual=2X, automated=4X
Iteration #3: manual=3X, automated=6X
Iteration #4: manual=4X, automated=8X
Iteration #5: manual=5X, automated=10X
Iteration #6: manual=6X, automated=12X
Iteration #7: manual=7X, automated=14X
Iteration #8: manual=8X, automated=16X
Iteration #9: manual=9X, automated=18X
Iteration #10: manual=10X, automated=20X
Iteration #11: manual=11X, automated=22X
Iteration #12: manual=12X, automated=24X

What is happening in scenario #2 is poor planning at the start of the automation. Not enough time was spent designing and implementing the test framework. Every time there is a new release of the application, the cost to maintain the automation is as much as the initial creation. We are not leveraging anything we did from the first iteration. Usually test automation is seen as a failure around the third or fourth iteration. When this happens, management is usually pretty apprehensive about trying test automation again. They didn't expect these sort of losses. If we use the initial estimate that manual testing costs $172,800 and assuming we abandoned test automation after iteration 3. The total cost of attempting automation is $216,000 ($43,200 more than just doing manual testing). And this is assuming we called it quits after iteration 3. Each iteration we hesitate on is costing the company an additional $14,400.

It is understandable why so many companies become disillusioned with test automation. Companies selling test automation software will pitch it as saving you 8 times the cost of manual testing. It looks like the cost of automation is going to be $21,600. Even if you double that and add $10,000 for the software ($53,200) it is better than one third the cost of manual testing. So you are expecting $55,000 and it ends of costing you $210,000 or more.

Scenario #3

Iteration #1: manual=X, automated=5X
Iteration #2: manual=2X, automated=6X
Iteration #3: manual=3X, automated=7X
Iteration #4: manual=4X, automated=8X
Iteration #5: manual=5X, automated=9X
Iteration #6: manual=6X, automated=10X
Iteration #7: manual=7X, automated=11X
Iteration #8: manual=8X, automated=12X
Iteration #9: manual=9X, automated=13X
Iteration #10: manual=10X, automated=14X
Iteration #11: manual=11X, automated=15X
Iteration #12: manual=12X, automated=16X

In scenario #3 we spent time creating a test framework but for some reason the cost of maintaining the test framework is the same as manual testing. Sometimes this is because we haven't really thought out the design or the application we are testing is in flux. Some applications are not ready for test automation. If your test automation efforts are failing, you need to look at why. If the application is changing so much that automators need to do some creative coding (which becomes difficult to maintain) or they end up throwing away a lot of the previous iteration automation, then you need to look at stabilizing the development process. Another thing you might have done wrong at this point is write different code for the different web browsers. If I have methods with:

if IE6 then do X
else if IE7 then do Y
else if FF3 then do Z

Or if I actually write one method for IE and another for Firefox then you are not really taking advantage of write once and run many. Additionally, you don't want to be too creative with the automation code. If you have to use a very complex regular expression to find a text field on all the different browsers then maybe the application is not ready for automation and you need to work with development to make the application friendlier to automation.

Scenario #4

Iteration #1: manual=X, automated=5X
Iteration #2: manual=2X, automated=5.5X
Iteration #3: manual=3X, automated=6X
Iteration #4: manual=4X, automated=6.5X
Iteration #5: manual=5X, automated=7X
Iteration #6: manual=6X, automated=7.5X
Iteration #7: manual=7X, automated=8X
Iteration #8: manual=8X, automated=8.5X
Iteration #9: manual=9X, automated=9X
Iteration #10: manual=10X, automated=9.5X
Iteration #11: manual=11X, automated=10X
Iteration #12: manual=12X, automated=10.5X

In scenario #4 we spent time creating a test framework and we were able to leverage the work in the previous iteration for the next iteration. Maintaining the automated test framework and adding new functionality after the initial iteration now costs us half as much as manual testing. At iteration #9 we are breaking even. By the end of the release we are looking at a cost savings of $21,600. This does not seem like a lot but if your automation team gets good at this or you hire someone with the right experience you can actually increase the saving. The cost of maintenance might not be linear. It could grow with each iteration. Or you might realize even more savings with someone who is already familiar with the pitfalls of automating.

Things to look for that might give you additional cost savings:

• Someone with experience working with development
• Someone with general experience creating automation frameworks
• Someone with experience automating a similar application (web based, desktop application, mobile device, etc.)
• Someone familiar with the automation tool you are using.

Obviously, someone who uses the same tool, has written automation for an application using similar technologies, a proven record of creating automation frameworks would be ideal and can work with development if the application is not friendly to automation tools.

I would say the most important factor is someone with experience working with development. If they are not comfortable working with development they might be set up for failure and no way to prevent it. Additionally, good automation is development. If the test automator is a developer, they should be comfortable working with development to make the application more amenable to automation.

Next would be experience creating an automation framework is the second most important factor. You can easily land yourself in scenario #2 if you don't know how to create a good test framework.

The second least important factor is testing similar applications. The technologies and problems for web testing are very different from desktop application testing. Additionally, if an application is using certain third party libraries, making it more amenable to automation might not be as easy. However, most automation is similar. Approximately 80% of the work will be the same regardless of the technology being tested. It is that last 20% which might make the difference. I'd also point out that there are dozen of resources on the various technologies and people within your company can help the automator understand the technology. Understanding the pitfalls and how to avoid them for automation frameworks and the soft skills to work with development are not as easy to find.

The least important factor is familiarity with the automation tool. This is not important if the automator has a proven record of learning new applications and technologies. For many experienced test automators, a new automation tool is trivial to learn. Test automators who know how to use a tool without understanding how and why it does when it does will be restricted to the tools they already know. If an automator understands the underlying principles behind all automation tools for a given technology then learning a new tool would be trivial.

How to reduce code duplication, part 2

In a previous entry I blogged about how to reduce code duplication. My solution is using libraries.

Let's say I have a web application for writing blogs. Okay, I'm actually going to use the Blogger software as my example. First thing I'm going to do is break the application down into manageable parts. Across the top is:

• Posting
• Settings
• Layout
• Monetize
• View Blog

Each one will take me to a new page. I typically automate with Selenium using Java. So I might create one class for each of these areas. However, when I select Posting I see it has the following subsections:

• New Post
• Edit Posts
• Edit Pages
• Comment Moderation

So I might want to create the following packages:

• com.google.blogger.posting
• com.google.blogger.settings
• com.google.blogger.layout
• com.google.blogger.monetize
• com.google.blogger.viewblog

Then within the com.google.blogger.posting package I'm going to create the following classes:

• NewPost
• EditPosts
• EditPages
• CommentModeration

Next I'm going to focus on one class at a time. Additionally, I'm going to create Java classes to represent the data on the application. For example, in the Posting section, on the New Post page I have the following:

• Title
• Link
• HTML (the HTML text in a text editor)
• Labels
• Post Options
• Reader comments
• Backlinks
• Post date and time
• Edit HTML Line Breaks
• Compose Settings

All this is data for the page so I'd create:

class NewPostForm {
        String title;
        String link;
        String htmlBody;
        String labels;
        PostOptions postOptions;
    }

    class PostOptions {
        boolean allowReaderComments;
        boolean allowBacklinks;
        boolean enterIsLineBreak;
        boolean htmlShownLiterally;
        boolean automaticPostDateTime;
        String scheduledAtPostDate;
        String scheduledAtPostTime;
    }

I would also add getters, setters and constructors to these classes, e.g.

public void setTitle(String title) {
        this.title = title;
    }

    public String getTitle() {
        return title;
    }

Now, if I create a library function to fill in a new post, I don't have to pass in a dozen or more parameters. I can create an instance of NewPostForm, populated it then pass it in to my library function. So back to the NewPost class. This is the library of routines to be used by my test suite:

class NewPost {
        Selenium selenium;

        public NewPost(Selenium selenium) {
            this.selenium = selenium;
        }

        public void gotoNewPost() {
            // if a link to New Post exists then
                // click the link
                // wait for the new post page to appear
                // this might be a page load so WaitForPageToLoad
                // or it might be AJAX so waitForCondition or
                // whatever the case may be
           // else
                // fail the test case
        }

        public void fillInNewPostForm(NewPostForm npf) {
            assertNotNull(npf);
            assertNewPostForm();
            if(npf.getTitle() != null) {
                // set the title input field to npf.getTitle()
            }
            // more of the same thing for each field in npf
        }

        public void assertNewPostForm() {
            assertTrue(selenium.isElementPresent(titleTextFieldLocator));
            assertTrue(selenium.isElementPresent(linkTextFieldLocator));
            // more of the same for all the inputs in the form
        }

        public void clickPublishPost() {
            // if a link for Publish post exists then
                // click the link
                // wait for the post to get published
            // else
                // fail the test case
        }
    }

I've left out things like the values for titleTextFieldLocator but that is easy enough. It would be something like:

String titleTextFieldLocator = "xpath=//input[@id='postingTitleField']";

As you build things up you will have a library of methods for the different 'pages' of the application. So if I wanted to test all the different ways of posting something I could have an instance of NewPost as a class variable for a test case then initialize the variable in the @Before method, just after the initialization of the selenium instance. For example:

class NewPostTestCases extends SeleneseTestCase {
    NewPost np;

    @Before
    public void setUp() throws Exception {
        super.setUp(); // initializes inherited selenium
        np = new NewPost(selenium);
    }

    @TestCase
    public void someTest() throws Exception {
        np.gotoNewPost();
        NewPostForm npf = new NewPostForm();
        npf.setTitle("How to reduce code duplication, part 2");
        npf.setHtmlBody("this is my new blog body");
        npf.setLabels("selenium, java, testing, development");
        np.fillInNewPostForm(npf);
        np.clickPublishPost();
        // do some sort of assertion to confirm the post went okay
    }

    @After
    public void tearDown() throws Exception {
        super.tearDown();
    }
}

And that is the basic idea. I'd used my IDE to generate all the getters/setters and constructors. I'd also let it suggest when I needed to throw an exception (I'd always throw the exception rather than try/catch and deal with it; junit was made assuming the exception gets thrown up to the TestRunner). And there are things like, put the datatypes in a different package, e.g. com.google.blogger.posting.datatypes for the NewPostForm datatype plus let the IDE tell you when to add the appropriate import statements.

Is Automated testing development?

I'm not talking about unit testing. I am talking about regression testing. There are a number of automation tools out there and for some applications you can just use the record and playback feature. WinRunner, SilkTest, RationalRobot, etc. all have a feature where you can turn on a recorder, manually walk through an application then save the script. Later you can play the script back; if nothing has changed the script should execute without error.

This is the theory. The reality is that most projects change and the scripts fail. You then have to take the time to re-record the script or edit the code so it matches the change in the application. Additionally, the scripts tend to make the application do things but the tester still needs to add code to the script to confirm the right things happen, e.g. assert statements or capture points.

So testers are creating, maintaining, enhancing and debugging source code. This sounds a lot like development work. Yet in most the places I've seen people doing automation and with most the people I've interviewed (and some I hired), very few have knowledge of software development.

Yesterday I was talking to someone using an automated script. The script worked fine for the person developing it but did not for the person I was talking to. It turns out that the script assumes relative paths to other things. If you don't run it from the right directory (not the directory the script is in) it fails to work. To fix this flaw the 'developer' added a command line option to the script. The logic was "If there is a $1 parameter, cd $1 else assume you are in the correct directory."

There was no comments in the script, they did not reassign the $1 variable to something more sensible and they checked for $1 deep in the script, i.e. not at the top.

The person I spoke with spent an hour trying to figure out what was wrong. She even spoke with the creator of the script and he couldn't figure out she was doing wrong.

A good development practice is a coding style guideline. Using appropriate comments, parsing input parameters near the beginning of the script and possibly writing it as a function. Developers working on a team have learned that a consistent style makes it easier for everyone to take over someone else's code. At first a new developer might want everyone to switch to their standard but once they come around everyone benefits.

Creators of automated regression tests never create a coding standard. In many cases they don't use source control. Additionally, they will pick automation tools that have poor or no debugging capabilities. Developers like Visual C++ or Java because the IDEs are so advanced. Once you get familiar with Eclipse or NetBeans, you could never imagine using Java from the command line again.

If developers are using powerful tools like Eclipse to develop their code, how is an automated tester going to keep up? Every time the developer makes a required change/enhancement to the application, the tester will have to maintain their scripts. If the developer can make the change in minutes but the tester takes hours, the cost of maintaining the automation will not be worth it.

I like the idea of pair programming where one person does the development and the other person codes the tests. Agile programmers are more thinking about unit testing when they describe this concept but why not have an integration or system level tester be a person with development skills?

I'm not saying that developers should start doing automation testing. Developers have a different mindset then a tester. I'm suggesting that testers should have some fundamental development training. If you hire QA or testing staff with development experience you will probably get better automation.

Additionally, if you are an automation tester, learn development techniques and apply them to your script development. Become more efficient. In many companies, they get automation tools but end up abandoning them because they become a maintenance nightmare. Maybe you will be the person who saves the tools and keeps the company using them.

Finally, automated testing is not WHAT you want to test. It is HOW you want to test. If I'm creating an application, I'll first come up with a list of requirements and design WHAT I want to create. If I decide I'm going to write the application in Java or C++, the language does not, for the most part, dictate WHAT I'm going to create. The automation tool you use comes at the implementation stage. You still need a test strategy, test plans and a list of test cases. Only then should you be looking at HOW you are going to automate the test cases.