One advantage of test automation - part 2

In my previous blog One advantage of test automation I talked about the saves of writing once and running many times. Essentially, if I have 8 configurations, it will take me 480 hours to test everything and I will test everything 12 times in a release I am looking at 5,760 hours of testing. Let's say the cost of a manual tester is $30/hour (salary, benefits, computer equipment, office space, vacation pay, support staff, etc.) then we are looking at $172,800 to manually test the application for the current release.

If I can write the automation once and run it 8 times (once for each configuration) I'm going to see significant savings. If automated testers cost twice as much, I'm going to see a cost savings of 4 times. Still looking good. So what I'm looking for is the cost of creating the initial framework but the cost of maintaining that framework needs to be less than 4 times as long as manual testing.

Here are four scenarios I have observed (the amount noted is the total cost from the beginning of the release):

Scenario #1

Iteration #1: manual=X, automated=X
Iteration #2: manual=2X, automated=2X
Iteration #3: manual=3X, automated=3X
Iteration #4: manual=4X, automated=4X
Iteration #5: manual=5X, automated=5X
Iteration #6: manual=6X, automated=6X
Iteration #7: manual=7X, automated=7X
Iteration #8: manual=8X, automated=8X
Iteration #9: manual=9X, automated=9X
Iteration #10: manual=10X, automated=10X
Iteration #11: manual=11X, automated=11X
Iteration #12: manual=12X, automated=12X

What is happening in scenario #1 is record and playback. The idea is that you buy the software, you record one iteration then play it back over and over. The cost of recording is the same as manual testing. You turn on record then manually test the application. Stop recording and save it. For the next iteration you simple play the recorded first iteration. You find there are timing issues. Changes to the application are causing tests to no longer work. Debugging what is wrong takes time. If you are lucky, it takes around the same amount of time as manual testing. But you paid for the software and training your staff. Let's say that is $20,000. You could have saved $20,000 if you just stuck to manual testing.

Scenario #2

Iteration #1: manual=X, automated=2X
Iteration #2: manual=2X, automated=4X
Iteration #3: manual=3X, automated=6X
Iteration #4: manual=4X, automated=8X
Iteration #5: manual=5X, automated=10X
Iteration #6: manual=6X, automated=12X
Iteration #7: manual=7X, automated=14X
Iteration #8: manual=8X, automated=16X
Iteration #9: manual=9X, automated=18X
Iteration #10: manual=10X, automated=20X
Iteration #11: manual=11X, automated=22X
Iteration #12: manual=12X, automated=24X

What is happening in scenario #2 is poor planning at the start of the automation. Not enough time was spent designing and implementing the test framework. Every time there is a new release of the application, the cost to maintain the automation is as much as the initial creation. We are not leveraging anything we did from the first iteration. Usually test automation is seen as a failure around the third or fourth iteration. When this happens, management is usually pretty apprehensive about trying test automation again. They didn't expect these sort of losses. If we use the initial estimate that manual testing costs $172,800 and assuming we abandoned test automation after iteration 3. The total cost of attempting automation is $216,000 ($43,200 more than just doing manual testing). And this is assuming we called it quits after iteration 3. Each iteration we hesitate on is costing the company an additional $14,400.

It is understandable why so many companies become disillusioned with test automation. Companies selling test automation software will pitch it as saving you 8 times the cost of manual testing. It looks like the cost of automation is going to be $21,600. Even if you double that and add $10,000 for the software ($53,200) it is better than one third the cost of manual testing. So you are expecting $55,000 and it ends of costing you $210,000 or more.

Scenario #3

Iteration #1: manual=X, automated=5X
Iteration #2: manual=2X, automated=6X
Iteration #3: manual=3X, automated=7X
Iteration #4: manual=4X, automated=8X
Iteration #5: manual=5X, automated=9X
Iteration #6: manual=6X, automated=10X
Iteration #7: manual=7X, automated=11X
Iteration #8: manual=8X, automated=12X
Iteration #9: manual=9X, automated=13X
Iteration #10: manual=10X, automated=14X
Iteration #11: manual=11X, automated=15X
Iteration #12: manual=12X, automated=16X

In scenario #3 we spent time creating a test framework but for some reason the cost of maintaining the test framework is the same as manual testing. Sometimes this is because we haven't really thought out the design or the application we are testing is in flux. Some applications are not ready for test automation. If your test automation efforts are failing, you need to look at why. If the application is changing so much that automators need to do some creative coding (which becomes difficult to maintain) or they end up throwing away a lot of the previous iteration automation, then you need to look at stabilizing the development process. Another thing you might have done wrong at this point is write different code for the different web browsers. If I have methods with:

if IE6 then do X
else if IE7 then do Y
else if FF3 then do Z

Or if I actually write one method for IE and another for Firefox then you are not really taking advantage of write once and run many. Additionally, you don't want to be too creative with the automation code. If you have to use a very complex regular expression to find a text field on all the different browsers then maybe the application is not ready for automation and you need to work with development to make the application friendlier to automation.

Scenario #4

Iteration #1: manual=X, automated=5X
Iteration #2: manual=2X, automated=5.5X
Iteration #3: manual=3X, automated=6X
Iteration #4: manual=4X, automated=6.5X
Iteration #5: manual=5X, automated=7X
Iteration #6: manual=6X, automated=7.5X
Iteration #7: manual=7X, automated=8X
Iteration #8: manual=8X, automated=8.5X
Iteration #9: manual=9X, automated=9X
Iteration #10: manual=10X, automated=9.5X
Iteration #11: manual=11X, automated=10X
Iteration #12: manual=12X, automated=10.5X

In scenario #4 we spent time creating a test framework and we were able to leverage the work in the previous iteration for the next iteration. Maintaining the automated test framework and adding new functionality after the initial iteration now costs us half as much as manual testing. At iteration #9 we are breaking even. By the end of the release we are looking at a cost savings of $21,600. This does not seem like a lot but if your automation team gets good at this or you hire someone with the right experience you can actually increase the saving. The cost of maintenance might not be linear. It could grow with each iteration. Or you might realize even more savings with someone who is already familiar with the pitfalls of automating.

Things to look for that might give you additional cost savings:

• Someone with experience working with development
• Someone with general experience creating automation frameworks
• Someone with experience automating a similar application (web based, desktop application, mobile device, etc.)
• Someone familiar with the automation tool you are using.

Obviously, someone who uses the same tool, has written automation for an application using similar technologies, a proven record of creating automation frameworks would be ideal and can work with development if the application is not friendly to automation tools.

I would say the most important factor is someone with experience working with development. If they are not comfortable working with development they might be set up for failure and no way to prevent it. Additionally, good automation is development. If the test automator is a developer, they should be comfortable working with development to make the application more amenable to automation.

Next would be experience creating an automation framework is the second most important factor. You can easily land yourself in scenario #2 if you don't know how to create a good test framework.

The second least important factor is testing similar applications. The technologies and problems for web testing are very different from desktop application testing. Additionally, if an application is using certain third party libraries, making it more amenable to automation might not be as easy. However, most automation is similar. Approximately 80% of the work will be the same regardless of the technology being tested. It is that last 20% which might make the difference. I'd also point out that there are dozen of resources on the various technologies and people within your company can help the automator understand the technology. Understanding the pitfalls and how to avoid them for automation frameworks and the soft skills to work with development are not as easy to find.

The least important factor is familiarity with the automation tool. This is not important if the automator has a proven record of learning new applications and technologies. For many experienced test automators, a new automation tool is trivial to learn. Test automators who know how to use a tool without understanding how and why it does when it does will be restricted to the tools they already know. If an automator understands the underlying principles behind all automation tools for a given technology then learning a new tool would be trivial.