Why each test case should start and end at the same state

I have seen a number of test automators struggling with creating a test suite. Their problem is that they are using xUnit style test cases but they want to change the way they were intended to be used.

For JUnit, the execution is:

• Run Before, open web browser
• Run Test Case #1
• Run After, close web browser
• Run Before, open web browser
• Run Test Case #2
• Run After, close web browser
• Run Before, open web browser
• Run Test Case #3
• Run After, close web browser

What test automators want is:

• Run Before, open web browser
• Run Test Case #1
• Run Test Case #2
• Run Test Case #3
• Run After, close web browser

The problem is, this is not how things work in JUnit. So they have been using static methods and helper classes to create the web browser. It looks like:

• Run Before, if browser == null [true], browser = open web browser
• Run Test Case #1
• Run After, if last test case [false], close web browser
• Run Before, if browser == null [false], browser = open web browser
• Run Test Case #2
• Run After, if last test case [false], close web browser
• Run Before, if browser == null [false], browser = open web browser
• Run Test Case #3, flag last test case
• Run After, if last test case [true], close web browser

This is essentially what test automators want because the After/Before calls between test cases do nothing and leave the web browser open.

Writing test cases which follow each other, i.e. the final state of test case n is the setup state for test case n+1, is a bad idea.

If you start with something simple and current it is not a bad idea.

Imagine the test suite growing and growing. A few months from now you have 5000 'test cases'. Everything is going great. Then they make a change to the application. Your test cases start failing. You investigate the first failure which happens to be test case 3476 of 5000. You need to run all the test cases from 1 to 3476 before you can get to the point it APPEARS to fail.

A few hours later you find out that things have changed in the application and it is a false negative, i.e. the problem is in the test automation and not the application. You fix it and run the test suite again. A few hours later you find out your fix didn't work. So you tweak it and run the test suite again. A few hours later you find the fix still isn't right. You try one more time. While the test suite is running for the third time that day, 5pm hits. Do you go home and check the results in the morning? Do you work late?

Let's say you work late and find test case 3476 is working again. But wait, test case 3788 fails now. You set a break point just before the failure point and run the test suite again. Even if you can fix the problems in one attempt, if there are multiple test cases needing maintenance you will still take days to just FIX the old test cases. Where do you find time to add new test cases for the next features? They will take just as long if you add them to the end of this chain.

Another thing to consider is, hopefully, you will get to a point that your test suite takes MANY hours to run. I've had test suites which took over 8 hours to run. What do you do? You could ask the Project Manager for more time. As a Project Manager he is going to look at the problem just like any other problem he deals with. If Tim has a task to do and it is going to take him 12 days to complete but we need it done in 4 days, break the task into 3 sub-tasks and bring in two more people to help Tim. For safety, bring in three more people. With four people (Tim plus the three new people) working on it, each person should take 3 days to do their part. Working in parallel, the whole task will be done in 3 days.

How does this apply to test automation? If one machine is taking 40 hours to run the test suite but we want it run daily. Out of a 24 hour day we need time for maintenance, building the application, backing up the system, etc. So let's say you have 8 hours to run the test suite. If it takes 40 hours to run the whole thing, 40 / 8 = 5 so get 5 computers and break the test suite into 5 parts. Heck, computers are a lot cheaper than people; get 10 computers and break the test suite into 10 parts. Now it takes 4 hours to run the test suite.

But how do you break apart the test suite? The way I write a test suite is to have the Before call setup for the test, I run the test case then the After call returns the system to the exact same state as before the test was run. This means all test cases start from the same point and end at the same point. I can run the tests in any order I want. I can run one, some or all the tests without worrying about how one test will affect another test. For me, if I have 5000 tests and I want to run them on 10 machines then I run tests 1 to 499 on machine 1, tests 500 to 999 runs on machine 2, tests 1000 to 1499 runs on machine 3, ..., tests 4500 to 5000 runs on machine 10.

If I try this and find that machine 3 is taking 7 hours and machines 5 and 9 are taking 3 hours and 2 hours, I can start moving some from machine 3 to machine 9, keep doing this until machine 9 is taking 4 hours and machine 3 is taking 5 hours. Then I can move some from machine 3 to machine 5 until they are both taking 4 hours.

Which test cases I move to which machine does not matter. It should take me seconds to make the change.

Other the other hand, if your test cases all depend on the previous test case, you will need to figure out where to make the first break, i.e. test case 1 to what? Once you figure that out, you'll have to figure out how to get the next test case into the correct state before it starts. You will have to do this 10 times and each time it could take you over a day to get things set up. In other words, it could take weeks just to get back to nightly build and test.

Bottom line, it feels like you are saving time by having the tests run one after the other without closing but in the long run it will cost you so much that you might have to abandon test automation or seriously jeopardize the project.

It is important to understand that if this happens a project manager has to deal with a lot more than testing, a LOT MORE. He will not want to hear why you need two EXTRA weeks. He will just expect you to find a solution and bring the project in on time and under budget. If fingers start pointing you can be sure that the project manager will throw the test automators 'under the bus'.