Test Automation Manifesto

I have been recently reading xUnit Test Patterns by Gerard Meszaros. Excellent book with a lot of things I learned the hard way. I also found an article by Gerard et al titled, "The Test Automation Manifesto". The manifesto is:

Automated tests should be:

1. Concise: As simple as possible and no simpler.
2. Self Checking: Test reports its own results; needs no human interpretation.
3. Repeatable: Test can be run many times in a row without human intervention.
4. Robust: Test produces same result now and forever. Tests are not affected by changes in the external environment.
5. Sufficient: Tests verify all the requirements of the software being tested.
6. Necessary: Everything in each test contributes to the specification of desired behavior.
7. Clear: Every statement is easy to understand
8. Efficient: Tests run in a reasonable amount of time.
9. Specific: Each test failure points to a specific piece of broken functionality; unit test failures provide “defect triangulation”
10. Independent: Each test can be run by itself or in a suite with an arbitrary set of other tests in any order.
11. Maintainable: Tests should be easy to understand and modify and extend.
12. Traceable: To and from the code it tests and to and from the requirements.

In his article he talks about bad code 'smells'. A 'smell' is something which you notice again and again as a problem. Even before you have clearly identified it, you can 'sniff' them out.

The first two code smells he talks about are THE reason record and playback test automation doesn't work.

When you use record and playback to automate, it will (a) hard-code test data and (b) duplicate code. If I'm testing an application which requires me to log in before each test, the recorder will record me logging in for each test case. If I do not refactor the log in to a function that all the test cases use and the developer changes the login, I would have a maintenance nightmare. Additionally, if the username and password change and the test data is hard-coded, I'd have to find all the instances and change them.

By the way, the idea of 'refactoring' is to change the code to be more maintainable without changing the way it runs. For example, if I had the following code:

// test#1
// put "darrell" in username text field
// put "password" in password text field
// click Submit button
//
// test#2
// put "darrell" in username text field
// put "password" in password text field
// click Submit button
//
// test#3
// put "darrell" in username text field
// put "password" in password text field
// click Submit button
//
// test#4
// put "darrell" in username text field
// put "password" in password text field
// click Submit button
//

and it worked as expected, I could refactor it into:

// login(username, password)
// put $username in username text field
// put $password in password text field
// click Submit button

// test#1
// login("darrell", "password")
//
// test#2
// login("darrell", "password")
//
// test#3
// login("darrell", "password")
//
// test#4
// login("darrell", "password")
//

This will run pretty much the same but now if the Submit button gets changed to a Login button, I don't have to change all 4 test cases (what if I have 40,000 test cases). I just change:

// login(username, password)
// put username in username text field
// put password in password text field
// click Login button

and it updates all the test cases. This code still has hard-coded data. How about changing this to put the data in a property file:

# login data
username=darrell
password=password

then change my code to:

// read property file
// username = getProperty("username")
// password = getProperty("password")

// test#1
// login(username, password)
//
// test#2
// login(username, password)
//
// test#3
// login(username, password)
//
// test#4
// login(username, password)
//

Now I can edit the property file if I want to change the username or password.

I picked property file to hold the test data but it could just as easily been a database, spreadsheet, text file, compiled resource, global variables, etc.