Test Driven Development

I've read a lot for and a lot against Test Driven Development (TDD) but I don't every remember something TDD has really helped me with. If you write a test then write the code which makes the test pass the end result is a piece of code which demonstrates how the application code works. I have used a number of open source applications and like every library/framework/application I have used the documentation ends up being way out of date. But if you are writing tests for all the features you add (before or after you write the code) you are essentially creating examples of how to use the library/framework/application. So even with the documentation falling out of date, so long as the code has a good base of unit tests, you can figure out how it works. I find this helpful in multiple ways. When I'm automating I tend to use tools like Selenium. Whenever I need to use a new feature of Selenium I can look at the documentation but ultimate, the unit tests for the framework tell me how it actually behaves. Additionally, the applications I am testing are written by people who understand TDD and have a good set of unit tests for each application. So I find it easy to under how new features are implemented by looking at the unit test for the application I am testing. For example, Selenium has a TakesScreenshot class. The source for it would be found at:

https://code.google.com/p/selenium/source/browse/java/client/src/org/openqa/selenium/TakesScreenshot.java

So if I want to see the tests for it, change 'src' to 'test' and change 'TakesScreenshot.java' to 'TakesScreenshotTest.java' for a full URL of:

https://code.google.com/p/selenium/source/browse/java/client/test/org/openqa/selenium/TakesScreenshotTest.java

When I look at this latter file I see many examples of how to use the TakesScreenshot class.

So you want to do unit testing

What is a unit test? Wikipedia describes unit testing as testing individual units of code in isolation. If the code has external dependencies, you simulate the dependencies using mock objects.

For example, if I am testing code which gets data from a database, hopefully access to the database is via something like ODBC or JDBC. In which case, it is possible to use a fake database (file system or memory based) rather than say an Oracle or SQL Server driver.

If my database connection is hard coded to a particular machine or assumes the machine is localhost then my first step is to refactor the code to remove this dependency.

Part of the purpose of having unit test cases is so that we can safely change the code and know we didn't break any existing functionality. So if we need to modify the code to be able to add unit tests we have a bit of a Catch-22 situation. The truth of the matter is, if we have been changing the code without unit tests, changing it one more time in order to add unit tests is actually a step in the right direction and no worse than previous development.

Another important feature of unit tests are speed. If I am adding a new feature and I want to be sure it hasn't broken anything, I want to know as soon as possible. I don't want to write the feature, run the tests and check the results tomorrow. Ideally, I want to know in seconds. Realistically, I might have to live with minutes at first.

Test runs should be automated. If I have to make a change and figure out what tests to run, run them and check the results there is a strong chance I will stop running them. Especially if I'm on a tight timeline.

Ideally, I would check in my code. This will fire a trigger which builds my code (not the entire product, just my code) and run the unit tests against it. Putting such a build system in place is a great deal of work but worth the effort. Every minute it takes to create this build system should be weighed against how much time developers spend testing their code before they check in, how many minutes testers spend finding bugs, how much time developers take understanding the bug and fixing it. Numerous studies have shown fixing bugs is much more expensive than never introducing them in the first place.

So what do we need so far?

First, we need a unit test framework. You wouldn't create your own replacement for JDBC/ODBC. So why create your own unit test framework. There are plenty of them out there.

Second, we need mocking frameworks for the technologies we are utilizing. Which mock object frameworks you require depends on what you are using in your application. If it is a web application, you might need to mock out the web server. If it accesses a database, you will need to mock out the database.

Third, we need a build system to automate the running and reporting of the unit tests. Reporting the results is important too. Most systems will either report back to the source control client or send you an email. If the tests run in, literally, seconds, you can afford to reject the checkin if a unit test fails. If it takes more than say 5 seconds, you might want to send an email when a checkin fails.

Fourth, we need commitment from management and the team. If you don't believe there is benefit to unit testing there will be no benefit to unit testing. Training people on how to create good unit tests and maintain them is critical. If I'm starting a new project and writing tests from the beginning it is easy but the majority of you will be adding unit tests to existing code.

The first three things are relatively easy to obtain. There are plenty of technologies and examples of people using them. The fourth requirement is the biggest reason adopting unit testing fails. If you don’t get buy in from everyone involved it just won’t work. The developers need to understand this will benefit them in the long run. The testers need to understand that less testing will be required and they need to focus on things unit testing will not catch. There will always be plenty of things to test. So there should be no fear unit testing will replace  integration or system testing. Management has to understand if they cut timelines for a project, they will not give developers time to write the unit tests. If you reward the Project Manager for getting the project out on time, he will get the project out on time even if it means giving developers no time for unit test creation. As a Project Manager, if reducing the number of issues AFTER the project has shipped is not a metric I’m evaluated on, I’m happy to ship a product which will make the next project difficult to get out on time.

So, you have the tools and you have buy in from everyone. Now what? If you have 100,000+ lines of code, where do you start writing unit tests? The answer is actually really simple. For example piece of code a developer touches, they should add unit tests. Bug fixing is the best place to start. I would FIRST write a unit test which would have caught the bug. Then I’d fix the bug and see the unit test pass.

By focusing on unit tests for bug fixes it reduces the need for regression testing, it focuses on the features customers are using and the developers are in that code anyways. If we need to refactor the code to support unit testing, might as well happen as we are changing the code. The code was broken when  we started the bug fix. So we’ll have to manually test the fix without unit tests. Hopefully, with a unit test in place, it will be the last time we manually test changes to this code.

If we are modifying the code for feature creation, not bug fixing, we want to write unit tests to confirm the current behaviour. Once we have a test which passes with the current code, we can add the feature and the tests should continue to pass.

At this point we know what we need and where to start. So let’s cover some of the how to write a unit test.

First, a unit test is going to be a function/method which calls our code. We want the name of the unit test to reflect what it is testing. When results are published they will go out to the developer but they will also be seen by the backup developer, project management and various other people as well. If I got an email telling me test17() failed I’m going to have to open the code and read what test17() is testing. You added comments and kept them up to date, right? Or course you didn’t. The comments shouldn’t be necessary. The test name should tell me what it is doing. If the test method was called, callingForgotPasswordWhenNoEmailInUserPreferences() then we all know what is being tested.

Second, what failed? Most unit test frameworks has assert statements. There is the basic fail() call but there are also things like AssertTrue, AssertEquals, AssertNotNull, etc. They can be called with just what you are checking or with a message and what you are checking. You don’t want to code any more than you have to but enough that someone receiving the results will know what failed. If the requirement for my software is “When a user clicks the Forgot Password button but they have not set an email address in their preferences, they should be presented with a message telling them to contact the system administrator.” Then the result message from my example here might be something like, “callingForgotPasswordWhenNoEmailInUserPreferences() failed. Was expecting: ‘No email address was set for your account. Please contact the System Administrator.’ but received: ‘No email address.’”. From this is it pretty clear what was expected and what we received instead. Failing to tell the user how to proceed should be considered a show stopper for the customer. On the other hand, if the result was: “callingForgotPasswordWhenNoEmailInUserPreferences() failed. Was expecting: ‘No email address was set for your account. Please contact the System Administrator.’ but received: ‘No email address was set for your account. Please contact the system administrator.’” the customer might consider this acceptable. We might even update the unit test case to ignore case so the test becomes a pass.

Unit test frameworks are pretty well established now. The general structure of a unit test is:

• set up for the test
• run the test
• assert the test passed
• clean up so the next test starts at the same point

The set up would be things like creating mock objects, initializing the inputs for the test, etc. The running of the test would be a call to the method being testing. Next would be an assert statement confirming that we received the expected results or side effect. Finally, clean up (often called tear down) the environment so it is at the exact same condition it was before the set up occurred.

Often you will group similar tests in one test suite. If I have 12 tests and they all require the same set up I will put them all in one suite. The code will then have one setUp() method that creates the environment for each test, one method for each test (12 methods in total for this example) and one tearDown(). The setUp() method will create any mock objects, initial global variables, etc. The test method will create anything particular to that test, call the method being tested then make an assert call. The tearDown() method will then clean up the environment so it is just like it was before the setUp() method was called. This is important because most unit test frameworks do no guarantee the order the tests will be run. Assuming one test starts where a previous test left off is just bad practice. I have worked on a project with 45,000 unit tests. All test are run as part of the nightly build. Rather than running all the tests on one machine, they are distributed to 238 different machines. If they all ran on one machine they would take 378 hours (over 2 weeks) to run. By distributing them over 238 computers they run in approximately 3 hours. However, if test1932 depends on test1931 and the two tests get sent to different machines, test1932 will not run correctly. Each test must be independent of all other tests. This will not seem important at first but 1 year later you might find yourself needing weeks (possibly months) to refactor all your unit tests. Moments like these often cause management to abandon unit testing.

This is unit testing is a nutshell. I will warn you, ‘the devil is in the details.’ Hiring someone who has gone through the pains of setting up a unit test framework is always a good idea. Either find a good consultant or hire someone full time to work on the framework for you. Some unit test frameworks are jUnit for Java, cppUnit for C++, nUnit for .NET, etc. Gerard Meszaros has written an excellent book called “xUnit Test Patterns: Refactoring Test Code”. In it he talks about “Test Smells”. Essentially, you can sometimes look at a piece of code and say, “This code stinks.” A code or test ‘smell’ is an indicating that the code has problems, i.e. it stinks. I have found reading Gerard Meszaros book I know what to look for before I do it. Originally the book was designed for people who created unit tests, found the tests have issues, i.e. they ‘smell’ and are looking to fix them, i.e. refactor. By reading the book, I avoid creating the bad unit tests in the first place.

Good luck and have fun!

How to do focused testing

After years of software testing I have found one constant... there is never enough time to test everything.

So how do I pick what I'm going to test? Here are a few tricks I use to focus what I'm going to test.

First, test what changed. I like to get a snapshot of the last release (source code) and a snapshot of the current release candidate then compare them. Often the source code is readable enough for any tester to understand what has changed. If you are not a programmer, not familiar with the particular language or the code requires a little indepth knowledge only the programmers have then you might want to talk with some of the developers about what you are looking at.

For example, if I am testing a shopping cart for an online web store and I see changes to a class called TaxRules then I'm going to look at the requirements for tax rules and see if there is anything new. Even if there aren't requirements for tax rules, I can talk to the person who wanted this feature (or improvement) and I can look at the code. Comments will be for humans and I'm human (no really, I am). Good code should also be self documenting. In other words, the variables and method names should almost read like English.

The next trick is to get someone to add a code coverage tool to the web site (or whatever it is you are testing). As you test the application (automated or manually) the coverage tool will record which parts of the code you executed. Look to see of the code which has changed was executed. Set the coverage tool so it records actions within the methods and not just method level data. This way you can see which branch conditions were called. If there is a loop which never gets entered or a portion of an if/elsif/else which does not executed, you know you need another test case. You could even ask the developer to relate the statements missed to user actions. Essentially, get them to tell you what test case you missed. :)

Finally, look at the inputs to the methods. Can you see a way to affect the inputs? Does the method handle situations where the data is bad? Developers think in terms of "make the program do what it is supposed to do." They always make it do the right thing when given the right data. As a tester you need to think in terms of "make sure the program doesn't do what it is not supposed to do." In other words, if you input bad data, it should report a nice, helpful message to the user or just restrict the user's ability to input the bad data.

As you do code coverage, you can keep notes with your test cases. Just a comment about how test case XXX should be run with class YYY is altered.

Ways to facilitate communication

One of the most important things on a team is communication. I have worked on teams were everything was put into a document. Unfortunately, as the project progressed, time became more and more precious. People needed to be more efficient and far too often, the documentation suffered. Decisions would be made between two individuals and the rest of the team would miss out.

So how do we make sure everything is well communicated? The simplest answer is to talk to each other.

In scrum you have sprints and planning meetings. What I would do is at the end of the week look at the back log (features and defects in the tracking system) and decide what I could get done be the end of next week. I might add a few notes to the defect report and assign it to myself.

On Monday morning we would have the planning meeting for the week. Everyone would decide what features and bug fixes should and could be completed that week. If there was anything which would take more than one week to complete, it need to be broken down into multiple, smaller tasks. No task should take more than one week. This means implementing and testing the feature/bug fix should happen the same week.

Everyone would pick the defects they were going to work on and give estimates for how long each would take. Every day we would meet and go over the status of the team. Have things changed? Did sales land a new contract but promise a feature we need to complete this week? Or is that feature multiple tasks and we want to start on one of those tasks this week? If yes, we reassess the plan and change priorities.

The key to all of this is that developers, business analysts, testers, project stakeholders all meet and decide what will be done each week and continue to track progress on a daily basis. Most people who do scrum understand this.

What about the time between each scrum? What if you need clarification on the feature you are working on? Whether you are a developer or a tester you might need further clarification. If everyone is in the same room you could just shout to the people involved. This could be disruptive to the other people in the room.

So how do you deal with it? We could create teams and partition the rooms. Each room would have a tester, a developer and a business analyst. What about the project manager? There will usually be one project manager and multiple developers. We cannot have one manager to each developer. Additionally, I have never worked on a team were the developer to tester ratio was 1:1. So this idea is not going to work.

What has worked for me in the past is everyone is on MSN Messenger or Google Talk. These tools let you talk with one or multiple people at once. You can set up aliases for the different groups you need to talk with. This is really helpful if you are working in an open concept office or a distributed team. In the case of an open concept office, shouting to all interested parties might work for you but it is disruptive to the people in the room who are not involved in the conversation. By using chat software I can ask quick questions of just the people who need to be involved. The same is true for people at remote locations. I can chat with everyone involved without having to book a conference room and have everyone call in.

An added benefit is the ability to opt out of a conversation. If I am working on something which requires my full concentration I can log out of chat. This send a signal to everyone that I cannot be disturbed. The danger of this is forgetting to log back into chat but hopefully, you will just get into the habit of remembering. Everyone checks email after they have been working hard on a feature which required their full attention. If someone is not logging back into chat and you need to talk with them, send them an email. When they are done it will be a gentle reminder to log back into chat.

If you find that chat is becoming too tedious and requires too much typing, switch to a meeting. Book a conference room and invite everyone to a 30 minute or 1 hour meeting. If the team is distributed you can use Skype or VOIP to get everyone calling in.

How does a conventional tester fit into an agile environment?

When I think 'agile' I think test driven development (TDD). In TDD, the programmers actually write the tests first then write code to make the tests pass. If you look at something like Eclipse IDE, I can write a jUnit class that creates the code to instantiates a class that does not exist yet. Eclipse will complain about the non-existent class and offer to create the class for me (without the body). If I write calls to methods which do not exist, Eclipse will complain about the non-existent methods and offer to write the method for me (without the body).

In Agile development you have sprints. A sprint is typically a week. On most agile projects I have worked on, the QA staff are involved from the very beginning of each sprint. Every Monday (or some other day) all the stakeholders (developers, business analysts, testers, etc.) get together to decide what will be completed in the next sprint. They will come up with "user stories" (sort of like requirements). Everyone signs off on the sprint (meaning the stories should cover all the requirements and from that all the test scenarios plus test cases).

As each feature is written, there is daily deployment (or more often). As QA Tester I would write automation to conduct Integration and System level tests.

The most important aspect of agile development is fast turn around. On a traditional SDLC, I would often not see a functioning application until weeks after a developer has created it. By the time I am testing the first feature the developer could be done with that feature and working on something totally different. If I file a defect report on a feature a developer completed 3 weeks ago, she has to stop what she is working on, recall the code for the feature I am testing, fix the bug then get back to the new feature she was originally working on. This can be quite disruptive.

With agile development, a developer makes a change, compiles the code, runs the unit tests. If something she did breaks a unit test she knows in seconds. If it passes the unit test she can check the change in. The QA Tester can then build and test the application.

With the frameworks I worked on, the application would get built automatically by a nightly build process. All unit tests would get run on it. There might also be static analysis tools run on the project before compiling. Once the project is built and unit tests pass it would get deployed to test machines and the QA automation suite would be run. This all happens after business hours. In the morning, I would examine the test results. Sometimes a change in the program breaks my automation so I would do some maintenance on the automation and re-run the tests which failed. If there was an actual failure in the application and my automation found it, I would file a defect report.

On a daily basis the team would meet to talk about what was done the previous 24 hours and what would be the goal for each person in the next 24 hours (this is scrum, a form of agile development). I would bring up any defects I found at the scrum.

Fast turn around and immediate feedback is key to a good agile development. I would write regression tests to make sure defects found and fixed do not creep back into the product but I would also work with the developers to bring the regression tests into the unit test framework. This way the defect can be caught sooner should it creep back in.

Because the number of regression test grow with each iteration (sprint), it is recommended that they be automated. This frees the test team to conduct manual test efforts towards new features.

The most important thing to understand as a tester in an agile development environment is that there is a STRONG belief in minimal documentation. Source code should be self-documenting. The reason for this, and I have seen this on EVER traditional development team, is because the documentation is often neglected. No documentation is often better than wrong documentation. If there is no documentation and the developer has not gotten to the point were their code is self-documenting, they will be forced to look at the code and figure out what it does, rather than read outdated documentation and believe it does something different than it really does.

To understand the requirements of the software, you look at the unit tests. These should also be self-documenting and there should be one unit test for each requirement (user story).

As a tester, I will not expect to get a set of requirements. If I need to, I will take notes at the planning meeting (beginning of each sprint) and update them at each daily meeting (scrum).

The key to success for the tester will be communication. Without communication between the tester, the developer, business analysts and project stakeholders, it will be extremely difficult for the tester to properly test the application.