A good defect report

What makes a good defect report? Let's look at the flip side.

A bad defect report is a defect report with insufficient information. If I file a report and the developer has to come back to me for more information, it is a bad report. If I write down incorrect information or make assumptions and report them as facts.

A bad defect report can also be a report with too much information. You might think I am wrong. How can you have too much information? Well, if the report goes on and on the developer will have to wade through all the information you provided to find the relevant information. The developer's time costs the company money. Also, how long time do you spend gathering and entering in information that is not useful or necessary? Your time costs the company money.

So like many things in life, you need balance. But how do you know what is the right amount of information?

Most defect tracking systems will have a form to fill out. As you are doing your testing, think about the spaces on that form. There will be:

• Which project you are testing.
• The version of the project. 
• A summary of the defect. 

This is where you need to start thinking. When someone is trying to find the defect, they will search the summary. Was there an error message? The title of an error dialog, the message in the dialog, the Exception from the log file (e.g. NullPointerException, IllegalFormatException, etc.). Try to put a keyword the Developer or Project Manager will associate with the problem. Error dialogs are good for the Project Manager and log messages are good for the Developer (and many Project Managers as well).

• The severity.

This should not be confused with the priority. The priority is how important fixing the defect is. On most projects this is decided by the Project Manager. The severity is how harmful this is to the customer. For example, let's say I found the following defect: If you enter "Bob" at the top of the page, scroll down two rows, press the Home key, press the shift key repeatedly for at least 37 times, turn the monitor off then back on again, while an audio CD is playing in the computer, it will short circuit the cause the monitor to burst into flames.

The severity of this is VERY high. Any time a computer program can cause your equipment to catch fire I would classify it as the highest possible priority. Now if the defect only happens with THE exact steps above and no other way then the priority would be extremely low, in my opinion.

• Steps to reproduce.

I see a lot of defect tracking systems that do not have a specific section for this  but it is very important. This also requires balance. I might be recording everything I did all day and after 5 hours of testing I find a bug. It would not be reasonable to list all 5 hours of testing. For me to enter all that information would probably take over 5 hours and for the developer to step through everything would take him at probably 5 hours. So this would triple the test time for each defect found. Additionally, I'd have to do all these steps when the defect is resolved to confirm it was fixed.

Often what I find myself doing is trying to find a short set of steps to reproduce the error. The more I know about the application the more I can guess at what went wrong. Hopefully this will help me determine a shorter test case to produce the problem.

• The environment.

If you are testing a web site, the type of web browser is important. The version might matter as well. When I test a web site I will use different browsers for the same test. If I see it looks fine in one browser but not the other I will mention it. What was the error/problem? What are the contributing factors. Is the amount of memory a factor? What about disk space or file permissions?

When you are trying to determine what to include and what not to, i.e. finding that balance, it is usually best to err on the side of caution. Anything you know is a factor must be included in the defect report. Anything you know does not contribute to the defect should be left out. Is there something which you are unsure about? Take a few seconds (maybe minutes) to try a few things to eliminate or confirm the information should be included. If I click on something in a web page and an error dialog pops up, I'd try a different web browser. If we supported four web browser, I have to test all of them anyways, try the test case with all four browsers. Now I know if it is a defect for all browsers or just some browsers and I can include that in the defect report.

On the other hand, if I think it might have something to do with the amount of memory available I might just put the amount of memory available in the defect report. The effort to free up more memory might be more than it is worth. Additionally, how much memory do I need to free up to determine memory was a contributing factor?

Finally, any files associated with the defect should be attached. Any log files but only if there is something in the log that looks unusual, like am Exception message or gibberish.