QA & Testing

Sunday, May 9, 2010

Down on your luck? We can make it worse.

The scammers are at it again. I recently received an email from ECIT GROUP LTD. The email appears to be coming from Workpolis. The From address appears to be Workopolis.com <workopolis.jobs@workopolis.ca>.

If you receive an email via GMail, you can open the menu with "Reply" and there will be an option for Show Original. When you show the original it will display the full email header. Here is the email header for the message I received:

Received-SPF: neutral (google.com: 193.34.167.216 is neither permitted nor denied by best guess record for domain of pbcpttnl@kristell.webhosters-direct.com) client-ip=193.34.167.216;
To: my.address@gmail.com
Subject: Executive Financial Manager Position
From: Workopolis.com <workopolis.jobs@workopolis.ca>
Reply-To: recruiter@ecitgroup.co.uk
MIME-Version: 1.0
Content-Type: text/html
Content-Transfer-Encoding: 8bit
Message-Id: <E1OAWQv-0005e2-PJ@kristell.webhosters-direct.com>
Date: Sat, 08 May 2010 00:54:13 +0200

Most email programs will take bits from the email header and display them, hiding the rest. You'll see the Subject, To, From and Date fields. You might see the Reply-To address as well. When you click reply, you will almost definitely see the Reply-To information.

Notice how the From address and the Reply-To address are completely different. That should be your first clue something is odd. Above the To field is the route the email took to get to me. You don't need to know a lot of the details but just look at all the machine names and notice that workopolis.ca does not appear in anything above the To field. On a quick glance it looks like the email came from kristell.webhosters-direct.com. I haven't checked but I would guess that www.webhosters-direct.com is a web hosting company. Anyone can buy an account from them and set up email accounts and a website.

The idea is, the criminal creates an email account and possibly a website using a general hosting company. Anyone can do this, including you. They will pay for the account in such a way as to make it hard to find out who they really are (stolen credit card). Minutes after they create the email accounts they spam millions of people with their scam email. A few hundred people respond and give them personal information (bank information, credit card, identity, etc.). They take all that information and disappear.

People like me will report them and MAYBE someone will investigate. They will find the site was created with a stolen credit card. The criminal logged in from a stolen account. Usually I can log into a company, from there log into a second company, to a third computer, etc. Different ways of logging in and hopefully somewhere in that chain of logins someone will not be keeping record of my log in. For example, I might have installed a virus on your computer and used your computer to log into the web hosting site. So when the police get the logs from the web hosting site, it leads back to your computer. When they get a warrant for your computer they find a virus on your computer and no logs. Dead end.

The best thing you can do is (a) don't fall victim to these scams and (b) tell your friends.

General rule of thumb is that NO ONE asks for personal information via an email. This scam says:

Have a chequing account at ROYAL BANK OF CANADA (RBC)

They are telling you right up front you have to have an RBC chequing account. This immediately rings warning bells for me. Other warning bells:

They promising you money. Sometimes it is millions but sometimes it is just a good income.
They want personal information. Not just credit card information. When you call RBC they validate you are who you say you are. The more information I can gather about you, the better chance I have of pretending to be you.
The From address and the Reply-To address are completely different.
They continually claim to be reputable. We are recognized by the Better Business Bureau, we are registered with the Chamber of Commerce, etc. Maybe they even are. But why do they feel the need to convince you they are legitimate?

Bottom line, if it sounds too good to be true it probably is. If something doesn't seem right, i.e. you have an odd feeling about it, no matter how small, investigate. I'd say 10 times out of 10 that odd feeling turns out to be right on the money and they are a scam.

Remember that the scammers have been doing this for years. They are MUCH better at this then you are. If logically it seems legitimate but your gut says something is wrong, your gut is probably right. The only way for you to find out what you are missing is to be taken or find someone else who has been taken. Sometimes I walk away from a situation without solid proof it is a scam but (a) at least I wasn't taken and (b) sooner or later I find out what the scam was and realize I was right to walk away.

Thursday, May 6, 2010

Organizing your automation

It has been a while since I have posted something to the blog. Life has been keeping me busy with things other than software testing.

I've been to a number of interviews in the past few weeks and I've been asked a number of questions. One of those questions was regarding traceability. If I have a number of automated tests and a set of requirements, how do I connect the two and how do I determine what has been run and what has not?

The first part to this is how are the requirements organized. If there is a tracking system and some unique identifier for the requirements, I want to use that identifier in my test automation. For example, a previous job the defect tracking system was used to manage requirements. In addition to categories like 'defect', 'enhancement', etc. there was a 'new feature' category. The business analyst would file a 'new feature' request into the system. Using mock ups and text descriptions, they would flesh out the requirements.

Each defect report has a unique tracking number. I would need to incorporate this number into the test automation. There might be one number for the requirement but a number of different test cases. Additionally, the number might be just a number, e.g. 28364. In the automation environment this might not stand out as a requirement. For this reason I would develop a convention where all requirements references would start with REQ-. Thus REQ-28364 could get added to the test case automation.

Ideally, you want the automation to be self-explanatory. If a test case fails, it would be helpful to know what failed without having to look up the requirements documentation. With automation like Selenium RC or Watij (both using Java) I can name the test case class the same as the requirement number, e.g. REQ-28364 but if I am looking at the test suite or test results it might not be obvious what this is. So I would create an annotation @REQ and put the requirement information in the comments of the source code.

The name of the class can then be used to indicate what the new feature is. The name of each test case would be a description of what is being tested. For example, if I'm adding Print support to an editor I might have the class name "PrintSupport" or "NewFeaturePrintSupport". The test cases might be:

canPrintAllPages
canPrintAllEvenPages
canPrintAllOddPages
canPrintRangeOfPages
canCancelPrintDialog

When when I look at the results for the test run I would see:

PrintSupport->canPrintAllPages: Pass
PrintSupport->canPrintAllEvenPages: Pass
PrintSupport->canPrintAllOddPages: Pass
PrintSupport->canPrintRangeOfPages: Pass
PrintSupport->canCancelPrintDialog: Fail

Very easy to see what is not working and what is.

The most important thing for tying requirements and automation together is creating a convention and sticking to it. To help you stick to it, edit the templates for the IDE. Whenever I create a new Test Case class, the template will have an @REQ field in the comments at the top of the class. I can even go one step further and have source control check for the @REQ field. If the field does not exist or it is blank, the check in of my automation will fail with an error telling me to add a requirement reference to the source code.

Thursday, March 25, 2010

How to develop an automation framework for a legacy application

If you join a team testing an application which already exists and may even be released to the customer but there is no automation in place, how do you start?

For many this can be an overwhelming task. The real answer is, one feature at a time.

Let's take for example the last project I worked on. It was created years before I joined the company and had a small base of customers who depended on it and tolerated the quirks and bugs of the system.

The first thing to do is pick an automation tool that will work for the application. Talk to the developers, business analysts and stakeholders to get a feel for where the project is going and keep that in mind. For example, current requirements are Windows and Internet Explorer 7. In the future, we would like to support Windows, Linux, Solaris, etc. and we want support for any and all browsers (Internet Explorer, Firefox, Opera, Safari, Mozilla, etc.). Additionally, we also have a Windows Mobile application but we would like to change to using the web browser on iPhone, Palm Pre or BlackBerry. So do we use a tool that supports all these possible future combinations or do we use a tool that works for the current requirements because the future requirements aren't written in stone and 95% of our customers for the next few years will be on Windows with Internet Explorer 7 (or IE8 in IE7 compatible mode).

The tools I found were a very mature product which only supported Internet Explorer or a newer product which supported everything but wasn't quite as mature. I selected Watij, a more mature product which only supported Internet Explorer. The majority of our customer were Internet Explorer customers, all the developers did their work using Firefox. The defects which appeared in one web browser and not another were typically layout issues. To date, all the layout issues were in Internet Explorer because the developers were using Firefox to do their design.

Next you want to think about code maintenance. This might seem strange; we don't have a single line of code and I'm thinking about how I'm going to maintain this code that doesn't exist.

Think about successful products. Microsoft Office was released 20 years ago. JBoss was started over 10 years ago. Photoshop 1.0 was released 20 years ago. The last proven web based application I tested was started in 2004 and is still going strong. Basically, a successful software project can exist for 10 to 20 years. How long do you think you have to create a test framework? If the software is going to be released within one year, the project manager is going to expect the framework and initial test cases to be written in the first release cycle. So you have less than a year to create your framework. Thus, 6 months to 1 year to create and use the framework on, hopefully, a project which will last for 10 to 20 years. So 90% of your time is going to be on maintenance. Even if we expect the framework to be used for 5 years you are still looking at 80% of your time spent on maintenance.

So plan out how to break the framework into small, manageable pieces. Look at something like a library. Take for example the Java APIs. There are over 200 packages with over 3700 classes. A typical class might have dozens of methods. Did this happen overnight? Absolutely not. James Gosling started work on Java (aka Oak) in 1991 and by 1995 v1.0 was released. So the 3700+ classes were developed over 19 years.

Creating a successful framework doesn't mean all the code has to be in place at version 1.0. It just means the structure has to be there. If you look at version 1.0 of Java, it was object oriented, it had packages, constructors, exceptions, etc. All the basic functionality of the language which exists today was there in v1.0. Was there support for XML, Xpath, SQL? I'm not sure but there didn't need to be. There just had to be enough to produce something useful.

So your framework just need to be structured in such a way that it will grow to something we can use 10 years from now. So how do you create something which will be comparable to Java, C++, etc.?

a dwarf standing on the shoulders of a giant may see farther than a giant himself.

Borrow the design of something like Java. If the language you are using for automation is going to be object oriented, use proven object oriented designs. Leverage the work of James Gosling. Look at the application you are testing. Can you break it apart into sub-sections? For a web application you have pages. Each page will have a variety of actions. Some will alter the current page using JavaScript and some will load a new page. Focus on the functionality of just that page. The new page loading will be in some other package, class, method.

Looking at things like MSDN library I noticed the original libraries where very detailed. To accomplish one thing you often had to call numerous library functions. Over the years Microsoft recognized that most programmers will call:

result1 = functionA();
result2 = functionB(result1);
result3 = functionC(result2);
result4 = functionD(result3);
print result4;

So they created a new library where you called:

print functionABCD();

If you look at the existing code in Microsoft you will see:

functionABCD()
{
    result1 = functionA();
    result2 = functionB(result1);
    result3 = functionC(result2);
    result4 = functionD(result3);
}

You want to build things up the same way. You can have separate projects, separate packages, different naming conventions. The choice is all up to you. Just remember that the division will look silly at first but by this time next year you will be happy you broken it down as much as you did. What I mean is you might find you have 2 packages, each package has 3 or 4 classes, each class has 5 or 6 methods. In a few years time you should find you have hundreds of packages, each package has dozens of classes and each class has numerous methods.

Additionally, do things on the page translate to data structures that need to get passed around? For example, on a web page you will have forms to fill in. Later you might need to edit the data you input. The data required for creation (filling in the initial form) and the data required for editing will be the same data. So create data structures that mirror the form. If the data on the form changes, you just need to edit the data structure. Use getters and setters to obfuscate away the implementation of the data structure. For example, you might store the date string as a Date or Calendar object. Later the web page may change from a text field to a read-only text field with a Calendar widget. If you hardcoded a string for the date field you will have to go and clean up all your automation. If you successful obfuscated away the data type, you should be able to just update one library call and the automation will continue to work. Imagine they decide to change the web form two years from now. You have 4,879 test cases which fill in that form. How long will it take to find and fix 4,879 test cases? How long will it take to update one library call?

Once you have decided on how to structure the framework you can start writing test cases. You might have noticed, I talked about designing the framework but I didn't write any of the code. The reason for this is because you want to have the test cases drive the code creation. If you have test plans in place and the priority of the test cases (conducted manually) is known, then the order you want to automate the test cases is also known. Just as you are not going to start manually testing a minor feature before you test a major feature, you are not going to start automating a minor feature before you automated a major feature.

My first test case might be something like:

class MyFirstTestCase {

    public void setUp() throws Exception {
    }

    public void testLoggingIn() throws Exception {
        String username = getProperty("username", "darrell");
        String password = getProperty("password", "mySecretPassw0rd");

        loginPage.goToTheLoginPage();
        loginPage.logIn(username, password);
        assertTrue(homePage.assertHomePage());
    }

    public void tearDown() throws Exception {
    }
}

If you enter this into an IDE like IntelliJ/IDEA or Eclipse you will get a lot of error messages. The getProperty method does not exist. The loginPage object does not exist. The goToTheLoginPage method does not exist. But the IDE has a helpful feature, it will make suggestions as to how to fix the errors.

It will tell you the getProperty method does not exist and do you want to create it. You can create it inside the current test class but won't other test classes need to get properties as well? So maybe you want to extend the test class and put the getProperty method in a super class. So I would go to the class definition and add an extends statement. Now I get an error in the extends statement. So I take the suggestion to create a new super class. The class will be empty and the error will go away. Now when I deal with the getProperty error, one of the suggestions is to create a method in the super class, so I do.

As you resolve each error, the test framework starts growing. You keep adding in more and more code. Once you have resolved all the errors in your first test case, you have JUST enough code in the test framework to run one test cases. The test case should now be runnable and you can add it to the nightly build process. Each night it will build the application, deploy it and run your one test. Next day, add another test. Are there things from the framework you can use for test case number two? Then reuse them.

If you find yourself putting code into more than one place, move the code to a library and change the two places you are using it into a call to the library.

At this point all your libraries will be fairly low level and work on one page at a time. What if I wanted to do a larger action? Maybe create account would create a user, login, add information about the user.

You might be tempted to put a higher level function call into one of the existing classes. If you do this you will have one library calling another library. This is not a good idea as it can lead to circular references.

What I have found happens is you end up writing helper methods in the test cases. So of the test case was originally:

import com.company.application.pages.LoginPage;
import com.company.application.pages.HomePage;
import com.company.application.pages.RegisterUserPage;
import com.company.application.pages.UserProfilePage;
import com.company.application.datatypes.UserProfile;

class MySecondTestCase extends MyTestCase {
    LoginPage loginPage;
    HomePage homePage;
    RegisterUserPage registerUserPage;
    UserProfilePage userProfilePage;

    public void setUp() throws Exception {
        super.setUp();
        loginPage = new LoginPage();
        homePage = new HomePage();
        registerUserPage = new RegisterUserPage();
        userProfilePage = new UserProfilePage();
    }

    public void testCreateUser() throws Exception {
        String username = getProperty("username", "darrell");
        String password = getProperty("password", "mySecretPassw0rd");

        loginPage.goToRegisterNewUser();
        registerUserPage.fillInForm(username, password, password);
        registerUserPage.submitForm();
        loginPage.goToTheLoginPage();
        loginPage.logIn(username, password);
        homePage.goToUserProfile();
        userProfilePage.goToEditUserProfile();
        UserProfile profile = new UserProfile();
        // code to set the various fields of the user profile
        // e.g. profile.setHobbies("scuba diving, travelling, programming");
        userProfilePage.fillInForm(profile);
        assertEquals(userProfilePage.getUserProfile(), profile);
    }
}

I might change it to:

import com.company.application.pages.LoginPage;
import com.company.application.pages.HomePage;
import com.company.application.pages.RegisterUserPage;
import com.company.application.pages.UserProfilePage;
import com.company.application.datatypes.UserProfile;

class MySecondTestCase extends MyTestCase {
    LoginPage loginPage;
    HomePage homePage;
    RegisterUserPage registerUserPage;
    UserProfilePage userProfilePage;

    public void setUp() throws Exception {
        super.setUp();
        loginPage = new LoginPage();
        homePage = new HomePage();
        registerUserPage = new RegisterUserPage();
        userProfilePage = new UserProfilePage();
    }

    public void testCreateUser() throws Exception {
        String username = getProperty("username", "darrell");
        String password = getProperty("password", "mySecretPassw0rd");

        registerUser(username, password);
        logInAndGoToUserProfile(username, password);
        UserProfile profile = createAUserProfile();
        updateUserProfile(profile);
        assertEquals(userProfilePage.getUserProfile(), profile);
    }

    private void registerUser(String username, String password) throws Exception {
        loginPage.goToRegisterNewUser();
        registerUserPage.fillInForm(username, password, password);
        registerUserPage.submitForm();
    }

    private void logInAndGoToUserProfile(username, password) throws Exception {
        loginPage.goToTheLoginPage();
        loginPage.logIn(username, password);
        homePage.goToUserProfile();
        userProfilePage.goToEditUserProfile();
    }

    private UserProfile createAUserProfile() throws Exception {
        UserProfile profile = new UserProfile();
        // code to set the various fields of the user profile
        // e.g. profile.setHobbies("scuba diving, travelling, programming");
        return profile;
    }

    private void updateUserProfile(UserProfile profile) throws Exception {
        userProfilePage.fillInForm(profile);
        userProfilePage.submitForm(); // submitting profile send us to home page
        homePage.goToUserProfile();
    }
}

This is okay but if I need to register a user, go to a user profile from the login page or update a user profile from more than one test cases, I'm going to have code duplication. You want to avoid code duplication. The more you duplicate code the more maintenance work you are creating PLUS there is a chance you will miss one of the duplicate pieces of code.

So you want to move some of these methods to a more common location. You could create packages which are feature and use case oriented rather than page oriented. So your initial test cases will be very low level and test a page at a time. Next you start creating test cases which test features, use cases or stories. You can continue to run the page test cases but now you have much more powerful libraries. Just like with my example of MSDN libraries. Twenty years ago, programmers had to call the page tests. Later they called methods which called the page tests.

You can create 'requirement' test cases which call the page methods to test end-to-end requirements. You can create 'user-defect' methods. How a user does something or the data they use might reveal a defect. So you can create packages for all user defect reports then create a class for each customer. In each class will be the high level library calls for how they achieve something and the data they used to find a defect. Now you can create test cases which cross reference to the defect number and call these user-defect methods. If a project manager wants to see if a defect has been fixed, he can look to see of the corresponding test case has passed.

You want to think about how the test cases are organized, how they are executed, etc. as a set of manual test cases. The automation should reflect this because a project manager, QA manager, stakeholder, etc. might request a specific subset of the tests be run at different stages of the project.

Finally, you might have noticed there is not a lot of error handling. To keep this article short I have not been putting in error handling but you do. If you give a Java library bad input it will throw an exception. If you give your library method bad input, it should throw an exception and end the test case. If you were manually testing the application and an error appeared on the screen, you don't keep executing the test case. You STOP and investigate. With automation, you need to anticipate were things will go wrong. Any time you are submitting user input, an error can occur. Any time you are receiving input from outside the application (network, printer, COM call, etc.) an error can occur. You code should be constantly checking for things which can go wrong. An analogy is, every time you look at the screen with your eyes, the automation should be scanning the application for problems/errors.

Tuesday, March 23, 2010

One advantage of test automation - part 2

In my previous blog One advantage of test automation I talked about the saves of writing once and running many times. Essentially, if I have 8 configurations, it will take me 480 hours to test everything and I will test everything 12 times in a release I am looking at 5,760 hours of testing. Let's say the cost of a manual tester is $30/hour (salary, benefits, computer equipment, office space, vacation pay, support staff, etc.) then we are looking at $172,800 to manually test the application for the current release.

If I can write the automation once and run it 8 times (once for each configuration) I'm going to see significant savings. If automated testers cost twice as much, I'm going to see a cost savings of 4 times. Still looking good. So what I'm looking for is the cost of creating the initial framework but the cost of maintaining that framework needs to be less than 4 times as long as manual testing.

Here are four scenarios I have observed (the amount noted is the total cost from the beginning of the release):

Scenario #1

Iteration #1: manual=X, automated=X
Iteration #2: manual=2X, automated=2X
Iteration #3: manual=3X, automated=3X
Iteration #4: manual=4X, automated=4X
Iteration #5: manual=5X, automated=5X
Iteration #6: manual=6X, automated=6X
Iteration #7: manual=7X, automated=7X
Iteration #8: manual=8X, automated=8X
Iteration #9: manual=9X, automated=9X
Iteration #10: manual=10X, automated=10X
Iteration #11: manual=11X, automated=11X
Iteration #12: manual=12X, automated=12X

What is happening in scenario #1 is record and playback. The idea is that you buy the software, you record one iteration then play it back over and over. The cost of recording is the same as manual testing. You turn on record then manually test the application. Stop recording and save it. For the next iteration you simple play the recorded first iteration. You find there are timing issues. Changes to the application are causing tests to no longer work. Debugging what is wrong takes time. If you are lucky, it takes around the same amount of time as manual testing. But you paid for the software and training your staff. Let's say that is $20,000. You could have saved $20,000 if you just stuck to manual testing.

Scenario #2

Iteration #1: manual=X, automated=2X
Iteration #2: manual=2X, automated=4X
Iteration #3: manual=3X, automated=6X
Iteration #4: manual=4X, automated=8X
Iteration #5: manual=5X, automated=10X
Iteration #6: manual=6X, automated=12X
Iteration #7: manual=7X, automated=14X
Iteration #8: manual=8X, automated=16X
Iteration #9: manual=9X, automated=18X
Iteration #10: manual=10X, automated=20X
Iteration #11: manual=11X, automated=22X
Iteration #12: manual=12X, automated=24X

What is happening in scenario #2 is poor planning at the start of the automation. Not enough time was spent designing and implementing the test framework. Every time there is a new release of the application, the cost to maintain the automation is as much as the initial creation. We are not leveraging anything we did from the first iteration. Usually test automation is seen as a failure around the third or fourth iteration. When this happens, management is usually pretty apprehensive about trying test automation again. They didn't expect these sort of losses. If we use the initial estimate that manual testing costs $172,800 and assuming we abandoned test automation after iteration 3. The total cost of attempting automation is $216,000 ($43,200 more than just doing manual testing). And this is assuming we called it quits after iteration 3. Each iteration we hesitate on is costing the company an additional $14,400.

It is understandable why so many companies become disillusioned with test automation. Companies selling test automation software will pitch it as saving you 8 times the cost of manual testing. It looks like the cost of automation is going to be $21,600. Even if you double that and add $10,000 for the software ($53,200) it is better than one third the cost of manual testing. So you are expecting $55,000 and it ends of costing you $210,000 or more.

Scenario #3

Iteration #1: manual=X, automated=5X
Iteration #2: manual=2X, automated=6X
Iteration #3: manual=3X, automated=7X
Iteration #4: manual=4X, automated=8X
Iteration #5: manual=5X, automated=9X
Iteration #6: manual=6X, automated=10X
Iteration #7: manual=7X, automated=11X
Iteration #8: manual=8X, automated=12X
Iteration #9: manual=9X, automated=13X
Iteration #10: manual=10X, automated=14X
Iteration #11: manual=11X, automated=15X
Iteration #12: manual=12X, automated=16X

In scenario #3 we spent time creating a test framework but for some reason the cost of maintaining the test framework is the same as manual testing. Sometimes this is because we haven't really thought out the design or the application we are testing is in flux. Some applications are not ready for test automation. If your test automation efforts are failing, you need to look at why. If the application is changing so much that automators need to do some creative coding (which becomes difficult to maintain) or they end up throwing away a lot of the previous iteration automation, then you need to look at stabilizing the development process. Another thing you might have done wrong at this point is write different code for the different web browsers. If I have methods with:

if IE6 then do X
else if IE7 then do Y
else if FF3 then do Z

Or if I actually write one method for IE and another for Firefox then you are not really taking advantage of write once and run many. Additionally, you don't want to be too creative with the automation code. If you have to use a very complex regular expression to find a text field on all the different browsers then maybe the application is not ready for automation and you need to work with development to make the application friendlier to automation.

Scenario #4

Iteration #1: manual=X, automated=5X
Iteration #2: manual=2X, automated=5.5X
Iteration #3: manual=3X, automated=6X
Iteration #4: manual=4X, automated=6.5X
Iteration #5: manual=5X, automated=7X
Iteration #6: manual=6X, automated=7.5X
Iteration #7: manual=7X, automated=8X
Iteration #8: manual=8X, automated=8.5X
Iteration #9: manual=9X, automated=9X
Iteration #10: manual=10X, automated=9.5X
Iteration #11: manual=11X, automated=10X
Iteration #12: manual=12X, automated=10.5X

In scenario #4 we spent time creating a test framework and we were able to leverage the work in the previous iteration for the next iteration. Maintaining the automated test framework and adding new functionality after the initial iteration now costs us half as much as manual testing. At iteration #9 we are breaking even. By the end of the release we are looking at a cost savings of $21,600. This does not seem like a lot but if your automation team gets good at this or you hire someone with the right experience you can actually increase the saving. The cost of maintenance might not be linear. It could grow with each iteration. Or you might realize even more savings with someone who is already familiar with the pitfalls of automating.

Things to look for that might give you additional cost savings:

Someone with experience working with development
Someone with general experience creating automation frameworks
Someone with experience automating a similar application (web based, desktop application, mobile device, etc.)
Someone familiar with the automation tool you are using.

Obviously, someone who uses the same tool, has written automation for an application using similar technologies, a proven record of creating automation frameworks would be ideal and can work with development if the application is not friendly to automation tools.

I would say the most important factor is someone with experience working with development. If they are not comfortable working with development they might be set up for failure and no way to prevent it. Additionally, good automation is development. If the test automator is a developer, they should be comfortable working with development to make the application more amenable to automation.

Next would be experience creating an automation framework is the second most important factor. You can easily land yourself in scenario #2 if you don't know how to create a good test framework.

The second least important factor is testing similar applications. The technologies and problems for web testing are very different from desktop application testing. Additionally, if an application is using certain third party libraries, making it more amenable to automation might not be as easy. However, most automation is similar. Approximately 80% of the work will be the same regardless of the technology being tested. It is that last 20% which might make the difference. I'd also point out that there are dozen of resources on the various technologies and people within your company can help the automator understand the technology. Understanding the pitfalls and how to avoid them for automation frameworks and the soft skills to work with development are not as easy to find.

The least important factor is familiarity with the automation tool. This is not important if the automator has a proven record of learning new applications and technologies. For many experienced test automators, a new automation tool is trivial to learn. Test automators who know how to use a tool without understanding how and why it does when it does will be restricted to the tools they already know. If an automator understands the underlying principles behind all automation tools for a given technology then learning a new tool would be trivial.

One advantage of test automation

One big advantage of test automation is testing different configurations. For example, when I was working at Quest Software I tested web applications for multiple configurations. There were different operating systems with different web browsers. Here are some of the combinations:

Windows 2000, Internet Explorer 6.x
Windows 2000, Internet Explorer 7.x
Windows 2000, Firefox 3.x
Windows 2000, Safari 3.x
Windows 2000, Safari 4.x
Windows XP, Internet Explorer 6.x
Windows XP, Internet Explorer 7.x
Windows XP, Firefox 3.x
Windows XP, Safari 3.x
Windows XP, Safari 4.x
Windows Vista, Internet Explorer 7.x
Windows Vista, Firefox 3.x
Windows Vista, Safari 4.x
Solaris 10, Firefox 3.x
Redhat Linux, Firefox 3.x
SuSE Linux, Firefox 3.x

Without automation, we would look at equivalence classes. Using our experience and knowledge of the different operating systems, we would estimate that Internet Explorer 6.x on Windows 2000 is going to behave the same as Internet Explorer 6.x on Windows XP and Vista. We might also assume Firefox 3.x on Redhat Linux and SuSE Linux is going to be equivalent. Additionally, we would look at customer base and see if we can reduce the combinations because customers are unlikely to be using some of them. For example, we found that most Windows customers who were using Firefox 3.x were also on Windows XP. So the list was trimmed down to:

Windows 2000, Internet Explorer 6.x
Windows XP, Firefox 3.x
Windows XP, Safari 3.x
Windows Vista, Internet Explorer 7.x
Windows Vista, Safari 4.x
Solaris 10, Firefox 3.x
SuSE Linux, Firefox 3.x

We selected SuSE Linux because we had encountered issues unique to SuSE Linux. All issues found in Firefox 3.x on Redhat Linux were similar to Firefox 3.x on Solaris 10.

Looking at the list, there are still 7 configurations. If manually testing each configuration takes two weeks and a test cycle is two weeks, we need 7 testers working on this full time. This also assumes everything goes fine and there aren't additional issues to content with. The reality was that we had to add a couple more configurations. The different desktop managers for Linux actually made a different on how things rendered. It was really exposing problems in the web browser but from a customer's point of view, most websites look fine but our application does not, therefore the problem is in our application not the browser. Non-technical customers don't want to hear the explanation why our application doesn't work with the Redhat Linux 4, Gnome desktop and Firefox 3.x.

So we had to add in another configuration:

Windows 2000, Internet Explorer 6.x
Windows XP, Firefox 3.x
Windows XP, Safari 3.x
Windows Vista, Internet Explorer 7.x
Windows Vista, Safari 4.x
Solaris 10, Firefox 3.x
SuSE Linux, KDE Desktop, Firefox 3.x
Redhat Linux 4, Gnome Desktop, Firefox 3.x

This adds one more tester to the test cycle. If we assume testers will be productive 6 hours a day (checking mail in the morning, lunch, meetings, etc.) then we are looking at 6 hours * 5 days * 2 weeks * 8 configurations for a total of 480 hours.

But what if I had an automation tool that ran on all the platforms and worked with all the different web browsers? I could write one test suite and run it with different configurations. One such tool is Selenium. It is written using JavaScript and Java and therefore runs on all these platforms.

The test automation can be run after hours. This means 24 hours a day and 7 days a week. We can also run the test suite in parallel. Even if the test suite ran as slow as a manual tester, I just need to add another computer to run the test suite. A computer costs a lot less then a manual tester.

This is a little naive however. You will need at least one tester to automate the test suite. Since automation is development, it should be someone who knows how to create test strategies, test plans, test cases and knows how to do software development. This one employee is probably going to cost more than a manual tester would. However, they are not going to cost 8 times as much.

The reality is that creating a maintainable test automation framework takes longer than 2 weeks. Additionally, as you find defects time will be lost filing the defect and following up on it. Other issues which occur with this is that the implementation of Selenium on the different platforms will present their own set of problems. Basically, the same reason we find issues with our web application on the different platforms is going to cause issues with Selenium as well.

So be forewarned, test automation is not as great as some people would lead you to believe. If done correctly, there will be some cost saving. The best thing is once you have a good test framework in place, the maintenance of it does yield significant cost saving. The problem is, most people don't put enough effort into the design and implementation of the framework to reap the benefits later.

One huge things which can derail test automation will also add cost to development. One project I worked on looked the same from the customer's point of view regardless of which configuration they were using but if you looked at source control they actually had different codes for the different configurations. The web pages being sent down from the server were slightly different depending on the configuration. The code was littered with lots of "if IE6 then do one thing else if IE7 do another thing else if FF3 do something TOTALLY different". Patches to the web browsers and operating systems often broke the application.

From an automation point of view, if the web page served for IE6 is different from the web page served for FF3 then it is as if we are testing two different web pages. The worst part is that the pages are often, initially, close enough that an automator will try to code once for both pages. The effort to create a single automation script that works for both configurations will be more time consuming then normally anticipated and the maintenance for this script will be greater than expected.

But if coded well, the application can be automated for one configuration. A few tweaks for different configurations and then run continuously on all configurations.

Saturday, March 20, 2010

Ways to facilitate communication

One of the most important things on a team is communication. I have worked on teams were everything was put into a document. Unfortunately, as the project progressed, time became more and more precious. People needed to be more efficient and far too often, the documentation suffered. Decisions would be made between two individuals and the rest of the team would miss out.

So how do we make sure everything is well communicated? The simplest answer is to talk to each other.

In scrum you have sprints and planning meetings. What I would do is at the end of the week look at the back log (features and defects in the tracking system) and decide what I could get done be the end of next week. I might add a few notes to the defect report and assign it to myself.

On Monday morning we would have the planning meeting for the week. Everyone would decide what features and bug fixes should and could be completed that week. If there was anything which would take more than one week to complete, it need to be broken down into multiple, smaller tasks. No task should take more than one week. This means implementing and testing the feature/bug fix should happen the same week.

Everyone would pick the defects they were going to work on and give estimates for how long each would take. Every day we would meet and go over the status of the team. Have things changed? Did sales land a new contract but promise a feature we need to complete this week? Or is that feature multiple tasks and we want to start on one of those tasks this week? If yes, we reassess the plan and change priorities.

The key to all of this is that developers, business analysts, testers, project stakeholders all meet and decide what will be done each week and continue to track progress on a daily basis. Most people who do scrum understand this.

What about the time between each scrum? What if you need clarification on the feature you are working on? Whether you are a developer or a tester you might need further clarification. If everyone is in the same room you could just shout to the people involved. This could be disruptive to the other people in the room.

So how do you deal with it? We could create teams and partition the rooms. Each room would have a tester, a developer and a business analyst. What about the project manager? There will usually be one project manager and multiple developers. We cannot have one manager to each developer. Additionally, I have never worked on a team were the developer to tester ratio was 1:1. So this idea is not going to work.

What has worked for me in the past is everyone is on MSN Messenger or Google Talk. These tools let you talk with one or multiple people at once. You can set up aliases for the different groups you need to talk with. This is really helpful if you are working in an open concept office or a distributed team. In the case of an open concept office, shouting to all interested parties might work for you but it is disruptive to the people in the room who are not involved in the conversation. By using chat software I can ask quick questions of just the people who need to be involved. The same is true for people at remote locations. I can chat with everyone involved without having to book a conference room and have everyone call in.

An added benefit is the ability to opt out of a conversation. If I am working on something which requires my full concentration I can log out of chat. This send a signal to everyone that I cannot be disturbed. The danger of this is forgetting to log back into chat but hopefully, you will just get into the habit of remembering. Everyone checks email after they have been working hard on a feature which required their full attention. If someone is not logging back into chat and you need to talk with them, send them an email. When they are done it will be a gentle reminder to log back into chat.

If you find that chat is becoming too tedious and requires too much typing, switch to a meeting. Book a conference room and invite everyone to a 30 minute or 1 hour meeting. If the team is distributed you can use Skype or VOIP to get everyone calling in.

Friday, March 19, 2010

How does a conventional tester fit into an agile environment?

When I think 'agile' I think test driven development (TDD). In TDD, the programmers actually write the tests first then write code to make the tests pass. If you look at something like Eclipse IDE, I can write a jUnit class that creates the code to instantiates a class that does not exist yet. Eclipse will complain about the non-existent class and offer to create the class for me (without the body). If I write calls to methods which do not exist, Eclipse will complain about the non-existent methods and offer to write the method for me (without the body).

In Agile development you have sprints. A sprint is typically a week. On most agile projects I have worked on, the QA staff are involved from the very beginning of each sprint. Every Monday (or some other day) all the stakeholders (developers, business analysts, testers, etc.) get together to decide what will be completed in the next sprint. They will come up with "user stories" (sort of like requirements). Everyone signs off on the sprint (meaning the stories should cover all the requirements and from that all the test scenarios plus test cases).

As each feature is written, there is daily deployment (or more often). As QA Tester I would write automation to conduct Integration and System level tests.

The most important aspect of agile development is fast turn around. On a traditional SDLC, I would often not see a functioning application until weeks after a developer has created it. By the time I am testing the first feature the developer could be done with that feature and working on something totally different. If I file a defect report on a feature a developer completed 3 weeks ago, she has to stop what she is working on, recall the code for the feature I am testing, fix the bug then get back to the new feature she was originally working on. This can be quite disruptive.

With agile development, a developer makes a change, compiles the code, runs the unit tests. If something she did breaks a unit test she knows in seconds. If it passes the unit test she can check the change in. The QA Tester can then build and test the application.

With the frameworks I worked on, the application would get built automatically by a nightly build process. All unit tests would get run on it. There might also be static analysis tools run on the project before compiling. Once the project is built and unit tests pass it would get deployed to test machines and the QA automation suite would be run. This all happens after business hours. In the morning, I would examine the test results. Sometimes a change in the program breaks my automation so I would do some maintenance on the automation and re-run the tests which failed. If there was an actual failure in the application and my automation found it, I would file a defect report.

On a daily basis the team would meet to talk about what was done the previous 24 hours and what would be the goal for each person in the next 24 hours (this is scrum, a form of agile development). I would bring up any defects I found at the scrum.

Fast turn around and immediate feedback is key to a good agile development. I would write regression tests to make sure defects found and fixed do not creep back into the product but I would also work with the developers to bring the regression tests into the unit test framework. This way the defect can be caught sooner should it creep back in.

Because the number of regression test grow with each iteration (sprint), it is recommended that they be automated. This frees the test team to conduct manual test efforts towards new features.

The most important thing to understand as a tester in an agile development environment is that there is a STRONG belief in minimal documentation. Source code should be self-documenting. The reason for this, and I have seen this on EVER traditional development team, is because the documentation is often neglected. No documentation is often better than wrong documentation. If there is no documentation and the developer has not gotten to the point were their code is self-documenting, they will be forced to look at the code and figure out what it does, rather than read outdated documentation and believe it does something different than it really does.

To understand the requirements of the software, you look at the unit tests. These should also be self-documenting and there should be one unit test for each requirement (user story).

As a tester, I will not expect to get a set of requirements. If I need to, I will take notes at the planning meeting (beginning of each sprint) and update them at each daily meeting (scrum).

The key to success for the tester will be communication. Without communication between the tester, the developer, business analysts and project stakeholders, it will be extremely difficult for the tester to properly test the application.