How to develop an automation framework for a legacy application

If you join a team testing an application which already exists and may even be released to the customer but there is no automation in place, how do you start?

For many this can be an overwhelming task. The real answer is, one feature at a time.

Let's take for example the last project I worked on. It was created years before I joined the company and had a small base of customers who depended on it and tolerated the quirks and bugs of the system.

The first thing to do is pick an automation tool that will work for the application. Talk to the developers, business analysts and stakeholders to get a feel for where the project is going and keep that in mind. For example, current requirements are Windows and Internet Explorer 7. In the future, we would like to support Windows, Linux, Solaris, etc. and we want support for any and all browsers (Internet Explorer, Firefox, Opera, Safari, Mozilla, etc.). Additionally, we also have a Windows Mobile application but we would like to change to using the web browser on iPhone, Palm Pre or BlackBerry. So do we use a tool that supports all these possible future combinations or do we use a tool that works for the current requirements because the future requirements aren't written in stone and 95% of our customers for the next few years will be on Windows with Internet Explorer 7 (or IE8 in IE7 compatible mode).

The tools I found were a very mature product which only supported Internet Explorer or a newer product which supported everything but wasn't quite as mature. I selected Watij, a more mature product which only supported Internet Explorer. The majority of our customer were Internet Explorer customers, all the developers did their work using Firefox. The defects which appeared in one web browser and not another were typically layout issues. To date, all the layout issues were in Internet Explorer because the developers were using Firefox to do their design.

Next you want to think about code maintenance. This might seem strange; we don't have a single line of code and I'm thinking about how I'm going to maintain this code that doesn't exist.

Think about successful products. Microsoft Office was released 20 years ago. JBoss was started over 10 years ago. Photoshop 1.0 was released 20 years ago. The last proven web based application I tested was started in 2004 and is still going strong. Basically, a successful software project can exist for 10 to 20 years. How long do you think you have to create a test framework? If the software is going to be released within one year, the project manager is going to expect the framework and initial test cases to be written in the first release cycle. So you have less than a year to create your framework. Thus, 6 months to 1 year to create and use the framework on, hopefully, a project which will last for 10 to 20 years. So 90% of your time is going to be on maintenance. Even if we expect the framework to be used for 5 years you are still looking at 80% of your time spent on maintenance.

So plan out how to break the framework into small, manageable pieces. Look at something like a library. Take for example the Java APIs. There are over 200 packages with over 3700 classes. A typical class might have dozens of methods. Did this happen overnight? Absolutely not. James Gosling started work on Java (aka Oak) in 1991 and by 1995 v1.0 was released. So the 3700+ classes were developed over 19 years.

Creating a successful framework doesn't mean all the code has to be in place at version 1.0. It just means the structure has to be there. If you look at version 1.0 of Java, it was object oriented, it had packages, constructors, exceptions, etc. All the basic functionality of the language which exists today was there in v1.0. Was there support for XML, Xpath, SQL? I'm not sure but there didn't need to be. There just had to be enough to produce something useful.

So your framework just need to be structured in such a way that it will grow to something we can use 10 years from now. So how do you create something which will be comparable to Java, C++, etc.?

a dwarf standing on the shoulders of a giant may see farther than a giant himself.

Borrow the design of something like Java. If the language you are using for automation is going to be object oriented, use proven object oriented designs. Leverage the work of James Gosling. Look at the application you are testing. Can you break it apart into sub-sections? For a web application you have pages. Each page will have a variety of actions. Some will alter the current page using JavaScript and some will load a new page. Focus on the functionality of just that page. The new page loading will be in some other package, class, method.

Looking at things like MSDN library I noticed the original libraries where very detailed. To accomplish one thing you often had to call numerous library functions. Over the years Microsoft recognized that most programmers will call:

result1 = functionA();
result2 = functionB(result1);
result3 = functionC(result2);
result4 = functionD(result3);
print result4;

So they created a new library where you called:

print functionABCD();

If you look at the existing code in Microsoft you will see:

functionABCD()
{
    result1 = functionA();
    result2 = functionB(result1);
    result3 = functionC(result2);
    result4 = functionD(result3);
}

You want to build things up the same way. You can have separate projects, separate packages, different naming conventions. The choice is all up to you. Just remember that the division will look silly at first but by this time next year you will be happy you broken it down as much as you did. What I mean is you might find you have 2 packages, each package has 3 or 4 classes, each class has 5 or 6 methods. In a few years time you should find you have hundreds of packages, each package has dozens of classes and each class has numerous methods.

Additionally, do things on the page translate to data structures that need to get passed around? For example, on a web page you will have forms to fill in. Later you might need to edit the data you input. The data required for creation (filling in the initial form) and the data required for editing will be the same data. So create data structures that mirror the form. If the data on the form changes, you just need to edit the data structure. Use getters and setters to obfuscate away the implementation of the data structure. For example, you might store the date string as a Date or Calendar object. Later the web page may change from a text field to a read-only text field with a Calendar widget. If you hardcoded a string for the date field you will have to go and clean up all your automation. If you successful obfuscated away the data type, you should be able to just update one library call and the automation will continue to work. Imagine they decide to change the web form two years from now. You have 4,879 test cases which fill in that form. How long will it take to find and fix 4,879 test cases? How long will it take to update one library call?

Once you have decided on how to structure the framework you can start writing test cases. You might have noticed, I talked about designing the framework but I didn't write any of the code. The reason for this is because you want to have the test cases drive the code creation. If you have test plans in place and the priority of the test cases (conducted manually) is known, then the order you want to automate the test cases is also known. Just as you are not going to start manually testing a minor feature before you test a major feature, you are not going to start automating a minor feature before you automated a major feature.

My first test case might be something like:

class MyFirstTestCase {

    public void setUp() throws Exception {
    }

    public void testLoggingIn() throws Exception {
        String username = getProperty("username", "darrell");
        String password = getProperty("password", "mySecretPassw0rd");

        loginPage.goToTheLoginPage();
        loginPage.logIn(username, password);
        assertTrue(homePage.assertHomePage());
    }

    public void tearDown() throws Exception {
    }
}

If you enter this into an IDE like IntelliJ/IDEA or Eclipse you will get a lot of error messages. The getProperty method does not exist. The loginPage object does not exist. The goToTheLoginPage method does not exist. But the IDE has a helpful feature, it will make suggestions as to how to fix the errors.

It will tell you the getProperty method does not exist and do you want to create it. You can create it inside the current test class but won't other test classes need to get properties as well? So maybe you want to extend the test class and put the getProperty method in a super class. So I would go to the class definition and add an extends statement. Now I get an error in the extends statement. So I take the suggestion to create a new super class. The class will be empty and the error will go away. Now when I deal with the getProperty error, one of the suggestions is to create a method in the super class, so I do.

As you resolve each error, the test framework starts growing. You keep adding in more and more code. Once you have resolved all the errors in your first test case, you have JUST enough code in the test framework to run one test cases. The test case should now be runnable and you can add it to the nightly build process. Each night it will build the application, deploy it and run your one test. Next day, add another test. Are there things from the framework you can use for test case number two? Then reuse them.

If you find yourself putting code into more than one place, move the code to a library and change the two places you are using it into a call to the library.

At this point all your libraries will be fairly low level and work on one page at a time. What if I wanted to do a larger action? Maybe create account would create a user, login, add information about the user.

You might be tempted to put a higher level function call into one of the existing classes. If you do this you will have one library calling another library. This is not a good idea as it can lead to circular references.

What I have found happens is you end up writing helper methods in the test cases. So of the test case was originally:

import com.company.application.pages.LoginPage;
import com.company.application.pages.HomePage;
import com.company.application.pages.RegisterUserPage;
import com.company.application.pages.UserProfilePage;
import com.company.application.datatypes.UserProfile;

class MySecondTestCase extends MyTestCase {
    LoginPage loginPage;
    HomePage homePage;
    RegisterUserPage registerUserPage;
    UserProfilePage userProfilePage;

    public void setUp() throws Exception {
        super.setUp();
        loginPage = new LoginPage();
        homePage = new HomePage();
        registerUserPage = new RegisterUserPage();
        userProfilePage = new UserProfilePage();
    }

    public void testCreateUser() throws Exception {
        String username = getProperty("username", "darrell");
        String password = getProperty("password", "mySecretPassw0rd");

        loginPage.goToRegisterNewUser();
        registerUserPage.fillInForm(username, password, password);
        registerUserPage.submitForm();
        loginPage.goToTheLoginPage();
        loginPage.logIn(username, password);
        homePage.goToUserProfile();
        userProfilePage.goToEditUserProfile();
        UserProfile profile = new UserProfile();
        // code to set the various fields of the user profile
        // e.g. profile.setHobbies("scuba diving, travelling, programming");
        userProfilePage.fillInForm(profile);
        assertEquals(userProfilePage.getUserProfile(), profile);
    }
}

I might change it to:

import com.company.application.pages.LoginPage;
import com.company.application.pages.HomePage;
import com.company.application.pages.RegisterUserPage;
import com.company.application.pages.UserProfilePage;
import com.company.application.datatypes.UserProfile;

class MySecondTestCase extends MyTestCase {
    LoginPage loginPage;
    HomePage homePage;
    RegisterUserPage registerUserPage;
    UserProfilePage userProfilePage;

    public void setUp() throws Exception {
        super.setUp();
        loginPage = new LoginPage();
        homePage = new HomePage();
        registerUserPage = new RegisterUserPage();
        userProfilePage = new UserProfilePage();
    }

    public void testCreateUser() throws Exception {
        String username = getProperty("username", "darrell");
        String password = getProperty("password", "mySecretPassw0rd");

        registerUser(username, password);
        logInAndGoToUserProfile(username, password);
        UserProfile profile = createAUserProfile();
        updateUserProfile(profile);
        assertEquals(userProfilePage.getUserProfile(), profile);
    }

    private void registerUser(String username, String password) throws Exception {
        loginPage.goToRegisterNewUser();
        registerUserPage.fillInForm(username, password, password);
        registerUserPage.submitForm();
    }

    private void logInAndGoToUserProfile(username, password) throws Exception {
        loginPage.goToTheLoginPage();
        loginPage.logIn(username, password);
        homePage.goToUserProfile();
        userProfilePage.goToEditUserProfile();
    }

    private UserProfile createAUserProfile() throws Exception {
        UserProfile profile = new UserProfile();
        // code to set the various fields of the user profile
        // e.g. profile.setHobbies("scuba diving, travelling, programming");
        return profile;
    }

    private void updateUserProfile(UserProfile profile) throws Exception {
        userProfilePage.fillInForm(profile);
        userProfilePage.submitForm(); // submitting profile send us to home page
        homePage.goToUserProfile();
    }
}

This is okay but if I need to register a user, go to a user profile from the login page or update a user profile from more than one test cases, I'm going to have code duplication. You want to avoid code duplication. The more you duplicate code the more maintenance work you are creating PLUS there is a chance you will miss one of the duplicate pieces of code.

So you want to move some of these methods to a more common location. You could create packages which are feature and use case oriented rather than page oriented. So your initial test cases will be very low level and test a page at a time. Next you start creating test cases which test features, use cases or stories. You can continue to run the page test cases but now you have much more powerful libraries. Just like with my example of MSDN libraries. Twenty years ago, programmers had to call the page tests. Later they called methods which called the page tests.

You can create 'requirement' test cases which call the page methods to test end-to-end requirements. You can create 'user-defect' methods. How a user does something or the data they use might reveal a defect. So you can create packages for all user defect reports then create a class for each customer. In each class will be the high level library calls for how they achieve something and the data they used to find a defect. Now you can create test cases which cross reference to the defect number and call these user-defect methods. If a project manager wants to see if a defect has been fixed, he can look to see of the corresponding test case has passed.

You want to think about how the test cases are organized, how they are executed, etc. as a set of manual test cases. The automation should reflect this because a project manager, QA manager, stakeholder, etc. might request a specific subset of the tests be run at different stages of the project.

Finally, you might have noticed there is not a lot of error handling. To keep this article short I have not been putting in error handling but you do. If you give a Java library bad input it will throw an exception. If you give your library method bad input, it should throw an exception and end the test case. If you were manually testing the application and an error appeared on the screen, you don't keep executing the test case. You STOP and investigate. With automation, you need to anticipate were things will go wrong. Any time you are submitting user input, an error can occur. Any time you are receiving input from outside the application (network, printer, COM call, etc.) an error can occur. You code should be constantly checking for things which can go wrong. An analogy is, every time you look at the screen with your eyes, the automation should be scanning the application for problems/errors.