maintenance versus performance

I was recently sent an example xpath for Selenium automation. The xpath, for a DIV, was:

"//*[@class="val1 val2 val3 val4"

My experience has been, especially with Internet Explorer, that xpath is very slow compared to say CSS. However, I find xpath easier to read (thus easier to maintain, especially in a team enviroinment) and certain things are just not possible in CSS.

So there is always a balance of maintenance versus performance. The golden rule for optimization is to never optimize right away. Right the code for maintainability and if you need to improve performance, then and only then optimize it.

The first thing I notice about the xpath above is the = operator. If the developer adds another value to the class attribute or the order of the values change, this will cause the automation to require maintenance. My experience has been that the values of multiple value attributes do change over time. So from a maintenance perspective I would write this as:

"//*[contains(@class,'val1') and contains(@class,'val2')
 and contains(@class,'val3') and contains(@class,'val4')]"

This way, if the order of the attributes change or the developer adds another value the xpath will still find the element.

The other things I'd change about the xpath is the use of //*. This will search every element for a match. Rather inefficient. It does have the nice feature that if a developer changes the element from a DIV to something else, the xpath will require no maintenance. However, and this is just from years of experience, I find it rare that a developer changes a DIV to something else. If they do, there is a huge amount of refactoring and using //* will not save you from any maintenance.

Since the likelihood of maintenance is low but it is such an obvious performance hit, I would actually use //div for the xpath.

How to do focused testing

After years of software testing I have found one constant... there is never enough time to test everything.

So how do I pick what I'm going to test? Here are a few tricks I use to focus what I'm going to test.

First, test what changed. I like to get a snapshot of the last release (source code) and a snapshot of the current release candidate then compare them. Often the source code is readable enough for any tester to understand what has changed. If you are not a programmer, not familiar with the particular language or the code requires a little indepth knowledge only the programmers have then you might want to talk with some of the developers about what you are looking at.

For example, if I am testing a shopping cart for an online web store and I see changes to a class called TaxRules then I'm going to look at the requirements for tax rules and see if there is anything new. Even if there aren't requirements for tax rules, I can talk to the person who wanted this feature (or improvement) and I can look at the code. Comments will be for humans and I'm human (no really, I am). Good code should also be self documenting. In other words, the variables and method names should almost read like English.

The next trick is to get someone to add a code coverage tool to the web site (or whatever it is you are testing). As you test the application (automated or manually) the coverage tool will record which parts of the code you executed. Look to see of the code which has changed was executed. Set the coverage tool so it records actions within the methods and not just method level data. This way you can see which branch conditions were called. If there is a loop which never gets entered or a portion of an if/elsif/else which does not executed, you know you need another test case. You could even ask the developer to relate the statements missed to user actions. Essentially, get them to tell you what test case you missed. :)

Finally, look at the inputs to the methods. Can you see a way to affect the inputs? Does the method handle situations where the data is bad? Developers think in terms of "make the program do what it is supposed to do." They always make it do the right thing when given the right data. As a tester you need to think in terms of "make sure the program doesn't do what it is not supposed to do." In other words, if you input bad data, it should report a nice, helpful message to the user or just restrict the user's ability to input the bad data.

As you do code coverage, you can keep notes with your test cases. Just a comment about how test case XXX should be run with class YYY is altered.