maintenance versus performance

I was recently sent an example xpath for Selenium automation. The xpath, for a DIV, was:

"//*[@class="val1 val2 val3 val4"

My experience has been, especially with Internet Explorer, that xpath is very slow compared to say CSS. However, I find xpath easier to read (thus easier to maintain, especially in a team enviroinment) and certain things are just not possible in CSS.

So there is always a balance of maintenance versus performance. The golden rule for optimization is to never optimize right away. Right the code for maintainability and if you need to improve performance, then and only then optimize it.

The first thing I notice about the xpath above is the = operator. If the developer adds another value to the class attribute or the order of the values change, this will cause the automation to require maintenance. My experience has been that the values of multiple value attributes do change over time. So from a maintenance perspective I would write this as:

"//*[contains(@class,'val1') and contains(@class,'val2')
 and contains(@class,'val3') and contains(@class,'val4')]"

This way, if the order of the attributes change or the developer adds another value the xpath will still find the element.

The other things I'd change about the xpath is the use of //*. This will search every element for a match. Rather inefficient. It does have the nice feature that if a developer changes the element from a DIV to something else, the xpath will require no maintenance. However, and this is just from years of experience, I find it rare that a developer changes a DIV to something else. If they do, there is a huge amount of refactoring and using //* will not save you from any maintenance.

Since the likelihood of maintenance is low but it is such an obvious performance hit, I would actually use //div for the xpath.

How to do focused testing

After years of software testing I have found one constant... there is never enough time to test everything.

So how do I pick what I'm going to test? Here are a few tricks I use to focus what I'm going to test.

First, test what changed. I like to get a snapshot of the last release (source code) and a snapshot of the current release candidate then compare them. Often the source code is readable enough for any tester to understand what has changed. If you are not a programmer, not familiar with the particular language or the code requires a little indepth knowledge only the programmers have then you might want to talk with some of the developers about what you are looking at.

For example, if I am testing a shopping cart for an online web store and I see changes to a class called TaxRules then I'm going to look at the requirements for tax rules and see if there is anything new. Even if there aren't requirements for tax rules, I can talk to the person who wanted this feature (or improvement) and I can look at the code. Comments will be for humans and I'm human (no really, I am). Good code should also be self documenting. In other words, the variables and method names should almost read like English.

The next trick is to get someone to add a code coverage tool to the web site (or whatever it is you are testing). As you test the application (automated or manually) the coverage tool will record which parts of the code you executed. Look to see of the code which has changed was executed. Set the coverage tool so it records actions within the methods and not just method level data. This way you can see which branch conditions were called. If there is a loop which never gets entered or a portion of an if/elsif/else which does not executed, you know you need another test case. You could even ask the developer to relate the statements missed to user actions. Essentially, get them to tell you what test case you missed. :)

Finally, look at the inputs to the methods. Can you see a way to affect the inputs? Does the method handle situations where the data is bad? Developers think in terms of "make the program do what it is supposed to do." They always make it do the right thing when given the right data. As a tester you need to think in terms of "make sure the program doesn't do what it is not supposed to do." In other words, if you input bad data, it should report a nice, helpful message to the user or just restrict the user's ability to input the bad data.

As you do code coverage, you can keep notes with your test cases. Just a comment about how test case XXX should be run with class YYY is altered.

Tools to determine how to use Selenium

Being able to view the source code to a page is helpful when trying to use Selenium. Many of the methods in Selenium require you to locate the element via something related to the source code. For example, if I wanted to click on an element I need to look at the source code. From the user's point of view they might see:

Click Me

But the underlying source code could be:

<span>

    <div>

        <table>

            <tbody>

                <tr><td><strong onclick="javascript:clickme();">Click Me</strong></td></tr>

            </tbody>

        </table>

    </div>

<span>

I might assume there is an anchor and I want to find the <a> element with the text 'Click Me'.  By looking at the source code I can see that I actually want to find a <strong> element and click on it.

Just viewing the source code isn't always sufficient however. AJAX and Web 2.0 means that the source code gets changed in memory after it is loaded. So what is actually available in the DOM (and Selenium) might be different than what you see when viewing the source code.

So how do you see what Selenium sees? A simple solution is to write your Selenium code to get to a point in the program then print out the source code using Selenium. Selenium 1.0 and 2.0 both have methods to return the HTML source code as a string, which you can print.

The problem with this is that you might have to write a program to print the source. Then with the source, find the code you want to click. Then run the program so it clicks the element then prints the new source. Then run the program to click the first element, click the second element and print the source. For n steps this will take O(n^2) run time. Not very efficient.

So how can you view the dynamically changing source? The answer is in browser tools.

If you are using IE7, search for "microsoft dev toolbar". When you install it, it gives you development tools which can view the page in real time. If you are using IE8, the tool is built in. Just press F12 and it will display the development tools.

If you are using Firefox, install the Firebug extension.

If you are using Google Chrome, right click an element and select Inspect to bring up the development tools.

Defensive programming

Had an interesting conversation with a programmer today. He created a program for updating pricing on books. It allowed you to update the price of one book at a time or to load a list of volume ids (sort of like ISBNs) and batch update the prices.

Many of our partners use Excel to manage their list of books. So the programmer was told the batch screen had to load Excel files.

My first thought when testing the application was, "How can users mess up an Excel file?"

The application required a header row with specific text. What if:

• there were extra columns
• a column was missing
• there was whitespace before or after the text
• there was no header row
• the text was all lowercase
• the text was all uppercase
• the text was camel cased (ThisIsCamelCase)

Tried all these and each one crashed the program. The programmer was using a third party library to access the Excel spreadsheet. I had used similar libraries in the past and knew these to be problems with those libraries. Essentially, you need to guard against bad data as these libraries don't.

When LAMP (Linux/Apache/MySQL/Php|Perl|Python) programmers access the database, they never assume the data coming in is safe. They check all the data before using it to access the database. This is because hackers will do things like input the following for First Name: "Hacker; DROP TABLE ..." or some other destructive text.

The lesson learned from this is to be a defensive programmer. LAMP programmers know this. I believe this to be true for people using third party libraries but also when using the code from someone else on your own team. If you assume the library does not handle edge cases and ensure the data has been checked before getting passed to the other programmer's code.

If you assume all code (third party libraries, your old code, code from other programmers on your team, etc.) does not handle code properly and check the inputs before passing them along to that code, your applications will be much more solid.

To the testers reading this, assume all programmers will assume someone else will take care of guarding against bad inputs. So think of all the bad inputs you can and send them in. You'll usually find they make it quite far before someone handles them. Often it will be the operating system that handles this with a crypt error dialog and a crash.

Asking for automation help

I've noticed a trend on help forums and groups. When people ask for help with their automation that often post what they tried and that it did not work.

They don't give any information about their environment, versions of the various software involved or the code they are trying to automate.

Asking for help with your automation is similar to filing a defect report.

• Specify a summary of the problem
• Give a description of the environment (OS, browser, etc.)
• Tell them what you did, what you expected to happen and what actually happened.

There is one additional piece of information you normally put in a defect report. You normally specify which build the defect appeared in. This is because the developer fixing the bug can find the appropriate code given a build number.

When asking people outside your organization for help with an automation problem, we don't have access to your source code. So a build number does not help us. What we need instead is a code snippet. If you are testing a web site, give us the relavent HTML code you are trying to automate.

Without these bits of information it would be impossible to solve the problem. At best, you will get a lot of guessing. If you are lucky, someone will make a lucky guess or say something which makes you realize what the problem really is. However, if you want a quicker response to your questions give us the information we need to reproduce the problem and solve it.

Test Web APIs

Testing web APIs essentially breaks down into:

• Building an HTTP Request
• Sending the HTTP Request
• Get the HTTP Response
• Verify the HTTP Response

If you look at RFC 2616 it will tell you everything you need to know about HTTP/1.1. However, if you are not used to reading RFC, this document can be quite confusing.

Here some highlights to an HTTP Request...

If you go to a website and they have a FORM, submitting the FORM will do an HTTP Request. The FORM tag will have an action attribute. In side the FORM will be INPUT, SELECT, BUTTON, etc. tags.

For example, the Google home page is a FORM. The action for the FORM is "/search". The INPUT has a name attribute of "q". If I enter "HTTP Request" into the INPUT field, my web browser builds an HTTP Request and sends it to Google. You can type the following URL into an address bar:

http://www.google.com/search?q=HTTP%20Request&hl=en

This is the same as entering 'HTTP Request' into the input field and clicking the Search button. The one oddity is the %20. A proper HTTP Request has no spaces in it. So you need to convert spaces to a hexidecimal value. The hexidecimal value 20 is ASCII for a space. Also symbols like / : & etc. have to be convert as well. If the symbol is part of the Request and not a value being passed you don't convert it. The general format is:

http://hostname:port/action?key=value&key=value&key=value

If you leave out the port it defaults to 80 (443 for https).

At this point you might be wondering, what does this have to do with API testing? A lot. Many APIs are implemented using HTTP Request/Response. So you need to send them an HTTP Request just like your web browser creates an HTTP Request. With the web browser, it adds a lot more you might not be aware of. The HTTP Request has the URL, a header and sometimes data. Your web browser will secretly add header information like:

Host: www.google.com
Accept-Language: en-us,en;q=0.5

Finally, the FORM can be a POST or a GET form. If it sends data using the URL, like the above examples, it is a GET request. If it sends the data in such a way the user doesn't see it, it is a POST request. For example, if I was logging into a website, I would use a POST so the password isn't visible in the address bar.

So now we have talked about all three parts. The URL, the HEADER and the DATA.

Let's say I design an API such that I expect some data to be in the URL, some in the header and some in the data. So lets say the URL will be:

https://www.mywebsite.com:8443/accounts/admin/requestAccount?company=Acme&auth=Darrell

In the header I want them to identify which machine the request is coming from and an API Key:

Request-Host: 127.0.0.1
Company-API-Key: foobar7

And finally, the data will be an XML file with the following information:

    <?xml version=\"1.0\" encoding=\"utf-8\"?>
    <RequestAccount>
    <CompanyName>Acme</CompanyName>
    <EmployeeID>10318630</EmployeeID>
    <FirstName>Bob</FirstName>
    <LastName>Johnson</LastName>
    </RequestAccount>

So how do I take this knowledge and using it? Let's say the XML data is saved in the text file foo.txt and I have curl installed on my machine (comes default on UNIX, Linux and Mac OS X machines; you can get and install a Windows version for free).

The curl command to use this would be:

curl -k -d @foo.txt -H "Request-Host: 127.0.0.1" -H "Company-API-Key: foobar7" -o response.txt https://www.mywebsite.com:8443/accounts/admin/requestAccount?company=Acme&auth=Darrell

The -k makes it so we don't worry about SSL certificates and authentication, the -d is used to send the data, the -H adds header information, the -o saves the response to the file.

The great thing about curl is you can try something, make a slight change, try again, make a slight change, try again. So if you wanted to try a bunch of different data files, you can do this quickly and easily with a shell for loop. You can play with the header information or leave/change some of the URL fields.

You can quickly probe and test a set of APIs using curl. There are more options to curl but this is the basic usage.

Upgrading Selenium 2.0b2 to 2.0b3 - Python

Selenium 2.0b3 was recently released. My current company is using Selenium 2.0 for web testing and the preferred language for the team is python. If you are familiar with python you might know that upgrading a site-package (like Selenium) is as simple as:

pip install selenium

One of the staff did this and immediately the test suite broke. The line of code which broke was:

host="localhost"
port=4444
browser="firefox"
wd = webdriver.Remote(command_executor='http://' + host + ':' + str(port) + '/wd/hub', browser_name=browser, platform='ANY')

One of the nice things about using Eclipse (my IDE of choice) is you can hold down the CTRL key and click a method to jump to the source code. So I held down the CTRL key (actually it was the Command key because I'm on a Mac) and clicked on Remote. This took me to the class for Remote. The __init__ method is the constructor.

When I looked at the constructor for 2.0b3 it was immediately obvious that the constructor had changed. The old constructor (2.0b2) was:

def __init__(self, command_executor='http://localhost:4444/wd/hub', browser_name='', platform='', version='', javascript_enabled=True)

but the new constructor (2.0b3) is:

def __init__(self, command_executor='http://127.0.0.1:4444/wd/hub', desired_capabilities=None, browser_profile=None)

So the first parameter, command_executor, is the same but all the other parameters have been changed to a desired_capabilities. So what is desired_capabilities? It is a python dictionary. The format for a dictionary is:

dc = { "browser_name" : browser, "platform" : 'ANY' }

This one little change almost fixed things. A little more digging into the source code and I found that the dictionary should be using "browserName" and there doesn't seem to be any support for platform anymore. Since we only care about the browser, I changed the dictionary to:

dc = { "browserName" : browser }

and this solved everything.

Lesson Learned: you have to dig into the source code to figure out what is going on.

P.S. reading the comments in the source code indicated that the start_session method, which uses the desired_capabilities, it still talks about the parameters browser_name and platform. It is clear that the comments have not been updated to reflect the code.