Defensive programming

Had an interesting conversation with a programmer today. He created a program for updating pricing on books. It allowed you to update the price of one book at a time or to load a list of volume ids (sort of like ISBNs) and batch update the prices.

Many of our partners use Excel to manage their list of books. So the programmer was told the batch screen had to load Excel files.

My first thought when testing the application was, "How can users mess up an Excel file?"

The application required a header row with specific text. What if:

• there were extra columns
• a column was missing
• there was whitespace before or after the text
• there was no header row
• the text was all lowercase
• the text was all uppercase
• the text was camel cased (ThisIsCamelCase)

Tried all these and each one crashed the program. The programmer was using a third party library to access the Excel spreadsheet. I had used similar libraries in the past and knew these to be problems with those libraries. Essentially, you need to guard against bad data as these libraries don't.

When LAMP (Linux/Apache/MySQL/Php|Perl|Python) programmers access the database, they never assume the data coming in is safe. They check all the data before using it to access the database. This is because hackers will do things like input the following for First Name: "Hacker; DROP TABLE ..." or some other destructive text.

The lesson learned from this is to be a defensive programmer. LAMP programmers know this. I believe this to be true for people using third party libraries but also when using the code from someone else on your own team. If you assume the library does not handle edge cases and ensure the data has been checked before getting passed to the other programmer's code.

If you assume all code (third party libraries, your old code, code from other programmers on your team, etc.) does not handle code properly and check the inputs before passing them along to that code, your applications will be much more solid.

To the testers reading this, assume all programmers will assume someone else will take care of guarding against bad inputs. So think of all the bad inputs you can and send them in. You'll usually find they make it quite far before someone handles them. Often it will be the operating system that handles this with a crypt error dialog and a crash.

Asking for automation help

I've noticed a trend on help forums and groups. When people ask for help with their automation that often post what they tried and that it did not work.

They don't give any information about their environment, versions of the various software involved or the code they are trying to automate.

Asking for help with your automation is similar to filing a defect report.

• Specify a summary of the problem
• Give a description of the environment (OS, browser, etc.)
• Tell them what you did, what you expected to happen and what actually happened.

There is one additional piece of information you normally put in a defect report. You normally specify which build the defect appeared in. This is because the developer fixing the bug can find the appropriate code given a build number.

When asking people outside your organization for help with an automation problem, we don't have access to your source code. So a build number does not help us. What we need instead is a code snippet. If you are testing a web site, give us the relavent HTML code you are trying to automate.

Without these bits of information it would be impossible to solve the problem. At best, you will get a lot of guessing. If you are lucky, someone will make a lucky guess or say something which makes you realize what the problem really is. However, if you want a quicker response to your questions give us the information we need to reproduce the problem and solve it.

Test Web APIs

Testing web APIs essentially breaks down into:

• Building an HTTP Request
• Sending the HTTP Request
• Get the HTTP Response
• Verify the HTTP Response

If you look at RFC 2616 it will tell you everything you need to know about HTTP/1.1. However, if you are not used to reading RFC, this document can be quite confusing.

Here some highlights to an HTTP Request...

If you go to a website and they have a FORM, submitting the FORM will do an HTTP Request. The FORM tag will have an action attribute. In side the FORM will be INPUT, SELECT, BUTTON, etc. tags.

For example, the Google home page is a FORM. The action for the FORM is "/search". The INPUT has a name attribute of "q". If I enter "HTTP Request" into the INPUT field, my web browser builds an HTTP Request and sends it to Google. You can type the following URL into an address bar:

http://www.google.com/search?q=HTTP%20Request&hl=en

This is the same as entering 'HTTP Request' into the input field and clicking the Search button. The one oddity is the %20. A proper HTTP Request has no spaces in it. So you need to convert spaces to a hexidecimal value. The hexidecimal value 20 is ASCII for a space. Also symbols like / : & etc. have to be convert as well. If the symbol is part of the Request and not a value being passed you don't convert it. The general format is:

http://hostname:port/action?key=value&key=value&key=value

If you leave out the port it defaults to 80 (443 for https).

At this point you might be wondering, what does this have to do with API testing? A lot. Many APIs are implemented using HTTP Request/Response. So you need to send them an HTTP Request just like your web browser creates an HTTP Request. With the web browser, it adds a lot more you might not be aware of. The HTTP Request has the URL, a header and sometimes data. Your web browser will secretly add header information like:

Host: www.google.com
Accept-Language: en-us,en;q=0.5

Finally, the FORM can be a POST or a GET form. If it sends data using the URL, like the above examples, it is a GET request. If it sends the data in such a way the user doesn't see it, it is a POST request. For example, if I was logging into a website, I would use a POST so the password isn't visible in the address bar.

So now we have talked about all three parts. The URL, the HEADER and the DATA.

Let's say I design an API such that I expect some data to be in the URL, some in the header and some in the data. So lets say the URL will be:

https://www.mywebsite.com:8443/accounts/admin/requestAccount?company=Acme&auth=Darrell

In the header I want them to identify which machine the request is coming from and an API Key:

Request-Host: 127.0.0.1
Company-API-Key: foobar7

And finally, the data will be an XML file with the following information:

    <?xml version=\"1.0\" encoding=\"utf-8\"?>
    <RequestAccount>
    <CompanyName>Acme</CompanyName>
    <EmployeeID>10318630</EmployeeID>
    <FirstName>Bob</FirstName>
    <LastName>Johnson</LastName>
    </RequestAccount>

So how do I take this knowledge and using it? Let's say the XML data is saved in the text file foo.txt and I have curl installed on my machine (comes default on UNIX, Linux and Mac OS X machines; you can get and install a Windows version for free).

The curl command to use this would be:

curl -k -d @foo.txt -H "Request-Host: 127.0.0.1" -H "Company-API-Key: foobar7" -o response.txt https://www.mywebsite.com:8443/accounts/admin/requestAccount?company=Acme&auth=Darrell

The -k makes it so we don't worry about SSL certificates and authentication, the -d is used to send the data, the -H adds header information, the -o saves the response to the file.

The great thing about curl is you can try something, make a slight change, try again, make a slight change, try again. So if you wanted to try a bunch of different data files, you can do this quickly and easily with a shell for loop. You can play with the header information or leave/change some of the URL fields.

You can quickly probe and test a set of APIs using curl. There are more options to curl but this is the basic usage.

Upgrading Selenium 2.0b2 to 2.0b3 - Python

Selenium 2.0b3 was recently released. My current company is using Selenium 2.0 for web testing and the preferred language for the team is python. If you are familiar with python you might know that upgrading a site-package (like Selenium) is as simple as:

pip install selenium

One of the staff did this and immediately the test suite broke. The line of code which broke was:

host="localhost"
port=4444
browser="firefox"
wd = webdriver.Remote(command_executor='http://' + host + ':' + str(port) + '/wd/hub', browser_name=browser, platform='ANY')

One of the nice things about using Eclipse (my IDE of choice) is you can hold down the CTRL key and click a method to jump to the source code. So I held down the CTRL key (actually it was the Command key because I'm on a Mac) and clicked on Remote. This took me to the class for Remote. The __init__ method is the constructor.

When I looked at the constructor for 2.0b3 it was immediately obvious that the constructor had changed. The old constructor (2.0b2) was:

def __init__(self, command_executor='http://localhost:4444/wd/hub', browser_name='', platform='', version='', javascript_enabled=True)

but the new constructor (2.0b3) is:

def __init__(self, command_executor='http://127.0.0.1:4444/wd/hub', desired_capabilities=None, browser_profile=None)

So the first parameter, command_executor, is the same but all the other parameters have been changed to a desired_capabilities. So what is desired_capabilities? It is a python dictionary. The format for a dictionary is:

dc = { "browser_name" : browser, "platform" : 'ANY' }

This one little change almost fixed things. A little more digging into the source code and I found that the dictionary should be using "browserName" and there doesn't seem to be any support for platform anymore. Since we only care about the browser, I changed the dictionary to:

dc = { "browserName" : browser }

and this solved everything.

Lesson Learned: you have to dig into the source code to figure out what is going on.

P.S. reading the comments in the source code indicated that the start_session method, which uses the desired_capabilities, it still talks about the parameters browser_name and platform. It is clear that the comments have not been updated to reflect the code.

Knowledge sharing

Joining a new company is always difficult at first. You spend a lot of time learning what you don't know about the company and the culture. Here are some ideas to make this normally unproductive time more productive.

Most companies have a wiki or central web site like Sharepoint. Create an employee page with pictures of the employees, there name, contact information (office/location, email, phone, etc.).

Next create a Subject Matter Expert (SME) table. Put the areas of expertise and who is the main and secondary contact. You could even work this from the other direction. As a manager, think about the needs of your department, create a table with one row for each subject your department needs an expert in. In the second column put the name of the person who is the SME for that subject. In the third column put the name of who would be your second choice for SME. Any row which does not have a second name means you need to create a backup for that area. Otherwise, when your one and only SME goes on vacation you could be in trouble. Or worse, they leave the company. Any row which has no SME at all means you need to either get someone to become that SME or you have a case for hiring someone.

Whenever someone new joins the company, take their picture, add an entry to the employee page and a link to the SME table. If they are an SME for a subject you have no one, make them the SME. If they are the SME for a subject you have one SME, make them the backup.

The next step in the process is for the SME to document what they know. Some people will believe documenting what they know will allow the company to get rid of them. The truth is that an SME's practical experience can never really be captured in a document. Most often this knowledge comes into play because the SME has left the company, is away on vacation or ill. If they don't document their knowledge, they will be critical to the company. Going on vacation will be discouraged. They will most likely be put under a great deal of stress. Sooner or later they will fall ill or quit.

The ultimate goal is to create a backup for the SME. If the SME is promoted or put on other activities, the company will require him to be available to help out the backup/replacement. If someone is replacing the SME, the SME will most likely become the backup until a suitable backup can be found. By the time the replacement is fully up to speed and a suitable backup has been found, the SME will probably be an SME for something new or they can at least make sure they are becoming the SME for something new and critical to the company.

Bottom line, if they are trying to phase you out it would be obvious. So don't worry and help your company share that knowledge.

a new type of source control - Git

At the new company they started with Subversion for source control but are now moving everything to Git.

I've worked with a few different source control servers and clients. The basic idea was to have a central repository which holds all the source code. You check in the source code to the central repository. Subsequent changes to the code are saved in the repository as changes or deltas.

If two or more people are working on the same project they have to set up a policy or convention. In some systems the team has to follow the policy. With other source control systems you can set up rules to enforce a policy.

For example, I check out a copy of the code, you check out a copy of the code. We both make changes to the file Foo.java. You check in your code and the repository is updated. When I go to check in my change it tells me there is a conflict. I have to merge my changes with your change. Only after that is done can I check in my changes.

In this model source is handled file by file. If our project has 37 files and only one has a conflict, the other 36 files will check in without issue.

With Git things are different. There is no central repository. I create a Git repository on my machine. I commit my changes to the local repository. When I want to share the changes with others I can push my repository to a central location or I can push it directly to someone on the team.

I pull a copy of the repository, you pull a copy of the repository, we both change Foo.java, you push your change then I attempt to push my change. Git will tell me there have been changes and I cannot push my repository to the central location. I need to pull the repository down and merge it with my changes. I can also use fetch rather than pull. Using fetch will do a pull then merge in one step.

The basic commands are:

git init
git clone
git status
git commit
git push
git pull
git rm

To create a repository on your local machine, use git init. If the central repository is located at the URL git:git@git-server:project.git then I can make a local copy using:

git clone git:git@git-server:project.git

Once I have a local copy I can use git status to see what has changed. If I look at the output of git status it will list the files which have changed and the untracked files. Issuing a git commit -a will add the changes to my LOCAL repository. To share this with others I would do a git push. If I'm in the directory which contains git controlled files, it will look at the .git/config file, determine where the "origin" repository is and push the files to there. You can also git push to another location (someone on your team or a sub-team working outside the main branch.

If you want to remove files from the repository you can use git rm. If you just rm a file, the next git pull will bring the file back. If you git rm then git commit, the file will be permanently remove from your LOCAL repository. A git push will make this permanent for everyone else.

floating point numbers and representation error

Floating point numbers can be a great source of error. When we think about pure mathematics it includes the concept of infinity. For example, between the numbers 0 and 1 are infinite fractions.

When you store a floating point number in memory it has a limited number of values. This means that it is impossible to represent all possible floating point numbers. Different languages have different ways to represent numbers. The more accurate the floating point numbers are, the more expensive, computationally, using them becomes.

This means that operations like 1.47 - 1.00 might result in 0.46999999997 because the math library cannot represent 0.47 and 0.46999999997 is the closest match.

If the program you are testing is financial and it only deals with dollar and cents, this representation error can be problematic.

As a programmer there is a simple solution to this problem. Use only integers (long or double long if possible) and treat them as cents. So the above example would become:

long n1 = 147;           // $1.47
long n2 = n1 - 100;      // $1.47 - $1.00
long dollars = n2 / 100; // 0
long cents = n2 % 100;   // 47
System.out.printf("$%d.%02d", dollars, cents);

This means if you have a program which uses dollars and cents, you want to check for representation error (assuming the programmer used floating point variables) but you also want to check for issues which might be integer related. So you want to consider things like integer overflow and underflow. See the article Are most binary searches broken? for a discussion of overflow.