xpath

I have been doing a lot of web testing. The general idea behind all UI test automation tools is to locate an element on a page then do something with it, set it, clear it, read it, etc.

For the web automation tools you can use:

(a) the position on the screen (x,y coordinations)
(b) the position in the DOM (e.g. /html/body/table/tbody/tr[2]/td[4]
(c) a unique attribute

The position on the screen never works. Different browsers, fonts, screen resized, etc. will change the layout and ultimately, change the screen position of elements. I wouldn't use this. Working with development to provide alternative means will be less work than maintaining automation with x,y coordinates.

The position within the DOM is a little brittle. When a browser is patched or a new browser needs to be supported it is not uncommon for the developers to throw in some span or div elements to help with layout. So the element /html/body/table might change to /html/body/div/span/table. The more precise the positioning information the more brittle it will be.

A unique attribute would be something like the id of a tag. For example:

Darrell

I can find this via the class or id attribute, or both. This is where xpath comes in handy. The automation tools I have been using (Watij, Selenium) can use xpath to locate an element. For my td element I can use:

//TD[@class='username-cell']

or

//TD[@id='username']

or

//TD[@class='username-cell' and @id='username']

The id attribute is required to be unique. So if the element has an id, that is the attribute to use. If you start an xpath with // it tells the tool to start searching anywhere in the DOM. Starting with a single / will start at the root. For a web page that will always be /html.

Xpath can be quite powerful in identifying elements. You have a few 'tricks' you want to use. First is that id=' foo' is not the same as id='foo '. The whitespace makes a difference. To get around this I would use:

//TD[contains(@id,'foo')]

Now the whitespace does not matter. you have to be careful with this however. If there are two matches, it depends on the automation tool as to what will happen. If I have:

darrellDarrell

then:

//TD[contains(@id,'user')]

will have unpredictable results. Not something you want in test automation. So how to get around this?

//TD[contains(@id,'user') and not(contains(@id,'username'))]

Any attribute inside the tag can be used via the @ symbol. You can do things like look at the style attribute using @style. Because the order of the things in style does not matter to the browser, the contains() function helps a lot.

Finally, the text between the open and close tag can be found using the text() function. So if I had:

Google

I can find it using:

//A[contains(text(),'Google')]

What about the difference between Google and google? For matching either you can use:

//A[contains(lower-case(text()),'google')]

This will take the text in the anchor (e.g. Google) and changing it to lower case (e.g. google) then comparing that resulting string to 'google'.

In addition to and there is an or keyword as well but I usually find it better to narrow things down (and will filter out) rather than build up the matches (or will combine).

There is a lot more to know about xpath. If you are curious, ask.

Additionally, I find a developer will put an id on something because he uses it from javascript to find it and all the elements underneath it. So if there is:

Darrell

Jerome

Mark

I would tend to use:

//DIV[contains(@id,'users')]/UL/LI[contains(text(),'Darrell')]

to find the element which contains 'Darrell'. The developer will tend to not want to change the structure under the id'd tag because it will cause them a lot of maintenance as well.

Test Automation Manifesto

I have been recently reading xUnit Test Patterns by Gerard Meszaros. Excellent book with a lot of things I learned the hard way. I also found an article by Gerard et al titled, "The Test Automation Manifesto". The manifesto is:

Automated tests should be:

1. Concise: As simple as possible and no simpler.
2. Self Checking: Test reports its own results; needs no human interpretation.
3. Repeatable: Test can be run many times in a row without human intervention.
4. Robust: Test produces same result now and forever. Tests are not affected by changes in the external environment.
5. Sufficient: Tests verify all the requirements of the software being tested.
6. Necessary: Everything in each test contributes to the specification of desired behavior.
7. Clear: Every statement is easy to understand
8. Efficient: Tests run in a reasonable amount of time.
9. Specific: Each test failure points to a specific piece of broken functionality; unit test failures provide “defect triangulation”
10. Independent: Each test can be run by itself or in a suite with an arbitrary set of other tests in any order.
11. Maintainable: Tests should be easy to understand and modify and extend.
12. Traceable: To and from the code it tests and to and from the requirements.

In his article he talks about bad code 'smells'. A 'smell' is something which you notice again and again as a problem. Even before you have clearly identified it, you can 'sniff' them out.

The first two code smells he talks about are THE reason record and playback test automation doesn't work.

When you use record and playback to automate, it will (a) hard-code test data and (b) duplicate code. If I'm testing an application which requires me to log in before each test, the recorder will record me logging in for each test case. If I do not refactor the log in to a function that all the test cases use and the developer changes the login, I would have a maintenance nightmare. Additionally, if the username and password change and the test data is hard-coded, I'd have to find all the instances and change them.

By the way, the idea of 'refactoring' is to change the code to be more maintainable without changing the way it runs. For example, if I had the following code:

// test#1
// put "darrell" in username text field
// put "password" in password text field
// click Submit button
//
// test#2
// put "darrell" in username text field
// put "password" in password text field
// click Submit button
//
// test#3
// put "darrell" in username text field
// put "password" in password text field
// click Submit button
//
// test#4
// put "darrell" in username text field
// put "password" in password text field
// click Submit button
//

and it worked as expected, I could refactor it into:

// login(username, password)
// put $username in username text field
// put $password in password text field
// click Submit button

// test#1
// login("darrell", "password")
//
// test#2
// login("darrell", "password")
//
// test#3
// login("darrell", "password")
//
// test#4
// login("darrell", "password")
//

This will run pretty much the same but now if the Submit button gets changed to a Login button, I don't have to change all 4 test cases (what if I have 40,000 test cases). I just change:

// login(username, password)
// put username in username text field
// put password in password text field
// click Login button

and it updates all the test cases. This code still has hard-coded data. How about changing this to put the data in a property file:

# login data
username=darrell
password=password

then change my code to:

// read property file
// username = getProperty("username")
// password = getProperty("password")

// test#1
// login(username, password)
//
// test#2
// login(username, password)
//
// test#3
// login(username, password)
//
// test#4
// login(username, password)
//

Now I can edit the property file if I want to change the username or password.

I picked property file to hold the test data but it could just as easily been a database, spreadsheet, text file, compiled resource, global variables, etc.

the math behind test automation

It has been a while since I had time to write to my blog. A lot has happened. In the world of test automation I created a kick ass test suite using Watij. Built out a wonderful set of libraries and had a tonne of reusable code. I could whip out a new test case in minutes. Complex test cases might take an hour or two.

New features and code changes required a minimal amount of work. This is usually key to the survival of a test automation framework. I could have used record and playback to create automation but small changes in the application would render the automation useless. Most test tools recommend re-recording the application. If recording the test case and adding in the appropriate verifications takes longer than manually testing the application then a record and playback automation tool is pointless.

You need to create reusable code. Software automation is development. I have always believed this and recently read an article on infoQ, by Dale H. Emery which reiterated this belief.

The only way a test automation suite pays off is by being maintainable. Here is the math:

-let x be the time it takes to run the test suite manually once
-let y be the cost for each unit of x
-let z be the number of iteration we need to run the full test suite
-therefore the cost of manual testing is xyz

-let a be the time it takes to create the automation test suite
-let b be the cost for each unit of a
-assume the time it takes to run the test suite is infinitesimally small
-therefore the cost of automated testing is ab (running the tests more than once does not incur any significant costs)

-so long as ab < xyz, the return on investment is worth it Companies selling test automation tools will often sell them with the idea that you can record and playback the automation almost as easily as manual testing. Even if a = 2x and b = 2y then ab < xyz == 2x2y < xyz == 2(xy) < xyz == 2 < z Or in english, if you run the automation more than twice, it pays for itself. Sounds pretty good, doesn't it? This is an incredible simplistic view. It does not take into consideration the time dealing with bugs in the automation tool, lack of support for new technologies or configurations, training the staff, learning how to avoid false positives, learning how to eliminate false negatives (these greatly increase the cost of automation especially if not handled well). After you take all that into consideration, the reality becomes that you need to run the test automation 6 to 10 times with no modification to break even. And everyone knows that the development team will not change the application in a 6 to 10 week period, right? Bottom line, even on the first release the test automation does not pay for itself. So you spent $10,000 on a test automation tool and you hired a consultant to use it and create your test suite for another $50,000 only to find out that it never really worked or therefore were additional costs. If you decide to drop the tool (which has an annual service contract), the sales team jumps into high gear and convinces you that maybe after the next release it will pay for itself. By the third release you'd start seeing a profit. You have invested thousands and want to believe you can make this money pit viable. You desperately believe the sales staff. The record and playback is a losing battle. The code is just not maintainable. There is no reuse of code. It is equivalent to cut&paste. Any intermediate developer will tell you that cut&paste is a bad thing. If you have a section of code that clicks the Login button and they change the way the button is identify, you find yourself changing a thousand lines of code. If the automation used a library call to click the Login button then one change is all you need for all test cases which click the Login button. And that is the trick. As an automation developer, I record a test case. I look at the code produced and refactor it so common operations are placed in a library. The next time I do a record, if I see the same chunk of code, I replace it with the library call. The problem now is remembering everything I put into a library and refactoring recording to use the library call. For this, structure your library. I like to use an object oriented language for automation. I can then use one class for each feature or 'screen' or 'window'. I basically break the application into 'screens' or states and model my test framework around that. Now my library has an easy to remember structure just like a language's set of libraries (e.g. Java Software Development Kit or C++ Standard Template Library). So now I'm developing a library similar to the C++ STL or Java SDK. If I'm a failed developer forced to do test automation, I'm probably not a good tester and I definitely don't have the skill set to create the STL. What else can I do to make the math work out for test automation? Look at maintenance and reporting. If a test framework cannot tell you quickly and easily how the state of testing is then it is not very good. You need to have good reporting capabilities. If the framework has this built in all the better. If it just happens as part of the development of a test case great. For my last test framework I added in log4j messages to all the library calls. If set to NONE it would output nothing. If set to INFO it would print out the test steps as them occurred with the data being used (helps to reproduce the test manually, if necessary). If set to DEBUG, tells you details and interworkings of the library calls. This is more for maintaining the code than reporting what is happening. As for code maintenance, you want something that works in a good debugger. Running to a point then examining the state of the application under test (AUT) is important for fixing broken automation. Being able to quickly and easily find, manipulate and verify elements of the AUT. Languages like Watij are Java based and take full advantage of the Java programming language. If I have an xpath that matches a dozen text fields I can ask Watij to give me a List then I can use the Iterator to look at each text field.

On the other hand, languages like Selenium are very popular but its native language is HTML tables. Each row (TR) in the table is a test step. The first column is the action, the second column is the locator and the third (optional) column is any extra parameters needed.

You can get Selenium-RC and a Java API but the Java support uses String[] rather than Lists and Collections. I can find out how many text fields match a given xpath but I have no way of locating them. I often find myself adding the support I've grown used to in Watij. This means more code and more maintenance.

The unfortunately thing is nothing is perfect. Watij is easier to use as a Java developer but it only works with Internet Explorer (I have found that IE locks up occasionally; the status bar indicates it is waiting for 1 more item to load but everything is loaded). IE is not a very stable platform and with Watij you have no choice but to deal with it. Selenium does not have the Java structure and support of Watij but I ran it continuously on IE and it never locked up (same AUT). Additionally, I can run Selenium on my Mac, a Linux box or Windows. It works with Safari, Firefox and IE.

Watij is easier to maintain but if the tests hang and never complete, Selenium seems to be the solution. Going forward I will try to write a set of miscellaneous libraries which mirror the Watij structure. Hopefully once that is done, Selenium will be just as powerful as Watij and far more stable.

A monthly thing

Seems life is turning my site into a monthly event. This month I'm going to talk about boundary test cases and a little about resumes.

When you are testing an application you want to be systematic. Even the simplest of applications can have so many possible combinations that testing all of them does not make sense. What you need to figure out is what subset is sufficient.

For example, a web site I was testing had security and a timeout. You had to log in and the system would automatically log you out after a certain period of inactivity.

Recently, they made the timeout period configurable. You would edit a config file and set a property. If the application.timeout property existed and had a value of 30 it would automatically log users out after 30 minutes of inactivity.

So, how do you test this. I could try all possible values but the application server takes 10 minutes to power up and 5 minutes to power down. Each value I'd use then means 15 minutes plus the timeout period. Even if it only supported up to 99 minutes it would take me one solid week to test it. If I took this long for every feature it would take me decades to test the application. In other words, before I finished testing it, the computer would be obsolete.

What I need to do is pick a subset. If 1 worked and 2 worked, I'd guess that 3, 4, 5, 6, etc. will work. If I understand the language the application was written in, I might be able to figure out special cases. It was written in Java. Is the timeout value stored in an int? long? Integer? If the data type is 8 bits then I know for Java there are no unsigned char so the data range is -127 to 128. What happens if I use 129? What happens if I use a negative number? The value 0 will be fine for the char but what will the timeout code do with a 0? Does 0 mean 'disable timeout'?

Turns out the programmer used a 32 bit number. This means it ranges from 2147483648 to -2147483647. So I could try a value of 2147483649 (MAX+1). If they are using the input as a String then using Integer class to convert it, the Integer class will throw an Exception. What about if I set the value to "twenty". Did they think to handle that. For the application I'm testing, the user is a Application Server Administrator for LARGE enterprise environments. I didn't test for "twenty" because our users wouldn't try that.

At this point you might be wonder, what about the resume thing? I see a lot of testers putting programming languages on their resumes. I test development tools. All my staff are programmers and I expect anyone I hire to be a programmer. So for me putting a language on your resume means you know how to program in that language. If you don't I have to put you on a different team (e.g. testing the user interface for application monitoring solutions). If I was hiring for someone testing a non-development application, the language on the resume might mean you understand the limitations of the language and can apply that to your testing.

For example, if you note you test Java and C applications, it means you understand that C has unsigned data types and Java does not. In Java the boundary cases are going to be 2^7, 2^15 and 2^31 but in C language it will be all the Java boundary cases PLUS 2^8, 2^16 and 2^32 (there are actually more but this shows the difference between Java and C applications). All the people I've interviewed had no idea why the language the application was written with made a difference. When I asked them why they put it on their resume they had no idea. If you don't know why something is worth putting on your resume ask. If no one can tell you, don't put it on your resume until someone can explain why.

Been a while

It has been a while since I posted to my blog. I've been reading less techie books and taking time to myself.

Been using Watij at work to do load testing. I am testing Foglight 5.0. Foglight is a application monitoring tool. You install it on a computer then deploy agents to other computers. The agents collect information and send it back to Foglight. Foglight saves the data in a database. A user can then log into the web console for Foglight and view the data. A Foglight cartridge is a package of agents, configuration files and schema information. You would have an Oracle Database cartridge. This would have an agent which monitors an Oracle database, sends back all the information an Oracle Database Administrator would be interested in then displays it in a manner the DBA would appreciate. The 'cartridge' has default dashboards (a dashboard is a chart/table/view of the agent data) and rules (a rule does things like email the DBA when the database crashes, if Foglight detects a bottleneck, someone tries to illegally access the database, etc.). There are other things like reports (PDF) and analysis tools.

So, if you load all the cartridges into Foglight (OracleDB, WebLogic, WebSphere, Windows, Solaris, AIX, HP-UX, Vmware, MySQL, DB2, etc.) you will have hundreds of different views. For example, just the Windows cartridge will have agents for DiskIO, FileSystem, CPU, Memory, EventLog, AppMonitor, WebMonitor, ApacheSvr, LogMonitor, etc. and each agent will have dozens of views.

Verifying all these views can be quite time consuming. Each dashboard has an associated URL. As a user of Foglight I would log into the console (username/password) then select a dashboard from a treeview. I could also type the URL into the address bar and go to the dashboard directly.

This is how I use Watij. I created a set of jUnit test cases. The setup() was starting IE and logging into the Foglight Console. Each test case [test*()] was loading a URL, i.e. dashboard. The tearDown() was logging out of the console and quitting IE.

One of the challenges I faced with Foglight's Web Console Framework (WCF) was the use of AJAX and client-side Javascript. A fair amount of the code was in the form of Javascript on the client side. This meant, the HTTP response would complete, Watij would see the HTTP request as done but the client (Internet Explorer) would still be processing the Javascript (many of the views were complex enough that a page would take an addition 1 to 5 seconds to render).

The solution: WCF has a GIF which they set the style="VISIBILITY: visible" when the page is rendering and it gets changed to style="VISIBILITY: hidden" when the rendering completes. So I just wrote a method which gets the CSS for the image as a string then uses the match method of Java String to search for "style=\".*VISIBILITY:[\s]*visible.*" and does a loop until this changes. Basically it is a:

do {
// sleep 250 milliseconds
// get the CSS in the string s
} while(s.matches(REGEX));

The moment the style changes from visible to anything else, the loop exits and I know the page is really done. As a double check I do a windowCapture from Watij then manually inspect the images.

Darrell

nmap and amap

Back in the day I used to frequent alt.2600. This was a usenet newsgroup. They are not as popular now a days. If you wanted to learn something about say C programming, you'd go to comp.lang.c and read the messages. It was a lot like a bulletin board. The alt.* newsgroups were easier to form and didn't require you to get a bunch of people to vote on whether or not you could form the group. You would constantly see things like alt.bork.bork.bork.swedish.chef or alt.wesley.crusher.must.die.

One alt group that was formed and lasted for quite some time was alt.2600. The frequency 2600 Hz was a tone AT&T used to indicate a line was not in use. A hacker found that a whistle you got with Captain Crunch cereal could transmit the 2600 Hz signal. He would call a long distance number, blow the whistle and AT&T would assume the line was not in use, thus he got long distance calls for free.

alt.2600 was devoted to hackers, like Captain Crunch (the nickname of the guy who discovered the 2600 Hz trick), sharing information about hacking. Some were ethical and some were not.

Today there is a 2600 magazine available. I was reading this magazine today and it had a nice article on tools like nmap and amap.

I was familiar with nmap. The nmap software is a Network MAPper. You can use it to probe a network. You pick a machine and nmap will probe the machine to see what ports are active. You can either listen, passively, for transmissions or you can actively send data to various ports and see what responses you get back. If you are passive, the machine operator would not know you are out there probing his/her network but you don't get a lot of information back this way. If you actively probe the network you get a lot more data back but the system operator will be able to detect you are probing their network.

The art of hacking seems to be a dying art. Many of the system operators out there today don't think to look for people probing their network. Most will try to keep some sort of logging. If you do something malicious they will check the logs to see who did it, i.e. they are more reactive then proactive.

So you can often probe networks so long as you don't do anything to make the system operator respond.

The amap program, I just read about, is an Application MAPper. Programs like nmap or netscan will probe a system but they can easily be tricked. For example, web servers are typically at port 80. If I have an application server running at port 80, mapping software might mistakenly assume it is a web server. The claim of the 2600 article is that amap is a little smarter than that.

I'll still have to download the source code and compile a version for myself.

You can find information about nmap at http://en.wikipedia.org/wiki/Nmap. For amap, you'll need to go to http://freeworld.thc.org/thc-amap/ and compile the program yourself. NOTE: a good way to hack someone is to give a novice a tool that lets them play without the source code. The novice downloads and runs the 'tool' only to find out they have installed a trojan horse on their system. You always want the source code *AND* you want to look at the source to see what it is doing. If there is any cryptic code like:

main(){int j=10024;char t[]=":@abcdefghijklmnopqrstuvwxyz.\n",*i=
"@zp:pf:qbogw\nxbz\nexke.z";char *strchr(const char *,int);while(
*i){ j+=strchr(t,*i++)-t;j%=sizeof t-1;putchar(t[j]);} return 0;}

Don't use it. If you don't understand what a piece of code does there is a good chance it is a trojan horse.

Safely probing web sites

The whole recruitmenttech.com / Bernard Haldane scam thing got me using my old telnet trick to examine the contents of a website. A number of bad eggs like to use security flaws in IE or Firefox to infect your computer. Most people try to make sure the security patches are up to date. But, there is always a period of time between when a virus is released and when a security patch is released to deal with it. If you visit the wrong website during that time you could be in for trouble.

What I like to do is avoid the security flaw by using a method the virus writers are not expecting. I like to use telnet. I tend to telnet from different operating systems as well. You might not have the luxury of a dozen different OS. You could consider Vmware or some other OS emulator.

Anyways, here is how it works. I'll use telnet from MSDOS. The connection for a web browser and for telnet is pretty much the same. Telnet defaults to port 23 and web browsers default to port 80. So if I wanted to use telnet to connect to say www.blogger.com I'd use:

telnet www.blogger.com 80

At this point the MSDOS telnet program will print nothing. If you press CTRL-] you get to the telnet settings. In there enter set localecho. Press ENTER twice, once to turn on local echo and once to get out of the telnet settings. You are now back at a blank screen again. Enter:

GET / HTTP/1.1
Host: www.bogus-computer.com

NOTE: you have to press enter twice at the end. Once to send the Host: field and once to signal the end of the HTTP header.

If you take too long to type things in, the computer at the other end will timeout and hang up on you. If you type it in quickly enough you'll get something back like:

HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/html; charset=ISO-8859-1
Set-Cookie: PREF=ID=0df2c62f96b9ffe7:TM=1203534740:LM=1203534740:S=0L2HwAZgQbyqrbmI; expires=Fri, 19-Feb-2010 19:12:20 GMT; path=/; domain=.google.com
Server: gws
Transfer-Encoding: chunked
Date: Wed, 20 Feb 2008 19:12:20 GMT

...

There should be a lot more as well. What I'm not posting here is all the stuff between <html> and </html>. What I have posted is the HTTP response header. Your web browser eats this and uses the information. For example, the Set-Cookie: field will make the web browser set a cookie.

The GET command is a standard HTML header command. The next part, /, is the path you want to get and the HTTP/1.1 is the protocol to use. Some web servers only work with HTTP/1.0, some only work with HTTP/1.1, most will use both.

If you wanted the web site, http://en.wikipedia.org/wiki/List_of_HTTP_headers then the sequence would be:

telnet en.wikipedia.org 80
GET /wiki/List_of_HTTP_headers HTTP/1.1
Host: www.bogus-computer.com

You'll notice that I always put a Host: field. Many web servers will not respond to robots or automation. They want to know who is talking to them. So if you don't include the computer name in the header, they just hang up or send you a HTTP/1.1 403 Forbidden response. Try using telnet to www.google.com and they will refuse you. They not only want the Host: information but they expect a number of other fields as well. If you go to the http://en.wikipedia.org/wiki/List_of_HTTP_headers web page, they talk about some of the common header fields and have a link to the HTTP standard on www.w3.org.

When you get the response back, you'll have to look through the body and see if there are other references, e.g. <SCRIPT> tags, which will create more GET requests. Your web browser is often getting the first page and from there doing multiple GET commands for the contents (each <IMG> tag is a GET, running Javascript will create more GET commands, etc.).

Once you have downloaded everything you can then look at it with a text editor and see if there is anything in it which could harm your computer. If you don't know how to read Javascript, this is obviously not an option for you but as a big nerd this is what I do. :^)

Hope you enjoy this. Let me know if you have any questions.

Happy hunting!