Mobile web testing

I've been working on a project recently which required testing on a mobile device. The project started in April this year and was focused solely on iOS.

When I looked into what was available for mobile testing I found a number of different tools:

• KIF
• Appium
• Frank
• Calabash
• EarlGray
• UI Automation

What I found next was a bother. I need a tool which would be used for testing the IPA we would ship from an app store. Tools like Frank and Calabash are great at automating tests but they required you to build a special version of the app. This would not be the same app you deployed to an app store.

This made it easy to eliminate those two great tools from my list of potential test automation tools.

I then looked into KIF and EarlGray. They had great reviews and looked really promising until I noticed Apple made significant changes to the UI Automation framework and broke KIF and EarlGray. So if I wanted to test against iOS 9.3 developed with XCode 7 and Swift I was probably not going to want to use KIF or EarlGray.

So the obvious choice was Appium. However, even Appium seem to be affected by the changes Apple announced at the July 2015 Developer Conference. :(

Since our iterations were one week and waiting to see who would 'fix' issues with their framework wasn't really an option, we branched Frank and started developing the app using Frank for UI testing. In the meantime we looked at UI Automation (our app was iOS only, so we didn't need to worry about Android support).

Initial use of UI Automation seemed good. So we started automating UI tests with it but continued to keep the Frank tests running in parallel. However after a few iterations we started to see maintaining the UI Automation tests was becoming increasingly difficult. Since I got into UI automation in 1998 I have found that failure to maintain a test automation framework is one of the common reasons for UI automation to fail. We didn't have a team of 20 QA Automation experts to keep the UI Automation framework going. :(

So I had a second look at the previously discarded frameworks. To my surprise and delight I found that support for them had been re-established and I took a second look at using Appium. 

Appium is definitely not fast and I'm looking for ways I can reduce the execution time of the Appium test suite (currently 15 minutes when run on hardware; I'd like to get it down to 5 minutes plus add more tests; maybe run tests in parallel on four or more phones).

Bottom line, Appium seems to be working well for us. I've created a page object model framework. In my next article I'll talk about using the Appium Inspector on a Mac laptop and things I found blocked me or slowed me down.