E-guide

Testing Mobile Apps

Criteria for quality assurance, tracking changes, and a review of 5
great tools for developers

E-guide

In this e-guide

In this e-guide:

App Quality Alliance compiles

mobile testing criteria

p.2

Five mobile testing tools for

developers to consider

p.6

How to track configuration

changes in mobile app testing
p.12

Page 1 of 14

Developers and service providers with a myriad of innovative

ideas are excited by the capabilities made possible by mobile
applications. Testing is important before and after product
release, as information is in a constant state of flux, and
customers expect the best from mobile apps.
Ahead, App Quality Alliance (AQuA) shares testing guidelines
that will give developers confidence that they are testing
mobile apps thoroughly. Next, we review five mobile testing
tools (Appium, Selendroid, ios-driver, Sauce Labs and Perfecto
Mobile), to kick-start your research in a growing software
market. Finally, software quality and testing veteran Amy
Reichert reviews tips on how to track configuration changes in
mobile app testing.

E-guide

In this e-guide
App Quality Alliance compiles
mobile testing criteria

p.2

Five mobile testing tools for

developers to consider

p.6

How to track configuration

changes in mobile app testing
p.12

App Quality Alliance compiles mobile

testing criteria
Gerie Owen, Test Architect/Business Solutions Analyst at Eversource

For development teams just entering the wonderful world of mobile

application development, mobile application testing can be full of unknowns.
Developers and quality engineers may find themselves wondering what
they've forgotten to test against, or worse -- missing bugs because they
didn't know to perform certain tests. The App Quality Alliance is pulling
together data from its members to create testing guidelines that will give
developers confidence that they are testing mobile apps thoroughly.
The App Quality Alliance (AQuA) is a non-profit group working with the
industry to improve the quality of mobile apps. Its members include AT&T,
LGE, Microsoft, Motorola, Oracle, Samsung and Sony Mobile.
SearchSoftwareQuality caught up with Martin Wrigley, executive director of
AQuA, for an interview at the Mobile App Europe conference in Potsdam,
Germany.
SearchSoftwareQuality: In your presentation, you spoke of 10 poor
practices still in use by many app developers. Can you describe a couple
of the most insidious?

Page 2 of 14

E-guide

In this e-guide
App Quality Alliance compiles
mobile testing criteria

p.2

Martin Wrigley: Apps need to work well in a real environment, in customers'

hands. But many developers use ideal environments for the functional
testing of their [apps]. This may mean that they have the fastest, newest
phone, or that they only use Wi-Fi connectivity, or that they forget that real
users fill up their phones' device memory with other apps, photos, music and
messages.

Five mobile testing tools for

developers to consider

p.6

How to track configuration

changes in mobile app testing
p.12

Developers don't always remember that a device can lose connectivity -and their app doesn't necessarily behave in a way that makes sense to the
user when that happens. It isn't good for an app to freeze when the device is
set on airplane mode -- but that still happens all too frequently.
There is no substitute for having a separate team (apart from the
development team) test an app on real devices in a real situation before
giving the app to users.
Tell us about AQuA's app testing criteria. How were
these developed?
Wrigley: AQuA has criteria for Java, Android and Apple
iOS apps. We are working on Microsoft criteria as well.
The criteria are designed to catch the majority of
common errors in apps. These criteria have been
developed over time, mostly in response to errors that our
members have found in apps. As we find new trends, such

Page 3 of 14

The testing criteria

are reviewed on an
ongoing basis, and
new versions are
released usually two
or three times a year,
as necessary.

E-guide

as one recently where apps were neglecting to release the camera on

devices, we add to or amend the criteria to catch these trends.

In this e-guide
App Quality Alliance compiles
mobile testing criteria

p.2

Five mobile testing tools for

developers to consider

p.6

How to track configuration

changes in mobile app testing
p.12

Our member representatives are involved in apps on a day-to-day basis,

either testing apps on devices, looking at the impact on the network, or
delivering apps to paying customers and having a reputation for quality.
The testing criteria are reviewed on an ongoing basis, and new versions are
released usually two or three times a year, as necessary. We receive a good
amount of feedback from the thousands of developers who have signed up
to receive information and updates from us, and we ask for new versions of
the criteria to be reviewed by them before we make updates public, in much
the same way that you should have someone test your app externally.
AQuA members debate each new addition to the testing criteria. We are
keen to ensure that they remain "right-sized" and relevant for apps that are
downloaded today. We consider the criteria to be a complete but minimum
set of tests needed to ensure a good customer experience.
How will following these criteria improve app quality?
Wrigley: If a developer follows the test criteria, he will avoid the top 10
errors and more. The testing criteria give developers a QA system in a box.
Since the launch of the Apple App Store, we've seen the introduction of
tools that could be used by designers as well as developers. We are seeing

Page 4 of 14

E-guide

In this e-guide
App Quality Alliance compiles
mobile testing criteria

p.2

Five mobile testing tools for

developers to consider

p.6

How to track configuration

changes in mobile app testing
p.12

apps that are much more attractive to look at, but the teams behind them
may not necessarily have the software engineering skills that the traditional
development teams may have had. As a consequence, they may not have
the testing expertise to look beyond the immediate functional testing that
people tend to do.
The AQuA testing criteria gives developers a cost-effective way of
performing a structured set of tests -- a checklist -- to avoid easily
overlooked errors that otherwise keep causing issues in the apps.
What testing advice do you have for teams building their first mobile
apps?
Wrigley: In the planning of your app, think about the types of tests you need
to do and don't skimp on them. It's always tempting to put in another feature
and miss out on some of the testing, but if that ends up with your app
crashing, that extra feature will not be much compensation.

Next article

Page 5 of 14

E-guide

In this e-guide
App Quality Alliance compiles
mobile testing criteria

p.2

Five mobile testing tools for developers to

consider
Matthew Heusser, Consultant, Writer and Change Agent focusing on Quality Software

Five mobile testing tools for

developers to consider

p.6

How to track configuration

changes in mobile app testing
p.12

Delivery and Justin Rohrman, Consulting Software Tester and Writer

Mobile testing tools are experiencing a growth spurt. New products and
services emerge nearly every day, and the time and energy needed to
evaluate them may make all the free trials in the world seem not so free. This
article looks at several mobile testing tools, listing their benefits, features
and tradeoffs to help testers and IT managers make smarter investment
decisions.
The first major challenge with mobile testing is the host -- that is, the device
the tests will run on. Unlike a laptop or desktop, where the tests are
developed and run on the same device, automated tests are typically
planned and programmed on a laptop and run on some mobile device that
may be plugged into the laptop and emulated. Or the mobile device might
run somewhere else and be connected to the test driver over the Internet.
Most large testing programs require testing on a handful, if not dozens, of
machines, making physical plug-and-play untenable. These challenges
should receive special attention when reviewing tools.

Page 6 of 14

E-guide

In this e-guide
App Quality Alliance compiles
mobile testing criteria

p.2

Five mobile testing tools for

developers to consider

p.6

How to track configuration

changes in mobile app testing
p.12

Appium
Appium is a small server written in node.js. Appium allows users to access
their Android or iOS environment where they have the server running.
Since Appium uses the WebDriver framework, it can run tests in any
language that WebDriver supports. Webdriver is a popular Web API that
developers can program in many languages, including Ruby, Python,
JavaScript and many more.

Page 7 of 14

E-guide

In this e-guide
App Quality Alliance compiles
mobile testing criteria

p.2

Five mobile testing tools for

developers to consider

p.6

How to track configuration

changes in mobile app testing
p.12
Appium interacting with iOs

On the plus side, Appium is free and based on open source components. On
the downside, that means setup can be a hassle; you have to install open
source prerequisites and make sure the versions are compatible.

Selendroid and ios-driver

Two other options fairly similar to Appium are Selendroid (which recently
replaced AndroidDriver) and ios-driver. They are both device-specific
implementations of Selenium WebDriver for mobile devices. Selendroid is a

Page 8 of 14

E-guide

framework for testing software on Android devices; ios-driver applies

specifically to iOS devices.

In this e-guide
App Quality Alliance compiles
mobile testing criteria

p.2

Five mobile testing tools for

developers to consider

p.6

How to track configuration

changes in mobile app testing

With Selendroid, testers can run tests on an emulator or on an actual device.

As of version 0.6.5, ios-driver does not support running tests on real devices.
But that feature may be added in the next release.
Testers using Selenium products can benefit from the massive community
of people that are also Selenium customers. Several websites offer free
community-based support and testers can rest assured that most problems
they experience will have been covered by someone else.

p.12

Sauce Labs
Sauce Labs offers a mobile testing product based on the Appium
framework. Strictly speaking, Sauce Labs uses the Appium tool to host the
tests, providing the environment for running digital tests. This approach
provides the same benefits of freedom of programming language and dual
(iOS, Android) platform support while removing the need for physical
hardware.
In other words, Sauce Labs uses the Appium framework, but gives testers
the device to run the tests on a virtual mobile device, then bills the device as
a metered service by the CPU hour. This service is built in the same way as

Page 9 of 14

E-guide

In this e-guide
App Quality Alliance compiles
mobile testing criteria

p.2

Five mobile testing tools for

developers to consider

p.6

How to track configuration

other Sauce Lab products, so testers get similar features, such as the
potential to watch videos of tests as they run, or to link those errors
reported in a test to a particular point in the video, making debugging much
easier.
However, unlike the previously mentioned tools, mobile testing supported by
the Sauce Labs platform is not free. The manual subscription plan is $12.00
a month, the individual plan is $49.00 a month and the small team plan is
$149.00 a month. Enterprises should contact Sauce Labs for further pricing
information.

changes in mobile app testing

p.12

Perfecto Mobile
Perfecto Mobile is not based on WebDriver. It has a native UI based on a
proprietary TCL-based programming language. This approach lets testers
who are not programmers select behaviors from a simple, no-coding
development environment and write expected results in simple English.
Tests designed with these proprietary tools may be more costly to move
away from than tests written with open-source tools.
In addition to its native UI, Perfecto Mobile provides a RESTful API for
integration with other interfaces including WebDriver and JUnit. Like Sauce
Labs, Perfecto Mobile can provide either simulators or cloud-based real
devices to run tests on. Perfecto Mobile recommends running tests on real

Page 10 of 14

E-guide

devices. Perfecto Mobile provides live scheduled demos and prerecorded

videos, but less extensive written documentation than the other products
listed here.

In this e-guide
App Quality Alliance compiles
mobile testing criteria

p.2

The choice is yours

Five mobile testing tools for
developers to consider

p.6

How to track configuration

changes in mobile app testing
p.12

Driving the user interface with mobile testing tools is massively popular in
the software world, and automation of mobile websites is quickly becoming
common practice. This growth has caused an explosion in companies
creating mobile testing tools. Now that testers have so many tools to choose
from, the choice is more complicated, often made worse by folklore
surrounding test automation and marketing material. Hopefully, this article
provides a starting point for testers' research.

Next article

Page 11 of 14

E-guide

In this e-guide
App Quality Alliance compiles
mobile testing criteria

p.2

How to track configuration changes in

mobile app testing
Amy Reichert, Software quality and testing veteran

Five mobile testing tools for

developers to consider

p.6

How to track configuration

changes in mobile app testing
p.12

Configuration changes are updated constantly in mobile application

development as development companies try to keep up with demand. Mobile
app testing, on the other hand, is already complicated by the number of
paths needed to ensure a quality product. The process is burdened by
testing requirements around screen size, platforms, connectivity and
configuration.
With an already full plate, how does a mobile app testing team add in testing
configuration changes in a release? Typically, configuration settings are not
the highest priority changes, so testing them falls behind other feature
testing commitments. You can try a couple of options to see what works for
the mobile app testing team's schedule.
The first is random selection. For example, if there are 20 configuration
changes pick the top 10 that are most important to clients, or that occur in
the most-used features. Or list each by priority or importance and by the
popularity of the associated feature and test your way down the list.

Page 12 of 14

E-guide

In this e-guide
App Quality Alliance compiles
mobile testing criteria

p.2

Testers also can divide the list between releases. Granted, not all features
will be tested in the initial release, but a group can be tested each release
until testing is completed. Create rotating test suites and mix them into your
regression testing executions. Once each has been thoroughly tested, go
back through and mix them at random or by feature and add them to the
regression test along with their matching feature.

Five mobile testing tools for

developers to consider

p.6

How to track configuration

changes in mobile app testing
p.12

Another logical option is to test them with the feature they control. If the
testing numbers are too high rotate the feature and create small subsets of
test suites that cover a piece of each feature. Switch out testing suites each
regression cycle. Each testing suite contains a portion of every feature so in
this way, full testing occurs over a series of regression testing events but all
features are covered to some extent.

Next Article

Page 13 of 14

E-guide

In this e-guide
App Quality Alliance compiles
mobile testing criteria

p.2

Five mobile testing tools for

developers to consider

p.6

How to track configuration

Getting more PRO+ exclusive content

This e-guide is made available to you, our member, through PRO+ Offersa
collection of free publications, training and special opportunities specifically
gathered from our partners and across our network of sites.
PRO+ Offers is a free benefit only available to members of the TechTarget
network of sites.

changes in mobile app testing

p.12

Take full advantage of your membership by

visiting http://pro.techtarget.com/ProLP/
Images; Fotalia
2015 TechTarget. No part of this publication may be transmitted or reproduced in any form or by any means without
written permission from the publisher.

Page 14 of 14