Mobile phone communication standards
have advanced from the basic analog communication channel to high-speed fourth
generation (4G) today. Today a smartphone / tablet user enjoys a ubiquitous status,
thanks to advancements in wireless network technology. Figure depicts the
evolution of wireless networks over the years and the impact it has on the
phones we use:
How important is mobile testing?
The adoption of smartphones and other
mobile devices is rapidly increasing. In the short term, mobile Internet usage
will take over desktop Internet usage and the world’s population will be
outnumbered by the quantity of active mobile devices. Traditional websites
started to have an optimized version for mobile devices, as a complement.
However, nowadays the focus is on the apps that are downloaded and installed on
the device. In some cases they are not a complement to a main product, but are
the main product themselves. Unlike mobile websites, apps can take advantage of
all the features of the mobile device and increase customer loyalty. Bearing in
mind that devices and operating systems (OS) are evolving continuously and the
app market is highly competitive, we notice that time to market tends to be
reduced in order to deliver before the competition. In this context, an
effective testing strategy becomes a critical success factor.
The testing scope
The mobile world is highly diverse.
Are you aware of all the variables that should be considered when designing a
test plan for mobile testing? The following is a list of most of them:
▪▪ One app per operating system. There will be
a different app per supported platform and each one should be tested. We have Android
and iOS, Symbian, BlackBerry OS and Windows Mobile. You will also need devices
for each platform.
▪▪ Operating system version – different
versions of the same OS may have different supported features. The app should
have both implementations in order to work properly on all OS versions. That is
something that should be checked. It will be important to have a device with
the last OS and one or two more with previous versions.
▪▪ Support for previous versions of the app –
when a new version of the app is released you should make sure that previous
versions, where users have not updated it yet, will not face bugs because of the
changes introduced.
▪▪ Device capabilities – smartphones can make
calls, but iPod Touch and some tablets cannot. The iPad version 1 has no
camera. How well does your app adapt to the available features?
▪▪ User settings – the user may have no email
account set up on the device. Some features like geolocation, push notification
alerts, and sound can be disabled by the user. How well can the App cope up
with such user settings?
▪▪ Interruptions – mobiles are all about
multitasking. How does your app do if the device receives an incoming call,
SMS, application alert, low battery alert, or application switch? There can be numerous
scenarios like this to be tested for an app.
▪▪ Features of your app – if your app adapts
the interface to the orientation (landscape or portrait), or supports
multilingual or other local configuration, these should be considered in the
testing scope. The languages with the longest words or special characters are
the more error prone in the visual presentation.
▪▪ Real connection – if the app is supposed to
be used through a slow connection (like 2G, 3G, Edge, GPRS, etc.) it should be
tested with that connection speed.
▪▪ Policies and guidelines – your app must comply
with the policies and guidelines of the distributor (Android Play Market, Apple
AppStore, etc.). It is worth checking them. If the app gets rejected, you may
lose valuable time in the process.
Finally, you may have one version for each
device generation – WAP for non-smartphones, HTML 4 for most smartphones, HTML5
Webapps for the new ones and tablet versions. If you want to see an example of
all this diversity, try accessing Twitter and Facebook with a desktop computer,
a tablet, a smartphone, a WAP-capable mobile phone, and also download their
apps for Android, iOS, Symbian, BlackBerry OS and Windows Phone. You will find
different products for each platform.
Mobile Devices or
simulators
The eternal question. The pros and
cons of each of these approaches must be carefully considered by enterprises when
formulating a mobile testing strategy. Here are some of the advantages and
disadvantages of each approach:
Handset Simulators
A handset simulator is a software
application which mimics all of the typical hardware and software features of a
typical mobile device in its real environment. Put simply, simulation is a
representation of a real system. Simulator gives us some feature to help us to
perform testing activities.
ADVANTAGES:
Lead time and cost
The big advantage is money. Designing,
building, testing, redesigning, rebuilding and retesting, is part of an project
life cycle and it certainly time consuming.
Simulations take the
building/rebuilding phase out of the loop by using the model already created in
the design phase. Most of the time, simulation testing is cheaper and faster
than performing multiple tests on the design each time.
Moreover, simulation systems are
easier. Most of the current simulators are free and mobile phone manufacturers
have made enormous efforts to ensure their platforms are easy to test and that
there is a wide range of solutions available.
Hardware connections
Simulation systems are sometimes used
with specialized hardware. But emulators are simply client software that runs
locally on your PC, they have less latency than real devices connected to the
local network or in the cloud. Depends on the application, this latency should
be carefully considered when the application runs on a real devices to avoid negatives
effects in our application. Simulators are great friends for development and QA
teams. They saves a lot of time and money.
DISADVANTAGES
Simulation errors
The first of these disadvantages is
simulation errors. Any incorrect key stroke has the potential to alter the
results of the simulation and give us wrong results. To get your simulation to
give us accurate results we must first run a base line to prove that it works
If simulations need to be acceptable
in the general community, there needs to be a lot of test scenario results to
simulate them.
Hardware-Software differences
Another aspect of testing on a
simulator is the differences between software and hardware. Simulators do not
reflect the specific hardware and software features of each supported device.
Performance
The last important aspect of
simulators is where the PC is running. Depending on the processing power of the
PC running the emulator and the type of handset or smartphone, with limited CPU
and memory, being used for testing, performance on the emulator may be
unrealistically good or bad.
Real Handsets
By definition, handsets are physical
resources which need to be managed. Unlike simulators, where additional “handsets”
can be either additional software installed or simple configuration, real
handsets are something physical to own.
For one, it is important to note that
testing on real handsets is not an optional task, it is always a given task. As
the simulators, there are some advantages with testing on real handsets.
ADVANTAGES:
User Experience
Mainly, the application will be run by
different kind of users and on different devices, and testing on real handsets
always gives us a true user experience, because is the only way to truly
understand the user experience. In addition, working with real handsets always
gives us accurate results and we will find actual application issues.
Performance Testing
Performance testing is always an
important task when testing. This task is difficult to perform with a simulator,
but, when using real handsets, it is easier to expose performance defects, as
well as defects which are the result of the handset itself or its environment.
In addition, we will find crashes, and memory leak issues which can not found
on an emulator
Interoperability Testing
Some of our mobile applications will
work through mobile communications networks. It is important to perform
Interoperability testing in a live network with real device, because if we
perform this kind of testing running on simulator, the network connection will
be different. With testing on real handsets we can find jitters, due to
interference and crosstalk with other signals which can affect the performance
of processors in our application, introduce clicks or other undesired effects
in audio signals, and loss of transmitted data between network devices. It is
important to note that the amount of tolerable jitter depends on the affected
application.
Security
A common concern amongst mobile users
is security. People are sensitive to data, such as bank account numbers that
remain on handsets, or if passwords are displayed on the screen. Testing for these
types of security concerns in a simulated environment are not a good use of
time because is the behavior of the actual handset that needs to be tested.
Adding to the challenge, not all handsets have the same security designs, so
each device must be individually tested. Real handsets have not been designed
with testing in mind. It means there are some disadvantages to consider when
the testing activities are performed with real handset.
Interoperability Testing
As commented above, interoperability
testing is performed in a real handset. But sometimes advantages become
disadvantages. Interoperability testing requires, not only handset costs,
personal costs and time; this kind of testing is hard (although there are some
expensive simulators for telecommunications networks) and takes too long.
Conclusion
We have seen both advantages and
disadvantages between testing on simulator and testing on real handsets. Taking
a look, simulators are great, but we should not assume that just because our
software works perfectly on an simulator, it will work in the same way on the
real device. However, sometimes there is just no substitute for the real thing.
For this reason, simulators are a very useful step in any testing program, but
should never be used to replace real handset testing, because it gives you
always a real
state of your application. A good
approach is to use simulator in the early stages of testing, when the maturity
level of the application is low and simulators accelerates the resolution of
problems, and the use of real handsets in the late stages. However, the
decision at what stage should be use a simulator or a real handset depends on
the organization. A lot of factors should be carefully considered, as
deadlines, costs, personal involved in QA activities and customer demands. Using
the combination a simulator and real handset, along with a solid test plan,
brings us to the point where our software is stable and appears to be working
correctly on a large number of devices.
