Why Memory Test?
Mobile
handsets have very limited memory compared to modern computers, and mobile
operating systems have a default behavior to terminate applications that are
using excessive memory and causing a poor user experience. Because of this
memory testing is exceptionally important for mobile applications to ensure
that each mobile application maintains optimized memory usage throughout the application
life cycle.
However
mobile application memory test is not that simple. This is mostly because each
mobile operating system and its memory management mechanisms are not standard
and unique. This is the main reason why some mobile applications smoothly run
on Apple IOS devices but crashed in Android devices.
Testing
mobile application memory is not simple like mobile functional test. Most of
the people depend on testing tools for this. However you should understand how
your mobile OS manage memory and adopt this knowledge into your test strategy
is the success factor of your test. This white paper is discussing simple test
strategy for Android and IOS mobile memory test.
Issues and Risks
Mobile OS Behavior and Memory Leaks
Memory
is something totally belongs to the Kernel of the system. Therefor memory
management is Kernels responsibility. As mobile developers we have limited
options to control memory management.
The
biggest memory killer is memory leaks, Observations or defects you find during
your memory test are mostly said memory leaks.
Most of the testers use some test tools and monitor mobile Ram and if
Ram usage is 100% for a transaction they raise the red flag. This is totally
wrong assumption in terms of memory leaks.
Memory
leak are something that your mobile applications reluctant to clear its own
memory objects witch he creates. Therefore memory leaks are totally depending
on OS kernels ability to clear application HEAP.
Android
kernel uses garbage collector (GC) for memory clean. This is mostly because
Android is Java and Java use lots of objects in the memory in order to execute
your application. This is what developers refer as classes and there instances.
GC needs a lot of head-room to run. Depending on the type of GC, you really
need 2 times to 8 times more memory than you are actually using to get good
performance. When memory is constrained the system slows down dramatically.
This is why Android applications usually use more memory then IOS Applications.
However
Objective C based IOS dose note use same approach. IOS was designed from the
outset to be memory efficient and avoid "Garbage Collection" of this
sort. Hence, the iPhone can run faster on lesser memory and is able to deliver
similar battery life to that of many Android phones boasting vastly larger
batteries.
More Memory Mean More Energy
Android
tends to keeps all your opened applications alive in within RAM, if there is
enough free RAM to do it. This increase energy consumption because your
applications are only initialized on the first run, and then kept alive while
you use your phone. These applications are however, not using any energy while
kept on your RAM, as the processes are generally not running unless background
processes exist.
For
Android, more RAM means more applications can be kept alive, thus faster
switching between applications for the user. Because of this Android tends to
waste empty RAM. However when Android need RAM and he don't have it, Android
will kill automatically the less used processes until he have enough RAM to run
active applications.
IOS
does the same thing, all recently used applications remain in memory in a
suspended state, so it comes down to how Android and iOS handle these suspended
apps.
Usually
the limitations enforce by IOS restrict developers and what they can do causes
them not to do much harm, while the freedom Android gives them often causes
memory leaks or resource hogging background processes. Android has made some
attempts to control this, such as killing apps that use too much CPU, but
still, these problems can be noticed by the user. This coupled with the fact
that every phone manufacturer adds its own apps when they ship your Android
device means every manufacturer can potentially add an app that will hog the
memory. With iOS you only have to worry about Apple, and they usually undergo
throw testing.
Development Best Practices Vs Memory Leaks
Android
uses Java which promotes a lot of good practices which optimize the Java code
specifically for Android devices, reducing the size of the app, the memory
usage, and the impact of Garbage Collection. However, in order to do this,
Android declares a lot of Java's good practices as bad practices, something not
all developers are aware of, which means they think they are writing good code,
when actually, it's quite the opposite. This is why garbage collection has such
an impact.
IOS
does not have this issue since there development environment xCode has enforced
their best practices and the application runs on controlled environment witch
generally tested by Apple. However modern hybrid development frameworks such as
Titanium, PhomeGap also generate IOS and Android applications. These
applications use underlying Framework libraries. This needs to consider for
your test strategy.
Test Approach
Android
and IOS ships with debugging tool called Android DDMS and IOS xCode
Instruments. Both these tools are capable of monitoring your application memory
usage. Therefore we are not going to discuss how to read said data form the
tool but we will discuss how to use them and find a leak.
With
the information we discuss in “Section 2” of this whitepaper, we should know
following information about our AUT in order to plan our memory test.
1.
Is
our AUT use bitmaps and where
2.
Code
abstractions and witch workflow actually execute it
3.
Use
of external libraries and execution points
4.
Use
of native hardware calls and execution points
5.
Performance
best practices followed by the development team
6.
Background
process if any and there execution and termination points
Then
you need to identify your test scenarios which cover above points. These are
your test cases for the memory test. Then you should understand the application
lifecycle. Application life cycle is defers form mobile OS to OS. However for
the simplicity of the test we can assume following 5 stages
1.
Not running - the app has been terminated or
has not been launched.
2.
Inactive - the app is in the foreground but
not receiving
3.
Active - the normal state of "in
use" for an app
4.
Background - the app is no longer on-screen
but is still executing code
5.
Suspended - the app is still resident in
memory but is not executing code
When
press the home button, the app moves from Active to Background. Most apps
usually then go from Background to Suspended in a matter of seconds. The first
technical caveat is that Suspended apps remain in the device's memory. This is
so they can resume more quickly when you go back to them. They're not using
processor time and they're not sucking battery power.
Test Execution
Then
use above behavior and identify your test execution plan. You must cover at
least above 5 items during our memory test. Then use either Android DDMS or
xCode Instruments to measure following for each test execution cycle. You need
to call GC manually before you take each reading.
1.
Heap
Size in (MB)
2.
Allocated
Heap (MB)
3.
Free
Heap (MB)
4.
No
of Objects
Your
Test execution will be as below.
|
Time
|
Test Activity
|
Heap Size (MB)
|
Allocated Heap (MB)
|
Free Heap (MB)
|
Heap Used
(%) |
No of Objects
|
|
12.45
|
Log into Dash Board and stay
for 5min
|
19.566
|
16.641
|
2.926
|
85.05%
|
55509
|
|
13.45
|
Push app into Foreground and
wait 5min
|
19.566
|
16.636
|
2.931
|
85.02%
|
55207
|
|
13.50
|
Push app into background and
wait 5min
|
19.566
|
16.633
|
2.933
|
85.01%
|
55192
|
|
13.55
|
Push app into Foreground and
wait 5min
|
19.566
|
16.646
|
2.921
|
85.07%
|
55430
|
|
14.00
|
Push app into background and
wait 5min
|
19.566
|
16.633
|
2.933
|
85.01%
|
55192
|
|
14.05
|
Push app into Foreground and
wait 5min
|
19.566
|
16.648
|
2.919
|
85.08%
|
55475
|
|
14.10
|
Logout
|
19.566
|
16.639
|
2.928
|
85.04%
|
55277
|
|
14,15
|
send the sign in screen to BG
and bring back to FG, Then stay in login screen
|
19.566
|
16.639
|
2.928
|
85.04%
|
55278
|
|
14.20
|
Stay in login screen
|
19.566
|
16.639
|
2.928
|
85.04%
|
55279
|
|
………
|
……………………………….
|
……
|
……
|
……
|
………
|
………
|
|
………
|
……………………………….
|
……
|
……
|
……
|
………
|
………
|
|
………
|
……………………………….
|
……
|
……
|
……
|
………
|
………
|
|
16.50
|
Logout
|
19.566
|
16.64
|
2.927
|
85.04%
|
55294
|
|
|
Await 2 hrs. to clear heap
|
|
|
|
|
|
Analyzing Test Results
Identifying Memory Leak
Once you
collect above data then you can plot below graph and check witch test scenario
fail to clear its heap after GC.
Please
refer to above charts. At the time of the application starts 7.023MB of Heap
(A) was used and when sending the application background and doing some other
task (Take photo) and calling the application back to foreground has increase
the heap usage to 7.148MB. However after logout Heap usage was 7.058MB. Then I
leave the handset for 2hours while calling GC for 15min intervals but suspected
leak was not cleared. This way we can identify witch test scenario/application
work flow is leaking memory
Analyze Heap Dump
Now
we need to get a binary dump of the heap memory using either Android DDMS or
xCode Instruments and analyze for witch objects or class actually cause this
problem.
There
are several open source heap analyzers in the internet and you can download
either one and open your AUT heap dump. Then create suspected memory leak
report and Object usage report to identify witch object stayed in the heap for
longer period of time.
