November 3th 2015 Debugging Native Android Code

August 5th 2015 Magnetic Declination | | December 1st 2015 Qt 5,5,1 is out

Debugging a crash in native Android C++ code is a hard thing to do. I’ll explain in the following:

If a crash occurred in native C++ Android code (e.g. a NULL pointer SIGSEGV), the Android system writes a tombstone file and logs the crash backtrace. So we fire up the Android Debug Bridge (the “adb” tool as part of the Android SDK) and read the most recent crash log via:

adb logcat -d -s DEBUG:I

Here is an example:

I/DEBUG   ( 2905): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
I/DEBUG   ( 2905): Build fingerprint: 'samsung/kltexx/klte:5.0/LRX21T/G900FXXU1BOA3:user/release-keys'
I/DEBUG   ( 2905): Revision: '14'
I/DEBUG   ( 2905): ABI: 'arm'
I/DEBUG   ( 2905): pid: 5918, tid: 6136, name: QtThread  >>> org.openterrain.ping <<<
I/DEBUG   ( 2905): signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0
I/DEBUG   ( 2905):     r0 a03b3544  r1 00000000  r2 00000000  r3 00000000
I/DEBUG   ( 2905):     r4 a03b38c0  r5 b394c640  r6 a07bd6dc  r7 b6ef7e04
I/DEBUG   ( 2905):     r8 a03b3970  r9 af8ff388  sl b6ea179d  fp a03b356c
I/DEBUG   ( 2905):     ip a08b4234  sp a03b34e0  lr a05303a8  pc a05303b8  cpsr 60010010
I/DEBUG   ( 2905): 
I/DEBUG   ( 2905): backtrace:
I/DEBUG   ( 2905):     #00 pc 0017c3b8  /data/app/org.openterrain.ping-2/lib/arm/libPing.so (CommonAPP::CommonAPP(QString, QString, QString, QWidget*)+1412)
I/DEBUG   ( 2905):     #01 pc 0014aab8  /data/app/org.openterrain.ping-2/lib/arm/libPing.so (ClientUI::ClientUI(SSLTransmissionQueueClient*, QWidget*)+132)
I/DEBUG   ( 2905):     #02 pc 00146d44  /data/app/org.openterrain.ping-2/lib/arm/libPing.so (main+8396)
I/DEBUG   ( 2905):     #03 pc 00019aa9  /data/data/org.openterrain.ping/qt-reserved-files/plugins/platforms/android/libqtforandroid.so
I/DEBUG   ( 2905):     #04 pc 000137bb  /system/lib/libc.so (__pthread_start(void*)+30)
I/DEBUG   ( 2905):     #05 pc 0001189b  /system/lib/libc.so (__start_thread+6)
I/DEBUG   ( 2905): 
I/DEBUG   ( 2905): Tombstone written to: /data/tombstones/tombstone_06

This yields the function in which the crash occurred, but no source code line number to refer to. Now we use the “ndk-stack” tool, which extracts the respective line numbers:

adb logcat -d -s DEBUG:I | <android-ndk dir>/ndk-stack -sym <build dir>

Here is the more readable translation of the above backtrace:

********** Crash dump: **********
Build fingerprint: 'samsung/kltexx/klte:5.0/LRX21T/G900FXXU1BOA3:user/release-keys'
pid: 5918, tid: 6136, name: QtThread  >>> org.openterrain.ping <<<
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0
Stack frame #00 pc 0017c3b8  /data/app/org.openterrain.ping-2/lib/arm/libPing.so (CommonAPP::CommonAPP(QString, QString, QString, QWidget*)+1412): Routine CommonAPP::CommonAPP(QString, QString, QString, QWidget*) at /home/roettger/Projects/libpong/build-Ping-Android_for_armeabi_v7a_GCC_4_9_Qt_5_5_0-Debug/src/../../pong/src/ui_app.cpp:12 (discriminator 16)
Stack frame #01 pc 0014aab8  /data/app/org.openterrain.ping-2/lib/arm/libPing.so (ClientUI::ClientUI(SSLTransmissionQueueClient*, QWidget*)+132): Routine ClientUI::ClientUI(SSLTransmissionQueueClient*, QWidget*) at /home/roettger/Projects/libpong/build-Ping-Android_for_armeabi_v7a_GCC_4_9_Qt_5_5_0-Debug/ping/../../pong/clientui.cpp:13 (discriminator 7)
Stack frame #02 pc 00146d44  /data/app/org.openterrain.ping-2/lib/arm/libPing.so (main+8396): Routine main at /home/roettger/Projects/libpong/build-Ping-Android_for_armeabi_v7a_GCC_4_9_Qt_5_5_0-Debug/ping/../../pong/main.cpp:376
Stack frame #03 pc 00019aa9  /data/data/org.openterrain.ping/qt-reserved-files/plugins/platforms/android/libqtforandroid.so
Stack frame #04 pc 000137bb  /system/lib/libc.so (__pthread_start(void*)+30)
Stack frame #05 pc 0001189b  /system/lib/libc.so (__start_thread+6)

The -sym option needs to refer to the directory where the shared objects reside (*.so with symbolic debug information).

As an example, the following command line does the job, when using Qt Creator to deploy a native C++ Qt app:

adb logcat -d -s DEBUG:I | .../android-ndk-r10e/ndk-stack -sym .../build-<app name>-Android_for_armeabi_v7a_GCC_4_9_Qt_5_5_0-Debug/<app name>/

So if you have your adb tool at hand debugging a crash is kind of complicated, but it works.

But what do we do, when the app crashes on the go? We may open a terminal and try to save the actual crash log in order to analyze it later on (do not forget the “-v time” option for logcat):

logcat -d -s -v time DEBUG:* CrashAnrDetector:* > /sdcard/crash.log.txt

But you won’t see anything in the logs unless you are on a rooted device. This is due to the Android 4.1 safety policy that no app (i.e. the terminal) can see information of another app (i.e. the crashed app). What to do? Maybe try to access the tombstone in “/data/tombstones/”. But of course, you get a “permission denied”, unless your device is rooted.

What is worse with the log is that it wraps around after ca. 45 minutes. At least on my Samsung S5 with Android 5.0 I see a constant hum of log messages, which quickly fill up the log space. If we cannot make it home quickly, we’ll loose all the debug information after 45 minutes.

So, one could think that we just need to shut down the phone to freeze the log. But no way, the log is a cyclic memory buffer. This means that if you shutdown the device, all the debug information is lost and only general crash reporting information is preserved. The latter does not contain a trace of the crash, so it is useless for debugging.

So bottom line, if you are seriously developing native C++ apps on Android, there are three options to analyze a crash that might happen on the go:

  • Carry a netbook with you, so that you can save the crash log via adb.
  • Get your hands on a rooted device to save the crash log in a shell.
  • Log all actions to a file by yourself (you will have to supply a signal handler to catch native crashes).

August 5th 2015 Magnetic Declination | | December 1st 2015 Qt 5,5,1 is out