A/B tests or split tests are controlled experiments just like many other test variants. The main difference is that A/B tests are conducted while the product is used by everyday users, not during the production phase in a laboratory environment. In an A/B test, users are randomly split into two (or more) groups. Each group is then presented a different version of the user interface, which ranges from slight variations in the interface (like the position or size of a button) to a completely different look and feel between the groups. Finally, measurements are collected for each variation in order to find out which variation is more successful than another.

A/B testing has a long history in web development and is used on a daily basis by successful companies such as Amazon, eBay and Microsoft in order to improve the usability of their websites. Web applications can be tested quite easily, and there are hundreds of readily available products on the market. This means that mobile applications can also be tested using these products, as long as they rely on web technologies (i.e. mobile web applications or hybrid mobile apps). However, web-based mobile apps do not deliver the same amount of usability and security as native application do.

We have built a system that enables comfortable and secure A/B testing of native mobile applications for Android. It uses a client-server architecture to allow programmers or designers to create new groups and deploy user interface changes whenever needed. Changed user interfaces are uploaded to a server, from where they are fetched by a mobile device whenever the respective app is started. The server randomly assigns mobile devices to groups and sends back the user interface of the right group to the device. For this to work, the mobile app has to contain an A/B testing module which fetches the user interface from the server and inflates it on the device. Finally, user interactions are logged with our aspect-oriented interaction logging framework, which allows for evaluating differences in the usability between the groups. A particular challenge in this project was to cope with larger changes in user interface (e.g. exchanging whole interface elements like buttons or sliders), which requires to load code at runtime in order to process the different inputs these elements deliver.

We created an example application for Android to showcase this problem. “Have A Break” is context-aware app reminding the user to take breaks at specified time intervals. The break interval and duration can be set in two ways. The first variant (Variant A) offers a button than can be pressed by the user, which in turn opens a popup menu allowing the user to set the break interval. In the second variant (Variant B), the buttons are exchanged for sliders which set the break interval, and the text above each slider automatically updates as the user changes one of the settings.



2014 C. Holzmann, P. Hutflesz – Multivariate Testing of Native Mobile Applications – Proceedings of the 12th International Conference on Advances in Mobile Computing and Multimedia (MoMM 2014), Kaohsiung, Taiwan, China, 2014. [acm] [pdf]
2013 F. Lettner, C. Holzmann, P. Hutflesz – Enabling A/B Testing of Native Mobile Applications by Remote User Interface Exchange – Proceedings of the 14th International Conference on Computer Aided Systems Theory (EUROCAST 2013), Las Palmas de Gran Canaria, Spain, 2013. [springer]


Patrick Hutflesz, Clemens Holzmann, Florian Lettner
Department of Mobile Computing, University of Applied Sciences Upper Austria, Austria
clemens.holzmann [at] fh-hagenberg.at