Conducting user tests with the aim to improve the usability of mobile applications is a critical competitive factor in this fast-moving market. Users are observed how they use a specific application, in order to calculate usability metrics such as the percentage of successfully executed tasks, the time required to perform certain tasks or the frequency of errors, such as those caused by wrong entries or navigation problems. In addition to the necessary expertise, primarily technical obstacles such as the heterogeneity of mobile platforms and the high resource requirements (time required, number of subjects, extensive infrastructure, etc.) are reasons why usability testing of mobile applications can be found mostly in large companies. Last but not least, it is the current lack of techniques and methods on mobile platforms which avoids a flexible and efficient way of testing under real-world conditions.

AUToMAte Sequence Visualization The primary goal of the project is therefore the development of an automated procedure for the effective and efficient implementation of usability tests on mobile platforms. It should support tests with an arbitrarily large number of users, over a long period of time, with the incorporation of the context of use (e.g. activity and viewing direction) and under real-world conditions. The developed procedure will be adapted to the requirements of companies in different sectors. It will be prototypically implemented using state-of-the-art technologies, integrated into the software development process of companies and exemplarily evaluated in pilot projects. Key results of the project will be (i) a set of science-based techniques and methods for the assessment and interpretation of usability metrics for mobile applications and (ii) a practical software prototype for the automated execution of usability tests and its practical application in different companies from the fields of automation systems, application development, and mobile commerce.

The scientific findings of the project will lead to long-term and interdisciplinary research activities at the Faculty of Computer Science, Communication and Media at the Campus Hagenberg and at the Faculty of Management at the Campus Steyr of the University of Applied Sciences Upper Austria. Moreover, the gained knowledge of the research project will be used in more than 10 courses at the University of Applied Sciences Upper Austria. The practical results are reflected in a software infrastructure for conducting automated usability tests, which will have a recommendation system for the generation of user interface improvements and will be provided to companies in the long run.

The project is funded by the Austria Research Promotion Agency (FFG) in the Cooperation and Innovation programme (COIN “Aufbau”). The duration of the project is from 06/2013 to 05/2016.

The Austrian Research Promotion Agency (FFG)

The Automate Toolkit Goes Open Source

Posted on Aug 23, 2016

The Automate Toolkit Goes Open Source

In May 2016, the AUToMAte (Automated Usability Testing of Mobile Applications) project, funded by the Austria Research Promotion Agency (FFG), has been finished successfully. The primary goal of the project was the development of an automated procedure for conducting usability tests on mobile platforms. Parts of the implementation have been published as open source project named automate toolkit under the GNU General Public License (GPL) version 3 (or later). It is hosted on GitHub and contains everything needed for tracking mobile device...

read more

Eye-Tracking Study for Company ILLUSTREE

Posted on Apr 15, 2016

Eye-Tracking Study for Company ILLUSTREE

In an ongoing follow-up activity for the R&D project AUToMAte, our project partners of the Campus Steyr of the University of Applied Sciences Upper Austria conduct usability tests for the user experience design company ILLUSTREE Great User Experiences GmbH. The tests are focused on evaluating gaze behavior using the SMI Eye Tracking Glasses, both in mobile and stationary settings. A combination of think aloud methods and eye tracking is used for the tests. The eye tracking data is analyzed using scanpaths (showing fixation points and...

read more

Follow-Up Project with emporia Telecom GmbH & Co. KG

Posted on Jan 14, 2016

Follow-Up Project with emporia Telecom GmbH & Co. KG

In a follow-up project for our R&D project AUToMAte, the new emporiaSMART phone developed by emporia Telecom GmbH & Co. KG will be evaluated. This Android-based smartphone, which was introduced at the Mobile World Congress 2015, has been designed to meet the needs of senior citizens and provides several unique features like the removable keyboard cover and a stylus for text input. We will test the emporiaSMART phone in the field with selected participants using an adapted version of our AUToMAte tool. Among traditional metrics like...

read more

Use of AUToMAte Software in SOCIALCARE Project

Posted on Sep 24, 2015

Use of AUToMAte Software in SOCIALCARE Project

The software framework for automated usability testing of mobile applications, which is being developed in the AUToMAte project, will be used in the EU-project SOCIALCARE. This project aims to support and promote social innovation processes of citizen empowerment using innovative technological solutions including state-of-the-art mobile devices and body sensors. The framework will be used to support our project partners in user testing of high-fidelity prototypes of their mobile solutions, which will be developed in the course of the...

read more

First Phase of the AUToMAte Project Successfully Completed

Posted on Sep 23, 2015

First Phase of the AUToMAte Project Successfully Completed

We are happy to announce that the first phase of the AUToMAte project, which started in June 2013 and ended in May 2015, has successfully been completed! All project goals have been met according to the funding guidelines of the Austria Research Promotion Agency (FFG) in the Cooperation and Innovation programme (COIN “Aufbau”). One of the goals in the first project phase was to acquire benchmark projects with a total sum of at least 10% of the overall AUToMAte project costs. We achieved this goal and would like to thank 6 Austrian companies...

read more

Mobile Usability Analysis of Universal.at Online Shop

Posted on Jun 2, 2015

Mobile Usability Analysis of Universal.at Online Shop

As a follow-up project for the R&D project AUToMAte, the online shop of Universal Versand GmbH has been evaluated by our project partners at the Campus Steyr of the University of Applied Sciences Upper Austria. The online shop, which is run by the company UNITO Versand & Dienstleistungen GmbH, has been evaluated with mobile Android and iOS devices (smartphones and tablets). Among other things, eye tracking glasses were used to analyze user interactions (fixations, fixation counts, scan paths, etc.) in order to improve the placement...

read more

Personal and Ubiquitous Computing (PUC) Journal Paper Accepted

Posted on Feb 1, 2015

Personal and Ubiquitous Computing (PUC) Journal Paper Accepted

We got the journal paper Only Play in Your Comfort Zone – Interaction Methods for Improving Security Awareness on Mobile Devices accepted for the Springer Journal Personal and Ubiquitous Computing (PUC). This paper has been written in cooperation with the Josef Ressel Center for User-friendly Secure Mobile Environments (u’Smile) of the University of Applied Sciences Upper Austria and the Institute for Embedded Systems of the University of Passau, Germany. Part of the research presented in this paper has been conducted in the course of...

read more

AUToMAte Demo Goes Online

Posted on Jan 30, 2015

AUToMAte Demo Goes Online

A first version of our software tool for testing the usability of mobile applications, which is currently under development in the R&D project AUToMAte, is now online at this link. The tool logs user interactions and context data while a mobile application is running, aggregates them on a server and visualizes results of an analysis of the data for the application developer. To get an impression of the capabilities of this tool, we created a simple web frontend where some of the data is already visualised; the frontend will be extended...

read more

Follow-Up Project with UNITO Versand & Dienstleistungen GmbH

Posted on Jan 8, 2015

Follow-Up Project with UNITO Versand & Dienstleistungen GmbH

A follow-up project with UNITO Versand & Dienstleistungen GmbH has been started for the R&D project AUToMAte. Unito runs some of the biggest e-commerce websites in Austria (www.quelle.at, www.universal.at, www.ottoversand.at) which are also optimized for different mobile device types (smartphones, tablets). The aim of the project is to analyze the usability of these m-commerce sites on smartphones. This is done by carrying out user tests using typical m-commerce tasks. Interactions are tracked and additionally mobile eye tracking...

read more

Follow-Up Project with ILLUSTREE Great User Experiences GmbH

Posted on Oct 14, 2014

Follow-Up Project with ILLUSTREE Great User Experiences GmbH

A follow-up project with ILLUSTREE Digitale Kommunikation GmbH has been started for the R&D project AUToMAte. The aim of this project is to analyze the usage of ILLUSTREE’s applications on different devices (tablets, smartphones, terminals). The primary focus lies on evaluating the gaze behavior using eye tracking technology, both in mobile and stationary settings. Our project partners at the Campus Steyr of the University of Applied Sciences Upper Austria will conduct the study at ILLUSTREE’s site in Vienna. ILLUSTREE is a...

read more

2 Papers Accepted for the 12th International Conference on Advances in Mobile Computing & Multimedia (MoMM 2014)

Posted on Oct 4, 2014

2 Papers Accepted for the 12th International Conference on Advances in Mobile Computing & Multimedia (MoMM 2014)

We got 2 full papers accepted for the 12th International Conference on Advances in Mobile Computing & Multimedia (MoMM 2014)! Both papers have been written in the course of the project AUToMAte. The paper Multivariate Testing of Native Mobile Applications describes an approach which not only supports A/B testing of native mobile applications, like some other commercial products, but allows for the more flexible multivariate testing of mobile applications. It is based the same mechanisms as A/B testing, but allows to compare a much higher...

read more

Paper Presentations at MobileHCI 2014

Posted on Sep 26, 2014

Paper Presentations at MobileHCI 2014

Christian Grossauer and Clemens Holzmann attended the 16th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI 2014) from September 23-26 in Toronto, Canada. They presented two paper, which have been written in the course of the project AUToMAte. Christian presented the full paper Mobile Interaction Analysis: Towards a Novel Concept for Interaction Sequence Mining, which describes a novel approach for automatically recognizing user tasks (e.g. writing an email) from logged user interactions (e.g....

read more

Follow-Up Project with ENGEL Austria GmbH

Posted on Jul 29, 2014

Follow-Up Project with ENGEL Austria GmbH

In September 2014, a follow-up project with ENGEL Austria GmbH for the project AUToMAte will start. ENGEL Austria GmbH is one of the world’s leading companies in the field of plastic machine manufacturing, with eight production plants worldwide as well as subsidiaries and representatives in over 85 countries. The company develops the software for their machine control units in-house. The control units provide a highly resistant, dirt-insensitive touchscreen for navigating through all sections of the user interface. The aim of this...

read more

2 Papers Accepted for the 16th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI 2014)

Posted on Jun 26, 2014

2 Papers Accepted for the 16th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI 2014)

We got 2 papers accepted for the 16th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI 2014)! Both papers have been written in the course of the project AUToMAte. In the full paper Mobile Interaction Analysis: Towards a Novel Concept for Interaction Sequence Mining, we describe a novel approach for automatically associating the usage of a mobile application with predefined user tasks (e.g. writing an email), based on logged user interactions like the order of visited pages and the time spent...

read more

Poster Presentation at NeuroIS 2014

Posted on Jun 11, 2014

Poster Presentation at NeuroIS 2014

At this year’s Gmunden Retreat on NeuroIS, which took place from June 5-7 in Gmunden, Austria, Werner Wetzlinger, Andreas Auinger and Harald Kindermann from the FH OÖ Campus Steyr presented ideas for combining interaction data with eye tracking data at the poster session of the conference. In their paper “Using eye tracking glasses to analyze mobile device interactions”, the features of the new SMI eye tracking glasses, which are used in the AUToMAte project, as well as our general approach of a combined analysis of gaze...

read more

Project with KEBA AG Successfully Completed

Posted on Jun 9, 2014

Project with KEBA AG Successfully Completed

The project Usability Testing of Mobile Operator Terminals with KEBA AG, which started in August 2013 as a follow-up project for AUToMAte, has been successfully completed. KEBA AG develops and manufactures control systems and mobile terminals for machines and robots, where some of them are already equipped with touch displays (e.g. KeTop T55). The aim of this project was the development of an easy-to-use software prototype for the automated logging and visualization of user interactions on mobile operator terminals. The following videos...

read more

Attendance of the Usability Stammtisch Linz

Posted on Apr 8, 2014

Attendance of the Usability Stammtisch Linz

Christian Grossauer and Clemens Holzmann attended the Usability Stammtisch Linz on April 7, 2014, where they presented and discussed the general ideas and the current status of the AUToMAte project. A particular focus was on eye tracking with the Pupil Pro eye tracker from Pupil Labs, which supports marker-based tracking of gaze positions in real-time and in natural settings. In the AUToMAte project, this will be used to record and analyze gaze positions of users while they are using a mobile application, together with touch interactions and...

read more

2 Full Papers Accepted for the 11th International Conference on Advances in Mobile Computing & Multimedia (MoMM 2013)

Posted on Sep 30, 2013

2 Full Papers Accepted for the 11th International Conference on Advances in Mobile Computing & Multimedia (MoMM 2013)

We got 2 full papers accepted for the 11th International Conference on Advances in Mobile Computing & Multimedia (MoMM 2013)! In the paper A Method for Mobile Download Conversion Rate Measurement based on Device Fingerprinting, we present a method which allows to determine if a mobile application install originates from a speci fic campaign. This requires that mobile devices are uniquely identi fied before referral to the mobile application store and after a successful install of the application. However, the mobile sandbox environment...

read more

Participation at the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp 2013)

Posted on Sep 16, 2013

Participation at the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp 2013)

Florian Lettner attended the Doctoral School at the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp 2013) from September 8-9 in Zurich, Switzerland. He presented his ongoing doctoral thesis with the working title “A Generic Concept for Task-driven Remote Usability Testing in Mobile Ecosystems”. The conference is a merger of the two most renown conferences in the field, Pervasive and UbiComp, and thus offers a single venue for the leading worldwide research in Pervasive and Ubiquitous...

read more

Follow-Up Project with KEBA AG

Posted on Aug 12, 2013

Follow-Up Project with KEBA AG

A first follow-up project with KEBA AG for the project AUToMAte has been started in August 2013. KEBA AG is an Austrian company that develops industrial products in the automation area. One of their main fields is the development of mobile operator terminals for machines and robots, some of them are already equipped with touch displays. The aim of this project, which is entitled Usability Testing of Mobile Operator Terminals, is the development of an easy-to-use software prototype for the automated tracking and analysis of user interactions...

read more

Start of the AUToMAte Project

Posted on Jun 5, 2013

Start of the AUToMAte Project

The kickoff meeting for the AUToMAte project has been carried out successfully. All project members, both from the Campus Hagenberg and the Campus Steyr of the University of Applied Sciences Upper Austria, attended the meeting. The project is funded by the  by the Austria Research Promotion Agency (FFG) in the Cooperation and Innovation programme (COIN “Aufbau”). The duration of the project is from 06/2013 to...

read more
To develop an automated procedure for the effective and efficient implementation of usability tests on mobile platforms, we will build up human resources and expertise in the field. To ensure the sustainability of the results, we will use scientific methods based on real-word requirements from project partners. This includes conducting a scientific requirements analysis, developing a widely applicable system design based on these requirements, implementing this design in a software prototype and providing a process model to show an effective and efficient way of using the AUToMAte infrastructure.Consequently the objectives of the project are:

  • A scientific requirements analysis: An extensive inquiry and analysis of industry specific requirements, which serves as foundation for identifying relevant usability metrics, integration scenarios of extern sensors (e.g. eye tracking), context information and analysis methods.
  • A widely applicable system design: An infrastructure implementation concept (system architecture, interface definitions, tools etc.) that is based on the results of the requirements analysis and takes into account the heterogeneity of platforms and devices. Industry specific requirements will be available as prototypic reference implementation (e.g. algorithms) in early project phases to enable the verification and validation in stages.
  • A software prototype: An implementation of a basic, self-scaling infrastructure for the automatic logging and evaluation of user interactions and context data of mobile applications.
  • A process model: A general framework and adaptable procedure formalized in a process model that facilitates the effective and efficient introduction as well as usage of the AUToMAte infrastructure in organizations. It consists of multiple phases and includes descriptions of all necessary activities, the corresponding results, applicable methods and tools, as well as organizational roles. It therefore shows how to use the technological tools in an organizational environment.

University of Applied Sciences Upper Austria

School of Informatics/Communications/Media, Campus Hagenberg

Christian Grossauer, MSc
Phone: +43 (0)50804 27123
E-Mail: christian.grossauer [at] fh-hagenberg.at
Andreas Guetz, BSc
Phone: +43 (0)50804 27176
E-Mail: andreas.guetz [at] fh-hagenberg.at
Prof. Dr. Michael Haller
Phone: +43 (0)50804 22127
E-Mail: michael.haller [at] fh-hagenberg.at
Prof. Dr. Clemens Holzmann (Project Leader)
Phone: +43 (0)50804 22824
E-Mail: clemens.holzmann [at] fh-hagenberg.at
Prof. Dipl.-Ing. Stephan Selinger
Phone: +43 (0)50804 22822
E-Mail: stephan.selinger [at] fh-hagenberg.at
Dustin Steiner, MSc
Phone: +43 (0)50804 27172
E-Mail: dustin.steiner [at] fh-hagenberg.at

 

University of Applied Sciences Upper Austria

School of Management, Campus Steyr

Prof. Dr. Andreas Auinger
Phone: +43 (0)50804 33413
E-Mail: andreas.auinger [at] fh-steyr.at
Prof. Dr. Dietmar Nedbal
Phone: +43 (0)50804 33415
E-Mail: dietmar.nedbal [at] fh-steyr.at
Mag. Werner Wetzlinger
Phone: +43 (0)50804 33451
E-Mail: werner.wetzlinger [at] fh-steyr.at

 

Former Members

Elisabeth Hackner, BSc
E-Mail: elisabeth.hackner [at] fh-hagenberg.at
Patrick Hutflesz, BSc
E-Mail: patrick.hutflesz [at] fh-hagenberg.at
Paul Latzelsperger, MSc
E-Mail: paul.latzelsperger [at] fh-hagenberg.at

Florian Lettner, MSc
E-Mail: florian.lettner [at] fh-hagenberg.at
KEBA - Automation by innovation.
ENGEL Austria GmbH
ILLUSTREE Great User Experiences GmbH
UNITO Versand & Dienstleistungen GmbH


emporia telecom

bluesource - mobile solutions gmbh
ImmobilienScout24 Österreich
mogree, mobile agreements, App Entwicklung, Apps, iPhone Entwicklung, Android Entwicklung, mobile Seiten, Tablets, iPad, Blackberry, Windows, Agentur, Linz, Wien, Oberösterreich, iPhone, Android, mobile Strategie, HTML5
STIWA Group


A1.net
runtastic
NTSwincash

Publications

2015 A. Riegler, C. Holzmann – UI-CAT: Calculating User Interface Complexity Metrics for Mobile Applications – Proceedings of the 14th International Conference on Mobile and Ubiquitous Multimedia (MUM 2015), Linz, Austria, 2015. [acm]
2015 C. Grossauer, C. Holzmann, D. Steiner, A. Guetz – Interaction Visualization and Analysis in Automation Industry – Proceedings of the 14th International Conference on Mobile and Ubiquitous Multimedia (MUM 2015), Linz, Austria, 2015. [acm] [pdf]
2015 P. Riedl, R. Mayrhofer, A. Möller, M. Kranz, F. Lettner, C. Holzmann, M. Koelle – Only Play in Your Comfort Zone – Interaction Methods for Improving Security Awareness on Mobile Devices – PERSONAL AND UBIQUITOUS COMPUTING, Vol. 19, No. 3, 2015, pp. 1-14. [springer] [pdf]
2015 C. Holzmann, J. Rieseneder – Creating Heat Maps for Mobile Applications with Dynamic Content – Extended Abstract Book of the 15th International Conference on Computer Aided Systems Theory (EUROCAST 2015), Las Palmas de Gran Canaria, Spain, 2015.
2014 C. Holzmann, P. Hutflesz – Multivariate Testing of Native Mobile Applications – Proceedings of the 12th International Conference on Advances in Mobile Computing and Multimedia (MoMM 2014), Kaohsiung, Taiwan, China, 2014. [acm] [pdf]
2014 W. Wetzlinger, D. Nedbal, A. Auinger, C. Grossauer, C. Holzmann, F. Lettner – Mobile Usability Testing Requirements and their Implementation in the Automation Engineering Industry – Proceedings of the 12th International Conference on Advances in Mobile Computing and Multimedia (MoMM 2014), Kaohsiung, Taiwan, China, 2014. [acm]
2014 F. Lettner, C. Grossauer, C. Holzmann – Mobile Interaction Analysis: Towards a Novel Concept for Interaction Sequence Mining – Proceedings of the 16th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI 2014), Toronto, Canada, 2014. [acm] [pdf] [slides]
2014 C. Holzmann, F. Lettner, C. Grossauer, W. Wetzlinger, P. Latzelsperger, C. Augdopler – Logging and Visualization of Touch Interactions on Teach Pendants – Proceedings of the 16th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI 2014), Toronto, Canada, 2014. [acm]
2014 W. Wetzlinger, A. Auinger, M. Dörflinger – Comparing Effectiveness, Efficiency, Ease of Use, Usability and User Experience When Using Tablets and Laptops – Proceedings of the 16th International Conference on Human-Computer Interaction (HCI International 2014), Crete, Greece, 2014 [springer]
2014 W. Wetzlinger, A. Auinger, H. Kindermann – Using eye tracking glasses to analyze mobile device interactions – Poster at the Gmunden Retreat on Neuro-Information-Systems 2014 (NeuroIS 2014), Gmunden, Austria, 2014.
2013 A. Schuch, C. Holzmann, F. Lettner – A Method for Mobile Download Conversion Rate Measurement based on Device Fingerprinting – Proceedings of the 11th International Conference on Advances in Mobile Computing and Multimedia (MoMM2013), Vienna, Austria, 2013. [acm] [pdf]
2013 F. Lettner – A Generic Concept for Task-driven Remote Usability Testing in Mobile Ecosystems – Adjunct Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp 2013), Zurich, Switzerland, 2013. [conference website]

Demo App and AUToMAte Web Frontend

For gathering interaction data of real users, we developed the Funny RSS app. The Funny RSS app is an Android App based on this xkcd comic, which suggests to replace certain (bad/negative) words of news articles with other (funnier/positive) words. The app includes the AUToMAte logging framework, therefore certain information can and will be tracked. The information consists of a variety of different kinds of data, like e.g. the user interactions on widgets, the type of interaction (tap, scale, fling etc.), device information (device type, carrier name, OS information, …) and also some context information (user activity [still, walking, …], language, current location etc.).

To get a basic impression what can be done with the data, we created a simple web frontend where some of the data is visualised. There are two projects available to choose: (1) Funny RSS (Demo) contains Funny RSS data tracked within the research group, during the period of two weeks. (2) Funny RSS (Store) contains Funny RSS data tracked from the public version available at the app store.

Download the Funny RSS App

List of feed items

List of feed items

Possible word substitutions

Possible word substitutions


Visit the Web Frontend (Alpha Version)

AUToMAte web frontend for the Funny RSS data

Access the Open Source Project at GitHub

A main result of the AUToMAte project was the development of a system for tracking and analysing mobile device usage. Parts of the implementation have been published as open source project named automate toolkit under the GNU General Public License (GPL) version 3 (or later). It is hosted on GitHub and contains everything needed for tracking mobile device usage and context information on Android phones.

GitHub repository of the toolkit.

Download the Automate Toolkit App from Google Play

The automate toolkit app, available via Google Play, provides an easy and customizable way to track the usage of arbitrary applications on Android. The tracked data is stored locally in CSV files, which can be processed further e.g. in statistical software like R. The automate toolkit is the perfect tool for supporting user studies or other tracking purposes on Android phones.

Android app for the automate toolkit.

Prof. Dr. Clemens Holzmann

Department of Mobile Computing
University of Applied Sciences Upper Austria
Softwarepark 11
4232 Hagenberg, Austria

Phone: +43 (0)50804 22824
Fax: +43 (0) 50804 22899
E-Mail: clemens.holzmann [at] fh-hagenberg.at