Almost every time we use a piece of technology, data about the usage is tracked. It seems like everybody is interested in how people behave, how they execute various tasks. This knowledge can be very valuable for marketing purposes, but it also allows developers to find bugs/errors in their applications. Focusing on the usability of applications, it would be very helpful to know what a user wants to do and match this expectation with the result of the executed actions. For example, if the developer knows that a user wants to add a contact to their list but fails to find the “add contact” button, the developer can take a closer look at this issue. Maybe the button was too small because the user was running while using the application, or because of an age-related visual impairment. Although we are not able to read the mind of the user, we can support developers to compare how users execute certain tasks. For this purpose, we need to know which tasks were executed.

Based on logged user interactions of mobile applications (e.g. order of visited pages, time spent on each page and number of interactions per page like e.g. checkbox selections or button clicks), we can associate each usage of the application – i.e. each interaction sequence – with a user task. Therefore, the designer or developer has to predefine the possible tasks (reference tasks) of the application beforehand. Using a semi-supervised clustering approach, features like the similarity of the visited pages (trajectory), the number of interactions or the time spent on each page are used to associate the interaction sequence to a certain task.

task recognition

Illustrates the different stages of the interaction sequence mining approach. First, user interaction sequences – i.e. ordered lists of UI states in mobile applications like page visits or dialogues popping up on a certain screen – are collected from real users in the field. Second, the sequences are preprocessed by eliminating cycles and combining equal sequences. Finally, the sequences are grouped and associated with reference tasks (e.g. writing an email).

Based on additional information that is stored for each screen a user visits (e.g. time on screen, number of interactions on a screen), we are able to compute measures that describe the similarity between one navigation path and another. For example, if designers record reference sequences for tasks, we can compute the similarity between these reference tasks and sequences that are generated by users. Algorithms we use and adopt are mainly derived from biomedical engineering, where similar approaches can be used in order to match DNA sequences. Computing the similarity between interaction sequences allows to find and group similar sequences that are propably executions of the same tasks. As a consequnce, this lays the foundation for further usability investigations. With our approach, designers will be able to measure standardized usability metrics such as effectiveness, efficiency or error rate. They will be able to find out where users perform probably better or where they potentially have issues executing a task without having to observe participants under controlled conditions. First experiments show that the developed approach works well for wizard-type mobile application.

In ongoing work, we are taking this concept to the next level. Instead of just comparing user sequences to predefined reference tasks, we experiment with clustering algorithms to automatically find tasks. The goal of this future work is to group user sequences according to their similarity to each other, without requiring the definition of reference tasks. As a major benefit, the result will not only be limited to the clusters defined by the reference tasks, designers will also be able to identify tasks that users did probably encounter on their own.


2014 F. Lettner, C. Grossauer, C. Holzmann – Mobile Interaction Analysis: Towards a Novel Concept for Interaction Sequence Mining – Proceedings of the 16th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI 2014), Toronto, Canada, 2014. [acm] [pdf] [slides]


Florian Lettner, Christian Grossauer, Clemens Holzmann
Department of Mobile Computing, University of Applied Sciences Upper Austria, Austria
florian.lettner [at]