Published on Jan 05, 2023
Researchers at the Chinese Academy of Sciences and Monash University have developed a new method for generating text input for mobile app testing based on a pre-trained large language model (LLM). QTypist was evaluated on 106 Android apps and automated testing tools, demonstrating a significant improvement in testing performance.
Automating mobile app testing is complicated by the need for text input generation, which is even challenging for human testers, according to the researchers. A consequence of this is the fact that different types of inputs may be required, including geolocation, addresses, and health measures, as well as the relationship that may exist between the different inputs required on successive input pages, resulting in validation restrictions. Furthermore, as one of the paper authors explained on Twitter, based on the input provided on one app view, other views will be presented accordingly.
A number of large language models (LLMs) have been demonstrated to be capable of writing essays, answering questions, and generating source code. Using LLMs, QTypist attempts to generate meaningful output that can be used as a text input within a mobile application.
As a first step, QTypist extracts context information for a GUI view by using a GUI testing tool, including metadata associated with input widgets, such as user hints, context information related to nearby widgets, and global context, such as the activity name.
The prompt generation step relies on the three categories of extracted information to construct a prompt based on linguistic patterns defined by authors working on 500 reference apps.
According to this process, 14 linguistic patterns are identified respectively related to the input widget, the local context, and the global context. For designing the patterns of the input widget, we employ keywords such as nouns (widget[n]), verbs (widget[v]) and prepositions (widget[prep]).
The prompt dataset is then used as an input to GPT-3, whose output is then used as input content. As part of the evaluation, this approach was compared with the baselines of a number of alternative approaches, including DroidBot, Humanoid, and others, as well as a human evaluation of the quality of the generated input. The researchers also conducted a usefulness evaluation of 106 Android apps available on Google Play by integrating QTypist with automated testing software. In all cases, QTypist was able to improve existing approaches.
While the initial research conducted by the team behind QTypist shows promise, more work is needed to extend its use to situations in which the app does not provide sufficient context information, as well as to apply it to scenarios beyond GUI testing.
