Construction of Semantic Model for GUI of Mobile Applications Using Deep Learning

Publication Date

1-1-2022

Document Type

Conference Proceeding

Publication Title

Proceedings - 4th IEEE International Conference on Artificial Intelligence Testing, AITest 2022

DOI

10.1109/AITest55621.2022.00010

First Page

7

Last Page

11

Abstract

Modeling the graphical user interface (GUI) of mobile applications is a crucial task for automated robotic testing. Pixel-based modeling methods are non-intrusive and thus have potential for truly black-box automation. However, existing modeling methods can hardly produce accurate models because they do not take the semantic and structural information of GUI into account. In this paper, we propose a layered semantic approach to modeling GUI for mobile applications to address this important problem. The proposed approach adopts a method for recognizing GUI elements and a novel strategy for semantics acquisition using deep learning. The model generated using the proposed approach can support the fully black-box automated robotic testing. We evaluate the approach by conducting a small experiment on 10 mobile applications. The results demonstrate that the proposed approach is effective in generating the GUI models.

Keywords

character recognition, deep learning, GUI modeling, object detection

Department

Computer Engineering

Link to Full Text

Share

COinS