Construction of Semantic Model for GUI of Mobile Applications Using Deep Learning
Proceedings - 4th IEEE International Conference on Artificial Intelligence Testing, AITest 2022
Modeling the graphical user interface (GUI) of mobile applications is a crucial task for automated robotic testing. Pixel-based modeling methods are non-intrusive and thus have potential for truly black-box automation. However, existing modeling methods can hardly produce accurate models because they do not take the semantic and structural information of GUI into account. In this paper, we propose a layered semantic approach to modeling GUI for mobile applications to address this important problem. The proposed approach adopts a method for recognizing GUI elements and a novel strategy for semantics acquisition using deep learning. The model generated using the proposed approach can support the fully black-box automated robotic testing. We evaluate the approach by conducting a small experiment on 10 mobile applications. The results demonstrate that the proposed approach is effective in generating the GUI models.
character recognition, deep learning, GUI modeling, object detection
Qingying Liu, Tao Zhang, Jerry Gao, Shaoying Liu, and Jing Cheng. "Construction of Semantic Model for GUI of Mobile Applications Using Deep Learning" Proceedings - 4th IEEE International Conference on Artificial Intelligence Testing, AITest 2022 (2022): 7-11. https://doi.org/10.1109/AITest55621.2022.00010