GUI Agents with Foundation Models: A Comprehensive Survey

Recent advances in foundation models, particularly Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs), have facilitated the development of intelligent agents capable of performing complex tasks. By leveraging the ability of (M)LLMs to process and interpret Graphical User Interfaces (GUIs), these agents can autonomously execute user instructions, simulating human-like interactions such as clicking and typing. This survey consolidates recent research on (M)LLM-based GUI agents, highlighting key innovations in data resources, frameworks, and applications. We begin by reviewing representative datasets and benchmarks, followed by an overview of a generalized, unified framework that encapsulates the essential components of prior studies, supported by a detailed taxonomy. Additionally, we explore relevant commercial applications. Drawing insights from existing work, we identify key challenges and propose future research directions. We hope this survey will inspire further advancements in the field of (M)LLM-based GUI agents.

@article{wang2025_2411.04890,
  title={ GUI Agents with Foundation Models: A Comprehensive Survey },
  author={ Shuai Wang and Weiwen Liu and Jingxuan Chen and Yuqi Zhou and Weinan Gan and Xingshan Zeng and Yuhan Che and Shuai Yu and Xinlong Hao and Kun Shao and Bin Wang and Chuhan Wu and Yasheng Wang and Ruiming Tang and Jianye Hao },
  journal={arXiv preprint arXiv:2411.04890},
  year={ 2025 }
}