Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception

29 January 2024

Jiabo Ye

Ji Zhang

Fei Huang

Jitao Sang

ArXiv PDF HTML

Papers citing "Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception"

29 / 79 papers shown

Title
Systematic Categorization, Construction and Evaluation of New Attacks against Multi-modal Mobile GUI Agents Yulong Yang Xinshan Yang Shuaidong Li Chenhao Lin Zhengyu Zhao Chao Shen Tianwei Zhang 35 1 0 12 Jul 2024
MobileFlow: A Multimodal LLM For Mobile GUI Agent Songqin Nong Jiali Zhu Rui Wu Jiongchao Jin Shuo Shan Xiutian Huang Wenhao Xu 27 7 0 05 Jul 2024
MobileExperts: A Dynamic Tool-Enabled Agent Team in Mobile Devices Jiayi Zhang Chuang Zhao Yihan Zhao Zhaoyang Yu Ming He Jianping Fan LLMAG 26 8 0 04 Jul 2024
MMedAgent: Learning to Use Medical Tools with Multi-modal Agent Binxu Li Tiankai Yan Yuanting Pan Zhe Xu Jie Luo Ruiyang Ji Shilong Liu Haoyu Dong Zihao Lin Yixin Wang LM&MA 31 24 0 02 Jul 2024
CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents Tianqi Xu Linyao Chen Dai-Jie Wu Yanjun Chen Zecheng Zhang ... Shilong Liu Bochen Qian Philip H. S. Torr Bernard Ghanem G. Li 38 14 0 01 Jul 2024
Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding Yue Fan Lei Ding Ching-Chen Kuo Shan Jiang Yang Zhao Xinze Guan Jie Yang Yi Zhang Xin Eric Wang 39 10 0 27 Jun 2024
GUICourse: From General Vision Language Models to Versatile GUI Agents Wentong Chen Junbo Cui Jinyi Hu Yujia Qin Junjie Fang ... Yupeng Huo Yuan Yao Yankai Lin Zhiyuan Liu Maosong Sun LLMAG 31 31 0 17 Jun 2024
GUI Odyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices Quanfeng Lu Wenqi Shao Zitao Liu Fanqing Meng Boxuan Li Botong Chen Siyuan Huang Kaipeng Zhang Yu Qiao Ping Luo 38 26 0 12 Jun 2024
MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents Luyuan Wang Yongyu Deng Yiwei Zha Guodong Mao Qinmin Wang Tianchen Min Wei Chen Shoufa Chen LLMAG 40 12 0 12 Jun 2024
Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration Junyang Wang Haiyang Xu Haitao Jia Xi Zhang Ming Yan Weizhou Shen Ji Zhang Fei Huang Jitao Sang LM&Ro LLMAG 29 44 0 03 Jun 2024
Graphic Design with Large Multimodal Model Yutao Cheng Zhao Zhang Maoke Yang Hui Nie Chunyuan Li Xinglong Wu Jie Shao 36 10 0 22 Apr 2024
Octopus v3: Technical Report for On-device Sub-billion Multimodal AI Agent Wei Chen Zhiyuan Li LLMAG 17 3 0 17 Apr 2024
MMInA: Benchmarking Multihop Multimodal Internet Agents Ziniu Zhang Shulin Tian Liangyu Chen Ziwei Liu LLMAG LM&Ro 27 13 0 15 Apr 2024
Autonomous Evaluation and Refinement of Digital Agents Jiayi Pan Yichi Zhang Nicholas Tomlin Yifei Zhou Sergey Levine Alane Suhr ELM 36 49 0 09 Apr 2024
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs Keen You Haotian Zhang E. Schoop Floris Weers Amanda Swearngin Jeffrey Nichols Yinfei Yang Zhe Gan MLLM 39 82 0 08 Apr 2024
Octopus v2: On-device language model for super agent Wei Chen Zhiyuan Li RALM 27 12 0 02 Apr 2024
Reframe Anything: LLM Agent for Open World Video Reframing Jiawang Cao Yongliang Wu Weiheng Chi Wenbo Zhu Ziyue Su Jay Wu 26 3 0 10 Mar 2024
TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document Yuliang Liu Biao Yang Qiang Liu Zhang Li Zhiyin Ma Shuo Zhang Xiang Bai MLLM VLM 36 87 0 07 Mar 2024
Large Multimodal Agents: A Survey Junlin Xie Zhihong Chen Ruifei Zhang Xiang Wan Guanbin Li LM&Ro LLMAG 37 38 0 23 Feb 2024
Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast Xiangming Gu Xiaosen Zheng Tianyu Pang Chao Du Qian Liu Ye Wang Jing Jiang Min-Bin Lin LLMAG LM&Ro 35 47 0 13 Feb 2024
UFO: A UI-Focused Agent for Windows OS Interaction Chaoyun Zhang Liqun Li Shilin He Xu Zhang Bo Qiao ... Yu Kang Qingwei Lin Saravan Rajmohan Dongmei Zhang Qi Zhang LLMAG 58 65 0 08 Feb 2024
MM-LLMs: Recent Advances in MultiModal Large Language Models Duzhen Zhang Yahan Yu Jiahua Dong Chenxing Li Dan Su Chenhui Chu Dong Yu OffRL LRM 37 173 0 24 Jan 2024
AppAgent: Multimodal Agents as Smartphone Users C. Zhang Zhao Yang Jiaxuan Liu Yucheng Han Xin Chen Zebiao Huang Bin-Bin Fu Gang Yu LM&Ro LLMAG 11 77 0 21 Dec 2023
mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration Qinghao Ye Haiyang Xu Jiabo Ye Mingshi Yan Anwen Hu Haowei Liu Qi Qian Ji Zhang Fei Huang Jingren Zhou MLLM VLM 116 367 0 07 Nov 2023
ControlLLM: Augment Language Models with Tools by Searching on Graphs Zhaoyang Liu Zeqiang Lai Zhangwei Gao Erfei Cui Ziheng Li ... Lewei Lu Qifeng Chen Yu Qiao Jifeng Dai Wenhai Wang MLLM 126 30 0 26 Oct 2023
MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning Jun Chen Deyao Zhu Xiaoqian Shen Xiang Li Zechun Liu Pengchuan Zhang Raghuraman Krishnamoorthi Vikas Chandra Yunyang Xiong Mohamed Elhoseiny MLLM 154 280 0 14 Oct 2023
A Survey on Multimodal Large Language Models Shukang Yin Chaoyou Fu Sirui Zhao Ke Li Xing Sun Tong Bill Xu Enhong Chen MLLM LRM 33 551 0 23 Jun 2023
Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction Danyang Zhang Zhennan Shen Rui Xie Situo Zhang Tianbao Xie ... Siyuan Chen Lu Chen Hongshen Xu Ruisheng Cao Kai Yu ELM LLMAG 26 3 0 14 May 2023
mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality Qinghao Ye Haiyang Xu Guohai Xu Jiabo Ye Ming Yan ... Junfeng Tian Qiang Qi Ji Zhang Feiyan Huang Jingren Zhou VLM MLLM 203 883 0 27 Apr 2023