Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs

8 April 2024

Papers citing "Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs"

50 / 75 papers shown

Title
Benchmarking Vision, Language, & Action Models in Procedurally Generated, Open Ended Action Environments Pranav Guruprasad Yangyue Wang Sudipta Chowdhury Harshvardhan Sikka LM&Ro VLM 47 0 0 08 May 2025
Unlocking User-oriented Pages: Intention-driven Black-box Scanner for Real-world Web Applications Weizhe Wang Yao Zhang Kaitai Liang Guangquan Xu Hongpeng Bai Qingyang Yan Xi Zheng Bin Wu 29 0 0 29 Apr 2025
PixelWeb: The First Web GUI Dataset with Pixel-Wise Labels Qi Yang Weichen Bi Haiyang Shen Y. Guo Yun Ma 29 0 0 23 Apr 2025
Guiding VLM Agents with Process Rewards at Inference Time for GUI Navigation Zhiyuan Hu Shiyun Xiong Yifan Zhang See-Kiong Ng Anh Tuan Luu Bo An Shuicheng Yan Bryan Hooi 19 0 0 22 Apr 2025
InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners Yuhang Liu Pengxiang Li C. Xie Xavier Hu Xiaotian Han Shengyu Zhang Hongxia Yang Fei Wu LLMAG LM&Ro LRM AI4CE 57 1 0 19 Apr 2025
TongUI: Building Generalized GUI Agents by Learning from Multimodal Web Tutorials Bofei Zhang Zirui Shang Zhi Gao Wang Zhang Rui Xie Xiaojian Ma Tao Yuan Xinxiao Wu Song-Chun Zhu Qing Li LLMAG 33 1 0 17 Apr 2025
SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills Boyuan Zheng Michael Y. Fatemi Xiaolong Jin Z. Wang Apurva Gandhi ... Yu Gu Jayanth Srinivasa Gaowen Liu Graham Neubig Yu Su CLL 27 0 0 09 Apr 2025
ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use Kaixin Li Ziyang Meng Hongzhan Lin Ziyang Luo Yuchen Tian Jing Ma Zhiyong Huang Tat-Seng Chua 27 7 0 04 Apr 2025
Navi-plus: Managing Ambiguous GUI Navigation Tasks with Follow-up Ziming Cheng Zhiyuan Huang Junting Pan Zhaohui Hou Mingjie Zhan 28 0 0 31 Mar 2025
SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understanding Mingze Xu Mingfei Gao Shiyu Li Jiasen Lu Zhe Gan Zhengfeng Lai Meng Cao Kai Kang Y. Yang Afshin Dehghan 45 1 0 24 Mar 2025
GUI-Xplore: Empowering Generalizable GUI Agents with One Exploration Yuchen Sun Shanhui Zhao Tao Yu Hao Wen Samith Va Mengwei Xu Yuanchun Li Chongyang Zhang LLMAG 54 0 0 22 Mar 2025
Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical Deployment Gaole Dai Shiqi Jiang Ting Cao Yuanchun Li Y. Yang Rui Tan Mo Li Lili Qiu 41 0 0 20 Mar 2025
UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction Shravan Nayak Xiangru Jian Kevin Qinghong Lin Juan A. Rodriguez Montek Kalsi ... David Vazquez Christopher Pal Perouz Taslakian Spandana Gella Sai Rajeswar 76 0 0 19 Mar 2025
MP-GUI: Modality Perception with MLLMs for GUI Understanding Ziwei Wang Weizhi Chen Leyang Yang Sheng Zhou Shengchu Zhao Hanbei Zhan Jiongchao Jin Liangcheng Li Zirui Shao Jiajun Bu 52 1 0 18 Mar 2025
MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs Erik Daxberger Nina Wenzel David Griffiths Haiming Gang Justin Lazarow ... Kai Kang Marcin Eichner Y. Yang Afshin Dehghan Peter Grasch 72 2 0 17 Mar 2025
DeskVision: Large Scale Desktop Region Captioning for Advanced GUI Agents Yibin Xu Liang Yang Hao Chen Hua Wang Zhi Chen Yaohua Tang 3DV 49 0 0 14 Mar 2025
AppAgentX: Evolving GUI Agents as Proficient Smartphone Users Wenjia Jiang Yangyang Zhuang Chenxi Song Xu Yang Chi Zhang Chi Zhang LLMAG 72 1 0 04 Mar 2025
UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interface Hao Tang Chenwei Xie Haiyang Wang Xiaoyi Bao Tingyu Weng Pandeng Li Yun Zheng Liwei Wang ObjD VLM 52 0 0 03 Mar 2025
Smoothing Grounding and Reasoning for MLLM-Powered GUI Agents with Query-Oriented Pivot Tasks Zongru Wu Pengzhou Cheng Zheng Wu Tianjie Ju Zhuosheng Zhang Gongshen Liu LRM 32 1 0 01 Mar 2025
Tracking the Copyright of Large Vision-Language Models through Parameter Learning Adversarial Images Yubo Wang Jianting Tang Chaohu Liu Linli Xu AAML 43 1 0 23 Feb 2025
GenComUI: Exploring Generative Visual Aids as Medium to Support Task-Oriented Human-Robot Communication Yate Ge Meiying Li Xipeng Huang Yuanda Hu Qi Wang Xiaohua Sun Weiwei Guo 34 0 0 15 Feb 2025
AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMs Hongxin Li Jingfan Chen Jingran Su Yuntao Chen Qing Li Zhaoxiang Zhang 53 0 0 04 Feb 2025
InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection Y. Liu Pengxiang Li Zishu Wei C. Xie Xueyu Hu Xinchen Xu Shengyu Zhang Xiaotian Han Hongxia Yang Fei Wu LLMAG LRM 45 11 0 08 Jan 2025
Visual Large Language Models for Generalized and Specialized Applications Yifan Li Zhixin Lai Wentao Bao Zhen Tan Anh Dao Kewei Sui Jiayi Shen Dong Liu Huan Liu Yu Kong VLM 83 10 0 06 Jan 2025
Towards Visual Grounding: A Survey Linhui Xiao Xiaoshan Yang X. Lan Yaowei Wang Changsheng Xu ObjD 39 3 0 31 Dec 2024
PC Agent: While You Sleep, AI Works -- A Cognitive Journey into Digital World Yanheng He Jiahe Jin Shijie Xia Jiadi Su Runze Fan Haoyang Zou Xiangkun Hu Pengfei Liu LLMAG 26 2 0 23 Dec 2024
Aria-UI: Visual Grounding for GUI Instructions Yuhao Yang Yue Wang Dongxu Li Ziyang Luo Bei Chen C. Huang Junnan Li LM&Ro LLMAG 92 14 0 20 Dec 2024
OmniVLM: A Token-Compressed, Sub-Billion-Parameter Vision-Language Model for Efficient On-Device Inference Wei Chen Zhiyuan Li Shuo Xin VLM MLLM 73 0 0 16 Dec 2024
Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuning Hai-Ming Xu Qi Chen Lei Wang Lingqiao Liu 57 0 0 14 Dec 2024
Falcon-UI: Understanding GUI Before Following User Instructions Huawen Shen Chang-Shu Liu Gengluo Li Xinlong Wang Yu Zhou Can Ma Xiangyang Ji LLMAG 72 4 0 12 Dec 2024
ShowUI: One Vision-Language-Action Model for GUI Visual Agent Kevin Qinghong Lin Linjie Li Difei Gao Z. Yang Shiwei Wu Zechen Bai Weixian Lei Lijuan Wang Mike Zheng Shou LLMAG 62 6 0 26 Nov 2024
GUI Agents with Foundation Models: A Comprehensive Survey Shuai Wang W. Liu Jingxuan Chen Weinan Gan Xingshan Zeng ... Bin Wang Chuhan Wu Yasheng Wang Ruiming Tang Jianye Hao LLMAG 56 12 0 07 Nov 2024
Foundations and Recent Trends in Multimodal Mobile Agents: A Survey Biao Wu Yanda Li Meng Fang Zirui Song Zhiwei Zhang Yunchao Wei L. Chen LM&Ro LLMAG OffRL AI4TS 31 3 0 04 Nov 2024
OS-ATLAS: A Foundation Action Model for Generalist GUI Agents Zhiyong Wu Zhenyu Wu Fangzhi Xu Yian Wang Qiushi Sun ... Kanzhi Cheng Zichen Ding L. Chen Paul Pu Liang Yu Qiao 37 6 0 30 Oct 2024
EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Data Xuetian Chen Hangcheng Li Jiaqing Liang Sihang Jiang Deqing Yang LLMAG 43 2 0 25 Oct 2024
OSCAR: Operating System Control via State-Aware Reasoning and Re-Planning Xiaoqiang Wang Bang Liu LLMAG LM&Ro LRM 28 6 0 24 Oct 2024
Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms Zhangheng Li Keen You H. Zhang Di Feng Harsh Agrawal Xiujun Li Mohana Prasad Sathya Moorthy Jeff Nichols Y. Yang Zhe Gan MLLM 29 18 0 24 Oct 2024
AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models Kim Sung-Bin Oh Hyun-Bin JungMok Lee Arda Senocak Joon Son Chung Tae-Hyun Oh MLLM VLM 23 2 0 23 Oct 2024
Beyond Browsing: API-Based Web Agents Yueqi Song Frank F. Xu Shuyan Zhou Graham Neubig 29 13 0 21 Oct 2024
Griffon-G: Bridging Vision-Language and Vision-Centric Tasks via Large Multimodal Models Yufei Zhan Hongyin Zhao Yousong Zhu Fan Yang Ming Tang Jinqiao Wang MLLM 43 1 0 21 Oct 2024
Harnessing Webpage UIs for Text-Rich Visual Understanding Junpeng Liu Tianyue Ou Yifan Song Yuxiao Qu Wai Lam Chenyan Xiong Wenhu Chen Graham Neubig Xiang Yue 70 5 0 17 Oct 2024
Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents Priyanshu Kumar Elaine Lau Saranya Vijayakumar Tu Trinh Scale Red Team ... Sean Hendryx Shuyan Zhou Matt Fredrikson Summer Yue Zifan Wang LLMAG 19 17 0 11 Oct 2024
Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents Boyu Gou Ruohan Wang Boyuan Zheng Yanan Xie Cheng Chang Yiheng Shu Huan Sun Yu Su LM&Ro LLMAG 53 48 0 07 Oct 2024
Dynamic Planning for LLM-based Graphical User Interface Automation Shaoqing Zhang Zhuosheng Zhang Kehai Chen Xinbei Ma Muyun Yang Tiejun Zhao Min Zhang LLMAG 16 2 0 01 Oct 2024
MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning Haotian Zhang Mingfei Gao Zhe Gan Philipp Dufter Nina Wenzel ... Haoxuan You Zirui Wang Afshin Dehghan Peter Grasch Yinfei Yang VLM MLLM 36 32 1 30 Sep 2024
VMAD: Visual-enhanced Multimodal Large Language Model for Zero-Shot Anomaly Detection Huilin Deng Hongchen Luo Wei Zhai Yang Cao Yu Kang 17 1 0 30 Sep 2024
Data Analysis in the Era of Generative AI J. Inala Chenglong Wang Steven Drucker Gonzalo Ramos Victor C. Dibia N. Riche Dave Brown Dan Marshall Jianfeng Gao 13 2 0 27 Sep 2024
MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understanding Qinzhuo Wu Weikai Xu Wei Liu Tao Tan Jianfeng Liu Ang Li Jian Luan Bin Wang Shuo Shang VLM 17 3 0 23 Sep 2024
MobileViews: A Large-Scale Mobile GUI Dataset Longxi Gao Li Zhang Shihe Wang Shangguang Wang Yuanchun Li Mengwei Xu 23 5 0 22 Sep 2024
A Survey on Multimodal Benchmarks: In the Era of Large AI Models Lin Li Guikun Chen Hanrong Shi Jun Xiao Long Chen 34 8 0 21 Sep 2024