GPT-4V(ision) is a Generalist Web Agent, if Grounded

3 January 2024

Huan Sun

Papers citing "GPT-4V(ision) is a Generalist Web Agent, if Grounded"

50 / 52 papers shown

Title
VLM Q-Learning: Aligning Vision-Language Models for Interactive Decision-Making Jake Grigsby Yuke Zhu Michael S Ryoo Juan Carlos Niebles OffRL VLM 31 0 0 06 May 2025
ScaleTrack: Scaling and back-tracking Automated GUI Agents Jing Huang Zhixiong Zeng WenKang Han Yufeng Zhong Liming Zheng Shuai Fu Jingyuan Chen Lin Ma 42 0 0 01 May 2025
Visual Test-time Scaling for GUI Agent Grounding Tiange Luo Lajanugen Logeswaran Justin Johnson Honglak Lee 48 0 0 01 May 2025
AndroidGen: Building an Android Language Agent under Data Scarcity Hanyu Lai Junjie Gao Xiao-Yang Liu Y. Xu S. Zhang Yuxiao Dong Jie Tang LLMAG 72 0 0 27 Apr 2025
Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks Yixin Cao Shibo Hong X. Li Jiahao Ying Yubo Ma ... Juanzi Li Aixin Sun Xuanjing Huang Tat-Seng Chua Yu Jiang ALM ELM 84 0 0 26 Apr 2025
NoEsis: Differentially Private Knowledge Transfer in Modular LLM Adaptation Rob Romijnders Stefanos Laskaridis Ali Shahin Shamsabadi Hamed Haddadi 54 0 0 25 Apr 2025
Toward a Human-Centered Evaluation Framework for Trustworthy LLM-Powered GUI Agents C. L. P. Chen Zhiping Zhang Ibrahim Khalilov Bingcan Guo Simret Araya Gebreegziabher Yanfang Ye Ziang Xiao Yaxing Yao Tianshi Li T. Li LLMAG ELM 87 0 0 24 Apr 2025
WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks Ivan Evtimov Arman Zharmagambetov Aaron Grattafiori Chuan Guo Kamalika Chaudhuri AAML 33 0 0 22 Apr 2025
UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesis Xinyi Liu Xiaoyi Zhang Ziyun Zhang Yan Lu 32 0 0 15 Apr 2025
Breaking the Data Barrier -- Building GUI Agents Through Task Generalization Junlei Zhang Zichen Ding Chang Ma Zijie Chen Qiushi Sun Zhenzhong Lan Junxian He 39 0 0 14 Apr 2025
An Illusion of Progress? Assessing the Current State of Web Agents Tianci Xue Weijian Qi Tianneng Shi Chan Hee Song Boyu Gou D. Song Huan Sun Yu Su LLMAG ELM 81 4 1 02 Apr 2025
A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Models Liangbo Ning Ziran Liang Zhuohang Jiang Haohao Qu Yujuan Ding ... Xiao Wei Shanru Lin Hui Liu Philip S. Yu Qing Li LLMAG LM&Ro 78 5 0 30 Mar 2025
UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction Shravan Nayak Xiangru Jian Kevin Qinghong Lin Juan A. Rodriguez Montek Kalsi ... David Vazquez Christopher Pal Perouz Taslakian Spandana Gella Sai Rajeswar 79 0 0 19 Mar 2025
A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrieval Yu Zhang Shutong Qiao Jiaqi Zhang Tzu-Heng Lin Chen Gao Y. Li LM&Ro LM&MA 76 0 0 07 Mar 2025
SpiritSight Agent: Advanced GUI Agent with One Look Zhiyuan Huang Ziming Cheng Junting Pan Zhaohui Hou Mingjie Zhan LLMAG 83 2 0 05 Mar 2025
A Practical Memory Injection Attack against LLM Agents Shen Dong Shaocheng Xu Pengfei He Y. Li Jiliang Tang Tianming Liu Hui Liu Zhen Xiang LLMAG AAML 41 2 0 05 Mar 2025
Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agents Vardaan Pahuja Yadong Lu Corby Rosset Boyu Gou Arindam Mitra Spencer Whitehead Yu Su Ahmed Awadallah LLMAG LM&Ro Presented at ResearchTrend Connect \| LLMAG on 14 Mar 2025 145 3 1 20 Feb 2025
Magma: A Foundation Model for Multimodal AI Agents Jianwei Yang Reuben Tan Qianhui Wu Ruijie Zheng Baolin Peng ... Seonghyeon Ye Joel Jang Yuquan Deng Lars Liden Jianfeng Gao VLM AI4TS 98 8 0 18 Feb 2025
AgentStudio: A Toolkit for Building General Virtual Agents Longtao Zheng Zhiyuan Huang Zhenghai Xue Xinrun Wang Bo An Shuicheng Yan 75 14 0 17 Feb 2025
WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning Zehan Qi Xiao-Chang Liu Iat Long Iong Hanyu Lai X. Sun ... Shuntian Yao Tianjie Zhang Wei Xu J. Tang Yuxiao Dong 93 14 0 28 Jan 2025
Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks Zhenhailong Wang Haiyang Xu Junyang Wang Xi Zhang Ming Yan J. Zhang Fei Huang Heng Ji 41 9 0 20 Jan 2025
The BrowserGym Ecosystem for Web Agent Research Thibault Le Sellier De Chezelles Maxime Gasse Alexandre Lacoste Alexandre Drouin Massimo Caccia ... Siva Reddy Quentin Cappart Graham Neubig Ruslan Salakhutdinov Nicolas Chapados LLMAG 96 9 0 06 Dec 2024
GUI Agents with Foundation Models: A Comprehensive Survey Shuai Wang W. Liu Jingxuan Chen Weinan Gan Xingshan Zeng ... Bin Wang Chuhan Wu Yasheng Wang Ruiming Tang Jianye Hao LLMAG 61 12 0 07 Nov 2024
Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms Zhangheng Li Keen You H. Zhang Di Feng Harsh Agrawal Xiujun Li Mohana Prasad Sathya Moorthy Jeff Nichols Y. Yang Zhe Gan MLLM 40 18 0 24 Oct 2024
VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Tasks Lawrence Jang Yinheng Li Charles Ding Justin Lin Paul Pu Liang Dan Zhao Rogerio Bonatti K. Koishida 28 5 0 24 Oct 2024
Lightweight Neural App Control Filippos Christianos Georgios Papoudakis Thomas Coste Jianye Hao Jun Wang Kun Shao LM&Ro 41 4 0 23 Oct 2024
Large Language Models Empowered Personalized Web Agents Hongru Cai Yongqi Li W. Wang Fengbin Zhu Xiaoyu Shen Wenjie Li Tat-Seng Chua LLMAG 41 12 0 22 Oct 2024
Beyond Browsing: API-Based Web Agents Yueqi Song Frank F. Xu Shuyan Zhou Graham Neubig 37 13 0 21 Oct 2024
SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation Jingxuan Chen Derek Yuen Bin Xie Y. Yang Gongwei Chen ... Liqiang Nie Yasheng Wang Jianye Hao Jun Wang Kun Shao LLMAG 31 5 0 19 Oct 2024
Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation Hyungjoo Chae Namyoung Kim Kai Tzu-iunn Ong Minju Gwak Gwanwoo Song Jihoon Kim S. Kim Dongha Lee Jinyoung Yeo LLMAG 30 13 0 17 Oct 2024
From Grounding to Planning: Benchmarking Bottlenecks in Web Agents Segev Shlomov Ben wiesel Aviad Sela Ido Levy Liane Galanti Roy Abitbol LLMAG 30 3 0 03 Sep 2024
GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning Zhen Xiang Linzhi Zheng Yanjie Li Junyuan Hong Qinbin Li ... Zidi Xiong Chulin Xie Carl Yang Dawn Song Bo Li LLMAG 45 22 0 13 Jun 2024
Open-Endedness is Essential for Artificial Superhuman Intelligence Edward Hughes Michael Dennis Jack Parker-Holder Feryal M. P. Behbahani Aditi Mavalankar Yuge Shi Tom Schaul Tim Rocktaschel LRM 32 18 0 06 Jun 2024
AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents Christopher Rawles Sarah Clinckemaillie Yifan Chang Jonathan Waltz Gabrielle Lau ... Daniel Toyama Robert Berry Divya Tyamagundlu Timothy Lillicrap Oriana Riva LLMAG 55 44 0 23 May 2024
A Multimodal Automated Interpretability Agent Tamar Rott Shaham Sarah Schwettmann Franklin Wang Achyuta Rajaram Evan Hernandez Jacob Andreas Antonio Torralba 16 17 0 22 Apr 2024
MMInA: Benchmarking Multihop Multimodal Internet Agents Ziniu Zhang Shulin Tian Liangyu Chen Ziwei Liu LLMAG LM&Ro 27 13 0 15 Apr 2024
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs Keen You Haotian Zhang E. Schoop Floris Weers Amanda Swearngin Jeffrey Nichols Yinfei Yang Zhe Gan MLLM 39 82 0 08 Apr 2024
Tur[k]ingBench: A Challenge Benchmark for Web Agents Kevin Xu Yeganeh Kordi Kate Sanders Yizhong Wang Adam Byerly Kate Sanders Adam Byerly Jingyu Zhang Benjamin Van Durme Daniel Khashabi LLMAG 60 6 0 18 Mar 2024
Android in the Zoo: Chain-of-Action-Thought for GUI Agents Jiwen Zhang Jihao Wu Yihua Teng Minghui Liao Nuo Xu Xiao Xiao Zhongyu Wei Duyu Tang LLMAG LM&Ro 27 50 0 05 Mar 2024
Towards Unified Alignment Between Agents, Humans, and Environment Zonghan Yang An Liu Zijun Liu Kai Liu Fangzhou Xiong ... Zhenhe Zhang Fuwen Luo Zhicheng Guo Peng Li Yang Liu 24 4 0 12 Feb 2024
WebLINX: Real-World Website Navigation with Multi-Turn Dialogue Xing Han Lù Zdeněk Kasner Siva Reddy 19 59 0 08 Feb 2024
Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception Junyang Wang Haiyang Xu Jiabo Ye Mingshi Yan Weizhou Shen Ji Zhang Fei Huang Jitao Sang 10 103 0 29 Jan 2024
SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents Kanzhi Cheng Qiushi Sun Yougang Chu Fangzhi Xu Yantao Li Jianbing Zhang Zhiyong Wu LLMAG 162 137 0 17 Jan 2024
CogAgent: A Visual Language Model for GUI Agents Wenyi Hong Weihan Wang Qingsong Lv Jiazheng Xu Wenmeng Yu ... Juanzi Li Bin Xu Yuxiao Dong Ming Ding Jie Tang MLLM 137 310 0 14 Dec 2023
Instruction Tuning with GPT-4 Baolin Peng Chunyuan Li Pengcheng He Michel Galley Jianfeng Gao SyDa ALM LM&MA 157 576 0 06 Apr 2023
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models Junnan Li Dongxu Li Silvio Savarese Steven C. H. Hoi VLM MLLM 244 4,186 0 30 Jan 2023
Understanding HTML with Large Language Models Izzeddin Gur Ofir Nachum Yingjie Miao Mustafa Safdari Austin Huang Aakanksha Chowdhery Sharan Narang Noah Fiedel Aleksandra Faust AI4CE 124 69 0 08 Oct 2022
Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding Kenton Lee Mandar Joshi Iulia Turc Hexiang Hu Fangyu Liu Julian Martin Eisenschlos Urvashi Khandelwal Peter Shaw Ming-Wei Chang Kristina Toutanova CLIP VLM 148 259 0 07 Oct 2022
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering Pan Lu Swaroop Mishra Tony Xia Liang Qiu Kai-Wei Chang Song-Chun Zhu Oyvind Tafjord Peter Clark A. Kalyan ELM ReLM LRM 198 1,089 0 20 Sep 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 301 11,730 0 04 Mar 2022