Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs

8 April 2024

Papers citing "Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs"

25 / 75 papers shown

Title
UI-JEPA: Towards Active Perception of User Intent through Onscreen User Activity Yicheng Fu R. Anantha Prabal Vashisht Jianpeng Cheng Etai Littwin 18 2 0 06 Sep 2024
A Survey on Evaluation of Multimodal Large Language Models Jiaxing Huang Jingyi Zhang LM&MA ELM LRM 43 20 0 28 Aug 2024
A Taxonomy of Architecture Options for Foundation Model-based Agents: Analysis and Decision Model Jingwen Zhou Qinghua Lu Jieshan Chen Liming Zhu Xiwei Xu Zhenchang Xing Stefan Harrer 31 0 0 06 Aug 2024
Caution for the Environment: Multimodal Agents are Susceptible to Environmental Distractions Xinbei Ma Yiting Wang Yao Yao Tongxin Yuan Aston Zhang Zhuosheng Zhang Hai Zhao AAML LLMAG 20 1 0 05 Aug 2024
MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities Weihao Yu Zhengyuan Yang Linfeng Ren Linjie Li Jianfeng Wang K. Lin Chung-Ching Lin Zicheng Liu Lijuan Wang Xinchao Wang VLM MLLM 23 17 0 01 Aug 2024
ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models Ming-Kuan Wu Xinyue Cai Jiayi Ji Jiale Li Oucheng Huang Gen Luo Hao Fei Xiaoshuai Sun Rongrong Ji MLLM 40 7 0 31 Jul 2024
MobileFlow: A Multimodal LLM For Mobile GUI Agent Songqin Nong Jiali Zhu Rui Wu Jiongchao Jin Shuo Shan Xiutian Huang Wenhao Xu 19 7 0 05 Jul 2024
MobileExperts: A Dynamic Tool-Enabled Agent Team in Mobile Devices Jiayi Zhang Chuang Zhao Yihan Zhao Zhaoyang Yu Ming He Jianping Fan LLMAG 18 1 0 04 Jul 2024
AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents Yuxiang Chai Siyuan Huang Yazhe Niu Han Xiao Liang Liu Dingyu Zhang Peng Gao Shuai Ren Hongsheng Li LLMAG 27 16 0 03 Jul 2024
MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs Yusu Qian Hanrong Ye J. Fauconnier Peter Grasch Yinfei Yang Zhe Gan 99 13 0 01 Jul 2024
Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding Yue Fan Lei Ding Ching-Chen Kuo Shan Jiang Yang Zhao Xinze Guan Jie Yang Yi Zhang Xin Eric Wang 31 10 0 27 Jun 2024
VideoGUI: A Benchmark for GUI Automation from Instructional Videos Kevin Qinghong Lin Linjie Li Difei Gao Qinchen Wu Mingyi Yan Zhengyuan Yang Lijuan Wang Mike Zheng Shou 34 10 0 14 Jun 2024
AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents Christopher Rawles Sarah Clinckemaillie Yifan Chang Jonathan Waltz Gabrielle Lau ... Daniel Toyama Robert Berry Divya Tyamagundlu Timothy Lillicrap Oriana Riva LLMAG 48 44 0 23 May 2024
GUing: A Mobile GUI Search Engine using a Vision-Language Model Jialiang Wei A. Courbis Thomas Lambolais Binbin Xu P. Bernard Gérard Dray Walid Maalej DiffM CLIP 19 1 0 30 Apr 2024
Android in the Zoo: Chain-of-Action-Thought for GUI Agents Jiwen Zhang Jihao Wu Yihua Teng Minghui Liao Nuo Xu Xiao Xiao Zhongyu Wei Duyu Tang LLMAG LM&Ro 14 50 0 05 Mar 2024
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models Chris Liu Renrui Zhang Longtian Qiu Siyuan Huang Weifeng Lin ... Hao Shao Pan Lu Hongsheng Li Yu Qiao Peng Gao MLLM 116 106 0 08 Feb 2024
ScreenAI: A Vision-Language Model for UI and Infographics Understanding Gilles Baechler Srinivas Sunkara Maria Wang Fedir Zubach Hassan Mansoor Vincent Etter Victor Carbune Jason Lin Jindong Chen Abhanshu Sharma 107 47 0 07 Feb 2024
Muffin or Chihuahua? Challenging Multimodal Large Language Models with Multipanel VQA Yue Fan Jing Gu KAI-QING Zhou Qianqi Yan Shan Jiang Ching-Chen Kuo Xinze Guan Xin Eric Wang 16 6 0 29 Jan 2024
SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents Kanzhi Cheng Qiushi Sun Yougang Chu Fangzhi Xu Yantao Li Jianbing Zhang Zhiyong Wu LLMAG 162 137 0 17 Jan 2024
CogAgent: A Visual Language Model for GUI Agents Wenyi Hong Weihan Wang Qingsong Lv Jiazheng Xu Wenmeng Yu ... Juanzi Li Bin Xu Yuxiao Dong Ming Ding Jie Tang MLLM 132 310 0 14 Dec 2023
LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models Hao Zhang Hongyang Li Feng Li Tianhe Ren Xueyan Zou ... Shijia Huang Jianfeng Gao Lei Zhang Chun-yue Li Jianwei Yang 87 68 0 05 Dec 2023
Multimodal Foundation Models: From Specialists to General-Purpose Assistants Chunyuan Li Zhe Gan Zhengyuan Yang Jianwei Yang Linjie Li Lijuan Wang Jianfeng Gao MLLM 105 221 0 18 Sep 2023
mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality Qinghao Ye Haiyang Xu Guohai Xu Jiabo Ye Ming Yan ... Junfeng Tian Qiang Qi Ji Zhang Feiyan Huang Jingren Zhou VLM MLLM 198 883 0 27 Apr 2023
Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding Kenton Lee Mandar Joshi Iulia Turc Hexiang Hu Fangyu Liu Julian Martin Eisenschlos Urvashi Khandelwal Peter Shaw Ming-Wei Chang Kristina Toutanova CLIP VLM 148 259 0 07 Oct 2022
Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels Xiaoyi Zhang Lilian de Greef Amanda Swearngin Samuel White Kyle I. Murray ... Jeffrey Nichols Jason Wu Chris Fleizach Aaron Everitt Jeffrey P. Bigham 147 163 0 13 Jan 2021