VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modeling

10 December 2021

Papers citing "VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modeling"

24 / 24 papers shown

Title
MP-GUI: Modality Perception with MLLMs for GUI Understanding Ziwei Wang Weizhi Chen Leyang Yang Sheng Zhou Shengchu Zhao Hanbei Zhan Jiongchao Jin Liangcheng Li Zirui Shao Jiajun Bu 60 1 0 18 Mar 2025
GUI Agents with Foundation Models: A Comprehensive Survey Shuai Wang W. Liu Jingxuan Chen Weinan Gan Xingshan Zeng ... Bin Wang Chuhan Wu Yasheng Wang Ruiming Tang Jianye Hao LLMAG 68 12 0 07 Nov 2024
Foundations and Recent Trends in Multimodal Mobile Agents: A Survey Biao Wu Yanda Li Meng Fang Zirui Song Zhiwei Zhang Yunchao Wei L. Chen LM&Ro LLMAG OffRL AI4TS 39 4 0 04 Nov 2024
Inferring Alt-text For UI Icons With Large Language Models During App Development Sabrina Haque Christoph Csallner VLM 31 0 0 26 Sep 2024
MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understanding Qinzhuo Wu Weikai Xu Wei Liu Tao Tan Jianfeng Liu Ang Li Jian Luan Bin Wang Shuo Shang VLM 32 10 0 23 Sep 2024
Tur[k]ingBench: A Challenge Benchmark for Web Agents Kevin Xu Yeganeh Kordi Kate Sanders Yizhong Wang Adam Byerly Kate Sanders Adam Byerly Jingyu Zhang Benjamin Van Durme Daniel Khashabi LLMAG 67 6 0 18 Mar 2024
Computer User Interface Understanding. A New Dataset and a Learning Framework Andrés Munoz Daniel Borrajo 27 0 0 15 Mar 2024
OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web Raghav Kapoor Y. Butala M. Russak Jing Yu Koh Kiran Kamble Waseem Alshikh Ruslan Salakhutdinov LLMAG 51 44 0 27 Feb 2024
AI Assistance for UX: A Literature Review Through Human-Centered AI Yuwen Lu Yuewen Yang Qinyi Zhao Chengzhi Zhang Toby Jia-Jun Li 9 16 0 08 Feb 2024
ScreenAI: A Vision-Language Model for UI and Infographics Understanding Gilles Baechler Srinivas Sunkara Maria Wang Fedir Zubach Hassan Mansoor Vincent Etter Victor Carbune Jason Lin Jindong Chen Abhanshu Sharma 115 47 0 07 Feb 2024
SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents Kanzhi Cheng Qiushi Sun Yougang Chu Fangzhi Xu Yantao Li Jianbing Zhang Zhiyong Wu LLMAG 170 138 0 17 Jan 2024
AutoDroid: LLM-powered Task Automation in Android Hao Wen Yuanchun Li Guohong Liu Shanhui Zhao Tao Yu Toby Jia-Jun Li Shiqi Jiang Yunhao Liu Yaqin Zhang Yunxin Liu 37 74 0 29 Aug 2023
Referring to Screen Texts with Voice Assistants Shruti Bhargava Anand Dhoot I. Jonsson Hoang Long Nguyen Alkesh Patel Hong-ye Yu Vincent Renkens 8 2 0 10 Jun 2023
Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding Kenton Lee Mandar Joshi Iulia Turc Hexiang Hu Fangyu Liu Julian Martin Eisenschlos Urvashi Khandelwal Peter Shaw Ming-Wei Chang Kristina Toutanova CLIP VLM 158 262 0 07 Oct 2022
MUG: Interactive Multimodal Grounding on User Interfaces Tao Li Gang Li Jingjie Zheng Purple Wang Yang Li LLMAG 30 8 0 29 Sep 2022
Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus Gang Li Yang Li 22 66 0 29 Sep 2022
Enabling Conversational Interaction with Mobile UI using Large Language Models Bryan Wang Gang Li Yang Li 171 132 0 18 Sep 2022
Multimodal Conversational AI: A Survey of Datasets and Approaches Anirudh S. Sundar Larry Heck 30 29 0 13 May 2022
Do BERTs Learn to Use Browser User Interface? Exploring Multi-Step Tasks with Unified Vision-and-Language BERTs Taichi Iki Akiko Aizawa LLMAG 11 6 0 15 Mar 2022
A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility Andrea Burns Deniz Arsan Sanjna Agrawal Ranjitha Kumar Kate Saenko Bryan A. Plummer 34 59 0 04 Feb 2022
SCENIC: A JAX Library for Computer Vision Research and Beyond Mostafa Dehghani A. Gritsenko Anurag Arnab Matthias Minderer Yi Tay 41 67 0 18 Oct 2021
Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning Bryan Wang Gang Li Xin Zhou Zhourong Chen Tovi Grossman Yang Li 162 152 0 07 Aug 2021
Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels Xiaoyi Zhang Lilian de Greef Amanda Swearngin Samuel White Kyle I. Murray ... Jeffrey Nichols Jason Wu Chris Fleizach Aaron Everitt Jeffrey P. Bigham 171 166 0 13 Jan 2021
Unified Vision-Language Pre-Training for Image Captioning and VQA Luowei Zhou Hamid Palangi Lei Zhang Houdong Hu Jason J. Corso Jianfeng Gao MLLM VLM 250 926 0 24 Sep 2019