ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine
Conversations

ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations

7 October 2023

Amanda Swearngin

Jeffrey Nichols

Papers citing "ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations"

9 / 9 papers shown

Title
A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Models Liangbo Ning Ziran Liang Zhuohang Jiang Haohao Qu Yujuan Ding ... Xiao Wei Shanru Lin Hui Liu Philip S. Yu Qing Li LLMAG LM&Ro 91 6 0 30 Mar 2025
GUI Agents with Foundation Models: A Comprehensive Survey Shuai Wang W. Liu Jingxuan Chen Weinan Gan Xingshan Zeng ... Bin Wang Chuhan Wu Yasheng Wang Ruiming Tang Jianye Hao LLMAG 68 13 0 07 Nov 2024
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs Keen You Haotian Zhang E. Schoop Floris Weers Amanda Swearngin Jeffrey Nichols Yinfei Yang Zhe Gan MLLM 45 82 0 08 Apr 2024
Instruction Tuning with GPT-4 Baolin Peng Chunyuan Li Pengcheng He Michel Galley Jianfeng Gao SyDa ALM LM&MA 159 579 0 06 Apr 2023
Enabling Conversational Interaction with Mobile UI using Large Language Models Bryan Wang Gang Li Yang Li 175 132 0 18 Sep 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 311 11,915 0 04 Mar 2022
ReverseORC: Reverse Engineering of Resizable User Interface Layouts with OR-Constraints Yue Jiang W. Stuerzlinger C. Lutteroth 16 18 0 23 Feb 2022
Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots Jason Wu Xiaoyi Zhang Jeffrey Nichols Jeffrey P. Bigham 3DV 163 71 0 17 Sep 2021
Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels Xiaoyi Zhang Lilian de Greef Amanda Swearngin Samuel White Kyle I. Murray ... Jeffrey Nichols Jason Wu Chris Fleizach Aaron Everitt Jeffrey P. Bigham 188 167 0 13 Jan 2021