WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics

30 January 2023

Papers citing "WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics"

36 / 36 papers shown

Title
Leveraging Vision-Language Models for Visual Grounding and Analysis of Automotive UI Benjamin Raphael Ernhofer Daniil Prokhorov Jannica Langner Dominik Bollmann 39 0 0 09 May 2025
Benchmarking Vision, Language, & Action Models in Procedurally Generated, Open Ended Action Environments Pranav Guruprasad Yangyue Wang Sudipta Chowdhury Harshvardhan Sikka LM&Ro VLM 156 0 0 08 May 2025
Multimodal graph representation learning for website generation based on visual sketch Tung D. Vu Chung Hoang Truong-Son Hy 3DV 56 0 0 25 Apr 2025
PixelWeb: The First Web GUI Dataset with Pixel-Wise Labels Qi Yang Weichen Bi Haiyang Shen Y. Guo Yun Ma 32 0 0 23 Apr 2025
LayoutCoT: Unleashing the Deep Reasoning Potential of Large Language Models for Layout Generation Hengyu Shi Junhao Su Huansheng Ning Xiaoming Wei Jialin Gao 3DV AI4TS LRM 52 0 0 15 Apr 2025
Explorer: Robust Collection of Interactable GUI Elements Iason Chaimalas Arnas Vyšniauskas Gabriel Brostow 26 0 0 12 Apr 2025
UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction Shravan Nayak Xiangru Jian Kevin Qinghong Lin Juan A. Rodriguez Montek Kalsi ... David Vazquez Christopher Pal Perouz Taslakian Spandana Gella Sai Rajeswar 189 0 0 19 Mar 2025
ShowUI: One Vision-Language-Action Model for GUI Visual Agent Kevin Qinghong Lin Linjie Li Difei Gao Z. Yang Shiwei Wu Zechen Bai Weixian Lei Lijuan Wang Mike Zheng Shou LLMAG 74 13 0 26 Nov 2024
Foundations and Recent Trends in Multimodal Mobile Agents: A Survey Biao Wu Yanda Li Meng Fang Zirui Song Zhiwei Zhang Yunchao Wei L. Chen LM&Ro LLMAG OffRL AI4TS 44 4 0 04 Nov 2024
Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms Zhangheng Li Keen You H. Zhang Di Feng Harsh Agrawal Xiujun Li Mohana Prasad Sathya Moorthy Jeff Nichols Y. Yang Zhe Gan MLLM 57 18 0 24 Oct 2024
Leopard: A Vision Language Model For Text-Rich Multi-Image Tasks Mengzhao Jia Wenhao Yu Kaixin Ma Tianqing Fang Zhihan Zhang Siru Ouyang Hongming Zhang Meng-Long Jiang Dong Yu VLM 29 5 0 02 Oct 2024
DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation Yi-Hao Peng Faria Huq Yue Jiang Jason Wu Amanda Li Jeffrey P. Bigham Amy Pavel DiffM 27 4 0 30 Sep 2024
The Impact of Element Ordering on LM Agent Performance Wayne Chi Ameet Talwalkar Chris Donahue 22 2 0 18 Sep 2024
UICrit: Enhancing Automated Design Evaluation with a UICritique Dataset Peitong Duan Chin-yi Chen Gang Li Bjoern Hartmann Yang Li 53 7 0 11 Jul 2024
Flowy: Supporting UX Design Decisions Through AI-Driven Pattern Annotation in Multi-Screen User Flows Yuwen Lu Ziang Tong Qinyi Zhao Yewon Oh Bryan Wang Toby Jia-Jun Li 44 6 0 23 Jun 2024
UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback Jason Wu E. Schoop Alan Leung Titus Barik Jeffrey P. Bigham Jeffrey Nichols 19 12 0 11 Jun 2024
MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI Modeling Sidong Feng Suyu Ma Han Wang David Kong Chunyang Chen 34 9 0 11 May 2024
Automating the Enterprise with Foundation Models Michael Wornow A. Narayan Krista Opsahl-Ong Quinn McIntyre Nigam H. Shah Christopher Ré AI4CE 36 9 0 03 May 2024
UIClip: A Data-driven Model for Assessing User Interface Design Jason Wu Yi-Hao Peng Amanda Li Amanda Swearngin Jeffrey P. Bigham Jeffrey Nichols HAI 21 6 0 18 Apr 2024
LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Automation Task Evaluation Li Lyna Zhang Shihe Wang Xianqing Jia Zhihan Zheng Yun-Yu Yan Longxi Gao Yuanchun Li Mengwei Xu LLMAG 30 10 0 12 Apr 2024
Computer User Interface Understanding. A New Dataset and a Learning Framework Andrés Munoz Daniel Borrajo 30 0 0 15 Mar 2024
AQuA: Automated Question-Answering in Software Tutorial Videos with Visual Anchors Saelyne Yang Jo Vermeulen G. Fitzmaurice Justin Matejka 16 5 0 08 Mar 2024
AI Assistance for UX: A Literature Review Through Human-Centered AI Yuwen Lu Yuewen Yang Qinyi Zhao Chengzhi Zhang Toby Jia-Jun Li 16 16 0 08 Feb 2024
WebLINX: Real-World Website Navigation with Multi-Turn Dialogue Xing Han Lù Zdeněk Kasner Siva Reddy 30 59 0 08 Feb 2024
Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automation Sunjae Lee Junyoung Choi Jungjae Lee Munim Hasan Wasi Hojun Choi Steven Y. Ko Sangeun Oh Insik Shin RALM 34 24 0 04 Dec 2023
Interactively Optimizing Layout Transfer for Vector Graphics Jeremy Warner Shuyao Zhou Björn Hartmann 30 1 0 20 Sep 2023
Natural Language Dataset Generation Framework for Visualizations Powered by Large Language Models Hyung-Kwon Ko Hyeon Jeon Gwanmo Park Dae Hyun Kim Nam Wook Kim Juho Kim Jinwook Seo 13 21 0 19 Sep 2023
Never-ending Learning of User Interfaces Jason Wu Rebecca Krosnick E. Schoop Amanda Swearngin Jeffrey P. Bigham Jeffrey Nichols VLM HAI 11 15 0 17 Aug 2023
MultiZoo & MultiBench: A Standardized Toolkit for Multimodal Deep Learning Paul Pu Liang Yiwei Lyu Xiang Fan Arav Agarwal Yun Cheng Louis-Philippe Morency Ruslan Salakhutdinov VLM 31 6 0 28 Jun 2023
ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshots Yu-Chung Hsiao Fedir Zubach Maria Wang Jindong Chen Victor Carbune Jason Lin Maria Wang Yun Zhu Jindong Chen RALM 157 25 0 16 Sep 2022
Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots Jason Wu Xiaoyi Zhang Jeffrey Nichols Jeffrey P. Bigham 3DV 163 71 0 17 Sep 2021
Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning Bryan Wang Gang Li Xin Zhou Zhourong Chen Tovi Grossman Yang Li 164 152 0 07 Aug 2021
Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels Xiaoyi Zhang Lilian de Greef Amanda Swearngin Samuel White Kyle I. Murray ... Jeffrey Nichols Jason Wu Chris Fleizach Aaron Everitt Jeffrey P. Bigham 188 167 0 13 Jan 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 253 1,989 0 31 Dec 2020
Domain-Adversarial Training of Neural Networks Yaroslav Ganin E. Ustinova Hana Ajakan Pascal Germain Hugo Larochelle François Laviolette M. Marchand Victor Lempitsky GAN OOD 177 9,327 0 28 May 2015
SMOTE: Synthetic Minority Over-sampling Technique Nitesh V. Chawla Kevin W. Bowyer Lawrence Hall W. Kegelmeyer AI4TS 163 25,247 0 09 Jun 2011