Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots

17 September 2021

Papers citing "Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots"

34 / 34 papers shown

Title
UISearch: Graph-Based Embeddings for Multimodal Enterprise UI Screenshots Retrieval Maroun Ayli Youssef Bakouny Tushar Sharma Nader Jalloul Hani Seifeddine Rima Kilany AI4TS 175 0 0 24 Nov 2025
UI-UG: A Unified MLLM for UI Understanding and Generation Hao Yang Weijie Qiu Ru Zhang Zhou Fang Ruichao Mao ... Maji Huang Longxiang Zhang Teng Guo Shuoyang Liu Hai Rao MLLM 124 0 0 29 Sep 2025
UISim: An Interactive Image-Based UI Simulator for Dynamic Mobile Environments Jiannan Xiang Yun Zhu Lei Shu Maria Wang Lijun Yu Gabriel Barcik James Lyon Srinivas Sunkara Jindong Chen 56 0 0 26 Sep 2025
WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generation Rabiul Awal Mahsa Massoud Aarash Feizi Zichao Li Suyuchen Wang ... Siva Reddy Juan A. Rodriguez Perouz Taslakian Spandana Gella Sai Rajeswar LRM 84 9 0 22 Aug 2025
ScreenCoder: Advancing Visual-to-Code Generation for Front-End Automation via Modular Multimodal Agents Yilei Jiang Y. Zheng Yuxuan Wan Jiaming Han Qunzhong Wang Michael R. Lyu Xiangyu Yue LLMAG 149 7 0 30 Jul 2025
ScreenAudit: Detecting Screen Reader Accessibility Errors in Mobile Apps Using Large Language ModelsInternational Conference on Human Factors in Computing Systems (CHI), 2025 Mingyuan Zhong Ruolin Chen Xia Chen James Fogarty Jacob O. Wobbrock 116 5 0 02 Apr 2025
Visual Prompting with Iterative Refinement for Design Critique Generation Peitong Duan Chin-Yi Cheng Bjoern Hartmann Yang Li 250 2 0 22 Dec 2024
AutoGameUI: Constructing High-Fidelity Game UIs via Multimodal Learning and Interactive Web-Based Tool Zhongliang Tang Mengchen Tan Fei Xia Qingrong Cheng Hao Jiang Yujiao Shi 124 0 0 06 Nov 2024
DreamStruct: Understanding Slides and User Interfaces via Synthetic Data GenerationEuropean Conference on Computer Vision (ECCV), 2024 Yi-Hao Peng Faria Huq Yue Jiang Jason Wu Amanda Li Jeffrey P. Bigham Amy Pavel DiffM 180 9 0 30 Sep 2024
Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approach Yuxuan Wan Chaozheng Wang Yi Dong Wenxuan Wang Shuqing Li Yintong Huo Michael R. Lyu 3DV 489 26 0 24 Jun 2024
CanvasPic: An Interactive Tool for Freely Generating Facial Images Based on Spatial Layout Jiafu Wei Chia-Ming Chang Xi Yang Takeo Igarashi 214 1 0 16 Apr 2024
MineXR: Mining Personalized Extended Reality InterfacesInternational Conference on Human Factors in Computing Systems (CHI), 2024 Hyunsung Cho Yukang Yan Kashyap Todi Mark Parent Missie Smith Tanya R. Jonker Hrvoje Benko David Lindlbauer 96 13 0 12 Mar 2024
WebLINX: Real-World Website Navigation with Multi-Turn Dialogue Xing Han Lù Zdeněk Kasner Siva Reddy 256 117 0 08 Feb 2024
ScreenAI: A Vision-Language Model for UI and Infographics Understanding Gilles Baechler Srinivas Sunkara Maria Wang Fedir Zubach Hassan Mansoor Vincent Etter Victor Carbune Jason Lin Jindong Chen Abhanshu Sharma 557 89 0 07 Feb 2024
UINav: A Practical Approach to Train On-Device Automation AgentsNorth American Chapter of the Association for Computational Linguistics (NAACL), 2023 Wei Li Fu-Lin Hsu Will Bishop Folawiyo Campbell-Ajala Max Lin Oriana Riva 417 4 0 15 Dec 2023
ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine ConversationsInternational Conference on Intelligent User Interfaces (IUI), 2023 Yue Jiang E. Schoop Amanda Swearngin Jeffrey Nichols MLLM 271 23 0 07 Oct 2023
Towards Automated Accessibility Report Generation for Mobile Apps Amanda Swearngin Jason Wu Xiaoyi Zhang Esteban Gomez Jen Coughenour ... Bhavya Garg Greg Hughes Adriana Hilliard Jeffrey P. Bigham Jeffrey Nichols 190 14 0 29 Sep 2023
Interactive Flexible Style Transfer for Vector GraphicsACM Symposium on User Interface Software and Technology (UIST), 2023 Jeremy Warner Kyu Won Kim Bjoern Hartmann 173 11 0 20 Sep 2023
Never-ending Learning of User InterfacesACM Symposium on User Interface Software and Technology (UIST), 2023 Jason Wu Rebecca Krosnick E. Schoop Amanda Swearngin Jeffrey P. Bigham Jeffrey Nichols VLM HAI 108 21 0 17 Aug 2023
TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIsIntelligent Computing (IC), 2023 Yaobo Liang Chenfei Wu Ting Song Wenshan Wu Yan Xia ... Shaoguang Mao Yuntao Wang Linjun Shou Ming Gong Nan Duan LLMAG CLL 208 232 0 29 Mar 2023
AidUI: Toward Automated Recognition of Dark Patterns in User InterfacesInternational Conference on Software Engineering (ICSE), 2023 S M Hasan Mansur Sabiha Salma Damilola Awofisayo Kevin Moran 193 34 0 12 Mar 2023
WebUI: A Dataset for Enhancing Visual UI Understanding with Web SemanticsInternational Conference on Human Factors in Computing Systems (CHI), 2023 Jason Wu Siyan Wang Siman Shen Yi-Hao Peng Jeffrey Nichols Jeffrey P. Bigham 199 85 0 30 Jan 2023
Screen Correspondence: Mapping Interchangeable Elements between UIs Jason Wu Amanda Swearngin Xiaoyi Zhang Jeffrey Nichols Jeffrey P. Bigham 169 8 0 20 Jan 2023
SpaceEditing: Integrating Human Knowledge into Deep Neural Networks via Interactive Latent Space Editing Jiafu Wei Ding Xia H. Xie Chia-Ming Chang Chuntao Li Xi Yang KELM 143 1 0 08 Dec 2022
Pix2Struct: Screenshot Parsing as Pretraining for Visual Language UnderstandingInternational Conference on Machine Learning (ICML), 2022 Kenton Lee Mandar Joshi Iulia Turc Hexiang Hu Fangyu Liu Julian Martin Eisenschlos Urvashi Khandelwal Peter Shaw Ming-Wei Chang Kristina Toutanova CLIP VLM 637 355 0 07 Oct 2022
Spotlight: Mobile UI Understanding using Vision-Language Models with a FocusInternational Conference on Learning Representations (ICLR), 2022 Gang Li Yang Li 214 79 0 29 Sep 2022
Enabling Conversational Interaction with Mobile UI using Large Language ModelsInternational Conference on Human Factors in Computing Systems (CHI), 2022 Bryan Wang Gang Li Yang Li 356 168 0 18 Sep 2022
Reflow: Automatically Improving Touch Interactions in Mobile Applications through Pixel-based Refinements Jason Wu Titus Barik Xiaoyi Zhang Colin S. Lea Jeffrey Nichols Jeffrey P. Bigham 86 5 0 15 Jul 2022
Auto-Icon+: An Automated End-to-End Code Generation Tool for Icon Designs in UI Development Sidong Feng Minmin Jiang Tingting Zhou Yankun Zhen Chunyang Chen 147 22 0 19 Apr 2022
GreaseVision: Rewriting the Rules of the Interface Siddhartha Datta Konrad Kollnig N. Shadbolt 151 5 0 07 Apr 2022
Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency AnalysisInternational Conference on Human Factors in Computing Systems (CHI), 2022 E. Schoop Xin Zhou Gang Li Zhourong Chen Björn Hartmann Yang Li HAI FAtt 160 40 0 05 Apr 2022
GANSpiration: Balancing Targeted and Serendipitous Inspiration in User Interface Design with Style-Based Generative Adversarial NetworkInternational Conference on Human Factors in Computing Systems (CHI), 2022 Mohammad Amin Mozaffari Xinyuan Zhang Jinghui Cheng Jin L. C. Guo 165 31 0 08 Mar 2022
Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at ScaleInternational Conference on Human Factors in Computing Systems (CHI), 2022 Gang Li Gilles Baechler Manuel Tragut Yang Li 218 56 0 11 Jan 2022
Understanding Mobile GUI: from Pixel-Words to Screen-Sentences Jingwen Fu Xiaoyi Zhang Yuwang Wang Wenjun Zeng Sam Yang Grayson Hilliard 190 16 0 25 May 2021

All Papers

Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots

Papers citing "Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots"