Predicting and Explaining Mobile UI Tappability with Vision Modeling and
Saliency Analysis

Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis

5 April 2022

Björn Hartmann

Papers citing "Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis"

16 / 16 papers shown

Title
MP-GUI: Modality Perception with MLLMs for GUI Understanding Ziwei Wang Weizhi Chen Leyang Yang Sheng Zhou Shengchu Zhao Hanbei Zhan Jiongchao Jin Liangcheng Li Zirui Shao Jiajun Bu 60 1 0 18 Mar 2025
DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation Yi-Hao Peng Faria Huq Yue Jiang Jason Wu Amanda Li Jeffrey P. Bigham Amy Pavel DiffM 25 4 0 30 Sep 2024
MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning Haotian Zhang Mingfei Gao Zhe Gan Philipp Dufter Nina Wenzel ... Haoxuan You Zirui Wang Afshin Dehghan Peter Grasch Yinfei Yang VLM MLLM 38 32 1 30 Sep 2024
Tell Me What's Next: Textual Foresight for Generic UI Representations Andrea Burns Kate Saenko Bryan A. Plummer LM&Ro AI4TS 31 4 0 12 Jun 2024
UIClip: A Data-driven Model for Assessing User Interface Design Jason Wu Yi-Hao Peng Amanda Li Amanda Swearngin Jeffrey P. Bigham Jeffrey Nichols HAI 21 6 0 18 Apr 2024
Unblind Text Inputs: Predicting Hint-text of Text Input in Mobile Apps via LLM Zhe Liu Chunyang Chen Junjie Wang Mengzhuo Chen Boyu Wu Yuekai Huang Jun Hu Qing Wang 27 10 0 03 Apr 2024
Generating Automatic Feedback on UI Mockups with Large Language Models Peitong Duan Jeremy Warner Yang Li Bjoern Hartmann 41 19 0 19 Mar 2024
Tappy: Predicting Tap Accuracy of User-Interface Elements by Reverse-Engineering Webpage Structures Hiroki Usuba Junichi Sato Naomi Sasaya Shota Yamanaka Fumiya Yamashita 16 2 0 05 Mar 2024
AI Assistance for UX: A Literature Review Through Human-Centered AI Yuwen Lu Yuewen Yang Qinyi Zhao Chengzhi Zhang Toby Jia-Jun Li 11 16 0 08 Feb 2024
ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations Yue Jiang E. Schoop Amanda Swearngin Jeffrey Nichols MLLM 13 17 0 07 Oct 2023
Interactive Flexible Style Transfer for Vector Graphics Jeremy Warner Kyu Won Kim Bjoern Hartmann 11 9 0 20 Sep 2023
Never-ending Learning of User Interfaces Jason Wu Rebecca Krosnick E. Schoop Amanda Swearngin Jeffrey P. Bigham Jeffrey Nichols VLM HAI 11 15 0 17 Aug 2023
Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus Gang Li Yang Li 22 67 0 29 Sep 2022
Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots Jason Wu Xiaoyi Zhang Jeffrey Nichols Jeffrey P. Bigham 3DV 163 71 0 17 Sep 2021
Understanding Mobile GUI: from Pixel-Words to Screen-Sentences Jingwen Fu Xiaoyi Zhang Yuwang Wang Wenjun Zeng Sam Yang Grayson Hilliard 21 14 0 25 May 2021
Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels Xiaoyi Zhang Lilian de Greef Amanda Swearngin Samuel White Kyle I. Murray ... Jeffrey Nichols Jason Wu Chris Fleizach Aaron Everitt Jeffrey P. Bigham 185 167 0 13 Jan 2021