Interactive Task and Concept Learning from Natural Language Instructions
and GUI Demonstrations

Interactive Task and Concept Learning from Natural Language Instructions and GUI Demonstrations

30 August 2019

Toby Jia-Jun Li

Marissa Radensky

Kirielle Singarajah

Tom Michael Mitchell

Papers citing "Interactive Task and Concept Learning from Natural Language Instructions and GUI Demonstrations"

14 / 14 papers shown

Title
AI Assistance for UX: A Literature Review Through Human-Centered AI Yuwen Lu Yuewen Yang Qinyi Zhao Chengzhi Zhang Toby Jia-Jun Li 34 16 0 08 Feb 2024
DrawTalking: Building Interactive Worlds by Sketching and Speaking K. Rosenberg Rubaiat Habib Kazi Li-Yi Wei Haijun Xia Ken Perlin 27 8 0 11 Jan 2024
AutoTask: Executing Arbitrary Voice Commands by Exploring and Learning from Mobile GUI Lihang Pan Bowen Wang Chun Yu Yuxuan Chen Xiangyu Zhang Yuanchun Shi 47 3 0 26 Dec 2023
AI for Low-Code for AI Nikitha Rao Jason Tsay Kiran Kate Vincent J. Hellendoorn Martin Hirzel 18 15 0 31 May 2023
VISAR: A Human-AI Argumentative Writing Assistant with Visual Programming and Rapid Draft Prototyping Zheng Zhang Jie Ying Gao Ranjodh Singh Dhaliwal Toby Jia-Jun Li 33 87 0 16 Apr 2023
Enabling Conversational Interaction with Mobile UI using Large Language Models Bryan Wang Gang Li Yang Li 186 132 0 18 Sep 2022
META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI Liangtai Sun Xingyu Chen Lu Chen Tianle Dai Zichen Zhu Kai Yu LLMAG 26 51 0 23 May 2022
Multimodal Conversational AI: A Survey of Datasets and Approaches Anirudh S. Sundar Larry Heck 48 29 0 13 May 2022
StoryBuddy: A Human-AI Collaborative Chatbot for Parent-Child Interactive Storytelling with Flexible Parental Involvement Zheng Zhang Ying Xu Yanhao Wang Bingsheng Yao Daniel E. Ritchie Tongshuang Wu Mo Yu Dakuo Wang Toby Jia-Jun Li 21 116 0 13 Feb 2022
Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning Bryan Wang Gang Li Xin Zhou Zhourong Chen Tovi Grossman Yang Li 167 154 0 07 Aug 2021
To Reuse or Not To Reuse? A Framework and System for Evaluating Summarized Knowledge Michael Xieyang Liu A. Kittur Brad A. Myers 11 8 0 11 Feb 2021
Screen2Vec: Semantic Embedding of GUI Screens and GUI Components Toby Jia-Jun Li Lindsay Popowski Tom Michael Mitchell Brad A. Myers 11 104 0 11 Jan 2021
Geno: A Developer Tool for Authoring Multimodal Interaction on Existing Web Applications Ritam Jyoti Sarmah Yunpeng Ding Di Wang Cheuk Yin Phipson Lee Toby Jia-Jun Li Xiang Ánthony' Chen 25 19 0 19 Jul 2020
VASTA: A Vision and Language-assisted Smartphone Task Automation System A. R. Sereshkeh Gary Leung K. Perumal Caleb Phillips Minfan Zhang Afsaneh Fazly Iqbal Mohomed 22 41 0 04 Nov 2019