Modular Visual Question Answering via Code Generation

8 June 2023

Papers citing "Modular Visual Question Answering via Code Generation"

42 / 42 papers shown

Title
Resource-efficient Inference with Foundation Model Programs Lunyiu Nie Zhimin Ding Kevin Yu Marco Cheung C. Jermaine S. Chaudhuri 30 0 0 09 Apr 2025
ENTER: Event Based Interpretable Reasoning for VideoQA Hammad A. Ayyubi Junzhang Liu Ali Asgarov Zaber Ibn Abdul Hakim Najibul Haque Sarker ... Md. Atabuzzaman Xudong Lin Naveen Reddy Dyava Shih-Fu Chang Chris Thomas NAI 147 2 0 24 Jan 2025
Generalist Virtual Agents: A Survey on Autonomous Agents Across Digital Platforms Minghe Gao Wendong Bu Bingchen Miao Yang Wu Yunfei Li Juncheng Billy Li Siliang Tang Qi Wu Yueting Zhuang Meng Wang LM&Ro 45 3 0 17 Nov 2024
Natural Language Inference Improves Compositionality in Vision-Language Models Paola Cascante-Bonilla Yu Hou Yang Trista Cao Hal Daumé III Rachel Rudinger ReLM CoGe VLM 52 3 0 29 Oct 2024
FISHNET: Financial Intelligence from Sub-querying, Harmonizing, Neural-Conditioning, Expert Swarms, and Task Planning Nicole Cho Nishan Srishankar Lucas Cecchi William Watson AIFin 32 1 0 25 Oct 2024
HumanEval-V: Evaluating Visual Understanding and Reasoning Abilities of Large Multimodal Models Through Coding Tasks Fengji Zhang Linquan Wu Huiyu Bai Guancheng Lin Xiao Li Xiao Yu Yue Wang Bei Chen Jacky Keung MLLM ELM LRM 32 0 0 16 Oct 2024
VoxelPrompt: A Vision-Language Agent for Grounded Medical Image Analysis Andrew Hoopes V. Butoi John Guttag Adrian V. Dalca MedIm LM&MA 35 1 0 10 Oct 2024
Grounding Language in Multi-Perspective Referential Communication Zineng Tang Lingjun Mao Alane Suhr 21 2 0 04 Oct 2024
Discovering Object Attributes by Prompting Large Language Models with Perception-Action APIs A. Mavrogiannis Dehao Yuan Yiannis Aloimonos LM&Ro 43 0 0 23 Sep 2024
Pyramid Coder: Hierarchical Code Generator for Compositional Visual Question Answering Ruoyue Shen Nakamasa Inoue Koichi Shinoda 26 1 0 30 Jul 2024
AdaCoder: Adaptive Prompt Compression for Programmatic Visual Question Answering Mahiro Ukai Shuhei Kurita Atsushi Hashimoto Yoshitaka Ushiku Nakamasa Inoue 18 0 0 28 Jul 2024
Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awareness Khyathi Raghavi Chandu Linjie Li Anas Awadalla Ximing Lu Jae Sung Park Jack Hessel Lijuan Wang Yejin Choi 47 2 0 02 Jul 2024
From the Least to the Most: Building a Plug-and-Play Visual Reasoner via Data Synthesis Chuanqi Cheng Jian-Yu Guan Wei Wu Rui Yan LRM 45 10 0 28 Jun 2024
Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning Brandon Huang Chancharik Mitra Assaf Arbelle Leonid Karlinsky Trevor Darrell Roei Herzig 41 12 0 21 Jun 2024
Automatic benchmarking of large multimodal models via iterative experiment programming Alessandro Conti Enrico Fini Paolo Rota Yiming Wang Massimiliano Mancini Elisa Ricci 43 0 0 18 Jun 2024
Real2Code: Reconstruct Articulated Objects via Code Generation Zhao Mandi Yijia Weng Dominik Bauer Shuran Song 45 16 0 12 Jun 2024
VQA Training Sets are Self-play Environments for Generating Few-shot Pools Tautvydas Misiunas Hassan Mansoor Jasper Uijlings Oriana Riva Victor Carbune LRM VLM 35 0 0 30 May 2024
LLM-based Hierarchical Concept Decomposition for Interpretable Fine-Grained Image Classification Renyi Qu Mark Yatskar 24 1 0 29 May 2024
MoReVQA: Exploring Modular Reasoning Models for Video Question Answering Juhong Min Shyamal Buch Arsha Nagrani Minsu Cho Cordelia Schmid LRM 44 20 0 09 Apr 2024
Self-Training Large Language Models for Improved Visual Program Synthesis With Visual Reinforcement Zaid Khan B. Vijaykumar S. Schulter Yun Fu Manmohan Chandraker LRM ReLM 31 6 0 06 Apr 2024
PropTest: Automatic Property Testing for Improved Visual Programming Jaywon Koo Ziyan Yang Paola Cascante-Bonilla Baishakhi Ray Vicente Ordonez LRM 29 2 0 25 Mar 2024
FlowMind: Automatic Workflow Generation with LLMs Zhen Zeng William Watson Nicole Cho Saba Rahimi Shayleen Reynolds T. Balch Manuela Veloso 39 26 0 17 Mar 2024
Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoning Tejas Srinivasan Jack Hessel Tanmay Gupta Bill Yuchen Lin Yejin Choi Jesse Thomason Khyathi Raghavi Chandu 24 7 0 23 Feb 2024
Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models Siddharth Karamcheti Suraj Nair Ashwin Balakrishna Percy Liang Thomas Kollar Dorsa Sadigh MLLM VLM 57 98 0 12 Feb 2024
TroVE: Inducing Verifiable and Efficient Toolboxes for Solving Programmatic Tasks Zhiruo Wang Daniel Fried Graham Neubig 25 19 0 23 Jan 2024
Prompting Large Vision-Language Models for Compositional Reasoning Timothy Ossowski Ming Jiang Junjie Hu CoGe VLM LRM 43 3 0 20 Jan 2024
Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as Programmers Aleksandar Stanić Sergi Caelles Michael Tschannen LRM VLM 27 9 0 03 Jan 2024
If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents Ke Yang Jiateng Liu John Wu Chaoqi Yang Yi Ren Fung ... Xu Cao Xingyao Wang Yiquan Wang Heng Ji Chengxiang Zhai LLMAG ELM 26 73 0 01 Jan 2024
Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models Yushi Hu Otilia Stretcu Chun-Ta Lu Krishnamurthy Viswanathan Kenji Hata Enming Luo Ranjay Krishna Ariel Fuxman VLM LRM MLLM 44 29 0 05 Dec 2023
Recursive Visual Programming Jiaxin Ge Sanjay Subramanian Baifeng Shi Roei Herzig Trevor Darrell 29 4 0 04 Dec 2023
Zero-Shot Video Question Answering with Procedural Programs Rohan Choudhury Koichiro Niinuma Kris M. Kitani László A. Jeni 19 21 0 01 Dec 2023
Compositional Chain-of-Thought Prompting for Large Multimodal Models Chancharik Mitra Brandon Huang Trevor Darrell Roei Herzig MLLM LRM 33 80 0 27 Nov 2023
Analyzing Modular Approaches for Visual Question Decomposition Apoorv Khandelwal Ellie Pavlick Chen Sun 45 4 0 10 Nov 2023
Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Models Gabriel H. Sarch Yue Wu Michael J. Tarr Katerina Fragkiadaki LM&Ro LLMAG 21 19 0 23 Oct 2023
API-Assisted Code Generation for Question Answering on Varied Table Structures Yihan Cao Shuyi Chen Ryan Liu Zhiruo Wang Daniel Fried LMTD 25 10 0 23 Oct 2023
ViperGPT: Visual Inference via Python Execution for Reasoning Dídac Surís Sachit Menon Carl Vondrick MLLM LRM ReLM 45 431 0 14 Mar 2023
Binding Language Models in Symbolic Languages Zhoujun Cheng Tianbao Xie Peng Shi Chengzu Li Rahul Nadkarni ... Dragomir R. Radev Mari Ostendorf Luke Zettlemoyer Noah A. Smith Tao Yu LMTD 119 198 0 06 Oct 2022
Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners Zhenhailong Wang Manling Li Ruochen Xu Luowei Zhou Jie Lei ... Chenguang Zhu Derek Hoiem Shih-Fu Chang Joey Tianyi Zhou Heng Ji MLLM VLM 170 137 0 22 May 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 313 11,953 0 04 Mar 2022
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation Junnan Li Dongxu Li Caiming Xiong S. Hoi MLLM BDL VLM CLIP 392 4,137 0 28 Jan 2022
COVR: A test-bed for Visually Grounded Compositional Generalization with real images Ben Bogin Shivanshu Gupta Matt Gardner Jonathan Berant CoGe 34 29 0 22 Sep 2021
An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA Zhengyuan Yang Zhe Gan Jianfeng Wang Xiaowei Hu Yumao Lu Zicheng Liu Lijuan Wang 174 402 0 10 Sep 2021