Improving Text-to-SQL Evaluation Methodology

23 June 2018

Catherine Finegan-Dollak

Jonathan K. Kummerfeld

Papers citing "Improving Text-to-SQL Evaluation Methodology"

50 / 175 papers shown

Title
Reinforcing Compositional Retrieval: Retrieving Step-by-Step for Composing Informative Contexts Quanyu Long Jianda Chen Zhengyuan Liu Nancy F. Chen Wenya Wang Sinno Jialin Pan KELM RALM LRM 123 0 0 15 Apr 2025
Learning to Substitute Components for Compositional Generalization Z. Li Gangwei Jiang Chenwang Wu Ying Wei Defu Lian Enhong Chen 57 0 0 28 Feb 2025
Text-to-SQL Domain Adaptation via Human-LLM Collaborative Data Annotation Yuan Tian Daniel Lee Fei Wu Tung Mai Kun Qian Siddhartha Sahai Tianyi Zhang Yunyao Li SyDa 45 0 0 21 Feb 2025
How Should We Build A Benchmark? Revisiting 274 Code-Related Benchmarks For LLMs Jialun Cao Yuk-Kit Chan Zixuan Ling Wenxuan Wang Shuqing Li ... Pinjia He Shuai Wang Zibin Zheng Michael R. Lyu S. Cheung ALM 69 2 0 18 Jan 2025
CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval Y. Liu Rui Meng Shafiq R. Joty Silvio Savarese Caiming Xiong Yingbo Zhou Semih Yavuz 92 3 0 19 Nov 2024
BIS: NL2SQL Service Evaluation Benchmark for Business Intelligence Scenarios Bora Caglayan Mingxue Wang John D. Kelleher Shen Fei Gui Tong Jiandong Ding Puchao Zhang 26 0 0 30 Oct 2024
PRACTIQ: A Practical Conversational Text-to-SQL dataset with Ambiguous and Unanswerable Queries Mingwen Dong Nischal Ashok Kumar Yiqun Hu Anuj Chauhan Chung-Wei Hang ... Wuwei Lan Henghui Zhu Jiarong Jiang Patrick K. L. Ng Zhiguo Wang 18 2 0 14 Oct 2024
Can Models Learn Skill Composition from Examples? Haoyu Zhao Simran Kaur Dingli Yu Anirudh Goyal Sanjeev Arora CoGe MoE 58 2 0 29 Sep 2024
ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficulty Xindi Wu Dingli Yu Yangsibo Huang Olga Russakovsky Sanjeev Arora CoGe EGVM 46 12 0 26 Aug 2024
SPINACH: SPARQL-Based Information Navigation for Challenging Real-World Questions Shicheng Liu Sina J. Semnani Harold Triedman Jialiang Xu Isaac Dan Zhao Monica S. Lam 30 6 0 16 Jul 2024
FuncEvalGMN: Evaluating Functional Correctness of SQL via Graph Matching Network Yi Zhan Yang Sun Han Weng Longjie Cui Guifeng Wang Jiajun Xie Yu Tian Xiaoming Yin Boyi Liu Dongchi Huang 34 0 0 09 Jul 2024
BookSQL: A Large Scale Text-to-SQL Dataset for Accounting Domain Rahul Kumar Amar Raja Dibbu Shrutendra Harsola Vignesh T. Subrahmaniam Ashutosh Modi 21 6 0 12 Jun 2024
EHR-SeqSQL : A Sequential Text-to-SQL Dataset For Interactively Exploring Electronic Health Records Jaehee Ryu Seonhee Cho Gyubok Lee Edward Choi 31 0 0 23 May 2024
TrustSQL: Benchmarking Text-to-SQL Reliability with Penalty-Based Scoring Gyubok Lee Woosog Chay Seonhee Cho Edward Choi LMTD 39 4 0 23 Mar 2024
Schema-Aware Multi-Task Learning for Complex Text-to-SQL Yangjun Wu Han Wang 27 0 0 09 Mar 2024
Archer: A Human-Labeled Text-to-SQL Dataset with Arithmetic, Commonsense and Hypothetical Reasoning Danna Zheng Mirella Lapata Jeff Z. Pan RALM 32 6 0 19 Feb 2024
Decomposition for Enhancing Attention: Improving LLM-based Text-to-SQL through Workflow Paradigm Yuanzhen Xie Xinzhou Jin Tao Xie Mingxiong Lin Liang Chen Chenyun Yu Lei Cheng Chengxiang Zhuo Bo Hu Zang Li 43 18 0 16 Feb 2024
Improving Generalization in Semantic Parsing by Increasing Natural Language Variation Irina Saparina Mirella Lapata 14 1 0 13 Feb 2024
Evaluating the Data Model Robustness of Text-to-SQL Systems Based on Real User Queries Jonathan Fürst Catherine Kosten Farhad Nooralahzadeh Yi Zhang Kurt Stockinger LMTD 11 7 0 13 Feb 2024
Compositional Generalization for Multi-label Text Classification: A Data-Augmentation Approach X. Chu Zhuang Li Jiahui Liu Lei Chen Yuanpei Cai Donghong Ji K. W. S. Au 42 8 0 18 Dec 2023
Leveraging Code to Improve In-context Learning for Semantic Parsing Ben Bogin Shivanshu Gupta Peter Clark Ashish Sabharwal 24 7 0 16 Nov 2023
Imagine the Unseen World: A Benchmark for Systematic Generalization in Visual World Models Yeongbin Kim Gautam Singh Junyeong Park Çağlar Gülçehre Sungjin Ahn OCL VLM 42 1 0 15 Nov 2023
A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases Juan Sequeda D. Allemang Bryon Jacob 8 26 0 13 Nov 2023
Data Factors for Better Compositional Generalization Xiang Zhou Yichen Jiang Mohit Bansal CoGe OOD 19 3 0 08 Nov 2023
Natural Language Interfaces for Tabular Data Querying and Visualization: A Survey Weixu Zhang Yifei Wang Yuanfeng Song Victor Junqiu Wei Yuxing Tian Yiyan Qi Jonathan H. Chan Raymond Chi-Wing Wong Haiqin Yang LMTD 41 15 0 27 Oct 2023
The Validity of Evaluation Results: Assessing Concurrence Across Compositionality Benchmarks Kaiser Sun Adina Williams Dieuwke Hupkes CoGe 16 6 0 26 Oct 2023
Structural generalization in COGS: Supertagging is (almost) all you need Alban Petit Caio Corro François Yvon NAI 32 1 0 21 Oct 2023
Improving Cross-Lingual Transfer through Subtree-Aware Word Reordering Ofir Arviv Dmitry Nikolaev Taelin Karidi Omri Abend LRM 32 3 0 20 Oct 2023
Battle of the Large Language Models: Dolly vs LLaMA vs Vicuna vs Guanaco vs Bard vs ChatGPT -- A Text-to-SQL Parsing Comparison Shuo Sun Yuchen Zhang Jiahuan Yan Yuze Gao Donovan Ong Bin Chen Jian Su ELM ALM 35 12 0 16 Oct 2023
Selective Demonstrations for Cross-domain Text-to-SQL Shuaichen Chang Eric Fosler-Lussier 13 19 0 10 Oct 2023
SIP: Injecting a Structural Inductive Bias into a Seq2Seq Model by Simulation Matthias Lindemann Alexander Koller Ivan Titov AI4CE 19 1 0 01 Oct 2023
Reranking for Natural Language Generation from Logical Forms: A Study based on Large Language Models Levon Haroutunian Zhuang Li Lucian Galescu Philip R. Cohen Raj Tumuluri Gholamreza Haffari LRM 26 1 0 21 Sep 2023
ExeDec: Execution Decomposition for Compositional Generalization in Neural Program Synthesis Kensen Shi Joey Hong Yinlin Deng Pengcheng Yin Manzil Zaheer Charles Sutton 18 17 0 26 Jul 2023
On Evaluation of Document Classification using RVL-CDIP Stefan Larson Gordon Lim Kevin Leach 26 3 0 21 Jun 2023
ScienceBenchmark: A Complex Real-World Benchmark for Evaluating Natural Language to SQL Systems Yi Zhang Jan Deriu George Katsogiannis-Meimarakis Catherine Kosten Georgia Koutrika Kurt Stockinger 20 18 0 07 Jun 2023
XSemPLR: Cross-Lingual Semantic Parsing in Multiple Natural Languages and Meaning Representations Yusen Zhang J. Wang Zhiguo Wang Rui Zhang VLM 63 9 0 07 Jun 2023
Learning to Substitute Spans towards Improving Compositional Generalization Zhaoyi Li Ying Wei Defu Lian 10 9 0 05 Jun 2023
Benchmarking Diverse-Modal Entity Linking with Generative Models Sijia Wang A. Li He Zhu Shenmin Zhang Chung-Wei Hang ... William Wang Zhiguo Wang Vittorio Castelli Bing Xiang Patrick K. L. Ng VLM 35 8 0 27 May 2023
Federated Learning for Semantic Parsing: Task Formulation, Evaluation Setup, New Algorithms Tianshu Zhang Changchang Liu Wei-Han Lee Yu-Chuan Su Huan Sun FedML 9 4 0 26 May 2023
UNITE: A Unified Benchmark for Text-to-SQL Evaluation Wuwei Lan Zhiguo Wang Anuj Chauhan Henghui Zhu A. Li ... Jiarong Jiang Stephen M. Ash Vittorio Castelli Patrick K. L. Ng Bing Xiang ELM LMTD 29 8 0 25 May 2023
CSS: A Large-scale Cross-schema Chinese Text-to-SQL Medical Dataset Hanchong Zhang Jieyu Li Lu Chen Ruisheng Cao Yunyang Zhang Yu Huang Yefeng Zheng Kai Yu 30 0 0 25 May 2023
SETI: Systematicity Evaluation of Textual Inference Xiyan Fu Anette Frank LRM 17 5 0 24 May 2023
Coverage-based Example Selection for In-Context Learning Shivanshu Gupta Matt Gardner Sameer Singh 18 40 0 24 May 2023
Improved Compositional Generalization by Generating Demonstrations for Meta-Learning Sam Spilsbury Alexander Ilin 42 1 0 22 May 2023
How to Prompt LLMs for Text-to-SQL: A Study in Zero-shot, Single-domain, and Cross-domain Settings Shuaichen Chang Eric Fosler-Lussier LRM 11 59 0 19 May 2023
Laziness Is a Virtue When It Comes to Compositionality in Neural Semantic Parsing M. Crouse Pavan Kapanipathi Subhajit Chaudhury Tahira Naseem Ramón Fernández Astudillo Achille Fokoue Tim Klinger NAI 25 3 0 07 May 2023
Sequential Query Encoding For Complex Query Answering on Knowledge Graphs Jiaxin Bai Tianshi Zheng Yangqiu Song 24 13 0 25 Feb 2023
On graph-based reentrancy-free semantic parsing Alban Petit Caio Corro GNN 21 3 0 15 Feb 2023
Compositional Exemplars for In-context Learning Jiacheng Ye Zhiyong Wu Jiangtao Feng Tao Yu Lingpeng Kong 30 111 0 11 Feb 2023
On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex Terry Yue Zhuo Zhuang Li Yujin Huang Fatemeh Shiri Weiqing Wang Gholamreza Haffari Yuan-Fang Li AAML 18 53 0 30 Jan 2023