Targeted Syntactic Evaluation of Language Models

27 August 2018

Papers citing "Targeted Syntactic Evaluation of Language Models"

50 / 71 papers shown

Title
FLUKE: A Linguistically-Driven and Task-Agnostic Framework for Robustness Evaluation Yulia Otmakhova Hung Thinh Truong Rahmad Mahendra Zenan Zhai Rongxin Zhu Daniel Beck Jey Han Lau ELM 65 0 0 24 Apr 2025
Do Large Language Models know who did what to whom? Joseph M. Denning Xiaohan Bryor Snefjella Idan A. Blank 50 1 0 23 Apr 2025
Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora Alex Warstadt Aaron Mueller Leshem Choshen E. Wilcox Chengxu Zhuang ... Rafael Mosquera Bhargavi Paranjape Adina Williams Tal Linzen Ryan Cotterell 38 106 0 10 Apr 2025
Language Models at the Syntax-Semantics Interface: A Case Study of the Long-Distance Binding of Chinese Reflexive ziji Xiulin Yang 35 0 0 02 Apr 2025
Language Models Fail to Introspect About Their Knowledge of Language Siyuan Song Jennifer Hu Kyle Mahowald LRM KELM HILM ELM 79 2 0 10 Mar 2025
A Distributional Perspective on Word Learning in Neural Language Models Filippo Ficarra Ryan Cotterell Alex Warstadt 46 1 0 09 Feb 2025
Language Models Largely Exhibit Human-like Constituent Ordering Preferences Ada Defne Tur Gaurav Kamath Siva Reddy 57 0 0 08 Feb 2025
What Goes Into a LM Acceptability Judgment? Rethinking the Impact of Frequency and Length Lindia Tjuatja Graham Neubig Tal Linzen Sophie Hao 29 1 0 04 Nov 2024
How to Make the Most of LLMs' Grammatical Knowledge for Acceptability Judgments Yusuke Ide Yuto Nishida Miyu Oba Miyu Oba Justin Vasselli Hidetaka Kamigaito Taro Watanabe 39 2 0 19 Aug 2024
Filtered Corpus Training (FiCT) Shows that Language Models can Generalize from Indirect Evidence Abhinav Patil Jaap Jumelet Yu Ying Chiu Andy Lapastora Peter Shen Lexie Wang Clevis Willrich Shane Steinert-Threlkeld 27 13 0 24 May 2024
Acquiring Linguistic Knowledge from Multimodal Input Theodor Amariucai Alexander Scott Warstadt CLL 29 2 0 27 Feb 2024
Neither hype nor gloom do DNNs justice Gaurav Malhotra Christian Tsvetkov B. D. Evans 19 117 0 08 Dec 2023
MELA: Multilingual Evaluation of Linguistic Acceptability Ziyin Zhang Yikang Liu Wei Huang Junyu Mao Rui Wang Hai Hu 22 3 0 15 Nov 2023
Uncovering Intermediate Variables in Transformers using Circuit Probing Michael A. Lepori Thomas Serre Ellie Pavlick 70 7 0 07 Nov 2023
A Geometric Notion of Causal Probing Clément Guerner Anej Svete Tianyu Liu Alex Warstadt Ryan Cotterell LLMSV 34 12 0 27 Jul 2023
Dissociating language and thought in large language models Kyle Mahowald Anna A. Ivanova I. Blank Nancy Kanwisher J. Tenenbaum Evelina Fedorenko ELM ReLM 23 209 0 16 Jan 2023
BUMP: A Benchmark of Unfaithful Minimal Pairs for Meta-Evaluation of Faithfulness Metrics Liang Ma Shuyang Cao IV RobertL.Logan Di Lu Shihao Ran Kecheng Zhang Joel R. Tetreault A. Jaimes 13 6 0 20 Dec 2022
Language model acceptability judgements are not always robust to context Koustuv Sinha Jon Gauthier Aaron Mueller Kanishka Misra Keren Fuentes R. Levy Adina Williams 11 17 0 18 Dec 2022
Syntactic Substitutability as Unsupervised Dependency Syntax Jasper Jian Siva Reddy 16 3 0 29 Nov 2022
Probing for Incremental Parse States in Autoregressive Language Models Tiwalayo Eisape Vineet Gangireddy R. Levy Yoon Kim 17 11 0 17 Nov 2022
Emergent Linguistic Structures in Neural Networks are Fragile Emanuele La Malfa Matthew Wicker Marta Kiatkowska 15 1 0 31 Oct 2022
DEMETR: Diagnosing Evaluation Metrics for Translation Marzena Karpinska N. Raj Katherine Thai Yixiao Song Ankita Gupta Mohit Iyyer 21 36 0 25 Oct 2022
The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlative Leonie Weissweiler Valentin Hofmann Abdullatif Köksal Hinrich Schütze 27 31 0 24 Oct 2022
RuCoLA: Russian Corpus of Linguistic Acceptability Vladislav Mikhailov T. Shamardina Max Ryabinin A. Pestova I. Smurov Ekaterina Artemova 22 28 0 23 Oct 2022
On the Transformation of Latent Space in Fine-Tuned NLP Models Nadir Durrani Hassan Sajjad Fahim Dalvi Firoj Alam 27 18 0 23 Oct 2022
Analyzing Text Representations under Tight Annotation Budgets: Measuring Structural Alignment César González-Gutiérrez Audi Primadhanty Francesco Cazzaro A. Quattoni 23 0 0 11 Oct 2022
Subject Verb Agreement Error Patterns in Meaningless Sentences: Humans vs. BERT Karim Lasri Olga Seminck Alessandro Lenci Thierry Poibeau 24 4 0 21 Sep 2022
Possible Stories: Evaluating Situated Commonsense Reasoning under Multiple Possible Scenarios Mana Ashida Saku Sugawara 51 6 0 16 Sep 2022
Visual Comparison of Language Model Adaptation R. Sevastjanova E. Cakmak Shauli Ravfogel Ryan Cotterell Mennatallah El-Assady VLM 23 16 0 17 Aug 2022
Towards Unsupervised Content Disentanglement in Sentence Representations via Syntactic Roles G. Felhi Joseph Le Roux Djamé Seddah DRL 16 5 0 22 Jun 2022
A computational psycholinguistic evaluation of the syntactic abilities of Galician BERT models at the interface of dependency resolution and training time Iria de-Dios-Flores Marcos Garcia 25 2 0 06 Jun 2022
Assessing the Limits of the Distributional Hypothesis in Semantic Spaces: Trait-based Relational Knowledge and the Impact of Co-occurrences Mark Anderson Jose Camacho-Collados 22 0 0 16 May 2022
Exploiting Inductive Bias in Transformers for Unsupervised Disentanglement of Syntax and Semantics with VAEs G. Felhi Joseph Le Roux Djamé Seddah DRL 21 2 0 12 May 2022
When a sentence does not introduce a discourse entity, Transformer-based models still sometimes refer to it Sebastian Schuster Tal Linzen 8 25 0 06 May 2022
Does BERT really agree ? Fine-grained Analysis of Lexical Dependence on a Syntactic Task Karim Lasri Alessandro Lenci Thierry Poibeau 16 13 0 14 Apr 2022
minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Models Kanishka Misra 17 58 0 24 Mar 2022
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models Aaron Mueller Robert Frank Tal Linzen Luheng Wang Sebastian Schuster AIMat 11 33 0 17 Mar 2022
Data-driven Model Generalizability in Crosslinguistic Low-resource Morphological Segmentation Zoey Liu Emily Tucker Prudhommeaux 25 4 0 05 Jan 2022
Monolingual and Cross-Lingual Acceptability Judgments with the Italian CoLA corpus Daniela Trotta R. Guarasci Elisa Leonardelli Sara Tonelli 42 30 0 24 Sep 2021
Controlled Evaluation of Grammatical Knowledge in Mandarin Chinese Language Models Yiwen Wang Jennifer Hu R. Levy Peng Qian 9 3 0 22 Sep 2021
How Does Adversarial Fine-Tuning Benefit BERT? J. Ebrahimi Hao Yang Wei Zhang AAML 18 4 0 31 Aug 2021
Neuron-level Interpretation of Deep NLP Models: A Survey Hassan Sajjad Nadir Durrani Fahim Dalvi MILM AI4CE 22 79 0 30 Aug 2021
Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probing Rowan Hall Maudslay Ryan Cotterell 21 33 0 04 Jun 2021
SyGNS: A Systematic Generalization Testbed Based on Natural Language Semantics Hitomi Yanaka K. Mineshima Kentaro Inui NAI AI4CE 25 11 0 02 Jun 2021
How Reliable are Model Diagnostics? V. Aribandi Yi Tay Donald Metzler 16 19 0 12 May 2021
Understanding by Understanding Not: Modeling Negation in Language Models Arian Hosseini Siva Reddy Dzmitry Bahdanau R. Devon Hjelm Alessandro Sordoni Aaron C. Courville 11 87 0 07 May 2021
Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Models Anne Beyer Sharid Loáiciga David Schlangen 11 15 0 07 May 2021
Refining Targeted Syntactic Evaluation of Language Models Benjamin Newman Kai-Siang Ang Julia Gong John Hewitt 14 43 0 19 Apr 2021
A multilabel approach to morphosyntactic probing Naomi Tachikawa Shapiro Amandalynne Paullada Shane Steinert-Threlkeld 27 10 0 17 Apr 2021
Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little Koustuv Sinha Robin Jia Dieuwke Hupkes J. Pineau Adina Williams Douwe Kiela 29 243 0 14 Apr 2021