Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little

14 April 2021

Robin Jia

Douwe Kiela

Papers citing "Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little"

50 / 165 papers shown

Title
A 5' UTR Language Model for Decoding Untranslated Regions of mRNA and Function Predictions Yanyi Chu Dan Yu Yupeng Li Kaixuan Huang Yue Shen Le Cong Jason Zhang Mengdi Wang 85 45 0 05 Oct 2023
Language Models as a Service: Overview of a New Paradigm and its Challenges Emanuele La Malfa Aleksandar Petrov Simon Frieder Christoph Weinhuber Ryan Burnell Raza Nazar Anthony Cohn Nigel Shadbolt Michael Wooldridge ALM ELM 30 3 0 28 Sep 2023
SCOB: Universal Text Understanding via Character-wise Supervised Contrastive Learning with Online Text Rendering for Bridging Domain Gap Daehee Kim Yoon Kim Donghyun Kim Yumin Lim Geewook Kim Taeho Kil 23 3 0 21 Sep 2023
ContextRef: Evaluating Referenceless Metrics For Image Description Generation Elisa Kreiss E. Zelikman Christopher Potts Nick Haber 19 5 0 21 Sep 2023
Multilingual Text Representation Fahim Faisal 19 0 0 02 Sep 2023
Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining? Fei-Yue Wang Liang Ding Jun Rao Ye Liu Li Shen Changxing Ding 32 15 0 24 Aug 2023
On Data Imbalance in Molecular Property Prediction with Pre-training Limin Wang Masatoshi Hanai Toyotaro Suzumura Shun Takashige Kenjiro Taura AI4CE 30 0 0 17 Aug 2023
ICSVR: Investigating Compositional and Syntactic Understanding in Video Retrieval Models Avinash Madasu Vasudev Lal CoGe 42 3 0 28 Jun 2023
Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion Isha Rawal Alexander Matyasko Shantanu Jaiswal Basura Fernando Cheston Tan 21 1 0 15 Jun 2023
Human-imperceptible, Machine-recognizable Images Fusheng Hao Fengxiang He Yikai Wang Fuxiang Wu Jing Zhang Jun Cheng Dacheng Tao AAML 19 0 0 06 Jun 2023
Does Character-level Information Always Improve DRS-based Semantic Parsing? Tomoya Kurosawa Hitomi Yanaka 16 0 0 04 Jun 2023
Revisiting the Role of Language Priors in Vision-Language Models Zhiqiu Lin Xinyue Chen Deepak Pathak Pengchuan Zhang Deva Ramanan VLM 23 22 0 02 Jun 2023
Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings Ta-Chung Chi Ting-Han Fan Li-Wei Chen Alexander I. Rudnicky Peter J. Ramadge VLM MILM 52 12 0 23 May 2023
A Better Way to Do Masked Language Model Scoring Carina Kauf Anna A. Ivanova 42 22 0 17 May 2023
Towards preserving word order importance through Forced Invalidation Hadeel Al-Negheimish Pranava Madhyastha Alessandra Russo 19 3 0 11 Apr 2023
Language Model Behavior: A Comprehensive Survey Tyler A. Chang Benjamin Bergen VLM LRM LM&MA 27 103 0 20 Mar 2023
Joint Representations of Text and Knowledge Graphs for Retrieval and Evaluation Teven Le Scao Claire Gardent 25 2 0 28 Feb 2023
Information-Restricted Neural Language Models Reveal Different Brain Regions' Sensitivity to Semantics, Syntax and Context Alexandre Pasquiou Yair Lakretz B. Thirion Christophe Pallier 11 16 0 28 Feb 2023
Can discrete information extraction prompts generalize across language models? Nathanaël Carraz Rakotonirina Roberto Dessì Fabio Petroni Sebastian Riedel Marco Baroni 15 7 0 20 Feb 2023
RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions Yunlong Wang Shuyuan Shen Brian Y. Lim 28 88 0 19 Feb 2023
Position Matters! Empirical Study of Order Effect in Knowledge-grounded Dialogue Hsuan Su Shachi H. Kumar Sahisnu Mazumder Wenda Chen R. Manuvinakurike Eda Okur Saurav Sahay L. Nachman Shang-Tse Chen Hung-yi Lee 23 3 0 12 Feb 2023
When are Lemons Purple? The Concept Association Bias of Vision-Language Models Yutaro Yamada Yingtian Tang Yoyo Zhang Ilker Yildirim CoGe 19 14 0 22 Dec 2022
Synthetic Pre-Training Tasks for Neural Machine Translation Zexue He Graeme W. Blackwood Rameswar Panda Julian McAuley Rogerio Feris 13 3 0 19 Dec 2022
Deep representation learning: Fundamentals, Perspectives, Applications, and Open Challenges K. T. Baghaei Amirreza Payandeh Pooya Fayyazsanavi Shahram Rahimi Zhiqian Chen Somayeh Bakhtiari Ramezani FaML AI4TS 30 6 0 27 Nov 2022
Local Structure Matters Most in Most Languages Louis Clouâtre Prasanna Parthasarathi Amal Zouaq Sarath Chandar 26 1 0 09 Nov 2022
Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes Louis Clouâtre Prasanna Parthasarathi Amal Zouaq Sarath Chandar 33 3 0 09 Nov 2022
Word Order Matters when you Increase Masking Karim Lasri Alessandro Lenci Thierry Poibeau 28 7 0 08 Nov 2022
Processing Long Legal Documents with Pre-trained Transformers: Modding LegalBERT and Longformer Dimitris Mamakas Petros Tsotsi Ion Androutsopoulos Ilias Chalkidis VLM AILaw 21 27 0 02 Nov 2022
Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality Anuj Diwan Layne Berry Eunsol Choi David F. Harwath Kyle Mahowald CoGe 103 41 0 01 Nov 2022
Emergent Linguistic Structures in Neural Networks are Fragile Emanuele La Malfa Matthew Wicker Marta Kiatkowska 15 1 0 31 Oct 2022
ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metrics Chantal Amrhein Nikita Moghe Liane Guillou ELM 26 22 0 27 Oct 2022
Causal Analysis of Syntactic Agreement Neurons in Multilingual Language Models Aaron Mueller Yudi Xia Tal Linzen MILM 34 9 0 25 Oct 2022
The Curious Case of Absolute Position Embeddings Koustuv Sinha Amirhossein Kazemnejad Siva Reddy J. Pineau Dieuwke Hupkes Adina Williams 83 15 0 23 Oct 2022
Can Pretrained Language Models (Yet) Reason Deductively? Moy Yuan Songbo Hu Ivan Vulić Anna Korhonen Zaiqiao Meng ReLM ELM LRM 34 8 0 12 Oct 2022
State-of-the-art generalisation research in NLP: A taxonomy and review Dieuwke Hupkes Mario Giulianelli Verna Dankers Mikel Artetxe Yanai Elazar ... Leila Khalatbari Maria Ryskina Rita Frieske Ryan Cotterell Zhijing Jin 108 93 0 06 Oct 2022
When and why vision-language models behave like bags-of-words, and what to do about it? Mert Yuksekgonul Federico Bianchi Pratyusha Kalluri Dan Jurafsky James Y. Zou VLM CoGe 28 362 0 04 Oct 2022
Downstream Datasets Make Surprisingly Good Pretraining Corpora Kundan Krishna Saurabh Garg Jeffrey P. Bigham Zachary Chase Lipton 38 30 0 28 Sep 2022
On the Effectiveness of Compact Biomedical Transformers Omid Rohanian Mohammadmahdi Nouriborji Samaneh Kouchaki David A. Clifton MedIm 18 31 0 07 Sep 2022
Do language models make human-like predictions about the coreferents of Italian anaphoric zero pronouns? J. Michaelov Benjamin Bergen 25 6 0 30 Aug 2022
Shortcut Learning of Large Language Models in Natural Language Understanding Mengnan Du Fengxiang He Na Zou Dacheng Tao Xia Hu KELM OffRL 28 83 0 25 Aug 2022
Compositional Evaluation on Japanese Textual Entailment and Similarity Hitomi Yanaka K. Mineshima 14 24 0 09 Aug 2022
LAD: Language Models as Data for Zero-Shot Dialog Shikib Mehri Yasemin Altun M. Eskénazi 20 25 0 28 Jul 2022
Unit Testing for Concepts in Neural Networks Charles Lovering Ellie Pavlick 23 28 0 28 Jul 2022
Position Prediction as an Effective Pretraining Strategy Shuangfei Zhai Navdeep Jaitly Jason Ramapuram Dan Busbridge Tatiana Likhomanenko Joseph Y. Cheng Walter A. Talbott Chen Huang Hanlin Goh J. Susskind ViT 35 23 0 15 Jul 2022
An Approach to Ensure Fairness in News Articles Shaina Raza Deepak John Reji Dora D. Liu Syed Raza Bashir Usman Naseem FaML 18 1 0 08 Jul 2022
The Role of Complex NLP in Transformers for Text Ranking? David Rau J. Kamps 19 10 0 06 Jul 2022
The Linguistic Blind Spot of Value-Aligned Agency, Natural and Artificial Travis LaCroix 20 3 0 02 Jul 2022
Insights into Pre-training via Simpler Synthetic Tasks Yuhuai Wu Felix Li Percy Liang AIMat 24 20 0 21 Jun 2022
Order-sensitive Shapley Values for Evaluating Conceptual Soundness of NLP Models Kaiji Lu Anupam Datta 13 0 0 01 Jun 2022
Revisiting Generative Commonsense Reasoning: A Pre-Ordering Approach Chao Zhao Faeze Brahman Tenghao Huang Snigdha Chaturvedi LRM 8 3 0 26 May 2022