Headless Language Models: Learning without Predicting with Contrastive Weight Tying

15 September 2023

Papers citing "Headless Language Models: Learning without Predicting with Contrastive Weight Tying"

7 / 7 papers shown

Title
Zipfian Whitening Sho Yokoi Han Bao Hiroto Kurita Hidetoshi Shimodaira 27 0 0 01 Nov 2024
In-Context Reinforcement Learning for Variable Action Spaces Viacheslav Sinii Alexander Nikulin Vladislav Kurenkov Ilya Zisman Sergey Kolesnikov 13 14 0 20 Dec 2023
Outliers Dimensions that Disrupt Transformers Are Driven by Frequency Giovanni Puccetti Anna Rogers Aleksandr Drozd F. Dell’Orletta 71 42 0 23 May 2022
TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning Yixuan Su Fangyu Liu Zaiqiao Meng Tian Lan Lei Shu Ehsan Shareghi Nigel Collier 128 57 0 07 Nov 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 245 1,977 0 31 Dec 2020
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models Phillip Rust Jonas Pfeiffer Ivan Vulić Sebastian Ruder Iryna Gurevych 69 234 0 31 Dec 2020
PubMedQA: A Dataset for Biomedical Research Question Answering Qiao Jin Bhuwan Dhingra Zhengping Liu William W. Cohen Xinghua Lu 202 806 0 13 Sep 2019