What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?

12 April 2022

Papers citing "What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?"

36 / 36 papers shown

Title
Retrieval Backward Attention without Additional Training: Enhance Embeddings of Large Language Models via Repetition Yifei Duan Raphael Shang Deng Liang Yongqiang Cai 80 0 0 28 Feb 2025
Registering Source Tokens to Target Language Spaces in Multilingual Neural Machine Translation Zhi Qu Yiran Wang Jiannan Mao Chenchen Ding Hideki Tanaka Masao Utiyama Taro Watanabe LRM 40 0 0 06 Jan 2025
ControlMM: Controllable Masked Motion Generation Ekkasit Pinyoanuntapong Muhammad Usama Saleem Korrawe Karunratanakul Pu Wang Hongfei Xue C. L. P. Chen Chuan Guo Junli Cao J. Ren Sergey Tulyakov VGen 29 4 0 14 Oct 2024
How Does Code Pretraining Affect Language Model Task Performance? Jackson Petty Sjoerd van Steenkiste Tal Linzen 60 8 0 06 Sep 2024
AdaNAT: Exploring Adaptive Policy for Token-Based Image Generation Zanlin Ni Yulin Wang Renping Zhou Rui Lu Jiayi Guo Jinyi Hu Zhiyuan Liu Yuan Yao Gao Huang 25 7 0 31 Aug 2024
Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretraining Qi Zhang Tianqi Du Haotian Huang Yifei Wang Yisen Wang 34 3 0 01 Jul 2024
From Instance Training to Instruction Learning: Task Adapters Generation from Instructions Huanxuan Liao Yao Xu Shizhu He Yuanzhe Zhang Yanchao Hao Shengping Liu Kang Liu Jun Zhao 39 1 0 18 Jun 2024
Benchmark Data Contamination of Large Language Models: A Survey Cheng Xu Shuhao Guan Derek Greene Mohand-Tahar Kechadi ELM ALM 38 38 0 06 Jun 2024
Scaling Laws for Discriminative Classification in Large Language Models Dean Wyatte Fatemeh Tahmasbi Ming Li Thomas Markovich 35 2 0 24 May 2024
Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models Jingyang Zhang Jingwei Sun Eric C. Yeats Ouyang Yang Martin Kuo Jianyi Zhang Hao Frank Yang Hai Li 32 41 0 03 Apr 2024
Where is the answer? Investigating Positional Bias in Language Model Knowledge Extraction Kuniaki Saito Kihyuk Sohn Chen-Yu Lee Yoshitaka Ushiku 62 2 0 16 Feb 2024
Convergences and Divergences between Automatic Assessment and Human Evaluation: Insights from Comparing ChatGPT-Generated Translation and Neural Machine Translation Zhaokun Jiang Ziyin Zhang EGVM 19 3 0 10 Jan 2024
CodeGen2: Lessons for Training LLMs on Programming and Natural Languages Erik Nijkamp A. Ghobadzadeh Caiming Xiong Silvio Savarese Yingbo Zhou 147 164 0 03 May 2023
BARTSmiles: Generative Masked Language Models for Molecular Representations Gayane Chilingaryan Hovhannes Tamoyan Ani Tevosyan N. Babayan L. Khondkaryan Karen Hambardzumyan Zaven Navoyan Hrant Khachatrian Armen Aghajanyan SSL 25 26 0 29 Nov 2022
On the Compositional Generalization Gap of In-Context Learning Arian Hosseini Ankit Vani Dzmitry Bahdanau Alessandro Sordoni Aaron C. Courville 19 24 0 15 Nov 2022
Large Language Models Struggle to Learn Long-Tail Knowledge Nikhil Kandpal H. Deng Adam Roberts Eric Wallace Colin Raffel RALM KELM 36 378 0 15 Nov 2022
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model BigScience Workshop : Teven Le Scao Angela Fan Christopher Akiki ... Zhongli Xie Zifan Ye M. Bras Younes Belkada Thomas Wolf VLM 83 2,301 0 09 Nov 2022
GPS: Genetic Prompt Search for Efficient Few-shot Learning Hanwei Xu Yujun Chen Yulun Du Nan Shao Yanggang Wang Haiyu Li Zhilin Yang VLM 14 28 0 31 Oct 2022
What Language Model to Train if You Have One Million GPU Hours? Teven Le Scao Thomas Wang Daniel Hesslow Lucile Saulnier Stas Bekman ... Lintang Sutawika Jaesung Tae Zheng-Xin Yong Julien Launay Iz Beltagy MoE AI4CE 225 103 0 27 Oct 2022
Towards Better Few-Shot and Finetuning Performance with Forgetful Causal Language Models Hao Liu Xinyang Geng Lisa Lee Igor Mordatch Sergey Levine Sharan Narang Pieter Abbeel KELM CLL 33 2 0 24 Oct 2022
Scaling Instruction-Finetuned Language Models Hyung Won Chung Le Hou Shayne Longpre Barret Zoph Yi Tay ... Jacob Devlin Adam Roberts Denny Zhou Quoc V. Le Jason W. Wei ReLM LRM 56 2,986 0 20 Oct 2022
Transcending Scaling Laws with 0.1% Extra Compute Yi Tay Jason W. Wei Hyung Won Chung Vinh Q. Tran David R. So ... Donald Metzler Slav Petrov N. Houlsby Quoc V. Le Mostafa Dehghani LRM 29 68 0 20 Oct 2022
VIMA: General Robot Manipulation with Multimodal Prompts Yunfan Jiang Agrim Gupta Zichen Zhang Guanzhi Wang Yongqiang Dou Yanjun Chen Li Fei-Fei Anima Anandkumar Yuke Zhu Linxi Fan LM&Ro 15 334 0 06 Oct 2022
Bidirectional Language Models Are Also Few-shot Learners Ajay Patel Bryan Li Mohammad Sadegh Rasooli Noah Constant Colin Raffel Chris Callison-Burch LRM 62 45 0 29 Sep 2022
AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model Saleh Soltan Shankar Ananthakrishnan Jack G. M. FitzGerald Rahul Gupta Wael Hamza ... Mukund Sridhar Fabian Triefenbach Apurv Verma Gökhan Tür Premkumar Natarajan 34 82 0 02 Aug 2022
Language Models are General-Purpose Interfaces Y. Hao Haoyu Song Li Dong Shaohan Huang Zewen Chi Wenhui Wang Shuming Ma Furu Wei MLLM 19 95 0 13 Jun 2022
UL2: Unifying Language Learning Paradigms Yi Tay Mostafa Dehghani Vinh Q. Tran Xavier Garcia Jason W. Wei ... Tal Schuster H. Zheng Denny Zhou N. Houlsby Donald Metzler AI4CE 52 293 0 10 May 2022
Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space Mor Geva Avi Caciularu Ke Wang Yoav Goldberg KELM 34 333 0 28 Mar 2022
Multitask Prompted Training Enables Zero-Shot Task Generalization Victor Sanh Albert Webson Colin Raffel Stephen H. Bach Lintang Sutawika ... T. Bers Stella Biderman Leo Gao Thomas Wolf Alexander M. Rush LRM 211 1,656 0 15 Oct 2021
What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers Boseop Kim Hyoungseok Kim Sang-Woo Lee Gichang Lee Donghyun Kwak ... Jaewook Kang Inho Kang Jung-Woo Ha W. Park Nako Sung VLM 241 121 0 10 Sep 2021
Carbon Emissions and Large Neural Network Training David A. Patterson Joseph E. Gonzalez Quoc V. Le Chen Liang Lluís-Miquel Munguía D. Rothchild David R. So Maud Texier J. Dean AI4CE 239 642 0 21 Apr 2021
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 280 3,843 0 18 Apr 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 248 1,986 0 31 Dec 2020
PubMedQA: A Dataset for Biomedical Research Question Answering Qiao Jin Bhuwan Dhingra Zhengping Liu William W. Cohen Xinghua Lu 205 807 0 13 Sep 2019
The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives Elena Voita Rico Sennrich Ivan Titov 188 181 0 03 Sep 2019
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 294 6,943 0 20 Apr 2018