A Survey on Transformers in NLP with Focus on Efficiency

A Survey on Transformers in NLP with Focus on Efficiency

15 May 2024

Saptarsi Goswami

Amlan Chakrabarti

Papers citing "A Survey on Transformers in NLP with Focus on Efficiency"

12 / 12 papers shown

Title
Attention Condensation via Sparsity Induced Regularized Training Eli Sason Darya Frolova Boris Nazarov Felix Goldberd 101 0 0 03 Mar 2025
Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning Barun Patra Saksham Singhal Shaohan Huang Zewen Chi Li Dong Furu Wei Vishrav Chaudhary Xia Song 56 23 0 26 Oct 2022
Deduplicating Training Data Makes Language Models Better Katherine Lee Daphne Ippolito A. Nystrom Chiyuan Zhang Douglas Eck Chris Callison-Burch Nicholas Carlini SyDa 237 588 0 14 Jul 2021
Carbon Emissions and Large Neural Network Training David A. Patterson Joseph E. Gonzalez Quoc V. Le Chen Liang Lluís-Miquel Munguía D. Rothchild David R. So Maud Texier J. Dean AI4CE 239 626 0 21 Apr 2021
Shortformer: Better Language Modeling using Shorter Inputs Ofir Press Noah A. Smith M. Lewis 219 87 0 31 Dec 2020
Cold-start Active Learning through Self-supervised Language Modeling Michelle Yuan Hsuan-Tien Lin Jordan L. Boyd-Graber 104 180 0 19 Oct 2020
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 249 1,982 0 28 Jul 2020
Pre-trained Models for Natural Language Processing: A Survey Xipeng Qiu Tianxiang Sun Yige Xu Yunfan Shao Ning Dai Xuanjing Huang LM&MA VLM 241 1,444 0 18 Mar 2020
Efficient Content-Based Sparse Attention with Routing Transformers Aurko Roy M. Saffar Ashish Vaswani David Grangier MoE 238 578 0 12 Mar 2020
Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference Timo Schick Hinrich Schütze 258 1,584 0 21 Jan 2020
SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents Ramesh Nallapati Feifei Zhai Bowen Zhou 207 1,249 0 14 Nov 2016
Effective Approaches to Attention-based Neural Machine Translation Thang Luong Hieu H. Pham Christopher D. Manning 214 7,687 0 17 Aug 2015