AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing

12 August 2021

Katikapalli Subramanyam Kalyan

Papers citing "AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing"

50 / 58 papers shown

Title
Non-Stationary Time Series Forecasting Based on Fourier Analysis and Cross Attention Mechanism Yuqi Xiong Yang Wen AI4TS 21 0 0 11 May 2025
SCFormer: Structured Channel-wise Transformer with Cumulative Historical State for Multivariate Time Series Forecasting Shiwei Guo Z. Chen Yupeng Ma Yunfei Han Yi Wang AI4TS 100 0 0 05 May 2025
SFi-Former: Sparse Flow Induced Attention for Graph Transformer Z. Li J. Q. Shi X. Zhang Miao Zhang B. Li 44 0 0 29 Apr 2025
Entity-aware Cross-lingual Claim Detection for Automated Fact-checking Rrubaa Panchendrarajan A. Zubiaga 45 0 0 19 Mar 2025
Assessing Text Classification Methods for Cyberbullying Detection on Social Media Platforms Adamu Gaston Philipo Doreen Sebastian Sarwatt Jianguo Ding M. Daneshmand Huansheng Ning 41 0 0 31 Dec 2024
Vulnerability Analysis of Transformer-based Optical Character Recognition to Adversarial Attacks Lucas Beerens D. Higham 26 1 0 28 Nov 2023
Evaluating multiple large language models in pediatric ophthalmology J. Holmes Rui Peng Yiwei Li Jinyu Hu Zheng Liu ... Wei Liu Hong Wei Jie Zou Tianming Liu Yi Shao AI4Ed ELM LM&MA 19 0 0 07 Nov 2023
An Ensemble Approach to Question Classification: Integrating Electra Transformer, GloVe, and LSTM Sanad Aburass O. Dorgham Maha Abu Rumman 22 3 0 13 Aug 2023
Multilingual Multiword Expression Identification Using Lateral Inhibition and Domain Adaptation Andrei-Marius Avram V. Mititelu V. Pais Dumitru-Clementin Cercel Stefan Trausan-Matu 25 3 0 17 Jun 2023
A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validation Xiaowei Huang Wenjie Ruan Wei Huang Gao Jin Yizhen Dong ... Sihao Wu Peipei Xu Dengyu Wu André Freitas Mustafa A. Mustafa ALM 27 81 0 19 May 2023
HICO-DET-SG and V-COCO-SG: New Data Splits for Evaluating the Systematic Generalization Performance of Human-Object Interaction Detection Models Kenta Takemoto Moyuru Yamada Tomotake Sasaki H. Akima 27 0 0 17 May 2023
IUST_NLP at SemEval-2023 Task 10: Explainable Detecting Sexism with Transformers and Task-adaptive Pretraining Hadi Mahmoudi 8 0 0 11 May 2023
SELFormer: Molecular Representation Learning via SELFIES Language Models Atakan Yüksel Erva Ulusoy Atabey Ünlü Tunca Dogan 25 54 0 10 Apr 2023
Safety Analysis in the Era of Large Language Models: A Case Study of STPA using ChatGPT Yi Qi Xingyu Zhao Siddartha Khastgir Xiaowei Huang 24 14 0 03 Apr 2023
Diffusing Graph Attention Daniel Glickman Eran Yahav GNN 38 3 0 01 Mar 2023
Mask-guided BERT for Few Shot Text Classification Wenxiong Liao Zheng Liu Haixing Dai Zihao Wu Yiyang Zhang ... Dajiang Zhu Tianming Liu Sheng R. Li Xiang Li Hongmin Cai VLM 36 39 0 21 Feb 2023
PSST! Prosodic Speech Segmentation with Transformers Nathan Roll C. Graham Simon Todd VLM 26 5 0 03 Feb 2023
On the Connection Between MPNN and Graph Transformer Chen Cai Truong Son-Hy Rose Yu Yusu Wang 28 50 0 27 Jan 2023
MaNLP@SMM4H22: BERT for Classification of Twitter Posts Keshav Kapur Rajitha Harikrishnan 25 3 0 12 Dec 2022
Block Format Error Bounds and Optimal Block Size Selection I. Soloveychik I. Lyubomirsky Xin Eric Wang S. Bhoja MQ 27 4 0 11 Oct 2022
Detecting Harmful Online Conversational Content towards LGBTQIA+ Individuals Jamell Dacon Harry Shomer Shaylynn Crum-Dacon Jiliang Tang 9 8 0 15 Jun 2022
E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generation Qihuang Zhong Liang Ding Juhua Liu Bo Du Dacheng Tao 29 27 0 30 May 2022
Recipe for a General, Powerful, Scalable Graph Transformer Ladislav Rampášek Mikhail Galkin Vijay Prakash Dwivedi A. Luu Guy Wolf Dominique Beaini 48 507 0 25 May 2022
UMass PCL at SemEval-2022 Task 4: Pre-trained Language Model Ensembles for Detecting Patronizing and Condescending Language David Koleczek Alexander Scarlatos Siddha Makarand Karkare Preshma Linet Pereira 19 0 0 18 Apr 2022
mGPT: Few-Shot Learners Go Multilingual Oleh Shliazhko Alena Fenogenova Maria Tikhonova Vladislav Mikhailov Anastasia Kozlova Tatiana Shavrina 31 148 0 15 Apr 2022
Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Survey Prajjwal Bhargava Vincent Ng ReLM LRM 34 62 0 28 Jan 2022
Video Transformers: A Survey Javier Selva A. S. Johansen Sergio Escalera Kamal Nasrollahi T. Moeslund Albert Clapés ViT 20 102 0 16 Jan 2022
Offensive Language Identification in Low-resourced Code-mixed Dravidian languages using Pseudo-labeling Adeep Hande Karthik Puranik Konthala Yasaswini R. Priyadharshini Sajeetha Thavareesan Anbukkarasi Sampath Kogilavani Shanmugavadivel D. Thenmozhi Bharathi Raja Chakravarthi 17 29 0 27 Aug 2021
FLEX: Unifying Evaluation for Few-Shot NLP Jonathan Bragg Arman Cohan Kyle Lo Iz Beltagy 197 104 0 15 Jul 2021
Deduplicating Training Data Makes Language Models Better Katherine Lee Daphne Ippolito A. Nystrom Chiyuan Zhang Douglas Eck Chris Callison-Burch Nicholas Carlini SyDa 237 590 0 14 Jul 2021
A Primer on Pretrained Multilingual Language Models Sumanth Doddapaneni Gowtham Ramesh Mitesh M. Khapra Anoop Kunchukuttan Pratyush Kumar LRM 35 73 0 01 Jul 2021
Larger-Scale Transformers for Multilingual Masked Language Modeling Naman Goyal Jingfei Du Myle Ott Giridhar Anantharaman Alexis Conneau 88 98 0 02 May 2021
MathBERT: A Pre-Trained Model for Mathematical Formula Understanding Shuai Peng Ke Yuan Liangcai Gao Zhi Tang AIMat 41 104 0 02 May 2021
MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs Zewen Chi Li Dong Shuming Ma Shaohan Huang Xian-Ling Mao Heyan Huang Furu Wei LRM 45 71 0 18 Apr 2021
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 280 3,843 0 18 Apr 2021
AMMU : A Survey of Transformer-based Biomedical Pretrained Language Models Katikapalli Subramanyam Kalyan A. Rajasekharan S. Sangeetha LM&MA MedIm 18 163 0 16 Apr 2021
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation Shuai Lu Daya Guo Shuo Ren Junjie Huang Alexey Svyatkovskiy ... Nan Duan Neel Sundaresan Shao Kun Deng Shengyu Fu Shujie Liu ELM 196 853 0 09 Feb 2021
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics Sebastian Gehrmann Tosin P. Adewumi Karmanya Aggarwal Pawan Sasanka Ammanamanchi Aremu Anuoluwapo ... Nishant Subramani Wei-ping Xu Diyi Yang Akhila Yerukola Jiawei Zhou VLM 246 283 0 02 Feb 2021
Transformers in Vision: A Survey Salman Khan Muzammal Naseer Munawar Hayat Syed Waqas Zamir F. Khan M. Shah ViT 225 2,428 0 04 Jan 2021
Making Pre-trained Language Models Better Few-shot Learners Tianyu Gao Adam Fisch Danqi Chen 241 1,916 0 31 Dec 2020
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models Phillip Rust Jonas Pfeiffer Ivan Vulić Sebastian Ruder Iryna Gurevych 69 234 0 31 Dec 2020
ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic Muhammad Abdul-Mageed AbdelRahim Elmadany El Moatez Billah Nagoudi VLM 60 447 0 27 Dec 2020
Extracting Training Data from Large Language Models Nicholas Carlini Florian Tramèr Eric Wallace Matthew Jagielski Ariel Herbert-Voss ... Tom B. Brown D. Song Ulfar Erlingsson Alina Oprea Colin Raffel MLAU SILM 267 1,808 0 14 Dec 2020
CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters Hicham El Boukkouri Olivier Ferret Thomas Lavergne Hiroshi Noji Pierre Zweigenbaum Junichi Tsujii 66 156 0 20 Oct 2020
A Benchmark for Lease Contract Review Spyretta Leivaditi Julien Rossi Evangelos Kanoulas AILaw 106 36 0 20 Oct 2020
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 251 2,012 0 28 Jul 2020
Pre-trained Models for Natural Language Processing: A Survey Xipeng Qiu Tianxiang Sun Yige Xu Yunfan Shao Ning Dai Xuanjing Huang LM&MA VLM 241 1,450 0 18 Mar 2020
A Survey on Contextual Embeddings Qi Liu Matt J. Kusner Phil Blunsom 214 146 0 16 Mar 2020
PhoBERT: Pre-trained language models for Vietnamese Dat Quoc Nguyen A. Nguyen 162 341 0 02 Mar 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 226 4,453 0 23 Jan 2020