Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning

17 June 2021

Papers citing "Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning"

50 / 74 papers shown

Title
Chronocept: Instilling a Sense of Time in Machines Krish Goel Sanskar Pandey KS Mahadevan Harsh Kumar Vishesh Khadaria 13 0 0 12 May 2025
When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformers Hongkang Li Yihua Zhang Shuai Zhang M. Wang Sijia Liu Pin-Yu Chen MoMe 57 2 0 15 Apr 2025
Personalized Federated Fine-tuning for Heterogeneous Data: An Automatic Rank Learning Approach via Two-Level LoRA Jie Hao Yuman Wu Ali Payani Myungjin Lee Mingrui Liu 29 1 0 05 Mar 2025
Provable Benefits of Unsupervised Pre-training and Transfer Learning via Single-Index Models Taj Jones-McCormick Aukosh Jagannath S. Sen 46 0 0 24 Feb 2025
Do we really have to filter out random noise in pre-training data for language models? Jinghan Ru Yuxin Xie Xianwei Zhuang Yuguo Yin Yuexian Zou 83 2 0 10 Feb 2025
A Probabilistic Model for Self-Supervised Learning Maximilian Fleissner P. Esser D. Ghoshdastidar SSL BDL 91 1 0 22 Jan 2025
Investigating the Impact of Model Complexity in Large Language Models Jing Luo Huiyuan Wang Weiran Huang 24 0 0 01 Oct 2024
AsthmaBot: Multi-modal, Multi-Lingual Retrieval Augmented Generation For Asthma Patient Support Adil Bahaj Mounir Ghogho 33 2 0 24 Sep 2024
Unlocking Memorization in Large Language Models with Dynamic Soft Prompting Zhepeng Wang Runxue Bao Yawen Wu Jackson Taylor Cao Xiao Feng Zheng Weiwen Jiang Shangqian Gao Yanfu Zhang PILM 33 7 0 20 Sep 2024
Geometric Self-Supervised Pretraining on 3D Protein Structures using Subgraphs Michail Chatzianastasis George Dasoulas Michalis Vazirgiannis SSL 19 0 0 20 Jun 2024
CTSyn: A Foundational Model for Cross Tabular Data Generation Xiaofeng Lin Chenheng Xu Matthew Yang Guang Cheng 32 3 0 07 Jun 2024
Mixture-of-Prompt-Experts for Multi-modal Semantic Understanding Zichen Wu Hsiu-Yuan Huang Fanyi Qu Yunfang Wu VLM MoE 24 3 0 17 Mar 2024
When can we Approximate Wide Contrastive Models with Neural Tangent Kernels and Principal Component Analysis? Gautham Govind Anil P. Esser D. Ghoshdastidar 24 1 0 13 Mar 2024
Low-Rank Approximation of Structural Redundancy for Self-Supervised Learning Kang Du Yu Xiang 17 0 0 10 Feb 2024
BayesPrompt: Prompting Large-Scale Pre-Trained Language Models on Few-shot Inference via Debiased Domain Abstraction Jiangmeng Li Fei Song Yifan Jin Wenwen Qiang Changwen Zheng Fuchun Sun Hui Xiong VLM 32 2 0 25 Jan 2024
Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models Yae Jee Cho Luyang Liu Zheng Xu Aldi Fahrezi Gauri Joshi 6 45 0 12 Jan 2024
MLPs Compass: What is learned when MLPs are combined with PLMs? Li Zhou Wenyu Chen Yong Cao DingYi Zeng Wanlong Liu Hong Qu 26 0 0 03 Jan 2024
If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents Ke Yang Jiateng Liu John Wu Chaoqi Yang Yi Ren Fung ... Xu Cao Xingyao Wang Yiquan Wang Heng Ji Chengxiang Zhai LLMAG ELM 18 71 0 01 Jan 2024
Latent Skill Discovery for Chain-of-Thought Reasoning Zifan Xu Haozhu Wang Dmitriy Bespalov Peter Stone Yanjun Qi ReLM LRM 46 2 0 07 Dec 2023
An Eye on Clinical BERT: Investigating Language Model Generalization for Diabetic Eye Disease Phenotyping Keith Harrigian Tina Tang Anthony Gonzales Cindy X. Cai Mark Dredze VLM 20 2 0 15 Nov 2023
Instructive Decoding: Instruction-Tuned Large Language Models are Self-Refiner from Noisy Instructions Taehyeon Kim Joonkee Kim Gihun Lee Se-Young Yun 17 11 0 01 Nov 2023
The Distributional Hypothesis Does Not Fully Explain the Benefits of Masked Language Model Pretraining Ting-Rui Chiang Dani Yogatama 20 1 0 25 Oct 2023
PCGPT: Procedural Content Generation via Transformers Sajad Mohaghegh Mohammad Amin Ramezan Dehnavi Golnoosh Abdollahinejad Matin Hashemi ViT 8 2 0 03 Oct 2023
Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiency Zhihan Liu Hao Hu Shenao Zhang Hongyi Guo Shuqi Ke Boyi Liu Zhaoran Wang LLMAG LRM 18 33 0 29 Sep 2023
Human-AI Interactions and Societal Pitfalls Francisco Castro Jian Gao Sébastien Martin 12 2 0 19 Sep 2023
Representation Learning Dynamics of Self-Supervised Models P. Esser Satyaki Mukherjee D. Ghoshdastidar SSL 19 2 0 05 Sep 2023
GradientCoin: A Peer-to-Peer Decentralized Large Language Models Yeqi Gao Zhao-quan Song Junze Yin 21 18 0 21 Aug 2023
Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regression Runtian Zhai Bing Liu Andrej Risteski Zico Kolter Pradeep Ravikumar SSL 25 9 0 01 Jun 2023
Universality and Limitations of Prompt Tuning Yihan Wang Jatin Chauhan Wei Wang Cho-Jui Hsieh 29 17 0 30 May 2023
Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrations Chenglei Si Dan Friedman Nitish Joshi Shi Feng Danqi Chen He He 8 42 0 22 May 2023
A Latent Space Theory for Emergent Abilities in Large Language Models Hui Jiang LRM 21 35 0 19 Apr 2023
Progressive-Hint Prompting Improves Reasoning in Large Language Models Chuanyang Zheng Zhengying Liu Enze Xie Zhenguo Li Yu Li LLMAG ReLM LRM 19 100 0 19 Apr 2023
The Learnability of In-Context Learning Noam Wies Yoav Levine Amnon Shashua 114 89 0 14 Mar 2023
PromptFusion: Decoupling Stability and Plasticity for Continual Learning Haoran Chen Zuxuan Wu Xintong Han Menglin Jia Yu-Gang Jiang CLL 104 12 0 13 Mar 2023
An Overview on Language Models: Recent Developments and Outlook Chengwei Wei Yun Cheng Wang Bin Wang C.-C. Jay Kuo 10 41 0 10 Mar 2023
Dynamic Prompting: A Unified Framework for Prompt Tuning Xianjun Yang Wei Cheng Xujiang Zhao Wenchao Yu Linda R. Petzold Haifeng Chen VLM 17 14 0 06 Mar 2023
On the Provable Advantage of Unsupervised Pretraining Jiawei Ge Shange Tang Jianqing Fan Chi Jin SSL 25 16 0 02 Mar 2023
Almanac: Retrieval-Augmented Language Models for Clinical Medicine C. Zakka Akash Chaurasia R. Shad Alex R. Dalal Jennifer L. Kim ... Kathleen Boyd Karen Hirsch C. Langlotz Joanna Nelson W. Hiesinger LM&MA 94 135 0 01 Mar 2023
Hiding Data Helps: On the Benefits of Masking for Sparse Coding Muthuraman Chidambaram Chenwei Wu Yu Cheng Rong Ge 8 0 0 24 Feb 2023
Task-Specific Skill Localization in Fine-tuned Language Models A. Panigrahi Nikunj Saunshi Haoyu Zhao Sanjeev Arora MoMe 21 66 0 13 Feb 2023
Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning Xinyi Wang Wanrong Zhu Michael Stephen Saxon Mark Steyvers William Yang Wang BDL 40 89 0 27 Jan 2023
A Stability Analysis of Fine-Tuning a Pre-Trained Model Z. Fu Anthony Man-Cho So Nigel Collier 17 3 0 24 Jan 2023
Reasoning with Language Model Prompting: A Survey Shuofei Qiao Yixin Ou Ningyu Zhang Xiang Chen Yunzhi Yao Shumin Deng Chuanqi Tan Fei Huang Huajun Chen ReLM ELM LRM 44 307 0 19 Dec 2022
Pivotal Role of Language Modeling in Recommender Systems: Enriching Task-specific and Task-agnostic Representation Learning Kyuyong Shin Hanock Kwak Wonjae Kim Jisu Jeong Seungjae Jung KyungHyun Kim Jung-Woo Ha Sang-Woo Lee 14 4 0 07 Dec 2022
A Theoretical Study of Inductive Biases in Contrastive Learning Jeff Z. HaoChen Tengyu Ma UQCV SSL 23 31 0 27 Nov 2022
Prototypical Fine-tuning: Towards Robust Performance Under Varying Data Sizes Yiqiao Jin Xiting Wang Y. Hao Yizhou Sun Xing Xie 17 11 0 24 Nov 2022
Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models Hong Liu Sang Michael Xie Zhiyuan Li Tengyu Ma AI4CE 24 49 0 25 Oct 2022
Enhancing Tabular Reasoning with Pattern Exploiting Training Abhilash Shankarampeta Vivek Gupta Shuo Zhang LMTD RALM ReLM 58 6 0 21 Oct 2022
Improving the Sample Efficiency of Prompt Tuning with Domain Adaptation Xu Guo Boyang Albert Li Han Yu VLM 31 22 0 06 Oct 2022
Complexity-Based Prompting for Multi-Step Reasoning Yao Fu Hao-Chun Peng Ashish Sabharwal Peter Clark Tushar Khot ReLM LRM 158 411 0 03 Oct 2022