Title
Towards Developmentally Plausible Rewards: Communicative Success as a Learning Signal for Interactive Language Models Lennart Stöpler Rufat Asadli Mitja Nikolaus Ryan Cotterell Alex Warstadt LRM 29 0 0 09 May 2025
SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning Tianjian Li Daniel Khashabi 50 0 0 05 May 2025
Demystifying optimized prompts in language models Rimon Melamed Lucas H. McCabe H. H. Huang 31 0 0 04 May 2025
Don't be lazy: CompleteP enables compute-efficient deep transformers Nolan Dey Bin Claire Zhang Lorenzo Noci Mufan Bill Li Blake Bordelon Shane Bergsma C. Pehlevan Boris Hanin Joel Hestness 35 0 0 02 May 2025
Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition Zhengfu He J. Wang Rui Lin Xuyang Ge Wentao Shu Qiong Tang J. Zhang Xipeng Qiu 68 0 0 29 Apr 2025
Improving Reasoning Performance in Large Language Models via Representation Engineering Bertram Højer Oliver Jarvis Stefan Heinrich LRM 67 1 0 28 Apr 2025
Hierarchical Attention Generates Better Proofs Jianlong Chen Chao Li Yang Yuan Andrew Chi-Chih Yao AIMat LRM 26 0 0 27 Apr 2025
Modes of Sequence Models and Learning Coefficients Zhongtian Chen Daniel Murfet 68 1 0 25 Apr 2025
Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models Tyler A. Chang Benjamin Bergen 38 0 0 21 Apr 2025
Energy-Based Reward Models for Robust Language Model Alignment Anamika Lochab Ruqi Zhang 36 0 0 17 Apr 2025
Outlier dimensions favor frequent tokens in language models Iuri Macocco Nora Graichen Gemma Boleda Marco Baroni 42 0 0 27 Mar 2025
Adaptive Rank Allocation: Speeding Up Modern Transformers with RaNA Adapters Roberto Garcia Jerry Liu Daniel Sorvisto Sabri Eyuboglu 81 0 0 23 Mar 2025
The Lucie-7B LLM and the Lucie Training Dataset: Open resources for multilingual language generation Olivier Gouvert Julie Hunter Jérôme Louradour Christophe Cerisara Evan Dufraisse Yaya Sy Laura Rivière Jean-Pierre Lorré OpenLLM-France community 61 0 0 15 Mar 2025
Exploiting Unstructured Sparsity in Fully Homomorphic Encrypted DNNs Aidan Ferguson Perry Gibson Lara DÁgata Parker McLeod Ferhat Yaman Amitabh Das Ian Colbert José Cano 56 0 0 12 Mar 2025
(How) Do Language Models Track State? Belinda Z. Li Zifan Carl Guo Jacob Andreas LRM 44 0 0 04 Mar 2025
Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuning Anh Tong Thanh Nguyen-Tang Dongeun Lee Duc Nguyen Toan M. Tran David Hall Cheongwoong Kang Jaesik Choi 31 0 0 03 Mar 2025
Predictive Data Selection: The Data That Predicts Is the Data That Teaches Kashun Shum Y. Huang Hongjian Zou Qi Ding Yixuan Liao X. Chen Qian Liu Junxian He 39 2 0 02 Mar 2025
FOReCAst: The Future Outcome Reasoning and Confidence Assessment Benchmark Zhangdie Yuan Zifeng Ding Andreas Vlachos AI4TS 63 0 0 27 Feb 2025
SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters Teng Xiao Yige Yuan Z. Chen Mingxiao Li Shangsong Liang Z. Ren V. Honavar 90 5 0 21 Feb 2025
Repetition Neurons: How Do Language Models Produce Repetitions? Tatsuya Hiraoka Kentaro Inui MILM 57 5 0 21 Feb 2025
DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs Minxuan Lv Zhenpeng Su Leiyu Pan Yizhe Xiong Zijia Lin ... Guiguang Ding Cheng Luo Di Zhang Kun Gai Songlin Hu MoE 39 0 0 18 Feb 2025
Prediction hubs are context-informed frequent tokens in LLMs Beatrix M. G. Nielsen Iuri Macocco Marco Baroni 117 1 0 17 Feb 2025
Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scale Fan Zhou Zengzhi Wang Qian Liu Junlong Li Pengfei Liu ALM 83 14 0 17 Feb 2025
A distributional simplicity bias in the learning dynamics of transformers Riccardo Rende Federica Gerace A. Laio Sebastian Goldt 62 7 0 17 Feb 2025
MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections Da Xiao Qingye Meng Shengping Li Xingyuan Yuan MoE AI4CE 52 0 0 13 Feb 2025
Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputers Siddharth Singh Prajwal Singhania Aditya K. Ranjan John Kirchenbauer Jonas Geiping ... Abhimanyu Hans Manli Shu Aditya Tomar Tom Goldstein A. Bhatele 82 2 0 12 Feb 2025
EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Models Xingrun Xing Zheng Liu Shitao Xiao Boyan Gao Yiming Liang Wanpeng Zhang Haokun Lin Guoqi Li Jiajun Zhang LRM 47 1 0 10 Feb 2025
MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations Kaixuan Huang Jiacheng Guo Zihao Li X. Ji Jiawei Ge ... Yangsibo Huang Chi Jin Xinyun Chen Chiyuan Zhang Mengdi Wang AAML LRM 69 7 0 10 Feb 2025
The Complexity of Learning Sparse Superposed Features with Feedback Akash Kumar 53 0 0 08 Feb 2025
Out-of-Distribution Detection using Synthetic Data Generation Momin Abbas Muneeza Azmat R. Horesh Mikhail Yurochkin 36 1 0 05 Feb 2025
The Pitfalls of "Security by Obscurity" And What They Mean for Transparent AI Peter Hall Olivia Mundahl Sunoo Park 63 0 0 30 Jan 2025
BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models Yibin Wang H. Shi Ligong Han Dimitris N. Metaxas Hao Wang BDL UQLM 93 6 0 28 Jan 2025
Merino: Entropy-driven Design for Generative Language Models on IoT Devices Youpeng Zhao Ming Lin Huadong Tang Qiang Wu Jun Wang 62 0 0 28 Jan 2025
Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs Nicolas Boizard Kevin El Haddad C´eline Hudelot Pierre Colombo 60 14 0 28 Jan 2025
ZETA: Leveraging Z-order Curves for Efficient Top-k Attention Qiuhao Zeng Jerry Huang Peng Lu Gezheng Xu Boxing Chen Charles X. Ling Boyu Wang 39 1 0 24 Jan 2025
The potential -- and the pitfalls -- of using pre-trained language models as cognitive science theories Raj Sanjay Shah Sashank Varma LRM 80 0 0 22 Jan 2025
Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Words Gouki Minegishi Hiroki Furuta Yusuke Iwasawa Y. Matsuo 49 1 0 09 Jan 2025
Large Language Monkeys: Scaling Inference Compute with Repeated Sampling Bradley Brown Jordan Juravsky Ryan Ehrlich Ronald Clark Quoc V. Le Christopher Ré Azalia Mirhoseini ALM LRM 76 207 0 03 Jan 2025
ConTrans: Weak-to-Strong Alignment Engineering via Concept Transplantation Weilong Dong Xinwei Wu Renren Jin Shaoyang Xu Deyi Xiong 45 6 0 31 Dec 2024
Out-of-distribution generalization via composition: a lens through induction heads in Transformers Jiajun Song Zhuoyan Xu Yiqiao Zhong 60 4 0 31 Dec 2024
Tracking the Feature Dynamics in LLM Training: A Mechanistic Study Yang Xu Y. Wang Hao Wang 54 1 0 23 Dec 2024
Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models Xiao Cui Mo Zhu Yulei Qin Liang Xie Wengang Zhou H. Li 75 2 0 19 Dec 2024
Beyond the Safety Bundle: Auditing the Helpful and Harmless Dataset Khaoula Chehbouni Jonathan Colaço-Carr Yash More Jackie CK Cheung G. Farnadi 68 0 0 12 Nov 2024
MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mamba Masakazu Yoshimura Teruaki Hayashi Yota Maeda Mamba 54 2 0 06 Nov 2024
What Goes Into a LM Acceptability Judgment? Rethinking the Impact of Frequency and Length Lindia Tjuatja Graham Neubig Tal Linzen Sophie Hao 26 1 0 04 Nov 2024
TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters Haiyang Wang Yue Fan Muhammad Ferjad Naeem Yongqin Xian J. E. Lenssen Liwei Wang F. Tombari Bernt Schiele 31 2 0 30 Oct 2024
$f$ -PO: Generalizing Preference Optimization with $f$ -divergence Minimization Jiaqi Han Mingjian Jiang Yuxuan Song J. Leskovec Stefano Ermon 43 3 0 29 Oct 2024
Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA Sangmin Bae Adam Fisch Hrayr Harutyunyan Ziwei Ji Seungyeon Kim Tal Schuster KELM 54 5 0 28 Oct 2024
Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptions Yujuan Fu Özlem Uzuner Meliha Yetisgen Fei Xia 33 3 0 24 Oct 2024
Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models Michael Noukhovitch Shengyi Huang Sophie Xhonneux Arian Hosseini Rishabh Agarwal Aaron C. Courville OffRL 74 4 0 23 Oct 2024