Do Long-Range Language Models Actually Use Long-Range Context?

19 September 2021

Simeng Sun

Kalpesh Krishna

Andrew Mattarella-Micke

Mohit Iyyer

RALM

ArXiv PDF HTML

Papers citing "Do Long-Range Language Models Actually Use Long-Range Context?"

50 / 76 papers shown

Title
A Survey on Transformer Context Extension: Approaches and Evaluation Yijun Liu Jinzheng Yu Yang Xu Zhongyang Li Qingfu Zhu LLMAG 66 0 0 17 Mar 2025
Optimizing open-domain question answering with graph-based retrieval augmented generation Joyce Cahoon Prerna Singh Nick Litombe Jonathan Larson Ha Trinh Yiwen Zhu A. Mueller Fotis Psallidas Carlo Curino 29 0 0 04 Mar 2025
PICLe: Pseudo-Annotations for In-Context Learning in Low-Resource Named Entity Detection Sepideh Mamooler Syrielle Montariol Alexander Mathis Antoine Bosselut 85 1 0 16 Dec 2024
MiniKV: Pushing the Limits of LLM Inference via 2-Bit Layer-Discriminative KV Cache Akshat Sharma Hangliang Ding Jianping Li Neel Dani Minjia Zhang 74 1 0 27 Nov 2024
Long Context RAG Performance of Large Language Models Quinn Leng Jacob P. Portes Sam Havens Matei A. Zaharia Michael Carbin AIFin RALM 3DV 41 8 0 05 Nov 2024
On Positional Bias of Faithfulness for Long-form Summarization David Wan Jesse Vig Mohit Bansal Shafiq R. Joty HILM 48 3 0 31 Oct 2024
What is Wrong with Perplexity for Long-context Language Modeling? Lizhe Fang Yifei Wang Zhaoyang Liu Chenheng Zhang Stefanie Jegelka Jinyang Gao Bolin Ding Yisen Wang 58 4 0 31 Oct 2024
Rolling the DICE on Idiomaticity: How LLMs Fail to Grasp Context Maggie Mi Aline Villavicencio Nafise Sadat Moosavi 45 1 0 21 Oct 2024
From Isolated Conversations to Hierarchical Schemas: Dynamic Tree Memory Representation for LLMs Alireza Rezazadeh Zichao Li Wei Wei Yujia Bao 30 4 0 17 Oct 2024
How much do contextualized representations encode long-range context? Simeng Sun Cheng-Ping Hsieh 39 0 0 16 Oct 2024
Parameter-Efficient Fine-Tuning via Selective Discrete Cosine Transform Yixian Shen Qi Bi Jia-Hong Huang Hongyi Zhu Anuj Pathania 25 1 0 09 Oct 2024
GEM-RAG: Graphical Eigen Memories For Retrieval Augmented Generation B. Rappazzo Yingheng Wang Aaron Ferber Carla P. Gomes VLM 18 0 0 23 Sep 2024
A Controlled Study on Long Context Extension and Generalization in LLMs Yi Lu Jing Nathan Yan Songlin Yang Justin T. Chiu Siyu Ren Fei Yuan Wenting Zhao Zhiyong Wu Alexander M. Rush 31 9 0 18 Sep 2024
Mamba-PTQ: Outlier Channels in Recurrent Large Language Models Alessandro Pierro Steven Abreu MQ Mamba 43 6 0 17 Jul 2024
Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers Chao Lou Zixia Jia Zilong Zheng Kewei Tu ODL 31 18 0 24 Jun 2024
One Thousand and One Pairs: A "novel" challenge for long-context language models Marzena Karpinska Katherine Thai Kyle Lo Tanya Goyal Mohit Iyyer LRM 39 40 0 24 Jun 2024
GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Models Shilong Li Yancheng He Hangyu Guo Xingyuan Bu Ge Bai ... Xingwei Qu Yangguang Li Wanli Ouyang Wenbo Su Bo Zheng RALM LLMAG 40 6 0 20 Jun 2024
HIRO: Hierarchical Information Retrieval Optimization Krish Goel Mahek Chandak RALM 37 1 0 14 Jun 2024
Analyzing Large Language Models for Classroom Discussion Assessment Nhat Tran Benjamin Pierce Diane Litman Richard Correnti L. Matsumura 27 1 0 12 Jun 2024
Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Models Longze Chen Ziqiang Liu Wanwei He Yunshui Li Run Luo Min Yang 29 9 0 28 May 2024
Towards Understanding the Working Mechanism of Text-to-Image Diffusion Model Mingyang Yi Aoxue Li Yi Xin Zhenguo Li DiffM 37 11 0 24 May 2024
SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Sound Haohe Liu Xuenan Xu Yiitan Yuan Mengyue Wu Wenwu Wang Mark D. Plumbley 32 18 0 30 Apr 2024
Make Your LLM Fully Utilize the Context Shengnan An Zexiong Ma Zeqi Lin Nanning Zheng Jian-Guang Lou SyDa 44 54 0 25 Apr 2024
CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models Zexuan Qiu Jingjing Li Shijue Huang Wanjun Zhong Irwin King ELM ALM 45 3 0 06 Mar 2024
Parallel Structures in Pre-training Data Yield In-Context Learning Yanda Chen Chen Zhao Zhou Yu Kathleen McKeown He He 29 11 0 19 Feb 2024
LongHeads: Multi-Head Attention is Secretly a Long Context Processor Yi Lu Xin Zhou Wei He Jun Zhao Tao Ji Tao Gui Qi Zhang Xuanjing Huang LLMAG 39 11 0 16 Feb 2024
Steering Conversational Large Language Models for Long Emotional Support Conversations Navid Madani Sougata Saha R. Srihari 19 3 0 16 Feb 2024
UniMem: Towards a Unified View of Long-Context Large Language Models Junjie Fang Likai Tang Hongzhe Bi Yujia Qin Si Sun ... Xiaodong Shi Sen Song Yankai Lin Zhiyuan Liu Maosong Sun 19 3 0 05 Feb 2024
RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval Parth Sarthi Salman Abdullah Aditi Tuli Shubh Khanna Anna Goldie Christopher D. Manning RALM 19 122 0 31 Jan 2024
The Power of Noise: Redefining Retrieval for RAG Systems Florin Cuconasu Giovanni Trappolini F. Siciliano Simone Filice Cesare Campagnano Y. Maarek Nicola Tonellotto Fabrizio Silvestri RALM 37 143 0 26 Jan 2024
Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMs Haritz Puerto Martin Tutek Somak Aditya Xiaodan Zhu Iryna Gurevych ReCod ReLM LRM 43 9 0 18 Jan 2024
Extending LLMs' Context Window with 100 Samples Yikai Zhang Junlong Li Pengfei Liu 24 11 0 13 Jan 2024
M4LE: A Multi-Ability Multi-Range Multi-Task Multi-Domain Long-Context Evaluation Benchmark for Large Language Models Wai-Chung Kwan Xingshan Zeng Yufei Wang Yusen Sun Liangyou Li Lifeng Shang Qun Liu Kam-Fai Wong ELM 89 10 0 30 Oct 2023
How Much Context Does My Attention-Based ASR System Need? Robert Flynn Anton Ragni 30 1 0 24 Oct 2023
S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models Fangyu Lei Qian Liu Yiming Huang Shizhu He Jun Zhao Kang Liu ELM LRM 25 12 0 23 Oct 2023
Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layer Qingru Zhang Dhananjay Ram Cole Hawkins Sheng Zha Tuo Zhao 27 15 0 19 Oct 2023
On Context Utilization in Summarization with Large Language Models Mathieu Ravaut Aixin Sun Nancy F. Chen Shafiq R. Joty 31 13 0 16 Oct 2023
CacheGen: KV Cache Compression and Streaming for Fast Language Model Serving Yuhan Liu Hanchen Li Yihua Cheng Siddhant Ray Yuyang Huang ... Ganesh Ananthanarayanan Michael Maire Henry Hoffmann Ari Holtzman Junchen Jiang 50 41 0 11 Oct 2023
Transformer-VQ: Linear-Time Transformers via Vector Quantization Albert Mohwald 26 15 0 28 Sep 2023
Attention Sorting Combats Recency Bias In Long Context Language Models A. Peysakhovich Adam Lerer LRM RALM 34 41 0 28 Sep 2023
BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models Zican Dong Tianyi Tang Junyi Li Wayne Xin Zhao Ji-Rong Wen RALM ALM 23 34 0 23 Sep 2023
BatchPrompt: Accomplish more with less Jianzhe Lin Maurice Diesendruck Liang Du Robin Abraham LRM 24 9 0 01 Sep 2023
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding Yushi Bai Xin Lv Jiajie Zhang Hong Lyu Jiankai Tang ... Aohan Zeng Lei Hou Yuxiao Dong Jie Tang Juanzi Li LLMAG RALM 26 492 0 28 Aug 2023
OctoPack: Instruction Tuning Code Large Language Models Niklas Muennighoff Qian Liu A. Zebaze Qinkai Zheng Binyuan Hui Terry Yue Zhuo Swayam Singh Xiangru Tang Leandro von Werra Shayne Longpre VLM ALM 60 117 0 14 Aug 2023
Generative Models as a Complex Systems Science: How can we make sense of large language model behavior? Ari Holtzman Peter West Luke Zettlemoyer AI4CE 23 13 0 31 Jul 2023
L-Eval: Instituting Standardized Evaluation for Long Context Language Models Chen An Shansan Gong Ming Zhong Xingjian Zhao Mukai Li Jun Zhang Lingpeng Kong Xipeng Qiu ELM ALM 37 133 0 20 Jul 2023
Lost in the Middle: How Language Models Use Long Contexts Nelson F. Liu Kevin Lin John Hewitt Ashwin Paranjape Michele Bevilacqua Fabio Petroni Percy Liang RALM 27 1,389 0 06 Jul 2023
Focused Transformer: Contrastive Training for Context Scaling Szymon Tworkowski Konrad Staniszewski Mikolaj Pacek Yuhuai Wu Henryk Michalewski Piotr Milo's 21 135 0 06 Jul 2023
Personality Traits in Large Language Models Gregory Serapio-García Mustafa Safdari Clément Crepy Luning Sun Stephen Fitz P. Romero Marwa Abdulhai Aleksandra Faust Maja J. Matarić LM&MA LLMAG 55 119 0 01 Jul 2023
Statler: State-Maintaining Language Models for Embodied Reasoning Takuma Yoneda Jiading Fang Peng Li Huanyu Zhang Tianchong Jiang Shengjie Lin Ben Picker David Yunis Hongyuan Mei Matthew R. Walter LM&Ro 20 32 0 30 Jun 2023