Found in the Middle: Calibrating Positional Attention Bias Improves Long
Context Utilization

Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization

23 June 2024

Yung-Sung Chuang

Chun-Liang Li

Long T. Le

Alexander Ratner

Papers citing "Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization"

15 / 15 papers shown

Title
Attention Mechanisms Perspective: Exploring LLM Processing of Graph-Structured Data Zhong Guan Likang Wu Hongke Zhao Ming He Jianpin Fan GNN 25 0 0 04 May 2025
Reconstructing Context: Evaluating Advanced Chunking Strategies for Retrieval-Augmented Generation Carlo Merola Jaspinder Singh RALM 49 0 0 28 Apr 2025
END: Early Noise Dropping for Efficient and Effective Context Denoising Hongye Jin Pei Chen Jingfeng Yang Z. Wang Meng-Long Jiang ... X. Zhang Zheng Li Tianyi Liu Huasheng Li Bing Yin 49 0 0 26 Feb 2025
Parallel Key-Value Cache Fusion for Position Invariant RAG Philhoon Oh Jinwoo Shin James Thorne 3DV 49 0 0 13 Jan 2025
Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks? Jonathan Roberts Kai Han Samuel Albanie LLMAG 62 0 0 07 Nov 2024
On the Loss of Context-awareness in General Instruction Fine-tuning Yihan Wang Andrew Bai Nanyun Peng Cho-Jui Hsieh 50 1 0 05 Nov 2024
Towards Fair RAG: On the Impact of Fair Ranking in Retrieval-Augmented Generation To Eun Kim Fernando Diaz 45 2 0 17 Sep 2024
Eliminating Position Bias of Language Models: A Mechanistic Approach Ziqi Wang Hanlin Zhang Xiner Li Kuan-Hao Huang Chi Han Shuiwang Ji Sham Kakade Hao Peng Heng Ji 46 12 0 01 Jul 2024
Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection Akari Asai Zeqiu Wu Yizhong Wang Avirup Sil Hannaneh Hajishirzi RALM 138 600 0 17 Oct 2023
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression Huiqiang Jiang Qianhui Wu Xufang Luo Dongsheng Li Chin-Yew Lin Yuqing Yang Lili Qiu RALM 96 179 0 10 Oct 2023
Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading Howard Chen Ramakanth Pasunuru Jason Weston Asli Celikyilmaz RALM 66 72 0 08 Oct 2023
Unlimiformer: Long-Range Transformers with Unlimited Length Input Amanda Bertsch Uri Alon Graham Neubig Matthew R. Gormley RALM 91 122 0 02 May 2023
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 234 690 0 27 Aug 2021
On-the-Fly Attention Modulation for Neural Generation Yue Dong Chandra Bhagavatula Ximing Lu Jena D. Hwang Antoine Bosselut Jackie C.K. Cheung Yejin Choi 35 11 0 02 Jan 2021
Effective Approaches to Attention-based Neural Machine Translation Thang Luong Hieu H. Pham Christopher D. Manning 208 7,687 0 17 Aug 2015