Attention Sorting Combats Recency Bias In Long Context Language Models

28 September 2023

Papers citing "Attention Sorting Combats Recency Bias In Long Context Language Models"

7 / 7 papers shown

Title
Eliminating Position Bias of Language Models: A Mechanistic Approach Ziqi Wang Hanlin Zhang Xiner Li Kuan-Hao Huang Chi Han Shuiwang Ji Sham Kakade Hao Peng Heng Ji 49 12 0 01 Jul 2024
Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell Taiming Lu Muhan Gao Kuai Yu Adam Byerly Daniel Khashabi 37 11 0 20 Jun 2024
Mitigate Position Bias in Large Language Models via Scaling a Single Dimension Yijiong Yu Huiqiang Jiang Xufang Luo Qianhui Wu Chin-Yew Lin Dongsheng Li Yuqing Yang Yongfeng Huang L. Qiu 35 9 0 04 Jun 2024
Where is the answer? Investigating Positional Bias in Language Model Knowledge Extraction Kuniaki Saito Kihyuk Sohn Chen-Yu Lee Yoshitaka Ushiku 46 2 0 16 Feb 2024
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 315 8,261 0 28 Jan 2022
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 234 690 0 27 Aug 2021
Distilling Knowledge from Reader to Retriever for Question Answering Gautier Izacard Edouard Grave RALM 170 249 0 08 Dec 2020