Long-context LLMs Struggle with Long In-context Learning

2 April 2024

Ge Zhang

Papers citing "Long-context LLMs Struggle with Long In-context Learning"

46 / 46 papers shown

Title
Recall with Reasoning: Chain-of-Thought Distillation for Mamba's Long-Context Memory and Extrapolation Junyu Ma Tianqing Fang Z. Zhang Hongming Zhang Haitao Mi Dong Yu ReLM RALM LRM 43 0 0 06 May 2025
Divide, Optimize, Merge: Fine-Grained LLM Agent Optimization at Scale Jiale Liu Yifan Zeng Shaokun Zhang Chi Zhang Malte Højmark-Bertelsen Marie Normann Gadeberg H. Wang Qingyun Wu 34 0 0 06 May 2025
Bye-bye, Bluebook? Automating Legal Procedure with Large Language Models Matthew Dahl AILaw ELM 45 0 0 05 May 2025
Retrieval-augmented in-context learning for multimodal large language models in disease classification Zaifu Zhan Shuang Zhou Xiaoshan Zhou Yongkang Xiao J. Wang Jiawen Deng He Zhu Yu Hou Rui Zhang 36 0 0 04 May 2025
Can LLMs Understand Time Series Anomalies? Zihao Zhou Rose Yu AI4TS 67 8 0 13 Mar 2025
HierarQ: Task-Aware Hierarchical Q-Former for Enhanced Video Understanding Shehreen Azad Vibhav Vineet Y. S. Rawat VLM 52 1 0 11 Mar 2025
LongEval: A Comprehensive Analysis of Long-Text Generation Through a Plan-based Paradigm Siwei Wu Y. Li Xingwei Qu Rishi Ravikumar Y. Li Tyler Loakman Shanghaoran Quan Xiaoyong Wei Shanghaoran Quan Xiaoyong Wei R. Batista-Navarro C. Lin 48 2 0 26 Feb 2025
Self-Memory Alignment: Mitigating Factual Hallucinations with Generalized Improvement Siyuan Zhang Y. Zhang Yinpeng Dong Hang Su HILM KELM 82 0 0 26 Feb 2025
InductionBench: LLMs Fail in the Simplest Complexity Class Wenyue Hua Tyler Wong Sun Fei Liangming Pan Adam Jardine William Yang Wang LRM 63 2 0 20 Feb 2025
ARISE: Iterative Rule Induction and Synthetic Data Generation for Text Classification Y. Meena Vaibhav Singh Ayush Maheshwari Amrith Krishna Ganesh Ramakrishnan AI4TS 53 0 0 09 Feb 2025
Context-Aware Hierarchical Merging for Long Document Summarization Litu Ou Mirella Lapata MoMe 88 1 0 03 Feb 2025
CBVLM: Training-free Explainable Concept-based Large Vision Language Models for Medical Image Classification Cristiano Patrício Isabel Rio-Torto J. S. Cardoso Luís F. Teixeira João C. Neves VLM 116 0 0 21 Jan 2025
Is Long Context All You Need? Leveraging LLM's Extended Context for NL2SQL Yeounoh Chung Gaurav Tarlok Kakkar Yu Gan Brenton Milne Fatma Ozcan RALM 56 5 0 21 Jan 2025
Revisiting In-Context Learning with Long Context Language Models Jinheon Baek Sun Jae Lee Prakhar Gupta Geunseob Oh Siddharth Dalmia 86 0 0 22 Dec 2024
AutoPrep: Natural Language Question-Aware Data Preparation with a Multi-Agent Framework Meihao Fan Ju Fan Nan Tang Lei Cao Guoliang Li Xiaoyong Du LMTD 115 0 0 10 Dec 2024
What is Wrong with Perplexity for Long-context Language Modeling? Lizhe Fang Yifei Wang Zhaoyang Liu Chenheng Zhang Stefanie Jegelka Jinyang Gao Bolin Ding Yisen Wang 49 4 0 31 Oct 2024
From Single to Multi: How LLMs Hallucinate in Multi-Document Summarization Catarina G. Belem Pouya Pezeskhpour Hayate Iso Seiji Maekawa Nikita Bhutani Estevam R. Hruschka HILM 65 1 0 17 Oct 2024
In-Context Learning Enables Robot Action Prediction in LLMs Yida Yin Zekai Wang Yuvan Sharma Dantong Niu Trevor Darrell Roei Herzig LM&Ro 44 1 0 16 Oct 2024
Let's Ask GNN: Empowering Large Language Model for Graph In-Context Learning Zhengyu Hu Yichuan Li Zhengyu Chen J. Wang Han Liu Kyumin Lee Kaize Ding GNN 82 1 0 09 Oct 2024
Narrative-of-Thought: Improving Temporal Reasoning of Large Language Models via Recounted Narratives Xinliang Frederick Zhang Nick Beauchamp Lu Wang LRM AI4CE 24 3 0 07 Oct 2024
In-context Learning in Presence of Spurious Correlations Hrayr Harutyunyan R. Darbinyan Samvel Karapetyan Hrant Khachatrian LRM 30 1 0 04 Oct 2024
How to Train Long-Context Language Models (Effectively) Tianyu Gao Alexander Wettig Howard Yen Danqi Chen RALM 62 36 0 03 Oct 2024
HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly Howard Yen Tianyu Gao Minmin Hou Ke Ding Daniel Fleischer Peter Izsak Moshe Wasserblat Danqi Chen ALM ELM 46 24 0 03 Oct 2024
Bridging Context Gaps: Leveraging Coreference Resolution for Long Contextual Understanding Yanming Liu Xinyue Peng Jiannan Cao Shi Bo Yanxin Shen Tianyu Du Sheng Cheng Xun Wang Jianwei Yin Xuhong Zhang 45 9 0 02 Oct 2024
From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future Haolin Jin Linghan Huang Haipeng Cai Jun Yan Bo Li Huaming Chen 53 24 0 05 Aug 2024
Grammar-based Game Description Generation using Large Language Models Tsunehiko Tanaka Edgar Simo-Serra 39 2 0 24 Jul 2024
Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting Zilong Wang Zifeng Wang Long Le Huaixiu Steven Zheng Swaroop Mishra ... Anush Mattapalli Ankur Taly Jingbo Shang Chen-Yu Lee Tomas Pfister RALM 70 30 0 11 Jul 2024
KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches Jiayi Yuan Hongyi Liu Shaochen Zhong Yu-Neng Chuang ... Hongye Jin V. Chaudhary Zhaozhuo Xu Zirui Liu Xia Hu 28 17 0 01 Jul 2024
DeciMamba: Exploring the Length Extrapolation Potential of Mamba Assaf Ben-Kish Itamar Zimerman Shady Abu Hussein Nadav Cohen Amir Globerson Lior Wolf Raja Giryes Mamba 58 12 0 20 Jun 2024
BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack Yuri Kuratov Aydar Bulatov Petr Anokhin Ivan Rodkin Dmitry Sorokin Artyom Sorokin Mikhail Burtsev RALM ALM LRM ReLM ELM 39 57 0 14 Jun 2024
Chain of Agents: Large Language Models Collaborating on Long-Context Tasks Yusen Zhang Ruoxi Sun Yanfei Chen Tomas Pfister Rui Zhang Sercan Ö. Arik RALM AI4CE LLMAG 42 28 0 04 Jun 2024
Mitigate Position Bias in Large Language Models via Scaling a Single Dimension Yijiong Yu Huiqiang Jiang Xufang Luo Qianhui Wu Chin-Yew Lin Dongsheng Li Yuqing Yang Yongfeng Huang L. Qiu 35 9 0 04 Jun 2024
Is In-Context Learning Sufficient for Instruction Following in LLMs? Hao Zhao Maksym Andriushchenko Francesco Croce Nicolas Flammarion 52 12 0 30 May 2024
(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts Minghao Wu Jiahao Xu Yulin Yuan Gholamreza Haffari Longyue Wang Weihua Luo Kaifu Zhang LLMAG 111 22 0 20 May 2024
Lean Attention: Hardware-Aware Scalable Attention Mechanism for the Decode-Phase of Transformers Rya Sanovar Srikant Bharadwaj Renée St. Amant Victor Rühle Saravan Rajmohan 44 6 0 17 May 2024
Gemma: Open Models Based on Gemini Research and Technology Gemma Team Gemma Team Thomas Mesnard Cassidy Hardin Robert Dadashi Surya Bhupatiraju ... Armand Joulin Noah Fiedel Evan Senter Alek Andreev Kathleen Kenealy VLM LLMAG 123 415 0 13 Mar 2024
E^2-LLM: Efficient and Extreme Length Extension of Large Language Models Jiaheng Liu Zhiqi Bai Yuanxing Zhang Chenchen Zhang Yu Zhang ... Wenbo Su Tiezheng Ge Jie Fu Wenhu Chen Bo Zheng 38 8 0 13 Jan 2024
PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training Dawei Zhu Nan Yang Liang Wang Yifan Song Wenhao Wu Furu Wei Sujian Li 52 77 0 19 Sep 2023
On the Creativity of Large Language Models Giorgio Franceschelli Mirco Musolesi 54 48 0 27 Mar 2023
Resurrecting Recurrent Neural Networks for Long Sequences Antonio Orvieto Samuel L. Smith Albert Gu Anushan Fernando Çağlar Gülçehre Razvan Pascanu Soham De 83 258 0 11 Mar 2023
PoliGraph: Automated Privacy Policy Analysis using Knowledge Graphs (Journal Version) Hao Cui R. Trimananda A. Markopoulou Scott Jordan 38 17 0 13 Oct 2022
GLM-130B: An Open Bilingual Pre-trained Model Aohan Zeng Xiao Liu Zhengxiao Du Zihan Wang Hanyu Lai ... Jidong Zhai Wenguang Chen Peng-Zhen Zhang Yuxiao Dong Jie Tang BDL LRM 240 1,070 0 05 Oct 2022
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 234 690 0 27 Aug 2021
What Makes Good In-Context Examples for GPT- $3$ ? Jiachang Liu Dinghan Shen Yizhe Zhang Bill Dolan Lawrence Carin Weizhu Chen AAML RALM 275 1,296 0 17 Jan 2021
Efficient Intent Detection with Dual Sentence Encoders I. Casanueva Tadas Temvcinas D. Gerz Matthew Henderson Ivan Vulić VLM 164 444 0 10 Mar 2020
SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents Ramesh Nallapati Feifei Zhai Bowen Zhou 190 1,249 0 14 Nov 2016