Zebra: Extending Context Window with Layerwise Grouped Local-Global
Attention

Zebra: Extending Context Window with Layerwise Grouped Local-Global Attention

14 December 2023

Dong Yu

Papers citing "Zebra: Extending Context Window with Layerwise Grouped Local-Global Attention"

13 / 13 papers shown

Title
Cognitive Memory in Large Language Models Lianlei Shan Shixian Luo Zezhou Zhu Yu Yuan Yong Wu LLMAG KELM 98 1 0 03 Apr 2025
Towards LifeSpan Cognitive Systems Yu Wang Chi Han Tongtong Wu Xiaoxin He Wangchunshu Zhou ... Zexue He Wei Wang Gholamreza Haffari Heng Ji Julian McAuley KELM CLL 91 1 0 20 Sep 2024
The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Serving Pai Zeng Zhenyu Ning Jieru Zhao Weihao Cui Mengwei Xu Liwei Guo Xusheng Chen Yizhou Shan LLMAG 40 4 0 18 May 2024
MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies Shengding Hu Yuge Tu Xu Han Chaoqun He Ganqu Cui ... Chaochao Jia Guoyang Zeng Dahai Li Zhiyuan Liu Maosong Sun MoE 38 282 0 09 Apr 2024
Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding Zhenyu (Allen) Zhang Runjin Chen Shiwei Liu Zhewei Yao Olatunji Ruwase Beidi Chen Xiaoxia Wu Zhangyang Wang 26 26 0 05 Mar 2024
Training-Free Long-Context Scaling of Large Language Models Chen An Fei Huang Jun Zhang Shansan Gong Xipeng Qiu Chang Zhou Lingpeng Kong ALM LRM 32 34 0 27 Feb 2024
M4LE: A Multi-Ability Multi-Range Multi-Task Multi-Domain Long-Context Evaluation Benchmark for Large Language Models Wai-Chung Kwan Xingshan Zeng Yufei Wang Yusen Sun Liangyou Li Lifeng Shang Qun Liu Kam-Fai Wong ELM 89 10 0 30 Oct 2023
LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models Chi Han Qifan Wang Hao Peng Wenhan Xiong Yu Chen Heng Ji Sinong Wang 42 47 0 30 Aug 2023
MultiDoc2Dial: Modeling Dialogues Grounded in Multiple Documents Song Feng S. Patel H. Wan Sachindra Joshi 49 66 0 26 Sep 2021
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 245 695 0 27 Aug 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 248 1,986 0 31 Dec 2020
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 251 2,009 0 28 Jul 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 243 1,815 0 17 Sep 2019