ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding

23 May 2023

Papers citing "ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding"

50 / 109 papers shown

Title
MedOdyssey: A Medical Domain Benchmark for Long Context Evaluation Up to 200K Tokens Yongqi Fan Hongli Sun Kui Xue Xiaofan Zhang Shaoting Zhang Tong Ruan 34 0 0 21 Jun 2024
Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell Taiming Lu Muhan Gao Kuai Yu Adam Byerly Daniel Khashabi 37 11 0 20 Jun 2024
Can Few-shot Work in Long-Context? Recycling the Context to Generate Demonstrations Arie Cattan Alon Jacovi Alex Fabrikant Jonathan Herzig Roee Aharoni ... Dror Marcus Avinatan Hassidim Yossi Matias Idan Szpektor Avi Caciularu RALM 34 0 0 19 Jun 2024
What Kinds of Tokens Benefit from Distant Text? An Analysis on Long Context Language Modeling Yutong Hu Quzhe Huang Kangcheng Luo Yansong Feng 48 1 0 17 Jun 2024
BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack Yuri Kuratov Aydar Bulatov Petr Anokhin Ivan Rodkin Dmitry Sorokin Artyom Sorokin Mikhail Burtsev RALM ALM LRM ReLM ELM 42 57 0 14 Jun 2024
Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling Liliang Ren Yang Liu Yadong Lu Yelong Shen Chen Liang Weizhu Chen Mamba 64 55 0 11 Jun 2024
RepoQA: Evaluating Long Context Code Understanding Jiawei Liu Jia Le Tian Vijay Daita Yuxiang Wei Yifeng Ding Yuhan Katherine Wang Jun Yang Lingming Zhang LLMAG 31 17 0 10 Jun 2024
Chain of Agents: Large Language Models Collaborating on Long-Context Tasks Yusen Zhang Ruoxi Sun Yanfei Chen Tomas Pfister Rui Zhang Sercan Ö. Arik RALM AI4CE LLMAG 42 28 0 04 Jun 2024
Block Transformer: Global-to-Local Language Modeling for Fast Inference Namgyu Ho Sangmin Bae Taehyeon Kim Hyunjik Jo Yireun Kim Tal Schuster Adam Fisch James Thorne Se-Young Yun 45 6 0 04 Jun 2024
Analyzing Temporal Complex Events with Large Language Models? A Benchmark towards Temporal, Long Context Understanding Zhihan Zhang Yixin Cao Chenchen Ye Yunshan Ma Lizi Liao Tat-Seng Chua 29 9 0 04 Jun 2024
You Only Cache Once: Decoder-Decoder Architectures for Language Models Yutao Sun Li Dong Yi Zhu Shaohan Huang Wenhui Wang Shuming Ma Quanlu Zhang Jianyong Wang Furu Wei VLM 25 52 0 08 May 2024
IceFormer: Accelerated Inference with Long-Sequence Transformers on CPUs Yuzhen Mao Martin Ester Ke Li 22 6 0 05 May 2024
Make Your LLM Fully Utilize the Context Shengnan An Zexiong Ma Zeqi Lin Nanning Zheng Jian-Guang Lou SyDa 44 52 0 25 Apr 2024
When Fuzzing Meets LLMs: Challenges and Opportunities Yu Jiang Jie Liang Fuchen Ma Yuanliang Chen Chijin Zhou ... Zhiyong Wu Jingzhou Fu Mingzhe Wang ShanShan Li Quan Zhang 34 8 0 25 Apr 2024
A Survey on the Memory Mechanism of Large Language Model based Agents Zeyu Zhang Xiaohe Bo Chen Ma Rui Li Xu Chen Quanyu Dai Jieming Zhu Zhenhua Dong Ji-Rong Wen LLMAG KELM 34 105 0 21 Apr 2024
Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs Woomin Song Seunghyuk Oh Sangwoo Mo Jaehyung Kim Sukmin Yun Jung-Woo Ha Jinwoo Shin 28 14 0 16 Apr 2024
DesignQA: A Multimodal Benchmark for Evaluating Large Language Models' Understanding of Engineering Documentation Anna C. Doris Daniele Grandi Ryan Tomich Md Ferdous Alam Hyunmin Cheong Faez Ahmed 36 14 0 11 Apr 2024
Select and Summarize: Scene Saliency for Movie Script Summarization Rohit Saxena Frank Keller 19 2 0 04 Apr 2024
Self-Organized Agents: A LLM Multi-Agent Framework toward Ultra Large-Scale Code Generation and Optimization Yoichi Ishibashi Yoshimasa Nishimura 21 30 0 02 Apr 2024
LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression Zhuoshi Pan Qianhui Wu Huiqiang Jiang Menglin Xia Xufang Luo ... Yuqing Yang Chin-Yew Lin H. V. Zhao Lili Qiu Dongmei Zhang VLM 35 88 0 19 Mar 2024
NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens Cunxiang Wang Ruoxi Ning Boqi Pan Tonghui Wu Qipeng Guo ... Guangsheng Bao Xiangkun Hu Zheng Zhang Qian Wang Yue Zhang RALM 71 3 0 18 Mar 2024
CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models Zexuan Qiu Jingjing Li Shijue Huang Wanjun Zhong Irwin King ELM ALM 45 3 0 06 Mar 2024
Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding Zhenyu (Allen) Zhang Runjin Chen Shiwei Liu Zhewei Yao Olatunji Ruwase Beidi Chen Xiaoxia Wu Zhangyang Wang 21 26 0 05 Mar 2024
Resonance RoPE: Improving Context Length Generalization of Large Language Models Suyuchen Wang I. Kobyzev Peng Lu Mehdi Rezagholizadeh Bang Liu 19 11 0 29 Feb 2024
Long-Context Language Modeling with Parallel Context Encoding Howard Yen Tianyu Gao Danqi Chen 27 42 0 26 Feb 2024
FanOutQA: A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Models Andrew Zhu Alyssa Hwang Liam Dugan Chris Callison-Burch ELM 40 0 0 21 Feb 2024
Fine-Grained Modeling of Narrative Context: A Coherence Perspective via Retrospective Questions Liyan Xu JiangNan Li Mo Yu Jie Zhou 25 3 0 21 Feb 2024
Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models Mosh Levy Alon Jacoby Yoav Goldberg 30 67 0 19 Feb 2024
In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Miss Yuri Kuratov Aydar Bulatov Petr Anokhin Dmitry Sorokin Artyom Sorokin Mikhail Burtsev RALM 117 32 0 16 Feb 2024
Data Engineering for Scaling Language Models to 128K Context Yao Fu Rameswar Panda Xinyao Niu Xiang Yue Hanna Hajishirzi Yoon Kim Hao-Chun Peng MoE 39 115 0 15 Feb 2024
AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers Reduan Achtibat Sayed Mohammad Vakilzadeh Hatefi Maximilian Dreyer Aakriti Jain Thomas Wiegand Sebastian Lapuschkin Wojciech Samek 12 24 0 08 Feb 2024
LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K Tao Yuan Xuefei Ning Dong Zhou Zhijie Yang Shiyao Li ... Dahua Lin Boxun Li Guohao Dai Shengen Yan Yu-Xiang Wang ALM 31 34 0 06 Feb 2024
PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models Haochen Tan Zhijiang Guo Zhan Shi Lu Xu Zhili Liu ... Xiaoguang Li Yasheng Wang Lifeng Shang Qun Liu Linqi Song 25 12 0 26 Jan 2024
The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Survey Saurav Pawar S.M. Towhidul Islam Tonmoy S. M. M. Zaman Vinija Jain Aman Chadha Amitava Das 24 24 0 15 Jan 2024
Transformers are Multi-State RNNs Matanel Oren Michael Hassid Nir Yarden Yossi Adi Roy Schwartz OffRL 19 34 0 11 Jan 2024
Zebra: Extending Context Window with Layerwise Grouped Local-Global Attention Kaiqiang Song Xiaoyang Wang Sangwoo Cho Xiaoman Pan Dong Yu 13 7 0 14 Dec 2023
Universal Self-Consistency for Large Language Model Generation Xinyun Chen Renat Aksitov Uri Alon Jie Jessie Ren Kefan Xiao Pengcheng Yin Sushant Prakash Charles Sutton Xuezhi Wang Denny Zhou LRM 24 65 0 29 Nov 2023
ChatGPT's One-year Anniversary: Are Open-Source Large Language Models Catching up? Hailin Chen Fangkai Jiao Xingxuan Li Chengwei Qin Mathieu Ravaut Ruochen Zhao Caiming Xiong Shafiq R. Joty ELM CLL AI4MH LRM ALM 77 27 0 28 Nov 2023
LooGLE: Can Long-Context Language Models Understand Long Contexts? Jiaqi Li Mengmeng Wang Zilong Zheng Muhan Zhang ELM RALM 24 106 0 08 Nov 2023
M4LE: A Multi-Ability Multi-Range Multi-Task Multi-Domain Long-Context Evaluation Benchmark for Large Language Models Wai-Chung Kwan Xingshan Zeng Yufei Wang Yusen Sun Liangyou Li Lifeng Shang Qun Liu Kam-Fai Wong ELM 89 10 0 30 Oct 2023
S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models Fangyu Lei Qian Liu Yiming Huang Shizhu He Jun Zhao Kang Liu ELM LRM 25 12 0 23 Oct 2023
AutoMix: Automatically Mixing Language Models Pranjal Aggarwal Aman Madaan Ankit Anand Srividya Pranavi Potharaju Swaroop Mishra ... Karthik Kappaganthu Yiming Yang Shyam Upadhyay Manaal Faruqui Mausam 40 17 0 19 Oct 2023
On Context Utilization in Summarization with Large Language Models Mathieu Ravaut Aixin Sun Nancy F. Chen Shafiq R. Joty 25 13 0 16 Oct 2023
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression Huiqiang Jiang Qianhui Wu Xufang Luo Dongsheng Li Chin-Yew Lin Yuqing Yang Lili Qiu RALM 101 179 0 10 Oct 2023
Retrieval meets Long Context Large Language Models Peng-Tao Xu Wei Ping Xianchao Wu Lawrence C. McAfee Chen Zhu Zihan Liu Sandeep Subramanian Evelina Bakhturina M. Shoeybi Bryan Catanzaro RALM LRM 14 79 0 04 Oct 2023
Effective Long-Context Scaling of Foundation Models Wenhan Xiong Jingyu Liu Igor Molybog Hejia Zhang Prajjwal Bhargava ... Dániel Baráth Sergey Edunov Mike Lewis Sinong Wang Hao Ma 26 202 0 27 Sep 2023
BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models Zican Dong Tianyi Tang Junyi Li Wayne Xin Zhao Ji-Rong Wen RALM ALM 20 34 0 23 Sep 2023
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding Yushi Bai Xin Lv Jiajie Zhang Hong Lyu Jiankai Tang ... Aohan Zeng Lei Hou Yuxiao Dong Jie Tang Juanzi Li LLMAG RALM 26 486 0 28 Aug 2023
L-Eval: Instituting Standardized Evaluation for Long Context Language Models Chen An Shansan Gong Ming Zhong Xingjian Zhao Mukai Li Jun Zhang Lingpeng Kong Xipeng Qiu ELM ALM 30 132 0 20 Jul 2023
Lost in the Middle: How Language Models Use Long Contexts Nelson F. Liu Kevin Lin John Hewitt Ashwin Paranjape Michele Bevilacqua Fabio Petroni Percy Liang RALM 27 1,380 0 06 Jul 2023