What changes when you randomly choose BPE merge operations? Not much

4 May 2023

Papers citing "What changes when you randomly choose BPE merge operations? Not much"

6 / 6 papers shown

Title
Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance Omer Goldman Avi Caciularu Matan Eyal Kris Cao Idan Szpektor Reut Tsarfaty 43 22 0 10 Mar 2024
Greed is All You Need: An Evaluation of Tokenizer Inference Methods Omri Uzan Craig W. Schmidt Chris Tanner Yuval Pinter 38 14 0 02 Mar 2024
Tokenization Is More Than Compression Craig W. Schmidt Varshini Reddy Haoran Zhang Alec Alameddine Omri Uzan Yuval Pinter Chris Tanner 38 28 0 28 Feb 2024
Tokenization and the Noiseless Channel Vilém Zouhar Clara Meister Juan Luis Gastaldi Li Du Mrinmaya Sachan Ryan Cotterell 25 31 0 29 Jun 2023
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation Yonghui Wu M. Schuster Z. Chen Quoc V. Le Mohammad Norouzi ... Alex Rudnick Oriol Vinyals G. Corrado Macduff Hughes J. Dean AIMat 716 6,743 0 26 Sep 2016
Effective Approaches to Attention-based Neural Machine Translation Thang Luong Hieu H. Pham Christopher D. Manning 218 7,923 0 17 Aug 2015