SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

19 August 2018

Papers citing "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing"

50 / 1,923 papers shown

Title
Revisiting N-Gram Models: Their Impact in Modern Neural Networks for Handwritten Text Recognition Solène Tarride Christopher Kermorvant 37 1 0 30 Apr 2024
Unknown Script: Impact of Script on Cross-Lingual Transfer Wondimagegnhue Tufa Ilia Markov Piek Vossen 45 0 0 29 Apr 2024
Decoding Radiologists' Intentions: A Novel System for Accurate Region Identification in Chest X-ray Image Analysis Akash Awasthi Safwan Ahmad Bryant Le Hien Nguyen 23 0 0 29 Apr 2024
A cost minimization approach to fix the vocabulary size in a tokenizer for an End-to-End ASR system Sunil Kumar Kopparapu Ashish Panda 31 0 0 29 Apr 2024
PatentGPT: A Large Language Model for Intellectual Property Zilong Bai Ruiji Zhang Linqing Chen Qijun Cai Yuan Zhong ... Fu Bian Xiaolong Gu Lisha Zhang Weilei Wang Changyang Tu 49 5 0 28 Apr 2024
Can Perplexity Predict Fine-Tuning Performance? An Investigation of Tokenization Effects on Sequential Language Models for Nepali Nishant Luitel Nirajan Bekoju Anand Kumar Sah Subarna Shakya 58 1 0 28 Apr 2024
Scaffold-BPE: Enhancing Byte Pair Encoding with Simple and Effective Scaffold Token Removal Haoran Lian Yizhe Xiong Jianwei Niu Shasha Mo Zhenpeng Su Zijia Lin Peng Liu Hui Chen Guiguang Ding 44 1 0 27 Apr 2024
Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities Kazuki Fujii Taishi Nakamura Mengsay Loem Hiroki Iida Masanari Ohi Kakeru Hattori Hirai Shota Sakae Mizuki Rio Yokota Naoaki Okazaki CLL 41 55 0 27 Apr 2024
Prefix Text as a Yarn: Eliciting Non-English Alignment in Foundation Language Model Runzhe Zhan Xinyi Yang Derek F. Wong Lidia S. Chao Yue Zhang 58 10 0 25 Apr 2024
Nyonic Technical Report Junfeng Tian Rui-cang Wang Cong Li Yudong Zhou Jun Liu Jun Wang 41 0 0 24 Apr 2024
Multi-Head Mixture-of-Experts Xun Wu Shaohan Huang Wenhui Wang Furu Wei MoE 47 12 0 23 Apr 2024
SpaceByte: Towards Deleting Tokenization from Large Language Modeling Kevin Slagle 37 3 0 22 Apr 2024
Less Peaky and More Accurate CTC Forced Alignment by Label Priors Ruizhe Huang Xiaohui Zhang Zhaoheng Ni Li Sun Moto Hira ... Vineel Pratap Sanjeev Khudanpur Shinji Watanabe Daniel Povey Sanjeev Khudanpur 29 4 0 22 Apr 2024
TartuNLP @ SIGTYP 2024 Shared Task: Adapting XLM-RoBERTa for Ancient and Historical Languages Aleksei Dorkin Kairit Sirts 38 1 0 19 Apr 2024
Simultaneous Interpretation Corpus Construction by Large Language Models in Distant Language Pair Yusuke Sakai Mana Makinae Hidetaka Kamigaito Taro Watanabe 40 4 0 18 Apr 2024
Neuron Specialization: Leveraging intrinsic task modularity for multilingual machine translation Shaomu Tan Di Wu Christof Monz MoMe 36 8 0 17 Apr 2024
Language Model Cascades: Token-level uncertainty and beyond Neha Gupta Harikrishna Narasimhan Wittawat Jitkrittum A. S. Rawat A. Menon Sanjiv Kumar UQLM 53 42 0 15 Apr 2024
TrafficVLM: A Controllable Visual Language Model for Traffic Video Captioning Quang Minh Dinh Minh Khoi Ho Anh Quan Dang Hung Phong Tran 45 6 0 14 Apr 2024
TransformerFAM: Feedback attention is working memory Dongseong Hwang Weiran Wang Zhuoyuan Huo K. Sim P. M. Mengibar 40 12 0 14 Apr 2024
The Role of Language Imbalance in Cross-lingual Generalisation: Insights from Cloned Language Experiments Anton Schäfer Shauli Ravfogel Thomas Hofmann Tiago Pimentel Imanol Schlag 68 3 0 11 Apr 2024
RecurrentGemma: Moving Past Transformers for Efficient Open Language Models Aleksandar Botev Soham De Samuel L. Smith Anushan Fernando George-Christian Muraru ... Koray Kavukcuoglu Demis Hassabis R. Hadsell Yee Whye Teh Nando de Frietas VLM RALM 37 28 0 11 Apr 2024
Interactive Prompt Debugging with Sequence Salience Ian Tenney Ryan Mullins Bin Du Shree Pandya Minsuk Kahng Lucas Dixon LRM 40 1 0 11 Apr 2024
High-Dimension Human Value Representation in Large Language Models Samuel Cahyawijaya Delong Chen Yejin Bang Leila Khalatbari Bryan Wilie Ziwei Ji Etsuko Ishii Pascale Fung 71 5 0 11 Apr 2024
Analyzing the Performance of Large Language Models on Code Summarization Rajarshi Haldar J. Hockenmaier 46 18 0 10 Apr 2024
On the Effect of (Near) Duplicate Subwords in Language Modelling Anton Schäfer Thomas Hofmann Imanol Schlag Tiago Pimentel 44 1 0 09 Apr 2024
Towards Robust Domain Generation Algorithm Classification Arthur Drichel Marc Meyer Ulrike Meyer AAML 44 3 0 09 Apr 2024
Interplay of Machine Translation, Diacritics, and Diacritization Wei-Rui Chen Ife Adebara Muhammad Abdul-Mageed 49 0 0 09 Apr 2024
Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model Xinrun Du Zhouliang Yu Songyang Gao Ding Pan Yuyang Cheng ... Tianyu Zheng Xinchen Luo Guorui Zhou Wenhu Chen Ge Zhang 48 17 0 05 Apr 2024
Training LLMs over Neurally Compressed Text Brian Lester Jaehoon Lee A. Alemi Jeffrey Pennington Adam Roberts Jascha Narain Sohl-Dickstein Noah Constant 45 6 0 04 Apr 2024
SemGrasp: Semantic Grasp Generation via Language Aligned Discretization Kailin Li Jingbo Wang Lixin Yang Cewu Lu Bo Dai 48 16 0 04 Apr 2024
Dynamic Neural Control Flow Execution: An Agent-Based Deep Equilibrium Approach for Binary Vulnerability Detection Litao Li Steven H. H. Ding Andrew Walenstein P. Charland Benjamin C. M. Fung 34 0 0 03 Apr 2024
PejorativITy: Disambiguating Pejorative Epithets to Improve Misogyny Detection in Italian Tweets Arianna Muti Federico Ruggeri Cagri Toraman Lorenzo Musetti Samuel Algherini Silvia Ronchi G. Saretto Caterina Zapparoli Alberto Barrón-Cedeño 25 3 0 03 Apr 2024
PhonologyBench: Evaluating Phonological Skills of Large Language Models Ashima Suvarna Harshita Khandelwal Nanyun Peng LM&MA 47 2 0 03 Apr 2024
Revisiting subword tokenization: A case study on affixal negation in large language models Thinh Hung Truong Yulia Otmakhova Karin Verspoor Trevor Cohn Timothy Baldwin 47 2 0 03 Apr 2024
Low-resource neural machine translation with morphological modeling Antoine Nzeyimana 39 4 0 03 Apr 2024
BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognition A. Haliassos Andreas Zinonos Rodrigo Mira Stavros Petridis Maja Pantic VLM SSL AI4TS 47 13 0 02 Apr 2024
MotionChain: Conversational Motion Controllers via Multimodal Prompts Biao Jiang Xin Chen C. Zhang Fukun Yin Zhuoyuan Li Gang Yu Jiayuan Fan VGen LRM 35 10 0 02 Apr 2024
Release of Pre-Trained Models for the Japanese Language Kei Sawada Tianyu Zhao Makoto Shing Kentaro Mitsui Akio Kaga Yukiya Hono Toshiaki Wakatsuki Koh Mitsuda 35 11 0 02 Apr 2024
Scaling Properties of Speech Language Models Santiago Cuervo R. Marxer 31 9 0 31 Mar 2024
A Systematic Analysis of Subwords and Cross-Lingual Transfer in Multilingual Translation Francois Meyer Jan Buys 39 1 0 29 Mar 2024
IDGenRec: LLM-RecSys Alignment with Textual ID Learning Juntao Tan Shuyuan Xu Wenyue Hua Yingqiang Ge Zelong Li Yongfeng Zhang 51 23 0 27 Mar 2024
CYCLE: Learning to Self-Refine the Code Generation Yangruibo Ding Marcus J. Min Gail E. Kaiser Baishakhi Ray 41 29 0 27 Mar 2024
Homogeneous Tokenizer Matters: Homogeneous Visual Tokenizer for Remote Sensing Image Understanding Run Shao Zhaoyang Zhang Chao Tao Yunsheng Zhang Chengli Peng Haifeng Li VLM 48 5 0 27 Mar 2024
Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction Inhwan Bae Junoh Lee Hae-Gon Jeon 36 15 0 27 Mar 2024
mALBERT: Is a Compact Multilingual BERT Model Still Worth It? Christophe Servan Sahar Ghannay Sophie Rosset 44 0 0 27 Mar 2024
Provably Secure Disambiguating Neural Linguistic Steganography Yuang Qi Kejiang Chen Kai Zeng Weiming Zhang Neng H. Yu 26 2 0 26 Mar 2024
Making Sentence Embeddings Robust to User-Generated Content Lydia Nishimwe Benoît Sagot Rachel Bawden 3DV 33 1 0 25 Mar 2024
Understanding Emergent Abilities of Language Models from the Loss Perspective Zhengxiao Du Aohan Zeng Yuxiao Dong Jie Tang UQCV LRM 73 47 0 23 Mar 2024
AI for Biomedicine in the Era of Large Language Models Zhenyu Bi Sajib Acharjee Dip Daniel Hajialigol Sindhura Kommu Hanwen Liu Meng Lu Xuan Wang LM&MA AI4CE 34 8 0 23 Mar 2024
Adapprox: Adaptive Approximation in Adam Optimization via Randomized Low-Rank Matrices Pengxiang Zhao Ping Li Yingjie Gu Yi Zheng Stephan Ludger Kölker Zhefeng Wang Xiaoming Yuan 29 1 0 22 Mar 2024