SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

19 August 2018

Papers citing "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing"

50 / 1,923 papers shown

Title
Learning Mutually Informed Representations for Characters and Subwords Yilin Wang Xinyi Hu Matthew R. Gormley 39 0 0 14 Nov 2023
Context Consistency between Training and Testing in Simultaneous Machine Translation M. Zhong Lemao Liu Kehai Chen Mingming Yang Min Zhang LRM 52 0 0 13 Nov 2023
Towards the Law of Capacity Gap in Distilling Language Models Chen Zhang Dawei Song Zheyu Ye Yan Gao ELM 40 20 0 13 Nov 2023
AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs Yassir Fathullah Chunyang Wu Egor Lakomkin Ke Li Junteng Jia Shangguan Yuan Jay Mahadeokar Ozlem Kalinli Christian Fuegen Michael Seltzer LM&MA MLLM AuLLM 29 35 0 12 Nov 2023
ReactionT5: a large-scale pre-trained model towards application of limited reaction data Tatsuya Sagawa Ryosuke Kojima AI4CE 18 7 0 12 Nov 2023
Tamil-Llama: A New Tamil Language Model Based on Llama 2 Abhinand Balachandran 17 32 0 10 Nov 2023
Proceedings of the 5th International Workshop on Reading Music Systems Jorge Calvo-Zaragoza Alexander Pacha Elona Shatri 32 0 0 07 Nov 2023
The Linear Representation Hypothesis and the Geometry of Large Language Models Kiho Park Yo Joong Choe Victor Veitch LLMSV MILM 45 146 0 07 Nov 2023
Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE Zeren Chen Ziqin Wang Zhen Wang Huayang Liu Zhen-fei Yin Si Liu Lu Sheng Wanli Ouyang Yu Qiao Jing Shao MoE 46 7 0 05 Nov 2023
Too Much Information: Keeping Training Simple for BabyLMs Lukas Edman Lisa Bylinina 32 4 0 03 Nov 2023
Server-side Rescoring of Spoken Entity-centric Knowledge Queries for Virtual Assistants Youyuan Zhang Sashank Gondala Thiago Fraga-Silva Christophe Van Gysel 48 2 0 02 Nov 2023
ACES: Translation Accuracy Challenge Sets at WMT 2023 Chantal Amrhein Nikita Moghe Liane Guillou ELM 32 3 0 02 Nov 2023
From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities Md Farhan Ishmam Md Sakib Hossain Shovon M. F. Mridha Nilanjan Dey 56 36 0 01 Nov 2023
The Unreasonable Effectiveness of Random Target Embeddings for Continuous-Output Neural Machine Translation Evgeniia Tokarchuk Vlad Niculae 27 2 0 31 Oct 2023
Towards a Deep Understanding of Multilingual End-to-End Speech Translation Haoran Sun Xiaohu Zhao Yikun Lei Shaolin Zhu Deyi Xiong 44 8 0 31 Oct 2023
Is Robustness Transferable across Languages in Multilingual Neural Machine Translation? Leiyu Pan Supryadi Supryadi Deyi Xiong AAML 31 0 0 31 Oct 2023
CreoleVal: Multilingual Multitask Benchmarks for Creoles Heather Lent Kushal Tatariya Raj Dabre Yiyi Chen Marcell Richard Fekete ... Miryam de Lhoneux Daniel Hershcovich Michel DeGraff Anders Sogaard Johannes Bjerva SLR 43 9 0 30 Oct 2023
Skywork: A More Open Bilingual Foundation Model Tianwen Wei Liang Zhao Lichang Zhang Bo Zhu Lijie Wang ... Yongyi Peng Xiaojuan Liang Shuicheng Yan Han Fang Yahui Zhou 38 93 0 30 Oct 2023
Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention Changjiang Gao Shujian Huang Jixing Li Jiajun Chen LRM ALM 47 7 0 29 Oct 2023
Probing LLMs for Joint Encoding of Linguistic Categories Giulio Starace Konstantinos Papakostas Rochelle Choenni Apostolos Panagiotopoulos Matteo Rosati Alina Leidinger Ekaterina Shutova 27 5 0 28 Oct 2023
Unified Segment-to-Segment Framework for Simultaneous Sequence Generation Shaolei Zhang Yang Feng 30 7 0 27 Oct 2023
Lil-Bevo: Explorations of Strategies for Training Language Models in More Humanlike Ways Venkata S Govindarajan Juan Diego Rodriguez Kaj Bostrom Kyle Mahowald 25 1 0 26 Oct 2023
Learning to Abstract with Nonparametric Variational Information Bottleneck Melika Behjati Fabio Fehr James Henderson SSL 32 1 0 26 Oct 2023
EMMA-X: An EM-like Multilingual Pre-training Algorithm for Cross-lingual Representation Learning Ping Guo Xiangpeng Wei Yue Hu Baosong Yang Dayiheng Liu Fei Huang Jun Xie 28 2 0 26 Oct 2023
CL-MASR: A Continual Learning Benchmark for Multilingual ASR Luca Della Libera Pooneh Mousavi Salah Zaiem Cem Subakan Mirco Ravanelli AuLLM CLL 53 13 0 25 Oct 2023
Enhanced Simultaneous Machine Translation with Word-level Policies Kang Kim Hankyu Cho 61 3 0 25 Oct 2023
Samsung R&D Institute Philippines at WMT 2023 Jan Christian Blaise Cruz 23 6 0 25 Oct 2023
MindLLM: Pre-training Lightweight Large Language Model from Scratch, Evaluations and Domain Applications Yizhe Yang Huashan Sun Jiawei Li Runheng Liu Yinghao Li Yuhang Liu Heyan Huang Yang Gao ALM LRM 16 8 0 24 Oct 2023
A Joint Matrix Factorization Analysis of Multilingual Representations Zheng Zhao Yftah Ziser Bonnie Webber Shay B. Cohen 32 2 0 24 Oct 2023
Leveraging Timestamp Information for Serialized Joint Streaming Recognition and Translation Sara Papi Peidong Wang Junkun Chen Jian Xue Naoyuki Kanda Jinyu Li Yashesh Gaur 34 3 0 23 Oct 2023
Code-Switching with Word Senses for Pretraining in Neural Machine Translation Vivek Iyer Edoardo Barba Alexandra Birch Jeff Z. Pan Roberto Navigli 20 3 0 21 Oct 2023
Ask Language Model to Clean Your Noisy Translation Data Quinten Bolding Baohao Liao Brandon James Denis Jun Luo Christof Monz 35 5 0 20 Oct 2023
The CHiME-7 Challenge: System Description and Performance of NeMo Team's DASR System T. Park He Huang Ante Jukić Kunal Dhawan Krishna C. Puvvada Nithin Rao Koluguri Nikolay Karpov A. Laptev Jagadeesh Balam Boris Ginsburg 37 6 0 18 Oct 2023
Direct Neural Machine Translation with Task-level Mixture of Experts models Isidora Chara Tourni Subhajit Naskar MoE 21 0 0 18 Oct 2023
SPEED: Speculative Pipelined Execution for Efficient Decoding Coleman Hooper Sehoon Kim Hiva Mohammadzadeh Hasan Genç Kurt Keutzer A. Gholami Y. Shao 32 35 0 18 Oct 2023
BUT CHiME-7 system description M. Karafiát Karel Veselý Igor Szöke Ladislav Mošner Karel Beneš Marcin Witkowski Germán Barchi L. Pepino 35 1 0 18 Oct 2023
ChapGTP, ILLC's Attempt at Raising a BabyLM: Improving Data Efficiency by Automatic Task Formation Jaap Jumelet Michael Hanna Marianne de Heer Kloots Anna Langedijk Charlotte Pouw Oskar van der Wal 29 3 0 17 Oct 2023
ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processing Quoc-Nam Nguyen Thang Chau Phan Duc-Vu Nguyen Kiet Van Nguyen 33 8 0 17 Oct 2023
IMTLab: An Open-Source Platform for Building, Evaluating, and Diagnosing Interactive Machine Translation Systems Xu Huang Zhirui Zhang Ruize Gao Yichao Du Lemao Liu Gouping Huang Shuming Shi Jiajun Chen Shujian Huang VLM 26 0 0 17 Oct 2023
Iterative Shallow Fusion of Backward Language Model for End-to-End Speech Recognition A. Ogawa Takafumi Moriya Naoyuki Kamo Naohiro Tawara Marc Delcroix 20 1 0 17 Oct 2023
Approximating Two-Layer Feedforward Networks for Efficient Transformers Róbert Csordás Kazuki Irie Jürgen Schmidhuber MoE 27 18 0 16 Oct 2023
Towards a Better Understanding of Variations in Zero-Shot Neural Machine Translation Performance Shaomu Tan Christof Monz 43 12 0 16 Oct 2023
Optimized Tokenization for Transcribed Error Correction Tomer Wullach Shlomo E. Chazan 32 0 0 16 Oct 2023
Prediction of Arabic Legal Rulings using Large Language Models Adel Ammar Anis Koubaa Bilel Benjdira Omar Najar Serry Sibaee AILaw ELM 25 7 0 16 Oct 2023
End-to-end Multichannel Speaker-Attributed ASR: Speaker Guided Decoder and Input Feature Analysis Can Cui Imran A. Sheikh Mostafa Sadeghi Emmanuel Vincent 29 4 0 16 Oct 2023
Personalization of CTC-based End-to-End Speech Recognition Using Pronunciation-Driven Subword Tokenization Zhihong Lei Ernest Pusateri Shiyi Han Leo Liu Mingbin Xu ... R. Travadi Youyuan Zhang Mirko Hannemann Man-Hung Siu Zhen Huang 23 9 0 16 Oct 2023
UvA-MT's Participation in the WMT23 General Translation Shared Task Di Wu Shaomu Tan David Stap Ali Araabi Christof Monz 32 3 0 15 Oct 2023
Generative Adversarial Training for Text-to-Speech Synthesis Based on Raw Phonetic Input and Explicit Prosody Modelling Tiberiu Boros Stefan Daniel Dumitrescu Ionut Mironica Radu Chivereanu GAN 22 1 0 14 Oct 2023
Embarrassingly Simple Text Watermarks Ryoma Sato Yuki Takezawa Han Bao Kenta Niwa Makoto Yamada WaLM 32 14 0 13 Oct 2023
Tokenizer Choice For LLM Training: Negligible or Crucial? Mehdi Ali Michael Fromm Klaudia Thellmann Richard Rutmann Max Lübbering ... Malte Ostendorff Samuel Weinbach R. Sifa Stefan Kesselheim Nicolas Flores-Herr 28 48 0 12 Oct 2023