SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

19 August 2018

Papers citing "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing"

50 / 1,923 papers shown

Title
Pfeed: Generating near real-time personalized feeds using precomputed embedding similarities B. Gebre Karoliina Ranta S. V. D. Elzen Ernst Kuiper Thijs Baars Tom Heskes 46 1 0 25 Feb 2024
ArabianGPT: Native Arabic GPT-based Large Language Model Anis Koubaa Adel Ammar L. Ghouti Omar Najar Serry Sibaee LM&MA 38 4 0 23 Feb 2024
Representing Online Handwriting for Recognition in Large Vision-Language Models Anastasiia Fadeeva Philippe Schlattner Andrii Maksai Mark Collier Efi Kokiopoulou Jesse Berent C. Musat 54 4 0 23 Feb 2024
Fine-tuning Large Language Models for Domain-specific Machine Translation Jiawei Zheng Hanghai Hong Xiaoli Wang Jingsong Su Yonggui Liang Shikai Wu ALM 52 34 0 23 Feb 2024
How Important Is Tokenization in French Medical Masked Language Models? Yanis Labrak Adrien Bazoge B. Daille Mickael Rouvier Richard Dufour 44 1 0 22 Feb 2024
The Impact of Word Splitting on the Semantic Content of Contextualized Word Representations Aina Garí Soler Matthieu Labeau Chloé Clavel VLM 47 2 0 22 Feb 2024
OmniPred: Language Models as Universal Regressors Xingyou Song Oscar Li Chansoo Lee Bangding Yang Daiyi Peng Sagi Perel Yutian Chen 62 14 0 22 Feb 2024
Subobject-level Image Tokenization Delong Chen Samuel Cahyawijaya Jianfeng Liu Baoyuan Wang Pascale Fung VLM OCL 60 7 0 22 Feb 2024
How do Hyenas deal with Human Speech? Speech Recognition and Translation with ConfHyena Marco Gaido Sara Papi Matteo Negri L. Bentivogli 46 1 0 20 Feb 2024
MORE-3S:Multimodal-based Offline Reinforcement Learning with Shared Semantic Spaces Tianyu Zheng Ge Zhang Xingwei Qu Ming Kuang Stephen W. Huang Zhaofeng He OffRL 58 1 0 20 Feb 2024
Emergent Word Order Universals from Cognitively-Motivated Language Models Tatsuki Kuribayashi Ryo Ueda Ryosuke Yoshida Yohei Oseki Ted Briscoe Timothy Baldwin 44 2 0 19 Feb 2024
Pushing the Limits of Zero-shot End-to-End Speech Translation Ioannis Tsiamas Gerard I. Gállego José A. R. Fonollosa Marta R. Costa-jussá 43 7 0 16 Feb 2024
BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains Yanis Labrak Adrien Bazoge Emmanuel Morin P. Gourraud Mickael Rouvier Richard Dufour 111 197 0 15 Feb 2024
Fast Vocabulary Transfer for Language Model Compression Leonidas Gee Andrea Zugarini Leonardo Rigutini Paolo Torroni 35 27 0 15 Feb 2024
Multi-word Tokenization for Sequence Compression Leonidas Gee Leonardo Rigutini Marco Ernandes Andrea Zugarini 18 8 0 15 Feb 2024
Knowledge of Pretrained Language Models on Surface Information of Tokens Tatsuya Hiraoka Naoaki Okazaki 32 1 0 15 Feb 2024
UniEnc-CASSNAT: An Encoder-only Non-autoregressive ASR for Speech SSL Models Ruchao Fan Natarajan Balaji Shankar Abeer Alwan 41 0 0 14 Feb 2024
Self-consistent context aware conformer transducer for speech recognition Konstantin Kolokolov Pavel Pekichev Karthik Raghunathan 22 0 0 09 Feb 2024
Text-to-Code Generation with Modality-relative Pre-training Fenia Christopoulou Guchun Zhang Gerasimos Lampouras AI4TS 26 1 0 08 Feb 2024
Offline Actor-Critic Reinforcement Learning Scales to Large Models Jost Tobias Springenberg A. Abdolmaleki Jingwei Zhang Oliver Groth Michael Bloesch ... Sarah Bechtle Steven Kapturowski Roland Hafner N. Heess Martin Riedmiller OffRL LRM 35 12 0 08 Feb 2024
Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generation Luca Beurer-Kellner Marc Fischer Martin Vechev 44 38 0 07 Feb 2024
Predicting positive transfer for improved low-resource speech recognition using acoustic pseudo-tokens Nay San Georgios Paraskevopoulos Aryaman Arora Xiluo He Prabhjot Kaur Oliver Adams Dan Jurafsky 42 7 0 03 Feb 2024
Towards Sustainable Workplace Mental Health: A Novel Approach to Early Intervention and Support David W. Vinson Mihael Arcan Paul-David Niland Fionn Delahunty AI4MH 33 1 0 02 Feb 2024
Sequence Shortening for Context-Aware Machine Translation Paweł Mąka Yusuf Can Semerci Jan Scholtes Gerasimos Spanakis 22 2 0 02 Feb 2024
IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition Zi-Jian Leng Amitrajit Bhattacharjee Hrudhai Rajasekhar Lizhe Zhang Elizabeth Bruda Hyeokhyen Kwon Thomas Plötz VLM 41 13 0 01 Feb 2024
Getting the most out of your tokenizer for pre-training and domain adaptation Gautier Dagan Gabriele Synnaeve Baptiste Rozière 36 20 0 01 Feb 2024
Disentangling the Roles of Target-Side Transfer and Regularization in Multilingual Machine Translation Yan Meng Christof Monz LRM 41 2 0 01 Feb 2024
Positional Encoding Helps Recurrent Neural Networks Handle a Large Vocabulary Takashi Morita 26 3 0 31 Jan 2024
EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain Wei Zhang Miaoxin Cai Tong Zhang Zhuang Yin Xuerui Mao 42 92 0 30 Jan 2024
SpeechBERTScore: Reference-Aware Automatic Evaluation of Speech Generation Leveraging NLP Evaluation Metrics Takaaki Saeki Soumi Maiti Shinnosuke Takamichi Shinji Watanabe Hiroshi Saruwatari 30 15 0 30 Jan 2024
TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguese N. Corrêa Sophia Falk Shiza Fatimah Aniket Sen N. D. Oliveira 30 9 0 30 Jan 2024
Byte Pair Encoding Is All You Need For Automatic Bengali Speech Recognition Ahnaf Mozib Samin 20 0 0 28 Jan 2024
Modular Adaptation of Multilingual Encoders to Written Swiss German Dialect Jannis Vamvas Noëmi Aepli Rico Sennrich 34 0 0 25 Jan 2024
TURNA: A Turkish Encoder-Decoder Language Model for Enhanced Understanding and Generation Gokcce Uludougan Zeynep Yirmibecsouglu Balal Furkan Akkurt Melikcsah Turker Onur Gungor S. Uskudarli 39 12 0 25 Jan 2024
MambaByte: Token-free Selective State Space Model Junxiong Wang Tushaar Gangavarapu Jing Nathan Yan Alexander M. Rush Mamba 44 37 0 24 Jan 2024
MaLA-500: Massive Language Adaptation of Large Language Models Peiqin Lin Shaoxiong Ji Jörg Tiedemann André F. T. Martins Hinrich Schütze ELM 36 15 0 24 Jan 2024
Keep Decoding Parallel with Effective Knowledge Distillation from Language Models to End-to-end Speech Recognisers Michael Hentschel Yuta Nishikawa Tatsuya Komatsu Yusuke Fujita 27 4 0 22 Jan 2024
Text-to-Image Cross-Modal Generation: A Systematic Review Maciej Żelaszczyk Jacek Mańdziuk 35 3 0 21 Jan 2024
Instructional Fingerprinting of Large Language Models Lyne Tchapmi Fei Wang Mingyu Derek Ma Pang Wei Koh Chaowei Xiao Muhao Chen WaLM 22 29 0 21 Jan 2024
Orion-14B: Open-source Multilingual Large Language Models Du Chen Yi Huang Xiaopu Li Yongqiang Li Yongqiang Liu Haihui Pan Leichao Xu Dacheng Zhang Zhipeng Zhang Kun Han 35 4 0 20 Jan 2024
Improving fine-grained understanding in image-text pre-training Ioana Bica Anastasija Ilić Matthias Bauer Goker Erdogan Matko Bovsnjak ... A. Gritsenko Matthias Minderer Charles Blundell Razvan Pascanu Jovana Mitrović VLM 30 22 0 18 Jan 2024
Efficient Training for Multilingual Visual Speech Recognition: Pre-training with Discretized Visual Speech Representation Minsu Kim Jeong Hun Yeo Se Jin Park J. Choi Y. Ro 27 5 0 18 Jan 2024
Deciphering Textual Authenticity: A Generalized Strategy through the Lens of Large Language Semantics for Detecting Human vs. Machine-Generated Text Mazal Bethany Brandon Wherry Emet Bethany Nishant Vishwamitra Anthony Rios Peyman Najafirad DeLMO 36 4 0 17 Jan 2024
A Generative Adversarial Attack for Multilingual Text Classifiers Tom Roth Inigo Jauregi Unanue A. Abuadbba Massimo Piccardi AAML 13 0 0 16 Jan 2024
Enhancing Document-level Translation of Large Language Model via Translation Mixed-instructions Yachao Li Junhui Li Jing Jiang Min Zhang 38 9 0 16 Jan 2024
Cross-Attention Watermarking of Large Language Models Folco Bertini Baldassini H. Nguyen Ching-Chung Chang Isao Echizen WaLM 25 1 0 12 Jan 2024
Distilling Vision-Language Models on Millions of Videos Yue Zhao Long Zhao Xingyi Zhou Jialin Wu Chun-Te Chu ... Hartwig Adam Ting Liu Boqing Gong Philipp Krahenbuhl Liangzhe Yuan VLM 41 13 0 11 Jan 2024
A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatars Ronglai Zuo Fangyun Wei Zenggui Chen Brian Mak Jiaolong Yang Xin Tong SLR 36 4 0 09 Jan 2024
Deep Learning in Physical Layer: Review on Data Driven End-to-End Communication Systems and their Enabling Semantic Applications Nazmul Islam Seokjoo Shin AI4CE 34 3 0 08 Jan 2024
An Exploratory Study on Automatic Identification of Assumptions in the Development of Deep Learning Frameworks Chen Yang Peng Liang Zinan Ma 32 0 0 08 Jan 2024