SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

19 August 2018

Papers citing "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing"

50 / 1,923 papers shown

Title
TokenVerse: Unifying Speech and NLP Tasks via Transducer-based ASR Shashi Kumar S. Madikeri Juan Zuluaga-Gomez Iuliia Nigmatulina Esaú Villatoro-Tello Sergio Burdisso P. Motlícek Karthik Pandia A. Ganapathiraju 46 0 0 05 Jul 2024
Serialized Output Training by Learned Dominance Ying Shi Lantian Li Shi Yin D. Wang Jiqing Han 23 4 0 04 Jul 2024
BM25S: Orders of magnitude faster lexical search via eager sparse scoring Xing Han Lù 33 32 0 04 Jul 2024
Codec-ASR: Training Performant Automatic Speech Recognition Systems with Discrete Speech Representations Kunal Dhawan Nithin Rao Koluguri Ante Jukić Ryan Langman Jagadeesh Balam Boris Ginsburg 54 1 0 03 Jul 2024
Single Character Perturbations Break LLM Alignment Leon Lin Hannah Brown Kenji Kawaguchi Michael Shieh AAML 230 2 0 03 Jul 2024
Enhancing Translation Accuracy of Large Language Models through Continual Pre-Training on Parallel Data Minato Kondo T. Utsuro Masaaki Nagata CLL 44 4 0 03 Jul 2024
A Case Study on Context-Aware Neural Machine Translation with Multi-Task Learning Ramakrishna Appicharla Baban Gain Santanu Pal Asif Ekbal Pushpak Bhattacharyya 23 1 0 03 Jul 2024
Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scale Wenzhen Zheng Wenbo Pan Xu Xu Libo Qin Li Yue Ming Zhou CLL 42 6 0 02 Jul 2024
How to Learn in a Noisy World? Self-Correcting the Real-World Data Noise in Machine Translation Yan Meng Di Wu Christof Monz 36 1 0 02 Jul 2024
Gloss2Text: Sign Language Gloss translation using LLMs and Semantically Aware Label Smoothing Pooya Fayyazsanavi Antonios Anastasopoulos Jana Kosecka SLR 34 1 0 01 Jul 2024
Calibrated Large Language Models for Binary Question Answering Patrizio Giovannotti Alexander Gammerman 36 1 0 01 Jul 2024
xSemAD: Explainable Semantic Anomaly Detection in Event Logs Using Sequence-to-Sequence Models Kiran Busch T. Kampik Henrik Leopold 20 2 0 28 Jun 2024
Token-Weighted RNN-T for Learning from Flawed Data Gil Keren Wei Zhou Ozlem Kalinli 43 0 0 26 Jun 2024
PharmaGPT: Domain-Specific Large Language Models for Bio-Pharmaceutical and Chemistry Linqing Chen Weilei Wang Zilong Bai Peng Xu Yan Fang ... Lisha Zhang Fu Bian Zhongkai Ye Lidong Pei Changyang Tu AI4MH LM&MA 53 2 0 26 Jun 2024
Efficient Document Ranking with Learnable Late Interactions Ziwei Ji Himanshu Jain Andreas Veit Sashank J. Reddi Sadeep Jayasumana A. S. Rawat A. Menon Felix X. Yu Sanjiv Kumar 46 1 0 25 Jun 2024
CharED: Character-wise Ensemble Decoding for Large Language Models Kevin Gu Eva Tuecke Dmitriy Katz R. Horesh David Alvarez-Melis Mikhail Yurochkin 33 2 0 25 Jun 2024
Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignment Paarth Neekhara Shehzeen Samarah Hussain Subhankar Ghosh Jason Chun Lok Li Rafael Valle Rohan Badlani Boris Ginsburg 58 11 0 25 Jun 2024
Data curation via joint example selection further accelerates multimodal learning Talfan Evans Nikhil Parthasarathy Hamza Merzic Olivier J. Hénaff 32 13 0 25 Jun 2024
Vaporetto: Efficient Japanese Tokenization Based on Improved Pointwise Linear Classification Koichi Akabe Shunsuke Kanda Yusuke Oda Shinsuke Mori 13 0 0 24 Jun 2024
Understanding and Mitigating Tokenization Bias in Language Models Buu Phan Marton Havasi Matthew Muckley Karen Ullrich 57 4 0 24 Jun 2024
Large Vocabulary Size Improves Large Language Models Sho Takase Ryokan Ri Shun Kiyono Takuya Kato 45 3 0 24 Jun 2024
Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers Xiuying Wei Skander Moalla Razvan Pascanu Çağlar Gülçehre 34 0 0 24 Jun 2024
Revisiting Interpolation Augmentation for Speech-to-Text Generation Chen Xu Jie Wang Xiaoqian Liu Qianqian Dong Chunliang Zhang Tong Xiao Jingbo Zhu Dapeng Man Wu Yang 43 1 0 22 Jun 2024
TacoLM: GaTed Attention Equipped Codec Language Model are Efficient Zero-Shot Text to Speech Synthesizers Yakun Song Zhuo Chen Xiaofei Wang Ziyang Ma Guanrou Yang Xie Chen AuLLM 46 3 0 22 Jun 2024
Speech Prefix-Tuning with RNNT Loss for Improving LLM Predictions M. Baskar Andrew Rosenberg Bhuvana Ramabhadran Neeraj Gaur Zhong Meng 39 3 0 20 Jun 2024
Exploring Design Choices for Building Language-Specific LLMs Atula Tejaswi Nilesh Gupta Eunsol Choi 29 10 0 20 Jun 2024
How to Compute the Probability of a Word Tiago Pimentel Clara Meister 37 14 0 20 Jun 2024
Infusing clinical knowledge into tokenisers for language models Abul Hasan Jinge Wu Quang Ngoc Nguyen Salomé Andres Imane Guellil Huayu Zhang Arlene Casey Beatrice Alex Bruce Guthrie Honghan Wu 46 1 0 20 Jun 2024
On the Evaluation Practices in Multilingual NLP: Can Machine Translation Offer an Alternative to Human Translations? Rochelle Choenni Sara Rajaee Christof Monz Ekaterina Shutova 39 1 0 20 Jun 2024
Lexically Grounded Subword Segmentation Jindřich Libovický Jindřich Helcl 48 1 0 19 Jun 2024
How effective is Multi-source pivoting for Translation of Low Resource Indian Languages? Pranav Gaikwad Meet Doshi Raj Dabre Pushpak Bhattacharyya 31 0 0 19 Jun 2024
Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More? Jinhyuk Lee Anthony Chen Zhuyun Dai Dheeru Dua Devendra Singh Sachan ... Jeremy R. Cole Sebastian Riedel Iftekhar Naim Ming-Wei Chang Kelvin Guu RALM LRM 58 31 0 19 Jun 2024
Nemotron-4 340B Technical Report Nvidia : Bo Adler Niket Agarwal Ashwath Aithal ... Jimmy Zhang Jing Zhang Vivienne Zhang Yian Zhang Chen Zhu 49 57 0 17 Jun 2024
Tokenization Falling Short: The Curse of Tokenization Yekun Chai Yewei Fang Qiwei Peng Xuhong Li 52 1 0 17 Jun 2024
Towards an End-to-End Framework for Invasive Brain Signal Decoding with Large Language Models Sheng Feng Heyang Liu Yu Wang Yanfeng Wang 24 3 0 17 Jun 2024
Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning Zebang Cheng Zhi-Qi Cheng Jun-Yan He Jingdong Sun Kai Wang Yuxiang Lin Zheng Lian Xiaojiang Peng Alexander G. Hauptmann MLLM 40 31 0 17 Jun 2024
Unveiling the Power of Source: Source-based Minimum Bayes Risk Decoding for Neural Machine Translation Boxuan Lyu Hidetaka Kamigaito Kotaro Funakoshi Manabu Okumura 43 0 0 17 Jun 2024
CoSTA: Code-Switched Speech Translation using Aligned Speech-Text Interleaving Bhavani Shankar Preethi Jyothi Pushpak Bhattacharyya 53 1 0 16 Jun 2024
Multilingual Large Language Models and Curse of Multilinguality Daniil Gurgurov Tanja Bäumel Tatiana Anikina 86 4 0 15 Jun 2024
CNVSRC 2023: The First Chinese Continuous Visual Speech Recognition Challenge Chen Chen Zehua Liu Xiaolou Li Lantian Li D. Wang 35 2 0 14 Jun 2024
UniBridge: A Unified Approach to Cross-Lingual Transfer Learning for Low-Resource Languages Trinh Pham Khoi M. Le Luu Anh Tuan 42 1 0 14 Jun 2024
4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities Roman Bachmann Oğuzhan Fatih Kar David Mizrahi Ali Garjani Mingfei Gao David Griffiths Jiaming Hu Afshin Dehghan Amir Zamir MoE VLM MLLM 49 14 0 13 Jun 2024
Transformer-based Model for ASR N-Best Rescoring and Rewriting Iwen E. Kang Christophe Van Gysel Man-Hung Siu 39 2 0 12 Jun 2024
An Empirical Study of Mamba-based Language Models R. Waleffe Wonmin Byeon Duncan Riach Brandon Norick V. Korthikanti ... Vartika Singh Jared Casper Jan Kautz M. Shoeybi Bryan Catanzaro 63 66 0 12 Jun 2024
PRoDeliberation: Parallel Robust Deliberation for End-to-End Spoken Language Understanding Trang Le Daniel Lazar Suyoun Kim Shan Jiang Duc Le Adithya Sagar Aleksandr Livshits Ahmed Aly Akshat Shrivastava 50 0 0 12 Jun 2024
Languages Transferred Within the Encoder: On Representation Transfer in Zero-Shot Multilingual Translation Zhi Qu Chenchen Ding Taro Watanabe 90 1 0 12 Jun 2024
A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Any Translation Zhengrui Ma Qingkai Fang Shaolei Zhang Shoutao Guo Yang Feng Min Zhang 53 10 0 11 Jun 2024
EAVE: Efficient Product Attribute Value Extraction via Lightweight Sparse-layer Interaction Li Yang Qifan Wang Jianfeng Chi Jiahao Liu Jingang Wang Fuli Feng Zenglin Xu Yi Fang Lifu Huang Dongfang Liu 46 1 0 10 Jun 2024
StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection Sara Papi Marco Gaido Matteo Negri L. Bentivogli 79 4 0 10 Jun 2024
Attention as a Hypernetwork Simon Schug Seijin Kobayashi Yassir Akram João Sacramento Razvan Pascanu GNN 37 3 0 09 Jun 2024