SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

19 August 2018

Papers citing "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing"

50 / 1,923 papers shown

Title
Interpreting token compositionality in LLMs: A robustness analysis Nura Aljaafari Danilo S. Carvalho André Freitas 35 1 0 16 Oct 2024
Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5 Thao Anh Dang Limor Raviv Lukas Galke 25 1 0 15 Oct 2024
LargePiG: Your Large Language Model is Secretly a Pointer Generator Zhongxiang Sun Zihua Si Xiaoxue Zang Kai Zheng Yang Song Xiao Zhang Jun Xu HILM RALM 42 0 0 15 Oct 2024
Transfer Learning with Foundational Models for Time Series Forecasting using Low-Rank Adaptations M. Germán-Morales A. J. Rivera-Rivas M. J. del Jesus Díaz C. J. Carmona AI4TS AI4CE 56 0 0 15 Oct 2024
ChakmaNMT: A Low-resource Machine Translation On Chakma Language Aunabil Chakma Aditya Chakma Soham Khisa Chumui Tripura Masum Hasan Rifat Shahriyar 26 0 0 14 Oct 2024
Predicting from Strings: Language Model Embeddings for Bayesian Optimization Tung Nguyen Qiuyi Zhang Bangding Yang Chansoo Lee J. Bornschein Yingjie Miao Sagi Perel Yutian Chen Xingyou Song BDL 31 3 0 14 Oct 2024
Text Classification using Graph Convolutional Networks: A Comprehensive Survey Syed Mustafa Haider Rizvi Ramsha Imran Arif Mahmood GNN OOD FaML 26 0 0 12 Oct 2024
Adapters for Altering LLM Vocabularies: What Languages Benefit the Most? HyoJung Han Akiko Eriguchi Haoran Xu Hieu T. Hoang Marine Carpuat Huda Khayrallah VLM 43 2 0 12 Oct 2024
OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling Linhui Xiao Xiaoshan Yang Fang Peng Yaowei Wang Changsheng Xu ObjD 40 5 0 10 Oct 2024
Self-Attention Mechanism in Multimodal Context for Banking Transaction Flow Cyrile Delestre Yoann Sola 34 0 0 10 Oct 2024
Transducer Consistency Regularization for Speech to Text Applications Cindy Tseng Yun Tang Vijendra Raj Apsingekar 42 0 0 09 Oct 2024
Generative Model for Less-Resourced Language with 1 billion parameters Domen Vreš Martin Božič Aljaž Potočnik Tomaž Martinčič Marko Robnik-Šikonja 26 1 0 09 Oct 2024
Inference over Unseen Entities, Relations and Literals on Knowledge Graphs Caglar Demir N'Dah Jean Kouagou Arnab Sharma Axel-Cyrille Ngonga Ngomo 28 0 0 09 Oct 2024
DEPT: Decoupled Embeddings for Pre-training Language Models Alex Iacob Lorenzo Sani Meghdad Kurmanji William F. Shen Xinchi Qiu Dongqi Cai Yan Gao Nicholas D. Lane VLM 224 0 0 07 Oct 2024
Language Model-Driven Data Pruning Enables Efficient Active Learning Abdul Hameed Azeemi I. Qazi Agha Ali Raza VLM 36 1 0 05 Oct 2024
Adaptive BPE Tokenization for Enhanced Vocabulary Adaptation in Finetuning Pretrained Language Models Gunjan Balde Soumyadeep Roy Mainack Mondal Niloy Ganguly 17 1 0 04 Oct 2024
Cross-lingual Transfer for Automatic Question Generation by Learning Interrogative Structures in Target Languages Seonjeong Hwang Yunsu Kim Gary Geunbae Lee 40 0 0 04 Oct 2024
MELODI: Exploring Memory Compression for Long Contexts Yinpeng Chen DeLesley Hutchins Aren Jansen Andrey Zhmoginov David Racz Jesper Andersen 38 2 0 04 Oct 2024
No Need to Talk: Asynchronous Mixture of Language Models Anastasiia Filippova Angelos Katharopoulos David Grangier Ronan Collobert MoE 46 0 0 04 Oct 2024
Morphological evaluation of subwords vocabulary used by BETO language model Óscar García-Sierra Ana Fernández-Pampillón Cesteros Miguel Ortega-Martín 41 0 0 03 Oct 2024
Selective Attention Improves Transformer Yaniv Leviathan Matan Kalman Yossi Matias 51 9 0 03 Oct 2024
HAINAN: Fast and Accurate Transducer for Hybrid-Autoregressive ASR Hainan Xu Travis M. Bartley Vladimir Bataev Boris Ginsburg 239 0 0 03 Oct 2024
Gold Panning in Vocabulary: An Adaptive Method for Vocabulary Expansion of Domain-Specific LLMs Chengyuan Liu Shihang Wang Lizhi Qing Kun Kuang Yangyang Kang Changlong Sun Fei Wu 36 0 0 02 Oct 2024
FedPT: Federated Proxy-Tuning of Large Language Models on Resource-Constrained Edge Devices Zhidong Gao Yu Zhang Zhenxiao Zhang Yanmin Gong Yuanxiong Guo 23 0 0 01 Oct 2024
AfriHuBERT: A self-supervised speech representation model for African languages Jesujoba Oluwadara Alabi Xuechen Liu Dietrich Klakow Junichi Yamagishi VLM 38 1 0 30 Sep 2024
Enhancing High-order Interaction Awareness in LLM-based Recommender Model Xinfeng Wang Jin Cui Fumiyo Fukumoto Yoshimi Suzuki 30 3 0 30 Sep 2024
Universal Medical Image Representation Learning with Compositional Decoders Kaini Wang Ling Yang Siping Zhou Guangquan Zhou Wentao Zhang Bin Cui Shuo Li SSL MedIm 36 0 0 30 Sep 2024
Exploring Language Model Generalization in Low-Resource Extractive QA Saptarshi Sengupta Wenpeng Yin Preslav Nakov Shreya Ghosh Suhang Wang 27 0 0 27 Sep 2024
Convolutional Signal Propagation: A Simple Scalable Algorithm for Hypergraphs Pavel Procházka Marek Dědič Lukáš Bajer GNN 37 0 0 26 Sep 2024
LangSAMP: Language-Script Aware Multilingual Pretraining Yihong Liu Haotian Ye Chunlan Ma Mingyang Wang Hinrich Schütze VLM 36 0 0 26 Sep 2024
How Transliterations Improve Crosslingual Alignment Yihong Liu Mingyang Wang Amir Hossein Kargaran Ayyoob Imani Orgest Xhelili Haotian Ye Chunlan Ma François Yvon Hinrich Schütze 42 2 0 25 Sep 2024
EuroLLM: Multilingual Language Models for Europe Pedro Henrique Martins Patrick Fernandes Joao Alves Nuno M. Guerreiro Ricardo Rei ... Pierre Colombo Barry Haddow José G. C. de Souza Alexandra Birch André F. T. Martins 37 20 0 24 Sep 2024
Multilingual Transfer and Domain Adaptation for Low-Resource Languages of Spain Yuanchang Luo Zhanglin Wu Daimeng Wei Hengchao Shang Zongyao Li ... Shaojun Li Jinlong Yang Yuhao Xie Jiawei Zheng Bin Wei Hao Yang 33 1 0 24 Sep 2024
Machine Translation Advancements of Low-Resource Indian Languages by Transfer Learning Bin Wei Jiawei Zhen Zongyao Li Zhanglin Wu Daimeng Wei ... Yuanchang Luo Hengchao Shang Jinlong Yang Yuhao Xie Hao Yang VLM 30 1 0 24 Sep 2024
dnaGrinder: a lightweight and high-capacity genomic foundation model Qihang Zhao Chi Zhang Weixiong Zhang 31 0 0 24 Sep 2024
HW-TSC's Submission to the CCMT 2024 Machine Translation Tasks Zhanglin Wu Yuanchang Luo Daimeng Wei Jiawei Zheng Bin Wei ... Jiaxin Guo Shaojun Li Mengli Zhu Ning Xie Hao Yang 45 1 0 23 Sep 2024
Choose the Final Translation from NMT and LLM hypotheses Using MBR Decoding: HW-TSC's Submission to the WMT24 General MT Shared Task Zhanglin Wu Daimeng Wei Zongyao Li Hengchao Shang Jiaxin Guo Shaojun Li Zhiqiang Rao Yuanchang Luo Ning Xie Hao Yang 37 4 0 23 Sep 2024
Cross-Domain Content Generation with Domain-Specific Small Language Models Ankit Maloo Abhinav Garg CLL 22 0 0 19 Sep 2024
An Efficient Self-Learning Framework For Interactive Spoken Dialog Systems Hitesh Tulsiani David M. Chan Shalini Ghosh Garima Lalwani Prabhat Pandey Ankish Bansal Sri Garimella Ariya Rastrow Björn Hoffmeister 33 0 0 16 Sep 2024
PixelBytes: Catching Unified Representation for Multimodal Generation Fabien Furfaro 26 0 0 16 Sep 2024
DomURLs_BERT: Pre-trained BERT-based Model for Malicious Domains and URLs Detection and Classification Abdelkader El Mahdaouy Salima Lamsiyah Meryem Janati Idrissi H. Alami Zakaria Yartaoui Ismail Berrada 21 3 0 13 Sep 2024
Optimizing Rare Word Accuracy in Direct Speech Translation with a Retrieval-and-Demonstration Approach Siqi Li Danni Liu Jan Niehues 33 0 0 13 Sep 2024
Retro-li: Small-Scale Retrieval Augmented Generation Supporting Noisy Similarity Searches and Domain Shift Generalization Gentiana Rashiti G. Karunaratne Mrinmaya Sachan Abu Sebastian Abbas Rahimi RALM 44 0 0 12 Sep 2024
TeXBLEU: Automatic Metric for Evaluate LaTeX Format Kyudan Jung N. Kim Hyongon Ryu Sieun Hyeon Seung-jun Lee Hyeok-jae Lee 39 0 0 10 Sep 2024
BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training Pavel Chizhov Catherine Arnett Elizaveta Korotkova Ivan P. Yamshchikov 48 2 0 06 Sep 2024
Open Language Data Initiative: Advancing Low-Resource Machine Translation for Karakalpak Mukhammadsaid Mamasaidov Abror Shopulatov VLM 29 4 0 06 Sep 2024
The AdEMAMix Optimizer: Better, Faster, Older Matteo Pagliardini Pierre Ablin David Grangier ODL 30 9 0 05 Sep 2024
Multi-modal Situated Reasoning in 3D Scenes Xiongkun Linghu Jiangyong Huang Xuesong Niu Xiaojian Ma Baoxiong Jia Siyuan Huang 43 12 0 04 Sep 2024
Resource-Efficient Adaptation of Speech Foundation Models for Multi-Speaker ASR Weiqing Wang Kunal Dhawan Taejin Park Krishna Puvvada Ivan Medennikov Somshubra Majumdar He Huang Jagadeesh Balam Boris Ginsburg 44 2 0 02 Sep 2024
Multi-Modal Multi-Granularity Tokenizer for Chu Bamboo Slip Scripts Yingfa Chen Chenlong Hu Cong Feng Chenyang Song Shi Yu Xu Han Zhiyuan Liu Maosong Sun 33 0 0 02 Sep 2024