SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

19 August 2018

Papers citing "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing"

50 / 1,923 papers shown

Title
RoBERTurk: Adjusting RoBERTa for Turkish Nuri Tas 27 1 0 07 Jan 2024
DiarizationLM: Speaker Diarization Post-Processing with Large Language Models Quan Wang Yiling Huang Guanlong Zhao Evan Clark Wei Xia Hank Liao AuLLM 33 8 0 07 Jan 2024
PIXAR: Auto-Regressive Language Modeling in Pixel Space Yintao Tai Xiyang Liao Alessandro Suglia Antonio Vergari MLLM 26 7 0 06 Jan 2024
Cheetah: Natural Language Generation for 517 African Languages Ife Adebara AbdelRahim Elmadany Muhammad Abdul-Mageed 29 4 0 02 Jan 2024
An Empirical Study of Scaling Law for OCR Miao Rang Zhenni Bi Chuanjian Liu Yunhe Wang Kai Han 50 6 0 29 Dec 2023
SentinelLMs: Encrypted Input Adaptation and Fine-tuning of Language Models for Private and Secure Inference Abhijit Mishra Mingda Li S. Deo SILM 13 2 0 28 Dec 2023
MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices Xiangxiang Chu Limeng Qiao Xinyang Lin Shuang Xu Yang Yang ... Fei Wei Xinyu Zhang Bo Zhang Xiaolin Wei Chunhua Shen MLLM 44 35 0 28 Dec 2023
Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognition Vahid Noroozi Somshubra Majumdar Ankur Kumar Jagadeesh Balam Boris Ginsburg 41 10 0 27 Dec 2023
PanGu- $π$ : Enhancing Language Model Architectures via Nonlinearity Compensation Yunhe Wang Hanting Chen Yehui Tang Tianyu Guo Kai Han ... Qinghua Xu Qun Liu Jun Yao Chao Xu Dacheng Tao 73 17 0 27 Dec 2023
Gemini Pro Defeated by GPT-4V: Evidence from Education Gyeong-Geon Lee Ehsan Latif Lehong Shi Xiaoming Zhai 34 22 0 27 Dec 2023
Dotless Representation of Arabic Text: Analysis and Modeling Maged S. Al-Shaibani Irfan Ahmad 25 0 0 26 Dec 2023
PersianLLaMA: Towards Building First Persian Large Language Model Mohammad Amin Abbasi A. Ghafouri Mahdi Firouzmandi Hassan Naderi B. Minaei-Bidgoli 29 9 0 25 Dec 2023
YAYI 2: Multilingual Open-Source Large Language Models Yin Luo Qingchao Kong Nan Xu Jia Cao Bao Hao ... Zhaoxin Yu Zhengda Luo Wenji Mao Lei Wang Dajun Zeng ALM OSLM 51 7 0 22 Dec 2023
Typhoon: Thai Large Language Models Kunat Pipatanakul Phatrasek Jirabovonvisut Potsawee Manakul Sittipong Sripaisarnmongkol Ruangsak Patomwong Pathomporn Chokchainant Kasima Tharnpipitchai 50 16 0 21 Dec 2023
Soft Alignment of Modality Space for End-to-end Speech Translation Yuhao Zhang Kaiqi Kou Bei Li Chen Xu Chunliang Zhang Tong Xiao Jingbo Zhu 31 0 0 18 Dec 2023
Binary Code Summarization: Benchmarking ChatGPT/GPT-4 and Other Large Language Models Xin Jin Jonathan Larson Weiwei Yang Zhiqiang Lin ELM 23 23 0 15 Dec 2023
VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generation Jinguo Zhu Xiaohan Ding Yixiao Ge Yuying Ge Sijie Zhao Hengshuang Zhao Xiaohua Wang Ying Shan ViT VLM 24 33 0 14 Dec 2023
Depicting Beyond Scores: Advancing Image Quality Assessment through Multi-modal Language Models Zhiyuan You Zheyuan Li Jinjin Gu Zhenfei Yin Tianfan Xue Chao Dong EGVM 29 35 0 14 Dec 2023
SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention Róbert Csordás Piotr Piekos Kazuki Irie Jürgen Schmidhuber MoE 28 14 0 13 Dec 2023
N-Gram Unsupervised Compoundation and Feature Injection for Better Symbolic Music Understanding Jinhao Tian Zuchao Li Jiajia Li Ping Wang 30 3 0 13 Dec 2023
MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception Yiran Qin Enshen Zhou Qichang Liu Zhen-fei Yin Lu Sheng Ruimao Zhang Yu Qiao Jing Shao LM&Ro 34 39 0 12 Dec 2023
Order Matters in the Presence of Dataset Imbalance for Multilingual Learning Dami Choi Derrick Xin Hamid Dadkhahi Justin Gilmer Ankush Garg Orhan Firat Chih-Kuan Yeh Andrew M. Dai Behrooz Ghorbani 57 3 0 11 Dec 2023
Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations Hakan Inan Kartikeya Upasani Jianfeng Chi Rashi Rungta Krithika Iyer ... Michael Tontchev Qing Hu Brian Fuller Davide Testuggine Madian Khabsa AI4MH 36 379 0 07 Dec 2023
Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognition Yukiya Hono Koh Mitsuda Tianyu Zhao Kentaro Mitsui Toshiaki Wakatsuki Kei Sawada AuLLM 47 8 0 06 Dec 2023
Evaluating Self-supervised Speech Models on a Taiwanese Hokkien Corpus Yi-Hui Chou Kalvin Chang Meng-Ju Wu Winston Ou Alice Wen-Hsin Bi ... Iu-Tshian Phoann Winnie Chang Chenxuan Cui Noel Chen Jiatong Shi 51 3 0 06 Dec 2023
Large Language Models on Graphs: A Comprehensive Survey Bowen Jin Gang Liu Chi Han Meng Jiang Heng Ji Jiawei Han AI4CE 44 141 0 05 Dec 2023
A Machine Learning Approach Towards SKILL Code Autocompletion Enrique Dehaerne Bappaditya Dey Wannes Meert 32 0 0 04 Dec 2023
Using Large Language Models to Accelerate Communication for Users with Severe Motor Impairments Shanqing Cai Subhashini Venugopalan Katie Seaver Xiang Xiao Katrin Tomanek ... Daniel E Vance Blair Casey Steve M. Gleason Philip Q. Nelson Michael P. Brenner 30 7 0 03 Dec 2023
On Significance of Subword tokenization for Low Resource and Efficient Named Entity Recognition: A case study in Marathi Harsh Chaudhari A. Patil Dhanashree Lavekar Pranav Khairnar Raviraj Joshi Sachin Pande 52 0 0 03 Dec 2023
INarIG: Iterative Non-autoregressive Instruct Generation Model For Word-Level Auto Completion Hengchao Shang Zongyao Li Daimeng Wei Jiaxin Guo Minghan Wang Xiaoyu Chen Lizhi Lei Hao Yang 32 0 0 30 Nov 2023
Leveraging VLM-Based Pipelines to Annotate 3D Objects Rishabh Kabra Loic Matthey Alexander Lerchner Niloy J. Mitra 34 6 0 29 Nov 2023
YUAN 2.0: A Large Language Model with Localized Filtering-based Attention Shaohua Wu Xudong Zhao Shenling Wang Jiangang Luo Lingjun Li ... Wei Wang Tong Yu Rongguo Zhang Jiahua Zhang Chao Wang OSLM 56 6 0 27 Nov 2023
Improving Word Sense Disambiguation in Neural Machine Translation with Salient Document Context Elijah Matthew Rippeth Marine Carpuat Kevin Duh Matt Post 20 0 0 27 Nov 2023
Learning to Skip for Language Modeling Dewen Zeng Nan Du Tao Wang Yuanzhong Xu Tao Lei Zhifeng Chen Claire Cui 25 11 0 26 Nov 2023
OpusCleaner and OpusTrainer, open source toolkits for training Machine Translation and Large language models Nikolay Bogoychev Jelmer van der Linde Graeme Nail Barry Haddow Jaume Zaragoza-Bernabeu Gema Ramírez-Sánchez Lukas Weymann Tudor N. Mateiu Jindvrich Helcl Mikko Aulamo VLM 21 1 0 24 Nov 2023
Machine Translation for Geéz Language A. Wassie 29 5 0 24 Nov 2023
PhayaThaiBERT: Enhancing a Pretrained Thai Language Model with Unassimilated Loanwords Panyut Sriwirote Jalinee Thapiang Vasan Timtong Attapol T. Rutherford 24 5 0 21 Nov 2023
Multi-teacher Distillation for Multilingual Spelling Correction Jingfen Zhang Xuan Guo S. Bodapati Christopher Potts KELM 29 3 0 20 Nov 2023
An Embodied Generalist Agent in 3D World Jiangyong Huang Silong Yong Xiaojian Ma Xiongkun Linghu Puhao Li Yan Wang Qing Li Song-Chun Zhu Baoxiong Jia Siyuan Huang LM&Ro 31 139 0 18 Nov 2023
JWSign: A Highly Multilingual Corpus of Bible Translations for more Diversity in Sign Language Processing Shester Gueuwou Sophie Siake Colin Leong Mathias Müller SLR 42 11 0 16 Nov 2023
WatME: Towards Lossless Watermarking Through Lexical Redundancy Liang Chen Yatao Bian Yang Deng Deng Cai Shuaiyi Li Peilin Zhao Kam-Fai Wong WaLM 42 7 0 16 Nov 2023
When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages Tyler A. Chang Catherine Arnett Zhuowen Tu Benjamin Bergen LRM 48 7 0 15 Nov 2023
Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Models J. Michaelov Catherine Arnett Tyler A. Chang Benjamin Bergen 41 12 0 15 Nov 2023
Memory Augmented Language Models through Mixture of Word Experts Cicero Nogueira dos Santos James Lee-Thorp Isaac Noble Chung-Ching Chang David C. Uthus MoE 32 8 0 15 Nov 2023
How Vocabulary Sharing Facilitates Multilingualism in LLaMA? Fei Yuan Shuai Yuan Zhiyong Wu Lei Li 42 10 0 15 Nov 2023
OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretraining Yihong Liu Peiqin Lin Mingyang Wang Hinrich Schütze 40 23 0 15 Nov 2023
Low-Rank Adaptation for Multilingual Summarization: An Empirical Study Chenxi Whitehouse Fantine Huot Jasmijn Bastings Mostafa Dehghani Chu-Cheng Lin Mirella Lapata 27 6 0 14 Nov 2023
Extending Multilingual Machine Translation through Imitation Learning Wen Lai Viktor Hangya Alexander Fraser LRM CLL 35 3 0 14 Nov 2023
Retrieve and Copy: Scaling ASR Personalization to Large Catalogs Sai Muralidhar Jayanthi Devang Kulshreshtha Saket Dingliwal S. Ronanki S. Bodapati 46 7 0 14 Nov 2023
On-the-Fly Fusion of Large Language Models and Machine Translation Hieu T. Hoang Huda Khayrallah Marcin Junczys-Dowmunt 41 3 0 14 Nov 2023