ResearchTrend.AI
  • Communities
  • Connect sessions
  • AI calendar
  • Organizations
  • Join Slack
  • Contact Sales
Papers
Communities
Social Events
Terms and Conditions
Pricing
Contact Sales
Parameter LabParameter LabTwitterGitHubLinkedInBlueskyYoutube

© 2026 ResearchTrend.AI, All rights reserved.

  1. Home
  2. Papers
  3. 1808.06226
  4. Cited By
SentencePiece: A simple and language independent subword tokenizer and
  detokenizer for Neural Text Processing

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

19 August 2018
Taku Kudo
John Richardson
ArXiv (abs)PDFHTMLGithub (10925★)

Papers citing "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing"

50 / 2,063 papers shown
Gloss2Text: Sign Language Gloss translation using LLMs and Semantically
  Aware Label Smoothing
Gloss2Text: Sign Language Gloss translation using LLMs and Semantically Aware Label Smoothing
Pooya Fayyazsanavi
Antonios Anastasopoulos
Jana Kosecka
SLR
213
4
0
01 Jul 2024
Calibrated Large Language Models for Binary Question Answering
Calibrated Large Language Models for Binary Question Answering
Patrizio Giovannotti
Alexander Gammerman
210
1
0
01 Jul 2024
xSemAD: Explainable Semantic Anomaly Detection in Event Logs Using
  Sequence-to-Sequence Models
xSemAD: Explainable Semantic Anomaly Detection in Event Logs Using Sequence-to-Sequence Models
Kiran Busch
T. Kampik
Henrik Leopold
103
5
0
28 Jun 2024
Token-Weighted RNN-T for Learning from Flawed Data
Token-Weighted RNN-T for Learning from Flawed Data
Gil Keren
Wei Zhou
Ozlem Kalinli
263
1
0
26 Jun 2024
PharmaGPT: Domain-Specific Large Language Models for Bio-Pharmaceutical
  and Chemistry
PharmaGPT: Domain-Specific Large Language Models for Bio-Pharmaceutical and Chemistry
Linqing Chen
Wentao Wu
Zilong Bai
Peng Xu
Yan Fang
...
Lisha Zhang
Fu Bian
Zhongkai Ye
Lidong Pei
Changyang Tu
AI4MHLM&MA
346
5
0
26 Jun 2024
Efficient Document Ranking with Learnable Late Interactions
Efficient Document Ranking with Learnable Late Interactions
Ziwei Ji
Himanshu Jain
Andreas Veit
Sashank J. Reddi
Sadeep Jayasumana
A. S. Rawat
A. Menon
Felix X. Yu
Sanjiv Kumar
242
9
0
25 Jun 2024
CharED: Character-wise Ensemble Decoding for Large Language Models
CharED: Character-wise Ensemble Decoding for Large Language Models
Kevin Gu
Eva Tuecke
Dmitriy Katz
R. Horesh
David Alvarez-Melis
Mikhail Yurochkin
249
3
0
25 Jun 2024
Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic
  Alignment
Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignment
Paarth Neekhara
Shehzeen Samarah Hussain
Subhankar Ghosh
Jason Chun Lok Li
Rafael Valle
Rohan Badlani
Boris Ginsburg
200
27
0
25 Jun 2024
Data curation via joint example selection further accelerates multimodal
  learning
Data curation via joint example selection further accelerates multimodal learning
Talfan Evans
Nikhil Parthasarathy
Hamza Merzic
Olivier J. Hénaff
301
25
0
25 Jun 2024
Vaporetto: Efficient Japanese Tokenization Based on Improved Pointwise
  Linear Classification
Vaporetto: Efficient Japanese Tokenization Based on Improved Pointwise Linear Classification
Koichi Akabe
Shunsuke Kanda
Yusuke Oda
Shinsuke Mori
82
0
0
24 Jun 2024
Understanding and Mitigating Tokenization Bias in Language Models
Understanding and Mitigating Tokenization Bias in Language Models
Buu Phan
Marton Havasi
Matthew Muckley
Karen Ullrich
257
11
0
24 Jun 2024
Building on Efficient Foundations: Effectively Training LLMs with
  Structured Feedforward Layers
Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers
Xiuying Wei
Skander Moalla
Razvan Pascanu
Çağlar Gülçehre
342
4
0
24 Jun 2024
Large Vocabulary Size Improves Large Language Models
Large Vocabulary Size Improves Large Language Models
Sho Takase
Ryokan Ri
Shun Kiyono
Takuya Kato
311
8
0
24 Jun 2024
Revisiting Interpolation Augmentation for Speech-to-Text Generation
Revisiting Interpolation Augmentation for Speech-to-Text Generation
Chen Xu
Jie Wang
Xiaoqian Liu
Qianqian Dong
Chunliang Zhang
Tong Xiao
Jingbo Zhu
Dapeng Man
Wu Yang
193
1
0
22 Jun 2024
TacoLM: GaTed Attention Equipped Codec Language Model are Efficient
  Zero-Shot Text to Speech Synthesizers
TacoLM: GaTed Attention Equipped Codec Language Model are Efficient Zero-Shot Text to Speech Synthesizers
Yakun Song
Zhuo Chen
Xiaofei Wang
Ziyang Ma
Guanrou Yang
Xie Chen
AuLLM
128
6
0
22 Jun 2024
Speech Prefix-Tuning with RNNT Loss for Improving LLM Predictions
Speech Prefix-Tuning with RNNT Loss for Improving LLM Predictions
M. Baskar
Andrew Rosenberg
Bhuvana Ramabhadran
Neeraj Gaur
Zhong Meng
181
3
0
20 Jun 2024
Exploring Design Choices for Building Language-Specific LLMs
Exploring Design Choices for Building Language-Specific LLMs
Atula Tejaswi
Nilesh Gupta
Eunsol Choi
248
21
0
20 Jun 2024
How to Compute the Probability of a Word
How to Compute the Probability of a Word
Tiago Pimentel
Clara Meister
244
33
0
20 Jun 2024
Infusing clinical knowledge into tokenisers for language models
Infusing clinical knowledge into tokenisers for language models
Abul Hasan
Jinge Wu
Quang Ngoc Nguyen
Salomé Andres
Imane Guellil
Huayu Zhang
Arlene Casey
Beatrice Alex
Bruce Guthrie
Honghan Wu
186
3
0
20 Jun 2024
On the Evaluation Practices in Multilingual NLP: Can Machine Translation
  Offer an Alternative to Human Translations?
On the Evaluation Practices in Multilingual NLP: Can Machine Translation Offer an Alternative to Human Translations?
Rochelle Choenni
Sara Rajaee
Christof Monz
Ekaterina Shutova
308
5
0
20 Jun 2024
Lexically Grounded Subword Segmentation
Lexically Grounded Subword SegmentationConference on Empirical Methods in Natural Language Processing (EMNLP), 2024
Jindřich Libovický
Jindřich Helcl
245
8
0
19 Jun 2024
How effective is Multi-source pivoting for Translation of Low Resource
  Indian Languages?
How effective is Multi-source pivoting for Translation of Low Resource Indian Languages?
Pranav Gaikwad
Meet Doshi
Mary Dabre
Pushpak Bhattacharyya
200
3
0
19 Jun 2024
Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?
Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?
Jinhyuk Lee
Anthony Chen
Zhuyun Dai
Dheeru Dua
Devendra Singh Sachan
...
Jeremy R. Cole
Sebastian Riedel
Iftekhar Naim
Ming-Wei Chang
Kelvin Guu
RALMLRM
228
52
0
19 Jun 2024
Nemotron-4 340B Technical Report
Nemotron-4 340B Technical Report
Nvidia
:
Bo Adler
Niket Agarwal
Ashwath Aithal
...
Jimmy Zhang
Jing Zhang
Vivienne Zhang
Yian Zhang
Chen Zhu
301
111
0
17 Jun 2024
Tokenization Falling Short: The Curse of Tokenization
Tokenization Falling Short: The Curse of Tokenization
Yekun Chai
Yewei Fang
Qiwei Peng
Xuhong Li
213
0
0
17 Jun 2024
Towards an End-to-End Framework for Invasive Brain Signal Decoding with
  Large Language Models
Towards an End-to-End Framework for Invasive Brain Signal Decoding with Large Language Models
Sheng Feng
Heyang Liu
Yu Wang
Yanfeng Wang
106
7
0
17 Jun 2024
Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with
  Instruction Tuning
Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning
Zebang Cheng
Zhi-Qi Cheng
Jun-Yan He
Yuxuan Zhou
Kai Wang
Yuxiang Lin
Zheng Lian
Xiaojiang Peng
Alexander G. Hauptmann
MLLM
248
115
0
17 Jun 2024
Unveiling the Power of Source: Source-based Minimum Bayes Risk Decoding for Neural Machine Translation
Unveiling the Power of Source: Source-based Minimum Bayes Risk Decoding for Neural Machine Translation
Boxuan Lyu
Hidetaka Kamigaito
Kotaro Funakoshi
Manabu Okumura
558
0
0
17 Jun 2024
CoSTA: Code-Switched Speech Translation using Aligned Speech-Text
  Interleaving
CoSTA: Code-Switched Speech Translation using Aligned Speech-Text Interleaving
Bhavani Shankar
Preethi Jyothi
Pushpak Bhattacharyya
313
5
0
16 Jun 2024
Multilingual Large Language Models and Curse of Multilinguality
Multilingual Large Language Models and Curse of Multilinguality
Daniil Gurgurov
Tanja Bäumel
Tatiana Anikina
304
12
0
15 Jun 2024
CNVSRC 2023: The First Chinese Continuous Visual Speech Recognition
  Challenge
CNVSRC 2023: The First Chinese Continuous Visual Speech Recognition ChallengeInterspeech (Interspeech), 2024
Chen Chen
Zehua Liu
Xiaolou Li
Lantian Li
D. Wang
187
6
0
14 Jun 2024
UniBridge: A Unified Approach to Cross-Lingual Transfer Learning for
  Low-Resource Languages
UniBridge: A Unified Approach to Cross-Lingual Transfer Learning for Low-Resource LanguagesAnnual Meeting of the Association for Computational Linguistics (ACL), 2024
Trinh Pham
Khoi M. Le
Luu Anh Tuan
363
4
0
14 Jun 2024
4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities
4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities
Roman Bachmann
Oğuzhan Fatih Kar
David Mizrahi
Ali Garjani
Mingfei Gao
David Griffiths
Jiaming Hu
Afshin Dehghan
Amir Zamir
MoEVLMMLLM
268
33
0
13 Jun 2024
Transformer-based Model for ASR N-Best Rescoring and Rewriting
Transformer-based Model for ASR N-Best Rescoring and Rewriting
Iwen E. Kang
Christophe Van Gysel
Man-Hung Siu
227
5
0
12 Jun 2024
An Empirical Study of Mamba-based Language Models
An Empirical Study of Mamba-based Language Models
R. Waleffe
Wonmin Byeon
Duncan Riach
Brandon Norick
V. Korthikanti
...
Vartika Singh
Jared Casper
Jan Kautz
Mohammad Shoeybi
Bryan Catanzaro
326
142
0
12 Jun 2024
PRoDeliberation: Parallel Robust Deliberation for End-to-End Spoken
  Language Understanding
PRoDeliberation: Parallel Robust Deliberation for End-to-End Spoken Language Understanding
Trang Le
Daniel Lazar
Suyoun Kim
Shan Jiang
Duc Le
Adithya Sagar
Aleksandr Livshits
Ahmed Aly
Akshat Shrivastava
191
0
0
12 Jun 2024
Languages Transferred Within the Encoder: On Representation Transfer in Zero-Shot Multilingual Translation
Languages Transferred Within the Encoder: On Representation Transfer in Zero-Shot Multilingual Translation
Zhi Qu
Chenchen Ding
Taro Watanabe
299
3
0
12 Jun 2024
A Non-autoregressive Generation Framework for End-to-End Simultaneous
  Speech-to-Any Translation
A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Any Translation
Zhengrui Ma
Qingkai Fang
Shaolei Zhang
Shoutao Guo
Yang Feng
Min Zhang
218
18
0
11 Jun 2024
EAVE: Efficient Product Attribute Value Extraction via Lightweight
  Sparse-layer Interaction
EAVE: Efficient Product Attribute Value Extraction via Lightweight Sparse-layer Interaction
Li Yang
Qifan Wang
Jianfeng Chi
Jiahao Liu
Jingang Wang
Fuli Feng
Zenglin Xu
Yi Fang
Lifu Huang
Dongfang Liu
189
3
0
10 Jun 2024
StreamAtt: Direct Streaming Speech-to-Text Translation with
  Attention-based Audio History Selection
StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History SelectionAnnual Meeting of the Association for Computational Linguistics (ACL), 2024
Sara Papi
Marco Gaido
Matteo Negri
L. Bentivogli
382
16
0
10 Jun 2024
Attention as a Hypernetwork
Attention as a HypernetworkInternational Conference on Learning Representations (ICLR), 2024
Simon Schug
Seijin Kobayashi
Yassir Akram
João Sacramento
Razvan Pascanu
GNN
269
9
0
09 Jun 2024
Exploring the Benefits of Tokenization of Discrete Acoustic Units
Exploring the Benefits of Tokenization of Discrete Acoustic UnitsInterspeech (Interspeech), 2024
Avihu Dekel
Raul Fernandez
158
3
0
08 Jun 2024
Large Language Model-guided Document Selection
Large Language Model-guided Document Selection
Xiang Kong
Tom Gunter
Ruoming Pang
192
7
0
07 Jun 2024
Recovering document annotations for sentence-level bitext
Recovering document annotations for sentence-level bitext
R. Wicks
Matt Post
Philipp Koehn
275
7
0
06 Jun 2024
Enhancing CTC-based speech recognition with diverse modeling units
Enhancing CTC-based speech recognition with diverse modeling units
Shiyi Han
Zhihong Lei
Mingbin Xu
Xingyu Na
Zhen Huang
339
1
0
05 Jun 2024
StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task
  Learning
StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning
Shaolei Zhang
Qingkai Fang
Shoutao Guo
Zhengrui Ma
Min Zhang
Yang Feng
258
19
0
05 Jun 2024
LCS: A Language Converter Strategy for Zero-Shot Neural Machine
  Translation
LCS: A Language Converter Strategy for Zero-Shot Neural Machine Translation
Zengkui Sun
Yijin Liu
Fandong Meng
Jinan Xu
Jinan Xu
Jie Zhou
323
2
0
05 Jun 2024
Xmodel-LM Technical Report
Xmodel-LM Technical Report
Yichuan Wang
Yang Liu
Yu Yan
Qun Wang
Xucheng Huang
Ling Jiang
OSLMALM
266
1
0
05 Jun 2024
Multi-word Term Embeddings Improve Lexical Product Retrieval
Multi-word Term Embeddings Improve Lexical Product Retrieval
Viktor Shcherbakov
Fedor Krasnov
172
0
0
03 Jun 2024
Applying Intrinsic Debiasing on Downstream Tasks: Challenges and
  Considerations for Machine Translation
Applying Intrinsic Debiasing on Downstream Tasks: Challenges and Considerations for Machine Translation
Bar Iluz
Yanai Elazar
Asaf Yehudai
Gabriel Stanovsky
198
4
0
02 Jun 2024
Previous
123...789...404142
Next