Neural Machine Translation of Rare Words with Subword Units

31 August 2015

Papers citing "Neural Machine Translation of Rare Words with Subword Units"

50 / 3,808 papers shown

Title
Unified Lexical Representation for Interpretable Visual-Language Alignment Yifan Li Yikai Wang Yanwei Fu Dongyu Ru Zheng-Wei Zhang Tong He VLM 42 4 0 25 Jul 2024
Automated Code-centric Software Vulnerability Assessment: How Far Are We? An Empirical Study in C/C++ Anh The Nguyen T. H. Le Muhammad Ali Babar 36 2 0 24 Jul 2024
Course-Correction: Safety Alignment Using Synthetic Preferences Rongwu Xu Yishuo Cai Zhenhong Zhou Renjie Gu Haiqin Weng Yan Liu Tianwei Zhang Wei Xu Han Qiu 37 4 0 23 Jul 2024
Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data? J. Hayase Alisa Liu Yejin Choi Sewoong Oh Noah A. Smith 41 10 0 23 Jul 2024
Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies Chaofan Tao Qian Liu Longxu Dou Niklas Muennighoff Zhongwei Wan Ping Luo Min Lin Ngai Wong PILM 60 46 0 18 Jul 2024
Fixed and Adaptive Simultaneous Machine Translation Strategies Using Adapters Abderrahmane Issam Yusuf Can Semerci Jan Scholtes Gerasimos Spanakis 44 0 0 18 Jul 2024
Beyond Next Token Prediction: Patch-Level Training for Large Language Models Chenze Shao Fandong Meng Jie Zhou 51 1 0 17 Jul 2024
BinaryAlign: Word Alignment as Binary Sequence Labeling Gaetan Lopez Latouche M. Carbonneau Ben Swanson 33 1 0 16 Jul 2024
Genomic Language Models: Opportunities and Challenges Gonzalo Benegas Chengzhong Ye C. Albors Jianan Canal Li Yun S. Song AI4CE LM&MA ELM 53 18 0 16 Jul 2024
Bilingual Adaptation of Monolingual Foundation Models Gurpreet Gosal Yishi Xu Gokul Ramakrishnan Rituraj Joshi Avraham Sheinin ... Rahul Pal Parvez Mullah Soundar Doraiswamy Mohamed El Karim Chami Preslav Nakov CLL 34 3 0 13 Jul 2024
SPIN: Hierarchical Segmentation with Subpart Granularity in Natural Images Josh Myers-Dean Jarek Reynolds Brian Price Yifei Fan Danna Gurari 49 2 0 12 Jul 2024
Towards Chapter-to-Chapter Context-Aware Literary Translation via Large Language Models Linghao Jin Li An Xuezhe Ma 34 0 0 12 Jul 2024
MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization Orevaoghene Ahia Sachin Kumar Hila Gonen Valentin Hoffman Tomasz Limisiewicz Yulia Tsvetkov Noah A. Smith 51 4 0 11 Jul 2024
Autoregressive Speech Synthesis without Vector Quantization Lingwei Meng Long Zhou Shujie Liu Sanyuan Chen Bing Han ... Jinyu Li Sheng Zhao Xixin Wu Helen Meng Furu Wei 54 33 0 11 Jul 2024
HDT: Hierarchical Document Transformer Haoyu He Markus Flicke Jan Buchmann Iryna Gurevych Andreas Geiger 43 0 0 11 Jul 2024
LLMs' morphological analyses of complex FST-generated Finnish words Anssi Moisio Mathias Creutz M. Kurimo 52 1 0 11 Jul 2024
Automata-based constraints for language model decoding Terry Koo Frederick Liu Luheng He AI4CE 52 16 0 11 Jul 2024
Adapting LLMs to Hebrew: Unveiling DictaLM 2.0 with Enhanced Vocabulary and Instruction Capabilities Shaltiel Shmidman Avi Shmidman Amir DN Cohen Moshe Koppel 43 2 0 09 Jul 2024
Exploring Scalability of Self-Training for Open-Vocabulary Temporal Action Localization Jeongseok Hyun Su Ho Han Hyolim Kang Joon-Young Lee Seon Joo Kim VLM 42 2 0 09 Jul 2024
Mobile Edge Intelligence for Large Language Models: A Contemporary Survey Guanqiao Qu Qiyuan Chen Wei Wei Zheng Lin Xianhao Chen Kaibin Huang 42 43 0 09 Jul 2024
Large Language Models Understand Layout Weiming Li Manni Duan Dong An Yan Shao 51 3 0 08 Jul 2024
How Effective are State Space Models for Machine Translation? Hugo Pitorro Pavlo Vasylenko Marcos Vinícius Treviso André F. T. Martins Mamba 45 3 0 07 Jul 2024
A Principled Framework for Evaluating on Typologically Diverse Languages Esther Ploeger Wessel Poelman Andreas Holck Høeg-Petersen Anders Schlichtkrull Miryam de Lhoneux Johannes Bjerva 36 1 0 06 Jul 2024
Statistical investigations into the geometry and homology of random programs Jon Sporring Ken Friis Larsen 18 0 0 05 Jul 2024
Toucan: Many-to-Many Translation for 150 African Language Pairs AbdelRahim Elmadany Ife Adebara Muhammad Abdul-Mageed 39 1 0 05 Jul 2024
TokenVerse: Unifying Speech and NLP Tasks via Transducer-based ASR Shashi Kumar S. Madikeri Juan Zuluaga-Gomez Iuliia Nigmatulina Esaú Villatoro-Tello Sergio Burdisso P. Motlícek Karthik Pandia A. Ganapathiraju 46 0 0 05 Jul 2024
On the Effectiveness of Acoustic BPE in Decoder-Only TTS Bohan Li Feiyu Shen Yiwei Guo Shuai Wang Xie Chen Kai Yu 39 2 0 04 Jul 2024
Improving Self Consistency in LLMs through Probabilistic Tokenization Ashutosh Sathe Divyanshu Aggarwal Sunayana Sitaram 45 4 0 04 Jul 2024
Continual Learning Optimizations for Auto-regressive Decoder of Multilingual ASR systems Chin Yuen Kwok J. Yip Eng Siong Chng CLL 46 1 0 04 Jul 2024
Single Character Perturbations Break LLM Alignment Leon Lin Hannah Brown Kenji Kawaguchi Michael Shieh AAML 203 2 0 03 Jul 2024
A Case Study on Context-Aware Neural Machine Translation with Multi-Task Learning Ramakrishna Appicharla Baban Gain Santanu Pal Asif Ekbal Pushpak Bhattacharyya 23 1 0 03 Jul 2024
Exploiting Dialect Identification in Automatic Dialectal Text Normalization Bashar Alhafni Sarah Al-Towaity Ziyad Fawzy Fatema Nassar Fadhl Eryani Houda Bouamor Nizar Habash 45 3 0 03 Jul 2024
A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understanding Jinghui Lu Haiyang Yu Yanjie Wang Yongjie Ye Jingqun Tang ... Qi Liu Hao Feng Han Wang Hao Liu Can Huang 50 19 0 02 Jul 2024
SignCLIP: Connecting Text and Sign Language by Contrastive Learning Zifan Jiang Gerard Sant Amit Moryossef Mathias Müller Rico Sennrich Sarah Ebling VLM CLIP 42 2 0 01 Jul 2024
How to Leverage Digit Embeddings to Represent Numbers? Jasivan Sivakumar N. Moosavi 31 0 0 01 Jul 2024
Too Late to Train, Too Early To Use? A Study on Necessity and Viability of Low-Resource Bengali LLMs Tamzeed Mahfuz Satak Kumar Dey Ruwad Naswan Hasnaen Adil Khondker Salman Sayeed Haz Sameen Shahgir 39 0 0 29 Jun 2024
Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs Sheridan Feucht David Atkinson Byron C. Wallace David Bau 50 7 0 28 Jun 2024
CharED: Character-wise Ensemble Decoding for Large Language Models Kevin Gu Eva Tuecke Dmitriy Katz R. Horesh David Alvarez-Melis Mikhail Yurochkin 33 2 0 25 Jun 2024
Understanding and Mitigating Tokenization Bias in Language Models Buu Phan Marton Havasi Matthew Muckley Karen Ullrich 52 3 0 24 Jun 2024
Large Vocabulary Size Improves Large Language Models Sho Takase Ryokan Ri Shun Kiyono Takuya Kato 45 3 0 24 Jun 2024
TacoLM: GaTed Attention Equipped Codec Language Model are Efficient Zero-Shot Text to Speech Synthesizers Yakun Song Zhuo Chen Xiaofei Wang Ziyang Ma Guanrou Yang Xie Chen AuLLM 46 3 0 22 Jun 2024
Large Language Models for Link Stealing Attacks Against Graph Neural Networks Faqian Guan Tianqing Zhu Hui Sun Wanlei Zhou Philip S. Yu AAML 37 0 0 22 Jun 2024
Unsupervised Morphological Tree Tokenizer Qingyang Zhu Xiang Hu Pengyu Ji Wei Wu Kewei Tu 39 0 0 21 Jun 2024
Enhancing Idiomatic Representation in Multiple Languages via an Adaptive Contrastive Triplet Loss Wei He M. Idiart Carolina Scarton Aline Villavicencio 42 2 0 21 Jun 2024
Brain-Like Language Processing via a Shallow Untrained Multihead Attention Network Badr AlKhamissi Greta Tuckute Antoine Bosselut Martin Schrimpf 76 6 0 21 Jun 2024
Exploring Design Choices for Building Language-Specific LLMs Atula Tejaswi Nilesh Gupta Eunsol Choi 29 10 0 20 Jun 2024
How to Compute the Probability of a Word Tiago Pimentel Clara Meister 37 14 0 20 Jun 2024
Infusing clinical knowledge into tokenisers for language models Abul Hasan Jinge Wu Quang Ngoc Nguyen Salomé Andres Imane Guellil Huayu Zhang Arlene Casey Beatrice Alex Bruce Guthrie Honghan Wu 46 1 0 20 Jun 2024
On the Evaluation Practices in Multilingual NLP: Can Machine Translation Offer an Alternative to Human Translations? Rochelle Choenni Sara Rajaee Christof Monz Ekaterina Shutova 39 1 0 20 Jun 2024
Lexically Grounded Subword Segmentation Jindřich Libovický Jindřich Helcl 43 1 0 19 Jun 2024