Translatotron 2: High-quality direct speech-to-speech translation with voice preservation

19 July 2021

Ye Jia

Michelle Tadmor Ramanovich

Tal Remez

Roi Pomerantz

ArXiv PDF HTML

Papers citing "Translatotron 2: High-quality direct speech-to-speech translation with voice preservation"

50 / 50 papers shown

Title
SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translation Keqi Deng Wenxi Chen Xie Chen P. Woodland 43 0 0 22 Apr 2025
MAVFlow: Preserving Paralinguistic Elements with Conditional Flow Matching for Zero-Shot AV2AV Multilingual Translation Sungwoo Cho J. Choi Sungnyun Kim Se-Young Yun 54 0 0 14 Mar 2025
Speech to Speech Translation with Translatotron: A State of the Art Review Jules R. Kala Emmanuel Adetiba Abdultaofeek Abayom Oluwatobi E. Dare Ayodele H. Ifijeh 142 0 0 21 Feb 2025
High-Fidelity Simultaneous Speech-To-Speech Translation Tom Labiausse Laurent Mazaré Edouard Grave P. Pérez Alexandre Défossez Neil Zeghidour 100 0 0 05 Feb 2025
A Unit-based System and Dataset for Expressive Direct Speech-to-Speech Translation Anna Min Chenxu Hu Yi Ren Hang Zhao 61 0 0 01 Feb 2025
Exploiting Phonological Similarities between African Languages to achieve Speech to Speech Translation P. Ochieng D. Kaburu 18 0 0 30 Oct 2024
Zero-shot Cross-lingual Voice Transfer for TTS Fadi Biadsy Youzheng Chen Isaac Elias Kyle Kastner Gary Wang Andrew Rosenberg Bhuvana Ramabhadran 25 1 0 20 Sep 2024
CTC-based Non-autoregressive Textless Speech-to-Speech Translation Qingkai Fang Zhengrui Ma Yan Zhou Min Zhang Yang Feng 45 0 0 11 Jun 2024
Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data? Qingkai Fang Shaolei Zhang Zhengrui Ma Min Zhang Yang Feng VLM 27 1 0 11 Jun 2024
A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Any Translation Zhengrui Ma Qingkai Fang Shaolei Zhang Shoutao Guo Yang Feng Min Zhang 48 9 0 11 Jun 2024
StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning Shaolei Zhang Qingkai Fang Shoutao Guo Zhengrui Ma Min Zhang Yang Feng 29 4 0 05 Jun 2024
SimulTron: On-Device Simultaneous Speech to Speech Translation A. Agranovich Eliya Nachmani Oleg Rybakov Yifan Ding Ye Jia Nadav Bar Heiga Zen Michelle Tadmor Ramanovich 39 0 0 04 Jun 2024
SeamlessExpressiveLM: Speech Language Model for Expressive Speech-to-Speech Translation with Chain-of-Thought Hongyu Gong Bandhav Veluri 38 0 0 30 May 2024
TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation Chenyang Le Yao Qian Dongmei Wang Long Zhou Shujie Liu ... Midia Yousefi Yanmin Qian Jinyu Li Sheng Zhao Michael Zeng 34 3 0 28 May 2024
MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservation Yifan Peng Ilia Kulikov Yilin Yang Sravya Popuri Hui Lu Changhan Wang Hongyu Gong 28 4 0 19 Mar 2024
Direct Punjabi to English speech translation using discrete units Prabhjot Kaur L. A. M. Bush Weisong Shi 24 0 0 25 Feb 2024
Efficient Training for Multilingual Visual Speech Recognition: Pre-training with Discretized Visual Speech Representation Minsu Kim Jeong Hun Yeo Se Jin Park J. Choi Y. Ro 17 5 0 18 Jan 2024
TranSentence: Speech-to-speech Translation via Language-agnostic Sentence-level Speech Encoding without Language-parallel Data Seung-Bin Kim Sang-Hoon Lee Seong-Whan Lee 22 4 0 17 Jan 2024
EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-to-Speech Models Maureen de Seyssel Antony DÁvirro Adina Williams Emmanuel Dupoux 30 3 0 21 Dec 2023
AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation J. Choi Se Jin Park Minsu Kim Y. Ro 14 12 0 05 Dec 2023
DiffS2UT: A Semantic Preserving Diffusion Model for Textless Direct Speech-to-Speech Translation Yongxin Zhu Zhujin Gao Xinyuan Zhou Zhongyi Ye Linli Xu 19 2 0 26 Oct 2023
DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translation Qingkai Fang Yan Zhou Yangzhou Feng 27 6 0 11 Oct 2023
Direct Text to Speech Translation System using Acoustic Units Victoria Mingote Pablo Gimeno Luis Vicente Sameer Khurana Antoine Laurent J. Duret 15 3 0 14 Sep 2023
Sparks of Large Audio Models: A Survey and Outlook S. Latif Moazzam Shoukat Fahad Shamshad Muhammad Usama Yi Ren ... Wenwu Wang Xulong Zhang Roberto Togneri Erik Cambria Björn W. Schuller LM&MA AuLLM 29 36 0 24 Aug 2023
Many-to-Many Spoken Language Translation via Unified Speech and Text Representation Learning with Unit-to-Unit Translation Minsu Kim J. Choi Dahun Kim Y. Ro 33 10 0 03 Aug 2023
Multilingual Speech-to-Speech Translation into Multiple Target Languages Hongyu Gong Ning Dong Sravya Popuri Vedanuj Goswami Ann Lee J. Pino 42 4 0 17 Jul 2023
Towards cross-language prosody transfer for dialog Jonathan Avila Nigel G. Ward 20 6 0 09 Jul 2023
AudioPaLM: A Large Language Model That Can Speak and Listen Paul Kishan Rubenstein Chulayuth Asawaroengchai D. Nguyen Ankur Bapna Zalan Borsos ... Neil Zeghidour Yu Zhang Zhishuai Zhang Lukás Zilka Christian Frank LM&MA AuLLM VLM 35 256 0 22 Jun 2023
PolyVoice: Language Models for Speech to Speech Translation Qianqian Dong Zhiying Huang Qiao Tian Chen Xu Tom Ko ... Lu Lu Zejun Ma Yuping Wang Mingxuan Wang Yuxuan Wang 20 22 0 05 Jun 2023
Learning When to Speak: Latency and Quality Trade-offs for Simultaneous Speech-to-Speech Translation with Offline Models Liam Dugan Anshul Wadhawan Kyle Spence Chris Callison-Burch Morgan McGuire Victor Zordan OffRL 17 2 0 01 Jun 2023
StyleS2ST: Zero-shot Style Transfer for Direct Speech-to-speech Translation Kun Song Yi Ren Yinjiao Lei Chunfeng Wang Kun Wei Linfu Xie Xiang Yin Zejun Ma 24 8 0 28 May 2023
Translatotron 3: Speech to Speech Translation with Monolingual Data Eliya Nachmani Alon Levkovitch Yi-Yang Ding Chulayutsh Asawaroengchai Heiga Zen Michelle Tadmor Ramanovich 13 13 0 27 May 2023
Duplex Diffusion Models Improve Speech-to-Speech Translation Xianchao Wu DiffM 13 4 0 22 May 2023
Enhancing Speech-to-Speech Translation with Multiple TTS Targets Jiatong Shi Yun Tang Ann Lee H. Inaguma Changhan Wang J. Pino Shinji Watanabe 30 9 0 10 Apr 2023
ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkit Brian Yan Jiatong Shi Yun Tang H. Inaguma Yifan Peng ... Zhaoheng Ni Moto Hira Soumi Maiti J. Pino Shinji Watanabe 14 20 0 10 Apr 2023
Transformers in Speech Processing: A Survey S. Latif Aun Zaidi Heriberto Cuayáhuitl Fahad Shamshad Moazzam Shoukat Junaid Qadir 35 46 0 21 Mar 2023
Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling Zi-Hua Zhang Long Zhou Chengyi Wang Sanyuan Chen Yu Wu ... Huaming Wang Jinyu Li Lei He Sheng Zhao Furu Wei VLM 28 170 0 07 Mar 2023
A Holistic Cascade System, benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translation Wen-Chin Huang Benjamin Peloquin Justine T. Kao Changhan Wang Hongyu Gong Elizabeth Salesky Yossi Adi Ann Lee Peng-Jen Chen 25 15 0 25 Jan 2023
UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units H. Inaguma Sravya Popuri Ilia Kulikov Peng-Jen Chen Changhan Wang Yu-An Chung Yun Tang Ann Lee Shinji Watanabe J. Pino 35 51 0 15 Dec 2022
Direct Speech-to-speech Translation without Textual Annotation using Bottleneck Features Junhui Zhang Junjie Pan Xiang Yin Zejun Ma 11 0 0 12 Dec 2022
Speech-to-Speech Translation For A Real-world Unwritten Language Peng-Jen Chen Ke M. Tran Yilin Yang Jingfei Du Justine T. Kao ... Sravya Popuri Changhan Wang J. Pino Wei-Ning Hsu Ann Lee 13 25 0 11 Nov 2022
SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations Paul-Ambroise Duquenne Hongyu Gong Ning Dong Jingfei Du Ann Lee Vedanuj Goswani Changhan Wang J. Pino Benoît Sagot Holger Schwenk 21 34 0 08 Nov 2022
Textless Direct Speech-to-Speech Translation with Discrete Speech Representation Xinjian Li Ye Jia Chung-Cheng Chiu 20 23 0 31 Oct 2022
Make More of Your Data: Minimal Effort Data Augmentation for Automatic Speech Recognition and Translation Tsz Kin Lam Shigehiko Schamoni Stefan Riezler VLM 25 8 0 27 Oct 2022
Simple and Effective Unsupervised Speech Translation Changhan Wang H. Inaguma Peng-Jen Chen Ilia Kulikov Yun Tang Wei-Ning Hsu Michael Auli J. Pino SSL 21 14 0 18 Oct 2022
WaveFit: An Iterative and Non-autoregressive Neural Vocoder based on Fixed-Point Iteration Yuma Koizumi Kohei Yatabe Heiga Zen M. Bacchiani DiffM 42 28 0 03 Oct 2022
Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation Ye Jia Yifan Ding Ankur Bapna Colin Cherry Yu Zhang Alexis Conneau Nobuyuki Morioka 31 20 0 24 Mar 2022
CVSS Corpus and Massively Multilingual Speech-to-Speech Translation Yeting Jia Michelle Tadmor Ramanovich Quan Wang Heiga Zen SLR 12 65 0 11 Jan 2022
Direct Simultaneous Speech-to-Speech Translation with Variational Monotonic Multihead Attention Xutai Ma Hongyu Gong Danni Liu Ann Lee Yun Tang Peng-Jen Chen Wei-Ning Hsu P. Koehn J. Pino 54 8 0 15 Oct 2021
Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis Ye Jia Yu Zhang Ron J. Weiss Quan Wang Jonathan Shen ... Z. Chen Patrick Nguyen Ruoming Pang Ignacio López Moreno Yonghui Wu 201 817 0 12 Jun 2018