Joint Training And Decoding for Multilingual End-to-End Simultaneous Speech Translation

Recent studies on end-to-end speech translation(ST) have facilitated the exploration of multilingual end-to-end ST and end-to-end simultaneous ST. In this paper, we investigate end-to-end simultaneous speech translation in a one-to-many multilingual setting which is closer to applications in real scenarios. We explore a separate decoder architecture and a unified architecture for joint synchronous training in this scenario. To further explore knowledge transfer across languages, we propose an asynchronous training strategy on the proposed unified decoder architecture. A multi-way aligned multilingual end-to-end ST dataset was curated as a benchmark testbed to evaluate our methods. Experimental results demonstrate the effectiveness of our models on the collected dataset. Our codes and data are available at:this https URL.

@article{huang2025_2503.11080,
  title={ Joint Training And Decoding for Multilingual End-to-End Simultaneous Speech Translation },
  author={ Wuwei Huang and Renren Jin and Wen Zhang and Jian Luan and Bin Wang and Deyi Xiong },
  journal={arXiv preprint arXiv:2503.11080},
  year={ 2025 }
}