Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM

24 May 2023

Chulayutsh Asawaroengchai

Soroosh Mariooryad

Ehud Rivlin

RJ Skerry-Ryan

Michelle Tadmor Ramanovich

AuLLM

ArXiv PDF HTML

Papers citing "Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM"

34 / 34 papers shown

Title
VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model Zuwei Long Yunhang Shen Chaoyou Fu Heting Gao Lijiang Li ... Jinlong Peng Haoyu Cao Ke Li R. Ji Xing Sun 30 0 0 06 May 2025
Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Play Yemin Shi Yu Shu Siwei Dong Guangyi Liu Jaward Sesay Jingwen Li Zhiting Hu AuLLM VLM 32 0 0 05 May 2025
LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesis Qingkai Fang Yan Zhou Shoutao Guo Shaolei Zhang Yang Feng AuLLM 48 0 0 05 May 2025
On The Landscape of Spoken Language Models: A Comprehensive Survey Siddhant Arora Kai-Wei Chang Chung-Ming Chien Yifan Peng Haibin Wu Yossi Adi Emmanuel Dupoux Hung-yi Lee Karen Livescu Shinji Watanabe 36 1 0 11 Apr 2025
VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generation Yuhao Wang Heyang Liu Ziyang Cheng Ronghua Wu Qunshan Gu Yanfeng Wang Yu Wang 37 0 0 05 Apr 2025
From Text to Graph: Leveraging Graph Neural Networks for Enhanced Explainability in NLP Fabio Yáñez-Romero Andrés Montoyo Armando Suárez Yoan Gutiérrez Ruslan Mitkov 41 0 0 02 Apr 2025
Chirp Localization via Fine-Tuned Transformer Model: A Proof-of-Concept Study N. Bahador M. Lankarany 34 0 0 24 Mar 2025
Nexus-O: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision Che Liu Yingji Zhang D. Zhang Weijie Zhang Chenggong Gong ... André Freitas Qifan Wang Z. Xu Rongjuncheng Zhang Yong Dai AuLLM 61 0 0 26 Feb 2025
Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction Tianpeng Li J. Liu Tao Zhang Yuanbo Fang Da Pan ... Guosheng Dong Jianhua Xu Haoze Sun Zenan Zhou Weipeng Chen AuLLM 45 3 0 24 Feb 2025
Slamming: Training a Speech Language Model on One GPU in a Day Gallil Maimon Avishai Elmakies Yossi Adi 38 3 0 19 Feb 2025
SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words Junyi Ao Yuancheng Wang Xiaohai Tian Dekun Chen J. Zhang Lu Lu Y. Wang Haizhou Li Z. Wu AuLLM 71 16 0 17 Jan 2025
Autoregressive Speech Synthesis with Next-Distribution Prediction Xinfa Zhu WenJie Tian Lei Xie VLM 165 4 0 22 Dec 2024
Continuous Speech Tokens Makes LLMs Robust Multi-Modality Learners Ze Yuan Yanqing Liu Shujie Liu Sheng Zhao AuLLM 74 0 0 06 Dec 2024
Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback Guan-Ting Lin Prashanth Gurunath Shivakumar Aditya Gourav Yile Gu Ankur Gandhe Hung-yi Lee I. Bulyko 21 8 0 04 Nov 2024
Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM Xiong Wang Yangze Li Chaoyou Fu Yunhang Shen Lei Xie Ke Li Xing Sun Long Ma AuLLM MLLM 29 25 0 01 Nov 2024
Get Large Language Models Ready to Speak: A Late-fusion Approach for Speech Generation Maohao Shen Shun Zhang Jilong Wu Zhiping Xiu Ehab AlBadawy Yiting Lu M. Seltzer Qing He 28 2 0 27 Oct 2024
Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decoding Tan Dat Nguyen Ji-Hoon Kim Jeongsoo Choi Shukjae Choi Jinseok Park Younglo Lee Joon Son Chung 26 0 0 17 Oct 2024
IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilities Xin Zhang Xiang Lyu Zhihao Du Qian Chen Dong Zhang ... Yuxuan Wang Bin Zhang Heng Lu Yaqian Zhou Xipeng Qiu AuLLM 22 5 0 09 Oct 2024
Recent Advances in Speech Language Models: A Survey Wenqian Cui Dianzhi Yu Xiaoqi Jiao Ziqiao Meng Guangyan Zhang Qichao Wang Yiwen Guo Irwin King AuLLM 57 14 0 01 Oct 2024
Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents Bandhav Veluri Benjamin Peloquin Bokai Yu Hongyu Gong Shyamnath Gollakota AuLLM OffRL 29 13 0 23 Sep 2024
Salmon: A Suite for Acoustic Language Model Evaluation Gallil Maimon Amit Roth Yossi Adi ELM AuLLM 49 5 0 11 Sep 2024
LAST: Language Model Aware Speech Tokenization A. Turetzky Yossi Adi 13 2 0 05 Sep 2024
SpeechPrompt: Prompting Speech Language Models for Speech Processing Tasks Kai-Wei Chang Haibin Wu Yu-Kai Wang Yuan-Kuei Wu Hua Shen Wei-Cheng Tseng Iu-thing Kang Shang-Wen Li Hung-yi Lee 28 1 0 23 Aug 2024
PSLM: Parallel Generation of Text and Speech with LLMs for Low-Latency Spoken Dialogue Systems Kentaro Mitsui Koh Mitsuda Toshiaki Wakatsuki Yukiya Hono Kei Sawada 25 2 0 18 Jun 2024
Teaching a Multilingual Large Language Model to Understand Multilingual Speech via Multi-Instructional Training Pavel Denisov Ngoc Thang Vu 23 1 0 16 Apr 2024
SpiRit-LM: Interleaved Spoken and Written Language Model Tu Nguyen Benjamin Muller Bokai Yu Marta R. Costa-jussá Maha Elbayad ... Itai Gat Gabriel Synnaeve Juan Pino Benoît Sagot Emmanuel Dupoux AuLLM VLM 36 32 0 08 Feb 2024
Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages Yu Zhang Wei Han James Qin Yongqiang Wang Ankur Bapna ... Pedro J. Moreno Chung-Cheng Chiu J. Schalkwyk Franccoise Beaufays Yonghui Wu VLM 77 249 0 02 Mar 2023
Residual Adapters for Few-Shot Text-to-Speech Speaker Adaptation Nobuyuki Morioka Heiga Zen Nanxin Chen Yu Zhang Yifan Ding 24 16 0 28 Oct 2022
GLM-130B: An Open Bilingual Pre-trained Model Aohan Zeng Xiao Liu Zhengxiao Du Zihan Wang Hanyu Lai ... Jidong Zhai Wenguang Chen Peng-Zhen Zhang Yuxiao Dong Jie Tang BDL LRM 240 1,070 0 05 Oct 2022
WaveFit: An Iterative and Non-autoregressive Neural Vocoder based on Fixed-Point Iteration Yuma Koizumi Kohei Yatabe Heiga Zen M. Bacchiani DiffM 34 28 0 03 Oct 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 315 8,261 0 28 Jan 2022
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing Junyi Ao Rui Wang Long Zhou Chengyi Wang Shuo Ren ... Yu Zhang Zhihua Wei Yao Qian Jinyu Li Furu Wei 110 148 0 14 Oct 2021
Generative Spoken Language Modeling from Raw Audio Kushal Lakhotia Evgeny Kharitonov Wei-Ning Hsu Yossi Adi Adam Polyak ... Tu Nguyen Jade Copet Alexei Baevski A. Mohamed Emmanuel Dupoux AuLLM 172 336 0 01 Feb 2021
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 220 3,054 0 23 Jan 2020