LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT

29 March 2022

Rui Wang

Haizhou Li

Papers citing "LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT"

46 / 46 papers shown

Title
StableQuant: Layer Adaptive Post-Training Quantization for Speech Foundation Models Yeona Hong Hyewon Han Woo-Jin Chung Hong-Goo Kang MQ 28 0 0 21 Apr 2025
Automatic Speech Recognition with BERT and CTC Transformers: A Review Noussaiba Djeffal Hamza Kheddar Djamel Addou A. Mazari Yassine Himeur 37 16 0 12 Oct 2024
Speech Representation Learning Revisited: The Necessity of Separate Learnable Parameters and Robust Data Augmentation Hemant Yadav Sunayana Sitaram R. Shah SSL 42 0 0 20 Aug 2024
Dynamic Encoder Size Based on Data-Driven Layer-wise Pruning for Speech Recognition Jingjing Xu Wei Zhou Zijian Yang Eugen Beck Ralf Schlueter 20 1 0 10 Jul 2024
One-pass Multiple Conformer and Foundation Speech Systems Compression and Quantization Using An All-in-one Neural Model Zhaoqing Li Haoning Xu Tianzi Wang Shoukang Hu Zengrui Jin Shujie Hu Jiajun Deng Mingyu Cui Mengzhe Geng Xunying Liu MQ 21 1 0 14 Jun 2024
GenDistiller: Distilling Pre-trained Language Models based on an Autoregressive Generative Model Yingying Gao Shilei Zhang Chao Deng Junlan Feng 19 0 0 12 Jun 2024
DAISY: Data Adaptive Self-Supervised Early Exit for Speech Representation Models T. Lin Hung-yi Lee Hao Tang 27 1 0 08 Jun 2024
A Large-Scale Evaluation of Speech Foundation Models Shu-Wen Yang Heng-Jui Chang Zili Huang Andy T. Liu Cheng-I Jeff Lai ... Kushal Lakhotia Shang-Wen Li Abdelrahman Mohamed Shinji Watanabe Hung-yi Lee 38 19 0 15 Apr 2024
Efficiently Distilling LLMs for Edge Applications Achintya Kundu Fabian Lim Aaron Chew L. Wynter Penny Chong Rhui Dih Lee 27 6 0 01 Apr 2024
Domain Adaptation in Intent Classification Systems: A Review Jesse Atuhurra Hidetaka Kamigaito Taro Watanabe Eric Nichols 35 0 0 26 Mar 2024
An Efficient End-to-End Approach to Noise Invariant Speech Features via Multi-Task Learning Heitor R. Guimarães Arthur Pimentel Anderson R. Avila Mehdi Rezagholizadeh Boxing Chen Tiago H. Falk 54 1 0 13 Mar 2024
Noise robust distillation of self-supervised speech models via correlation metrics Fabian Ritter Gutierrez Kuan-Po Huang Dianwen Ng Jeremy H. M. Wong Hung-yi Lee Chng Eng Siong Nancy F. Chen 16 1 0 19 Dec 2023
Efficiency-oriented approaches for self-supervised speech representation learning Luis Lugo Valentin Vielzeuf SSL 19 1 0 18 Dec 2023
STaR: Distilling Speech Temporal Relation for Lightweight Speech Self-Supervised Learning Models Kangwook Jang Sungnyun Kim Hoi-Rim Kim 26 1 0 14 Dec 2023
Distilling HuBERT with LSTMs via Decoupled Knowledge Distillation Danilo de Oliveira Timo Gerkmann VLM 20 3 0 18 Sep 2023
CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoders Heng-Jui Chang Ning Dong Ruslan Mavlyutov Sravya Popuri Yu-An Chung 40 6 0 14 Sep 2023
Mispronunciation detection using self-supervised speech representations Jazmín Vidal Pablo Riera Luciana Ferrer 6 1 0 30 Jul 2023
A Survey of Techniques for Optimizing Transformer Inference Krishna Teja Chitty-Venkata Sparsh Mittal M. Emani V. Vishwanath Arun Somani 29 62 0 16 Jul 2023
On-Device Constrained Self-Supervised Speech Representation Learning for Keyword Spotting via Knowledge Distillation Gene-Ping Yang Yue Gu Qingming Tang Dongsu Du Yuzong Liu 14 5 0 06 Jul 2023
Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Compute William Chen Xuankai Chang Yifan Peng Zhaoheng Ni Soumi Maiti Shinji Watanabe SSL 15 25 0 11 Jun 2023
Task-Agnostic Structured Pruning of Speech Representation Models Haoyu Wang Siyuan Wang Weiqiang Zhang Hongbin Suo Yulong Wan VLM 16 14 0 02 Jun 2023
DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Models Yifan Peng Yui Sudo Muhammad Shakeel Shinji Watanabe 13 37 0 28 May 2023
One-Step Knowledge Distillation and Fine-Tuning in Using Large Pre-Trained Self-Supervised Learning Models for Speaker Verification Ju-Sung Heo Chan-yeong Lim Ju-ho Kim Hyun-Seo Shin Ha-Jin Yu 11 2 0 27 May 2023
Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillation Kangwook Jang Sungnyun Kim Se-Young Yun Hoi-Rim Kim 19 5 0 19 May 2023
Whisper-KDQ: A Lightweight Whisper via Guided Knowledge Distillation and Quantization for Efficient ASR Hang Shao Wei Wang Bei Liu Xun Gong Haoyu Wang Y. Qian 94 10 0 18 May 2023
DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learning Alexander H. Liu Heng-Jui Chang Michael Auli Wei-Ning Hsu James R. Glass 19 24 0 17 May 2023
Small-footprint slimmable networks for keyword spotting Zuhaib Akhtar Mohammad Omar Khursheed Dongsu Du Yuzong Liu 11 2 0 21 Apr 2023
Weight-sharing Supernet for Searching Specialized Acoustic Event Classification Networks Across Device Constraints Guan-Ting Lin Qingming Tang Chieh-Chi Kao Viktor Rozgic Chao Wang 15 0 0 18 Mar 2023
DistillW2V2: A Small and Streaming Wav2vec 2.0 Based ASR Model Yanzhe Fu Yueteng Kang Songjun Cao Long Ma 6 7 0 16 Mar 2023
I3D: Transformer architectures with input-dependent dynamic depth for speech recognition Yifan Peng Jaesong Lee Shinji Watanabe 22 19 0 14 Mar 2023
Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models: A Comparative Study Salah Zaiem Robin Algayres Titouan Parcollet S. Essid Mirco Ravanelli 43 14 0 12 Mar 2023
Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understanding Yifan Peng Kwangyoun Kim Felix Wu Prashant Sridhar Shinji Watanabe 19 34 0 27 Feb 2023
RobustDistiller: Compressing Universal Speech Representations for Enhanced Environment Robustness Heitor R. Guimarães Arthur Pimentel Anderson R. Avila Mehdi Rezagholizadeh Boxing Chen Tiago H. Falk 61 10 0 18 Feb 2023
Continual Learning for On-Device Speech Recognition using Disentangled Conformers Anuj Diwan Ching-Feng Yeh Wei-Ning Hsu Paden Tomasello Eunsol Choi David F. Harwath Abdel-rahman Mohamed CLL BDL 18 7 0 02 Dec 2022
Compressing Transformer-based self-supervised models for speech processing Tzu-Quan Lin Tsung-Huan Yang Chun-Yao Chang Kuang-Ming Chen Tzu-hsun Feng Hung-yi Lee Hao Tang 25 6 0 17 Nov 2022
Improving the Robustness of DistilHuBERT to Unseen Noisy Conditions via Data Augmentation, Curriculum Learning, and Multi-Task Enhancement Heitor R. Guimarães Arthur Pimentel Anderson R. Avila Mehdi Rezagholizadeh Tiago H. Falk 22 3 0 12 Nov 2022
Once-for-All Sequence Compression for Self-Supervised Speech Models Hsuan-Jui Chen Yen Meng Hung-yi Lee 8 4 0 04 Nov 2022
Application of Knowledge Distillation to Multi-task Speech Representation Learning Mine Kerpicci V. Nguyen Shuhua Zhang Erik M. Visser 20 0 0 29 Oct 2022
Exploring Effective Distillation of Self-Supervised Speech Models for Automatic Speech Recognition Yujin Wang Changli Tang Ziyang Ma Zhisheng Zheng Xie Chen Weiqiang Zhang 29 1 0 27 Oct 2022
Real-time Speech Interruption Analysis: From Cloud to Client Deployment Quchen Fu Szu-Wei Fu Yaran Fan Yu-Huan Wu Zhuo Chen J. Gupchup Ross Cutler 26 0 0 24 Oct 2022
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning Tzu-hsun Feng Annie Dong Ching-Feng Yeh Shu-Wen Yang Tzu-Quan Lin ... Xuankai Chang Shinji Watanabe Abdel-rahman Mohamed Shang-Wen Li Hung-yi Lee ELM SSL 19 33 0 16 Oct 2022
SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model Yi-Jen Shih Hsuan-Fu Wang Heng-Jui Chang Layne Berry Hung-yi Lee David F. Harwath VLM CLIP 38 32 0 03 Oct 2022
FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learning Yeonghyeon Lee Kangwook Jang Jahyun Goo Youngmoon Jung Hoi-Rim Kim 10 28 0 01 Jul 2022
Adversarial Speaker Distillation for Countermeasure Model on Automatic Speaker Verification Yen-Lun Liao Xuan-Bo Chen Chung-Che Wang J. Jang AAML 15 8 0 31 Mar 2022
Oracle Teacher: Leveraging Target Information for Better Knowledge Distillation of CTC Models J. Yoon H. Kim Hyeon Seung Lee Sunghwan Ahn N. Kim 26 1 0 05 Nov 2021
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing Junyi Ao Rui Wang Long Zhou Chengyi Wang Shuo Ren ... Yu Zhang Zhihua Wei Yao Qian Jinyu Li Furu Wei 110 192 0 14 Oct 2021