DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT

5 October 2021

Papers citing "DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT"

50 / 116 papers shown

Title
A Small and Fast BERT for Chinese Medical Punctuation Restoration Tongtao Ling Chen Liao Lei Chen Shilei Huang Yi Liu MedIm 11 1 0 24 Aug 2023
On-Device Constrained Self-Supervised Speech Representation Learning for Keyword Spotting via Knowledge Distillation Gene-Ping Yang Yue Gu Qingming Tang Dongsu Du Yuzong Liu 6 5 0 06 Jul 2023
What Do Self-Supervised Speech Models Know About Words? Ankita Pasad C. Chien Shane Settle Karen Livescu SSL 16 26 0 30 Jun 2023
Accelerating Transducers through Adjacent Token Merging Yuang Li Yu-Huan Wu Jinyu Li Shujie Liu 17 4 0 28 Jun 2023
Toward Leveraging Pre-Trained Self-Supervised Frontends for Automatic Singing Voice Understanding Tasks: Three Case Studies Yuya Yamamoto 12 2 0 22 Jun 2023
Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Compute William Chen Xuankai Chang Yifan Peng Zhaoheng Ni Soumi Maiti Shinji Watanabe SSL 10 25 0 11 Jun 2023
Task-Agnostic Structured Pruning of Speech Representation Models Haoyu Wang Siyuan Wang Weiqiang Zhang Hongbin Suo Yulong Wan VLM 14 14 0 02 Jun 2023
DistilXLSR: A Light Weight Cross-Lingual Speech Representation Model Haoyu Wang Siyuan Wang Weiqiang Zhang Jinfeng Bai 19 2 0 02 Jun 2023
Speech Self-Supervised Representation Benchmarking: Are We Doing it Right? Salah Zaiem Youcef Kemiche Titouan Parcollet S. Essid Mirco Ravanelli SSL 4 23 0 01 Jun 2023
Accurate and Structured Pruning for Efficient Automatic Speech Recognition Huiqiang Jiang Li Lyna Zhang Yuang Li Yu-Huan Wu Shijie Cao Ting Cao Yuqing Yang Jinyu Li Mao Yang Lili Qiu CVBM 28 9 0 31 May 2023
Leveraging Semantic Information for Efficient Self-Supervised Emotion Recognition with Audio-Textual Distilled Models Danilo de Oliveira N. Prabhu Timo Gerkmann 15 5 0 30 May 2023
DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Models Yifan Peng Yui Sudo Muhammad Shakeel Shinji Watanabe 11 37 0 28 May 2023
One-Step Knowledge Distillation and Fine-Tuning in Using Large Pre-Trained Self-Supervised Learning Models for Speaker Verification Ju-Sung Heo Chan-yeong Lim Ju-ho Kim Hyun-Seo Shin Ha-Jin Yu 6 2 0 27 May 2023
ABC-KD: Attention-Based-Compression Knowledge Distillation for Deep Learning-Based Noise Suppression Yixin Wan Yuan-yuan Zhou Xiulian Peng Kai-Wei Chang Yan Lu 17 3 0 26 May 2023
Masked Modeling Duo for Speech: Specializing General-Purpose Audio Representation to Speech using Denoising Distillation Daisuke Niizumi Daiki Takeuchi Yasunori Ohishi Noboru Harada K. Kashino 34 3 0 23 May 2023
Can Self-Supervised Neural Representations Pre-Trained on Human Speech distinguish Animal Callers? Eklavya Sarkar Mathew Magimai.-Doss 8 11 0 23 May 2023
Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillation Kangwook Jang Sungnyun Kim Se-Young Yun Hoi-Rim Kim 10 5 0 19 May 2023
Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clustering Heng-Jui Chang Alexander H. Liu James R. Glass SSL 17 20 0 18 May 2023
Whisper-KDQ: A Lightweight Whisper via Guided Knowledge Distillation and Quantization for Efficient ASR Hang Shao Wei Wang Bei Liu Xun Gong Haoyu Wang Y. Qian 89 9 0 18 May 2023
DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learning Alexander H. Liu Heng-Jui Chang Michael Auli Wei-Ning Hsu James R. Glass 13 24 0 17 May 2023
Knowledge Distillation from Multiple Foundation Models for End-to-End Speech Recognition Xiaoyu Yang Qiujia Li C. Zhang P. Woodland 10 6 0 20 Mar 2023
DistillW2V2: A Small and Streaming Wav2vec 2.0 Based ASR Model Yanzhe Fu Yueteng Kang Songjun Cao Long Ma 6 7 0 16 Mar 2023
I3D: Transformer architectures with input-dependent dynamic depth for speech recognition Yifan Peng Jaesong Lee Shinji Watanabe 17 16 0 14 Mar 2023
Lightweight feature encoder for wake-up word detection based on self-supervised speech representation Hyungjun Lim Younggwan Kim Ki-Woong Yeom E. Seo Hoodong Lee Stanley Jungkyu Choi Honglak Lee 13 1 0 14 Mar 2023
Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models: A Comparative Study Salah Zaiem Robin Algayres Titouan Parcollet S. Essid Mirco Ravanelli 35 14 0 12 Mar 2023
UNFUSED: UNsupervised Finetuning Using SElf supervised Distillation Ashish Seth Sreyan Ghosh S. Umesh Dinesh Manocha 17 0 0 10 Mar 2023
Self-supervised speech representation learning for keyword-spotting with light-weight transformers Chenyang Gao Yue Gu Francesco Calivá Yuzong Liu OffRL 22 3 0 07 Mar 2023
Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understanding Yifan Peng Kwangyoun Kim Felix Wu Prashant Sridhar Shinji Watanabe 11 34 0 27 Feb 2023
A low latency attention module for streaming self-supervised speech representation learning Jianbo Ma Siqi Pan Deepak Chandran A. Fanelli Richard Cartwright 12 0 0 27 Feb 2023
Ensemble knowledge distillation of self-supervised speech models Kuan-Po Huang Tzu-hsun Feng Yu-Kuan Fu Tsung-Yuan Hsu Po-Chieh Yen Wei-Cheng Tseng Kai-Wei Chang Hung-yi Lee 20 16 0 24 Feb 2023
RobustDistiller: Compressing Universal Speech Representations for Enhanced Environment Robustness Heitor R. Guimarães Arthur Pimentel Anderson R. Avila Mehdi Rezagholizadeh Boxing Chen Tiago H. Falk 32 10 0 18 Feb 2023
Don't Be So Sure! Boosting ASR Decoding via Confidence Relaxation Tomer Wullach Shlomo E. Chazan 16 1 0 27 Dec 2022
Efficient Speech Representation Learning with Low-Bit Quantization Ching-Feng Yeh Wei-Ning Hsu Paden Tomasello Abdel-rahman Mohamed MQ 12 9 0 14 Dec 2022
Model Extraction Attack against Self-supervised Speech Models Tsung-Yuan Hsu Chen An Li Tung-Yu Wu Hung-yi Lee 14 1 0 29 Nov 2022
Compressing Transformer-based self-supervised models for speech processing Tzu-Quan Lin Tsung-Huan Yang Chun-Yao Chang Kuang-Ming Chen Tzu-hsun Feng Hung-yi Lee Hao Tang 14 6 0 17 Nov 2022
Improving the Robustness of DistilHuBERT to Unseen Noisy Conditions via Data Augmentation, Curriculum Learning, and Multi-Task Enhancement Heitor R. Guimarães Arthur Pimentel Anderson R. Avila Mehdi Rezagholizadeh Tiago H. Falk 17 3 0 12 Nov 2022
Enhancing and Adversarial: Improve ASR with Speaker Labels Wei Zhou Haotian Wu Jingjing Xu Mohammad Zeineldeen Christoph Luscher Ralf Schluter Hermann Ney 6 8 0 11 Nov 2022
Once-for-All Sequence Compression for Self-Supervised Speech Models Hsuan-Jui Chen Yen Meng Hung-yi Lee 6 4 0 04 Nov 2022
Application of Knowledge Distillation to Multi-task Speech Representation Learning Mine Kerpicci V. Nguyen Shuhua Zhang Erik M. Visser 12 0 0 29 Oct 2022
Exploring Effective Distillation of Self-Supervised Speech Models for Automatic Speech Recognition Yujin Wang Changli Tang Ziyang Ma Zhisheng Zheng Xie Chen Weiqiang Zhang 19 1 0 27 Oct 2022
Efficient Utilization of Large Pre-Trained Models for Low Resource ASR Peter Vieting Christoph Luscher Julian Dierkes Ralf Schluter Hermann Ney 30 5 0 26 Oct 2022
Fast Yet Effective Speech Emotion Recognition with Self-distillation Zhao Ren Thanh Tam Nguyen Yi Chang Björn W. Schuller 10 11 0 26 Oct 2022
Real-time Speech Interruption Analysis: From Cloud to Client Deployment Quchen Fu Szu-Wei Fu Yaran Fan Yu-Huan Wu Zhuo Chen J. Gupchup Ross Cutler 23 0 0 24 Oct 2022
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning Tzu-hsun Feng Annie Dong Ching-Feng Yeh Shu-Wen Yang Tzu-Quan Lin ... Xuankai Chang Shinji Watanabe Abdel-rahman Mohamed Shang-Wen Li Hung-yi Lee ELM SSL 13 33 0 16 Oct 2022
$RedApt: An Adaptor for wav2vec 2 Encoding \\ Faster and Smaller Speech Translation without Quality Compromise$ RedApt: An Adaptor for wav2vec 2 Encoding \\ Faster and Smaller Speech Translation without Quality Compromise Jinming Zhao Haomiao Yang Gholamreza Haffari Ehsan Shareghi VLM 11 2 0 16 Oct 2022
Improving generalizability of distilled self-supervised speech processing models under distorted settings Kuan-Po Huang Yu-Kuan Fu Tsung-Yuan Hsu Fabian Ritter Gutierrez Fan Wang Liang-Hsuan Tseng Yu Zhang Hung-yi Lee 6 13 0 14 Oct 2022
On Compressing Sequences for Self-Supervised Speech Models Yen Meng Hsuan-Jui Chen Jiatong Shi Shinji Watanabe Paola García Hung-yi Lee Hao Tang SSL 8 14 0 13 Oct 2022
On the Utility of Self-supervised Models for Prosody-related Tasks Guan-Ting Lin Chiyu Feng Wei-Ping Huang Yuan Tseng Tzu-Han Lin Chen An Li Hung-yi Lee Nigel G. Ward 11 47 0 13 Oct 2022
SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model Yi-Jen Shih Hsuan-Fu Wang Heng-Jui Chang Layne Berry Hung-yi Lee David F. Harwath VLM CLIP 38 32 0 03 Oct 2022
Analyzing Robustness of End-to-End Neural Models for Automatic Speech Recognition Goutham Rajendran Wei Zou 10 2 0 17 Aug 2022