HAT: Hardware-Aware Transformers for Efficient Natural Language Processing

28 May 2020

Zhijian Liu

Chuang Gan

Song Han

ArXiv PDF HTML

Papers citing "HAT: Hardware-Aware Transformers for Efficient Natural Language Processing"

50 / 56 papers shown

Title
Enhancing Video Understanding: Deep Neural Networks for Spatiotemporal Analysis Amir Hosein Fadaei M. Dehaqani 42 0 0 11 Feb 2025
Merino: Entropy-driven Design for Generative Language Models on IoT Devices Youpeng Zhao Ming Lin Huadong Tang Qiang Wu Jun Wang 75 0 0 28 Jan 2025
Multi-objective Differentiable Neural Architecture Search R. Sukthanker Arber Zela B. Staffler Samuel Dooley Josif Grabocka Frank Hutter 40 1 0 28 Feb 2024
TransAxx: Efficient Transformers with Approximate Computing Dimitrios Danopoulos Georgios Zervakis Dimitrios Soudris Jörg Henkel ViT 42 2 0 12 Feb 2024
DistDNAS: Search Efficient Feature Interactions within 2 Hours Tunhou Zhang W. Wen Igor Fedorov Xi Liu Buyun Zhang ... Wen-Yen Chen Yiping Han Feng Yan Hai Helen Li Yiran Chen 13 1 0 01 Nov 2023
Evolutionary Neural Architecture Search for Transformer in Knowledge Tracing Shangshang Yang Xiaoshan Yu Ye Tian Xueming Yan Haiping Ma Xingyi Zhang ViT KELM AI4Ed 16 2 0 02 Oct 2023
InstaTune: Instantaneous Neural Architecture Search During Fine-Tuning S. N. Sridhar Souvik Kundu Sairam Sundaresan Maciej Szankin Anthony Sarah 17 3 0 29 Aug 2023
Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers Tobias Christian Nauen Sebastián M. Palacio Federico Raue Andreas Dengel 37 3 0 18 Aug 2023
Open-TransMind: A New Baseline and Benchmark for 1st Foundation Model Challenge of Intelligent Transportation Yifeng Shi Feng Lv Xinliang Wang Chunlong Xia Shaojie Li Shu-Zhen Yang Teng Xi Gang Zhang VLM 35 13 0 12 Apr 2023
SwiftTron: An Efficient Hardware Accelerator for Quantized Transformers Alberto Marchisio David Durà Maurizio Capra Maurizio Martina Guido Masera Muhammad Shafique 26 17 0 08 Apr 2023
System-status-aware Adaptive Network for Online Streaming Video Understanding Lin Geng Foo Jia Gong Zhipeng Fan J. Liu AI4TS 27 15 0 28 Mar 2023
EdgeTran: Co-designing Transformers for Efficient Inference on Mobile Edge Platforms Shikhar Tuli N. Jha 34 3 0 24 Mar 2023
DetOFA: Efficient Training of Once-for-All Networks for Object Detection Using Path Filter Yuiko Sakuma Masato Ishii T. Narihira 28 2 0 23 Mar 2023
Transformers in Speech Processing: A Survey S. Latif Aun Zaidi Heriberto Cuayáhuitl Fahad Shamshad Moazzam Shoukat Junaid Qadir 42 47 0 21 Mar 2023
The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deployment Jared Fernandez Jacob Kahn Clara Na Yonatan Bisk Emma Strubell FedML 25 10 0 13 Feb 2023
6-DoF Robotic Grasping with Transformer Zhenjie Zhao Han Yu Hang Wu Xuebo Zhang ViT 28 0 0 29 Jan 2023
Vision Transformer Computation and Resilience for Dynamic Inference Kavya Sreedhar Jason Clemons Rangharajan Venkatesan S. Keckler M. Horowitz 24 2 0 06 Dec 2022
HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformers Peiyan Dong Mengshu Sun Alec Lu Yanyue Xie Li-Yu Daisy Liu ... Xin Meng Z. Li Xue Lin Zhenman Fang Yanzhi Wang ViT 26 58 0 15 Nov 2022
Efficiently Scaling Transformer Inference Reiner Pope Sholto Douglas Aakanksha Chowdhery Jacob Devlin James Bradbury Anselm Levskaya Jonathan Heek Kefan Xiao Shivani Agrawal J. Dean 32 292 0 09 Nov 2022
NASA: Neural Architecture Search and Acceleration for Hardware Inspired Hybrid Networks Huihong Shi Haoran You Yang Katie Zhao Zhongfeng Wang Yingyan Lin 56 7 0 24 Oct 2022
Efficient Methods for Natural Language Processing: A Survey Marcos Vinícius Treviso Ji-Ung Lee Tianchu Ji Betty van Aken Qingqing Cao ... Emma Strubell Niranjan Balasubramanian Leon Derczynski Iryna Gurevych Roy Schwartz 28 109 0 31 Aug 2022
Neural Architecture Search on Efficient Transformers and Beyond Zexiang Liu Dong Li Kaiyue Lu Zhen Qin Weixuan Sun Jiacheng Xu Yiran Zhong 27 19 0 28 Jul 2022
UFO: Unified Feature Optimization Teng Xi Yifan Sun Deli Yu Bi Li Nan Peng ... Haocheng Feng Junyu Han Jingtuo Liu Errui Ding Jingdong Wang 32 10 0 21 Jul 2022
EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithm Jiangning Zhang Xiangtai Li Yabiao Wang Chengjie Wang Yibo Yang Yong Liu Dacheng Tao ViT 30 32 0 19 Jun 2022
EfficientFormer: Vision Transformers at MobileNet Speed Yanyu Li Geng Yuan Yang Wen Eric Hu Georgios Evangelidis Sergey Tulyakov Yanzhi Wang Jian Ren ViT 16 346 0 02 Jun 2022
A Hardware-Aware Framework for Accelerating Neural Architecture Search Across Modalities Daniel Cummings Anthony Sarah S. N. Sridhar Maciej Szankin J. P. Muñoz Sairam Sundaresan 28 8 0 19 May 2022
PVNAS: 3D Neural Architecture Search with Point-Voxel Convolution Zhijian Liu Haotian Tang Shengyu Zhao Kevin Shao Song Han 3DPC 13 39 0 25 Apr 2022
SplitNets: Designing Neural Architectures for Efficient Distributed Computing on Head-Mounted Systems Xin Dong B. D. Salvo Meng Li Chiao Liu Zhongnan Qu H. T. Kung Ziyun Li 3DGS 21 20 0 10 Apr 2022
Accelerating Neural Architecture Exploration Across Modalities Using Genetic Algorithms Daniel Cummings S. N. Sridhar Anthony Sarah Maciej Szankin AI4CE 25 0 0 25 Feb 2022
EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation Tao Ge Si-Qing Chen Furu Wei MoE 22 21 0 16 Feb 2022
Fast Monte-Carlo Approximation of the Attention Mechanism Hyunjun Kim Jeonggil Ko 17 2 0 30 Jan 2022
Representing Long-Range Context for Graph Neural Networks with Global Attention Zhanghao Wu Paras Jain Matthew A. Wright Azalia Mirhoseini Joseph E. Gonzalez Ion Stoica GNN 35 258 0 21 Jan 2022
Searching for TrioNet: Combining Convolution with Local and Global Self-Attention Huaijin Pi Huiyu Wang Yingwei Li Zizhang Li Alan Yuille ViT 19 3 0 15 Nov 2021
The NiuTrans System for the WMT21 Efficiency Task Chenglong Wang Chi Hu Yongyu Mu Zhongxiang Yan Siming Wu ... Hang Cao Bei Li Ye Lin Tong Xiao Jingbo Zhu 19 2 0 16 Sep 2021
EfficientBERT: Progressively Searching Multilayer Perceptron via Warm-up Knowledge Distillation Chenhe Dong Guangrun Wang Hang Xu Jiefeng Peng Xiaozhe Ren Xiaodan Liang 16 28 0 15 Sep 2021
Group Fisher Pruning for Practical Network Compression Liyang Liu Shilong Zhang Zhanghui Kuang Aojun Zhou Jingliang Xue Xinjiang Wang Yimin Chen Wenming Yang Q. Liao Wayne Zhang 25 146 0 02 Aug 2021
QuantumNAS: Noise-Adaptive Search for Robust Quantum Circuits Hanrui Wang Yongshan Ding Jiaqi Gu Zirui Li Yujun Lin D. Pan Frederic T. Chong Song Han 14 170 0 22 Jul 2021
AutoFormer: Searching Transformers for Visual Recognition Minghao Chen Houwen Peng Jianlong Fu Haibin Ling ViT 36 259 0 01 Jul 2021
HELP: Hardware-Adaptive Efficient Latency Prediction for NAS via Meta-Learning Hayeon Lee Sewoong Lee Song Chong S. Hwang 13 26 0 16 Jun 2021
FEAR: A Simple Lightweight Method to Rank Architectures Debadeepta Dey Shital C. Shah Sébastien Bubeck OOD 22 4 0 07 Jun 2021
Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model Jiangning Zhang Chao Xu Jian Li Wenzhou Chen Yabiao Wang Ying Tai Shuo Chen Chengjie Wang Feiyue Huang Yong Liu 27 22 0 31 May 2021
Memory-Efficient Differentiable Transformer Architecture Search Yuekai Zhao Li Dong Yelong Shen Zhihua Zhang Furu Wei Weizhu Chen ViT 24 17 0 31 May 2021
Dynamic-OFA: Runtime DNN Architecture Switching for Performance Scaling on Heterogeneous Embedded Platforms W. Lou Lei Xun Amin Sabet Jia Bi Jonathon S. Hare G. Merrett AI4CE 20 28 0 08 May 2021
Translational NLP: A New Paradigm and General Principles for Natural Language Processing Research Denis R. Newman-Griffis J. Lehman Carolyn Rose H. Hochheiser 24 20 0 16 Apr 2021
Enabling Design Methodologies and Future Trends for Edge AI: Specialization and Co-design Cong Hao Jordan Dotzel Jinjun Xiong Luca Benini Zhiru Zhang Deming Chen 50 34 0 25 Mar 2021
Scalable Vision Transformers with Hierarchical Pooling Zizheng Pan Bohan Zhuang Jing Liu Haoyu He Jianfei Cai ViT 25 126 0 19 Mar 2021
AlphaNet: Improved Training of Supernets with Alpha-Divergence Dilin Wang Chengyue Gong Meng Li Qiang Liu Vikas Chandra 147 44 0 16 Feb 2021
Dancing along Battery: Enabling Transformer with Run-time Reconfigurability on Mobile Devices Yuhong Song Weiwen Jiang Bingbing Li Panjie Qi Qingfeng Zhuge E. Sha Sakyasingha Dasgupta Yiyu Shi Caiwen Ding 15 18 0 12 Feb 2021
A Comprehensive Survey on Hardware-Aware Neural Architecture Search Hadjer Benmeziane K. E. Maghraoui Hamza Ouarnoughi Smail Niar Martin Wistuba Naigang Wang 26 95 0 22 Jan 2021
Transformers in Vision: A Survey Salman Khan Muzammal Naseer Munawar Hayat Syed Waqas Zamir F. Khan M. Shah ViT 227 2,428 0 04 Jan 2021