Can Vision Transformers Learn without Natural Images?

Can Vision Transformers Learn without Natural Images?

24 March 2021

Kodai Nakashima

Hirokatsu Kataoka

Asato Matsumoto

Papers citing "Can Vision Transformers Learn without Natural Images?"

12 / 12 papers shown

Title
Formula-Supervised Sound Event Detection: Pre-Training Without Real Data Yuto Shibata Keitaro Tanaka Yoshiaki Bando Keisuke Imoto Hirokatsu Kataoka Yoshimitsu Aoki 26 0 0 06 Apr 2025
$FractalAD: A simple industrial anomaly detection method using fractal anomaly generation and backbone knowledge distillation$ FractalAD: A simple industrial anomaly detection method using fractal anomaly generation and backbone knowledge distillation X. Xia Weijie Lv Xing He Nan Li Chuanqi Liu Ning Ding 27 1 0 30 Jan 2023
EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithm Jiangning Zhang Xiangtai Li Yabiao Wang Chengjie Wang Yibo Yang Yong Liu Dacheng Tao ViT 30 32 0 19 Jun 2022
PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures Dan Hendrycks Andy Zou Mantas Mazeika Leonard Tang Bo-wen Li D. Song Jacob Steinhardt UQCV 23 136 0 09 Dec 2021
Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model Jiangning Zhang Chao Xu Jian Li Wenzhou Chen Yabiao Wang Ying Tai Shuo Chen Chengjie Wang Feiyue Huang Yong Liu 27 22 0 31 May 2021
Post-training deep neural network pruning via layer-wise calibration Ivan Lazarevich Alexander Kozlov Nikita Malinin 3DPC 8 25 0 30 Apr 2021
Pre-training without Natural Images Hirokatsu Kataoka Kazushige Okayasu Asato Matsumoto Eisuke Yamagata Ryosuke Yamada Nakamasa Inoue Akio Nakamura Y. Satoh 79 116 0 21 Jan 2021
Meta Pseudo Labels Hieu H. Pham Zihang Dai Qizhe Xie Minh-Thang Luong Quoc V. Le VLM 250 656 0 23 Mar 2020
Improved Baselines with Momentum Contrastive Learning Xinlei Chen Haoqi Fan Ross B. Girshick Kaiming He SSL 264 3,369 0 09 Mar 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 226 4,460 0 23 Jan 2020
Boosting Self-Supervised Learning via Knowledge Transfer M. Noroozi Ananth Vinjimoor Paolo Favaro Hamed Pirsiavash SSL 207 292 0 01 May 2018
Aggregated Residual Transformations for Deep Neural Networks Saining Xie Ross B. Girshick Piotr Dollár Z. Tu Kaiming He 294 10,216 0 16 Nov 2016