What Does BERT Look At? An Analysis of BERT's Attention

11 June 2019

Kevin Clark

Urvashi Khandelwal

Omer Levy

Christopher D. Manning

MILM

ArXiv PDF HTML

Papers citing "What Does BERT Look At? An Analysis of BERT's Attention"

50 / 885 papers shown

Title
Towards Faithful Model Explanation in NLP: A Survey Qing Lyu Marianna Apidianaki Chris Callison-Burch XAI 109 107 0 22 Sep 2022
SPACE-3: Unified Dialog Model Pre-training for Task-Oriented Dialog Understanding and Generation Wanwei He Yinpei Dai Min Yang Jian Sun Fei Huang Luo Si Yongbin Li 22 60 0 14 Sep 2022
Pre-Training a Graph Recurrent Network for Language Representation Yile Wang Linyi Yang Zhiyang Teng M. Zhou Yue Zhang GNN 35 1 0 08 Sep 2022
Does Attention Mechanism Possess the Feature of Human Reading? A Perspective of Sentiment Classification Task Leilei Zhao Yingyi Zhang Chengzhi Zhang 27 2 0 08 Sep 2022
On the Effectiveness of Compact Biomedical Transformers Omid Rohanian Mohammadmahdi Nouriborji Samaneh Kouchaki David A. Clifton MedIm 18 31 0 07 Sep 2022
Analyzing Transformers in Embedding Space Guy Dar Mor Geva Ankit Gupta Jonathan Berant 19 83 0 06 Sep 2022
Mind the Gap! Injecting Commonsense Knowledge for Abstractive Dialogue Summarization Seungone Kim Se June Joo Hyungjoo Chae Chaehyeong Kim Seung-won Hwang Jinyoung Yeo 21 20 0 02 Sep 2022
Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignment Mustafa Shukor Guillaume Couairon Matthieu Cord VLM CLIP 19 27 0 29 Aug 2022
Survey: Exploiting Data Redundancy for Optimization of Deep Learning Jou-An Chen Wei Niu Bin Ren Yanzhi Wang Xipeng Shen 23 24 0 29 Aug 2022
Interpreting Embedding Spaces by Conceptualization Adi Simhi Shaul Markovitch 24 5 0 22 Aug 2022
A Syntax Aware BERT for Identifying Well-Formed Queries in a Curriculum Framework Avinash Madasu Anvesh Rao Vijjini 22 0 0 21 Aug 2022
Lost in Context? On the Sense-wise Variance of Contextualized Word Embeddings Yile Wang Yue Zhang 19 4 0 20 Aug 2022
Pretrained Language Encoders are Natural Tagging Frameworks for Aspect Sentiment Triplet Extraction Yanjie Gou Yinjie Lei Lingqiao Liu Yong Dai Chun-Yen Shen Yongqi Tong ViT 21 0 0 20 Aug 2022
Disentangled Modeling of Domain and Relevance for Adaptable Dense Retrieval Jingtao Zhan Qingyao Ai Yiqun Liu Jiaxin Mao Xiaohui Xie M. Zhang Shaoping Ma 30 10 0 11 Aug 2022
Attention Hijacking in Trojan Transformers Weimin Lyu Songzhu Zheng Teng Ma Haibin Ling Chao Chen 27 6 0 09 Aug 2022
Learning to Learn to Predict Performance Regressions in Production at Meta M. Beller Hongyu Li V. Nair V. Murali Imad Ahmad Jürgen Cito Drew Carlson Gareth Ari Aye Wes Dyer 31 5 0 08 Aug 2022
Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks Tilman Raukur A. Ho Stephen Casper Dylan Hadfield-Menell AAML AI4CE 23 124 0 27 Jul 2022
Fine-Tuning BERT for Automatic ADME Semantic Labeling in FDA Drug Labeling to Enhance Product-Specific Guidance Assessment Yiwen Shi Jing Wang Ping Ren Taha ValizadehAslani Yi Zhang Meng Hu Hualou Liang AI4MH AAML 22 16 0 25 Jul 2022
What does Transformer learn about source code? Kechi Zhang Ge Li Zhi Jin ViT 20 8 0 18 Jul 2022
STI: Turbocharge NLP Inference at the Edge via Elastic Pipelining Liwei Guo Wonkyo Choe F. Lin 19 14 0 11 Jul 2022
Betti numbers of attention graphs is all you really need Laida Kushnareva D. Piontkovski Irina Piontkovskaya GNN 17 2 0 05 Jul 2022
Probing via Prompting Jiaoda Li Ryan Cotterell Mrinmaya Sachan 29 13 0 04 Jul 2022
Linguistically inspired roadmap for building biologically reliable protein language models Mai Ha Vu Rahmad Akbar Philippe A. Robert B. Swiatczak Victor Greiff G. K. Sandve Dag Trygve Tryslew Haug 39 35 0 03 Jul 2022
The Topological BERT: Transforming Attention into Topology for Natural Language Processing Ilan Perez Raphael Reinauer 22 17 0 30 Jun 2022
Analyzing Encoded Concepts in Transformer Language Models Hassan Sajjad Nadir Durrani Fahim Dalvi Firoj Alam A. Khan Jia Xu 11 40 0 27 Jun 2022
Towards Unsupervised Content Disentanglement in Sentence Representations via Syntactic Roles G. Felhi Joseph Le Roux Djamé Seddah DRL 16 5 0 22 Jun 2022
Vicinity Vision Transformer Weixuan Sun Zhen Qin Huiyuan Deng Jianyuan Wang Yi Zhang Kaihao Zhang Nick Barnes Stan Birchfield Lingpeng Kong Yiran Zhong ViT 34 31 0 21 Jun 2022
Methods for Estimating and Improving Robustness of Language Models Michal Stefánik 6 1 0 16 Jun 2022
SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features Juri Opitz Anette Frank 26 33 0 14 Jun 2022
Unsupervised and Few-shot Parsing from Pretrained Language Models Zhiyuan Zeng Deyi Xiong 13 4 0 10 Jun 2022
Learning to Estimate Shapley Values with Vision Transformers Ian Covert Chanwoo Kim Su-In Lee FAtt 25 34 0 10 Jun 2022
STNDT: Modeling Neural Population Activity with a Spatiotemporal Transformer Trung Le Eli Shlizerman 25 22 0 09 Jun 2022
Unveiling Transformers with LEGO: a synthetic reasoning task Yi Zhang A. Backurs Sébastien Bubeck Ronen Eldan Suriya Gunasekar Tal Wagner LRM 28 85 0 09 Jun 2022
Challenges in Applying Explainability Methods to Improve the Fairness of NLP Models Esma Balkir S. Kiritchenko I. Nejadgholi Kathleen C. Fraser 21 36 0 08 Jun 2022
Dict-TTS: Learning to Pronounce with Prior Dictionary Knowledge for Text-to-Speech Ziyue Jiang Zhe Su Zhou Zhao Qian Yang Yi Ren Jinglin Liu Zhe Ye 24 4 0 05 Jun 2022
Transformer with Fourier Integral Attentions T. Nguyen Minh Pham Tam Nguyen Khai Nguyen Stanley J. Osher Nhat Ho 17 4 0 01 Jun 2022
Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation Verna Dankers Christopher G. Lucas Ivan Titov 35 36 0 30 May 2022
CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behavior Eldar David Abraham Karel DÓosterlinck Amir Feder Y. Gat Atticus Geiger Christopher Potts Roi Reichart Zhengxuan Wu CML 28 43 0 27 May 2022
What Dense Graph Do You Need for Self-Attention? Yuxing Wang Chu-Tak Lee Qipeng Guo Zhangyue Yin Yunhua Zhou Xuanjing Huang Xipeng Qiu GNN 6 4 0 27 May 2022
Revisiting Generative Commonsense Reasoning: A Pre-Ordering Approach Chao Zhao Faeze Brahman Tenghao Huang Snigdha Chaturvedi LRM 19 3 0 26 May 2022
Leveraging Dependency Grammar for Fine-Grained Offensive Language Detection using Graph Convolutional Networks Divyam Goel Raksha Sharma GNN 11 4 0 26 May 2022
Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformer Javier Ferrando Gerard I. Gállego Belen Alastruey Carlos Escolano Marta R. Costa-jussá 22 44 0 23 May 2022
Outliers Dimensions that Disrupt Transformers Are Driven by Frequency Giovanni Puccetti Anna Rogers Aleksandr Drozd F. Dell’Orletta 71 42 0 23 May 2022
What Drives the Use of Metaphorical Language? Negative Insights from Abstractness, Affect, Discourse Coherence and Contextualized Word Representations P. Piccirilli Sabine Schulte im Walde 22 4 0 23 May 2022
A Graph Enhanced BERT Model for Event Prediction LI DU Xiao Ding Yue Zhang Kai Xiong Ting Liu Bing Qin 30 10 0 22 May 2022
Life after BERT: What do Other Muppets Understand about Language? Vladislav Lialin Kevin Zhao Namrata Shivagunde Anna Rumshisky 39 6 0 21 May 2022
Exploring Extreme Parameter Compression for Pre-trained Language Models Yuxin Ren Benyou Wang Lifeng Shang Xin Jiang Qun Liu 28 18 0 20 May 2022
Automated Scoring for Reading Comprehension via In-context BERT Tuning Nigel Fernandez Aritra Ghosh Naiming Liu Zichao Wang Benoît Choffin Richard Baraniuk Andrew S. Lan 12 20 0 19 May 2022
Acceptability Judgements via Examining the Topology of Attention Maps D. Cherniavskii Eduard Tulchinskii Vladislav Mikhailov Irina Proskurina Laida Kushnareva Ekaterina Artemova S. Barannikov Irina Piontkovskaya D. Piontkovski Evgeny Burnaev 770 19 0 19 May 2022
Assessing the Limits of the Distributional Hypothesis in Semantic Spaces: Trait-based Relational Knowledge and the Impact of Co-occurrences Mark Anderson Jose Camacho-Collados 32 0 0 16 May 2022