BERTology Meets Biology: Interpreting Attention in Protein Language Models

26 June 2020

Papers citing "BERTology Meets Biology: Interpreting Attention in Protein Language Models"

23 / 73 papers shown

Title
ReLSO: A Transformer-based Model for Latent Space Optimization and Generation of Proteins Egbert Castro Abhinav Godavarthi Julian Rubinfien Kevin B. Givechian Dhananjay Bhaskar Smita Krishnaswamy 38 5 0 24 Jan 2022
OntoProtein: Protein Pretraining With Gene Ontology Embedding Ningyu Zhang Zhen Bi Xiaozhuan Liang Shuyang Cheng Haosen Hong Shumin Deng J. Lian Qiang Zhang Huajun Chen 99 90 0 23 Jan 2022
An Effective GCN-based Hierarchical Multi-label classification for Protein Function Prediction Kyudam Choi Yurim Lee Cheongwon Kim MinSung Yoon 14 7 0 06 Dec 2021
Pre-training Co-evolutionary Protein Representation via A Pairwise Masked Language Model Liang He Shizhuo Zhang Lijun Wu Huanhuan Xia Fusong Ju ... Jianwei Zhu Pan Deng Bin Shao Tao Qin Tie-Yan Liu 26 31 0 29 Oct 2021
Pre-trained Language Models in Biomedical Domain: A Systematic Survey Benyou Wang Qianqian Xie Jiahuan Pei Zhihong Chen Prayag Tiwari Zhao Li Jie Fu LM&MA AI4CE 37 163 0 11 Oct 2021
Automated and Explainable Ontology Extension Based on Deep Learning: A Case Study in the Chemical Domain A. Memariani Martin Glauer Fabian Neuhaus Till Mossakowski Janna Hastings 20 5 0 19 Sep 2021
Cross-lingual Transfer of Monolingual Models Evangelia Gogoulou Ariel Ekgren T. Isbister Magnus Sahlgren 27 17 0 15 Sep 2021
Towards Interpreting Zoonotic Potential of Betacoronavirus Sequences With Attention Kahini Wadhawan Payel Das Barbara A. Han Ilya R. Fischhoff Adrian C. Castellanos A. Varsani Kush R. Varshney 17 4 0 18 Aug 2021
VitaLITy: Promoting Serendipitous Discovery of Academic Literature with Transformers & Visual Analytics Arpit Narechania Alireza Karduni Ryan Wesslen Emily Wall 43 25 0 07 Aug 2021
Large-Scale Chemical Language Representations Capture Molecular Structure and Properties Jerret Ross Brian M. Belgodere Vijil Chenthamarakshan Inkit Padhi Youssef Mroueh Payel Das AI4CE 21 272 0 17 Jun 2021
Transformers with Competitive Ensembles of Independent Mechanisms Alex Lamb Di He Anirudh Goyal Guolin Ke Chien-Feng Liao Mirco Ravanelli Yoshua Bengio MoE 16 23 0 27 Feb 2021
Generative Capacity of Probabilistic Protein Sequence Models Francisco McGee Quentin Novinger R. Levy Vincenzo Carnevale A. Haldane 22 34 0 03 Dec 2020
What is a meaningful representation of protein sequences? N. Detlefsen Søren Hauberg Wouter Boomsma 8 111 0 28 Nov 2020
Is Transfer Learning Necessary for Protein Landscape Prediction? Amir Shanehsazzadeh David Belanger David Dohan SSL 12 61 0 31 Oct 2020
Language Models are Open Knowledge Graphs Chenguang Wang Xiao Liu D. Song SSL KELM 24 135 0 22 Oct 2020
Fixed-Length Protein Embeddings using Contextual Lenses Amir Shanehsazzadeh David Belanger David Dohan 8 1 0 15 Oct 2020
Rethinking Attention with Performers K. Choromanski Valerii Likhosherstov David Dohan Xingyou Song Andreea Gane ... Afroz Mohiuddin Lukasz Kaiser David Belanger Lucy J. Colwell Adrian Weller 8 1,517 0 30 Sep 2020
Navigating Human Language Models with Synthetic Agents Philip G. Feldman Antonio Bucchiarone 15 4 0 10 Aug 2020
Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers K. Choromanski Valerii Likhosherstov David Dohan Xingyou Song Andreea Gane ... Peter Hawkins Jared Davis David Belanger Lucy J. Colwell Adrian Weller 28 84 0 05 Jun 2020
Calibration of Pre-trained Transformers Shrey Desai Greg Durrett UQLM 243 289 0 17 Mar 2020
What you can cram into a single vector: Probing sentence embeddings for linguistic properties Alexis Conneau Germán Kruszewski Guillaume Lample Loïc Barrault Marco Baroni 201 882 0 03 May 2018
A causal framework for explaining the predictions of black-box sequence-to-sequence models David Alvarez-Melis Tommi Jaakkola CML 227 201 0 06 Jul 2017
Towards A Rigorous Science of Interpretable Machine Learning Finale Doshi-Velez Been Kim XAI FaML 251 3,683 0 28 Feb 2017