BERTology Meets Biology: Interpreting Attention in Protein Language Models

26 June 2020

Papers citing "BERTology Meets Biology: Interpreting Attention in Protein Language Models"

50 / 73 papers shown

Title
PLM-eXplain: Divide and Conquer the Protein Embedding Space Jan van Eck Dea Gogishvili Wilson Silva Sanne Abeln 23 0 0 09 Apr 2025
Interpretable Multimodal Learning for Tumor Protein-Metal Binding: Progress, Challenges, and Perspectives Xiaokun Liu Sayedmohammadreza Rastegari Yijun Huang Sxe Chang Cheong Weikang Liu ... Sina Tabakhi Xianyuan Liu Zheqing Zhu Wei Sang Haiping Lu 29 0 0 04 Apr 2025
Towards Interpretable Protein Structure Prediction with Sparse Autoencoders Nithin Parsan David J. Yang John J. Yang 53 0 0 11 Mar 2025
evoBPE: Evolutionary Protein Sequence Tokenization Burak Suyunu Özdeniz Dolu Arzucan Özgür 58 0 0 11 Mar 2025
Protein Large Language Models: A Comprehensive Survey Yijia Xiao Wanjia Zhao Junkai Zhang Yiqiao Jin Han Zhang ... Xiao Luo Yu-Jie Zhang James Y. Zou Y. Sun Wei Wang LM&MA AI4CE 54 3 0 21 Feb 2025
Transformers trained on proteins can learn to attend to Euclidean distance Isaac Ellmen Constantin Schneider Matthew I.J. Raybould Charlotte M. Deane 79 0 0 03 Feb 2025
Linguistic Laws Meet Protein Sequences: A Comparative Analysis of Subword Tokenization Methods Burak Suyunu Enes Taylan Arzucan Özgür 62 2 0 26 Nov 2024
Computational design of target-specific linear peptide binders with TransformerBeta Haowen Zhao Francesco A. Aprile Barbara Bravi 29 0 0 07 Oct 2024
FoodPuzzle: Developing Large Language Model Agents as Flavor Scientists Tenghao Huang Donghee Lee John Sweeney Jiatong Shi Emily Steliotes Matthew Lange Jonathan May Muhao Chen 39 2 0 19 Sep 2024
ProteinGPT: Multimodal LLM for Protein Property Prediction and Structure Understanding Yijia Xiao Edward Sun Yiqiao Jin Qifan Wang Wei Wang 47 9 0 21 Aug 2024
Towards Interpretable Visuo-Tactile Predictive Models for Soft Robot Interactions Enrico Donato T. G. Thuruthel Egidio Falotico 23 1 0 16 Jul 2024
MatText: Do Language Models Need More than Text & Scale for Materials Modeling? Nawaf Alampara Santiago Miret K. Jablonka 50 8 0 25 Jun 2024
A Critical Study of What Code-LLMs (Do Not) Learn Abhinav Anand Shweta Verma Krishna Narasimhan Mira Mezini 40 4 0 17 Jun 2024
Extracting Protein-Protein Interactions (PPIs) from Biomedical Literature using Attention-based Relational Context Information Gilchan Park S. McCorkle Carlos Soto Ian K. Blaby Shinjae Yoo 22 6 0 08 Mar 2024
ProtChatGPT: Towards Understanding Proteins with Large Language Models Chao Wang Hehe Fan Ruijie Quan Yi Yang 26 12 0 15 Feb 2024
A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules? Agustinus Kristiadi Felix Strieth-Kalthoff Marta Skreta Pascal Poupart Alán Aspuru-Guzik Geoff Pleiss 30 20 0 07 Feb 2024
Identification of Knowledge Neurons in Protein Language Models Divya Nori Shivali Singireddy M. T. Have MILM 13 2 0 17 Dec 2023
BEND: Benchmarking DNA Language Models on biologically meaningful tasks Frederikke Isa Marin Felix Teufel Marc Horlacher Dennis Madsen Dennis Pultz Ole Winther Wouter Boomsma 12 33 0 21 Nov 2023
Understanding the Natural Language of DNA using Encoder-Decoder Foundation Models with Byte-level Precision Aditya Malusare Harish Kothandaraman Dipesh Tamboli N. Lanman Vaneet Aggarwal AI4CE 27 2 0 04 Nov 2023
GPCR-BERT: Interpreting Sequential Design of G Protein Coupled Receptors Using Protein Language Models Seongwon Kim P. Mollaei Akshay Antony Rishikesh Magar A. Farimani 17 10 0 30 Oct 2023
A Hierarchical Training Paradigm for Antibody Structure-sequence Co-design Fang Wu Stan Z. Li 27 17 0 30 Oct 2023
A Meta-Learning Perspective on Transformers for Causal Language Modeling Xinbo Wu L. Varshney 29 6 0 09 Oct 2023
A Comprehensive Review of Generative AI in Healthcare Yasin Shokrollahi Sahar Yarmohammadtoosky Matthew M. Nikahd Pengfei Dong Xianqi Li Linxia Gu MedIm AI4CE 19 19 0 01 Oct 2023
AntiBARTy Diffusion for Property Guided Antibody Design Jordan Venderley DiffM 16 1 0 22 Sep 2023
Insights Into the Inner Workings of Transformer Models for Protein Function Prediction M. Wenzel Erik Grüner Nils Strodthoff ViT 19 7 0 07 Sep 2023
Comparative Performance Evaluation of Large Language Models for Extracting Molecular Interactions and Pathway Knowledge Gilchan Park Byung-Jun Yoon Xihaier Luo Vanessa López-Marrero Shinjae Yoo Francis J. Alexander 38 10 0 17 Jul 2023
Graph Denoising Diffusion for Inverse Protein Folding Kai Yi Bingxin Zhou Yiqing Shen Pietro Lio' Yu Guang Wang DiffM 11 41 0 29 Jun 2023
A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks Saidul Islam Hanae Elmekki Ahmed Elsebai Jamal Bentahar Najat Drawel Gaith Rjoub Witold Pedrycz ViT MedIm 19 171 0 11 Jun 2023
Multi-level Protein Representation Learning for Blind Mutational Effect Prediction Y. Tan Bingxin Zhou Yuanhong Jiang Yu Wang Liang Hong 21 2 0 08 Jun 2023
Enhancing the Protein Tertiary Structure Prediction by Multiple Sequence Alignment Generation Le Zhang Jiayang Chen Tao Shen Yu-Hu Li S. Sun 11 5 0 02 Jun 2023
PromptAid: Prompt Exploration, Perturbation, Testing and Iteration using Visual Analytics for Large Language Models Aditi Mishra Utkarsh Soni Anjana Arunkumar Jinbin Huang Bum Chul Kwon Chris Bryan LRM 28 33 0 04 Apr 2023
Protein Representation Learning via Knowledge Enhanced Primary Structure Modeling Hong-Yu Zhou Yunxiang Fu Zhicheng Zhang Cheng Bian Yizhou Yu 22 8 0 30 Jan 2023
Ontology Pre-training for Poison Prediction Martin Glauer Fabian Neuhaus Till Mossakowski Janna Hastings 13 4 0 20 Jan 2023
Ankh: Optimized Protein Language Model Unlocks General-Purpose Modelling Ahmed Elnaggar Hazem Essam Wafaa Salah-Eldin Walid Moustafa Mohamed Elkerdawy Charlotte Rochereau B. Rost 153 86 0 16 Jan 2023
Probabilistic thermal stability prediction through sparsity promoting transformer representation Yevgen Zainchkovskyy J. Ferkinghoff-Borg Anja Bennett T. Egebjerg Nikolai Lorenzen Per Greisen Søren Hauberg Carsten Stahlhut AI4CE 11 0 0 10 Nov 2022
Influence Functions for Sequence Tagging Models Sarthak Jain Varun Manjunatha Byron C. Wallace A. Nenkova TDI 30 8 0 25 Oct 2022
ImmunoLingo: Linguistics-based formalization of the antibody language Mai Ha Vu Philippe A. Robert Rahmad Akbar B. Swiatczak G. K. Sandve Dag Trygve Tryslew Haug Victor Greiff AI4CE 28 8 0 26 Sep 2022
Can Offline Reinforcement Learning Help Natural Language Understanding? Ziqi Zhang Yile Wang Yue Zhang Donglin Wang OffRL 20 0 0 15 Sep 2022
Linguistically inspired roadmap for building biologically reliable protein language models Mai Ha Vu Rahmad Akbar Philippe A. Robert B. Swiatczak Victor Greiff G. K. Sandve Dag Trygve Tryslew Haug 39 35 0 03 Jul 2022
Transformer Neural Networks Attending to Both Sequence and Structure for Protein Prediction Tasks Anowarul Kabir Amarda Shehu ViT 15 3 0 17 Jun 2022
Exploring evolution-aware & -free protein language models as protein function predictors Min Hu Fajie Yuan Kevin Kaichuang Yang Fusong Ju Jingyu Su Hongya Wang Fei Yang Qiuyang Ding 13 36 0 14 Jun 2022
Learning to Estimate Shapley Values with Vision Transformers Ian Covert Chanwoo Kim Su-In Lee FAtt 25 34 0 10 Jun 2022
Visualizing and Explaining Language Models Adrian M. P. Braşoveanu Razvan Andonie MILM VLM 29 4 0 30 Apr 2022
Generative power of a protein language model trained on multiple sequence alignments Damiano Sgarbossa Umberto Lupo Anne-Florence Bitbol 19 32 0 14 Apr 2022
Structure-aware Protein Self-supervised Learning Can Chen Jingbo Zhou Fan Wang Xue Liu Dejing Dou SSL 13 65 0 06 Apr 2022
Protein language models trained on multiple sequence alignments learn phylogenetic relationships Umberto Lupo Damiano Sgarbossa Anne-Florence Bitbol 24 34 0 29 Mar 2022
What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Code Yao Wan Wei-Ye Zhao Hongyu Zhang Yulei Sui Guandong Xu Hairong Jin 27 105 0 14 Feb 2022
Prompt-Guided Injection of Conformation to Pre-trained Protein Model Qiang Zhang Zeyuan Wang Yuqiang Han Haoran Yu Xurui Jin Huajun Chen 23 3 0 07 Feb 2022
Transformers and the representation of biomedical background knowledge Oskar Wysocki Zili Zhou Paul O'Regan D. Ferreira Magdalena Wysocka Dónal Landers Idiap Research Institute MedIm 13 14 0 04 Feb 2022
Regression Transformer: Concurrent sequence regression and generation for molecular language modeling Jannis Born Matteo Manica 13 91 0 01 Feb 2022