Language Models are Surprisingly Fragile to Drug Names in Biomedical
Benchmarks

Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks

17 June 2024

Leo Anthony Celi

Hugo J. W. L. Aerts

Thomas Hartvigsen

Danielle S. Bitterman

Papers citing "Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks"

6 / 6 papers shown

Title
Do "New Snow Tablets" Contain Snow? Large Language Models Over-Rely on Names to Identify Ingredients of Chinese Drugs Sifan Li Yujun Cai Bryan Hooi Nanyun Peng Y. Wang 19 0 0 03 Apr 2025
Paloma: A Benchmark for Evaluating Language Model Fit Ian H. Magnusson Akshita Bhagia Valentin Hofmann Luca Soldaini A. Jha ... Iz Beltagy Hanna Hajishirzi Noah A. Smith Kyle Richardson Jesse Dodge 126 21 0 16 Dec 2023
Don't Make Your LLM an Evaluation Benchmark Cheater Kun Zhou Yutao Zhu Zhipeng Chen Wentong Chen Wayne Xin Zhao Xu Chen Yankai Lin Ji-Rong Wen Jiawei Han ELM 102 136 0 03 Nov 2023
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 236 1,508 0 31 Dec 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 220 3,054 0 23 Jan 2020
PubMedQA: A Dataset for Biomedical Research Question Answering Qiao Jin Bhuwan Dhingra Zhengping Liu William W. Cohen Xinghua Lu 196 791 0 13 Sep 2019