Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherence

5 July 2021

Alexander Miserlis Hoyle

Pranav Goel

Denis Peskov

Andrew Hian-Cheong

Jordan L. Boyd-Graber

Philip Resnik

ArXiv PDF HTML

Papers citing "Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherence"

50 / 51 papers shown

Title
From Precision to Perception: User-Centred Evaluation of Keyword Extraction Algorithms for Internet-Scale Contextual Advertising Jingwen Cai Sara Leckner Johanna Björklund 36 0 0 30 Apr 2025
Dynamic Embedded Topic Models: properties and recommendations based on diverse corpora Elisabeth Fittschen Bella Xia Leib Celnik Paul Dilley Tom Lippincott 44 0 0 27 Apr 2025
Evaluating Negative Sampling Approaches for Neural Topic Models Suman Adhya Avishek Lahiri Debarshi Kumar Sanyal Partha Pratim Das BDL 45 0 0 23 Mar 2025
Uncovering Latent Arguments in Social Media Messaging by Employing LLMs-in-the-Loop Strategy Tunazzina Islam Dan Goldwasser 82 3 0 28 Jan 2025
Enhancing Topic Interpretability for Neural Topic Modeling through Topic-wise Contrastive Learning Xin Gao Yang Lin Ruiqing Li Yasha Wang Xu Chu Xinyu Ma Hailong Yu 31 1 0 23 Dec 2024
Multi-environment Topic Models Dominic Sobhani Amir Feder David M. Blei 23 0 0 31 Oct 2024
Reliability of Topic Modeling Kayla Schroeder Zach Wood-Doughty 27 0 0 30 Oct 2024
Tethering Broken Themes: Aligning Neural Topic Models with Labels and Authors M. Nagda Phil Ostheimer Sophie Fellenz 34 0 0 22 Oct 2024
Embedded Topic Models Enhanced by Wikification Takashi Shibuya Takehito Utsuro AI4TS 15 0 0 03 Oct 2024
Qualitative Insights Tool (QualIT): LLM Enhanced Topic Modeling Satya Kapoor Alex Gil S. Bhaduri Anshul Mittal Rutu Mulkar 20 1 0 24 Sep 2024
Benchmarks as Microscopes: A Call for Model Metrology Michael Stephen Saxon Ari Holtzman Peter West William Yang Wang Naomi Saphra 31 10 0 22 Jul 2024
FASTopic: A Fast, Adaptive, Stable, and Transferable Topic Modeling Paradigm Xiaobao Wu Thong Nguyen Delvin Ce Zhang William Yang Wang A. Luu 37 4 0 28 May 2024
Human-interpretable clustering of short-text using large language models Justin K. Miller Tristram J. Alexander VLM 32 3 0 12 May 2024
Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooM Michelle S. Lam Janice Teoh James A. Landay Jeffrey Heer Michael S. Bernstein 21 40 0 18 Apr 2024
GINopic: Topic Modeling with Graph Isomorphism Network Suman Adhya Debarshi Kumar Sanyal 31 4 0 02 Apr 2024
Neural Multimodal Topic Modeling: A Comprehensive Evaluation Felipe González-Pizarro Giuseppe Carenini VGen 29 1 0 26 Mar 2024
Membership Inference Attacks and Privacy in Topic Modeling Nico Manzonelli Wanrong Zhang Salil P. Vadhan 34 1 0 07 Mar 2024
Probabilistic Topic Modelling with Transformer Representations Arik Reuter Anton Thielmann Christoph Weisser Benjamin Säfken Thomas Kneib 26 1 0 06 Mar 2024
The Geometric Structure of Topic Models Johannes Hirth Tom Hanika 26 0 0 06 Mar 2024
Topic Modeling as Multi-Objective Contrastive Optimization Thong Nguyen Xiaobao Wu Xinshuai Dong Cong-Duy Nguyen See-Kiong Ng A. Luu 35 6 0 12 Feb 2024
A Survey on Neural Topic Models: Methods, Applications, and Challenges Xiaobao Wu Thong Nguyen A. Luu BDL 18 33 0 27 Jan 2024
Prompting Large Language Models for Topic Modeling Han Wang Nirmalendu Prakash N. Hoang Ming Shan Hee Usman Naseem Roy Ka-Wei Lee 20 23 0 15 Dec 2023
PromptMTopic: Unsupervised Multimodal Topic Modeling of Memes using Large Language Models Nirmalendu Prakash Han Wang N. Hoang Ming Shan Hee Roy Ka-Wei Lee 21 11 0 11 Dec 2023
Revisiting Topic-Guided Language Models Carolina Zheng Keyon Vafa David M. Blei BDL 27 1 0 04 Dec 2023
Labeled Interactive Topic Models Kyle Seelman Mozhi Zhang Jordan L. Boyd-Graber 13 0 0 15 Nov 2023
Towards the TopMost: A Topic Modeling System Toolkit Xiaobao Wu Fengjun Pan A. Luu 23 14 0 13 Sep 2023
Evaluating Dynamic Topic Models Charu James M. Nagda Nooshin Haji Ghassemi Marius Kloft Sophie Fellenz 16 1 0 12 Sep 2023
Deep Dive into the Language of International Relations: NLP-based Analysis of UNESCO's Summary Records Joanna Wojciechowska Mateusz Sypniewski Maria Śmigielska Igor Kamiñski Emilia Wisnios Hanna Schreiber Bartosz Pieliñski 21 2 0 31 Jul 2023
vONTSS: vMF based semi-supervised neural topic modeling with optimal transport Weijie Xu Xiaoyu Jiang Srinivasan H. Sengamedu Francis Iannacci Jinjin Zhao OT BDL 21 17 0 03 Jul 2023
Contextualized Topic Coherence Metrics Hamed Rahimi Jacob Louis Hoover David M. Mimno Hubert Naacke Camélia Constantin B. Amann 27 16 0 23 May 2023
Revisiting Automated Topic Model Evaluation with Large Language Models Dominik Stammbach Vilém Zouhar Alexander Miserlis Hoyle Mrinmaya Sachan Elliott Ash ELM ALM 26 4 0 20 May 2023
Computational thematics: Comparing algorithms for clustering the genres of literary fiction Oleg Sobchuk A. Šeļa 17 7 0 18 May 2023
Interactive Concept Learning for Uncovering Latent Themes in Large Text Collections Maria Leonor Pacheco Tunazzina Islam Lyle Ungar Ming Yin Dan Goldwasser 40 12 0 08 May 2023
Bridging Nations: Quantifying the Role of Multilinguals in Communication on Social Media Julia Mendelsohn Sayan Ghosh David Jurgens Ceren Budak 12 1 0 07 Apr 2023
Topics in the Haystack: Extracting and Evaluating Topics beyond Coherence Anton Thielmann Q. Seifert Arik Reuter Elisabeth Bergherr Benjamin Säfken 17 2 0 30 Mar 2023
Do Neural Topic Models Really Need Dropout? Analysis of the Effect of Dropout in Topic Modeling Suman Adhya Avishek Lahiri Debarshi Kumar Sanyal 27 2 0 28 Mar 2023
Improving Contextualized Topic Models with Negative Sampling Suman Adhya Avishek Lahiri Debarshi Kumar Sanyal Partha Pratim Das 20 9 0 27 Mar 2023
Deep learning for COVID-19 topic modelling via Twitter: Alpha, Delta and Omicron Janhavi Lande Arti Pillay Rohitash Chandra 16 9 0 28 Feb 2023
A Two-Sided Discussion of Preregistration of NLP Research Anders Søgaard Daniel Hershcovich Miryam de Lhoneux OnRL AI4CE 20 3 0 20 Feb 2023
You Are What You Talk About: Inducing Evaluative Topics for Personality Analysis Josip Jukić Iva Vukojević Jan vSnajder 17 4 0 01 Feb 2023
Sensemaking About Contraceptive Methods Across Online Platforms LeAnn McDowall Maria Antoniak David M. Mimno 24 3 0 23 Jan 2023
Topics in Contextualised Attention Embeddings Mozhgan Talebpour A. G. S. D. Herrera Shoaib Jameel 21 2 0 11 Jan 2023
Effective Seed-Guided Topic Discovery by Integrating Multiple Types of Contexts Yu Zhang Yunyi Zhang Martin Michalski Yucheng Jiang Yu Meng Jiawei Han 54 16 0 12 Dec 2022
Moving beyond word lists: towards abstractive topic labels for human-like topics of scientific documents Domenic Rosati 21 0 0 28 Oct 2022
Are Neural Topic Models Broken? Alexander Miserlis Hoyle Pranav Goel Rupak Sarkar Philip Resnik 33 14 0 28 Oct 2022
No Pattern, No Recognition: a Survey about Reproducibility and Distortion Issues of Text Clustering and Topic Modeling Marília Costa Rosendo Silva Felipe Siqueira João Pedro M. Tarrega João Vitor P. Beinotti Augusto Sousa Nunes M. Gardini Vinicius Silva Nádia Félix Felipe da Silva A. C. P. L. F. D. Carvalho 23 2 0 02 Aug 2022
A Joint Learning Approach for Semi-supervised Neural Topic Modeling Jeffrey C. Chiu Rajat Mittal Neehal Tumma Abhishek Sharma Finale Doshi-Velez BDL DRL 13 4 0 07 Apr 2022
BERTopic: Neural topic modeling with a class-based TF-IDF procedure M. Grootendorst 39 1,372 0 11 Mar 2022
HTMOT : Hierarchical Topic Modelling Over Time Judicael Poumay A. Ittoo 14 3 0 22 Nov 2021
Topic Modeling with Contextualized Word Representation Clusters Laure Thompson David M. Mimno 102 82 0 23 Oct 2020