What Ingredients Make for an Effective Crowdsourcing Protocol for
Difficult NLU Data Collection Tasks?

What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks?

1 June 2021

Papers citing "What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks?"

14 / 14 papers shown

Title
Leveraging Human Feedback to Scale Educational Datasets: Combining Crowdworkers and Comparative Judgement Owen Henkel Libby Hills 11 1 0 22 May 2023
Cost-efficient Crowdsourcing for Span-based Sequence Labeling: Worker Selection and Data Augmentation Yujie Wang Chaorui Huang Liner Yang Zhixuan Fang Yaping Huang Yang Liu Erhong Yang 31 0 0 11 May 2023
CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation Abhilasha Ravichander Matt Gardner Ana Marasović 33 34 0 01 Nov 2022
Possible Stories: Evaluating Situated Commonsense Reasoning under Multiple Possible Scenarios Mana Ashida Saku Sugawara 65 6 0 16 Sep 2022
"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Dataset Eric Michael Smith Melissa Hall Melanie Kambadur Eleonora Presani Adina Williams 79 130 0 18 May 2022
Don't Blame the Annotator: Bias Already Starts in the Annotation Instructions Mihir Parmar Swaroop Mishra Mor Geva Chitta Baral 36 55 0 01 May 2022
What Makes Reading Comprehension Questions Difficult? Saku Sugawara Nikita Nangia Alex Warstadt Sam Bowman ELM RALM 20 13 0 12 Mar 2022
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation Alisa Liu Swabha Swayamdipta Noah A. Smith Yejin Choi 70 211 0 16 Jan 2022
QuALITY: Question Answering with Long Input Texts, Yes! Richard Yuanzhe Pang Alicia Parrish Nitish Joshi Nikita Nangia Jason Phang ... Vishakh Padmakumar Johnny Ma Jana Thompson He He Sam Bowman RALM 30 141 0 16 Dec 2021
SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets Ann Yuan Daphne Ippolito Vitaly Nikolaev Chris Callison-Burch Andy Coenen Sebastian Gehrmann SyDa 112 20 0 11 Nov 2021
CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge Yasumasa Onoe Michael J.Q. Zhang Eunsol Choi Greg Durrett HILM 40 85 0 03 Sep 2021
ANLIzing the Adversarial Natural Language Inference Dataset Adina Williams Tristan Thrush Douwe Kiela AAML 179 46 0 24 Oct 2020
Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets Mor Geva Yoav Goldberg Jonathan Berant 242 320 0 21 Aug 2019
Hypothesis Only Baselines in Natural Language Inference Adam Poliak Jason Naradowsky Aparajita Haldar Rachel Rudinger Benjamin Van Durme 190 576 0 02 May 2018