To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation

22 July 2021

Tom Kocmi

C. Federmann

Roman Grundkiewicz

Marcin Junczys-Dowmunt

Hitokazu Matsushita

Arul Menezes

ArXiv PDF HTML

Papers citing "To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation"

27 / 127 papers shown

Title
Adam Mickiewicz University at WMT 2022: NER-Assisted and Quality-Aware Neural Machine Translation Artur Nowakowski Gabriela Pałka Kamil Guttmann Miko Pokrywka 17 5 0 07 Sep 2022
Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translation Nuno M. Guerreiro Elena Voita André F. T. Martins HILM 16 51 0 10 Aug 2022
SMART: Sentences as Basic Units for Text Evaluation Reinald Kim Amplayo Peter J. Liu Yao-Min Zhao Shashi Narayan 22 21 0 01 Aug 2022
Benchmarking Azerbaijani Neural Machine Translation Chih-Chen Chen William Chen 13 0 0 29 Jul 2022
GODEL: Large-Scale Pre-Training for Goal-Directed Dialog Baolin Peng Michel Galley Pengcheng He Chris Brockett Lars Liden E. Nouri Zhou Yu Bill Dolan Jianfeng Gao VLM 39 73 0 22 Jun 2022
When does Parameter-Efficient Transfer Learning Work for Machine Translation? A. Ustun Asa Cooper Stickland 27 7 0 23 May 2022
PreQuEL: Quality Estimation of Machine Translation Outputs in Advance Shachar Don-Yehiya Leshem Choshen Omri Abend 17 10 0 18 May 2022
Consistent Human Evaluation of Machine Translation across Language Pairs Daniel Licht Cynthia Gao Janice Lam Francisco Guzman Mona T. Diab Philipp Koehn 32 17 0 17 May 2022
The Devil is in the Details: On the Pitfalls of Vocabulary Selection in Neural Machine Translation Tobias Domhan Eva Hasler Ke M. Tran Sony Trenous Bill Byrne F. Hieber 11 5 0 13 May 2022
Quantifying Synthesis and Fusion and their Impact on Machine Translation Arturo Oncevay Duygu Ataman N. V. Berkel Barry Haddow Alexandra Birch Johannes Bjerva 12 3 0 06 May 2022
Original or Translated? A Causal Analysis of the Impact of Translationese on Machine Translation Performance Jingwei Ni Zhijing Jin Markus Freitag Mrinmaya Sachan Bernhard Schölkopf 22 14 0 04 May 2022
Non-Autoregressive Machine Translation: It's Not as Fast as it Seems Jindvrich Helcl Barry Haddow Alexandra Birch 14 19 0 04 May 2022
Quality-Aware Decoding for Neural Machine Translation Patrick Fernandes António Farinhas Ricardo Rei José G. C. de Souza Perez Ogayo Graham Neubig André F. T. Martins 31 57 0 02 May 2022
UniTE: Unified Translation Evaluation Yu Wan Dayiheng Liu Baosong Yang Haibo Zhang Boxing Chen Derek F. Wong Lidia S. Chao 24 41 0 28 Apr 2022
Spurious Correlations in Reference-Free Evaluation of Text Generation Esin Durmus Faisal Ladhak Tatsunori Hashimoto 12 30 0 21 Apr 2022
Disentangling Uncertainty in Machine Translation Evaluation Chrysoula Zerva T. Glushkova Ricardo Rei André F.T. Martins UD UQCV 28 9 0 13 Apr 2022
Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation Jannis Bulian Christian Buck Wojciech Gajewski Benjamin Boerschinger Tal Schuster 22 43 0 15 Feb 2022
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text Sebastian Gehrmann Elizabeth Clark Thibault Sellam ELM AI4CE 58 181 0 14 Feb 2022
Data Scaling Laws in NMT: The Effect of Noise and Architecture Yamini Bansal Behrooz Ghorbani Ankush Garg Biao Zhang M. Krikun Colin Cherry Behnam Neyshabur Orhan Firat 29 47 0 04 Feb 2022
DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence Wei-Ye Zhao Michael Strube Steffen Eger 19 37 0 26 Jan 2022
TYPIC: A Corpus of Template-Based Diagnostic Comments on Argumentation Shoichi Naito Shintaro Sawada C. Nakagawa Naoya Inoue Kenshi Yamaguchi Iori Shimizu Farjana Sultana Mim Keshav Singh Kentaro Inui 17 3 0 18 Jan 2022
BEAMetrics: A Benchmark for Language Generation Evaluation Evaluation Thomas Scialom Felix Hill 20 7 0 18 Oct 2021
Alternative Input Signals Ease Transfer in Multilingual Machine Translation Simeng Sun Angela Fan James Cross Vishrav Chaudhary C. Tran Philipp Koehn Francisco Guzman 22 16 0 15 Oct 2021
Multilingual Document-Level Translation Enables Zero-Shot Transfer From Sentences to Documents Biao Zhang Ankur Bapna Melvin Johnson A. Dabirmoghaddam N. Arivazhagan Orhan Firat 26 12 0 21 Sep 2021
Don't Go Far Off: An Empirical Study on Neural Poetry Translation Tuhin Chakrabarty Arkadiy Saakyan Smaranda Muresan 12 16 0 07 Sep 2021
Survey of Low-Resource Machine Translation Barry Haddow Rachel Bawden Antonio Valerio Miceli Barone Jindvrich Helcl Alexandra Birch AIMat 27 147 0 01 Sep 2021
Digging Errors in NMT: Evaluating and Understanding Model Errors from Partial Hypothesis Space Jianhao Yan Chenming Wu Fandong Meng Jie Zhou ELM LRM 8 2 0 29 Jun 2021