Please use this identifier to cite or link to this item: https://hdl.handle.net/10419/313157 
Authors: 
Year of Publication: 
2023
Citation: 
[Journal:] AStA Wirtschafts- und Sozialstatistisches Archiv [ISSN:] 1863-8163 [Volume:] 17 [Issue:] 3 [Publisher:] Springer [Place:] Berlin, Heidelberg [Year:] 2023 [Pages:] 331-353
Publisher: 
Springer, Berlin, Heidelberg
Abstract: 
The quality of Machine Learning (ML) applications is commonly assessed by quantifying how well an algorithm fits its respective training data. Yet, a perfect model that learns from and reproduces erroneous data will always be flawed in its real-world application. Hence, a comprehensive assessment of ML quality must include an additional data perspective, especially for models trained on human-annotated data. For the collection of human-annotated training data, best practices often do not exist and leave researchers to make arbitrary decisions when collecting annotations. Decisions about the selection of annotators or label options may affect training data quality and model performance. In this paper, I will outline and summarize previous research and approaches to the collection of annotated training data. I look at data annotation and its quality confounders from two perspectives: the set of annotators and the strategy of data collection. The paper will highlight the various implementations of text and image annotation collection and stress the importance of careful task construction. I conclude by illustrating the consequences for future research and applications of data annotation. The paper is intended give readers a starting point on annotated data quality research and stress the necessity of thoughtful consideration of the annotation collection process to researchers and practitioners.
Subjects: 
Data quality
Data annotation
Training data
Human annotation
Research synthesis
Persistent Identifier of the first edition: 
Creative Commons License: 
cc-by Logo
Document Type: 
Article
Document Version: 
Published Version

Files in This Item:
File
Size





Items in EconStor are protected by copyright, with all rights reserved, unless otherwise indicated.