Iterative Quality Control Strategies for Expert Medical Image Labeling Article Swipe
Beverly Freeman
,
Naama Hammel
,
Sonia Phene
,
Abigail E. Huang
,
R. Ackermann
,
Olga Kanzheleva
,
Miles Hutson
,
Caitlin Taggart
,
Quang Duong
,
Rory Sayres
·
YOU?
·
· 2021
· Open Access
·
· DOI: https://doi.org/10.1609/hcomp.v9i1.18940
YOU?
·
· 2021
· Open Access
·
· DOI: https://doi.org/10.1609/hcomp.v9i1.18940
Data quality is a key concern for artificial intelligence (AI) efforts that rely on crowdsourced data collection. In the domain of medicine in particular, labeled data must meet high quality standards, or the resulting AI may perpetuate biases or lead to patient harm. What are the challenges involved in expert medical labeling? How do AI practitioners address such challenges? In this study, we interviewed members of teams developing AI for medical imaging in four subdomains (ophthalmology, radiology, pathology, and dermatology) about their quality-related practices. We describe one instance of low-quality labeling being caught by automated monitoring. The more proactive strategy, however, is to partner with experts in a collaborative, iterative process prior to the start of high-volume data collection. Best practices including 1) co-designing labeling tasks and instructional guidelines with experts, 2) piloting and revising the tasks and guidelines, and 3) onboarding workers enable teams to identify and address issues before they proliferate.
Related Topics
Concepts
Onboarding
Computer science
Quality (philosophy)
Process (computing)
Crowdsourcing
Data science
Control (management)
Best practice
Data collection
Knowledge management
Artificial intelligence
Psychology
World Wide Web
Operating system
Economics
Mathematics
Social psychology
Philosophy
Statistics
Epistemology
Management
Metadata
- Type
- article
- Language
- en
- Landing Page
- https://doi.org/10.1609/hcomp.v9i1.18940
- https://ojs.aaai.org/index.php/HCOMP/article/download/18940/18705
- OA Status
- diamond
- Cited By
- 14
- References
- 51
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W3211353562
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W3211353562Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.1609/hcomp.v9i1.18940Digital Object Identifier
- Title
-
Iterative Quality Control Strategies for Expert Medical Image LabelingWork title
- Type
-
articleOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2021Year of publication
- Publication date
-
2021-10-04Full publication date if available
- Authors
-
Beverly Freeman, Naama Hammel, Sonia Phene, Abigail E. Huang, R. Ackermann, Olga Kanzheleva, Miles Hutson, Caitlin Taggart, Quang Duong, Rory SayresList of authors in order
- Landing page
-
https://doi.org/10.1609/hcomp.v9i1.18940Publisher landing page
- PDF URL
-
https://ojs.aaai.org/index.php/HCOMP/article/download/18940/18705Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
diamondOpen access status per OpenAlex
- OA URL
-
https://ojs.aaai.org/index.php/HCOMP/article/download/18940/18705Direct OA link when available
- Concepts
-
Onboarding, Computer science, Quality (philosophy), Process (computing), Crowdsourcing, Data science, Control (management), Best practice, Data collection, Knowledge management, Artificial intelligence, Psychology, World Wide Web, Operating system, Economics, Mathematics, Social psychology, Philosophy, Statistics, Epistemology, ManagementTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
14Total citation count in OpenAlex
- Citations by year (recent)
-
2025: 2, 2024: 3, 2023: 8, 2022: 1Per-year citation counts (last 5 years)
- References (count)
-
51Number of works referenced by this work
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W3211353562 |
|---|---|
| doi | https://doi.org/10.1609/hcomp.v9i1.18940 |
| ids.doi | https://doi.org/10.1609/hcomp.v9i1.18940 |
| ids.mag | 3211353562 |
| ids.openalex | https://openalex.org/W3211353562 |
| fwci | 0.95993674 |
| type | article |
| title | Iterative Quality Control Strategies for Expert Medical Image Labeling |
| biblio.issue | |
| biblio.volume | 9 |
| biblio.last_page | 71 |
| biblio.first_page | 60 |
| topics[0].id | https://openalex.org/T11636 |
| topics[0].field.id | https://openalex.org/fields/27 |
| topics[0].field.display_name | Medicine |
| topics[0].score | 0.9980000257492065 |
| topics[0].domain.id | https://openalex.org/domains/4 |
| topics[0].domain.display_name | Health Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/2718 |
| topics[0].subfield.display_name | Health Informatics |
| topics[0].display_name | Artificial Intelligence in Healthcare and Education |
| topics[1].id | https://openalex.org/T13953 |
| topics[1].field.id | https://openalex.org/fields/27 |
| topics[1].field.display_name | Medicine |
| topics[1].score | 0.9868000149726868 |
| topics[1].domain.id | https://openalex.org/domains/4 |
| topics[1].domain.display_name | Health Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/2739 |
| topics[1].subfield.display_name | Public Health, Environmental and Occupational Health |
| topics[1].display_name | Digital Imaging in Medicine |
| topics[2].id | https://openalex.org/T10862 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9836999773979187 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1702 |
| topics[2].subfield.display_name | Artificial Intelligence |
| topics[2].display_name | AI in cancer detection |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C2779185108 |
| concepts[0].level | 2 |
| concepts[0].score | 0.6685990691184998 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q7091744 |
| concepts[0].display_name | Onboarding |
| concepts[1].id | https://openalex.org/C41008148 |
| concepts[1].level | 0 |
| concepts[1].score | 0.6066702604293823 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[1].display_name | Computer science |
| concepts[2].id | https://openalex.org/C2779530757 |
| concepts[2].level | 2 |
| concepts[2].score | 0.5904151201248169 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q1207505 |
| concepts[2].display_name | Quality (philosophy) |
| concepts[3].id | https://openalex.org/C98045186 |
| concepts[3].level | 2 |
| concepts[3].score | 0.5421280860900879 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q205663 |
| concepts[3].display_name | Process (computing) |
| concepts[4].id | https://openalex.org/C62230096 |
| concepts[4].level | 2 |
| concepts[4].score | 0.48118525743484497 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q275969 |
| concepts[4].display_name | Crowdsourcing |
| concepts[5].id | https://openalex.org/C2522767166 |
| concepts[5].level | 1 |
| concepts[5].score | 0.4470430016517639 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q2374463 |
| concepts[5].display_name | Data science |
| concepts[6].id | https://openalex.org/C2775924081 |
| concepts[6].level | 2 |
| concepts[6].score | 0.4344622492790222 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q55608371 |
| concepts[6].display_name | Control (management) |
| concepts[7].id | https://openalex.org/C184356942 |
| concepts[7].level | 2 |
| concepts[7].score | 0.42745086550712585 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q830382 |
| concepts[7].display_name | Best practice |
| concepts[8].id | https://openalex.org/C133462117 |
| concepts[8].level | 2 |
| concepts[8].score | 0.42173510789871216 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q4929239 |
| concepts[8].display_name | Data collection |
| concepts[9].id | https://openalex.org/C56739046 |
| concepts[9].level | 1 |
| concepts[9].score | 0.3862575888633728 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q192060 |
| concepts[9].display_name | Knowledge management |
| concepts[10].id | https://openalex.org/C154945302 |
| concepts[10].level | 1 |
| concepts[10].score | 0.28056615591049194 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[10].display_name | Artificial intelligence |
| concepts[11].id | https://openalex.org/C15744967 |
| concepts[11].level | 0 |
| concepts[11].score | 0.18618234992027283 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q9418 |
| concepts[11].display_name | Psychology |
| concepts[12].id | https://openalex.org/C136764020 |
| concepts[12].level | 1 |
| concepts[12].score | 0.1479400396347046 |
| concepts[12].wikidata | https://www.wikidata.org/wiki/Q466 |
| concepts[12].display_name | World Wide Web |
| concepts[13].id | https://openalex.org/C111919701 |
| concepts[13].level | 1 |
| concepts[13].score | 0.0 |
| concepts[13].wikidata | https://www.wikidata.org/wiki/Q9135 |
| concepts[13].display_name | Operating system |
| concepts[14].id | https://openalex.org/C162324750 |
| concepts[14].level | 0 |
| concepts[14].score | 0.0 |
| concepts[14].wikidata | https://www.wikidata.org/wiki/Q8134 |
| concepts[14].display_name | Economics |
| concepts[15].id | https://openalex.org/C33923547 |
| concepts[15].level | 0 |
| concepts[15].score | 0.0 |
| concepts[15].wikidata | https://www.wikidata.org/wiki/Q395 |
| concepts[15].display_name | Mathematics |
| concepts[16].id | https://openalex.org/C77805123 |
| concepts[16].level | 1 |
| concepts[16].score | 0.0 |
| concepts[16].wikidata | https://www.wikidata.org/wiki/Q161272 |
| concepts[16].display_name | Social psychology |
| concepts[17].id | https://openalex.org/C138885662 |
| concepts[17].level | 0 |
| concepts[17].score | 0.0 |
| concepts[17].wikidata | https://www.wikidata.org/wiki/Q5891 |
| concepts[17].display_name | Philosophy |
| concepts[18].id | https://openalex.org/C105795698 |
| concepts[18].level | 1 |
| concepts[18].score | 0.0 |
| concepts[18].wikidata | https://www.wikidata.org/wiki/Q12483 |
| concepts[18].display_name | Statistics |
| concepts[19].id | https://openalex.org/C111472728 |
| concepts[19].level | 1 |
| concepts[19].score | 0.0 |
| concepts[19].wikidata | https://www.wikidata.org/wiki/Q9471 |
| concepts[19].display_name | Epistemology |
| concepts[20].id | https://openalex.org/C187736073 |
| concepts[20].level | 1 |
| concepts[20].score | 0.0 |
| concepts[20].wikidata | https://www.wikidata.org/wiki/Q2920921 |
| concepts[20].display_name | Management |
| keywords[0].id | https://openalex.org/keywords/onboarding |
| keywords[0].score | 0.6685990691184998 |
| keywords[0].display_name | Onboarding |
| keywords[1].id | https://openalex.org/keywords/computer-science |
| keywords[1].score | 0.6066702604293823 |
| keywords[1].display_name | Computer science |
| keywords[2].id | https://openalex.org/keywords/quality |
| keywords[2].score | 0.5904151201248169 |
| keywords[2].display_name | Quality (philosophy) |
| keywords[3].id | https://openalex.org/keywords/process |
| keywords[3].score | 0.5421280860900879 |
| keywords[3].display_name | Process (computing) |
| keywords[4].id | https://openalex.org/keywords/crowdsourcing |
| keywords[4].score | 0.48118525743484497 |
| keywords[4].display_name | Crowdsourcing |
| keywords[5].id | https://openalex.org/keywords/data-science |
| keywords[5].score | 0.4470430016517639 |
| keywords[5].display_name | Data science |
| keywords[6].id | https://openalex.org/keywords/control |
| keywords[6].score | 0.4344622492790222 |
| keywords[6].display_name | Control (management) |
| keywords[7].id | https://openalex.org/keywords/best-practice |
| keywords[7].score | 0.42745086550712585 |
| keywords[7].display_name | Best practice |
| keywords[8].id | https://openalex.org/keywords/data-collection |
| keywords[8].score | 0.42173510789871216 |
| keywords[8].display_name | Data collection |
| keywords[9].id | https://openalex.org/keywords/knowledge-management |
| keywords[9].score | 0.3862575888633728 |
| keywords[9].display_name | Knowledge management |
| keywords[10].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[10].score | 0.28056615591049194 |
| keywords[10].display_name | Artificial intelligence |
| keywords[11].id | https://openalex.org/keywords/psychology |
| keywords[11].score | 0.18618234992027283 |
| keywords[11].display_name | Psychology |
| keywords[12].id | https://openalex.org/keywords/world-wide-web |
| keywords[12].score | 0.1479400396347046 |
| keywords[12].display_name | World Wide Web |
| language | en |
| locations[0].id | doi:10.1609/hcomp.v9i1.18940 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4387284122 |
| locations[0].source.issn | 2769-1330, 2769-1349 |
| locations[0].source.type | journal |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | 2769-1330 |
| locations[0].source.is_core | True |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | Proceedings of the AAAI Conference on Human Computation and Crowdsourcing |
| locations[0].source.host_organization | |
| locations[0].source.host_organization_name | |
| locations[0].source.host_organization_lineage | |
| locations[0].license | |
| locations[0].pdf_url | https://ojs.aaai.org/index.php/HCOMP/article/download/18940/18705 |
| locations[0].version | publishedVersion |
| locations[0].raw_type | journal-article |
| locations[0].license_id | |
| locations[0].is_accepted | True |
| locations[0].is_published | True |
| locations[0].raw_source_name | Proceedings of the AAAI Conference on Human Computation and Crowdsourcing |
| locations[0].landing_page_url | https://doi.org/10.1609/hcomp.v9i1.18940 |
| indexed_in | crossref |
| authorships[0].author.id | https://openalex.org/A5038748737 |
| authorships[0].author.orcid | https://orcid.org/0009-0009-9489-2161 |
| authorships[0].author.display_name | Beverly Freeman |
| authorships[0].countries | US |
| authorships[0].affiliations[0].institution_ids | https://openalex.org/I1291425158 |
| authorships[0].affiliations[0].raw_affiliation_string | |
| authorships[0].institutions[0].id | https://openalex.org/I1291425158 |
| authorships[0].institutions[0].ror | https://ror.org/00njsd438 |
| authorships[0].institutions[0].type | company |
| authorships[0].institutions[0].lineage | https://openalex.org/I1291425158, https://openalex.org/I4210128969 |
| authorships[0].institutions[0].country_code | US |
| authorships[0].institutions[0].display_name | Google (United States) |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Beverly Freeman |
| authorships[0].is_corresponding | False |
| authorships[0].raw_affiliation_strings | |
| authorships[1].author.id | https://openalex.org/A5003577779 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-1284-0061 |
| authorships[1].author.display_name | Naama Hammel |
| authorships[1].countries | US |
| authorships[1].affiliations[0].institution_ids | https://openalex.org/I1291425158 |
| authorships[1].affiliations[0].raw_affiliation_string | |
| authorships[1].institutions[0].id | https://openalex.org/I1291425158 |
| authorships[1].institutions[0].ror | https://ror.org/00njsd438 |
| authorships[1].institutions[0].type | company |
| authorships[1].institutions[0].lineage | https://openalex.org/I1291425158, https://openalex.org/I4210128969 |
| authorships[1].institutions[0].country_code | US |
| authorships[1].institutions[0].display_name | Google (United States) |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Naama Hammel |
| authorships[1].is_corresponding | False |
| authorships[1].raw_affiliation_strings | |
| authorships[2].author.id | https://openalex.org/A5026843481 |
| authorships[2].author.orcid | |
| authorships[2].author.display_name | Sonia Phene |
| authorships[2].countries | US |
| authorships[2].affiliations[0].institution_ids | https://openalex.org/I1291425158 |
| authorships[2].affiliations[0].raw_affiliation_string | |
| authorships[2].institutions[0].id | https://openalex.org/I1291425158 |
| authorships[2].institutions[0].ror | https://ror.org/00njsd438 |
| authorships[2].institutions[0].type | company |
| authorships[2].institutions[0].lineage | https://openalex.org/I1291425158, https://openalex.org/I4210128969 |
| authorships[2].institutions[0].country_code | US |
| authorships[2].institutions[0].display_name | Google (United States) |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Sonia Phene |
| authorships[2].is_corresponding | False |
| authorships[2].raw_affiliation_strings | |
| authorships[3].author.id | https://openalex.org/A5112695586 |
| authorships[3].author.orcid | |
| authorships[3].author.display_name | Abigail E. Huang |
| authorships[3].countries | US |
| authorships[3].affiliations[0].institution_ids | https://openalex.org/I1291425158 |
| authorships[3].affiliations[0].raw_affiliation_string | |
| authorships[3].institutions[0].id | https://openalex.org/I1291425158 |
| authorships[3].institutions[0].ror | https://ror.org/00njsd438 |
| authorships[3].institutions[0].type | company |
| authorships[3].institutions[0].lineage | https://openalex.org/I1291425158, https://openalex.org/I4210128969 |
| authorships[3].institutions[0].country_code | US |
| authorships[3].institutions[0].display_name | Google (United States) |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Abigail Huang |
| authorships[3].is_corresponding | False |
| authorships[3].raw_affiliation_strings | |
| authorships[4].author.id | https://openalex.org/A5069810248 |
| authorships[4].author.orcid | https://orcid.org/0000-0001-8757-6878 |
| authorships[4].author.display_name | R. Ackermann |
| authorships[4].countries | US |
| authorships[4].affiliations[0].institution_ids | https://openalex.org/I1291425158 |
| authorships[4].affiliations[0].raw_affiliation_string | |
| authorships[4].institutions[0].id | https://openalex.org/I1291425158 |
| authorships[4].institutions[0].ror | https://ror.org/00njsd438 |
| authorships[4].institutions[0].type | company |
| authorships[4].institutions[0].lineage | https://openalex.org/I1291425158, https://openalex.org/I4210128969 |
| authorships[4].institutions[0].country_code | US |
| authorships[4].institutions[0].display_name | Google (United States) |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Rebecca Ackermann |
| authorships[4].is_corresponding | False |
| authorships[4].raw_affiliation_strings | |
| authorships[5].author.id | https://openalex.org/A5077397088 |
| authorships[5].author.orcid | |
| authorships[5].author.display_name | Olga Kanzheleva |
| authorships[5].countries | US |
| authorships[5].affiliations[0].institution_ids | https://openalex.org/I1291425158 |
| authorships[5].affiliations[0].raw_affiliation_string | |
| authorships[5].institutions[0].id | https://openalex.org/I1291425158 |
| authorships[5].institutions[0].ror | https://ror.org/00njsd438 |
| authorships[5].institutions[0].type | company |
| authorships[5].institutions[0].lineage | https://openalex.org/I1291425158, https://openalex.org/I4210128969 |
| authorships[5].institutions[0].country_code | US |
| authorships[5].institutions[0].display_name | Google (United States) |
| authorships[5].author_position | middle |
| authorships[5].raw_author_name | Olga Kanzheleva |
| authorships[5].is_corresponding | False |
| authorships[5].raw_affiliation_strings | |
| authorships[6].author.id | https://openalex.org/A5019798634 |
| authorships[6].author.orcid | |
| authorships[6].author.display_name | Miles Hutson |
| authorships[6].countries | US |
| authorships[6].affiliations[0].institution_ids | https://openalex.org/I1291425158 |
| authorships[6].affiliations[0].raw_affiliation_string | |
| authorships[6].institutions[0].id | https://openalex.org/I1291425158 |
| authorships[6].institutions[0].ror | https://ror.org/00njsd438 |
| authorships[6].institutions[0].type | company |
| authorships[6].institutions[0].lineage | https://openalex.org/I1291425158, https://openalex.org/I4210128969 |
| authorships[6].institutions[0].country_code | US |
| authorships[6].institutions[0].display_name | Google (United States) |
| authorships[6].author_position | middle |
| authorships[6].raw_author_name | Miles Hutson |
| authorships[6].is_corresponding | False |
| authorships[6].raw_affiliation_strings | |
| authorships[7].author.id | https://openalex.org/A5010870339 |
| authorships[7].author.orcid | |
| authorships[7].author.display_name | Caitlin Taggart |
| authorships[7].countries | US |
| authorships[7].affiliations[0].institution_ids | https://openalex.org/I1291425158 |
| authorships[7].affiliations[0].raw_affiliation_string | |
| authorships[7].institutions[0].id | https://openalex.org/I1291425158 |
| authorships[7].institutions[0].ror | https://ror.org/00njsd438 |
| authorships[7].institutions[0].type | company |
| authorships[7].institutions[0].lineage | https://openalex.org/I1291425158, https://openalex.org/I4210128969 |
| authorships[7].institutions[0].country_code | US |
| authorships[7].institutions[0].display_name | Google (United States) |
| authorships[7].author_position | middle |
| authorships[7].raw_author_name | Caitlin Taggart |
| authorships[7].is_corresponding | False |
| authorships[7].raw_affiliation_strings | |
| authorships[8].author.id | https://openalex.org/A5110021473 |
| authorships[8].author.orcid | |
| authorships[8].author.display_name | Quang Duong |
| authorships[8].countries | US |
| authorships[8].affiliations[0].institution_ids | https://openalex.org/I1291425158 |
| authorships[8].affiliations[0].raw_affiliation_string | |
| authorships[8].institutions[0].id | https://openalex.org/I1291425158 |
| authorships[8].institutions[0].ror | https://ror.org/00njsd438 |
| authorships[8].institutions[0].type | company |
| authorships[8].institutions[0].lineage | https://openalex.org/I1291425158, https://openalex.org/I4210128969 |
| authorships[8].institutions[0].country_code | US |
| authorships[8].institutions[0].display_name | Google (United States) |
| authorships[8].author_position | middle |
| authorships[8].raw_author_name | Quang Duong |
| authorships[8].is_corresponding | False |
| authorships[8].raw_affiliation_strings | |
| authorships[9].author.id | https://openalex.org/A5058399428 |
| authorships[9].author.orcid | https://orcid.org/0000-0002-8269-5779 |
| authorships[9].author.display_name | Rory Sayres |
| authorships[9].countries | US |
| authorships[9].affiliations[0].institution_ids | https://openalex.org/I1291425158 |
| authorships[9].affiliations[0].raw_affiliation_string | |
| authorships[9].institutions[0].id | https://openalex.org/I1291425158 |
| authorships[9].institutions[0].ror | https://ror.org/00njsd438 |
| authorships[9].institutions[0].type | company |
| authorships[9].institutions[0].lineage | https://openalex.org/I1291425158, https://openalex.org/I4210128969 |
| authorships[9].institutions[0].country_code | US |
| authorships[9].institutions[0].display_name | Google (United States) |
| authorships[9].author_position | last |
| authorships[9].raw_author_name | Rory Sayres |
| authorships[9].is_corresponding | False |
| authorships[9].raw_affiliation_strings | |
| has_content.pdf | True |
| has_content.grobid_xml | True |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://ojs.aaai.org/index.php/HCOMP/article/download/18940/18705 |
| open_access.oa_status | diamond |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Iterative Quality Control Strategies for Expert Medical Image Labeling |
| has_fulltext | True |
| is_retracted | False |
| updated_date | 2025-11-06T03:46:38.306776 |
| primary_topic.id | https://openalex.org/T11636 |
| primary_topic.field.id | https://openalex.org/fields/27 |
| primary_topic.field.display_name | Medicine |
| primary_topic.score | 0.9980000257492065 |
| primary_topic.domain.id | https://openalex.org/domains/4 |
| primary_topic.domain.display_name | Health Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/2718 |
| primary_topic.subfield.display_name | Health Informatics |
| primary_topic.display_name | Artificial Intelligence in Healthcare and Education |
| related_works | https://openalex.org/W1490650877, https://openalex.org/W3000722155, https://openalex.org/W1553944497, https://openalex.org/W2383310903, https://openalex.org/W2613733560, https://openalex.org/W2803870934, https://openalex.org/W2945404560, https://openalex.org/W4286485160, https://openalex.org/W3125074139, https://openalex.org/W2044432453 |
| cited_by_count | 14 |
| counts_by_year[0].year | 2025 |
| counts_by_year[0].cited_by_count | 2 |
| counts_by_year[1].year | 2024 |
| counts_by_year[1].cited_by_count | 3 |
| counts_by_year[2].year | 2023 |
| counts_by_year[2].cited_by_count | 8 |
| counts_by_year[3].year | 2022 |
| counts_by_year[3].cited_by_count | 1 |
| locations_count | 1 |
| best_oa_location.id | doi:10.1609/hcomp.v9i1.18940 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4387284122 |
| best_oa_location.source.issn | 2769-1330, 2769-1349 |
| best_oa_location.source.type | journal |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | 2769-1330 |
| best_oa_location.source.is_core | True |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | Proceedings of the AAAI Conference on Human Computation and Crowdsourcing |
| best_oa_location.source.host_organization | |
| best_oa_location.source.host_organization_name | |
| best_oa_location.source.host_organization_lineage | |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://ojs.aaai.org/index.php/HCOMP/article/download/18940/18705 |
| best_oa_location.version | publishedVersion |
| best_oa_location.raw_type | journal-article |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | True |
| best_oa_location.is_published | True |
| best_oa_location.raw_source_name | Proceedings of the AAAI Conference on Human Computation and Crowdsourcing |
| best_oa_location.landing_page_url | https://doi.org/10.1609/hcomp.v9i1.18940 |
| primary_location.id | doi:10.1609/hcomp.v9i1.18940 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4387284122 |
| primary_location.source.issn | 2769-1330, 2769-1349 |
| primary_location.source.type | journal |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | 2769-1330 |
| primary_location.source.is_core | True |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | Proceedings of the AAAI Conference on Human Computation and Crowdsourcing |
| primary_location.source.host_organization | |
| primary_location.source.host_organization_name | |
| primary_location.source.host_organization_lineage | |
| primary_location.license | |
| primary_location.pdf_url | https://ojs.aaai.org/index.php/HCOMP/article/download/18940/18705 |
| primary_location.version | publishedVersion |
| primary_location.raw_type | journal-article |
| primary_location.license_id | |
| primary_location.is_accepted | True |
| primary_location.is_published | True |
| primary_location.raw_source_name | Proceedings of the AAAI Conference on Human Computation and Crowdsourcing |
| primary_location.landing_page_url | https://doi.org/10.1609/hcomp.v9i1.18940 |
| publication_date | 2021-10-04 |
| publication_year | 2021 |
| referenced_works | https://openalex.org/W2974737693, https://openalex.org/W2896881735, https://openalex.org/W6645210854, https://openalex.org/W6763678638, https://openalex.org/W2970837303, https://openalex.org/W2910707576, https://openalex.org/W3010595911, https://openalex.org/W2105413315, https://openalex.org/W2167431761, https://openalex.org/W2782152208, https://openalex.org/W2581082771, https://openalex.org/W2557738935, https://openalex.org/W2949074827, https://openalex.org/W2117470435, https://openalex.org/W3088298690, https://openalex.org/W1997504572, https://openalex.org/W3005653987, https://openalex.org/W2098865355, https://openalex.org/W2115206325, https://openalex.org/W2909334458, https://openalex.org/W6743716967, https://openalex.org/W2762741128, https://openalex.org/W2187291759, https://openalex.org/W3105070630, https://openalex.org/W6733997263, https://openalex.org/W2946474696, https://openalex.org/W2998175747, https://openalex.org/W3044088237, https://openalex.org/W4232379246, https://openalex.org/W4200634839, https://openalex.org/W2902802452, https://openalex.org/W3118813946, https://openalex.org/W3031679542, https://openalex.org/W3009312460, https://openalex.org/W2772246530, https://openalex.org/W1499884756, https://openalex.org/W2916105049, https://openalex.org/W2883147591, https://openalex.org/W2950504429, https://openalex.org/W3104634615, https://openalex.org/W3122383744, https://openalex.org/W4287662897, https://openalex.org/W2916545656, https://openalex.org/W4280545690, https://openalex.org/W2752099845, https://openalex.org/W2588363885, https://openalex.org/W1979290264, https://openalex.org/W3100511085, https://openalex.org/W2739352035, https://openalex.org/W2963772355, https://openalex.org/W2507467374 |
| referenced_works_count | 51 |
| abstract_inverted_index.a | 3, 107 |
| abstract_inverted_index.1) | 122 |
| abstract_inverted_index.2) | 131 |
| abstract_inverted_index.3) | 140 |
| abstract_inverted_index.AI | 34, 54, 68 |
| abstract_inverted_index.In | 17, 59 |
| abstract_inverted_index.We | 84 |
| abstract_inverted_index.by | 93 |
| abstract_inverted_index.do | 53 |
| abstract_inverted_index.in | 22, 48, 72, 106 |
| abstract_inverted_index.is | 2, 101 |
| abstract_inverted_index.of | 20, 65, 88, 115 |
| abstract_inverted_index.on | 13 |
| abstract_inverted_index.or | 31, 38 |
| abstract_inverted_index.to | 40, 102, 112, 145 |
| abstract_inverted_index.we | 62 |
| abstract_inverted_index.How | 52 |
| abstract_inverted_index.The | 96 |
| abstract_inverted_index.and | 78, 126, 133, 137, 139, 147 |
| abstract_inverted_index.are | 44 |
| abstract_inverted_index.for | 6, 69 |
| abstract_inverted_index.key | 4 |
| abstract_inverted_index.may | 35 |
| abstract_inverted_index.one | 86 |
| abstract_inverted_index.the | 18, 32, 45, 113, 135 |
| abstract_inverted_index.(AI) | 9 |
| abstract_inverted_index.Best | 119 |
| abstract_inverted_index.Data | 0 |
| abstract_inverted_index.What | 43 |
| abstract_inverted_index.data | 15, 25, 117 |
| abstract_inverted_index.four | 73 |
| abstract_inverted_index.high | 28 |
| abstract_inverted_index.lead | 39 |
| abstract_inverted_index.meet | 27 |
| abstract_inverted_index.more | 97 |
| abstract_inverted_index.must | 26 |
| abstract_inverted_index.rely | 12 |
| abstract_inverted_index.such | 57 |
| abstract_inverted_index.that | 11 |
| abstract_inverted_index.they | 151 |
| abstract_inverted_index.this | 60 |
| abstract_inverted_index.with | 104, 129 |
| abstract_inverted_index.about | 80 |
| abstract_inverted_index.being | 91 |
| abstract_inverted_index.harm. | 42 |
| abstract_inverted_index.prior | 111 |
| abstract_inverted_index.start | 114 |
| abstract_inverted_index.tasks | 125, 136 |
| abstract_inverted_index.teams | 66, 144 |
| abstract_inverted_index.their | 81 |
| abstract_inverted_index.before | 150 |
| abstract_inverted_index.biases | 37 |
| abstract_inverted_index.caught | 92 |
| abstract_inverted_index.domain | 19 |
| abstract_inverted_index.enable | 143 |
| abstract_inverted_index.expert | 49 |
| abstract_inverted_index.issues | 149 |
| abstract_inverted_index.study, | 61 |
| abstract_inverted_index.address | 56, 148 |
| abstract_inverted_index.concern | 5 |
| abstract_inverted_index.efforts | 10 |
| abstract_inverted_index.experts | 105 |
| abstract_inverted_index.imaging | 71 |
| abstract_inverted_index.labeled | 24 |
| abstract_inverted_index.medical | 50, 70 |
| abstract_inverted_index.members | 64 |
| abstract_inverted_index.partner | 103 |
| abstract_inverted_index.patient | 41 |
| abstract_inverted_index.process | 110 |
| abstract_inverted_index.quality | 1, 29 |
| abstract_inverted_index.workers | 142 |
| abstract_inverted_index.describe | 85 |
| abstract_inverted_index.experts, | 130 |
| abstract_inverted_index.however, | 100 |
| abstract_inverted_index.identify | 146 |
| abstract_inverted_index.instance | 87 |
| abstract_inverted_index.involved | 47 |
| abstract_inverted_index.labeling | 90, 124 |
| abstract_inverted_index.medicine | 21 |
| abstract_inverted_index.piloting | 132 |
| abstract_inverted_index.revising | 134 |
| abstract_inverted_index.automated | 94 |
| abstract_inverted_index.including | 121 |
| abstract_inverted_index.iterative | 109 |
| abstract_inverted_index.labeling? | 51 |
| abstract_inverted_index.practices | 120 |
| abstract_inverted_index.proactive | 98 |
| abstract_inverted_index.resulting | 33 |
| abstract_inverted_index.strategy, | 99 |
| abstract_inverted_index.artificial | 7 |
| abstract_inverted_index.challenges | 46 |
| abstract_inverted_index.developing | 67 |
| abstract_inverted_index.guidelines | 128 |
| abstract_inverted_index.onboarding | 141 |
| abstract_inverted_index.pathology, | 77 |
| abstract_inverted_index.perpetuate | 36 |
| abstract_inverted_index.practices. | 83 |
| abstract_inverted_index.radiology, | 76 |
| abstract_inverted_index.standards, | 30 |
| abstract_inverted_index.subdomains | 74 |
| abstract_inverted_index.challenges? | 58 |
| abstract_inverted_index.collection. | 16, 118 |
| abstract_inverted_index.guidelines, | 138 |
| abstract_inverted_index.high-volume | 116 |
| abstract_inverted_index.interviewed | 63 |
| abstract_inverted_index.low-quality | 89 |
| abstract_inverted_index.monitoring. | 95 |
| abstract_inverted_index.particular, | 23 |
| abstract_inverted_index.co-designing | 123 |
| abstract_inverted_index.crowdsourced | 14 |
| abstract_inverted_index.dermatology) | 79 |
| abstract_inverted_index.intelligence | 8 |
| abstract_inverted_index.proliferate. | 152 |
| abstract_inverted_index.instructional | 127 |
| abstract_inverted_index.practitioners | 55 |
| abstract_inverted_index.collaborative, | 108 |
| abstract_inverted_index.(ophthalmology, | 75 |
| abstract_inverted_index.quality-related | 82 |
| cited_by_percentile_year.max | 99 |
| cited_by_percentile_year.min | 89 |
| countries_distinct_count | 1 |
| institutions_distinct_count | 10 |
| sustainable_development_goals[0].id | https://metadata.un.org/sdg/17 |
| sustainable_development_goals[0].score | 0.4099999964237213 |
| sustainable_development_goals[0].display_name | Partnerships for the goals |
| citation_normalized_percentile.value | 0.76531049 |
| citation_normalized_percentile.is_in_top_1_percent | False |
| citation_normalized_percentile.is_in_top_10_percent | False |