CODA-19: Using a Non-Expert Crowd to Annotate Research Aspects on 10,000+ Abstracts in the COVID-19 Open Research Dataset Article Swipe
This paper introduces CODA-19, a human-annotated dataset that codes the Background, Purpose, Method, Finding/Contribution, and Other sections of 10,966 English abstracts in the COVID-19 Open Research Dataset. CODA-19 was created by 248 crowd workers from Amazon Mechanical Turk within 10 days, and achieved labeling quality comparable to that of experts. Each abstract was annotated by nine different workers, and the final labels were acquired by majority vote. The inter-annotator agreement (Cohen's kappa) between the crowd and the biomedical expert (0.741) is comparable to inter-expert agreement (0.788). CODA-19's labels have an accuracy of 82.2% when compared to the biomedical expert's labels, while the accuracy between experts was 85.0%. Reliable human annotations help scientists access and integrate the rapidly accelerating coronavirus literature, and also serve as the battery of AI/NLP research, but obtaining expert annotations can be slow. We demonstrated that a non-expert crowd can be rapidly employed at scale to join the fight against COVID-19.
Related Topics
Concepts
Coda
Coronavirus disease 2019 (COVID-19)
Computer science
Crowdsourcing
Artificial intelligence
Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2)
Quality (philosophy)
2019-20 coronavirus outbreak
Natural language processing
Information retrieval
World Wide Web
Medicine
Seismology
Geology
Epistemology
Outbreak
Virology
Disease
Pathology
Infectious disease (medical specialty)
Philosophy
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://export.arxiv.org/pdf/2005.02367
- OA Status
- green
- Cited By
- 11
- References
- 57
- Related Works
- 20
- OpenAlex ID
- https://openalex.org/W3049669010
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W3049669010Canonical identifier for this work in OpenAlex
- Title
-
CODA-19: Using a Non-Expert Crowd to Annotate Research Aspects on 10,000+ Abstracts in the COVID-19 Open Research DatasetWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2020Year of publication
- Publication date
-
2020-08-17Full publication date if available
- Authors
-
Ting-Hao Huang, Chieh-Yang Huang, Ding Cc, Hsu Y, Giles ClList of authors in order
- Landing page
-
https://export.arxiv.org/pdf/2005.02367Publisher landing page
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://export.arxiv.org/pdf/2005.02367Direct OA link when available
- Concepts
-
Coda, Coronavirus disease 2019 (COVID-19), Computer science, Crowdsourcing, Artificial intelligence, Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), Quality (philosophy), 2019-20 coronavirus outbreak, Natural language processing, Information retrieval, World Wide Web, Medicine, Seismology, Geology, Epistemology, Outbreak, Virology, Disease, Pathology, Infectious disease (medical specialty), PhilosophyTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
11Total citation count in OpenAlex
- Citations by year (recent)
-
2022: 2, 2021: 6, 2020: 3Per-year citation counts (last 5 years)
- References (count)
-
57Number of works referenced by this work
- Related works (count)
-
20Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W3049669010 |
|---|---|
| doi | |
| ids.mag | 3049669010 |
| ids.openalex | https://openalex.org/W3049669010 |
| fwci | |
| type | preprint |
| title | CODA-19: Using a Non-Expert Crowd to Annotate Research Aspects on 10,000+ Abstracts in the COVID-19 Open Research Dataset |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T11147 |
| topics[0].field.id | https://openalex.org/fields/33 |
| topics[0].field.display_name | Social Sciences |
| topics[0].score | 0.9901000261306763 |
| topics[0].domain.id | https://openalex.org/domains/2 |
| topics[0].domain.display_name | Social Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/3312 |
| topics[0].subfield.display_name | Sociology and Political Science |
| topics[0].display_name | Misinformation and Its Impacts |
| topics[1].id | https://openalex.org/T10028 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9890000224113464 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Topic Modeling |
| topics[2].id | https://openalex.org/T12262 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9853000044822693 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1702 |
| topics[2].subfield.display_name | Artificial Intelligence |
| topics[2].display_name | Hate Speech and Cyberbullying Detection |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C3962253 |
| concepts[0].level | 2 |
| concepts[0].score | 0.948418140411377 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q852466 |
| concepts[0].display_name | Coda |
| concepts[1].id | https://openalex.org/C3008058167 |
| concepts[1].level | 4 |
| concepts[1].score | 0.6900708079338074 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q84263196 |
| concepts[1].display_name | Coronavirus disease 2019 (COVID-19) |
| concepts[2].id | https://openalex.org/C41008148 |
| concepts[2].level | 0 |
| concepts[2].score | 0.6518798470497131 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[2].display_name | Computer science |
| concepts[3].id | https://openalex.org/C62230096 |
| concepts[3].level | 2 |
| concepts[3].score | 0.5964641571044922 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q275969 |
| concepts[3].display_name | Crowdsourcing |
| concepts[4].id | https://openalex.org/C154945302 |
| concepts[4].level | 1 |
| concepts[4].score | 0.48202574253082275 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[4].display_name | Artificial intelligence |
| concepts[5].id | https://openalex.org/C3007834351 |
| concepts[5].level | 5 |
| concepts[5].score | 0.47271057963371277 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q82069695 |
| concepts[5].display_name | Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) |
| concepts[6].id | https://openalex.org/C2779530757 |
| concepts[6].level | 2 |
| concepts[6].score | 0.4346185326576233 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q1207505 |
| concepts[6].display_name | Quality (philosophy) |
| concepts[7].id | https://openalex.org/C3006700255 |
| concepts[7].level | 3 |
| concepts[7].score | 0.4274039566516876 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q81068910 |
| concepts[7].display_name | 2019-20 coronavirus outbreak |
| concepts[8].id | https://openalex.org/C204321447 |
| concepts[8].level | 1 |
| concepts[8].score | 0.4241904616355896 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q30642 |
| concepts[8].display_name | Natural language processing |
| concepts[9].id | https://openalex.org/C23123220 |
| concepts[9].level | 1 |
| concepts[9].score | 0.3361148238182068 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q816826 |
| concepts[9].display_name | Information retrieval |
| concepts[10].id | https://openalex.org/C136764020 |
| concepts[10].level | 1 |
| concepts[10].score | 0.19703149795532227 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q466 |
| concepts[10].display_name | World Wide Web |
| concepts[11].id | https://openalex.org/C71924100 |
| concepts[11].level | 0 |
| concepts[11].score | 0.09061512351036072 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q11190 |
| concepts[11].display_name | Medicine |
| concepts[12].id | https://openalex.org/C165205528 |
| concepts[12].level | 1 |
| concepts[12].score | 0.06147852540016174 |
| concepts[12].wikidata | https://www.wikidata.org/wiki/Q83371 |
| concepts[12].display_name | Seismology |
| concepts[13].id | https://openalex.org/C127313418 |
| concepts[13].level | 0 |
| concepts[13].score | 0.0 |
| concepts[13].wikidata | https://www.wikidata.org/wiki/Q1069 |
| concepts[13].display_name | Geology |
| concepts[14].id | https://openalex.org/C111472728 |
| concepts[14].level | 1 |
| concepts[14].score | 0.0 |
| concepts[14].wikidata | https://www.wikidata.org/wiki/Q9471 |
| concepts[14].display_name | Epistemology |
| concepts[15].id | https://openalex.org/C116675565 |
| concepts[15].level | 2 |
| concepts[15].score | 0.0 |
| concepts[15].wikidata | https://www.wikidata.org/wiki/Q3241045 |
| concepts[15].display_name | Outbreak |
| concepts[16].id | https://openalex.org/C159047783 |
| concepts[16].level | 1 |
| concepts[16].score | 0.0 |
| concepts[16].wikidata | https://www.wikidata.org/wiki/Q7215 |
| concepts[16].display_name | Virology |
| concepts[17].id | https://openalex.org/C2779134260 |
| concepts[17].level | 2 |
| concepts[17].score | 0.0 |
| concepts[17].wikidata | https://www.wikidata.org/wiki/Q12136 |
| concepts[17].display_name | Disease |
| concepts[18].id | https://openalex.org/C142724271 |
| concepts[18].level | 1 |
| concepts[18].score | 0.0 |
| concepts[18].wikidata | https://www.wikidata.org/wiki/Q7208 |
| concepts[18].display_name | Pathology |
| concepts[19].id | https://openalex.org/C524204448 |
| concepts[19].level | 3 |
| concepts[19].score | 0.0 |
| concepts[19].wikidata | https://www.wikidata.org/wiki/Q788926 |
| concepts[19].display_name | Infectious disease (medical specialty) |
| concepts[20].id | https://openalex.org/C138885662 |
| concepts[20].level | 0 |
| concepts[20].score | 0.0 |
| concepts[20].wikidata | https://www.wikidata.org/wiki/Q5891 |
| concepts[20].display_name | Philosophy |
| keywords[0].id | https://openalex.org/keywords/coda |
| keywords[0].score | 0.948418140411377 |
| keywords[0].display_name | Coda |
| keywords[1].id | https://openalex.org/keywords/coronavirus-disease-2019 |
| keywords[1].score | 0.6900708079338074 |
| keywords[1].display_name | Coronavirus disease 2019 (COVID-19) |
| keywords[2].id | https://openalex.org/keywords/computer-science |
| keywords[2].score | 0.6518798470497131 |
| keywords[2].display_name | Computer science |
| keywords[3].id | https://openalex.org/keywords/crowdsourcing |
| keywords[3].score | 0.5964641571044922 |
| keywords[3].display_name | Crowdsourcing |
| keywords[4].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[4].score | 0.48202574253082275 |
| keywords[4].display_name | Artificial intelligence |
| keywords[5].id | https://openalex.org/keywords/severe-acute-respiratory-syndrome-coronavirus-2 |
| keywords[5].score | 0.47271057963371277 |
| keywords[5].display_name | Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) |
| keywords[6].id | https://openalex.org/keywords/quality |
| keywords[6].score | 0.4346185326576233 |
| keywords[6].display_name | Quality (philosophy) |
| keywords[7].id | https://openalex.org/keywords/2019-20-coronavirus-outbreak |
| keywords[7].score | 0.4274039566516876 |
| keywords[7].display_name | 2019-20 coronavirus outbreak |
| keywords[8].id | https://openalex.org/keywords/natural-language-processing |
| keywords[8].score | 0.4241904616355896 |
| keywords[8].display_name | Natural language processing |
| keywords[9].id | https://openalex.org/keywords/information-retrieval |
| keywords[9].score | 0.3361148238182068 |
| keywords[9].display_name | Information retrieval |
| keywords[10].id | https://openalex.org/keywords/world-wide-web |
| keywords[10].score | 0.19703149795532227 |
| keywords[10].display_name | World Wide Web |
| keywords[11].id | https://openalex.org/keywords/medicine |
| keywords[11].score | 0.09061512351036072 |
| keywords[11].display_name | Medicine |
| keywords[12].id | https://openalex.org/keywords/seismology |
| keywords[12].score | 0.06147852540016174 |
| keywords[12].display_name | Seismology |
| language | en |
| locations[0].id | mag:3049669010 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | |
| locations[0].version | submittedVersion |
| locations[0].raw_type | |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | arXiv (Cornell University) |
| locations[0].landing_page_url | http://export.arxiv.org/pdf/2005.02367 |
| authorships[0].author.id | https://openalex.org/A5083675499 |
| authorships[0].author.orcid | https://orcid.org/0000-0001-7021-4627 |
| authorships[0].author.display_name | Ting-Hao Huang |
| authorships[0].countries | US |
| authorships[0].affiliations[0].institution_ids | https://openalex.org/I130769515 |
| authorships[0].affiliations[0].raw_affiliation_string | Pennsylvania State University |
| authorships[0].institutions[0].id | https://openalex.org/I130769515 |
| authorships[0].institutions[0].ror | https://ror.org/04p491231 |
| authorships[0].institutions[0].type | education |
| authorships[0].institutions[0].lineage | https://openalex.org/I130769515 |
| authorships[0].institutions[0].country_code | US |
| authorships[0].institutions[0].display_name | Pennsylvania State University |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Huang T |
| authorships[0].is_corresponding | False |
| authorships[0].raw_affiliation_strings | Pennsylvania State University |
| authorships[1].author.id | https://openalex.org/A5102729368 |
| authorships[1].author.orcid | https://orcid.org/0009-0001-6736-9959 |
| authorships[1].author.display_name | Chieh-Yang Huang |
| authorships[1].countries | US |
| authorships[1].affiliations[0].institution_ids | https://openalex.org/I130769515 |
| authorships[1].affiliations[0].raw_affiliation_string | Pennsylvania State University |
| authorships[1].institutions[0].id | https://openalex.org/I130769515 |
| authorships[1].institutions[0].ror | https://ror.org/04p491231 |
| authorships[1].institutions[0].type | education |
| authorships[1].institutions[0].lineage | https://openalex.org/I130769515 |
| authorships[1].institutions[0].country_code | US |
| authorships[1].institutions[0].display_name | Pennsylvania State University |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Huang C |
| authorships[1].is_corresponding | False |
| authorships[1].raw_affiliation_strings | Pennsylvania State University |
| authorships[2].author.id | https://openalex.org/A5029595501 |
| authorships[2].author.orcid | |
| authorships[2].author.display_name | Ding Cc |
| authorships[2].countries | US |
| authorships[2].affiliations[0].institution_ids | https://openalex.org/I130769515 |
| authorships[2].affiliations[0].raw_affiliation_string | Pennsylvania State University |
| authorships[2].institutions[0].id | https://openalex.org/I130769515 |
| authorships[2].institutions[0].ror | https://ror.org/04p491231 |
| authorships[2].institutions[0].type | education |
| authorships[2].institutions[0].lineage | https://openalex.org/I130769515 |
| authorships[2].institutions[0].country_code | US |
| authorships[2].institutions[0].display_name | Pennsylvania State University |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Ding Cc |
| authorships[2].is_corresponding | False |
| authorships[2].raw_affiliation_strings | Pennsylvania State University |
| authorships[3].author.id | https://openalex.org/A5113329699 |
| authorships[3].author.orcid | |
| authorships[3].author.display_name | Hsu Y |
| authorships[3].countries | US |
| authorships[3].affiliations[0].institution_ids | https://openalex.org/I180670191 |
| authorships[3].affiliations[0].raw_affiliation_string | §University of California, San Francisco |
| authorships[3].institutions[0].id | https://openalex.org/I180670191 |
| authorships[3].institutions[0].ror | https://ror.org/043mz5j54 |
| authorships[3].institutions[0].type | education |
| authorships[3].institutions[0].lineage | https://openalex.org/I180670191 |
| authorships[3].institutions[0].country_code | US |
| authorships[3].institutions[0].display_name | University of California, San Francisco |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Hsu Y |
| authorships[3].is_corresponding | False |
| authorships[3].raw_affiliation_strings | §University of California, San Francisco |
| authorships[4].author.id | https://openalex.org/A5027012255 |
| authorships[4].author.orcid | |
| authorships[4].author.display_name | Giles Cl |
| authorships[4].countries | US |
| authorships[4].affiliations[0].institution_ids | https://openalex.org/I74973139 |
| authorships[4].affiliations[0].raw_affiliation_string | Carnegie Mellon University |
| authorships[4].institutions[0].id | https://openalex.org/I74973139 |
| authorships[4].institutions[0].ror | https://ror.org/05x2bcf33 |
| authorships[4].institutions[0].type | education |
| authorships[4].institutions[0].lineage | https://openalex.org/I74973139 |
| authorships[4].institutions[0].country_code | US |
| authorships[4].institutions[0].display_name | Carnegie Mellon University |
| authorships[4].author_position | last |
| authorships[4].raw_author_name | Giles Cl |
| authorships[4].is_corresponding | False |
| authorships[4].raw_affiliation_strings | Carnegie Mellon University |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | http://export.arxiv.org/pdf/2005.02367 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2020-08-21T00:00:00 |
| display_name | CODA-19: Using a Non-Expert Crowd to Annotate Research Aspects on 10,000+ Abstracts in the COVID-19 Open Research Dataset |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-10-10T17:16:08.811792 |
| primary_topic.id | https://openalex.org/T11147 |
| primary_topic.field.id | https://openalex.org/fields/33 |
| primary_topic.field.display_name | Social Sciences |
| primary_topic.score | 0.9901000261306763 |
| primary_topic.domain.id | https://openalex.org/domains/2 |
| primary_topic.domain.display_name | Social Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/3312 |
| primary_topic.subfield.display_name | Sociology and Political Science |
| primary_topic.display_name | Misinformation and Its Impacts |
| related_works | https://openalex.org/W3022014289, https://openalex.org/W3020786614, https://openalex.org/W3175973996, https://openalex.org/W1984439870, https://openalex.org/W2898991253, https://openalex.org/W3177058128, https://openalex.org/W3106074546, https://openalex.org/W2402617095, https://openalex.org/W2558737512, https://openalex.org/W3140526452, https://openalex.org/W1761766145, https://openalex.org/W3154048149, https://openalex.org/W2947644469, https://openalex.org/W2093044076, https://openalex.org/W2917014163, https://openalex.org/W3162331882, https://openalex.org/W2591689480, https://openalex.org/W3159259047, https://openalex.org/W2400083450, https://openalex.org/W3185930382 |
| cited_by_count | 11 |
| counts_by_year[0].year | 2022 |
| counts_by_year[0].cited_by_count | 2 |
| counts_by_year[1].year | 2021 |
| counts_by_year[1].cited_by_count | 6 |
| counts_by_year[2].year | 2020 |
| counts_by_year[2].cited_by_count | 3 |
| locations_count | 1 |
| best_oa_location.id | mag:3049669010 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | arXiv (Cornell University) |
| best_oa_location.landing_page_url | http://export.arxiv.org/pdf/2005.02367 |
| primary_location.id | mag:3049669010 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | |
| primary_location.version | submittedVersion |
| primary_location.raw_type | |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | arXiv (Cornell University) |
| primary_location.landing_page_url | http://export.arxiv.org/pdf/2005.02367 |
| publication_date | 2020-08-17 |
| publication_year | 2020 |
| referenced_works | https://openalex.org/W2963341956, https://openalex.org/W1982022486, https://openalex.org/W2952174148, https://openalex.org/W1969605785, https://openalex.org/W2591407402, https://openalex.org/W2442495973, https://openalex.org/W2517720908, https://openalex.org/W2163471296, https://openalex.org/W1550206324, https://openalex.org/W2186614768, https://openalex.org/W2142560900, https://openalex.org/W2067284187, https://openalex.org/W2579538068, https://openalex.org/W2969439587, https://openalex.org/W1232149359, https://openalex.org/W2757521750, https://openalex.org/W2139531859, https://openalex.org/W2147994374, https://openalex.org/W2006802446, https://openalex.org/W2953126493, https://openalex.org/W2788496822, https://openalex.org/W2898725686, https://openalex.org/W1515842608, https://openalex.org/W2121678130, https://openalex.org/W3020786614, https://openalex.org/W2402929825, https://openalex.org/W2766801974, https://openalex.org/W212864694, https://openalex.org/W2101234009, https://openalex.org/W2993277840, https://openalex.org/W2970771982, https://openalex.org/W2106101484, https://openalex.org/W2140336868, https://openalex.org/W1583385464, https://openalex.org/W2047351079, https://openalex.org/W2116705992, https://openalex.org/W2032570581, https://openalex.org/W2806882588, https://openalex.org/W2163361328, https://openalex.org/W1595345130, https://openalex.org/W2945165791, https://openalex.org/W2252173290, https://openalex.org/W2035682999, https://openalex.org/W2955594819, https://openalex.org/W2885925344, https://openalex.org/W2337191715, https://openalex.org/W2741949568, https://openalex.org/W2949541494, https://openalex.org/W2890308578, https://openalex.org/W1968466431, https://openalex.org/W2962903510, https://openalex.org/W3017169475, https://openalex.org/W170711724, https://openalex.org/W3101836065, https://openalex.org/W205532704, https://openalex.org/W1840435438, https://openalex.org/W2148008364 |
| referenced_works_count | 57 |
| abstract_inverted_index.a | 4, 139 |
| abstract_inverted_index.10 | 39 |
| abstract_inverted_index.We | 136 |
| abstract_inverted_index.an | 89 |
| abstract_inverted_index.as | 123 |
| abstract_inverted_index.at | 146 |
| abstract_inverted_index.be | 134, 143 |
| abstract_inverted_index.by | 30, 54, 64 |
| abstract_inverted_index.in | 21 |
| abstract_inverted_index.is | 80 |
| abstract_inverted_index.of | 17, 48, 91, 126 |
| abstract_inverted_index.to | 46, 82, 95, 148 |
| abstract_inverted_index.248 | 31 |
| abstract_inverted_index.The | 67 |
| abstract_inverted_index.and | 14, 41, 58, 75, 113, 120 |
| abstract_inverted_index.but | 129 |
| abstract_inverted_index.can | 133, 142 |
| abstract_inverted_index.the | 9, 22, 59, 73, 76, 96, 101, 115, 124, 150 |
| abstract_inverted_index.was | 28, 52, 105 |
| abstract_inverted_index.Each | 50 |
| abstract_inverted_index.Open | 24 |
| abstract_inverted_index.This | 0 |
| abstract_inverted_index.Turk | 37 |
| abstract_inverted_index.also | 121 |
| abstract_inverted_index.from | 34 |
| abstract_inverted_index.have | 88 |
| abstract_inverted_index.help | 110 |
| abstract_inverted_index.join | 149 |
| abstract_inverted_index.nine | 55 |
| abstract_inverted_index.that | 7, 47, 138 |
| abstract_inverted_index.were | 62 |
| abstract_inverted_index.when | 93 |
| abstract_inverted_index.82.2% | 92 |
| abstract_inverted_index.Other | 15 |
| abstract_inverted_index.codes | 8 |
| abstract_inverted_index.crowd | 32, 74, 141 |
| abstract_inverted_index.days, | 40 |
| abstract_inverted_index.fight | 151 |
| abstract_inverted_index.final | 60 |
| abstract_inverted_index.human | 108 |
| abstract_inverted_index.paper | 1 |
| abstract_inverted_index.scale | 147 |
| abstract_inverted_index.serve | 122 |
| abstract_inverted_index.slow. | 135 |
| abstract_inverted_index.vote. | 66 |
| abstract_inverted_index.while | 100 |
| abstract_inverted_index.10,966 | 18 |
| abstract_inverted_index.85.0%. | 106 |
| abstract_inverted_index.AI/NLP | 127 |
| abstract_inverted_index.Amazon | 35 |
| abstract_inverted_index.access | 112 |
| abstract_inverted_index.expert | 78, 131 |
| abstract_inverted_index.kappa) | 71 |
| abstract_inverted_index.labels | 61, 87 |
| abstract_inverted_index.within | 38 |
| abstract_inverted_index.(0.741) | 79 |
| abstract_inverted_index.CODA-19 | 27 |
| abstract_inverted_index.English | 19 |
| abstract_inverted_index.Method, | 12 |
| abstract_inverted_index.against | 152 |
| abstract_inverted_index.battery | 125 |
| abstract_inverted_index.between | 72, 103 |
| abstract_inverted_index.created | 29 |
| abstract_inverted_index.dataset | 6 |
| abstract_inverted_index.experts | 104 |
| abstract_inverted_index.labels, | 99 |
| abstract_inverted_index.quality | 44 |
| abstract_inverted_index.rapidly | 116, 144 |
| abstract_inverted_index.workers | 33 |
| abstract_inverted_index.(0.788). | 85 |
| abstract_inverted_index.(Cohen's | 70 |
| abstract_inverted_index.CODA-19, | 3 |
| abstract_inverted_index.COVID-19 | 23 |
| abstract_inverted_index.Dataset. | 26 |
| abstract_inverted_index.Purpose, | 11 |
| abstract_inverted_index.Reliable | 107 |
| abstract_inverted_index.Research | 25 |
| abstract_inverted_index.abstract | 51 |
| abstract_inverted_index.accuracy | 90, 102 |
| abstract_inverted_index.achieved | 42 |
| abstract_inverted_index.acquired | 63 |
| abstract_inverted_index.compared | 94 |
| abstract_inverted_index.employed | 145 |
| abstract_inverted_index.expert's | 98 |
| abstract_inverted_index.experts. | 49 |
| abstract_inverted_index.labeling | 43 |
| abstract_inverted_index.majority | 65 |
| abstract_inverted_index.sections | 16 |
| abstract_inverted_index.workers, | 57 |
| abstract_inverted_index.CODA-19's | 86 |
| abstract_inverted_index.COVID-19. | 153 |
| abstract_inverted_index.abstracts | 20 |
| abstract_inverted_index.agreement | 69, 84 |
| abstract_inverted_index.annotated | 53 |
| abstract_inverted_index.different | 56 |
| abstract_inverted_index.integrate | 114 |
| abstract_inverted_index.obtaining | 130 |
| abstract_inverted_index.research, | 128 |
| abstract_inverted_index.Mechanical | 36 |
| abstract_inverted_index.biomedical | 77, 97 |
| abstract_inverted_index.comparable | 45, 81 |
| abstract_inverted_index.introduces | 2 |
| abstract_inverted_index.non-expert | 140 |
| abstract_inverted_index.scientists | 111 |
| abstract_inverted_index.Background, | 10 |
| abstract_inverted_index.annotations | 109, 132 |
| abstract_inverted_index.coronavirus | 118 |
| abstract_inverted_index.literature, | 119 |
| abstract_inverted_index.accelerating | 117 |
| abstract_inverted_index.demonstrated | 137 |
| abstract_inverted_index.inter-expert | 83 |
| abstract_inverted_index.human-annotated | 5 |
| abstract_inverted_index.inter-annotator | 68 |
| abstract_inverted_index.Finding/Contribution, | 13 |
| cited_by_percentile_year | |
| countries_distinct_count | 1 |
| institutions_distinct_count | 5 |
| citation_normalized_percentile |