Detecting Crowdsourced Test Report Consistency for Mobile Apps with Deep Image Understanding and Text Analysis. Article Swipe
Shengcheng Yu
,
Chunrong Fang
,
Kai Mei
,
Yexiao Yun
,
Zhenfei Cao
,
Zhihao Cao
,
Zhenyu Chen
·
YOU?
·
· 2021
· Open Access
·
YOU?
·
· 2021
· Open Access
·
Crowdsourced testing, as a distinct testing paradigm, has attracted much attention in software testing, especially in mobile application (app) testing field. Compared with in-house testing, crowdsourced testing outperforms because it utilize the diverse testing environments of different crowdworkers faced with the mobile testing fragmentation problem. However, crowdsourced testing also brings some problem. The crowdworkers involved are with different expertise, and they are not professional testers. Therefore, the reports they may submit are numerous and with uneven quality. App developers have to distinguish high-quality reports from low-quality ones to help the bug revealing and fixing. Some crowdworkers would submit inconsistent test reports, which means the textual descriptions are not focusing on the attached bug occurring screenshots. Such reports cause the waste on both time and human resources of app developing and testing. To solve such a problem, we propose ReCoDe in this paper, which is designed to detect the consistency of crowdsourced test reports via deep image-and-text fusion understanding. First, according to a pre-conducted survey, ReCoDe classifies the crowdsourced test reports into 10 categories, which covers the vast majority of reported problems in the test reports. Then, for each category of bugs, we have distinct processing models. The models have a deep fusion understanding on both image information and textual descriptions. We also have conducted an experiment to evaluate ReCoDe, and the results show the effectiveness of ReCoDe to detect consistency crowdsourced test reports.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- https://arxiv.org/pdf/2108.07401v2
- OA Status
- green
- Cited By
- 3
- References
- 38
- Related Works
- 20
- OpenAlex ID
- https://openalex.org/W3195363835
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W3195363835Canonical identifier for this work in OpenAlex
- Title
-
Detecting Crowdsourced Test Report Consistency for Mobile Apps with Deep Image Understanding and Text Analysis.Work title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2021Year of publication
- Publication date
-
2021-08-17Full publication date if available
- Authors
-
Shengcheng Yu, Chunrong Fang, Kai Mei, Yexiao Yun, Zhenfei Cao, Zhihao Cao, Zhenyu ChenList of authors in order
- Landing page
-
https://arxiv.org/pdf/2108.07401v2Publisher landing page
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2108.07401v2Direct OA link when available
- Concepts
-
Computer science, Consistency (knowledge bases), Crowdsourcing, Quality (philosophy), Test (biology), Information retrieval, Data science, Mobile apps, Annotation, Artificial intelligence, World Wide Web, Data mining, Philosophy, Biology, Epistemology, PaleontologyTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
3Total citation count in OpenAlex
- Citations by year (recent)
-
2025: 1, 2023: 1, 2022: 1Per-year citation counts (last 5 years)
- References (count)
-
38Number of works referenced by this work
- Related works (count)
-
20Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W3195363835 |
|---|---|
| doi | |
| ids.mag | 3195363835 |
| ids.openalex | https://openalex.org/W3195363835 |
| fwci | |
| type | preprint |
| title | Detecting Crowdsourced Test Report Consistency for Mobile Apps with Deep Image Understanding and Text Analysis. |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10743 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9994999766349792 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1712 |
| topics[0].subfield.display_name | Software |
| topics[0].display_name | Software Testing and Debugging Techniques |
| topics[1].id | https://openalex.org/T10260 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9980999827384949 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1710 |
| topics[1].subfield.display_name | Information Systems |
| topics[1].display_name | Software Engineering Research |
| topics[2].id | https://openalex.org/T11241 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.996999979019165 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1711 |
| topics[2].subfield.display_name | Signal Processing |
| topics[2].display_name | Advanced Malware Detection Techniques |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C41008148 |
| concepts[0].level | 0 |
| concepts[0].score | 0.7592182159423828 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[0].display_name | Computer science |
| concepts[1].id | https://openalex.org/C2776436953 |
| concepts[1].level | 2 |
| concepts[1].score | 0.7313987612724304 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q5163215 |
| concepts[1].display_name | Consistency (knowledge bases) |
| concepts[2].id | https://openalex.org/C62230096 |
| concepts[2].level | 2 |
| concepts[2].score | 0.7284343242645264 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q275969 |
| concepts[2].display_name | Crowdsourcing |
| concepts[3].id | https://openalex.org/C2779530757 |
| concepts[3].level | 2 |
| concepts[3].score | 0.5597885847091675 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q1207505 |
| concepts[3].display_name | Quality (philosophy) |
| concepts[4].id | https://openalex.org/C2777267654 |
| concepts[4].level | 2 |
| concepts[4].score | 0.5391021966934204 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q3519023 |
| concepts[4].display_name | Test (biology) |
| concepts[5].id | https://openalex.org/C23123220 |
| concepts[5].level | 1 |
| concepts[5].score | 0.5043627023696899 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q816826 |
| concepts[5].display_name | Information retrieval |
| concepts[6].id | https://openalex.org/C2522767166 |
| concepts[6].level | 1 |
| concepts[6].score | 0.44389185309410095 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q2374463 |
| concepts[6].display_name | Data science |
| concepts[7].id | https://openalex.org/C2988145974 |
| concepts[7].level | 2 |
| concepts[7].score | 0.43667858839035034 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q620615 |
| concepts[7].display_name | Mobile apps |
| concepts[8].id | https://openalex.org/C2776321320 |
| concepts[8].level | 2 |
| concepts[8].score | 0.4176808297634125 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q857525 |
| concepts[8].display_name | Annotation |
| concepts[9].id | https://openalex.org/C154945302 |
| concepts[9].level | 1 |
| concepts[9].score | 0.3633095324039459 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[9].display_name | Artificial intelligence |
| concepts[10].id | https://openalex.org/C136764020 |
| concepts[10].level | 1 |
| concepts[10].score | 0.35125046968460083 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q466 |
| concepts[10].display_name | World Wide Web |
| concepts[11].id | https://openalex.org/C124101348 |
| concepts[11].level | 1 |
| concepts[11].score | 0.3356934189796448 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q172491 |
| concepts[11].display_name | Data mining |
| concepts[12].id | https://openalex.org/C138885662 |
| concepts[12].level | 0 |
| concepts[12].score | 0.0 |
| concepts[12].wikidata | https://www.wikidata.org/wiki/Q5891 |
| concepts[12].display_name | Philosophy |
| concepts[13].id | https://openalex.org/C86803240 |
| concepts[13].level | 0 |
| concepts[13].score | 0.0 |
| concepts[13].wikidata | https://www.wikidata.org/wiki/Q420 |
| concepts[13].display_name | Biology |
| concepts[14].id | https://openalex.org/C111472728 |
| concepts[14].level | 1 |
| concepts[14].score | 0.0 |
| concepts[14].wikidata | https://www.wikidata.org/wiki/Q9471 |
| concepts[14].display_name | Epistemology |
| concepts[15].id | https://openalex.org/C151730666 |
| concepts[15].level | 1 |
| concepts[15].score | 0.0 |
| concepts[15].wikidata | https://www.wikidata.org/wiki/Q7205 |
| concepts[15].display_name | Paleontology |
| keywords[0].id | https://openalex.org/keywords/computer-science |
| keywords[0].score | 0.7592182159423828 |
| keywords[0].display_name | Computer science |
| keywords[1].id | https://openalex.org/keywords/consistency |
| keywords[1].score | 0.7313987612724304 |
| keywords[1].display_name | Consistency (knowledge bases) |
| keywords[2].id | https://openalex.org/keywords/crowdsourcing |
| keywords[2].score | 0.7284343242645264 |
| keywords[2].display_name | Crowdsourcing |
| keywords[3].id | https://openalex.org/keywords/quality |
| keywords[3].score | 0.5597885847091675 |
| keywords[3].display_name | Quality (philosophy) |
| keywords[4].id | https://openalex.org/keywords/test |
| keywords[4].score | 0.5391021966934204 |
| keywords[4].display_name | Test (biology) |
| keywords[5].id | https://openalex.org/keywords/information-retrieval |
| keywords[5].score | 0.5043627023696899 |
| keywords[5].display_name | Information retrieval |
| keywords[6].id | https://openalex.org/keywords/data-science |
| keywords[6].score | 0.44389185309410095 |
| keywords[6].display_name | Data science |
| keywords[7].id | https://openalex.org/keywords/mobile-apps |
| keywords[7].score | 0.43667858839035034 |
| keywords[7].display_name | Mobile apps |
| keywords[8].id | https://openalex.org/keywords/annotation |
| keywords[8].score | 0.4176808297634125 |
| keywords[8].display_name | Annotation |
| keywords[9].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[9].score | 0.3633095324039459 |
| keywords[9].display_name | Artificial intelligence |
| keywords[10].id | https://openalex.org/keywords/world-wide-web |
| keywords[10].score | 0.35125046968460083 |
| keywords[10].display_name | World Wide Web |
| keywords[11].id | https://openalex.org/keywords/data-mining |
| keywords[11].score | 0.3356934189796448 |
| keywords[11].display_name | Data mining |
| language | en |
| locations[0].id | mag:3195363835 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | |
| locations[0].version | submittedVersion |
| locations[0].raw_type | |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | arXiv (Cornell University) |
| locations[0].landing_page_url | https://arxiv.org/pdf/2108.07401v2 |
| authorships[0].author.id | https://openalex.org/A5101519109 |
| authorships[0].author.orcid | https://orcid.org/0000-0003-4640-8637 |
| authorships[0].author.display_name | Shengcheng Yu |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Shengcheng Yu |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5075174750 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-9930-7111 |
| authorships[1].author.display_name | Chunrong Fang |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Chunrong Fang |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5114526266 |
| authorships[2].author.orcid | https://orcid.org/0000-0001-6231-3182 |
| authorships[2].author.display_name | Kai Mei |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Kai Mei |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5039912987 |
| authorships[3].author.orcid | https://orcid.org/0009-0001-3903-9872 |
| authorships[3].author.display_name | Yexiao Yun |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Yexiao Yun |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5059341940 |
| authorships[4].author.orcid | https://orcid.org/0009-0000-9679-3831 |
| authorships[4].author.display_name | Zhenfei Cao |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Zhenfei Cao |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5114526255 |
| authorships[5].author.orcid | https://orcid.org/0009-0009-4707-2099 |
| authorships[5].author.display_name | Zhihao Cao |
| authorships[5].author_position | middle |
| authorships[5].raw_author_name | Zhihao Cao |
| authorships[5].is_corresponding | False |
| authorships[6].author.id | https://openalex.org/A5100422935 |
| authorships[6].author.orcid | https://orcid.org/0000-0002-9592-7022 |
| authorships[6].author.display_name | Zhenyu Chen |
| authorships[6].author_position | last |
| authorships[6].raw_author_name | Zhenyu Chen |
| authorships[6].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2108.07401v2 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Detecting Crowdsourced Test Report Consistency for Mobile Apps with Deep Image Understanding and Text Analysis. |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-10-10T17:16:08.811792 |
| primary_topic.id | https://openalex.org/T10743 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9994999766349792 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1712 |
| primary_topic.subfield.display_name | Software |
| primary_topic.display_name | Software Testing and Debugging Techniques |
| related_works | https://openalex.org/W3131806503, https://openalex.org/W2290009368, https://openalex.org/W2605032615, https://openalex.org/W2093044076, https://openalex.org/W2994747757, https://openalex.org/W3187382105, https://openalex.org/W3010672471, https://openalex.org/W2969439587, https://openalex.org/W1916584426, https://openalex.org/W3099488617, https://openalex.org/W3198191527, https://openalex.org/W3086623535, https://openalex.org/W3136526765, https://openalex.org/W3134191702, https://openalex.org/W2900867162, https://openalex.org/W1944317963, https://openalex.org/W2991759854, https://openalex.org/W2888631336, https://openalex.org/W2914579363, https://openalex.org/W3194717064 |
| cited_by_count | 3 |
| counts_by_year[0].year | 2025 |
| counts_by_year[0].cited_by_count | 1 |
| counts_by_year[1].year | 2023 |
| counts_by_year[1].cited_by_count | 1 |
| counts_by_year[2].year | 2022 |
| counts_by_year[2].cited_by_count | 1 |
| locations_count | 1 |
| best_oa_location.id | mag:3195363835 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | arXiv (Cornell University) |
| best_oa_location.landing_page_url | https://arxiv.org/pdf/2108.07401v2 |
| primary_location.id | mag:3195363835 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | |
| primary_location.version | submittedVersion |
| primary_location.raw_type | |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | arXiv (Cornell University) |
| primary_location.landing_page_url | https://arxiv.org/pdf/2108.07401v2 |
| publication_date | 2021-08-17 |
| publication_year | 2021 |
| referenced_works | https://openalex.org/W2552110825, https://openalex.org/W2953461273, https://openalex.org/W3089375544, https://openalex.org/W2095873049, https://openalex.org/W3082097605, https://openalex.org/W2770141444, https://openalex.org/W2513201734, https://openalex.org/W3008024462, https://openalex.org/W2139270835, https://openalex.org/W2802838214, https://openalex.org/W3160987621, https://openalex.org/W3048516901, https://openalex.org/W2995460200, https://openalex.org/W1977603982, https://openalex.org/W2130146200, https://openalex.org/W2963341956, https://openalex.org/W3103843169, https://openalex.org/W3160414745, https://openalex.org/W3122319236, https://openalex.org/W2970608575, https://openalex.org/W2084236619, https://openalex.org/W2514090203, https://openalex.org/W3181736594, https://openalex.org/W2241093273, https://openalex.org/W2963295463, https://openalex.org/W2794908093, https://openalex.org/W1024906986, https://openalex.org/W1999910737, https://openalex.org/W2522973809, https://openalex.org/W2744629383, https://openalex.org/W1981109290, https://openalex.org/W2091646623, https://openalex.org/W3000390917, https://openalex.org/W3163060204, https://openalex.org/W2963403868, https://openalex.org/W2147527908, https://openalex.org/W1998916198, https://openalex.org/W2754054654 |
| referenced_works_count | 38 |
| abstract_inverted_index.a | 3, 134, 161, 199 |
| abstract_inverted_index.10 | 171 |
| abstract_inverted_index.We | 210 |
| abstract_inverted_index.an | 214 |
| abstract_inverted_index.as | 2 |
| abstract_inverted_index.in | 11, 15, 139, 181 |
| abstract_inverted_index.is | 143 |
| abstract_inverted_index.it | 29 |
| abstract_inverted_index.of | 35, 126, 149, 178, 189, 225 |
| abstract_inverted_index.on | 109, 120, 203 |
| abstract_inverted_index.to | 80, 87, 145, 160, 216, 227 |
| abstract_inverted_index.we | 136, 191 |
| abstract_inverted_index.App | 77 |
| abstract_inverted_index.The | 52, 196 |
| abstract_inverted_index.and | 59, 73, 92, 123, 129, 207, 219 |
| abstract_inverted_index.app | 127 |
| abstract_inverted_index.are | 55, 61, 71, 106 |
| abstract_inverted_index.bug | 90, 112 |
| abstract_inverted_index.for | 186 |
| abstract_inverted_index.has | 7 |
| abstract_inverted_index.may | 69 |
| abstract_inverted_index.not | 62, 107 |
| abstract_inverted_index.the | 31, 40, 66, 89, 103, 110, 118, 147, 166, 175, 182, 220, 223 |
| abstract_inverted_index.via | 153 |
| abstract_inverted_index. To | 131 |
| abstract_inverted_index.Some | 94 |
| abstract_inverted_index.Such | 115 |
| abstract_inverted_index.also | 48, 211 |
| abstract_inverted_index.both | 121, 204 |
| abstract_inverted_index.deep | 154, 200 |
| abstract_inverted_index.each | 187 |
| abstract_inverted_index.from | 84 |
| abstract_inverted_index.have | 79, 192, 198, 212 |
| abstract_inverted_index.help | 88 |
| abstract_inverted_index.into | 170 |
| abstract_inverted_index.much | 9 |
| abstract_inverted_index.ones | 86 |
| abstract_inverted_index.show | 222 |
| abstract_inverted_index.some | 50 |
| abstract_inverted_index.such | 133 |
| abstract_inverted_index.test | 99, 151, 168, 183, 231 |
| abstract_inverted_index.they | 60, 68 |
| abstract_inverted_index.this | 140 |
| abstract_inverted_index.time | 122 |
| abstract_inverted_index.vast | 176 |
| abstract_inverted_index.with | 22, 39, 56, 74 |
| abstract_inverted_index.(app) | 18 |
| abstract_inverted_index.Then, | 185 |
| abstract_inverted_index.bugs, | 190 |
| abstract_inverted_index.cause | 117 |
| abstract_inverted_index.faced | 38 |
| abstract_inverted_index.human | 124 |
| abstract_inverted_index.image | 205 |
| abstract_inverted_index.means | 102 |
| abstract_inverted_index.solve | 132 |
| abstract_inverted_index.waste | 119 |
| abstract_inverted_index.which | 101, 142, 173 |
| abstract_inverted_index.would | 96 |
| abstract_inverted_index.First, | 158 |
| abstract_inverted_index.ReCoDe | 138, 164, 226 |
| abstract_inverted_index.brings | 49 |
| abstract_inverted_index.covers | 174 |
| abstract_inverted_index.detect | 146, 228 |
| abstract_inverted_index.field. | 20 |
| abstract_inverted_index.fusion | 156, 201 |
| abstract_inverted_index.mobile | 16, 41 |
| abstract_inverted_index.models | 197 |
| abstract_inverted_index.paper, | 141 |
| abstract_inverted_index.submit | 70, 97 |
| abstract_inverted_index.uneven | 75 |
| abstract_inverted_index.ReCoDe, | 218 |
| abstract_inverted_index.because | 28 |
| abstract_inverted_index.diverse | 32 |
| abstract_inverted_index.fixing. | 93 |
| abstract_inverted_index.models. | 195 |
| abstract_inverted_index.propose | 137 |
| abstract_inverted_index.reports | 67, 83, 116, 152, 169 |
| abstract_inverted_index.results | 221 |
| abstract_inverted_index.survey, | 163 |
| abstract_inverted_index.testing | 5, 19, 26, 33, 42, 47 |
| abstract_inverted_index.textual | 104, 208 |
| abstract_inverted_index.utilize | 30 |
| abstract_inverted_index.Compared | 21 |
| abstract_inverted_index.However, | 45 |
| abstract_inverted_index.attached | 111 |
| abstract_inverted_index.category | 188 |
| abstract_inverted_index.designed | 144 |
| abstract_inverted_index.distinct | 4, 193 |
| abstract_inverted_index.evaluate | 217 |
| abstract_inverted_index.focusing | 108 |
| abstract_inverted_index.in-house | 23 |
| abstract_inverted_index.involved | 54 |
| abstract_inverted_index.majority | 177 |
| abstract_inverted_index.numerous | 72 |
| abstract_inverted_index.problem, | 135 |
| abstract_inverted_index.problem. | 44, 51 |
| abstract_inverted_index.problems | 180 |
| abstract_inverted_index.quality. | 76 |
| abstract_inverted_index.reported | 179 |
| abstract_inverted_index.reports, | 100 |
| abstract_inverted_index.reports. | 184, 232 |
| abstract_inverted_index.software | 12 |
| abstract_inverted_index.testers. | 64 |
| abstract_inverted_index.testing, | 1, 13, 24 |
| abstract_inverted_index.testing. | 130 |
| abstract_inverted_index.according | 159 |
| abstract_inverted_index.attention | 10 |
| abstract_inverted_index.attracted | 8 |
| abstract_inverted_index.conducted | 213 |
| abstract_inverted_index.different | 36, 57 |
| abstract_inverted_index.occurring | 113 |
| abstract_inverted_index.paradigm, | 6 |
| abstract_inverted_index.resources | 125 |
| abstract_inverted_index.revealing | 91 |
| abstract_inverted_index.Therefore, | 65 |
| abstract_inverted_index.classifies | 165 |
| abstract_inverted_index.developers | 78 |
| abstract_inverted_index.developing | 128 |
| abstract_inverted_index.especially | 14 |
| abstract_inverted_index.experiment | 215 |
| abstract_inverted_index.expertise, | 58 |
| abstract_inverted_index.processing | 194 |
| abstract_inverted_index.application | 17 |
| abstract_inverted_index.categories, | 172 |
| abstract_inverted_index.consistency | 148, 229 |
| abstract_inverted_index.distinguish | 81 |
| abstract_inverted_index.information | 206 |
| abstract_inverted_index.low-quality | 85 |
| abstract_inverted_index.outperforms | 27 |
| abstract_inverted_index.Crowdsourced | 0 |
| abstract_inverted_index.crowdsourced | 25, 46, 150, 167, 230 |
| abstract_inverted_index.crowdworkers | 37, 53, 95 |
| abstract_inverted_index.descriptions | 105 |
| abstract_inverted_index.environments | 34 |
| abstract_inverted_index.high-quality | 82 |
| abstract_inverted_index.inconsistent | 98 |
| abstract_inverted_index.professional | 63 |
| abstract_inverted_index.screenshots. | 114 |
| abstract_inverted_index.descriptions. | 209 |
| abstract_inverted_index.effectiveness | 224 |
| abstract_inverted_index.fragmentation | 43 |
| abstract_inverted_index.pre-conducted | 162 |
| abstract_inverted_index.understanding | 202 |
| abstract_inverted_index.image-and-text | 155 |
| abstract_inverted_index.understanding. | 157 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 7 |
| citation_normalized_percentile |