Knowing When to Stop: Final Results vs. Work Involved in Systematic Review Database Searching Article Swipe
Jennifer S. Walker
,
Rachael Posey
,
Karen Crowell
·
YOU?
·
· 2019
· Open Access
·
· DOI: https://doi.org/10.17615/36ey-p097
YOU?
·
· 2019
· Open Access
·
· DOI: https://doi.org/10.17615/36ey-p097
Objectives: To examine the relationship between the amount of work involved for the systematic review (SR) team for each additional database searched in an SR versus the number of unique articles included in the final review that were retrieved from each additional database searched. Methods: We reviewed SRs and meta-analyses published in JAMA, Lancet, and Annals of Internal Medicine from January 2012 to December 2015, selecting those journals as high impact factor journals that have published over more than 50 SRs per year since 2012. We required that all papers in our analysis report a complete search strategy, with the total number of citations retrieved from each database and a complete list of articles included in the final analysis. We then used lists of included journals from each database to determine in which database(s) each of the included articles were indexed, and we calculated the percentage of articles included in each SR that could be located by searching each database. We then compared the number of results added by searching each additional database to the total number of papers added to the review phase to estimate the amount of work required for each additional paper identified. Results: Ninety-seven SRs met our inclusion criteria. These SRs included an average of 48.13 journal articles and searched an average of 4.43 databases each. Of these, the journal articles included in 16 SRs could all be found in one database; the articles included in 58 SRs could all be found in two databases. For 20 SRs, all included articles could be found in three databases, and the remaining three SRs included articles that could be found in four databases. For 96 SRs, over 90% of articles could be found in two databases. Conclusions: In total, 99% of articles included in each SR were found in two databases, with the majority being found in PubMed/MEDLINE, Embase, or Cochrane. SRs that found articles in three or more databases screened an additional 923 records in order to find one additional included article, plus an additional 2410 records from databases that did not return any additional included articles, adding an average of 756 hours of work to each SR. SRs for which articles were found in four databases screened an average of 1440 records in order to find one additional included article, plus an additional 8963 records from databases that did not return any additional included articles.
Related Topics
Concepts
Metadata
- Type
- article
- Language
- en
- Landing Page
- https://cdr.lib.unc.edu/concern/scholarly_works/v692t8717
- OA Status
- green
- Cited By
- 2
- Related Works
- 14
- OpenAlex ID
- https://openalex.org/W2949477478
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W2949477478Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.17615/36ey-p097Digital Object Identifier
- Title
-
Knowing When to Stop: Final Results vs. Work Involved in Systematic Review Database SearchingWork title
- Type
-
articleOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2019Year of publication
- Publication date
-
2019-08-17Full publication date if available
- Authors
-
Jennifer S. Walker, Rachael Posey, Karen CrowellList of authors in order
- Landing page
-
https://cdr.lib.unc.edu/concern/scholarly_works/v692t8717Publisher landing page
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://doi.org/10.17615/36ey-p097Direct OA link when available
- Concepts
-
Database, Work (physics), Computer science, Information retrieval, Engineering, Mechanical engineeringTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
2Total citation count in OpenAlex
- Citations by year (recent)
-
2021: 2Per-year citation counts (last 5 years)
- Related works (count)
-
14Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W2949477478 |
|---|---|
| doi | https://doi.org/10.17615/36ey-p097 |
| ids.doi | https://doi.org/10.17615/36ey-p097 |
| ids.mag | 2949477478 |
| ids.openalex | https://openalex.org/W2949477478 |
| fwci | 0.36917188 |
| type | article |
| title | Knowing When to Stop: Final Results vs. Work Involved in Systematic Review Database Searching |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10206 |
| topics[0].field.id | https://openalex.org/fields/18 |
| topics[0].field.display_name | Decision Sciences |
| topics[0].score | 0.13840000331401825 |
| topics[0].domain.id | https://openalex.org/domains/2 |
| topics[0].domain.display_name | Social Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1804 |
| topics[0].subfield.display_name | Statistics, Probability and Uncertainty |
| topics[0].display_name | Meta-analysis and systematic reviews |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C77088390 |
| concepts[0].level | 1 |
| concepts[0].score | 0.576202929019928 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q8513 |
| concepts[0].display_name | Database |
| concepts[1].id | https://openalex.org/C18762648 |
| concepts[1].level | 2 |
| concepts[1].score | 0.515182614326477 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q42213 |
| concepts[1].display_name | Work (physics) |
| concepts[2].id | https://openalex.org/C41008148 |
| concepts[2].level | 0 |
| concepts[2].score | 0.4958363473415375 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[2].display_name | Computer science |
| concepts[3].id | https://openalex.org/C23123220 |
| concepts[3].level | 1 |
| concepts[3].score | 0.3988545536994934 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q816826 |
| concepts[3].display_name | Information retrieval |
| concepts[4].id | https://openalex.org/C127413603 |
| concepts[4].level | 0 |
| concepts[4].score | 0.17900437116622925 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q11023 |
| concepts[4].display_name | Engineering |
| concepts[5].id | https://openalex.org/C78519656 |
| concepts[5].level | 1 |
| concepts[5].score | 0.0 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q101333 |
| concepts[5].display_name | Mechanical engineering |
| keywords[0].id | https://openalex.org/keywords/database |
| keywords[0].score | 0.576202929019928 |
| keywords[0].display_name | Database |
| keywords[1].id | https://openalex.org/keywords/work |
| keywords[1].score | 0.515182614326477 |
| keywords[1].display_name | Work (physics) |
| keywords[2].id | https://openalex.org/keywords/computer-science |
| keywords[2].score | 0.4958363473415375 |
| keywords[2].display_name | Computer science |
| keywords[3].id | https://openalex.org/keywords/information-retrieval |
| keywords[3].score | 0.3988545536994934 |
| keywords[3].display_name | Information retrieval |
| keywords[4].id | https://openalex.org/keywords/engineering |
| keywords[4].score | 0.17900437116622925 |
| keywords[4].display_name | Engineering |
| language | en |
| locations[0].id | mag:2949477478 |
| locations[0].is_oa | False |
| locations[0].source | |
| locations[0].license | |
| locations[0].pdf_url | |
| locations[0].version | |
| locations[0].raw_type | |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | https://cdr.lib.unc.edu/concern/scholarly_works/v692t8717 |
| locations[1].id | doi:10.17615/36ey-p097 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S7407051488 |
| locations[1].source.type | repository |
| locations[1].source.is_oa | False |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | UNC Libraries |
| locations[1].source.host_organization | |
| locations[1].source.host_organization_name | |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article-journal |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.17615/36ey-p097 |
| indexed_in | datacite |
| authorships[0].author.id | https://openalex.org/A5112854208 |
| authorships[0].author.orcid | |
| authorships[0].author.display_name | Jennifer S. Walker |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Jennifer S. Walker |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5081271509 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-1638-9351 |
| authorships[1].author.display_name | Rachael Posey |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Rachael Posey |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5006775351 |
| authorships[2].author.orcid | |
| authorships[2].author.display_name | Karen Crowell |
| authorships[2].author_position | last |
| authorships[2].raw_author_name | Karen Crowell |
| authorships[2].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://doi.org/10.17615/36ey-p097 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Knowing When to Stop: Final Results vs. Work Involved in Systematic Review Database Searching |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10206 |
| primary_topic.field.id | https://openalex.org/fields/18 |
| primary_topic.field.display_name | Decision Sciences |
| primary_topic.score | 0.13840000331401825 |
| primary_topic.domain.id | https://openalex.org/domains/2 |
| primary_topic.domain.display_name | Social Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1804 |
| primary_topic.subfield.display_name | Statistics, Probability and Uncertainty |
| primary_topic.display_name | Meta-analysis and systematic reviews |
| related_works | https://openalex.org/W2111616997, https://openalex.org/W991155967, https://openalex.org/W2992972214, https://openalex.org/W2733611476, https://openalex.org/W2902604499, https://openalex.org/W2345269694, https://openalex.org/W2152723599, https://openalex.org/W2526108156, https://openalex.org/W2042037644, https://openalex.org/W2765636693, https://openalex.org/W2184567745, https://openalex.org/W2995621663, https://openalex.org/W2937759506, https://openalex.org/W1576095762 |
| cited_by_count | 2 |
| counts_by_year[0].year | 2021 |
| counts_by_year[0].cited_by_count | 2 |
| locations_count | 2 |
| best_oa_location.id | doi:10.17615/36ey-p097 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S7407051488 |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | False |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | UNC Libraries |
| best_oa_location.source.host_organization | |
| best_oa_location.source.host_organization_name | |
| best_oa_location.license | |
| best_oa_location.pdf_url | |
| best_oa_location.version | |
| best_oa_location.raw_type | article-journal |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | https://doi.org/10.17615/36ey-p097 |
| primary_location.id | mag:2949477478 |
| primary_location.is_oa | False |
| primary_location.source | |
| primary_location.license | |
| primary_location.pdf_url | |
| primary_location.version | |
| primary_location.raw_type | |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | https://cdr.lib.unc.edu/concern/scholarly_works/v692t8717 |
| publication_date | 2019-08-17 |
| publication_year | 2019 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 94, 109 |
| abstract_inverted_index.16 | 227 |
| abstract_inverted_index.20 | 250 |
| abstract_inverted_index.50 | 79 |
| abstract_inverted_index.58 | 240 |
| abstract_inverted_index.96 | 276 |
| abstract_inverted_index.In | 289 |
| abstract_inverted_index.Of | 220 |
| abstract_inverted_index.SR | 24, 151, 297 |
| abstract_inverted_index.To | 1 |
| abstract_inverted_index.We | 45, 85, 119, 160 |
| abstract_inverted_index.an | 23, 206, 214, 323, 336, 351, 371, 385 |
| abstract_inverted_index.as | 68 |
| abstract_inverted_index.be | 154, 231, 244, 256, 270, 283 |
| abstract_inverted_index.by | 156, 168 |
| abstract_inverted_index.in | 22, 32, 51, 90, 115, 131, 149, 226, 233, 239, 246, 258, 272, 285, 295, 300, 308, 317, 327, 367, 376 |
| abstract_inverted_index.of | 8, 28, 56, 102, 112, 123, 135, 146, 165, 177, 188, 208, 216, 280, 292, 353, 356, 373 |
| abstract_inverted_index.or | 311, 319 |
| abstract_inverted_index.to | 62, 129, 173, 180, 184, 329, 358, 378 |
| abstract_inverted_index.we | 142 |
| abstract_inverted_index.756 | 354 |
| abstract_inverted_index.90% | 279 |
| abstract_inverted_index.923 | 325 |
| abstract_inverted_index.99% | 291 |
| abstract_inverted_index.For | 249, 275 |
| abstract_inverted_index.SR. | 360 |
| abstract_inverted_index.SRs | 47, 80, 198, 204, 228, 241, 265, 313, 361 |
| abstract_inverted_index.all | 88, 230, 243, 252 |
| abstract_inverted_index.and | 48, 54, 108, 141, 212, 261 |
| abstract_inverted_index.any | 346, 395 |
| abstract_inverted_index.did | 343, 392 |
| abstract_inverted_index.for | 11, 17, 191, 362 |
| abstract_inverted_index.met | 199 |
| abstract_inverted_index.not | 344, 393 |
| abstract_inverted_index.one | 234, 331, 380 |
| abstract_inverted_index.our | 91, 200 |
| abstract_inverted_index.per | 81 |
| abstract_inverted_index.the | 3, 6, 12, 26, 33, 99, 116, 136, 144, 163, 174, 181, 186, 222, 236, 262, 304 |
| abstract_inverted_index.two | 247, 286, 301 |
| abstract_inverted_index.(SR) | 15 |
| abstract_inverted_index.1440 | 374 |
| abstract_inverted_index.2012 | 61 |
| abstract_inverted_index.2410 | 338 |
| abstract_inverted_index.4.43 | 217 |
| abstract_inverted_index.8963 | 387 |
| abstract_inverted_index.SRs, | 251, 277 |
| abstract_inverted_index.each | 18, 40, 106, 127, 134, 150, 158, 170, 192, 296, 359 |
| abstract_inverted_index.find | 330, 379 |
| abstract_inverted_index.four | 273, 368 |
| abstract_inverted_index.from | 39, 59, 105, 126, 340, 389 |
| abstract_inverted_index.have | 74 |
| abstract_inverted_index.high | 69 |
| abstract_inverted_index.list | 111 |
| abstract_inverted_index.more | 77, 320 |
| abstract_inverted_index.over | 76, 278 |
| abstract_inverted_index.plus | 335, 384 |
| abstract_inverted_index.team | 16 |
| abstract_inverted_index.than | 78 |
| abstract_inverted_index.that | 36, 73, 87, 152, 268, 314, 342, 391 |
| abstract_inverted_index.then | 120, 161 |
| abstract_inverted_index.used | 121 |
| abstract_inverted_index.were | 37, 139, 298, 365 |
| abstract_inverted_index.with | 98, 303 |
| abstract_inverted_index.work | 9, 189, 357 |
| abstract_inverted_index.year | 82 |
| abstract_inverted_index.2012. | 84 |
| abstract_inverted_index.2015, | 64 |
| abstract_inverted_index.48.13 | 209 |
| abstract_inverted_index.JAMA, | 52 |
| abstract_inverted_index.These | 203 |
| abstract_inverted_index.added | 167, 179 |
| abstract_inverted_index.being | 306 |
| abstract_inverted_index.could | 153, 229, 242, 255, 269, 282 |
| abstract_inverted_index.each. | 219 |
| abstract_inverted_index.final | 34, 117 |
| abstract_inverted_index.found | 232, 245, 257, 271, 284, 299, 307, 315, 366 |
| abstract_inverted_index.hours | 355 |
| abstract_inverted_index.lists | 122 |
| abstract_inverted_index.order | 328, 377 |
| abstract_inverted_index.paper | 194 |
| abstract_inverted_index.phase | 183 |
| abstract_inverted_index.since | 83 |
| abstract_inverted_index.those | 66 |
| abstract_inverted_index.three | 259, 264, 318 |
| abstract_inverted_index.total | 100, 175 |
| abstract_inverted_index.which | 132, 363 |
| abstract_inverted_index.Annals | 55 |
| abstract_inverted_index.adding | 350 |
| abstract_inverted_index.amount | 7, 187 |
| abstract_inverted_index.factor | 71 |
| abstract_inverted_index.impact | 70 |
| abstract_inverted_index.number | 27, 101, 164, 176 |
| abstract_inverted_index.papers | 89, 178 |
| abstract_inverted_index.report | 93 |
| abstract_inverted_index.return | 345, 394 |
| abstract_inverted_index.review | 14, 35, 182 |
| abstract_inverted_index.search | 96 |
| abstract_inverted_index.these, | 221 |
| abstract_inverted_index.total, | 290 |
| abstract_inverted_index.unique | 29 |
| abstract_inverted_index.versus | 25 |
| abstract_inverted_index.Embase, | 310 |
| abstract_inverted_index.January | 60 |
| abstract_inverted_index.Lancet, | 53 |
| abstract_inverted_index.average | 207, 215, 352, 372 |
| abstract_inverted_index.between | 5 |
| abstract_inverted_index.examine | 2 |
| abstract_inverted_index.journal | 210, 223 |
| abstract_inverted_index.located | 155 |
| abstract_inverted_index.records | 326, 339, 375, 388 |
| abstract_inverted_index.results | 166 |
| abstract_inverted_index.December | 63 |
| abstract_inverted_index.Internal | 57 |
| abstract_inverted_index.Medicine | 58 |
| abstract_inverted_index.Methods: | 44 |
| abstract_inverted_index.Results: | 196 |
| abstract_inverted_index.analysis | 92 |
| abstract_inverted_index.article, | 334, 383 |
| abstract_inverted_index.articles | 30, 113, 138, 147, 211, 224, 237, 254, 267, 281, 293, 316, 364 |
| abstract_inverted_index.compared | 162 |
| abstract_inverted_index.complete | 95, 110 |
| abstract_inverted_index.database | 20, 42, 107, 128, 172 |
| abstract_inverted_index.estimate | 185 |
| abstract_inverted_index.included | 31, 114, 124, 137, 148, 205, 225, 238, 253, 266, 294, 333, 348, 382, 397 |
| abstract_inverted_index.indexed, | 140 |
| abstract_inverted_index.involved | 10 |
| abstract_inverted_index.journals | 67, 72, 125 |
| abstract_inverted_index.majority | 305 |
| abstract_inverted_index.required | 86, 190 |
| abstract_inverted_index.reviewed | 46 |
| abstract_inverted_index.screened | 322, 370 |
| abstract_inverted_index.searched | 21, 213 |
| abstract_inverted_index.Cochrane. | 312 |
| abstract_inverted_index.analysis. | 118 |
| abstract_inverted_index.articles, | 349 |
| abstract_inverted_index.articles. | 398 |
| abstract_inverted_index.citations | 103 |
| abstract_inverted_index.criteria. | 202 |
| abstract_inverted_index.database. | 159 |
| abstract_inverted_index.database; | 235 |
| abstract_inverted_index.databases | 218, 321, 341, 369, 390 |
| abstract_inverted_index.determine | 130 |
| abstract_inverted_index.inclusion | 201 |
| abstract_inverted_index.published | 50, 75 |
| abstract_inverted_index.remaining | 263 |
| abstract_inverted_index.retrieved | 38, 104 |
| abstract_inverted_index.searched. | 43 |
| abstract_inverted_index.searching | 157, 169 |
| abstract_inverted_index.selecting | 65 |
| abstract_inverted_index.strategy, | 97 |
| abstract_inverted_index.additional | 19, 41, 171, 193, 324, 332, 337, 347, 381, 386, 396 |
| abstract_inverted_index.calculated | 143 |
| abstract_inverted_index.databases, | 260, 302 |
| abstract_inverted_index.databases. | 248, 274, 287 |
| abstract_inverted_index.percentage | 145 |
| abstract_inverted_index.systematic | 13 |
| abstract_inverted_index.Objectives: | 0 |
| abstract_inverted_index.database(s) | 133 |
| abstract_inverted_index.identified. | 195 |
| abstract_inverted_index.Conclusions: | 288 |
| abstract_inverted_index.Ninety-seven | 197 |
| abstract_inverted_index.relationship | 4 |
| abstract_inverted_index.meta-analyses | 49 |
| abstract_inverted_index.PubMed/MEDLINE, | 309 |
| cited_by_percentile_year.max | 95 |
| cited_by_percentile_year.min | 93 |
| countries_distinct_count | 0 |
| institutions_distinct_count | 3 |
| citation_normalized_percentile.value | 0.6637419 |
| citation_normalized_percentile.is_in_top_1_percent | False |
| citation_normalized_percentile.is_in_top_10_percent | False |