A scoping review on metrics to quantify reproducibility: a multitude of questions leads to a multitude of metrics Article Swipe

View

Rachel Heyard , Samuel Pawel , Joris Frese , Bernhard Voelkl , Hanno Wuerbel , Sarah McCann , Leonhard Held , Kimberley E. Wever , Helena Hartmann , Louise Townsin , Stephanie Zellers ·

YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.31222/osf.io/apdxk_v2

*Background:* Reproducibility is recognized as essential to scientific progress and integrity. Replication studies and large-scale replication projects, aiming to quantify different aspects of reproducibility, have become more common. Since no standardized approach to measuring reproducibility exists,a diverse set of metrics has emerged and a comprehensive overview is needed.*Methods:* We conducted a scoping review to identify large-scale replication projects that used metrics and methodological papers that proposed or discussed metrics. The project list was compiled by the authors. For the methodological papers, we searched Scopus, MedLine, PsycINFO andEconLit. Records were screened in duplicate against predefined inclusion criteria. Demographic information on included records and information on reproducibility metrics used, suggested or discussed was extracted.*Results:* We identified 49 large-scale projects and 97 methodological papers, and extracted 50 metrics. The metrics were characterized based on type (formulas and/or statistical models, frameworks, graphical representations, studies and questionnaires, algorithms), input required, and appropriate application scenarios. Each metric addresses a distinct question.*Conclusions:* Our review provides a comprehensive resource in the form of a “live”, interactive table for future replication teams and meta-researchers, offering support in how to select the most appropriate metrics that are aligned with research questions and project goals.

Related Topics

Feeding The Multitude

Concepts

Multitude Data science Computer science Reproducibility Political science Mathematics Statistics Law

Metadata

Type: review
Language: en
Landing Page: https://doi.org/10.31222/osf.io/apdxk_v2
OA Status: gold
Related Works: 10
OpenAlex ID: https://openalex.org/W4409318685

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4409318685

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.31222/osf.io/apdxk_v2

Digital Object Identifier
Title: A scoping review on metrics to quantify reproducibility: a multitude of questions leads to a multitude of metrics

Work title
Type: review

OpenAlex work type
Language: en

Primary language
Publication year: 2025

Year of publication
Publication date: 2025-04-10

Full publication date if available
Authors: Rachel Heyard, Samuel Pawel, Joris Frese, Bernhard Voelkl, Hanno Wuerbel, Sarah McCann, Leonhard Held, Kimberley E. Wever, Helena Hartmann, Louise Townsin, Stephanie Zellers

List of authors in order
Landing page: https://doi.org/10.31222/osf.io/apdxk_v2

Publisher landing page
Open access: Yes

Whether a free full text is available
OA status: gold

Open access status per OpenAlex
OA URL: https://doi.org/10.31222/osf.io/apdxk_v2

Direct OA link when available
Concepts: Multitude, Data science, Computer science, Reproducibility, Political science, Mathematics, Statistics, Law

Top concepts (fields/topics) attached by OpenAlex
Cited by: 0

Total citation count in OpenAlex
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4409318685
doi	https://doi.org/10.31222/osf.io/apdxk_v2
ids.doi	https://doi.org/10.31222/osf.io/apdxk_v2
ids.openalex	https://openalex.org/W4409318685
fwci
type	review
title	A scoping review on metrics to quantify reproducibility: a multitude of questions leads to a multitude of metrics
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C2780565519
concepts[0].level	2
concepts[0].score	0.9917877912521362
concepts[0].wikidata	https://www.wikidata.org/wiki/Q1208937
concepts[0].display_name	Multitude
concepts[1].id	https://openalex.org/C2522767166
concepts[1].level	1
concepts[1].score	0.44542092084884644
concepts[1].wikidata	https://www.wikidata.org/wiki/Q2374463
concepts[1].display_name	Data science
concepts[2].id	https://openalex.org/C41008148
concepts[2].level	0
concepts[2].score	0.4427623748779297
concepts[2].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[2].display_name	Computer science
concepts[3].id	https://openalex.org/C9893847
concepts[3].level	2
concepts[3].score	0.4140357971191406
concepts[3].wikidata	https://www.wikidata.org/wiki/Q1425625
concepts[3].display_name	Reproducibility
concepts[4].id	https://openalex.org/C17744445
concepts[4].level	0
concepts[4].score	0.17517000436782837
concepts[4].wikidata	https://www.wikidata.org/wiki/Q36442
concepts[4].display_name	Political science
concepts[5].id	https://openalex.org/C33923547
concepts[5].level	0
concepts[5].score	0.17255577445030212
concepts[5].wikidata	https://www.wikidata.org/wiki/Q395
concepts[5].display_name	Mathematics
concepts[6].id	https://openalex.org/C105795698
concepts[6].level	1
concepts[6].score	0.12246447801589966
concepts[6].wikidata	https://www.wikidata.org/wiki/Q12483
concepts[6].display_name	Statistics
concepts[7].id	https://openalex.org/C199539241
concepts[7].level	1
concepts[7].score	0.0
concepts[7].wikidata	https://www.wikidata.org/wiki/Q7748
concepts[7].display_name	Law
keywords[0].id	https://openalex.org/keywords/multitude
keywords[0].score	0.9917877912521362
keywords[0].display_name	Multitude
keywords[1].id	https://openalex.org/keywords/data-science
keywords[1].score	0.44542092084884644
keywords[1].display_name	Data science
keywords[2].id	https://openalex.org/keywords/computer-science
keywords[2].score	0.4427623748779297
keywords[2].display_name	Computer science
keywords[3].id	https://openalex.org/keywords/reproducibility
keywords[3].score	0.4140357971191406
keywords[3].display_name	Reproducibility
keywords[4].id	https://openalex.org/keywords/political-science
keywords[4].score	0.17517000436782837
keywords[4].display_name	Political science
keywords[5].id	https://openalex.org/keywords/mathematics
keywords[5].score	0.17255577445030212
keywords[5].display_name	Mathematics
keywords[6].id	https://openalex.org/keywords/statistics
keywords[6].score	0.12246447801589966
keywords[6].display_name	Statistics
language	en
locations[0].id	doi:10.31222/osf.io/apdxk_v2
locations[0].is_oa	True
locations[0].source
locations[0].license	cc-by
locations[0].pdf_url
locations[0].version	acceptedVersion
locations[0].raw_type	posted-content
locations[0].license_id	https://openalex.org/licenses/cc-by
locations[0].is_accepted	True
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	https://doi.org/10.31222/osf.io/apdxk_v2
indexed_in	crossref
authorships[0].author.id	https://openalex.org/A5085918982
authorships[0].author.orcid	https://orcid.org/0000-0002-7531-4333
authorships[0].author.display_name	Rachel Heyard
authorships[0].author_position	first
authorships[0].raw_author_name	Rachel Heyard
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5038190384
authorships[1].author.orcid	https://orcid.org/0000-0003-2779-320X
authorships[1].author.display_name	Samuel Pawel
authorships[1].author_position	middle
authorships[1].raw_author_name	Samuel Pawel
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5032406666
authorships[2].author.orcid	https://orcid.org/0000-0002-5871-997X
authorships[2].author.display_name	Joris Frese
authorships[2].author_position	middle
authorships[2].raw_author_name	Joris Frese
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5035897648
authorships[3].author.orcid	https://orcid.org/0000-0001-5454-2508
authorships[3].author.display_name	Bernhard Voelkl
authorships[3].author_position	middle
authorships[3].raw_author_name	Bernhard Voelkl
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5027072532
authorships[4].author.orcid	https://orcid.org/0000-0002-2934-3010
authorships[4].author.display_name	Hanno Wuerbel
authorships[4].author_position	middle
authorships[4].raw_author_name	Hanno Würbel
authorships[4].is_corresponding	False
authorships[5].author.id	https://openalex.org/A5045342203
authorships[5].author.orcid	https://orcid.org/0000-0003-4737-2349
authorships[5].author.display_name	Sarah McCann
authorships[5].author_position	middle
authorships[5].raw_author_name	Sarah McCann
authorships[5].is_corresponding	False
authorships[6].author.id	https://openalex.org/A5078043213
authorships[6].author.orcid	https://orcid.org/0000-0002-8686-5325
authorships[6].author.display_name	Leonhard Held
authorships[6].author_position	middle
authorships[6].raw_author_name	Leonhard Held
authorships[6].is_corresponding	False
authorships[7].author.id	https://openalex.org/A5064022439
authorships[7].author.orcid	https://orcid.org/0000-0003-3635-3660
authorships[7].author.display_name	Kimberley E. Wever
authorships[7].author_position	middle
authorships[7].raw_author_name	Kimberley E. Wever
authorships[7].is_corresponding	False
authorships[8].author.id	https://openalex.org/A5029109834
authorships[8].author.orcid	https://orcid.org/0000-0002-1331-6683
authorships[8].author.display_name	Helena Hartmann
authorships[8].author_position	middle
authorships[8].raw_author_name	Helena Hartmann
authorships[8].is_corresponding	False
authorships[9].author.id	https://openalex.org/A5032137854
authorships[9].author.orcid	https://orcid.org/0000-0002-2212-1626
authorships[9].author.display_name	Louise Townsin
authorships[9].author_position	middle
authorships[9].raw_author_name	Louise Townsin
authorships[9].is_corresponding	False
authorships[10].author.id	https://openalex.org/A5039496797
authorships[10].author.orcid	https://orcid.org/0000-0001-8927-3483
authorships[10].author.display_name	Stephanie Zellers
authorships[10].author_position	last
authorships[10].raw_author_name	Stephanie Zellers
authorships[10].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://doi.org/10.31222/osf.io/apdxk_v2
open_access.oa_status	gold
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	A scoping review on metrics to quantify reproducibility: a multitude of questions leads to a multitude of metrics
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T03:46:38.306776
primary_topic
related_works	https://openalex.org/W4391375266, https://openalex.org/W2899084033, https://openalex.org/W2748952813, https://openalex.org/W2417432647, https://openalex.org/W2371114124, https://openalex.org/W2168950483, https://openalex.org/W1975110769, https://openalex.org/W2378535610, https://openalex.org/W4213029470, https://openalex.org/W2798951784
cited_by_count	0
locations_count	1
best_oa_location.id	doi:10.31222/osf.io/apdxk_v2
best_oa_location.is_oa	True
best_oa_location.source
best_oa_location.license	cc-by
best_oa_location.pdf_url
best_oa_location.version	acceptedVersion
best_oa_location.raw_type	posted-content
best_oa_location.license_id	https://openalex.org/licenses/cc-by
best_oa_location.is_accepted	True
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	https://doi.org/10.31222/osf.io/apdxk_v2
primary_location.id	doi:10.31222/osf.io/apdxk_v2
primary_location.is_oa	True
primary_location.source
primary_location.license	cc-by
primary_location.pdf_url
primary_location.version	acceptedVersion
primary_location.raw_type	posted-content
primary_location.license_id	https://openalex.org/licenses/cc-by
primary_location.is_accepted	True
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	https://doi.org/10.31222/osf.io/apdxk_v2
publication_date	2025-04-10
publication_year	2025
referenced_works_count	0
abstract_inverted_index.a	43, 50, 152, 158, 165
abstract_inverted_index.49	114
abstract_inverted_index.50	123
abstract_inverted_index.97	118
abstract_inverted_index.We	48, 112
abstract_inverted_index.as	4
abstract_inverted_index.by	74
abstract_inverted_index.in	90, 161, 177
abstract_inverted_index.is	2, 46
abstract_inverted_index.no	29
abstract_inverted_index.of	22, 38, 164
abstract_inverted_index.on	98, 103, 130
abstract_inverted_index.or	66, 108
abstract_inverted_index.to	6, 18, 32, 53, 179
abstract_inverted_index.we	81
abstract_inverted_index.For	77
abstract_inverted_index.Our	155
abstract_inverted_index.The	69, 125
abstract_inverted_index.and	9, 13, 42, 61, 101, 117, 121, 140, 145, 173, 191
abstract_inverted_index.are	186
abstract_inverted_index.for	169
abstract_inverted_index.has	40
abstract_inverted_index.how	178
abstract_inverted_index.set	37
abstract_inverted_index.the	75, 78, 162, 181
abstract_inverted_index.was	72, 110
abstract_inverted_index.Each	149
abstract_inverted_index.form	163
abstract_inverted_index.have	24
abstract_inverted_index.list	71
abstract_inverted_index.more	26
abstract_inverted_index.most	182
abstract_inverted_index.that	58, 64, 185
abstract_inverted_index.type	131
abstract_inverted_index.used	59
abstract_inverted_index.were	88, 127
abstract_inverted_index.with	188
abstract_inverted_index.Since	28
abstract_inverted_index.based	129
abstract_inverted_index.input	143
abstract_inverted_index.table	168
abstract_inverted_index.teams	172
abstract_inverted_index.used,	106
abstract_inverted_index.aiming	17
abstract_inverted_index.and/or	133
abstract_inverted_index.become	25
abstract_inverted_index.future	170
abstract_inverted_index.goals.	193
abstract_inverted_index.metric	150
abstract_inverted_index.papers	63
abstract_inverted_index.review	52, 156
abstract_inverted_index.select	180
abstract_inverted_index.Records	87
abstract_inverted_index.Scopus,	83
abstract_inverted_index.against	92
abstract_inverted_index.aligned	187
abstract_inverted_index.aspects	21
abstract_inverted_index.common.	27
abstract_inverted_index.diverse	36
abstract_inverted_index.emerged	41
abstract_inverted_index.metrics	39, 60, 105, 126, 184
abstract_inverted_index.models,	135
abstract_inverted_index.papers,	80, 120
abstract_inverted_index.project	70, 192
abstract_inverted_index.records	100
abstract_inverted_index.scoping	51
abstract_inverted_index.studies	12, 139
abstract_inverted_index.support	176
abstract_inverted_index.MedLine,	84
abstract_inverted_index.PsycINFO	85
abstract_inverted_index.approach	31
abstract_inverted_index.authors.	76
abstract_inverted_index.compiled	73
abstract_inverted_index.distinct	153
abstract_inverted_index.exists,a	35
abstract_inverted_index.identify	54
abstract_inverted_index.included	99
abstract_inverted_index.metrics.	68, 124
abstract_inverted_index.offering	175
abstract_inverted_index.overview	45
abstract_inverted_index.progress	8
abstract_inverted_index.projects	57, 116
abstract_inverted_index.proposed	65
abstract_inverted_index.provides	157
abstract_inverted_index.quantify	19
abstract_inverted_index.research	189
abstract_inverted_index.resource	160
abstract_inverted_index.screened	89
abstract_inverted_index.searched	82
abstract_inverted_index.(formulas	132
abstract_inverted_index.addresses	151
abstract_inverted_index.conducted	49
abstract_inverted_index.criteria.	95
abstract_inverted_index.different	20
abstract_inverted_index.discussed	67, 109
abstract_inverted_index.duplicate	91
abstract_inverted_index.essential	5
abstract_inverted_index.extracted	122
abstract_inverted_index.graphical	137
abstract_inverted_index.inclusion	94
abstract_inverted_index.measuring	33
abstract_inverted_index.projects,	16
abstract_inverted_index.questions	190
abstract_inverted_index.required,	144
abstract_inverted_index.suggested	107
abstract_inverted_index.identified	113
abstract_inverted_index.integrity.	10
abstract_inverted_index.predefined	93
abstract_inverted_index.recognized	3
abstract_inverted_index.scenarios.	148
abstract_inverted_index.scientific	7
abstract_inverted_index.Demographic	96
abstract_inverted_index.Replication	11
abstract_inverted_index.andEconLit.	86
abstract_inverted_index.application	147
abstract_inverted_index.appropriate	146, 183
abstract_inverted_index.frameworks,	136
abstract_inverted_index.information	97, 102
abstract_inverted_index.interactive	167
abstract_inverted_index.large-scale	14, 55, 115
abstract_inverted_index.replication	15, 56, 171
abstract_inverted_index.statistical	134
abstract_inverted_index.“live”,	166
abstract_inverted_index.algorithms),	142
abstract_inverted_index.standardized	30
abstract_inverted_index.Background:	0
abstract_inverted_index.characterized	128
abstract_inverted_index.comprehensive	44, 159
abstract_inverted_index.methodological	62, 79, 119
abstract_inverted_index.Reproducibility	1
abstract_inverted_index.questionnaires,	141
abstract_inverted_index.reproducibility	34, 104
abstract_inverted_index.representations,	138
abstract_inverted_index.reproducibility,	23
abstract_inverted_index.meta-researchers,	174
abstract_inverted_index.needed.Methods:	47
abstract_inverted_index.extracted.Results:	111
abstract_inverted_index.question.Conclusions:	154
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	11
citation_normalized_percentile