Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects Article Swipe
Rachel Heyard
,
Leonhard Held
·
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.31222/osf.io/e9nw2_v2
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.31222/osf.io/e9nw2_v2
Recent large-scale replication projects (RPs) have estimated concerningly low reproducibility rates. Further, they all reported substantial degrees of shrinkage of effect size, where the replica tion effect size was found to be, on average, much smaller than the original effect size. Within these RPs, the included original-replication study-pairs can vary substantially with respect to aspects of study design, outcome measures, and descriptive features of both original and replication study population and study team. When broader claims about the reproducibility of an entire field are based on aggregations of such heterogeneous data, it becomes imperative to conduct a rigorous analysis of the amount and sources of shrinkage and heterogeneity within and between study-pairs included. Methodology from the meta-analysis literature provides an approach for quantifying the heterogeneity present in RPs, as additive or multiplicative parameter. Meta-regression methodology further allows for an investigation of the sources of shrinkage and heterogeneity. We propose the use of location-scale meta-regressions as a means to directly relate the identified characteristics with shrinkage (represented by the location) and the heterogeneity variance (represented by the scale). This could also provide valuable insights into drivers and factors associated with high or low reproducibility rates and therefore contextualise results of PRs. The proposed methodology is illustrated using data from the Replication Project Psychology and the Replication Project Experimental Economics. All analysis scripts and data are available online.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- https://doi.org/10.31222/osf.io/e9nw2_v2
- https://osf.io/e9nw2_v2/download
- OA Status
- gold
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4407953217
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4407953217Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.31222/osf.io/e9nw2_v2Digital Object Identifier
- Title
-
Meta-regression to explain shrinkage and heterogeneity in large-scale replication projectsWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2025Year of publication
- Publication date
-
2025-02-26Full publication date if available
- Authors
-
Rachel Heyard, Leonhard HeldList of authors in order
- Landing page
-
https://doi.org/10.31222/osf.io/e9nw2_v2Publisher landing page
- PDF URL
-
https://osf.io/e9nw2_v2/downloadDirect link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
goldOpen access status per OpenAlex
- OA URL
-
https://osf.io/e9nw2_v2/downloadDirect OA link when available
- Concepts
-
Replication (statistics), Shrinkage, Meta-regression, Scale (ratio), Regression, Regression analysis, Econometrics, Computer science, Economics, Statistics, Meta-analysis, Geography, Mathematics, Medicine, Cartography, Internal medicineTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4407953217 |
|---|---|
| doi | https://doi.org/10.31222/osf.io/e9nw2_v2 |
| ids.doi | https://doi.org/10.31222/osf.io/e9nw2_v2 |
| ids.openalex | https://openalex.org/W4407953217 |
| fwci | 0.0 |
| type | preprint |
| title | Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T11986 |
| topics[0].field.id | https://openalex.org/fields/18 |
| topics[0].field.display_name | Decision Sciences |
| topics[0].score | 0.8876000046730042 |
| topics[0].domain.id | https://openalex.org/domains/2 |
| topics[0].domain.display_name | Social Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1802 |
| topics[0].subfield.display_name | Information Systems and Management |
| topics[0].display_name | Scientific Computing and Data Management |
| topics[1].id | https://openalex.org/T11407 |
| topics[1].field.id | https://openalex.org/fields/22 |
| topics[1].field.display_name | Engineering |
| topics[1].score | 0.881600022315979 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/2204 |
| topics[1].subfield.display_name | Biomedical Engineering |
| topics[1].display_name | Innovative Microfluidic and Catalytic Techniques Innovation |
| topics[2].id | https://openalex.org/T10743 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.8442999720573425 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1712 |
| topics[2].subfield.display_name | Software |
| topics[2].display_name | Software Testing and Debugging Techniques |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C12590798 |
| concepts[0].level | 2 |
| concepts[0].score | 0.857563853263855 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q3933199 |
| concepts[0].display_name | Replication (statistics) |
| concepts[1].id | https://openalex.org/C180145272 |
| concepts[1].level | 2 |
| concepts[1].score | 0.7556864619255066 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q7504144 |
| concepts[1].display_name | Shrinkage |
| concepts[2].id | https://openalex.org/C124936440 |
| concepts[2].level | 3 |
| concepts[2].score | 0.7101601362228394 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q6822263 |
| concepts[2].display_name | Meta-regression |
| concepts[3].id | https://openalex.org/C2778755073 |
| concepts[3].level | 2 |
| concepts[3].score | 0.6250358819961548 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q10858537 |
| concepts[3].display_name | Scale (ratio) |
| concepts[4].id | https://openalex.org/C83546350 |
| concepts[4].level | 2 |
| concepts[4].score | 0.5567615628242493 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q1139051 |
| concepts[4].display_name | Regression |
| concepts[5].id | https://openalex.org/C152877465 |
| concepts[5].level | 2 |
| concepts[5].score | 0.4641830325126648 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q208042 |
| concepts[5].display_name | Regression analysis |
| concepts[6].id | https://openalex.org/C149782125 |
| concepts[6].level | 1 |
| concepts[6].score | 0.40949174761772156 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q160039 |
| concepts[6].display_name | Econometrics |
| concepts[7].id | https://openalex.org/C41008148 |
| concepts[7].level | 0 |
| concepts[7].score | 0.33963918685913086 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[7].display_name | Computer science |
| concepts[8].id | https://openalex.org/C162324750 |
| concepts[8].level | 0 |
| concepts[8].score | 0.33498215675354004 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q8134 |
| concepts[8].display_name | Economics |
| concepts[9].id | https://openalex.org/C105795698 |
| concepts[9].level | 1 |
| concepts[9].score | 0.2410784363746643 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q12483 |
| concepts[9].display_name | Statistics |
| concepts[10].id | https://openalex.org/C95190672 |
| concepts[10].level | 2 |
| concepts[10].score | 0.20776233077049255 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q815382 |
| concepts[10].display_name | Meta-analysis |
| concepts[11].id | https://openalex.org/C205649164 |
| concepts[11].level | 0 |
| concepts[11].score | 0.18445241451263428 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q1071 |
| concepts[11].display_name | Geography |
| concepts[12].id | https://openalex.org/C33923547 |
| concepts[12].level | 0 |
| concepts[12].score | 0.17905429005622864 |
| concepts[12].wikidata | https://www.wikidata.org/wiki/Q395 |
| concepts[12].display_name | Mathematics |
| concepts[13].id | https://openalex.org/C71924100 |
| concepts[13].level | 0 |
| concepts[13].score | 0.11381688714027405 |
| concepts[13].wikidata | https://www.wikidata.org/wiki/Q11190 |
| concepts[13].display_name | Medicine |
| concepts[14].id | https://openalex.org/C58640448 |
| concepts[14].level | 1 |
| concepts[14].score | 0.07096177339553833 |
| concepts[14].wikidata | https://www.wikidata.org/wiki/Q42515 |
| concepts[14].display_name | Cartography |
| concepts[15].id | https://openalex.org/C126322002 |
| concepts[15].level | 1 |
| concepts[15].score | 0.0 |
| concepts[15].wikidata | https://www.wikidata.org/wiki/Q11180 |
| concepts[15].display_name | Internal medicine |
| keywords[0].id | https://openalex.org/keywords/replication |
| keywords[0].score | 0.857563853263855 |
| keywords[0].display_name | Replication (statistics) |
| keywords[1].id | https://openalex.org/keywords/shrinkage |
| keywords[1].score | 0.7556864619255066 |
| keywords[1].display_name | Shrinkage |
| keywords[2].id | https://openalex.org/keywords/meta-regression |
| keywords[2].score | 0.7101601362228394 |
| keywords[2].display_name | Meta-regression |
| keywords[3].id | https://openalex.org/keywords/scale |
| keywords[3].score | 0.6250358819961548 |
| keywords[3].display_name | Scale (ratio) |
| keywords[4].id | https://openalex.org/keywords/regression |
| keywords[4].score | 0.5567615628242493 |
| keywords[4].display_name | Regression |
| keywords[5].id | https://openalex.org/keywords/regression-analysis |
| keywords[5].score | 0.4641830325126648 |
| keywords[5].display_name | Regression analysis |
| keywords[6].id | https://openalex.org/keywords/econometrics |
| keywords[6].score | 0.40949174761772156 |
| keywords[6].display_name | Econometrics |
| keywords[7].id | https://openalex.org/keywords/computer-science |
| keywords[7].score | 0.33963918685913086 |
| keywords[7].display_name | Computer science |
| keywords[8].id | https://openalex.org/keywords/economics |
| keywords[8].score | 0.33498215675354004 |
| keywords[8].display_name | Economics |
| keywords[9].id | https://openalex.org/keywords/statistics |
| keywords[9].score | 0.2410784363746643 |
| keywords[9].display_name | Statistics |
| keywords[10].id | https://openalex.org/keywords/meta-analysis |
| keywords[10].score | 0.20776233077049255 |
| keywords[10].display_name | Meta-analysis |
| keywords[11].id | https://openalex.org/keywords/geography |
| keywords[11].score | 0.18445241451263428 |
| keywords[11].display_name | Geography |
| keywords[12].id | https://openalex.org/keywords/mathematics |
| keywords[12].score | 0.17905429005622864 |
| keywords[12].display_name | Mathematics |
| keywords[13].id | https://openalex.org/keywords/medicine |
| keywords[13].score | 0.11381688714027405 |
| keywords[13].display_name | Medicine |
| keywords[14].id | https://openalex.org/keywords/cartography |
| keywords[14].score | 0.07096177339553833 |
| keywords[14].display_name | Cartography |
| language | en |
| locations[0].id | doi:10.31222/osf.io/e9nw2_v2 |
| locations[0].is_oa | True |
| locations[0].source | |
| locations[0].license | cc-by |
| locations[0].pdf_url | https://osf.io/e9nw2_v2/download |
| locations[0].version | acceptedVersion |
| locations[0].raw_type | posted-content |
| locations[0].license_id | https://openalex.org/licenses/cc-by |
| locations[0].is_accepted | True |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | https://doi.org/10.31222/osf.io/e9nw2_v2 |
| indexed_in | crossref |
| authorships[0].author.id | https://openalex.org/A5085918982 |
| authorships[0].author.orcid | https://orcid.org/0000-0002-7531-4333 |
| authorships[0].author.display_name | Rachel Heyard |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Rachel Heyard |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5078043213 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-8686-5325 |
| authorships[1].author.display_name | Leonhard Held |
| authorships[1].author_position | last |
| authorships[1].raw_author_name | Leonhard Held |
| authorships[1].is_corresponding | False |
| has_content.pdf | True |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://osf.io/e9nw2_v2/download |
| open_access.oa_status | gold |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T11986 |
| primary_topic.field.id | https://openalex.org/fields/18 |
| primary_topic.field.display_name | Decision Sciences |
| primary_topic.score | 0.8876000046730042 |
| primary_topic.domain.id | https://openalex.org/domains/2 |
| primary_topic.domain.display_name | Social Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1802 |
| primary_topic.subfield.display_name | Information Systems and Management |
| primary_topic.display_name | Scientific Computing and Data Management |
| related_works | https://openalex.org/W3089004032, https://openalex.org/W31220157, https://openalex.org/W2312753042, https://openalex.org/W4289356671, https://openalex.org/W2389155397, https://openalex.org/W2165884543, https://openalex.org/W3186837933, https://openalex.org/W2368989808, https://openalex.org/W1969346022, https://openalex.org/W2034959125 |
| cited_by_count | 0 |
| locations_count | 1 |
| best_oa_location.id | doi:10.31222/osf.io/e9nw2_v2 |
| best_oa_location.is_oa | True |
| best_oa_location.source | |
| best_oa_location.license | cc-by |
| best_oa_location.pdf_url | https://osf.io/e9nw2_v2/download |
| best_oa_location.version | acceptedVersion |
| best_oa_location.raw_type | posted-content |
| best_oa_location.license_id | https://openalex.org/licenses/cc-by |
| best_oa_location.is_accepted | True |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | https://doi.org/10.31222/osf.io/e9nw2_v2 |
| primary_location.id | doi:10.31222/osf.io/e9nw2_v2 |
| primary_location.is_oa | True |
| primary_location.source | |
| primary_location.license | cc-by |
| primary_location.pdf_url | https://osf.io/e9nw2_v2/download |
| primary_location.version | acceptedVersion |
| primary_location.raw_type | posted-content |
| primary_location.license_id | https://openalex.org/licenses/cc-by |
| primary_location.is_accepted | True |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | https://doi.org/10.31222/osf.io/e9nw2_v2 |
| publication_date | 2025-02-26 |
| publication_year | 2025 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 96, 155 |
| abstract_inverted_index.We | 147 |
| abstract_inverted_index.an | 80, 119, 138 |
| abstract_inverted_index.as | 128, 154 |
| abstract_inverted_index.by | 166, 174 |
| abstract_inverted_index.in | 126 |
| abstract_inverted_index.is | 203 |
| abstract_inverted_index.it | 91 |
| abstract_inverted_index.of | 17, 19, 55, 63, 79, 87, 99, 104, 140, 143, 151, 198 |
| abstract_inverted_index.on | 32, 85 |
| abstract_inverted_index.or | 130, 190 |
| abstract_inverted_index.to | 30, 53, 94, 157 |
| abstract_inverted_index.All | 218 |
| abstract_inverted_index.The | 200 |
| abstract_inverted_index.all | 13 |
| abstract_inverted_index.and | 60, 66, 70, 102, 106, 109, 145, 169, 185, 194, 212, 221 |
| abstract_inverted_index.are | 83, 223 |
| abstract_inverted_index.be, | 31 |
| abstract_inverted_index.can | 48 |
| abstract_inverted_index.for | 121, 137 |
| abstract_inverted_index.low | 8, 191 |
| abstract_inverted_index.the | 23, 37, 44, 77, 100, 115, 123, 141, 149, 160, 167, 170, 175, 208, 213 |
| abstract_inverted_index.use | 150 |
| abstract_inverted_index.was | 28 |
| abstract_inverted_index.PRs. | 199 |
| abstract_inverted_index.RPs, | 43, 127 |
| abstract_inverted_index.This | 177 |
| abstract_inverted_index.When | 73 |
| abstract_inverted_index.also | 179 |
| abstract_inverted_index.both | 64 |
| abstract_inverted_index.data | 206, 222 |
| abstract_inverted_index.from | 114, 207 |
| abstract_inverted_index.have | 5 |
| abstract_inverted_index.high | 189 |
| abstract_inverted_index.into | 183 |
| abstract_inverted_index.much | 34 |
| abstract_inverted_index.size | 27 |
| abstract_inverted_index.such | 88 |
| abstract_inverted_index.than | 36 |
| abstract_inverted_index.they | 12 |
| abstract_inverted_index.tion | 25 |
| abstract_inverted_index.vary | 49 |
| abstract_inverted_index.with | 51, 163, 188 |
| abstract_inverted_index.(RPs) | 4 |
| abstract_inverted_index.about | 76 |
| abstract_inverted_index.based | 84 |
| abstract_inverted_index.could | 178 |
| abstract_inverted_index.data, | 90 |
| abstract_inverted_index.field | 82 |
| abstract_inverted_index.found | 29 |
| abstract_inverted_index.means | 156 |
| abstract_inverted_index.rates | 193 |
| abstract_inverted_index.size, | 21 |
| abstract_inverted_index.size. | 40 |
| abstract_inverted_index.study | 56, 68, 71 |
| abstract_inverted_index.team. | 72 |
| abstract_inverted_index.these | 42 |
| abstract_inverted_index.using | 205 |
| abstract_inverted_index.where | 22 |
| abstract_inverted_index.Recent | 0 |
| abstract_inverted_index.Within | 41 |
| abstract_inverted_index.allows | 136 |
| abstract_inverted_index.amount | 101 |
| abstract_inverted_index.claims | 75 |
| abstract_inverted_index.effect | 20, 26, 39 |
| abstract_inverted_index.entire | 81 |
| abstract_inverted_index.rates. | 10 |
| abstract_inverted_index.relate | 159 |
| abstract_inverted_index.within | 108 |
| abstract_inverted_index.Project | 210, 215 |
| abstract_inverted_index.aspects | 54 |
| abstract_inverted_index.becomes | 92 |
| abstract_inverted_index.between | 110 |
| abstract_inverted_index.broader | 74 |
| abstract_inverted_index.conduct | 95 |
| abstract_inverted_index.degrees | 16 |
| abstract_inverted_index.design, | 57 |
| abstract_inverted_index.drivers | 184 |
| abstract_inverted_index.factors | 186 |
| abstract_inverted_index.further | 135 |
| abstract_inverted_index.online. | 225 |
| abstract_inverted_index.outcome | 58 |
| abstract_inverted_index.present | 125 |
| abstract_inverted_index.propose | 148 |
| abstract_inverted_index.provide | 180 |
| abstract_inverted_index.replica | 24 |
| abstract_inverted_index.respect | 52 |
| abstract_inverted_index.results | 197 |
| abstract_inverted_index.scale). | 176 |
| abstract_inverted_index.scripts | 220 |
| abstract_inverted_index.smaller | 35 |
| abstract_inverted_index.sources | 103, 142 |
| abstract_inverted_index.Further, | 11 |
| abstract_inverted_index.additive | 129 |
| abstract_inverted_index.analysis | 98, 219 |
| abstract_inverted_index.approach | 120 |
| abstract_inverted_index.average, | 33 |
| abstract_inverted_index.directly | 158 |
| abstract_inverted_index.features | 62 |
| abstract_inverted_index.included | 45 |
| abstract_inverted_index.insights | 182 |
| abstract_inverted_index.original | 38, 65 |
| abstract_inverted_index.projects | 3 |
| abstract_inverted_index.proposed | 201 |
| abstract_inverted_index.provides | 118 |
| abstract_inverted_index.reported | 14 |
| abstract_inverted_index.rigorous | 97 |
| abstract_inverted_index.valuable | 181 |
| abstract_inverted_index.variance | 172 |
| abstract_inverted_index.available | 224 |
| abstract_inverted_index.estimated | 6 |
| abstract_inverted_index.included. | 112 |
| abstract_inverted_index.location) | 168 |
| abstract_inverted_index.measures, | 59 |
| abstract_inverted_index.shrinkage | 18, 105, 144, 164 |
| abstract_inverted_index.therefore | 195 |
| abstract_inverted_index.Economics. | 217 |
| abstract_inverted_index.Psychology | 211 |
| abstract_inverted_index.associated | 187 |
| abstract_inverted_index.identified | 161 |
| abstract_inverted_index.imperative | 93 |
| abstract_inverted_index.literature | 117 |
| abstract_inverted_index.parameter. | 132 |
| abstract_inverted_index.population | 69 |
| abstract_inverted_index.Methodology | 113 |
| abstract_inverted_index.Replication | 209, 214 |
| abstract_inverted_index.descriptive | 61 |
| abstract_inverted_index.illustrated | 204 |
| abstract_inverted_index.large-scale | 1 |
| abstract_inverted_index.methodology | 134, 202 |
| abstract_inverted_index.quantifying | 122 |
| abstract_inverted_index.replication | 2, 67 |
| abstract_inverted_index.study-pairs | 47, 111 |
| abstract_inverted_index.substantial | 15 |
| abstract_inverted_index.(represented | 165, 173 |
| abstract_inverted_index.Experimental | 216 |
| abstract_inverted_index.aggregations | 86 |
| abstract_inverted_index.concerningly | 7 |
| abstract_inverted_index.contextualise | 196 |
| abstract_inverted_index.heterogeneity | 107, 124, 171 |
| abstract_inverted_index.heterogeneous | 89 |
| abstract_inverted_index.investigation | 139 |
| abstract_inverted_index.meta-analysis | 116 |
| abstract_inverted_index.substantially | 50 |
| abstract_inverted_index.heterogeneity. | 146 |
| abstract_inverted_index.location-scale | 152 |
| abstract_inverted_index.multiplicative | 131 |
| abstract_inverted_index.Meta-regression | 133 |
| abstract_inverted_index.characteristics | 162 |
| abstract_inverted_index.reproducibility | 9, 78, 192 |
| abstract_inverted_index.meta-regressions | 153 |
| abstract_inverted_index.original-replication | 46 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 2 |
| citation_normalized_percentile.value | 0.04812471 |
| citation_normalized_percentile.is_in_top_1_percent | False |
| citation_normalized_percentile.is_in_top_10_percent | True |