Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects Article Swipe
Rachel Heyard
,
Leonhard Held
·
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.31222/osf.io/e9nw2_v3
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.31222/osf.io/e9nw2_v3
Recent large-scale replication projects (RPs) have estimated concerningly low reproducibility rates. Further, they reported substantial degrees of shrinkage of effect size, where the replication effect size was found to be, on average, much smaller than the original effect size. Within these RPs, the included original-replication study-pairs can vary with respect to aspects of study design, outcome measures, and descriptive features of both original and replication study population and study team. This often results in between-study-pair heterogeneity, i.e., variation in effect size differences across study-pairs that goes beyond expected statistical variation. When broader claims about the reproducibility of an entire field are based on such heterogeneous data, it becomes imperative to conduct a rigorous analysis of the amount and sources of shrinkage and heterogeneity withinand between included study-pairs. Methodology from the meta-analysis literature provides an approach for quantifying the heterogeneity present in RPs with an additive or multiplicative parameter. Meta-regression methodology further allows for an investigation into the sources of shrinkage and heterogeneity. We propose the use of location-scale meta-regressions as a means to directly relate the identified characteristics with shrinkage (represented by the location) and heterogeneity (represented by the scale). This provides valuable insights into drivers and factors associated with high or low reproducibility rates and therefore contextualises results of PRs. The proposed methodology is illustrated using publicly available data from the Replication Project Psychology and the Replication Project Experimental Economics. All analysis scripts and data are available online.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- https://doi.org/10.31222/osf.io/e9nw2_v3
- https://osf.io/e9nw2_v3/download
- OA Status
- gold
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4410889995
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4410889995Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.31222/osf.io/e9nw2_v3Digital Object Identifier
- Title
-
Meta-regression to explain shrinkage and heterogeneity in large-scale replication projectsWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2025Year of publication
- Publication date
-
2025-05-30Full publication date if available
- Authors
-
Rachel Heyard, Leonhard HeldList of authors in order
- Landing page
-
https://doi.org/10.31222/osf.io/e9nw2_v3Publisher landing page
- PDF URL
-
https://osf.io/e9nw2_v3/downloadDirect link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
goldOpen access status per OpenAlex
- OA URL
-
https://osf.io/e9nw2_v3/downloadDirect OA link when available
- Concepts
-
Replication (statistics), Shrinkage, Meta-regression, Scale (ratio), Regression, Regression analysis, Econometrics, Economics, Statistics, Mathematics, Geography, Meta-analysis, Medicine, Cartography, Internal medicineTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4410889995 |
|---|---|
| doi | https://doi.org/10.31222/osf.io/e9nw2_v3 |
| ids.doi | https://doi.org/10.31222/osf.io/e9nw2_v3 |
| ids.openalex | https://openalex.org/W4410889995 |
| fwci | 0.0 |
| type | preprint |
| title | Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T11986 |
| topics[0].field.id | https://openalex.org/fields/18 |
| topics[0].field.display_name | Decision Sciences |
| topics[0].score | 0.8876000046730042 |
| topics[0].domain.id | https://openalex.org/domains/2 |
| topics[0].domain.display_name | Social Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1802 |
| topics[0].subfield.display_name | Information Systems and Management |
| topics[0].display_name | Scientific Computing and Data Management |
| topics[1].id | https://openalex.org/T11407 |
| topics[1].field.id | https://openalex.org/fields/22 |
| topics[1].field.display_name | Engineering |
| topics[1].score | 0.881600022315979 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/2204 |
| topics[1].subfield.display_name | Biomedical Engineering |
| topics[1].display_name | Innovative Microfluidic and Catalytic Techniques Innovation |
| topics[2].id | https://openalex.org/T10743 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.8442999720573425 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1712 |
| topics[2].subfield.display_name | Software |
| topics[2].display_name | Software Testing and Debugging Techniques |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C12590798 |
| concepts[0].level | 2 |
| concepts[0].score | 0.8764917850494385 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q3933199 |
| concepts[0].display_name | Replication (statistics) |
| concepts[1].id | https://openalex.org/C180145272 |
| concepts[1].level | 2 |
| concepts[1].score | 0.7139565944671631 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q7504144 |
| concepts[1].display_name | Shrinkage |
| concepts[2].id | https://openalex.org/C124936440 |
| concepts[2].level | 3 |
| concepts[2].score | 0.687359094619751 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q6822263 |
| concepts[2].display_name | Meta-regression |
| concepts[3].id | https://openalex.org/C2778755073 |
| concepts[3].level | 2 |
| concepts[3].score | 0.6170421838760376 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q10858537 |
| concepts[3].display_name | Scale (ratio) |
| concepts[4].id | https://openalex.org/C83546350 |
| concepts[4].level | 2 |
| concepts[4].score | 0.5566615462303162 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q1139051 |
| concepts[4].display_name | Regression |
| concepts[5].id | https://openalex.org/C152877465 |
| concepts[5].level | 2 |
| concepts[5].score | 0.45722121000289917 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q208042 |
| concepts[5].display_name | Regression analysis |
| concepts[6].id | https://openalex.org/C149782125 |
| concepts[6].level | 1 |
| concepts[6].score | 0.40836387872695923 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q160039 |
| concepts[6].display_name | Econometrics |
| concepts[7].id | https://openalex.org/C162324750 |
| concepts[7].level | 0 |
| concepts[7].score | 0.2744305431842804 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q8134 |
| concepts[7].display_name | Economics |
| concepts[8].id | https://openalex.org/C105795698 |
| concepts[8].level | 1 |
| concepts[8].score | 0.2738661766052246 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q12483 |
| concepts[8].display_name | Statistics |
| concepts[9].id | https://openalex.org/C33923547 |
| concepts[9].level | 0 |
| concepts[9].score | 0.19670650362968445 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q395 |
| concepts[9].display_name | Mathematics |
| concepts[10].id | https://openalex.org/C205649164 |
| concepts[10].level | 0 |
| concepts[10].score | 0.16999930143356323 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q1071 |
| concepts[10].display_name | Geography |
| concepts[11].id | https://openalex.org/C95190672 |
| concepts[11].level | 2 |
| concepts[11].score | 0.1670987904071808 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q815382 |
| concepts[11].display_name | Meta-analysis |
| concepts[12].id | https://openalex.org/C71924100 |
| concepts[12].level | 0 |
| concepts[12].score | 0.09365320205688477 |
| concepts[12].wikidata | https://www.wikidata.org/wiki/Q11190 |
| concepts[12].display_name | Medicine |
| concepts[13].id | https://openalex.org/C58640448 |
| concepts[13].level | 1 |
| concepts[13].score | 0.06695085763931274 |
| concepts[13].wikidata | https://www.wikidata.org/wiki/Q42515 |
| concepts[13].display_name | Cartography |
| concepts[14].id | https://openalex.org/C126322002 |
| concepts[14].level | 1 |
| concepts[14].score | 0.0 |
| concepts[14].wikidata | https://www.wikidata.org/wiki/Q11180 |
| concepts[14].display_name | Internal medicine |
| keywords[0].id | https://openalex.org/keywords/replication |
| keywords[0].score | 0.8764917850494385 |
| keywords[0].display_name | Replication (statistics) |
| keywords[1].id | https://openalex.org/keywords/shrinkage |
| keywords[1].score | 0.7139565944671631 |
| keywords[1].display_name | Shrinkage |
| keywords[2].id | https://openalex.org/keywords/meta-regression |
| keywords[2].score | 0.687359094619751 |
| keywords[2].display_name | Meta-regression |
| keywords[3].id | https://openalex.org/keywords/scale |
| keywords[3].score | 0.6170421838760376 |
| keywords[3].display_name | Scale (ratio) |
| keywords[4].id | https://openalex.org/keywords/regression |
| keywords[4].score | 0.5566615462303162 |
| keywords[4].display_name | Regression |
| keywords[5].id | https://openalex.org/keywords/regression-analysis |
| keywords[5].score | 0.45722121000289917 |
| keywords[5].display_name | Regression analysis |
| keywords[6].id | https://openalex.org/keywords/econometrics |
| keywords[6].score | 0.40836387872695923 |
| keywords[6].display_name | Econometrics |
| keywords[7].id | https://openalex.org/keywords/economics |
| keywords[7].score | 0.2744305431842804 |
| keywords[7].display_name | Economics |
| keywords[8].id | https://openalex.org/keywords/statistics |
| keywords[8].score | 0.2738661766052246 |
| keywords[8].display_name | Statistics |
| keywords[9].id | https://openalex.org/keywords/mathematics |
| keywords[9].score | 0.19670650362968445 |
| keywords[9].display_name | Mathematics |
| keywords[10].id | https://openalex.org/keywords/geography |
| keywords[10].score | 0.16999930143356323 |
| keywords[10].display_name | Geography |
| keywords[11].id | https://openalex.org/keywords/meta-analysis |
| keywords[11].score | 0.1670987904071808 |
| keywords[11].display_name | Meta-analysis |
| keywords[12].id | https://openalex.org/keywords/medicine |
| keywords[12].score | 0.09365320205688477 |
| keywords[12].display_name | Medicine |
| keywords[13].id | https://openalex.org/keywords/cartography |
| keywords[13].score | 0.06695085763931274 |
| keywords[13].display_name | Cartography |
| language | en |
| locations[0].id | doi:10.31222/osf.io/e9nw2_v3 |
| locations[0].is_oa | True |
| locations[0].source | |
| locations[0].license | cc-by |
| locations[0].pdf_url | https://osf.io/e9nw2_v3/download |
| locations[0].version | acceptedVersion |
| locations[0].raw_type | posted-content |
| locations[0].license_id | https://openalex.org/licenses/cc-by |
| locations[0].is_accepted | True |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | https://doi.org/10.31222/osf.io/e9nw2_v3 |
| indexed_in | crossref |
| authorships[0].author.id | https://openalex.org/A5085918982 |
| authorships[0].author.orcid | https://orcid.org/0000-0002-7531-4333 |
| authorships[0].author.display_name | Rachel Heyard |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Rachel Heyard |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5078043213 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-8686-5325 |
| authorships[1].author.display_name | Leonhard Held |
| authorships[1].author_position | last |
| authorships[1].raw_author_name | Leonhard Held |
| authorships[1].is_corresponding | False |
| has_content.pdf | True |
| has_content.grobid_xml | True |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://osf.io/e9nw2_v3/download |
| open_access.oa_status | gold |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T03:46:38.306776 |
| primary_topic.id | https://openalex.org/T11986 |
| primary_topic.field.id | https://openalex.org/fields/18 |
| primary_topic.field.display_name | Decision Sciences |
| primary_topic.score | 0.8876000046730042 |
| primary_topic.domain.id | https://openalex.org/domains/2 |
| primary_topic.domain.display_name | Social Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1802 |
| primary_topic.subfield.display_name | Information Systems and Management |
| primary_topic.display_name | Scientific Computing and Data Management |
| related_works | https://openalex.org/W3089004032, https://openalex.org/W31220157, https://openalex.org/W2312753042, https://openalex.org/W4289356671, https://openalex.org/W2389155397, https://openalex.org/W2165884543, https://openalex.org/W3186837933, https://openalex.org/W2368989808, https://openalex.org/W1969346022, https://openalex.org/W2034959125 |
| cited_by_count | 0 |
| locations_count | 1 |
| best_oa_location.id | doi:10.31222/osf.io/e9nw2_v3 |
| best_oa_location.is_oa | True |
| best_oa_location.source | |
| best_oa_location.license | cc-by |
| best_oa_location.pdf_url | https://osf.io/e9nw2_v3/download |
| best_oa_location.version | acceptedVersion |
| best_oa_location.raw_type | posted-content |
| best_oa_location.license_id | https://openalex.org/licenses/cc-by |
| best_oa_location.is_accepted | True |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | https://doi.org/10.31222/osf.io/e9nw2_v3 |
| primary_location.id | doi:10.31222/osf.io/e9nw2_v3 |
| primary_location.is_oa | True |
| primary_location.source | |
| primary_location.license | cc-by |
| primary_location.pdf_url | https://osf.io/e9nw2_v3/download |
| primary_location.version | acceptedVersion |
| primary_location.raw_type | posted-content |
| primary_location.license_id | https://openalex.org/licenses/cc-by |
| primary_location.is_accepted | True |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | https://doi.org/10.31222/osf.io/e9nw2_v3 |
| publication_date | 2025-05-30 |
| publication_year | 2025 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 111, 170 |
| abstract_inverted_index.We | 162 |
| abstract_inverted_index.an | 97, 133, 143, 153 |
| abstract_inverted_index.as | 169 |
| abstract_inverted_index.by | 181, 187 |
| abstract_inverted_index.in | 73, 78, 140 |
| abstract_inverted_index.is | 214 |
| abstract_inverted_index.it | 106 |
| abstract_inverted_index.of | 16, 18, 52, 60, 96, 114, 119, 158, 166, 209 |
| abstract_inverted_index.on | 30, 102 |
| abstract_inverted_index.or | 145, 201 |
| abstract_inverted_index.to | 28, 50, 109, 172 |
| abstract_inverted_index.All | 231 |
| abstract_inverted_index.RPs | 141 |
| abstract_inverted_index.The | 211 |
| abstract_inverted_index.and | 57, 63, 67, 117, 121, 160, 184, 196, 205, 225, 234 |
| abstract_inverted_index.are | 100, 236 |
| abstract_inverted_index.be, | 29 |
| abstract_inverted_index.can | 46 |
| abstract_inverted_index.for | 135, 152 |
| abstract_inverted_index.low | 8, 202 |
| abstract_inverted_index.the | 22, 35, 42, 94, 115, 129, 137, 156, 164, 175, 182, 188, 221, 226 |
| abstract_inverted_index.use | 165 |
| abstract_inverted_index.was | 26 |
| abstract_inverted_index.PRs. | 210 |
| abstract_inverted_index.RPs, | 41 |
| abstract_inverted_index.This | 70, 190 |
| abstract_inverted_index.When | 90 |
| abstract_inverted_index.both | 61 |
| abstract_inverted_index.data | 219, 235 |
| abstract_inverted_index.from | 128, 220 |
| abstract_inverted_index.goes | 85 |
| abstract_inverted_index.have | 5 |
| abstract_inverted_index.high | 200 |
| abstract_inverted_index.into | 155, 194 |
| abstract_inverted_index.much | 32 |
| abstract_inverted_index.size | 25, 80 |
| abstract_inverted_index.such | 103 |
| abstract_inverted_index.than | 34 |
| abstract_inverted_index.that | 84 |
| abstract_inverted_index.they | 12 |
| abstract_inverted_index.vary | 47 |
| abstract_inverted_index.with | 48, 142, 178, 199 |
| abstract_inverted_index.(RPs) | 4 |
| abstract_inverted_index.about | 93 |
| abstract_inverted_index.based | 101 |
| abstract_inverted_index.data, | 105 |
| abstract_inverted_index.field | 99 |
| abstract_inverted_index.found | 27 |
| abstract_inverted_index.i.e., | 76 |
| abstract_inverted_index.means | 171 |
| abstract_inverted_index.often | 71 |
| abstract_inverted_index.rates | 204 |
| abstract_inverted_index.size, | 20 |
| abstract_inverted_index.size. | 38 |
| abstract_inverted_index.study | 53, 65, 68 |
| abstract_inverted_index.team. | 69 |
| abstract_inverted_index.these | 40 |
| abstract_inverted_index.using | 216 |
| abstract_inverted_index.where | 21 |
| abstract_inverted_index.Recent | 0 |
| abstract_inverted_index.Within | 39 |
| abstract_inverted_index.across | 82 |
| abstract_inverted_index.allows | 151 |
| abstract_inverted_index.amount | 116 |
| abstract_inverted_index.beyond | 86 |
| abstract_inverted_index.claims | 92 |
| abstract_inverted_index.effect | 19, 24, 37, 79 |
| abstract_inverted_index.entire | 98 |
| abstract_inverted_index.rates. | 10 |
| abstract_inverted_index.relate | 174 |
| abstract_inverted_index.Project | 223, 228 |
| abstract_inverted_index.aspects | 51 |
| abstract_inverted_index.becomes | 107 |
| abstract_inverted_index.between | 124 |
| abstract_inverted_index.broader | 91 |
| abstract_inverted_index.conduct | 110 |
| abstract_inverted_index.degrees | 15 |
| abstract_inverted_index.design, | 54 |
| abstract_inverted_index.drivers | 195 |
| abstract_inverted_index.factors | 197 |
| abstract_inverted_index.further | 150 |
| abstract_inverted_index.online. | 238 |
| abstract_inverted_index.outcome | 55 |
| abstract_inverted_index.present | 139 |
| abstract_inverted_index.propose | 163 |
| abstract_inverted_index.respect | 49 |
| abstract_inverted_index.results | 72, 208 |
| abstract_inverted_index.scale). | 189 |
| abstract_inverted_index.scripts | 233 |
| abstract_inverted_index.smaller | 33 |
| abstract_inverted_index.sources | 118, 157 |
| abstract_inverted_index.Further, | 11 |
| abstract_inverted_index.additive | 144 |
| abstract_inverted_index.analysis | 113, 232 |
| abstract_inverted_index.approach | 134 |
| abstract_inverted_index.average, | 31 |
| abstract_inverted_index.directly | 173 |
| abstract_inverted_index.expected | 87 |
| abstract_inverted_index.features | 59 |
| abstract_inverted_index.included | 43, 125 |
| abstract_inverted_index.insights | 193 |
| abstract_inverted_index.original | 36, 62 |
| abstract_inverted_index.projects | 3 |
| abstract_inverted_index.proposed | 212 |
| abstract_inverted_index.provides | 132, 191 |
| abstract_inverted_index.publicly | 217 |
| abstract_inverted_index.reported | 13 |
| abstract_inverted_index.rigorous | 112 |
| abstract_inverted_index.valuable | 192 |
| abstract_inverted_index.available | 218, 237 |
| abstract_inverted_index.estimated | 6 |
| abstract_inverted_index.location) | 183 |
| abstract_inverted_index.measures, | 56 |
| abstract_inverted_index.shrinkage | 17, 120, 159, 179 |
| abstract_inverted_index.therefore | 206 |
| abstract_inverted_index.variation | 77 |
| abstract_inverted_index.withinand | 123 |
| abstract_inverted_index.Economics. | 230 |
| abstract_inverted_index.Psychology | 224 |
| abstract_inverted_index.associated | 198 |
| abstract_inverted_index.identified | 176 |
| abstract_inverted_index.imperative | 108 |
| abstract_inverted_index.literature | 131 |
| abstract_inverted_index.parameter. | 147 |
| abstract_inverted_index.population | 66 |
| abstract_inverted_index.variation. | 89 |
| abstract_inverted_index.Methodology | 127 |
| abstract_inverted_index.Replication | 222, 227 |
| abstract_inverted_index.descriptive | 58 |
| abstract_inverted_index.differences | 81 |
| abstract_inverted_index.illustrated | 215 |
| abstract_inverted_index.large-scale | 1 |
| abstract_inverted_index.methodology | 149, 213 |
| abstract_inverted_index.quantifying | 136 |
| abstract_inverted_index.replication | 2, 23, 64 |
| abstract_inverted_index.statistical | 88 |
| abstract_inverted_index.study-pairs | 45, 83 |
| abstract_inverted_index.substantial | 14 |
| abstract_inverted_index.(represented | 180, 186 |
| abstract_inverted_index.Experimental | 229 |
| abstract_inverted_index.concerningly | 7 |
| abstract_inverted_index.study-pairs. | 126 |
| abstract_inverted_index.heterogeneity | 122, 138, 185 |
| abstract_inverted_index.heterogeneous | 104 |
| abstract_inverted_index.investigation | 154 |
| abstract_inverted_index.meta-analysis | 130 |
| abstract_inverted_index.contextualises | 207 |
| abstract_inverted_index.heterogeneity, | 75 |
| abstract_inverted_index.heterogeneity. | 161 |
| abstract_inverted_index.location-scale | 167 |
| abstract_inverted_index.multiplicative | 146 |
| abstract_inverted_index.Meta-regression | 148 |
| abstract_inverted_index.characteristics | 177 |
| abstract_inverted_index.reproducibility | 9, 95, 203 |
| abstract_inverted_index.meta-regressions | 168 |
| abstract_inverted_index.between-study-pair | 74 |
| abstract_inverted_index.original-replication | 44 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 2 |
| citation_normalized_percentile.value | 0.1547837 |
| citation_normalized_percentile.is_in_top_1_percent | False |
| citation_normalized_percentile.is_in_top_10_percent | True |