The influence of study selection criteria on statistical conclusion validity: A simulation study Article Swipe

View

Allyson Hanson , Jessica A. R. Logan , Jeffrey Shero ·

YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.31234/osf.io/j6t3h_v1

Tier 2 interventions are designed for students who are not making sufficient progress despite receiving whole-group instruction. These interventions target a specific population, necessitating clear eligibility criteria for screening participants. While efficacy studies of Tier 2 interventions often generalize their findings to struggling readers, the eligibility criteria for selecting study samples can vary across studies. In our simulation study, we compared effect size estimates from different samples, each screened using commonly used eligibility criteria. Despite each method accurately recovering the true effect on average across 500 iterations, the range of estimates showed substantial variation, with significant under- and overestimation in individual iterations. We concluded that the perceived effectiveness of Tier 2 interventions may fluctuate based on how the sample is screened for eligibility. We propose future research to address this methodological challenge while balancing the practical demands of Tier 2 intervention studies.

Related Topics

Concepts

Selection (genetic algorithm) Statistics Econometrics Computer science Psychology Mathematics Machine learning

Metadata

Type: preprint
Language: en
Landing Page: https://doi.org/10.31234/osf.io/j6t3h_v1
OA Status: gold
Related Works: 10
OpenAlex ID: https://openalex.org/W4407898844

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4407898844

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.31234/osf.io/j6t3h_v1

Digital Object Identifier
Title: The influence of study selection criteria on statistical conclusion validity: A simulation study

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2025

Year of publication
Publication date: 2025-02-24

Full publication date if available
Authors: Allyson Hanson, Jessica A. R. Logan, Jeffrey Shero

List of authors in order
Landing page: https://doi.org/10.31234/osf.io/j6t3h_v1

Publisher landing page
Open access: Yes

Whether a free full text is available
OA status: gold

Open access status per OpenAlex
OA URL: https://doi.org/10.31234/osf.io/j6t3h_v1

Direct OA link when available
Concepts: Selection (genetic algorithm), Statistics, Econometrics, Computer science, Psychology, Mathematics, Machine learning

Top concepts (fields/topics) attached by OpenAlex
Cited by: 0

Total citation count in OpenAlex
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4407898844
doi	https://doi.org/10.31234/osf.io/j6t3h_v1
ids.doi	https://doi.org/10.31234/osf.io/j6t3h_v1
ids.openalex	https://openalex.org/W4407898844
fwci	0.0
type	preprint
title	The influence of study selection criteria on statistical conclusion validity: A simulation study
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T11918
topics[0].field.id	https://openalex.org/fields/18
topics[0].field.display_name	Decision Sciences
topics[0].score	0.23360000550746918
topics[0].domain.id	https://openalex.org/domains/2
topics[0].domain.display_name	Social Sciences
topics[0].subfield.id	https://openalex.org/subfields/1803
topics[0].subfield.display_name	Management Science and Operations Research
topics[0].display_name	Forecasting Techniques and Applications
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C81917197
concepts[0].level	2
concepts[0].score	0.731323778629303
concepts[0].wikidata	https://www.wikidata.org/wiki/Q628760
concepts[0].display_name	Selection (genetic algorithm)
concepts[1].id	https://openalex.org/C105795698
concepts[1].level	1
concepts[1].score	0.5422350764274597
concepts[1].wikidata	https://www.wikidata.org/wiki/Q12483
concepts[1].display_name	Statistics
concepts[2].id	https://openalex.org/C149782125
concepts[2].level	1
concepts[2].score	0.4353322386741638
concepts[2].wikidata	https://www.wikidata.org/wiki/Q160039
concepts[2].display_name	Econometrics
concepts[3].id	https://openalex.org/C41008148
concepts[3].level	0
concepts[3].score	0.428523451089859
concepts[3].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[3].display_name	Computer science
concepts[4].id	https://openalex.org/C15744967
concepts[4].level	0
concepts[4].score	0.3749344050884247
concepts[4].wikidata	https://www.wikidata.org/wiki/Q9418
concepts[4].display_name	Psychology
concepts[5].id	https://openalex.org/C33923547
concepts[5].level	0
concepts[5].score	0.27736133337020874
concepts[5].wikidata	https://www.wikidata.org/wiki/Q395
concepts[5].display_name	Mathematics
concepts[6].id	https://openalex.org/C119857082
concepts[6].level	1
concepts[6].score	0.17073053121566772
concepts[6].wikidata	https://www.wikidata.org/wiki/Q2539
concepts[6].display_name	Machine learning
keywords[0].id	https://openalex.org/keywords/selection
keywords[0].score	0.731323778629303
keywords[0].display_name	Selection (genetic algorithm)
keywords[1].id	https://openalex.org/keywords/statistics
keywords[1].score	0.5422350764274597
keywords[1].display_name	Statistics
keywords[2].id	https://openalex.org/keywords/econometrics
keywords[2].score	0.4353322386741638
keywords[2].display_name	Econometrics
keywords[3].id	https://openalex.org/keywords/computer-science
keywords[3].score	0.428523451089859
keywords[3].display_name	Computer science
keywords[4].id	https://openalex.org/keywords/psychology
keywords[4].score	0.3749344050884247
keywords[4].display_name	Psychology
keywords[5].id	https://openalex.org/keywords/mathematics
keywords[5].score	0.27736133337020874
keywords[5].display_name	Mathematics
keywords[6].id	https://openalex.org/keywords/machine-learning
keywords[6].score	0.17073053121566772
keywords[6].display_name	Machine learning
language	en
locations[0].id	doi:10.31234/osf.io/j6t3h_v1
locations[0].is_oa	True
locations[0].source
locations[0].license	cc-by
locations[0].pdf_url
locations[0].version	acceptedVersion
locations[0].raw_type	posted-content
locations[0].license_id	https://openalex.org/licenses/cc-by
locations[0].is_accepted	True
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	https://doi.org/10.31234/osf.io/j6t3h_v1
indexed_in	crossref
authorships[0].author.id	https://openalex.org/A5076437251
authorships[0].author.orcid	https://orcid.org/0009-0000-0891-6038
authorships[0].author.display_name	Allyson Hanson
authorships[0].author_position	first
authorships[0].raw_author_name	Allyson Hanson
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5058436364
authorships[1].author.orcid	https://orcid.org/0000-0003-3113-4346
authorships[1].author.display_name	Jessica A. R. Logan
authorships[1].author_position	middle
authorships[1].raw_author_name	Jessica A. R. Logan
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5056949372
authorships[2].author.orcid	https://orcid.org/0000-0002-2180-8979
authorships[2].author.display_name	Jeffrey Shero
authorships[2].author_position	last
authorships[2].raw_author_name	Jeffrey Shero
authorships[2].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://doi.org/10.31234/osf.io/j6t3h_v1
open_access.oa_status	gold
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	The influence of study selection criteria on statistical conclusion validity: A simulation study
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T03:46:38.306776
primary_topic.id	https://openalex.org/T11918
primary_topic.field.id	https://openalex.org/fields/18
primary_topic.field.display_name	Decision Sciences
primary_topic.score	0.23360000550746918
primary_topic.domain.id	https://openalex.org/domains/2
primary_topic.domain.display_name	Social Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1803
primary_topic.subfield.display_name	Management Science and Operations Research
primary_topic.display_name	Forecasting Techniques and Applications
related_works	https://openalex.org/W4391375266, https://openalex.org/W2899084033, https://openalex.org/W2748952813, https://openalex.org/W2390279801, https://openalex.org/W4391913857, https://openalex.org/W2358668433, https://openalex.org/W4396701345, https://openalex.org/W2376932109, https://openalex.org/W2001405890, https://openalex.org/W4396696052
cited_by_count	0
locations_count	1
best_oa_location.id	doi:10.31234/osf.io/j6t3h_v1
best_oa_location.is_oa	True
best_oa_location.source
best_oa_location.license	cc-by
best_oa_location.pdf_url
best_oa_location.version	acceptedVersion
best_oa_location.raw_type	posted-content
best_oa_location.license_id	https://openalex.org/licenses/cc-by
best_oa_location.is_accepted	True
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	https://doi.org/10.31234/osf.io/j6t3h_v1
primary_location.id	doi:10.31234/osf.io/j6t3h_v1
primary_location.is_oa	True
primary_location.source
primary_location.license	cc-by
primary_location.pdf_url
primary_location.version	acceptedVersion
primary_location.raw_type	posted-content
primary_location.license_id	https://openalex.org/licenses/cc-by
primary_location.is_accepted	True
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	https://doi.org/10.31234/osf.io/j6t3h_v1
publication_date	2025-02-24
publication_year	2025
referenced_works_count	0
abstract_inverted_index.2	1, 35, 110, 139
abstract_inverted_index.a	20
abstract_inverted_index.In	55
abstract_inverted_index.We	102, 123
abstract_inverted_index.in	99
abstract_inverted_index.is	119
abstract_inverted_index.of	33, 89, 108, 137
abstract_inverted_index.on	82, 115
abstract_inverted_index.to	41, 127
abstract_inverted_index.we	59
abstract_inverted_index.500	85
abstract_inverted_index.and	97
abstract_inverted_index.are	3, 8
abstract_inverted_index.can	51
abstract_inverted_index.for	5, 27, 47, 121
abstract_inverted_index.how	116
abstract_inverted_index.may	112
abstract_inverted_index.not	9
abstract_inverted_index.our	56
abstract_inverted_index.the	44, 79, 87, 105, 117, 134
abstract_inverted_index.who	7
abstract_inverted_index.Tier	0, 34, 109, 138
abstract_inverted_index.each	67, 75
abstract_inverted_index.from	64
abstract_inverted_index.size	62
abstract_inverted_index.that	104
abstract_inverted_index.this	129
abstract_inverted_index.true	80
abstract_inverted_index.used	71
abstract_inverted_index.vary	52
abstract_inverted_index.with	94
abstract_inverted_index.These	17
abstract_inverted_index.While	30
abstract_inverted_index.based	114
abstract_inverted_index.clear	24
abstract_inverted_index.often	37
abstract_inverted_index.range	88
abstract_inverted_index.study	49
abstract_inverted_index.their	39
abstract_inverted_index.using	69
abstract_inverted_index.while	132
abstract_inverted_index.across	53, 84
abstract_inverted_index.effect	61, 81
abstract_inverted_index.future	125
abstract_inverted_index.making	10
abstract_inverted_index.method	76
abstract_inverted_index.sample	118
abstract_inverted_index.showed	91
abstract_inverted_index.study,	58
abstract_inverted_index.target	19
abstract_inverted_index.under-	96
abstract_inverted_index.Despite	74
abstract_inverted_index.address	128
abstract_inverted_index.average	83
abstract_inverted_index.demands	136
abstract_inverted_index.despite	13
abstract_inverted_index.propose	124
abstract_inverted_index.samples	50
abstract_inverted_index.studies	32
abstract_inverted_index.commonly	70
abstract_inverted_index.compared	60
abstract_inverted_index.criteria	26, 46
abstract_inverted_index.designed	4
abstract_inverted_index.efficacy	31
abstract_inverted_index.findings	40
abstract_inverted_index.progress	12
abstract_inverted_index.readers,	43
abstract_inverted_index.research	126
abstract_inverted_index.samples,	66
abstract_inverted_index.screened	68, 120
abstract_inverted_index.specific	21
abstract_inverted_index.students	6
abstract_inverted_index.studies.	54, 141
abstract_inverted_index.balancing	133
abstract_inverted_index.challenge	131
abstract_inverted_index.concluded	103
abstract_inverted_index.criteria.	73
abstract_inverted_index.different	65
abstract_inverted_index.estimates	63, 90
abstract_inverted_index.fluctuate	113
abstract_inverted_index.perceived	106
abstract_inverted_index.practical	135
abstract_inverted_index.receiving	14
abstract_inverted_index.screening	28
abstract_inverted_index.selecting	48
abstract_inverted_index.accurately	77
abstract_inverted_index.generalize	38
abstract_inverted_index.individual	100
abstract_inverted_index.recovering	78
abstract_inverted_index.simulation	57
abstract_inverted_index.struggling	42
abstract_inverted_index.sufficient	11
abstract_inverted_index.variation,	93
abstract_inverted_index.eligibility	25, 45, 72
abstract_inverted_index.iterations,	86
abstract_inverted_index.iterations.	101
abstract_inverted_index.population,	22
abstract_inverted_index.significant	95
abstract_inverted_index.substantial	92
abstract_inverted_index.whole-group	15
abstract_inverted_index.eligibility.	122
abstract_inverted_index.instruction.	16
abstract_inverted_index.intervention	140
abstract_inverted_index.effectiveness	107
abstract_inverted_index.interventions	2, 18, 36, 111
abstract_inverted_index.necessitating	23
abstract_inverted_index.participants.	29
abstract_inverted_index.methodological	130
abstract_inverted_index.overestimation	98
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	3
citation_normalized_percentile.value	0.06200621
citation_normalized_percentile.is_in_top_1_percent	False
citation_normalized_percentile.is_in_top_10_percent	True