Evaluation of clinical prediction models (part 3): calculating the sample size required for an external validation study Article Swipe
Richard D Riley
,
Kym I E Snell
,
Lucinda Archer
,
Joie Ensor
,
Thomas P. A. Debray
,
Ben Van Calster
,
Maarten van Smeden
,
Gary S. Collins
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.1136/bmj-2023-074821
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.1136/bmj-2023-074821
An external validation study evaluates the performance of a prediction model in new data, but many of these studies are too small to provide reliable answers. In the third article of their series on model evaluation, Riley and colleagues describe how to calculate the sample size required for external validation studies, and propose to avoid rules of thumb by tailoring calculations to the model and setting at hand.
Related Topics
Concepts
Metadata
- Type
- article
- Language
- en
- Landing Page
- https://doi.org/10.1136/bmj-2023-074821
- https://www.bmj.com/content/bmj/384/bmj-2023-074821.full.pdf
- OA Status
- hybrid
- Cited By
- 100
- References
- 32
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4391100790
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4391100790Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.1136/bmj-2023-074821Digital Object Identifier
- Title
-
Evaluation of clinical prediction models (part 3): calculating the sample size required for an external validation studyWork title
- Type
-
articleOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-01-22Full publication date if available
- Authors
-
Richard D Riley, Kym I E Snell, Lucinda Archer, Joie Ensor, Thomas P. A. Debray, Ben Van Calster, Maarten van Smeden, Gary S. CollinsList of authors in order
- Landing page
-
https://doi.org/10.1136/bmj-2023-074821Publisher landing page
- PDF URL
-
https://www.bmj.com/content/bmj/384/bmj-2023-074821.full.pdfDirect link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
hybridOpen access status per OpenAlex
- OA URL
-
https://www.bmj.com/content/bmj/384/bmj-2023-074821.full.pdfDirect OA link when available
- Concepts
-
Sample size determination, Computer science, Rule of thumb, Sample (material), Data mining, Model validation, Predictive modelling, Machine learning, Statistics, Data science, Algorithm, Mathematics, Chromatography, ChemistryTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
100Total citation count in OpenAlex
- Citations by year (recent)
-
2025: 71, 2024: 29Per-year citation counts (last 5 years)
- References (count)
-
32Number of works referenced by this work
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4391100790 |
|---|---|
| doi | https://doi.org/10.1136/bmj-2023-074821 |
| ids.doi | https://doi.org/10.1136/bmj-2023-074821 |
| ids.pmid | https://pubmed.ncbi.nlm.nih.gov/38253388 |
| ids.openalex | https://openalex.org/W4391100790 |
| fwci | 87.81568212 |
| mesh[0].qualifier_ui | |
| mesh[0].descriptor_ui | D006801 |
| mesh[0].is_major_topic | False |
| mesh[0].qualifier_name | |
| mesh[0].descriptor_name | Humans |
| mesh[1].qualifier_ui | |
| mesh[1].descriptor_ui | D018401 |
| mesh[1].is_major_topic | False |
| mesh[1].qualifier_name | |
| mesh[1].descriptor_name | Sample Size |
| mesh[2].qualifier_ui | |
| mesh[2].descriptor_ui | D015233 |
| mesh[2].is_major_topic | True |
| mesh[2].qualifier_name | |
| mesh[2].descriptor_name | Models, Statistical |
| mesh[3].qualifier_ui | |
| mesh[3].descriptor_ui | D011379 |
| mesh[3].is_major_topic | False |
| mesh[3].qualifier_name | |
| mesh[3].descriptor_name | Prognosis |
| mesh[4].qualifier_ui | |
| mesh[4].descriptor_ui | D006801 |
| mesh[4].is_major_topic | False |
| mesh[4].qualifier_name | |
| mesh[4].descriptor_name | Humans |
| mesh[5].qualifier_ui | |
| mesh[5].descriptor_ui | D018401 |
| mesh[5].is_major_topic | False |
| mesh[5].qualifier_name | |
| mesh[5].descriptor_name | Sample Size |
| mesh[6].qualifier_ui | |
| mesh[6].descriptor_ui | D015233 |
| mesh[6].is_major_topic | True |
| mesh[6].qualifier_name | |
| mesh[6].descriptor_name | Models, Statistical |
| mesh[7].qualifier_ui | |
| mesh[7].descriptor_ui | D011379 |
| mesh[7].is_major_topic | False |
| mesh[7].qualifier_name | |
| mesh[7].descriptor_name | Prognosis |
| mesh[8].qualifier_ui | |
| mesh[8].descriptor_ui | D006801 |
| mesh[8].is_major_topic | False |
| mesh[8].qualifier_name | |
| mesh[8].descriptor_name | Humans |
| mesh[9].qualifier_ui | |
| mesh[9].descriptor_ui | D018401 |
| mesh[9].is_major_topic | False |
| mesh[9].qualifier_name | |
| mesh[9].descriptor_name | Sample Size |
| mesh[10].qualifier_ui | |
| mesh[10].descriptor_ui | D015233 |
| mesh[10].is_major_topic | True |
| mesh[10].qualifier_name | |
| mesh[10].descriptor_name | Models, Statistical |
| mesh[11].qualifier_ui | |
| mesh[11].descriptor_ui | D011379 |
| mesh[11].is_major_topic | False |
| mesh[11].qualifier_name | |
| mesh[11].descriptor_name | Prognosis |
| mesh[12].qualifier_ui | |
| mesh[12].descriptor_ui | D006801 |
| mesh[12].is_major_topic | False |
| mesh[12].qualifier_name | |
| mesh[12].descriptor_name | Humans |
| mesh[13].qualifier_ui | |
| mesh[13].descriptor_ui | D018401 |
| mesh[13].is_major_topic | False |
| mesh[13].qualifier_name | |
| mesh[13].descriptor_name | Sample Size |
| mesh[14].qualifier_ui | |
| mesh[14].descriptor_ui | D015233 |
| mesh[14].is_major_topic | True |
| mesh[14].qualifier_name | |
| mesh[14].descriptor_name | Models, Statistical |
| mesh[15].qualifier_ui | |
| mesh[15].descriptor_ui | D011379 |
| mesh[15].is_major_topic | False |
| mesh[15].qualifier_name | |
| mesh[15].descriptor_name | Prognosis |
| mesh[16].qualifier_ui | |
| mesh[16].descriptor_ui | D006801 |
| mesh[16].is_major_topic | False |
| mesh[16].qualifier_name | |
| mesh[16].descriptor_name | Humans |
| mesh[17].qualifier_ui | |
| mesh[17].descriptor_ui | D018401 |
| mesh[17].is_major_topic | False |
| mesh[17].qualifier_name | |
| mesh[17].descriptor_name | Sample Size |
| mesh[18].qualifier_ui | |
| mesh[18].descriptor_ui | D015233 |
| mesh[18].is_major_topic | True |
| mesh[18].qualifier_name | |
| mesh[18].descriptor_name | Models, Statistical |
| mesh[19].qualifier_ui | |
| mesh[19].descriptor_ui | D011379 |
| mesh[19].is_major_topic | False |
| mesh[19].qualifier_name | |
| mesh[19].descriptor_name | Prognosis |
| mesh[20].qualifier_ui | |
| mesh[20].descriptor_ui | D006801 |
| mesh[20].is_major_topic | False |
| mesh[20].qualifier_name | |
| mesh[20].descriptor_name | Humans |
| mesh[21].qualifier_ui | |
| mesh[21].descriptor_ui | D018401 |
| mesh[21].is_major_topic | False |
| mesh[21].qualifier_name | |
| mesh[21].descriptor_name | Sample Size |
| mesh[22].qualifier_ui | |
| mesh[22].descriptor_ui | D015233 |
| mesh[22].is_major_topic | True |
| mesh[22].qualifier_name | |
| mesh[22].descriptor_name | Models, Statistical |
| mesh[23].qualifier_ui | |
| mesh[23].descriptor_ui | D011379 |
| mesh[23].is_major_topic | False |
| mesh[23].qualifier_name | |
| mesh[23].descriptor_name | Prognosis |
| mesh[24].qualifier_ui | |
| mesh[24].descriptor_ui | D006801 |
| mesh[24].is_major_topic | False |
| mesh[24].qualifier_name | |
| mesh[24].descriptor_name | Humans |
| mesh[25].qualifier_ui | |
| mesh[25].descriptor_ui | D018401 |
| mesh[25].is_major_topic | False |
| mesh[25].qualifier_name | |
| mesh[25].descriptor_name | Sample Size |
| mesh[26].qualifier_ui | |
| mesh[26].descriptor_ui | D015233 |
| mesh[26].is_major_topic | True |
| mesh[26].qualifier_name | |
| mesh[26].descriptor_name | Models, Statistical |
| mesh[27].qualifier_ui | |
| mesh[27].descriptor_ui | D011379 |
| mesh[27].is_major_topic | False |
| mesh[27].qualifier_name | |
| mesh[27].descriptor_name | Prognosis |
| mesh[28].qualifier_ui | |
| mesh[28].descriptor_ui | D006801 |
| mesh[28].is_major_topic | False |
| mesh[28].qualifier_name | |
| mesh[28].descriptor_name | Humans |
| mesh[29].qualifier_ui | |
| mesh[29].descriptor_ui | D018401 |
| mesh[29].is_major_topic | False |
| mesh[29].qualifier_name | |
| mesh[29].descriptor_name | Sample Size |
| mesh[30].qualifier_ui | |
| mesh[30].descriptor_ui | D015233 |
| mesh[30].is_major_topic | True |
| mesh[30].qualifier_name | |
| mesh[30].descriptor_name | Models, Statistical |
| mesh[31].qualifier_ui | |
| mesh[31].descriptor_ui | D011379 |
| mesh[31].is_major_topic | False |
| mesh[31].qualifier_name | |
| mesh[31].descriptor_name | Prognosis |
| mesh[32].qualifier_ui | |
| mesh[32].descriptor_ui | D006801 |
| mesh[32].is_major_topic | False |
| mesh[32].qualifier_name | |
| mesh[32].descriptor_name | Humans |
| mesh[33].qualifier_ui | |
| mesh[33].descriptor_ui | D018401 |
| mesh[33].is_major_topic | False |
| mesh[33].qualifier_name | |
| mesh[33].descriptor_name | Sample Size |
| mesh[34].qualifier_ui | |
| mesh[34].descriptor_ui | D015233 |
| mesh[34].is_major_topic | True |
| mesh[34].qualifier_name | |
| mesh[34].descriptor_name | Models, Statistical |
| mesh[35].qualifier_ui | |
| mesh[35].descriptor_ui | D011379 |
| mesh[35].is_major_topic | False |
| mesh[35].qualifier_name | |
| mesh[35].descriptor_name | Prognosis |
| type | article |
| title | Evaluation of clinical prediction models (part 3): calculating the sample size required for an external validation study |
| awards[0].id | https://openalex.org/G194756405 |
| awards[0].funder_id | https://openalex.org/F4320334626 |
| awards[0].display_name | |
| awards[0].funder_award_id | MR/V038168/1 |
| awards[0].funder_display_name | Medical Research Council |
| awards[1].id | https://openalex.org/G2515644237 |
| awards[1].funder_id | https://openalex.org/F4320334627 |
| awards[1].display_name | |
| awards[1].funder_award_id | EP/Y018516/1 |
| awards[1].funder_display_name | Engineering and Physical Sciences Research Council |
| biblio.issue | |
| biblio.volume | 384 |
| biblio.last_page | e074821 |
| biblio.first_page | e074821 |
| topics[0].id | https://openalex.org/T10206 |
| topics[0].field.id | https://openalex.org/fields/18 |
| topics[0].field.display_name | Decision Sciences |
| topics[0].score | 0.9908000230789185 |
| topics[0].domain.id | https://openalex.org/domains/2 |
| topics[0].domain.display_name | Social Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1804 |
| topics[0].subfield.display_name | Statistics, Probability and Uncertainty |
| topics[0].display_name | Meta-analysis and systematic reviews |
| topics[1].id | https://openalex.org/T10218 |
| topics[1].field.id | https://openalex.org/fields/27 |
| topics[1].field.display_name | Medicine |
| topics[1].score | 0.9890000224113464 |
| topics[1].domain.id | https://openalex.org/domains/4 |
| topics[1].domain.display_name | Health Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/2713 |
| topics[1].subfield.display_name | Epidemiology |
| topics[1].display_name | Sepsis Diagnosis and Treatment |
| topics[2].id | https://openalex.org/T10804 |
| topics[2].field.id | https://openalex.org/fields/20 |
| topics[2].field.display_name | Economics, Econometrics and Finance |
| topics[2].score | 0.9814000129699707 |
| topics[2].domain.id | https://openalex.org/domains/2 |
| topics[2].domain.display_name | Social Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/2002 |
| topics[2].subfield.display_name | Economics and Econometrics |
| topics[2].display_name | Health Systems, Economic Evaluations, Quality of Life |
| funders[0].id | https://openalex.org/F4320319985 |
| funders[0].ror | https://ror.org/054225q67 |
| funders[0].display_name | Cancer Research UK |
| funders[1].id | https://openalex.org/F4320322308 |
| funders[1].ror | https://ror.org/05f950310 |
| funders[1].display_name | KU Leuven |
| funders[2].id | https://openalex.org/F4320334626 |
| funders[2].ror | https://ror.org/03x94j517 |
| funders[2].display_name | Medical Research Council |
| funders[3].id | https://openalex.org/F4320334627 |
| funders[3].ror | https://ror.org/0439y7842 |
| funders[3].display_name | Engineering and Physical Sciences Research Council |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C129848803 |
| concepts[0].level | 2 |
| concepts[0].score | 0.7531198263168335 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q2564360 |
| concepts[0].display_name | Sample size determination |
| concepts[1].id | https://openalex.org/C41008148 |
| concepts[1].level | 0 |
| concepts[1].score | 0.7260023951530457 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[1].display_name | Computer science |
| concepts[2].id | https://openalex.org/C89246107 |
| concepts[2].level | 2 |
| concepts[2].score | 0.6758548617362976 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q1398821 |
| concepts[2].display_name | Rule of thumb |
| concepts[3].id | https://openalex.org/C198531522 |
| concepts[3].level | 2 |
| concepts[3].score | 0.49875426292419434 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q485146 |
| concepts[3].display_name | Sample (material) |
| concepts[4].id | https://openalex.org/C124101348 |
| concepts[4].level | 1 |
| concepts[4].score | 0.46275874972343445 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q172491 |
| concepts[4].display_name | Data mining |
| concepts[5].id | https://openalex.org/C3019813237 |
| concepts[5].level | 2 |
| concepts[5].score | 0.4605255722999573 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q65089264 |
| concepts[5].display_name | Model validation |
| concepts[6].id | https://openalex.org/C45804977 |
| concepts[6].level | 2 |
| concepts[6].score | 0.4261902868747711 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q7239673 |
| concepts[6].display_name | Predictive modelling |
| concepts[7].id | https://openalex.org/C119857082 |
| concepts[7].level | 1 |
| concepts[7].score | 0.29606741666793823 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q2539 |
| concepts[7].display_name | Machine learning |
| concepts[8].id | https://openalex.org/C105795698 |
| concepts[8].level | 1 |
| concepts[8].score | 0.2674790024757385 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q12483 |
| concepts[8].display_name | Statistics |
| concepts[9].id | https://openalex.org/C2522767166 |
| concepts[9].level | 1 |
| concepts[9].score | 0.2353660762310028 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q2374463 |
| concepts[9].display_name | Data science |
| concepts[10].id | https://openalex.org/C11413529 |
| concepts[10].level | 1 |
| concepts[10].score | 0.1507163643836975 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q8366 |
| concepts[10].display_name | Algorithm |
| concepts[11].id | https://openalex.org/C33923547 |
| concepts[11].level | 0 |
| concepts[11].score | 0.11016497015953064 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q395 |
| concepts[11].display_name | Mathematics |
| concepts[12].id | https://openalex.org/C43617362 |
| concepts[12].level | 1 |
| concepts[12].score | 0.0 |
| concepts[12].wikidata | https://www.wikidata.org/wiki/Q170050 |
| concepts[12].display_name | Chromatography |
| concepts[13].id | https://openalex.org/C185592680 |
| concepts[13].level | 0 |
| concepts[13].score | 0.0 |
| concepts[13].wikidata | https://www.wikidata.org/wiki/Q2329 |
| concepts[13].display_name | Chemistry |
| keywords[0].id | https://openalex.org/keywords/sample-size-determination |
| keywords[0].score | 0.7531198263168335 |
| keywords[0].display_name | Sample size determination |
| keywords[1].id | https://openalex.org/keywords/computer-science |
| keywords[1].score | 0.7260023951530457 |
| keywords[1].display_name | Computer science |
| keywords[2].id | https://openalex.org/keywords/rule-of-thumb |
| keywords[2].score | 0.6758548617362976 |
| keywords[2].display_name | Rule of thumb |
| keywords[3].id | https://openalex.org/keywords/sample |
| keywords[3].score | 0.49875426292419434 |
| keywords[3].display_name | Sample (material) |
| keywords[4].id | https://openalex.org/keywords/data-mining |
| keywords[4].score | 0.46275874972343445 |
| keywords[4].display_name | Data mining |
| keywords[5].id | https://openalex.org/keywords/model-validation |
| keywords[5].score | 0.4605255722999573 |
| keywords[5].display_name | Model validation |
| keywords[6].id | https://openalex.org/keywords/predictive-modelling |
| keywords[6].score | 0.4261902868747711 |
| keywords[6].display_name | Predictive modelling |
| keywords[7].id | https://openalex.org/keywords/machine-learning |
| keywords[7].score | 0.29606741666793823 |
| keywords[7].display_name | Machine learning |
| keywords[8].id | https://openalex.org/keywords/statistics |
| keywords[8].score | 0.2674790024757385 |
| keywords[8].display_name | Statistics |
| keywords[9].id | https://openalex.org/keywords/data-science |
| keywords[9].score | 0.2353660762310028 |
| keywords[9].display_name | Data science |
| keywords[10].id | https://openalex.org/keywords/algorithm |
| keywords[10].score | 0.1507163643836975 |
| keywords[10].display_name | Algorithm |
| keywords[11].id | https://openalex.org/keywords/mathematics |
| keywords[11].score | 0.11016497015953064 |
| keywords[11].display_name | Mathematics |
| language | en |
| locations[0].id | doi:10.1136/bmj-2023-074821 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4393917726 |
| locations[0].source.issn | 1756-1833 |
| locations[0].source.type | journal |
| locations[0].source.is_oa | False |
| locations[0].source.issn_l | 1756-1833 |
| locations[0].source.is_core | True |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | BMJ |
| locations[0].source.host_organization | |
| locations[0].source.host_organization_name | |
| locations[0].license | cc-by |
| locations[0].pdf_url | https://www.bmj.com/content/bmj/384/bmj-2023-074821.full.pdf |
| locations[0].version | publishedVersion |
| locations[0].raw_type | journal-article |
| locations[0].license_id | https://openalex.org/licenses/cc-by |
| locations[0].is_accepted | True |
| locations[0].is_published | True |
| locations[0].raw_source_name | BMJ |
| locations[0].landing_page_url | https://doi.org/10.1136/bmj-2023-074821 |
| locations[1].id | pmid:38253388 |
| locations[1].is_oa | False |
| locations[1].source.id | https://openalex.org/S4306525036 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | False |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | PubMed |
| locations[1].source.host_organization | https://openalex.org/I1299303238 |
| locations[1].source.host_organization_name | National Institutes of Health |
| locations[1].source.host_organization_lineage | https://openalex.org/I1299303238 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | publishedVersion |
| locations[1].raw_type | |
| locations[1].license_id | |
| locations[1].is_accepted | True |
| locations[1].is_published | True |
| locations[1].raw_source_name | BMJ (Clinical research ed.) |
| locations[1].landing_page_url | https://pubmed.ncbi.nlm.nih.gov/38253388 |
| locations[2].id | pmh:oai:lirias2repo.kuleuven.be:20.500.12942/736826 |
| locations[2].is_oa | True |
| locations[2].source.id | https://openalex.org/S4306401954 |
| locations[2].source.issn | |
| locations[2].source.type | repository |
| locations[2].source.is_oa | False |
| locations[2].source.issn_l | |
| locations[2].source.is_core | False |
| locations[2].source.is_in_doaj | False |
| locations[2].source.display_name | Lirias (KU Leuven) |
| locations[2].source.host_organization | https://openalex.org/I99464096 |
| locations[2].source.host_organization_name | KU Leuven |
| locations[2].source.host_organization_lineage | https://openalex.org/I99464096 |
| locations[2].license | cc-by |
| locations[2].pdf_url | |
| locations[2].version | acceptedVersion |
| locations[2].raw_type | info:eu-repo/semantics/publishedVersion |
| locations[2].license_id | https://openalex.org/licenses/cc-by |
| locations[2].is_accepted | True |
| locations[2].is_published | False |
| locations[2].raw_source_name | Bmj-British Medical Journal, vol. 384, Art.No. ARTN e074821 |
| locations[2].landing_page_url | https://lirias.kuleuven.be/handle/20.500.12942/736826 |
| locations[3].id | pmh:oai:open-archive.highwire.org:bmj:384/jan22_1/e074821 |
| locations[3].is_oa | False |
| locations[3].source | |
| locations[3].license | |
| locations[3].pdf_url | |
| locations[3].version | submittedVersion |
| locations[3].raw_type | TEXT |
| locations[3].license_id | |
| locations[3].is_accepted | False |
| locations[3].is_published | False |
| locations[3].raw_source_name | |
| locations[3].landing_page_url | http://www.bmj.com/cgi/content/short/384/jan22_1/e074821 |
| locations[4].id | pmh:oai:pubmedcentral.nih.gov:11778934 |
| locations[4].is_oa | True |
| locations[4].source.id | https://openalex.org/S2764455111 |
| locations[4].source.issn | |
| locations[4].source.type | repository |
| locations[4].source.is_oa | False |
| locations[4].source.issn_l | |
| locations[4].source.is_core | False |
| locations[4].source.is_in_doaj | False |
| locations[4].source.display_name | PubMed Central |
| locations[4].source.host_organization | https://openalex.org/I1299303238 |
| locations[4].source.host_organization_name | National Institutes of Health |
| locations[4].source.host_organization_lineage | https://openalex.org/I1299303238 |
| locations[4].license | other-oa |
| locations[4].pdf_url | |
| locations[4].version | submittedVersion |
| locations[4].raw_type | Text |
| locations[4].license_id | https://openalex.org/licenses/other-oa |
| locations[4].is_accepted | False |
| locations[4].is_published | False |
| locations[4].raw_source_name | BMJ |
| locations[4].landing_page_url | https://www.ncbi.nlm.nih.gov/pmc/articles/11778934 |
| locations[5].id | pmh:oai:pure.atira.dk:openaire_cris_publications/190aae8f-4020-409f-9483-fb6c3f35c2e5 |
| locations[5].is_oa | True |
| locations[5].source.id | https://openalex.org/S4306400216 |
| locations[5].source.issn | |
| locations[5].source.type | repository |
| locations[5].source.is_oa | False |
| locations[5].source.issn_l | |
| locations[5].source.is_core | False |
| locations[5].source.is_in_doaj | False |
| locations[5].source.display_name | Research Portal (King's College London) |
| locations[5].source.host_organization | https://openalex.org/I183935753 |
| locations[5].source.host_organization_name | King's College London |
| locations[5].source.host_organization_lineage | https://openalex.org/I183935753 |
| locations[5].license | other-oa |
| locations[5].pdf_url | |
| locations[5].version | submittedVersion |
| locations[5].raw_type | article |
| locations[5].license_id | https://openalex.org/licenses/other-oa |
| locations[5].is_accepted | False |
| locations[5].is_published | False |
| locations[5].raw_source_name | Riley, R D, Snell, K I E, Archer, L, Ensor, J, Debray, T P A, Van Calster, B, Van Smeden, M & Collins, G S 2024, 'Evaluation of clinical prediction models (part 3): calculating the sample size required for an external validation study', BMJ, vol. 384, e074821. https://doi.org/10.1136/bmj-2023-074821 |
| locations[5].landing_page_url | https://research.birmingham.ac.uk/en/publications/190aae8f-4020-409f-9483-fb6c3f35c2e5 |
| indexed_in | crossref, pubmed |
| authorships[0].author.id | https://openalex.org/A5040942978 |
| authorships[0].author.orcid | https://orcid.org/0000-0001-8699-0735 |
| authorships[0].author.display_name | Richard D Riley |
| authorships[0].countries | GB |
| authorships[0].affiliations[0].institution_ids | https://openalex.org/I4210162943 |
| authorships[0].affiliations[0].raw_affiliation_string | National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK |
| authorships[0].affiliations[1].institution_ids | https://openalex.org/I79619799 |
| authorships[0].affiliations[1].raw_affiliation_string | Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham B15 2TT, UK |
| authorships[0].institutions[0].id | https://openalex.org/I4210162943 |
| authorships[0].institutions[0].ror | https://ror.org/05ccjmp23 |
| authorships[0].institutions[0].type | facility |
| authorships[0].institutions[0].lineage | https://openalex.org/I34931013, https://openalex.org/I4210162943 |
| authorships[0].institutions[0].country_code | GB |
| authorships[0].institutions[0].display_name | NIHR Birmingham Biomedical Research Centre |
| authorships[0].institutions[1].id | https://openalex.org/I79619799 |
| authorships[0].institutions[1].ror | https://ror.org/03angcq70 |
| authorships[0].institutions[1].type | education |
| authorships[0].institutions[1].lineage | https://openalex.org/I79619799 |
| authorships[0].institutions[1].country_code | GB |
| authorships[0].institutions[1].display_name | University of Birmingham |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Richard D Riley |
| authorships[0].is_corresponding | True |
| authorships[0].raw_affiliation_strings | Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham B15 2TT, UK, National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK |
| authorships[1].author.id | https://openalex.org/A5049252097 |
| authorships[1].author.orcid | https://orcid.org/0000-0001-9373-6591 |
| authorships[1].author.display_name | Kym I E Snell |
| authorships[1].countries | GB |
| authorships[1].affiliations[0].institution_ids | https://openalex.org/I79619799 |
| authorships[1].affiliations[0].raw_affiliation_string | Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham B15 2TT, UK |
| authorships[1].affiliations[1].institution_ids | https://openalex.org/I4210162943 |
| authorships[1].affiliations[1].raw_affiliation_string | National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK |
| authorships[1].institutions[0].id | https://openalex.org/I4210162943 |
| authorships[1].institutions[0].ror | https://ror.org/05ccjmp23 |
| authorships[1].institutions[0].type | facility |
| authorships[1].institutions[0].lineage | https://openalex.org/I34931013, https://openalex.org/I4210162943 |
| authorships[1].institutions[0].country_code | GB |
| authorships[1].institutions[0].display_name | NIHR Birmingham Biomedical Research Centre |
| authorships[1].institutions[1].id | https://openalex.org/I79619799 |
| authorships[1].institutions[1].ror | https://ror.org/03angcq70 |
| authorships[1].institutions[1].type | education |
| authorships[1].institutions[1].lineage | https://openalex.org/I79619799 |
| authorships[1].institutions[1].country_code | GB |
| authorships[1].institutions[1].display_name | University of Birmingham |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Kym I E Snell |
| authorships[1].is_corresponding | False |
| authorships[1].raw_affiliation_strings | Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham B15 2TT, UK, National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK |
| authorships[2].author.id | https://openalex.org/A5086736480 |
| authorships[2].author.orcid | https://orcid.org/0000-0003-2504-2613 |
| authorships[2].author.display_name | Lucinda Archer |
| authorships[2].countries | GB |
| authorships[2].affiliations[0].institution_ids | https://openalex.org/I4210162943 |
| authorships[2].affiliations[0].raw_affiliation_string | National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK |
| authorships[2].affiliations[1].institution_ids | https://openalex.org/I79619799 |
| authorships[2].affiliations[1].raw_affiliation_string | Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham B15 2TT, UK |
| authorships[2].institutions[0].id | https://openalex.org/I4210162943 |
| authorships[2].institutions[0].ror | https://ror.org/05ccjmp23 |
| authorships[2].institutions[0].type | facility |
| authorships[2].institutions[0].lineage | https://openalex.org/I34931013, https://openalex.org/I4210162943 |
| authorships[2].institutions[0].country_code | GB |
| authorships[2].institutions[0].display_name | NIHR Birmingham Biomedical Research Centre |
| authorships[2].institutions[1].id | https://openalex.org/I79619799 |
| authorships[2].institutions[1].ror | https://ror.org/03angcq70 |
| authorships[2].institutions[1].type | education |
| authorships[2].institutions[1].lineage | https://openalex.org/I79619799 |
| authorships[2].institutions[1].country_code | GB |
| authorships[2].institutions[1].display_name | University of Birmingham |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Lucinda Archer |
| authorships[2].is_corresponding | False |
| authorships[2].raw_affiliation_strings | Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham B15 2TT, UK, National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK |
| authorships[3].author.id | https://openalex.org/A5009368181 |
| authorships[3].author.orcid | https://orcid.org/0000-0001-7481-0282 |
| authorships[3].author.display_name | Joie Ensor |
| authorships[3].countries | GB |
| authorships[3].affiliations[0].institution_ids | https://openalex.org/I79619799 |
| authorships[3].affiliations[0].raw_affiliation_string | Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham B15 2TT, UK |
| authorships[3].affiliations[1].institution_ids | https://openalex.org/I4210162943 |
| authorships[3].affiliations[1].raw_affiliation_string | National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK |
| authorships[3].institutions[0].id | https://openalex.org/I4210162943 |
| authorships[3].institutions[0].ror | https://ror.org/05ccjmp23 |
| authorships[3].institutions[0].type | facility |
| authorships[3].institutions[0].lineage | https://openalex.org/I34931013, https://openalex.org/I4210162943 |
| authorships[3].institutions[0].country_code | GB |
| authorships[3].institutions[0].display_name | NIHR Birmingham Biomedical Research Centre |
| authorships[3].institutions[1].id | https://openalex.org/I79619799 |
| authorships[3].institutions[1].ror | https://ror.org/03angcq70 |
| authorships[3].institutions[1].type | education |
| authorships[3].institutions[1].lineage | https://openalex.org/I79619799 |
| authorships[3].institutions[1].country_code | GB |
| authorships[3].institutions[1].display_name | University of Birmingham |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Joie Ensor |
| authorships[3].is_corresponding | False |
| authorships[3].raw_affiliation_strings | Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham B15 2TT, UK, National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK |
| authorships[4].author.id | https://openalex.org/A5002390430 |
| authorships[4].author.orcid | https://orcid.org/0000-0002-1790-2719 |
| authorships[4].author.display_name | Thomas P. A. Debray |
| authorships[4].countries | NL |
| authorships[4].affiliations[0].institution_ids | https://openalex.org/I193662353, https://openalex.org/I3018483916 |
| authorships[4].affiliations[0].raw_affiliation_string | Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands |
| authorships[4].institutions[0].id | https://openalex.org/I3018483916 |
| authorships[4].institutions[0].ror | https://ror.org/0575yy874 |
| authorships[4].institutions[0].type | healthcare |
| authorships[4].institutions[0].lineage | https://openalex.org/I3018483916 |
| authorships[4].institutions[0].country_code | NL |
| authorships[4].institutions[0].display_name | University Medical Center Utrecht |
| authorships[4].institutions[1].id | https://openalex.org/I193662353 |
| authorships[4].institutions[1].ror | https://ror.org/04pp8hn57 |
| authorships[4].institutions[1].type | education |
| authorships[4].institutions[1].lineage | https://openalex.org/I193662353 |
| authorships[4].institutions[1].country_code | NL |
| authorships[4].institutions[1].display_name | Utrecht University |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Thomas P A Debray |
| authorships[4].is_corresponding | False |
| authorships[4].raw_affiliation_strings | Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands |
| authorships[5].author.id | https://openalex.org/A5086320066 |
| authorships[5].author.orcid | https://orcid.org/0000-0003-1613-7450 |
| authorships[5].author.display_name | Ben Van Calster |
| authorships[5].countries | BE, NL |
| authorships[5].affiliations[0].institution_ids | https://openalex.org/I99464096 |
| authorships[5].affiliations[0].raw_affiliation_string | Department of Development and Regeneration, KU Leuven, Leuven, Belgium |
| authorships[5].affiliations[1].institution_ids | https://openalex.org/I2800006345 |
| authorships[5].affiliations[1].raw_affiliation_string | Department of Biomedical Data Sciences, Leiden University Medical Centre, Leiden, Netherlands |
| authorships[5].institutions[0].id | https://openalex.org/I99464096 |
| authorships[5].institutions[0].ror | https://ror.org/05f950310 |
| authorships[5].institutions[0].type | education |
| authorships[5].institutions[0].lineage | https://openalex.org/I99464096 |
| authorships[5].institutions[0].country_code | BE |
| authorships[5].institutions[0].display_name | KU Leuven |
| authorships[5].institutions[1].id | https://openalex.org/I2800006345 |
| authorships[5].institutions[1].ror | https://ror.org/05xvt9f17 |
| authorships[5].institutions[1].type | healthcare |
| authorships[5].institutions[1].lineage | https://openalex.org/I2800006345 |
| authorships[5].institutions[1].country_code | NL |
| authorships[5].institutions[1].display_name | Leiden University Medical Center |
| authorships[5].author_position | middle |
| authorships[5].raw_author_name | Ben van Calster |
| authorships[5].is_corresponding | False |
| authorships[5].raw_affiliation_strings | Department of Biomedical Data Sciences, Leiden University Medical Centre, Leiden, Netherlands, Department of Development and Regeneration, KU Leuven, Leuven, Belgium |
| authorships[6].author.id | https://openalex.org/A5067519537 |
| authorships[6].author.orcid | https://orcid.org/0000-0002-5529-1541 |
| authorships[6].author.display_name | Maarten van Smeden |
| authorships[6].countries | NL |
| authorships[6].affiliations[0].institution_ids | https://openalex.org/I193662353, https://openalex.org/I3018483916 |
| authorships[6].affiliations[0].raw_affiliation_string | Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands |
| authorships[6].institutions[0].id | https://openalex.org/I3018483916 |
| authorships[6].institutions[0].ror | https://ror.org/0575yy874 |
| authorships[6].institutions[0].type | healthcare |
| authorships[6].institutions[0].lineage | https://openalex.org/I3018483916 |
| authorships[6].institutions[0].country_code | NL |
| authorships[6].institutions[0].display_name | University Medical Center Utrecht |
| authorships[6].institutions[1].id | https://openalex.org/I193662353 |
| authorships[6].institutions[1].ror | https://ror.org/04pp8hn57 |
| authorships[6].institutions[1].type | education |
| authorships[6].institutions[1].lineage | https://openalex.org/I193662353 |
| authorships[6].institutions[1].country_code | NL |
| authorships[6].institutions[1].display_name | Utrecht University |
| authorships[6].author_position | middle |
| authorships[6].raw_author_name | Maarten van Smeden |
| authorships[6].is_corresponding | False |
| authorships[6].raw_affiliation_strings | Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands |
| authorships[7].author.id | https://openalex.org/A5060891981 |
| authorships[7].author.orcid | https://orcid.org/0000-0002-2772-2316 |
| authorships[7].author.display_name | Gary S. Collins |
| authorships[7].countries | GB |
| authorships[7].affiliations[0].institution_ids | https://openalex.org/I2801809904, https://openalex.org/I40120149 |
| authorships[7].affiliations[0].raw_affiliation_string | Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford, UK |
| authorships[7].institutions[0].id | https://openalex.org/I2801809904 |
| authorships[7].institutions[0].ror | https://ror.org/0036ate90 |
| authorships[7].institutions[0].type | healthcare |
| authorships[7].institutions[0].lineage | https://openalex.org/I2801809904, https://openalex.org/I4210147381 |
| authorships[7].institutions[0].country_code | GB |
| authorships[7].institutions[0].display_name | Nuffield Orthopaedic Centre |
| authorships[7].institutions[1].id | https://openalex.org/I40120149 |
| authorships[7].institutions[1].ror | https://ror.org/052gg0110 |
| authorships[7].institutions[1].type | education |
| authorships[7].institutions[1].lineage | https://openalex.org/I40120149 |
| authorships[7].institutions[1].country_code | GB |
| authorships[7].institutions[1].display_name | University of Oxford |
| authorships[7].author_position | last |
| authorships[7].raw_author_name | Gary S Collins |
| authorships[7].is_corresponding | False |
| authorships[7].raw_affiliation_strings | Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford, UK |
| has_content.pdf | True |
| has_content.grobid_xml | True |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://www.bmj.com/content/bmj/384/bmj-2023-074821.full.pdf |
| open_access.oa_status | hybrid |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Evaluation of clinical prediction models (part 3): calculating the sample size required for an external validation study |
| has_fulltext | True |
| is_retracted | False |
| updated_date | 2025-11-06T03:46:38.306776 |
| primary_topic.id | https://openalex.org/T10206 |
| primary_topic.field.id | https://openalex.org/fields/18 |
| primary_topic.field.display_name | Decision Sciences |
| primary_topic.score | 0.9908000230789185 |
| primary_topic.domain.id | https://openalex.org/domains/2 |
| primary_topic.domain.display_name | Social Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1804 |
| primary_topic.subfield.display_name | Statistics, Probability and Uncertainty |
| primary_topic.display_name | Meta-analysis and systematic reviews |
| related_works | https://openalex.org/W4377832668, https://openalex.org/W4388813866, https://openalex.org/W2999104021, https://openalex.org/W1481656249, https://openalex.org/W2850804095, https://openalex.org/W2162280767, https://openalex.org/W3011166658, https://openalex.org/W2402853020, https://openalex.org/W4285561001, https://openalex.org/W3047225376 |
| cited_by_count | 100 |
| counts_by_year[0].year | 2025 |
| counts_by_year[0].cited_by_count | 71 |
| counts_by_year[1].year | 2024 |
| counts_by_year[1].cited_by_count | 29 |
| locations_count | 6 |
| best_oa_location.id | doi:10.1136/bmj-2023-074821 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4393917726 |
| best_oa_location.source.issn | 1756-1833 |
| best_oa_location.source.type | journal |
| best_oa_location.source.is_oa | False |
| best_oa_location.source.issn_l | 1756-1833 |
| best_oa_location.source.is_core | True |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | BMJ |
| best_oa_location.source.host_organization | |
| best_oa_location.source.host_organization_name | |
| best_oa_location.license | cc-by |
| best_oa_location.pdf_url | https://www.bmj.com/content/bmj/384/bmj-2023-074821.full.pdf |
| best_oa_location.version | publishedVersion |
| best_oa_location.raw_type | journal-article |
| best_oa_location.license_id | https://openalex.org/licenses/cc-by |
| best_oa_location.is_accepted | True |
| best_oa_location.is_published | True |
| best_oa_location.raw_source_name | BMJ |
| best_oa_location.landing_page_url | https://doi.org/10.1136/bmj-2023-074821 |
| primary_location.id | doi:10.1136/bmj-2023-074821 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4393917726 |
| primary_location.source.issn | 1756-1833 |
| primary_location.source.type | journal |
| primary_location.source.is_oa | False |
| primary_location.source.issn_l | 1756-1833 |
| primary_location.source.is_core | True |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | BMJ |
| primary_location.source.host_organization | |
| primary_location.source.host_organization_name | |
| primary_location.license | cc-by |
| primary_location.pdf_url | https://www.bmj.com/content/bmj/384/bmj-2023-074821.full.pdf |
| primary_location.version | publishedVersion |
| primary_location.raw_type | journal-article |
| primary_location.license_id | https://openalex.org/licenses/cc-by |
| primary_location.is_accepted | True |
| primary_location.is_published | True |
| primary_location.raw_source_name | BMJ |
| primary_location.landing_page_url | https://doi.org/10.1136/bmj-2023-074821 |
| publication_date | 2024-01-22 |
| publication_year | 2024 |
| referenced_works | https://openalex.org/W2115709314, https://openalex.org/W2297716127, https://openalex.org/W4242807082, https://openalex.org/W4390876555, https://openalex.org/W2084164487, https://openalex.org/W3153657475, https://openalex.org/W1997812541, https://openalex.org/W2018195004, https://openalex.org/W2122892545, https://openalex.org/W2225109326, https://openalex.org/W4238788310, https://openalex.org/W4200283820, https://openalex.org/W3129661430, https://openalex.org/W3094834023, https://openalex.org/W4390674087, https://openalex.org/W1994682257, https://openalex.org/W2898582579, https://openalex.org/W2226880313, https://openalex.org/W2983520337, https://openalex.org/W2045030989, https://openalex.org/W2152394445, https://openalex.org/W3153982323, https://openalex.org/W3120017613, https://openalex.org/W2996480032, https://openalex.org/W2078271269, https://openalex.org/W1952905287, https://openalex.org/W2474667434, https://openalex.org/W4233080080, https://openalex.org/W3012413426, https://openalex.org/W2898559532, https://openalex.org/W2896476100, https://openalex.org/W2981991941 |
| referenced_works_count | 32 |
| abstract_inverted_index.a | 8 |
| abstract_inverted_index.An | 0 |
| abstract_inverted_index.In | 26 |
| abstract_inverted_index.at | 66 |
| abstract_inverted_index.by | 58 |
| abstract_inverted_index.in | 11 |
| abstract_inverted_index.of | 7, 16, 30, 56 |
| abstract_inverted_index.on | 33 |
| abstract_inverted_index.to | 22, 41, 53, 61 |
| abstract_inverted_index.and | 37, 51, 64 |
| abstract_inverted_index.are | 19 |
| abstract_inverted_index.but | 14 |
| abstract_inverted_index.for | 47 |
| abstract_inverted_index.how | 40 |
| abstract_inverted_index.new | 12 |
| abstract_inverted_index.the | 5, 27, 43, 62 |
| abstract_inverted_index.too | 20 |
| abstract_inverted_index.many | 15 |
| abstract_inverted_index.size | 45 |
| abstract_inverted_index.Riley | 36 |
| abstract_inverted_index.avoid | 54 |
| abstract_inverted_index.data, | 13 |
| abstract_inverted_index.hand. | 67 |
| abstract_inverted_index.model | 10, 34, 63 |
| abstract_inverted_index.rules | 55 |
| abstract_inverted_index.small | 21 |
| abstract_inverted_index.study | 3 |
| abstract_inverted_index.their | 31 |
| abstract_inverted_index.these | 17 |
| abstract_inverted_index.third | 28 |
| abstract_inverted_index.thumb | 57 |
| abstract_inverted_index.sample | 44 |
| abstract_inverted_index.series | 32 |
| abstract_inverted_index.article | 29 |
| abstract_inverted_index.propose | 52 |
| abstract_inverted_index.provide | 23 |
| abstract_inverted_index.setting | 65 |
| abstract_inverted_index.studies | 18 |
| abstract_inverted_index.answers. | 25 |
| abstract_inverted_index.describe | 39 |
| abstract_inverted_index.external | 1, 48 |
| abstract_inverted_index.reliable | 24 |
| abstract_inverted_index.required | 46 |
| abstract_inverted_index.studies, | 50 |
| abstract_inverted_index.calculate | 42 |
| abstract_inverted_index.evaluates | 4 |
| abstract_inverted_index.tailoring | 59 |
| abstract_inverted_index.colleagues | 38 |
| abstract_inverted_index.prediction | 9 |
| abstract_inverted_index.validation | 2, 49 |
| abstract_inverted_index.evaluation, | 35 |
| abstract_inverted_index.performance | 6 |
| abstract_inverted_index.calculations | 60 |
| cited_by_percentile_year.max | 100 |
| cited_by_percentile_year.min | 99 |
| corresponding_author_ids | https://openalex.org/A5040942978 |
| countries_distinct_count | 3 |
| institutions_distinct_count | 8 |
| corresponding_institution_ids | https://openalex.org/I4210162943, https://openalex.org/I79619799 |
| citation_normalized_percentile.value | 0.99931549 |
| citation_normalized_percentile.is_in_top_1_percent | True |
| citation_normalized_percentile.is_in_top_10_percent | True |