Whose Personae? Synthetic Persona Experiments in LLM Research and Pathways to Transparency Article Swipe

PDF

Jan Batzner , Volker Stocker , Bo Tang , Anusha Natarajan , Qinhao Chen , Stefan Schmid , Gjergji Kasneci ·

YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.1609/aies.v8i1.36553

Synthetic personae experiments have become a prominent method in Large Language Model alignment research, yet the representativeness and ecological validity of these personae vary considerably between studies. Through a review of 63 peer-reviewed studies published between 2023 and 2025 in leading NLP and AI venues, we reveal a critical gap: task and population of interest are often underspecified in persona-based experiments, despite personalization being fundamentally dependent on these criteria. Our analysis shows substantial differences in user representation, with most studies focusing on limited sociodemographic attributes and only 35% discussing the representativeness of their LLM personae. Based on our findings, we introduce a persona transparency checklist that emphasizes representative sampling, explicit grounding in empirical data, and enhanced ecological validity. Our work provides both a comprehensive assessment of current practices and practical guidelines to improve the rigor and ecological validity of persona-based evaluations in language model alignment research.

Related Topics

Concepts

No concepts available.

Metadata

Type: preprint
Language: en
Landing Page: https://doi.org/10.1609/aies.v8i1.36553
PDF: https://ojs.aaai.org/index.php/AIES/article/download/36553/38691
OA Status: bronze
OpenAlex ID: https://openalex.org/W4415230806

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4415230806

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.1609/aies.v8i1.36553

Digital Object Identifier
Title: Whose Personae? Synthetic Persona Experiments in LLM Research and Pathways to Transparency

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2025

Year of publication
Publication date: 2025-10-15

Full publication date if available
Authors: Jan Batzner, Volker Stocker, Bo Tang, Anusha Natarajan, Qinhao Chen, Stefan Schmid, Gjergji Kasneci

List of authors in order
Landing page: https://doi.org/10.1609/aies.v8i1.36553

Publisher landing page
PDF URL: https://ojs.aaai.org/index.php/AIES/article/download/36553/38691

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: bronze

Open access status per OpenAlex
OA URL: https://ojs.aaai.org/index.php/AIES/article/download/36553/38691

Direct OA link when available
Cited by: 0

Total citation count in OpenAlex

Full payload

id	https://openalex.org/W4415230806
doi	https://doi.org/10.1609/aies.v8i1.36553
ids.doi	https://doi.org/10.1609/aies.v8i1.36553
ids.openalex	https://openalex.org/W4415230806
fwci	0.0
type	preprint
title	Whose Personae? Synthetic Persona Experiments in LLM Research and Pathways to Transparency
biblio.issue	1
biblio.volume	8
biblio.last_page	354
biblio.first_page	343
topics[0].id	https://openalex.org/T14074
topics[0].field.id	https://openalex.org/fields/17
topics[0].field.display_name	Computer Science
topics[0].score	0.9976999759674072
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/1709
topics[0].subfield.display_name	Human-Computer Interaction
topics[0].display_name	Persona Design and Applications
topics[1].id	https://openalex.org/T11024
topics[1].field.id	https://openalex.org/fields/33
topics[1].field.display_name	Social Sciences
topics[1].score	0.9057999849319458
topics[1].domain.id	https://openalex.org/domains/2
topics[1].domain.display_name	Social Sciences
topics[1].subfield.id	https://openalex.org/subfields/3312
topics[1].subfield.display_name	Sociology and Political Science
topics[1].display_name	Information Systems Theories and Implementation
is_xpac	False
apc_list
apc_paid
language	en
locations[0].id	doi:10.1609/aies.v8i1.36553
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S5407048695
locations[0].source.issn	3065-8365
locations[0].source.type	journal
locations[0].source.is_oa	False
locations[0].source.issn_l	3065-8365
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	Proceedings of the AAAI/ACM Conference on AI Ethics and Society
locations[0].source.host_organization
locations[0].source.host_organization_name
locations[0].license
locations[0].pdf_url	https://ojs.aaai.org/index.php/AIES/article/download/36553/38691
locations[0].version	publishedVersion
locations[0].raw_type	journal-article
locations[0].license_id
locations[0].is_accepted	True
locations[0].is_published	True
locations[0].raw_source_name	Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society
locations[0].landing_page_url	https://doi.org/10.1609/aies.v8i1.36553
locations[1].id	pmh:oai:arXiv.org:2512.00461
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license
locations[1].pdf_url	https://arxiv.org/pdf/2512.00461
locations[1].version	acceptedVersion
locations[1].raw_type	text
locations[1].license_id
locations[1].is_accepted	True
locations[1].is_published	False
locations[1].raw_source_name
locations[1].landing_page_url	http://arxiv.org/abs/2512.00461
indexed_in	arxiv, crossref
authorships[0].author.id	https://openalex.org/A5117865366
authorships[0].author.orcid
authorships[0].author.display_name	Jan Batzner
authorships[0].countries	DE
authorships[0].affiliations[0].institution_ids	https://openalex.org/I4210117750
authorships[0].affiliations[0].raw_affiliation_string	Weizenbaum Institute Columbia University Technical University Munich
authorships[0].institutions[0].id	https://openalex.org/I4210117750
authorships[0].institutions[0].ror	https://ror.org/023kksk09
authorships[0].institutions[0].type	facility
authorships[0].institutions[0].lineage	https://openalex.org/I176453806, https://openalex.org/I2800804238, https://openalex.org/I2801702882, https://openalex.org/I315704651, https://openalex.org/I4210117750, https://openalex.org/I4577782, https://openalex.org/I46043019, https://openalex.org/I4923324, https://openalex.org/I75951250
authorships[0].institutions[0].country_code	DE
authorships[0].institutions[0].display_name	Weizenbaum Institute
authorships[0].author_position	first
authorships[0].raw_author_name	Jan Batzner
authorships[0].is_corresponding	False
authorships[0].raw_affiliation_strings	Weizenbaum Institute Columbia University Technical University Munich
authorships[1].author.id	https://openalex.org/A5091751818
authorships[1].author.orcid
authorships[1].author.display_name	Volker Stocker
authorships[1].countries	DE
authorships[1].affiliations[0].institution_ids	https://openalex.org/I4210117750
authorships[1].affiliations[0].raw_affiliation_string	Weizenbaum Institute Technical University Berlin
authorships[1].institutions[0].id	https://openalex.org/I4210117750
authorships[1].institutions[0].ror	https://ror.org/023kksk09
authorships[1].institutions[0].type	facility
authorships[1].institutions[0].lineage	https://openalex.org/I176453806, https://openalex.org/I2800804238, https://openalex.org/I2801702882, https://openalex.org/I315704651, https://openalex.org/I4210117750, https://openalex.org/I4577782, https://openalex.org/I46043019, https://openalex.org/I4923324, https://openalex.org/I75951250
authorships[1].institutions[0].country_code	DE
authorships[1].institutions[0].display_name	Weizenbaum Institute
authorships[1].author_position	middle
authorships[1].raw_author_name	Volker Stocker
authorships[1].is_corresponding	False
authorships[1].raw_affiliation_strings	Weizenbaum Institute Technical University Berlin
authorships[2].author.id	https://openalex.org/A5103174018
authorships[2].author.orcid	https://orcid.org/0009-0008-8605-9455
authorships[2].author.display_name	Bo Tang
authorships[2].countries	US
authorships[2].affiliations[0].institution_ids	https://openalex.org/I78577930
authorships[2].affiliations[0].raw_affiliation_string	Columbia University
authorships[2].institutions[0].id	https://openalex.org/I78577930
authorships[2].institutions[0].ror	https://ror.org/00hj8s172
authorships[2].institutions[0].type	education
authorships[2].institutions[0].lineage	https://openalex.org/I78577930
authorships[2].institutions[0].country_code	US
authorships[2].institutions[0].display_name	Columbia University
authorships[2].author_position	middle
authorships[2].raw_author_name	Bingjun Tang
authorships[2].is_corresponding	False
authorships[2].raw_affiliation_strings	Columbia University
authorships[3].author.id	https://openalex.org/A5120016076
authorships[3].author.orcid
authorships[3].author.display_name	Anusha Natarajan
authorships[3].countries	US
authorships[3].affiliations[0].institution_ids	https://openalex.org/I78577930
authorships[3].affiliations[0].raw_affiliation_string	Columbia University
authorships[3].institutions[0].id	https://openalex.org/I78577930
authorships[3].institutions[0].ror	https://ror.org/00hj8s172
authorships[3].institutions[0].type	education
authorships[3].institutions[0].lineage	https://openalex.org/I78577930
authorships[3].institutions[0].country_code	US
authorships[3].institutions[0].display_name	Columbia University
authorships[3].author_position	middle
authorships[3].raw_author_name	Anusha Natarajan
authorships[3].is_corresponding	False
authorships[3].raw_affiliation_strings	Columbia University
authorships[4].author.id	https://openalex.org/A5090559181
authorships[4].author.orcid
authorships[4].author.display_name	Qinhao Chen
authorships[4].countries	US
authorships[4].affiliations[0].institution_ids	https://openalex.org/I78577930
authorships[4].affiliations[0].raw_affiliation_string	Columbia University
authorships[4].institutions[0].id	https://openalex.org/I78577930
authorships[4].institutions[0].ror	https://ror.org/00hj8s172
authorships[4].institutions[0].type	education
authorships[4].institutions[0].lineage	https://openalex.org/I78577930
authorships[4].institutions[0].country_code	US
authorships[4].institutions[0].display_name	Columbia University
authorships[4].author_position	middle
authorships[4].raw_author_name	Qinhao Chen
authorships[4].is_corresponding	False
authorships[4].raw_affiliation_strings	Columbia University
authorships[5].author.id	https://openalex.org/A5066080641
authorships[5].author.orcid	https://orcid.org/0000-0002-7798-1711
authorships[5].author.display_name	Stefan Schmid
authorships[5].countries	DE
authorships[5].affiliations[0].institution_ids	https://openalex.org/I4210117750
authorships[5].affiliations[0].raw_affiliation_string	Weizenbaum Institute Technical University Berlin
authorships[5].institutions[0].id	https://openalex.org/I4210117750
authorships[5].institutions[0].ror	https://ror.org/023kksk09
authorships[5].institutions[0].type	facility
authorships[5].institutions[0].lineage	https://openalex.org/I176453806, https://openalex.org/I2800804238, https://openalex.org/I2801702882, https://openalex.org/I315704651, https://openalex.org/I4210117750, https://openalex.org/I4577782, https://openalex.org/I46043019, https://openalex.org/I4923324, https://openalex.org/I75951250
authorships[5].institutions[0].country_code	DE
authorships[5].institutions[0].display_name	Weizenbaum Institute
authorships[5].author_position	middle
authorships[5].raw_author_name	Stefan Schmid
authorships[5].is_corresponding	False
authorships[5].raw_affiliation_strings	Weizenbaum Institute Technical University Berlin
authorships[6].author.id	https://openalex.org/A5024434748
authorships[6].author.orcid	https://orcid.org/0000-0002-3123-7268
authorships[6].author.display_name	Gjergji Kasneci
authorships[6].affiliations[0].raw_affiliation_string	Technical University Munich
authorships[6].author_position	last
authorships[6].raw_author_name	Gjergji Kasneci
authorships[6].is_corresponding	False
authorships[6].raw_affiliation_strings	Technical University Munich
has_content.pdf	True
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://ojs.aaai.org/index.php/AIES/article/download/36553/38691
open_access.oa_status	bronze
open_access.any_repository_has_fulltext	False
created_date	2025-10-16T00:00:00
display_name	Whose Personae? Synthetic Persona Experiments in LLM Research and Pathways to Transparency
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T03:46:38.306776
primary_topic.id	https://openalex.org/T14074
primary_topic.field.id	https://openalex.org/fields/17
primary_topic.field.display_name	Computer Science
primary_topic.score	0.9976999759674072
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1709
primary_topic.subfield.display_name	Human-Computer Interaction
primary_topic.display_name	Persona Design and Applications
cited_by_count	0
locations_count	2
best_oa_location.id	doi:10.1609/aies.v8i1.36553
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S5407048695
best_oa_location.source.issn	3065-8365
best_oa_location.source.type	journal
best_oa_location.source.is_oa	False
best_oa_location.source.issn_l	3065-8365
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	Proceedings of the AAAI/ACM Conference on AI Ethics and Society
best_oa_location.source.host_organization
best_oa_location.source.host_organization_name
best_oa_location.license
best_oa_location.pdf_url	https://ojs.aaai.org/index.php/AIES/article/download/36553/38691
best_oa_location.version	publishedVersion
best_oa_location.raw_type	journal-article
best_oa_location.license_id
best_oa_location.is_accepted	True
best_oa_location.is_published	True
best_oa_location.raw_source_name	Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society
best_oa_location.landing_page_url	https://doi.org/10.1609/aies.v8i1.36553
primary_location.id	doi:10.1609/aies.v8i1.36553
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S5407048695
primary_location.source.issn	3065-8365
primary_location.source.type	journal
primary_location.source.is_oa	False
primary_location.source.issn_l	3065-8365
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	Proceedings of the AAAI/ACM Conference on AI Ethics and Society
primary_location.source.host_organization
primary_location.source.host_organization_name
primary_location.license
primary_location.pdf_url	https://ojs.aaai.org/index.php/AIES/article/download/36553/38691
primary_location.version	publishedVersion
primary_location.raw_type	journal-article
primary_location.license_id
primary_location.is_accepted	True
primary_location.is_published	True
primary_location.raw_source_name	Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society
primary_location.landing_page_url	https://doi.org/10.1609/aies.v8i1.36553
publication_date	2025-10-15
publication_year	2025
referenced_works_count	0
abstract_inverted_index.a	5, 28, 47, 101, 122
abstract_inverted_index.63	31
abstract_inverted_index.AI	43
abstract_inverted_index.in	8, 39, 58, 74, 111, 141
abstract_inverted_index.of	20, 30, 53, 91, 125, 138
abstract_inverted_index.on	66, 81, 96
abstract_inverted_index.to	131
abstract_inverted_index.we	45, 99
abstract_inverted_index.35%	87
abstract_inverted_index.LLM	93
abstract_inverted_index.NLP	41
abstract_inverted_index.Our	69, 118
abstract_inverted_index.and	17, 37, 42, 51, 85, 114, 128, 135
abstract_inverted_index.are	55
abstract_inverted_index.our	97
abstract_inverted_index.the	15, 89, 133
abstract_inverted_index.yet	14
abstract_inverted_index.2023	36
abstract_inverted_index.2025	38
abstract_inverted_index.both	121
abstract_inverted_index.gap:	49
abstract_inverted_index.have	3
abstract_inverted_index.most	78
abstract_inverted_index.only	86
abstract_inverted_index.task	50
abstract_inverted_index.that	105
abstract_inverted_index.user	75
abstract_inverted_index.vary	23
abstract_inverted_index.with	77
abstract_inverted_index.work	119
abstract_inverted_index.Based	95
abstract_inverted_index.Large	9
abstract_inverted_index.Model	11
abstract_inverted_index.being	63
abstract_inverted_index.data,	113
abstract_inverted_index.model	143
abstract_inverted_index.often	56
abstract_inverted_index.rigor	134
abstract_inverted_index.shows	71
abstract_inverted_index.their	92
abstract_inverted_index.these	21, 67
abstract_inverted_index.become	4
abstract_inverted_index.method	7
abstract_inverted_index.reveal	46
abstract_inverted_index.review	29
abstract_inverted_index.Through	27
abstract_inverted_index.between	25, 35
abstract_inverted_index.current	126
abstract_inverted_index.despite	61
abstract_inverted_index.improve	132
abstract_inverted_index.leading	40
abstract_inverted_index.limited	82
abstract_inverted_index.persona	102
abstract_inverted_index.studies	33, 79
abstract_inverted_index.venues,	44
abstract_inverted_index.Language	10
abstract_inverted_index.analysis	70
abstract_inverted_index.critical	48
abstract_inverted_index.enhanced	115
abstract_inverted_index.explicit	109
abstract_inverted_index.focusing	80
abstract_inverted_index.interest	54
abstract_inverted_index.language	142
abstract_inverted_index.personae	1, 22
abstract_inverted_index.provides	120
abstract_inverted_index.studies.	26
abstract_inverted_index.validity	19, 137
abstract_inverted_index.Synthetic	0
abstract_inverted_index.alignment	12, 144
abstract_inverted_index.checklist	104
abstract_inverted_index.criteria.	68
abstract_inverted_index.dependent	65
abstract_inverted_index.empirical	112
abstract_inverted_index.findings,	98
abstract_inverted_index.grounding	110
abstract_inverted_index.introduce	100
abstract_inverted_index.personae.	94
abstract_inverted_index.practical	129
abstract_inverted_index.practices	127
abstract_inverted_index.prominent	6
abstract_inverted_index.published	34
abstract_inverted_index.research,	13
abstract_inverted_index.research.	145
abstract_inverted_index.sampling,	108
abstract_inverted_index.validity.	117
abstract_inverted_index.assessment	124
abstract_inverted_index.attributes	84
abstract_inverted_index.discussing	88
abstract_inverted_index.ecological	18, 116, 136
abstract_inverted_index.emphasizes	106
abstract_inverted_index.guidelines	130
abstract_inverted_index.population	52
abstract_inverted_index.differences	73
abstract_inverted_index.evaluations	140
abstract_inverted_index.experiments	2
abstract_inverted_index.substantial	72
abstract_inverted_index.considerably	24
abstract_inverted_index.experiments,	60
abstract_inverted_index.transparency	103
abstract_inverted_index.comprehensive	123
abstract_inverted_index.fundamentally	64
abstract_inverted_index.peer-reviewed	32
abstract_inverted_index.persona-based	59, 139
abstract_inverted_index.representative	107
abstract_inverted_index.underspecified	57
abstract_inverted_index.personalization	62
abstract_inverted_index.representation,	76
abstract_inverted_index.sociodemographic	83
abstract_inverted_index.representativeness	16, 90
cited_by_percentile_year
countries_distinct_count	2
institutions_distinct_count	7
citation_normalized_percentile.value	0.54118648
citation_normalized_percentile.is_in_top_1_percent	False
citation_normalized_percentile.is_in_top_10_percent	True