UM_FHS at TREC 2024 PLABA: Exploration of Fine-tuning and AI agent approach for plain language adaptations of biomedical text Article Swipe

PDF

Primož Kocbek , Leon Kopitar , Zhihong Zhang , Emin D. Aydin , Maxim Topaz , Gregor Štiglic ·

YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2502.14144

This paper describes our submissions to the TREC 2024 PLABA track with the aim to simplify biomedical abstracts for a K8-level audience (13-14 years old students). We tested three approaches using OpenAI's gpt-4o and gpt-4o-mini models: baseline prompt engineering, a two-AI agent approach, and fine-tuning. Adaptations were evaluated using qualitative metrics (5-point Likert scales for simplicity, accuracy, completeness, and brevity) and quantitative readability scores (Flesch-Kincaid grade level, SMOG Index). Results indicated that the two-agent approach and baseline prompt engineering with gpt-4o-mini models show superior qualitative performance, while fine-tuned models excelled in accuracy and completeness but were less simple. The evaluation results demonstrated that prompt engineering with gpt-4o-mini outperforms iterative improvement strategies via two-agent approach as well as fine-tuning with gpt-4o. We intend to expand our investigation of the results and explore advanced evaluations.

Related Topics

Computer Science

Plain Text

Artificial Intelligence

Philosophy

Computer Security

Concepts

Computer science Plain text Natural language processing Plain language Artificial intelligence Linguistics Philosophy Computer security Encryption

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2502.14144
PDF: https://arxiv.org/pdf/2502.14144
OA Status: green
Related Works: 10
OpenAlex ID: https://openalex.org/W4407806344

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4407806344

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2502.14144

Digital Object Identifier
Title: UM_FHS at TREC 2024 PLABA: Exploration of Fine-tuning and AI agent approach for plain language adaptations of biomedical text

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2025

Year of publication
Publication date: 2025-02-19

Full publication date if available
Authors: Primož Kocbek, Leon Kopitar, Zhihong Zhang, Emin D. Aydin, Maxim Topaz, Gregor Štiglic

List of authors in order
Landing page: https://arxiv.org/abs/2502.14144

Publisher landing page
PDF URL: https://arxiv.org/pdf/2502.14144

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2502.14144

Direct OA link when available
Concepts: Computer science, Plain text, Natural language processing, Plain language, Artificial intelligence, Linguistics, Philosophy, Computer security, Encryption

Top concepts (fields/topics) attached by OpenAlex
Cited by: 0

Total citation count in OpenAlex
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4407806344
doi	https://doi.org/10.48550/arxiv.2502.14144
ids.doi	https://doi.org/10.48550/arxiv.2502.14144
ids.openalex	https://openalex.org/W4407806344
fwci
type	preprint
title	UM_FHS at TREC 2024 PLABA: Exploration of Fine-tuning and AI agent approach for plain language adaptations of biomedical text
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T10028
topics[0].field.id	https://openalex.org/fields/17
topics[0].field.display_name	Computer Science
topics[0].score	0.963699996471405
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/1702
topics[0].subfield.display_name	Artificial Intelligence
topics[0].display_name	Topic Modeling
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C41008148
concepts[0].level	0
concepts[0].score	0.5103363394737244
concepts[0].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[0].display_name	Computer science
concepts[1].id	https://openalex.org/C46503548
concepts[1].level	3
concepts[1].score	0.48785400390625
concepts[1].wikidata	https://www.wikidata.org/wiki/Q1145976
concepts[1].display_name	Plain text
concepts[2].id	https://openalex.org/C204321447
concepts[2].level	1
concepts[2].score	0.45145121216773987
concepts[2].wikidata	https://www.wikidata.org/wiki/Q30642
concepts[2].display_name	Natural language processing
concepts[3].id	https://openalex.org/C2777083192
concepts[3].level	2
concepts[3].score	0.4406340420246124
concepts[3].wikidata	https://www.wikidata.org/wiki/Q1814648
concepts[3].display_name	Plain language
concepts[4].id	https://openalex.org/C154945302
concepts[4].level	1
concepts[4].score	0.4300966262817383
concepts[4].wikidata	https://www.wikidata.org/wiki/Q11660
concepts[4].display_name	Artificial intelligence
concepts[5].id	https://openalex.org/C41895202
concepts[5].level	1
concepts[5].score	0.2988351583480835
concepts[5].wikidata	https://www.wikidata.org/wiki/Q8162
concepts[5].display_name	Linguistics
concepts[6].id	https://openalex.org/C138885662
concepts[6].level	0
concepts[6].score	0.08521753549575806
concepts[6].wikidata	https://www.wikidata.org/wiki/Q5891
concepts[6].display_name	Philosophy
concepts[7].id	https://openalex.org/C38652104
concepts[7].level	1
concepts[7].score	0.05916190147399902
concepts[7].wikidata	https://www.wikidata.org/wiki/Q3510521
concepts[7].display_name	Computer security
concepts[8].id	https://openalex.org/C148730421
concepts[8].level	2
concepts[8].score	0.0
concepts[8].wikidata	https://www.wikidata.org/wiki/Q141090
concepts[8].display_name	Encryption
keywords[0].id	https://openalex.org/keywords/computer-science
keywords[0].score	0.5103363394737244
keywords[0].display_name	Computer science
keywords[1].id	https://openalex.org/keywords/plain-text
keywords[1].score	0.48785400390625
keywords[1].display_name	Plain text
keywords[2].id	https://openalex.org/keywords/natural-language-processing
keywords[2].score	0.45145121216773987
keywords[2].display_name	Natural language processing
keywords[3].id	https://openalex.org/keywords/plain-language
keywords[3].score	0.4406340420246124
keywords[3].display_name	Plain language
keywords[4].id	https://openalex.org/keywords/artificial-intelligence
keywords[4].score	0.4300966262817383
keywords[4].display_name	Artificial intelligence
keywords[5].id	https://openalex.org/keywords/linguistics
keywords[5].score	0.2988351583480835
keywords[5].display_name	Linguistics
keywords[6].id	https://openalex.org/keywords/philosophy
keywords[6].score	0.08521753549575806
keywords[6].display_name	Philosophy
keywords[7].id	https://openalex.org/keywords/computer-security
keywords[7].score	0.05916190147399902
keywords[7].display_name	Computer security
language	en
locations[0].id	pmh:oai:arXiv.org:2502.14144
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2502.14144
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2502.14144
locations[1].id	doi:10.48550/arxiv.2502.14144
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2502.14144
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5022754001
authorships[0].author.orcid	https://orcid.org/0000-0002-9064-5085
authorships[0].author.display_name	Primož Kocbek
authorships[0].author_position	first
authorships[0].raw_author_name	Kocbek, Primoz
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5067333520
authorships[1].author.orcid	https://orcid.org/0000-0002-6647-9988
authorships[1].author.display_name	Leon Kopitar
authorships[1].author_position	middle
authorships[1].raw_author_name	Kopitar, Leon
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5100419475
authorships[2].author.orcid	https://orcid.org/0000-0002-9948-8936
authorships[2].author.display_name	Zhihong Zhang
authorships[2].author_position	middle
authorships[2].raw_author_name	Zhang, Zhihong
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5110259561
authorships[3].author.orcid
authorships[3].author.display_name	Emin D. Aydin
authorships[3].author_position	middle
authorships[3].raw_author_name	Aydin, Emirhan
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5021465135
authorships[4].author.orcid	https://orcid.org/0000-0002-2358-9837
authorships[4].author.display_name	Maxim Topaz
authorships[4].author_position	middle
authorships[4].raw_author_name	Topaz, Maxim
authorships[4].is_corresponding	False
authorships[5].author.id	https://openalex.org/A5017783286
authorships[5].author.orcid	https://orcid.org/0000-0002-0183-8679
authorships[5].author.display_name	Gregor Štiglic
authorships[5].author_position	last
authorships[5].raw_author_name	Stiglic, Gregor
authorships[5].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2502.14144
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	UM_FHS at TREC 2024 PLABA: Exploration of Fine-tuning and AI agent approach for plain language adaptations of biomedical text
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T10028
primary_topic.field.id	https://openalex.org/fields/17
primary_topic.field.display_name	Computer Science
primary_topic.score	0.963699996471405
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1702
primary_topic.subfield.display_name	Artificial Intelligence
primary_topic.display_name	Topic Modeling
related_works	https://openalex.org/W2207738994, https://openalex.org/W3204019825, https://openalex.org/W4404840725, https://openalex.org/W3214310931, https://openalex.org/W4399238075, https://openalex.org/W2998443060, https://openalex.org/W3128842176, https://openalex.org/W4221158077, https://openalex.org/W4385321705, https://openalex.org/W856095404
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2502.14144
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2502.14144
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2502.14144
primary_location.id	pmh:oai:arXiv.org:2502.14144
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2502.14144
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2502.14144
publication_date	2025-02-19
publication_year	2025
referenced_works_count	0
abstract_inverted_index.a	19, 39
abstract_inverted_index.We	26, 120
abstract_inverted_index.as	114, 116
abstract_inverted_index.in	90
abstract_inverted_index.of	126
abstract_inverted_index.to	5, 14, 122
abstract_inverted_index.The	98
abstract_inverted_index.aim	13
abstract_inverted_index.and	33, 43, 58, 60, 75, 92, 129
abstract_inverted_index.but	94
abstract_inverted_index.for	18, 54
abstract_inverted_index.old	24
abstract_inverted_index.our	3, 124
abstract_inverted_index.the	6, 12, 72, 127
abstract_inverted_index.via	111
abstract_inverted_index.2024	8
abstract_inverted_index.SMOG	67
abstract_inverted_index.TREC	7
abstract_inverted_index.This	0
abstract_inverted_index.less	96
abstract_inverted_index.show	82
abstract_inverted_index.that	71, 102
abstract_inverted_index.well	115
abstract_inverted_index.were	46, 95
abstract_inverted_index.with	11, 79, 105, 118
abstract_inverted_index.PLABA	9
abstract_inverted_index.agent	41
abstract_inverted_index.grade	65
abstract_inverted_index.paper	1
abstract_inverted_index.three	28
abstract_inverted_index.track	10
abstract_inverted_index.using	30, 48
abstract_inverted_index.while	86
abstract_inverted_index.years	23
abstract_inverted_index.(13-14	22
abstract_inverted_index.Likert	52
abstract_inverted_index.expand	123
abstract_inverted_index.gpt-4o	32
abstract_inverted_index.intend	121
abstract_inverted_index.level,	66
abstract_inverted_index.models	81, 88
abstract_inverted_index.prompt	37, 77, 103
abstract_inverted_index.scales	53
abstract_inverted_index.scores	63
abstract_inverted_index.tested	27
abstract_inverted_index.two-AI	40
abstract_inverted_index.Index).	68
abstract_inverted_index.Results	69
abstract_inverted_index.explore	130
abstract_inverted_index.gpt-4o.	119
abstract_inverted_index.metrics	50
abstract_inverted_index.models:	35
abstract_inverted_index.results	100, 128
abstract_inverted_index.simple.	97
abstract_inverted_index.(5-point	51
abstract_inverted_index.K8-level	20
abstract_inverted_index.OpenAI's	31
abstract_inverted_index.accuracy	91
abstract_inverted_index.advanced	131
abstract_inverted_index.approach	74, 113
abstract_inverted_index.audience	21
abstract_inverted_index.baseline	36, 76
abstract_inverted_index.brevity)	59
abstract_inverted_index.excelled	89
abstract_inverted_index.simplify	15
abstract_inverted_index.superior	83
abstract_inverted_index.abstracts	17
abstract_inverted_index.accuracy,	56
abstract_inverted_index.approach,	42
abstract_inverted_index.describes	2
abstract_inverted_index.evaluated	47
abstract_inverted_index.indicated	70
abstract_inverted_index.iterative	108
abstract_inverted_index.two-agent	73, 112
abstract_inverted_index.approaches	29
abstract_inverted_index.biomedical	16
abstract_inverted_index.evaluation	99
abstract_inverted_index.fine-tuned	87
abstract_inverted_index.strategies	110
abstract_inverted_index.students).	25
abstract_inverted_index.Adaptations	45
abstract_inverted_index.engineering	78, 104
abstract_inverted_index.fine-tuning	117
abstract_inverted_index.gpt-4o-mini	34, 80, 106
abstract_inverted_index.improvement	109
abstract_inverted_index.outperforms	107
abstract_inverted_index.qualitative	49, 84
abstract_inverted_index.readability	62
abstract_inverted_index.simplicity,	55
abstract_inverted_index.submissions	4
abstract_inverted_index.completeness	93
abstract_inverted_index.demonstrated	101
abstract_inverted_index.engineering,	38
abstract_inverted_index.evaluations.	132
abstract_inverted_index.fine-tuning.	44
abstract_inverted_index.performance,	85
abstract_inverted_index.quantitative	61
abstract_inverted_index.completeness,	57
abstract_inverted_index.investigation	125
abstract_inverted_index.(Flesch-Kincaid	64
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	6
citation_normalized_percentile