LLaMEA-BO: A Large Language Model Evolutionary Algorithm for Automatically Generating Bayesian Optimization Algorithms Article Swipe

PDF

Wenhu Li , Niki van Stein , Thomas Bäck , Elena Raponi ·

YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2505.21034

Bayesian optimization (BO) is a powerful class of algorithms for optimizing expensive black-box functions, but designing effective BO algorithms remains a manual, expertise-driven task. Recent advancements in Large Language Models (LLMs) have opened new avenues for automating scientific discovery, including the automatic design of optimization algorithms. While prior work has used LLMs within optimization loops or to generate non-BO algorithms, we tackle a new challenge: Using LLMs to automatically generate full BO algorithm code. Our framework uses an evolution strategy to guide an LLM in generating Python code that preserves the key components of BO algorithms: An initial design, a surrogate model, and an acquisition function. The LLM is prompted to produce multiple candidate algorithms, which are evaluated on the established Black-Box Optimization Benchmarking (BBOB) test suite from the COmparing Continuous Optimizers (COCO) platform. Based on their performance, top candidates are selected, combined, and mutated via controlled prompt variations, enabling iterative refinement. Despite no additional fine-tuning, the LLM-generated algorithms outperform state-of-the-art BO baselines in 19 (out of 24) BBOB functions in dimension 5 and generalize well to higher dimensions, and different tasks (from the Bayesmark framework). This work demonstrates that LLMs can serve as algorithmic co-designers, offering a new paradigm for automating BO development and accelerating the discovery of novel algorithmic combinations. The source code is provided at https://github.com/Ewendawi/LLaMEA-BO.

Related Topics

C (Programming Language)

Language

Java (Programming Language)

Large Magellanic Cloud

Jerry & Marge Go Large

First Language

Slovene Language

Concepts

No concepts available.

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2505.21034
PDF: https://arxiv.org/pdf/2505.21034
OA Status: green
OpenAlex ID: https://openalex.org/W4415036730

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4415036730

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2505.21034

Digital Object Identifier
Title: LLaMEA-BO: A Large Language Model Evolutionary Algorithm for Automatically Generating Bayesian Optimization Algorithms

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2025

Year of publication
Publication date: 2025-05-27

Full publication date if available
Authors: Wenhu Li, Niki van Stein, Thomas Bäck, Elena Raponi

List of authors in order
Landing page: https://arxiv.org/abs/2505.21034

Publisher landing page
PDF URL: https://arxiv.org/pdf/2505.21034

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2505.21034

Direct OA link when available
Cited by: 0

Total citation count in OpenAlex

Full payload

id	https://openalex.org/W4415036730
doi	https://doi.org/10.48550/arxiv.2505.21034
ids.doi	https://doi.org/10.48550/arxiv.2505.21034
ids.openalex	https://openalex.org/W4415036730
fwci
type	preprint
title	LLaMEA-BO: A Large Language Model Evolutionary Algorithm for Automatically Generating Bayesian Optimization Algorithms
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T12031
topics[0].field.id	https://openalex.org/fields/17
topics[0].field.display_name	Computer Science
topics[0].score	0.9422000050544739
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/1702
topics[0].subfield.display_name	Artificial Intelligence
topics[0].display_name	Speech and dialogue systems
topics[1].id	https://openalex.org/T10028
topics[1].field.id	https://openalex.org/fields/17
topics[1].field.display_name	Computer Science
topics[1].score	0.9251000285148621
topics[1].domain.id	https://openalex.org/domains/3
topics[1].domain.display_name	Physical Sciences
topics[1].subfield.id	https://openalex.org/subfields/1702
topics[1].subfield.display_name	Artificial Intelligence
topics[1].display_name	Topic Modeling
topics[2].id	https://openalex.org/T12535
topics[2].field.id	https://openalex.org/fields/17
topics[2].field.display_name	Computer Science
topics[2].score	0.9164000153541565
topics[2].domain.id	https://openalex.org/domains/3
topics[2].domain.display_name	Physical Sciences
topics[2].subfield.id	https://openalex.org/subfields/1702
topics[2].subfield.display_name	Artificial Intelligence
topics[2].display_name	Machine Learning and Data Classification
is_xpac	False
apc_list
apc_paid
language	en
locations[0].id	pmh:oai:arXiv.org:2505.21034
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2505.21034
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2505.21034
locations[1].id	doi:10.48550/arxiv.2505.21034
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2505.21034
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5035083368
authorships[0].author.orcid	https://orcid.org/0000-0002-7795-5003
authorships[0].author.display_name	Wenhu Li
authorships[0].author_position	first
authorships[0].raw_author_name	Li, Wenhu
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5111231256
authorships[1].author.orcid
authorships[1].author.display_name	Niki van Stein
authorships[1].author_position	middle
authorships[1].raw_author_name	van Stein, Niki
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5062646838
authorships[2].author.orcid	https://orcid.org/0000-0001-6768-1478
authorships[2].author.display_name	Thomas Bäck
authorships[2].author_position	middle
authorships[2].raw_author_name	Bäck, Thomas
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5089662417
authorships[3].author.orcid	https://orcid.org/0000-0001-6841-7409
authorships[3].author.display_name	Elena Raponi
authorships[3].author_position	last
authorships[3].raw_author_name	Raponi, Elena
authorships[3].is_corresponding	False
has_content.pdf	True
has_content.grobid_xml	True
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2505.21034
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	LLaMEA-BO: A Large Language Model Evolutionary Algorithm for Automatically Generating Bayesian Optimization Algorithms
has_fulltext	True
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T12031
primary_topic.field.id	https://openalex.org/fields/17
primary_topic.field.display_name	Computer Science
primary_topic.score	0.9422000050544739
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1702
primary_topic.subfield.display_name	Artificial Intelligence
primary_topic.display_name	Speech and dialogue systems
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2505.21034
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2505.21034
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2505.21034
primary_location.id	pmh:oai:arXiv.org:2505.21034
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2505.21034
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2505.21034
publication_date	2025-05-27
publication_year	2025
referenced_works_count	0
abstract_inverted_index.5	172
abstract_inverted_index.a	4, 20, 62, 99, 197
abstract_inverted_index.19	164
abstract_inverted_index.An	96
abstract_inverted_index.BO	17, 71, 94, 161, 202
abstract_inverted_index.an	77, 82, 103
abstract_inverted_index.as	193
abstract_inverted_index.at	217
abstract_inverted_index.in	26, 84, 163, 170
abstract_inverted_index.is	3, 108, 215
abstract_inverted_index.no	153
abstract_inverted_index.of	7, 43, 93, 166, 208
abstract_inverted_index.on	118, 135
abstract_inverted_index.or	55
abstract_inverted_index.to	56, 67, 80, 110, 176
abstract_inverted_index.we	60
abstract_inverted_index.24)	167
abstract_inverted_index.LLM	83, 107
abstract_inverted_index.Our	74
abstract_inverted_index.The	106, 212
abstract_inverted_index.and	102, 143, 173, 179, 204
abstract_inverted_index.are	116, 140
abstract_inverted_index.but	14
abstract_inverted_index.can	191
abstract_inverted_index.for	9, 35, 200
abstract_inverted_index.has	49
abstract_inverted_index.key	91
abstract_inverted_index.new	33, 63, 198
abstract_inverted_index.the	40, 90, 119, 128, 156, 183, 206
abstract_inverted_index.top	138
abstract_inverted_index.via	145
abstract_inverted_index.(BO)	2
abstract_inverted_index.(out	165
abstract_inverted_index.BBOB	168
abstract_inverted_index.LLMs	51, 66, 190
abstract_inverted_index.This	186
abstract_inverted_index.code	87, 214
abstract_inverted_index.from	127
abstract_inverted_index.full	70
abstract_inverted_index.have	31
abstract_inverted_index.test	125
abstract_inverted_index.that	88, 189
abstract_inverted_index.used	50
abstract_inverted_index.uses	76
abstract_inverted_index.well	175
abstract_inverted_index.work	48, 187
abstract_inverted_index.(from	182
abstract_inverted_index.Based	134
abstract_inverted_index.Large	27
abstract_inverted_index.Using	65
abstract_inverted_index.While	46
abstract_inverted_index.class	6
abstract_inverted_index.code.	73
abstract_inverted_index.guide	81
abstract_inverted_index.loops	54
abstract_inverted_index.novel	209
abstract_inverted_index.prior	47
abstract_inverted_index.serve	192
abstract_inverted_index.suite	126
abstract_inverted_index.task.	23
abstract_inverted_index.tasks	181
abstract_inverted_index.their	136
abstract_inverted_index.which	115
abstract_inverted_index.(BBOB)	124
abstract_inverted_index.(COCO)	132
abstract_inverted_index.(LLMs)	30
abstract_inverted_index.Models	29
abstract_inverted_index.Python	86
abstract_inverted_index.Recent	24
abstract_inverted_index.design	42
abstract_inverted_index.higher	177
abstract_inverted_index.model,	101
abstract_inverted_index.non-BO	58
abstract_inverted_index.opened	32
abstract_inverted_index.prompt	147
abstract_inverted_index.source	213
abstract_inverted_index.tackle	61
abstract_inverted_index.within	52
abstract_inverted_index.Despite	152
abstract_inverted_index.avenues	34
abstract_inverted_index.design,	98
abstract_inverted_index.initial	97
abstract_inverted_index.manual,	21
abstract_inverted_index.mutated	144
abstract_inverted_index.produce	111
abstract_inverted_index.remains	19
abstract_inverted_index.Bayesian	0
abstract_inverted_index.Language	28
abstract_inverted_index.enabling	149
abstract_inverted_index.generate	57, 69
abstract_inverted_index.multiple	112
abstract_inverted_index.offering	196
abstract_inverted_index.paradigm	199
abstract_inverted_index.powerful	5
abstract_inverted_index.prompted	109
abstract_inverted_index.provided	216
abstract_inverted_index.strategy	79
abstract_inverted_index.Bayesmark	184
abstract_inverted_index.Black-Box	121
abstract_inverted_index.COmparing	129
abstract_inverted_index.algorithm	72
abstract_inverted_index.automatic	41
abstract_inverted_index.baselines	162
abstract_inverted_index.black-box	12
abstract_inverted_index.candidate	113
abstract_inverted_index.combined,	142
abstract_inverted_index.designing	15
abstract_inverted_index.different	180
abstract_inverted_index.dimension	171
abstract_inverted_index.discovery	207
abstract_inverted_index.effective	16
abstract_inverted_index.evaluated	117
abstract_inverted_index.evolution	78
abstract_inverted_index.expensive	11
abstract_inverted_index.framework	75
abstract_inverted_index.function.	105
abstract_inverted_index.functions	169
abstract_inverted_index.including	39
abstract_inverted_index.iterative	150
abstract_inverted_index.platform.	133
abstract_inverted_index.preserves	89
abstract_inverted_index.selected,	141
abstract_inverted_index.surrogate	100
abstract_inverted_index.Continuous	130
abstract_inverted_index.Optimizers	131
abstract_inverted_index.additional	154
abstract_inverted_index.algorithms	8, 18, 158
abstract_inverted_index.automating	36, 201
abstract_inverted_index.candidates	139
abstract_inverted_index.challenge:	64
abstract_inverted_index.components	92
abstract_inverted_index.controlled	146
abstract_inverted_index.discovery,	38
abstract_inverted_index.functions,	13
abstract_inverted_index.generalize	174
abstract_inverted_index.generating	85
abstract_inverted_index.optimizing	10
abstract_inverted_index.outperform	159
abstract_inverted_index.scientific	37
abstract_inverted_index.acquisition	104
abstract_inverted_index.algorithmic	194, 210
abstract_inverted_index.algorithms,	59, 114
abstract_inverted_index.algorithms.	45
abstract_inverted_index.algorithms:	95
abstract_inverted_index.development	203
abstract_inverted_index.dimensions,	178
abstract_inverted_index.established	120
abstract_inverted_index.framework).	185
abstract_inverted_index.refinement.	151
abstract_inverted_index.variations,	148
abstract_inverted_index.Benchmarking	123
abstract_inverted_index.Optimization	122
abstract_inverted_index.accelerating	205
abstract_inverted_index.advancements	25
abstract_inverted_index.demonstrates	188
abstract_inverted_index.fine-tuning,	155
abstract_inverted_index.optimization	1, 44, 53
abstract_inverted_index.performance,	137
abstract_inverted_index.LLM-generated	157
abstract_inverted_index.automatically	68
abstract_inverted_index.co-designers,	195
abstract_inverted_index.combinations.	211
abstract_inverted_index.expertise-driven	22
abstract_inverted_index.state-of-the-art	160
abstract_inverted_index.https://github.com/Ewendawi/LLaMEA-BO.	218
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	4
citation_normalized_percentile