BOND: Aligning LLMs with Best-of-N Distillation Article Swipe

PDF

Pier Giuseppe Sessa , Robert Dadashi , Léonard Hussenot , Johan Ferret , Nino Vieillard , Alexandre Ramé , Bobak Shariari , Sarah Perrin , Abe Friesen , Geoffrey Cideron , Sertan Girgin , Piotr Stańczyk , Andrea Michi , Danila Sinopalnikov , Sabela Ramos , Amélie Héliou , Aliaksei Severyn , Matt Hoffman , Nikola Momchev , Olivier Bachem ·

YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2407.14622

Reinforcement learning from human feedback (RLHF) is a key driver of quality and safety in state-of-the-art large language models. Yet, a surprisingly simple and strong inference-time strategy is Best-of-N sampling that selects the best generation among N candidates. In this paper, we propose Best-of-N Distillation (BOND), a novel RLHF algorithm that seeks to emulate Best-of-N but without its significant computational overhead at inference time. Specifically, BOND is a distribution matching algorithm that forces the distribution of generations from the policy to get closer to the Best-of-N distribution. We use the Jeffreys divergence (a linear combination of forward and backward KL) to balance between mode-covering and mode-seeking behavior, and derive an iterative formulation that utilizes a moving anchor for efficiency. We demonstrate the effectiveness of our approach and several design choices through experiments on abstractive summarization and Gemma models. Aligning Gemma policies with BOND outperforms other RLHF algorithms by improving results on several benchmarks.

Related Topics

Concepts

Bond Distillation Business Economics Chemistry Organic chemistry Finance

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2407.14622
PDF: https://arxiv.org/pdf/2407.14622
OA Status: green
Cited By: 1
Related Works: 10
OpenAlex ID: https://openalex.org/W4406072085

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4406072085

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2407.14622

Digital Object Identifier
Title: BOND: Aligning LLMs with Best-of-N Distillation

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2024

Year of publication
Publication date: 2024-07-19

Full publication date if available
Authors: Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Nino Vieillard, Alexandre Ramé, Bobak Shariari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, Sertan Girgin, Piotr Stańczyk, Andrea Michi, Danila Sinopalnikov, Sabela Ramos, Amélie Héliou, Aliaksei Severyn, Matt Hoffman, Nikola Momchev, Olivier Bachem

List of authors in order
Landing page: https://arxiv.org/abs/2407.14622

Publisher landing page
PDF URL: https://arxiv.org/pdf/2407.14622

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2407.14622

Direct OA link when available
Concepts: Bond, Distillation, Business, Economics, Chemistry, Organic chemistry, Finance

Top concepts (fields/topics) attached by OpenAlex
Cited by: 1

Total citation count in OpenAlex
Citations by year (recent): 2025: 1

Per-year citation counts (last 5 years)
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4406072085
doi	https://doi.org/10.48550/arxiv.2407.14622
ids.doi	https://doi.org/10.48550/arxiv.2407.14622
ids.openalex	https://openalex.org/W4406072085
fwci
type	preprint
title	BOND: Aligning LLMs with Best-of-N Distillation
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T10791
topics[0].field.id	https://openalex.org/fields/22
topics[0].field.display_name	Engineering
topics[0].score	0.7631000280380249
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/2207
topics[0].subfield.display_name	Control and Systems Engineering
topics[0].display_name	Advanced Control Systems Optimization
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C69738904
concepts[0].level	2
concepts[0].score	0.6489981412887573
concepts[0].wikidata	https://www.wikidata.org/wiki/Q11693
concepts[0].display_name	Bond
concepts[1].id	https://openalex.org/C204030448
concepts[1].level	2
concepts[1].score	0.5593911409378052
concepts[1].wikidata	https://www.wikidata.org/wiki/Q101017
concepts[1].display_name	Distillation
concepts[2].id	https://openalex.org/C144133560
concepts[2].level	0
concepts[2].score	0.3852052688598633
concepts[2].wikidata	https://www.wikidata.org/wiki/Q4830453
concepts[2].display_name	Business
concepts[3].id	https://openalex.org/C162324750
concepts[3].level	0
concepts[3].score	0.3450556993484497
concepts[3].wikidata	https://www.wikidata.org/wiki/Q8134
concepts[3].display_name	Economics
concepts[4].id	https://openalex.org/C185592680
concepts[4].level	0
concepts[4].score	0.2765577435493469
concepts[4].wikidata	https://www.wikidata.org/wiki/Q2329
concepts[4].display_name	Chemistry
concepts[5].id	https://openalex.org/C178790620
concepts[5].level	1
concepts[5].score	0.16317424178123474
concepts[5].wikidata	https://www.wikidata.org/wiki/Q11351
concepts[5].display_name	Organic chemistry
concepts[6].id	https://openalex.org/C10138342
concepts[6].level	1
concepts[6].score	0.133754163980484
concepts[6].wikidata	https://www.wikidata.org/wiki/Q43015
concepts[6].display_name	Finance
keywords[0].id	https://openalex.org/keywords/bond
keywords[0].score	0.6489981412887573
keywords[0].display_name	Bond
keywords[1].id	https://openalex.org/keywords/distillation
keywords[1].score	0.5593911409378052
keywords[1].display_name	Distillation
keywords[2].id	https://openalex.org/keywords/business
keywords[2].score	0.3852052688598633
keywords[2].display_name	Business
keywords[3].id	https://openalex.org/keywords/economics
keywords[3].score	0.3450556993484497
keywords[3].display_name	Economics
keywords[4].id	https://openalex.org/keywords/chemistry
keywords[4].score	0.2765577435493469
keywords[4].display_name	Chemistry
keywords[5].id	https://openalex.org/keywords/organic-chemistry
keywords[5].score	0.16317424178123474
keywords[5].display_name	Organic chemistry
keywords[6].id	https://openalex.org/keywords/finance
keywords[6].score	0.133754163980484
keywords[6].display_name	Finance
language	en
locations[0].id	pmh:oai:arXiv.org:2407.14622
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2407.14622
locations[0].version	submittedVersion
locations[0].raw_type
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2407.14622
locations[1].id	doi:10.48550/arxiv.2407.14622
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license	cc-by
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id	https://openalex.org/licenses/cc-by
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2407.14622
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5035760608
authorships[0].author.orcid	https://orcid.org/0000-0001-8986-8815
authorships[0].author.display_name	Pier Giuseppe Sessa
authorships[0].author_position	first
authorships[0].raw_author_name	Sessa, Pier Giuseppe
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5043577372
authorships[1].author.orcid
authorships[1].author.display_name	Robert Dadashi
authorships[1].author_position	middle
authorships[1].raw_author_name	Dadashi, Robert
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5079135104
authorships[2].author.orcid
authorships[2].author.display_name	Léonard Hussenot
authorships[2].author_position	middle
authorships[2].raw_author_name	Hussenot, Léonard
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5087706654
authorships[3].author.orcid
authorships[3].author.display_name	Johan Ferret
authorships[3].author_position	middle
authorships[3].raw_author_name	Ferret, Johan
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5071735499
authorships[4].author.orcid
authorships[4].author.display_name	Nino Vieillard
authorships[4].author_position	middle
authorships[4].raw_author_name	Vieillard, Nino
authorships[4].is_corresponding	False
authorships[5].author.id	https://openalex.org/A5057217987
authorships[5].author.orcid
authorships[5].author.display_name	Alexandre Ramé
authorships[5].author_position	middle
authorships[5].raw_author_name	Ramé, Alexandre
authorships[5].is_corresponding	False
authorships[6].author.id	https://openalex.org/A5115771709
authorships[6].author.orcid
authorships[6].author.display_name	Bobak Shariari
authorships[6].author_position	middle
authorships[6].raw_author_name	Shariari, Bobak
authorships[6].is_corresponding	False
authorships[7].author.id	https://openalex.org/A5031705919
authorships[7].author.orcid	https://orcid.org/0000-0002-4488-6689
authorships[7].author.display_name	Sarah Perrin
authorships[7].author_position	middle
authorships[7].raw_author_name	Perrin, Sarah
authorships[7].is_corresponding	False
authorships[8].author.id	https://openalex.org/A5039658137
authorships[8].author.orcid
authorships[8].author.display_name	Abe Friesen
authorships[8].author_position	middle
authorships[8].raw_author_name	Friesen, Abe
authorships[8].is_corresponding	False
authorships[9].author.id	https://openalex.org/A5028486593
authorships[9].author.orcid
authorships[9].author.display_name	Geoffrey Cideron
authorships[9].author_position	middle
authorships[9].raw_author_name	Cideron, Geoffrey
authorships[9].is_corresponding	False
authorships[10].author.id	https://openalex.org/A5103445258
authorships[10].author.orcid
authorships[10].author.display_name	Sertan Girgin
authorships[10].author_position	middle
authorships[10].raw_author_name	Girgin, Sertan
authorships[10].is_corresponding	False
authorships[11].author.id	https://openalex.org/A5013012032
authorships[11].author.orcid	https://orcid.org/0000-0003-0124-2936
authorships[11].author.display_name	Piotr Stańczyk
authorships[11].author_position	middle
authorships[11].raw_author_name	Stanczyk, Piotr
authorships[11].is_corresponding	False
authorships[12].author.id	https://openalex.org/A5046831695
authorships[12].author.orcid	https://orcid.org/0009-0001-4797-3593
authorships[12].author.display_name	Andrea Michi
authorships[12].author_position	middle
authorships[12].raw_author_name	Michi, Andrea
authorships[12].is_corresponding	False
authorships[13].author.id	https://openalex.org/A5025950368
authorships[13].author.orcid
authorships[13].author.display_name	Danila Sinopalnikov
authorships[13].author_position	middle
authorships[13].raw_author_name	Sinopalnikov, Danila
authorships[13].is_corresponding	False
authorships[14].author.id	https://openalex.org/A5035211063
authorships[14].author.orcid	https://orcid.org/0000-0001-6656-9732
authorships[14].author.display_name	Sabela Ramos
authorships[14].author_position	middle
authorships[14].raw_author_name	Ramos, Sabela
authorships[14].is_corresponding	False
authorships[15].author.id	https://openalex.org/A5023095596
authorships[15].author.orcid
authorships[15].author.display_name	Amélie Héliou
authorships[15].author_position	middle
authorships[15].raw_author_name	Héliou, Amélie
authorships[15].is_corresponding	False
authorships[16].author.id	https://openalex.org/A5080125160
authorships[16].author.orcid	https://orcid.org/0009-0003-2954-4167
authorships[16].author.display_name	Aliaksei Severyn
authorships[16].author_position	middle
authorships[16].raw_author_name	Severyn, Aliaksei
authorships[16].is_corresponding	False
authorships[17].author.id	https://openalex.org/A5108813586
authorships[17].author.orcid
authorships[17].author.display_name	Matt Hoffman
authorships[17].author_position	middle
authorships[17].raw_author_name	Hoffman, Matt
authorships[17].is_corresponding	False
authorships[18].author.id	https://openalex.org/A5049028265
authorships[18].author.orcid
authorships[18].author.display_name	Nikola Momchev
authorships[18].author_position	middle
authorships[18].raw_author_name	Momchev, Nikola
authorships[18].is_corresponding	False
authorships[19].author.id	https://openalex.org/A5089890773
authorships[19].author.orcid
authorships[19].author.display_name	Olivier Bachem
authorships[19].author_position	last
authorships[19].raw_author_name	Bachem, Olivier
authorships[19].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2407.14622
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	BOND: Aligning LLMs with Best-of-N Distillation
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T10791
primary_topic.field.id	https://openalex.org/fields/22
primary_topic.field.display_name	Engineering
primary_topic.score	0.7631000280380249
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/2207
primary_topic.subfield.display_name	Control and Systems Engineering
primary_topic.display_name	Advanced Control Systems Optimization
related_works	https://openalex.org/W4391375266, https://openalex.org/W3026162553, https://openalex.org/W2344382886, https://openalex.org/W19111321, https://openalex.org/W2412887479, https://openalex.org/W32245304, https://openalex.org/W1502198272, https://openalex.org/W2017540542, https://openalex.org/W2953684491, https://openalex.org/W4285338581
cited_by_count	1
counts_by_year[0].year	2025
counts_by_year[0].cited_by_count	1
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2407.14622
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2407.14622
best_oa_location.version	submittedVersion
best_oa_location.raw_type
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2407.14622
primary_location.id	pmh:oai:arXiv.org:2407.14622
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2407.14622
primary_location.version	submittedVersion
primary_location.raw_type
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2407.14622
publication_date	2024-07-19
publication_year	2024
referenced_works_count	0
abstract_inverted_index.N	36
abstract_inverted_index.a	7, 20, 46, 67, 114
abstract_inverted_index.(a	92
abstract_inverted_index.In	38
abstract_inverted_index.We	87, 119
abstract_inverted_index.an	109
abstract_inverted_index.at	61
abstract_inverted_index.by	147
abstract_inverted_index.in	14
abstract_inverted_index.is	6, 27, 66
abstract_inverted_index.of	10, 75, 95, 123
abstract_inverted_index.on	132, 150
abstract_inverted_index.to	52, 80, 83, 100
abstract_inverted_index.we	41
abstract_inverted_index.KL)	99
abstract_inverted_index.and	12, 23, 97, 104, 107, 126, 135
abstract_inverted_index.but	55
abstract_inverted_index.for	117
abstract_inverted_index.get	81
abstract_inverted_index.its	57
abstract_inverted_index.key	8
abstract_inverted_index.our	124
abstract_inverted_index.the	32, 73, 78, 84, 89, 121
abstract_inverted_index.use	88
abstract_inverted_index.BOND	65, 142
abstract_inverted_index.RLHF	48, 145
abstract_inverted_index.Yet,	19
abstract_inverted_index.best	33
abstract_inverted_index.from	2, 77
abstract_inverted_index.that	30, 50, 71, 112
abstract_inverted_index.this	39
abstract_inverted_index.with	141
abstract_inverted_index.Gemma	136, 139
abstract_inverted_index.among	35
abstract_inverted_index.human	3
abstract_inverted_index.large	16
abstract_inverted_index.novel	47
abstract_inverted_index.other	144
abstract_inverted_index.seeks	51
abstract_inverted_index.time.	63
abstract_inverted_index.(RLHF)	5
abstract_inverted_index.anchor	116
abstract_inverted_index.closer	82
abstract_inverted_index.derive	108
abstract_inverted_index.design	128
abstract_inverted_index.driver	9
abstract_inverted_index.forces	72
abstract_inverted_index.linear	93
abstract_inverted_index.moving	115
abstract_inverted_index.paper,	40
abstract_inverted_index.policy	79
abstract_inverted_index.safety	13
abstract_inverted_index.simple	22
abstract_inverted_index.strong	24
abstract_inverted_index.(BOND),	45
abstract_inverted_index.balance	101
abstract_inverted_index.between	102
abstract_inverted_index.choices	129
abstract_inverted_index.emulate	53
abstract_inverted_index.forward	96
abstract_inverted_index.models.	18, 137
abstract_inverted_index.propose	42
abstract_inverted_index.quality	11
abstract_inverted_index.results	149
abstract_inverted_index.selects	31
abstract_inverted_index.several	127, 151
abstract_inverted_index.through	130
abstract_inverted_index.without	56
abstract_inverted_index.Aligning	138
abstract_inverted_index.Jeffreys	90
abstract_inverted_index.approach	125
abstract_inverted_index.backward	98
abstract_inverted_index.feedback	4
abstract_inverted_index.language	17
abstract_inverted_index.learning	1
abstract_inverted_index.matching	69
abstract_inverted_index.overhead	60
abstract_inverted_index.policies	140
abstract_inverted_index.sampling	29
abstract_inverted_index.strategy	26
abstract_inverted_index.utilizes	113
abstract_inverted_index.Best-of-N	28, 43, 54, 85
abstract_inverted_index.algorithm	49, 70
abstract_inverted_index.behavior,	106
abstract_inverted_index.improving	148
abstract_inverted_index.inference	62
abstract_inverted_index.iterative	110
abstract_inverted_index.algorithms	146
abstract_inverted_index.divergence	91
abstract_inverted_index.generation	34
abstract_inverted_index.abstractive	133
abstract_inverted_index.benchmarks.	152
abstract_inverted_index.candidates.	37
abstract_inverted_index.combination	94
abstract_inverted_index.demonstrate	120
abstract_inverted_index.efficiency.	118
abstract_inverted_index.experiments	131
abstract_inverted_index.formulation	111
abstract_inverted_index.generations	76
abstract_inverted_index.outperforms	143
abstract_inverted_index.significant	58
abstract_inverted_index.Distillation	44
abstract_inverted_index.distribution	68, 74
abstract_inverted_index.mode-seeking	105
abstract_inverted_index.surprisingly	21
abstract_inverted_index.Reinforcement	0
abstract_inverted_index.Specifically,	64
abstract_inverted_index.computational	59
abstract_inverted_index.distribution.	86
abstract_inverted_index.effectiveness	122
abstract_inverted_index.mode-covering	103
abstract_inverted_index.summarization	134
abstract_inverted_index.inference-time	25
abstract_inverted_index.state-of-the-art	15
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	20
citation_normalized_percentile