Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuning Article Swipe

PDF

Kaiwen Wang , Rahul Kidambi , Ryan Sullivan , A. Agarwal , Christoph Dann , Andrea Michi , Marco Gelmi , Yunxuan Li , Raghav Gupta , Avinava Dubey , Alexandre Ramé , Johan Ferret , Geoffrey Cideron , Le Hou , Hongkun Yu , Amr Ahmed , Aranyak Mehta , Léonard Hussenot , Olivier Bachem , Edouard Leurent ·

YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2407.15762

Reward-based finetuning is crucial for aligning language policies with intended behaviors (e.g., creativity and safety). A key challenge is to develop steerable language models that trade-off multiple (conflicting) objectives in a flexible and efficient manner. This paper presents Conditional Language Policy (CLP), a general framework for finetuning language models on multiple objectives. Building on techniques from multi-task training and parameter-efficient finetuning, CLP learn steerable models that effectively trade-off conflicting objectives at inference time. Notably, this does not require training or maintaining multiple models to achieve different trade-offs between the objectives. Through extensive experiments and ablations on two summarization datasets, we show that CLP learns steerable language models that outperform and Pareto-dominate the existing approaches for multi-objective finetuning.

Related Topics

Computer Science

Concepts

Computer science

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2407.15762
PDF: https://arxiv.org/pdf/2407.15762
OA Status: green
Related Works: 10
OpenAlex ID: https://openalex.org/W4402853841

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4402853841

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2407.15762

Digital Object Identifier
Title: Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuning

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2024

Year of publication
Publication date: 2024-07-22

Full publication date if available
Authors: Kaiwen Wang, Rahul Kidambi, Ryan Sullivan, A. Agarwal, Christoph Dann, Andrea Michi, Marco Gelmi, Yunxuan Li, Raghav Gupta, Avinava Dubey, Alexandre Ramé, Johan Ferret, Geoffrey Cideron, Le Hou, Hongkun Yu, Amr Ahmed, Aranyak Mehta, Léonard Hussenot, Olivier Bachem, Edouard Leurent

List of authors in order
Landing page: https://arxiv.org/abs/2407.15762

Publisher landing page
PDF URL: https://arxiv.org/pdf/2407.15762

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2407.15762

Direct OA link when available
Concepts: Computer science

Top concepts (fields/topics) attached by OpenAlex
Cited by: 0

Total citation count in OpenAlex
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4402853841
doi	https://doi.org/10.48550/arxiv.2407.15762
ids.doi	https://doi.org/10.48550/arxiv.2407.15762
ids.openalex	https://openalex.org/W4402853841
fwci
type	preprint
title	Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuning
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T12373
topics[0].field.id	https://openalex.org/fields/12
topics[0].field.display_name	Arts and Humanities
topics[0].score	0.0934000015258789
topics[0].domain.id	https://openalex.org/domains/2
topics[0].domain.display_name	Social Sciences
topics[0].subfield.id	https://openalex.org/subfields/1203
topics[0].subfield.display_name	Language and Linguistics
topics[0].display_name	Linguistic research and analysis
topics[1].id	https://openalex.org/T10034
topics[1].field.id	https://openalex.org/fields/12
topics[1].field.display_name	Arts and Humanities
topics[1].score	0.08940000087022781
topics[1].domain.id	https://openalex.org/domains/2
topics[1].domain.display_name	Social Sciences
topics[1].subfield.id	https://openalex.org/subfields/1203
topics[1].subfield.display_name	Language and Linguistics
topics[1].display_name	Syntax, Semantics, Linguistic Variation
topics[2].id	https://openalex.org/T10181
topics[2].field.id	https://openalex.org/fields/17
topics[2].field.display_name	Computer Science
topics[2].score	0.08330000191926956
topics[2].domain.id	https://openalex.org/domains/3
topics[2].domain.display_name	Physical Sciences
topics[2].subfield.id	https://openalex.org/subfields/1702
topics[2].subfield.display_name	Artificial Intelligence
topics[2].display_name	Natural Language Processing Techniques
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C41008148
concepts[0].level	0
concepts[0].score	0.513916015625
concepts[0].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[0].display_name	Computer science
keywords[0].id	https://openalex.org/keywords/computer-science
keywords[0].score	0.513916015625
keywords[0].display_name	Computer science
language	en
locations[0].id	pmh:oai:arXiv.org:2407.15762
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license	cc-by-nc-sa
locations[0].pdf_url	https://arxiv.org/pdf/2407.15762
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id	https://openalex.org/licenses/cc-by-nc-sa
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2407.15762
locations[1].id	doi:10.48550/arxiv.2407.15762
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2407.15762
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5100722760
authorships[0].author.orcid	https://orcid.org/0000-0003-3857-4790
authorships[0].author.display_name	Kaiwen Wang
authorships[0].author_position	first
authorships[0].raw_author_name	Wang, Kaiwen
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5057958491
authorships[1].author.orcid
authorships[1].author.display_name	Rahul Kidambi
authorships[1].author_position	middle
authorships[1].raw_author_name	Kidambi, Rahul
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5107580321
authorships[2].author.orcid
authorships[2].author.display_name	Ryan Sullivan
authorships[2].author_position	middle
authorships[2].raw_author_name	Sullivan, Ryan
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5101783911
authorships[3].author.orcid	https://orcid.org/0009-0005-4916-1045
authorships[3].author.display_name	A. Agarwal
authorships[3].author_position	middle
authorships[3].raw_author_name	Agarwal, Alekh
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5015966835
authorships[4].author.orcid
authorships[4].author.display_name	Christoph Dann
authorships[4].author_position	middle
authorships[4].raw_author_name	Dann, Christoph
authorships[4].is_corresponding	False
authorships[5].author.id	https://openalex.org/A5046831695
authorships[5].author.orcid	https://orcid.org/0009-0001-4797-3593
authorships[5].author.display_name	Andrea Michi
authorships[5].author_position	middle
authorships[5].raw_author_name	Michi, Andrea
authorships[5].is_corresponding	False
authorships[6].author.id	https://openalex.org/A5104224050
authorships[6].author.orcid
authorships[6].author.display_name	Marco Gelmi
authorships[6].author_position	middle
authorships[6].raw_author_name	Gelmi, Marco
authorships[6].is_corresponding	False
authorships[7].author.id	https://openalex.org/A5107646464
authorships[7].author.orcid
authorships[7].author.display_name	Yunxuan Li
authorships[7].author_position	middle
authorships[7].raw_author_name	Li, Yunxuan
authorships[7].is_corresponding	False
authorships[8].author.id	https://openalex.org/A5018229381
authorships[8].author.orcid	https://orcid.org/0000-0002-9546-2632
authorships[8].author.display_name	Raghav Gupta
authorships[8].author_position	middle
authorships[8].raw_author_name	Gupta, Raghav
authorships[8].is_corresponding	False
authorships[9].author.id	https://openalex.org/A5036039474
authorships[9].author.orcid
authorships[9].author.display_name	Avinava Dubey
authorships[9].author_position	middle
authorships[9].raw_author_name	Dubey, Avinava
authorships[9].is_corresponding	False
authorships[10].author.id	https://openalex.org/A5057217987
authorships[10].author.orcid
authorships[10].author.display_name	Alexandre Ramé
authorships[10].author_position	middle
authorships[10].raw_author_name	Ramé, Alexandre
authorships[10].is_corresponding	False
authorships[11].author.id	https://openalex.org/A5087706654
authorships[11].author.orcid
authorships[11].author.display_name	Johan Ferret
authorships[11].author_position	middle
authorships[11].raw_author_name	Ferret, Johan
authorships[11].is_corresponding	False
authorships[12].author.id	https://openalex.org/A5028486593
authorships[12].author.orcid
authorships[12].author.display_name	Geoffrey Cideron
authorships[12].author_position	middle
authorships[12].raw_author_name	Cideron, Geoffrey
authorships[12].is_corresponding	False
authorships[13].author.id	https://openalex.org/A5101561297
authorships[13].author.orcid
authorships[13].author.display_name	Le Hou
authorships[13].author_position	middle
authorships[13].raw_author_name	Hou, Le
authorships[13].is_corresponding	False
authorships[14].author.id	https://openalex.org/A5100537890
authorships[14].author.orcid
authorships[14].author.display_name	Hongkun Yu
authorships[14].author_position	middle
authorships[14].raw_author_name	Yu, Hongkun
authorships[14].is_corresponding	False
authorships[15].author.id	https://openalex.org/A5009154893
authorships[15].author.orcid	https://orcid.org/0000-0002-7749-7911
authorships[15].author.display_name	Amr Ahmed
authorships[15].author_position	middle
authorships[15].raw_author_name	Ahmed, Amr
authorships[15].is_corresponding	False
authorships[16].author.id	https://openalex.org/A5075894805
authorships[16].author.orcid	https://orcid.org/0000-0002-6132-4901
authorships[16].author.display_name	Aranyak Mehta
authorships[16].author_position	middle
authorships[16].raw_author_name	Mehta, Aranyak
authorships[16].is_corresponding	False
authorships[17].author.id	https://openalex.org/A5079135104
authorships[17].author.orcid
authorships[17].author.display_name	Léonard Hussenot
authorships[17].author_position	middle
authorships[17].raw_author_name	Hussenot, Léonard
authorships[17].is_corresponding	False
authorships[18].author.id	https://openalex.org/A5089890773
authorships[18].author.orcid
authorships[18].author.display_name	Olivier Bachem
authorships[18].author_position	middle
authorships[18].raw_author_name	Bachem, Olivier
authorships[18].is_corresponding	False
authorships[19].author.id	https://openalex.org/A5107646463
authorships[19].author.orcid
authorships[19].author.display_name	Edouard Leurent
authorships[19].author_position	last
authorships[19].raw_author_name	Leurent, Edouard
authorships[19].is_corresponding	False
has_content.pdf	True
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2407.15762
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2024-09-26T00:00:00
display_name	Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuning
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T12373
primary_topic.field.id	https://openalex.org/fields/12
primary_topic.field.display_name	Arts and Humanities
primary_topic.score	0.0934000015258789
primary_topic.domain.id	https://openalex.org/domains/2
primary_topic.domain.display_name	Social Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1203
primary_topic.subfield.display_name	Language and Linguistics
primary_topic.display_name	Linguistic research and analysis
related_works	https://openalex.org/W4391375266, https://openalex.org/W2748952813, https://openalex.org/W2390279801, https://openalex.org/W2358668433, https://openalex.org/W4396701345, https://openalex.org/W2376932109, https://openalex.org/W2001405890, https://openalex.org/W4396696052, https://openalex.org/W2382290278, https://openalex.org/W4395014643
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2407.15762
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license	cc-by-nc-sa
best_oa_location.pdf_url	https://arxiv.org/pdf/2407.15762
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id	https://openalex.org/licenses/cc-by-nc-sa
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2407.15762
primary_location.id	pmh:oai:arXiv.org:2407.15762
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license	cc-by-nc-sa
primary_location.pdf_url	https://arxiv.org/pdf/2407.15762
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id	https://openalex.org/licenses/cc-by-nc-sa
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2407.15762
publication_date	2024-07-22
publication_year	2024
referenced_works_count	0
abstract_inverted_index.A	15
abstract_inverted_index.a	30, 42
abstract_inverted_index.at	70
abstract_inverted_index.in	29
abstract_inverted_index.is	2, 18
abstract_inverted_index.on	49, 53, 95
abstract_inverted_index.or	79
abstract_inverted_index.to	19, 83
abstract_inverted_index.we	99
abstract_inverted_index.CLP	61, 102
abstract_inverted_index.and	13, 32, 58, 93, 109
abstract_inverted_index.for	4, 45, 114
abstract_inverted_index.key	16
abstract_inverted_index.not	76
abstract_inverted_index.the	88, 111
abstract_inverted_index.two	96
abstract_inverted_index.This	35
abstract_inverted_index.does	75
abstract_inverted_index.from	55
abstract_inverted_index.show	100
abstract_inverted_index.that	24, 65, 101, 107
abstract_inverted_index.this	74
abstract_inverted_index.with	8
abstract_inverted_index.learn	62
abstract_inverted_index.paper	36
abstract_inverted_index.time.	72
abstract_inverted_index.(CLP),	41
abstract_inverted_index.(e.g.,	11
abstract_inverted_index.Policy	40
abstract_inverted_index.learns	103
abstract_inverted_index.models	23, 48, 64, 82, 106
abstract_inverted_index.Through	90
abstract_inverted_index.achieve	84
abstract_inverted_index.between	87
abstract_inverted_index.crucial	3
abstract_inverted_index.develop	20
abstract_inverted_index.general	43
abstract_inverted_index.manner.	34
abstract_inverted_index.require	77
abstract_inverted_index.Building	52
abstract_inverted_index.Language	39
abstract_inverted_index.Notably,	73
abstract_inverted_index.aligning	5
abstract_inverted_index.existing	112
abstract_inverted_index.flexible	31
abstract_inverted_index.intended	9
abstract_inverted_index.language	6, 22, 47, 105
abstract_inverted_index.multiple	26, 50, 81
abstract_inverted_index.policies	7
abstract_inverted_index.presents	37
abstract_inverted_index.safety).	14
abstract_inverted_index.training	57, 78
abstract_inverted_index.ablations	94
abstract_inverted_index.behaviors	10
abstract_inverted_index.challenge	17
abstract_inverted_index.datasets,	98
abstract_inverted_index.different	85
abstract_inverted_index.efficient	33
abstract_inverted_index.extensive	91
abstract_inverted_index.framework	44
abstract_inverted_index.inference	71
abstract_inverted_index.steerable	21, 63, 104
abstract_inverted_index.trade-off	25, 67
abstract_inverted_index.approaches	113
abstract_inverted_index.creativity	12
abstract_inverted_index.finetuning	1, 46
abstract_inverted_index.multi-task	56
abstract_inverted_index.objectives	28, 69
abstract_inverted_index.outperform	108
abstract_inverted_index.techniques	54
abstract_inverted_index.trade-offs	86
abstract_inverted_index.Conditional	38
abstract_inverted_index.conflicting	68
abstract_inverted_index.effectively	66
abstract_inverted_index.experiments	92
abstract_inverted_index.finetuning,	60
abstract_inverted_index.finetuning.	116
abstract_inverted_index.maintaining	80
abstract_inverted_index.objectives.	51, 89
abstract_inverted_index.Reward-based	0
abstract_inverted_index.(conflicting)	27
abstract_inverted_index.summarization	97
abstract_inverted_index.Pareto-dominate	110
abstract_inverted_index.multi-objective	115
abstract_inverted_index.parameter-efficient	59
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	20
citation_normalized_percentile