Privately Aligning Language Models with Reinforcement Learning Article Swipe
Fan Wu
,
Huseyin A. Inan
,
Artūrs Bačkurs
,
Varun Chandrasekaran
,
Janardhan Kulkarni
,
Robert B. Sim
·
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2310.16960
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2310.16960
Positioned between pre-training and user deployment, aligning large language models (LLMs) through reinforcement learning (RL) has emerged as a prevailing strategy for training instruction following-models such as ChatGPT. In this work, we initiate the study of privacy-preserving alignment of LLMs through Differential Privacy (DP) in conjunction with RL. Following the influential work of Ziegler et al. (2020), we study two dominant paradigms: (i) alignment via RL without human in the loop (e.g., positive review generation) and (ii) alignment via RL from human feedback (RLHF) (e.g., summarization in a human-preferred way). We give a new DP framework to achieve alignment via RL, and prove its correctness. Our experimental results validate the effectiveness of our approach, offering competitive utility while ensuring strong privacy protections.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2310.16960
- https://arxiv.org/pdf/2310.16960
- OA Status
- green
- Cited By
- 1
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4387994968
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4387994968Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2310.16960Digital Object Identifier
- Title
-
Privately Aligning Language Models with Reinforcement LearningWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2023Year of publication
- Publication date
-
2023-10-25Full publication date if available
- Authors
-
Fan Wu, Huseyin A. Inan, Artūrs Bačkurs, Varun Chandrasekaran, Janardhan Kulkarni, Robert B. SimList of authors in order
- Landing page
-
https://arxiv.org/abs/2310.16960Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2310.16960Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2310.16960Direct OA link when available
- Concepts
-
Reinforcement learning, Automatic summarization, Computer science, Correctness, Software deployment, Artificial intelligence, Human–computer interaction, Software engineering, Programming languageTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
1Total citation count in OpenAlex
- Citations by year (recent)
-
2024: 1Per-year citation counts (last 5 years)
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4387994968 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2310.16960 |
| ids.doi | https://doi.org/10.48550/arxiv.2310.16960 |
| ids.openalex | https://openalex.org/W4387994968 |
| fwci | |
| type | preprint |
| title | Privately Aligning Language Models with Reinforcement Learning |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10764 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9843999743461609 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Privacy-Preserving Technologies in Data |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C97541855 |
| concepts[0].level | 2 |
| concepts[0].score | 0.8460264205932617 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q830687 |
| concepts[0].display_name | Reinforcement learning |
| concepts[1].id | https://openalex.org/C170858558 |
| concepts[1].level | 2 |
| concepts[1].score | 0.7881953120231628 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q1394144 |
| concepts[1].display_name | Automatic summarization |
| concepts[2].id | https://openalex.org/C41008148 |
| concepts[2].level | 0 |
| concepts[2].score | 0.6814044713973999 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[2].display_name | Computer science |
| concepts[3].id | https://openalex.org/C55439883 |
| concepts[3].level | 2 |
| concepts[3].score | 0.622978687286377 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q360812 |
| concepts[3].display_name | Correctness |
| concepts[4].id | https://openalex.org/C105339364 |
| concepts[4].level | 2 |
| concepts[4].score | 0.5541257858276367 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q2297740 |
| concepts[4].display_name | Software deployment |
| concepts[5].id | https://openalex.org/C154945302 |
| concepts[5].level | 1 |
| concepts[5].score | 0.4063907265663147 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[5].display_name | Artificial intelligence |
| concepts[6].id | https://openalex.org/C107457646 |
| concepts[6].level | 1 |
| concepts[6].score | 0.33955127000808716 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q207434 |
| concepts[6].display_name | Human–computer interaction |
| concepts[7].id | https://openalex.org/C115903868 |
| concepts[7].level | 1 |
| concepts[7].score | 0.24155297875404358 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q80993 |
| concepts[7].display_name | Software engineering |
| concepts[8].id | https://openalex.org/C199360897 |
| concepts[8].level | 1 |
| concepts[8].score | 0.22078382968902588 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q9143 |
| concepts[8].display_name | Programming language |
| keywords[0].id | https://openalex.org/keywords/reinforcement-learning |
| keywords[0].score | 0.8460264205932617 |
| keywords[0].display_name | Reinforcement learning |
| keywords[1].id | https://openalex.org/keywords/automatic-summarization |
| keywords[1].score | 0.7881953120231628 |
| keywords[1].display_name | Automatic summarization |
| keywords[2].id | https://openalex.org/keywords/computer-science |
| keywords[2].score | 0.6814044713973999 |
| keywords[2].display_name | Computer science |
| keywords[3].id | https://openalex.org/keywords/correctness |
| keywords[3].score | 0.622978687286377 |
| keywords[3].display_name | Correctness |
| keywords[4].id | https://openalex.org/keywords/software-deployment |
| keywords[4].score | 0.5541257858276367 |
| keywords[4].display_name | Software deployment |
| keywords[5].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[5].score | 0.4063907265663147 |
| keywords[5].display_name | Artificial intelligence |
| keywords[6].id | https://openalex.org/keywords/human–computer-interaction |
| keywords[6].score | 0.33955127000808716 |
| keywords[6].display_name | Human–computer interaction |
| keywords[7].id | https://openalex.org/keywords/software-engineering |
| keywords[7].score | 0.24155297875404358 |
| keywords[7].display_name | Software engineering |
| keywords[8].id | https://openalex.org/keywords/programming-language |
| keywords[8].score | 0.22078382968902588 |
| keywords[8].display_name | Programming language |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2310.16960 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2310.16960 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2310.16960 |
| locations[1].id | doi:10.48550/arxiv.2310.16960 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2310.16960 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5100380478 |
| authorships[0].author.orcid | https://orcid.org/0000-0001-5051-3005 |
| authorships[0].author.display_name | Fan Wu |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Wu, Fan |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5071827392 |
| authorships[1].author.orcid | https://orcid.org/0000-0001-7465-7214 |
| authorships[1].author.display_name | Huseyin A. Inan |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Inan, Huseyin A. |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5084855970 |
| authorships[2].author.orcid | |
| authorships[2].author.display_name | Artūrs Bačkurs |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Backurs, Arturs |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5009092703 |
| authorships[3].author.orcid | https://orcid.org/0000-0003-0661-6378 |
| authorships[3].author.display_name | Varun Chandrasekaran |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Chandrasekaran, Varun |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5043137436 |
| authorships[4].author.orcid | https://orcid.org/0009-0005-9980-7836 |
| authorships[4].author.display_name | Janardhan Kulkarni |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Kulkarni, Janardhan |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5090308773 |
| authorships[5].author.orcid | https://orcid.org/0000-0002-2855-7455 |
| authorships[5].author.display_name | Robert B. Sim |
| authorships[5].author_position | last |
| authorships[5].raw_author_name | Sim, Robert |
| authorships[5].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2310.16960 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Privately Aligning Language Models with Reinforcement Learning |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10764 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9843999743461609 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Privacy-Preserving Technologies in Data |
| related_works | https://openalex.org/W2366403280, https://openalex.org/W1495108544, https://openalex.org/W2091301346, https://openalex.org/W3148229873, https://openalex.org/W2150160875, https://openalex.org/W4242223894, https://openalex.org/W1517524280, https://openalex.org/W4323520239, https://openalex.org/W4389760904, https://openalex.org/W4306886878 |
| cited_by_count | 1 |
| counts_by_year[0].year | 2024 |
| counts_by_year[0].cited_by_count | 1 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2310.16960 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2310.16960 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2310.16960 |
| primary_location.id | pmh:oai:arXiv.org:2310.16960 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2310.16960 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2310.16960 |
| publication_date | 2023-10-25 |
| publication_year | 2023 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 18, 87, 92 |
| abstract_inverted_index.DP | 94 |
| abstract_inverted_index.In | 28 |
| abstract_inverted_index.RL | 65, 79 |
| abstract_inverted_index.We | 90 |
| abstract_inverted_index.as | 17, 26 |
| abstract_inverted_index.et | 54 |
| abstract_inverted_index.in | 44, 68, 86 |
| abstract_inverted_index.of | 35, 38, 52, 111 |
| abstract_inverted_index.to | 96 |
| abstract_inverted_index.we | 31, 57 |
| abstract_inverted_index.(i) | 62 |
| abstract_inverted_index.Our | 105 |
| abstract_inverted_index.RL, | 100 |
| abstract_inverted_index.RL. | 47 |
| abstract_inverted_index.al. | 55 |
| abstract_inverted_index.and | 3, 75, 101 |
| abstract_inverted_index.for | 21 |
| abstract_inverted_index.has | 15 |
| abstract_inverted_index.its | 103 |
| abstract_inverted_index.new | 93 |
| abstract_inverted_index.our | 112 |
| abstract_inverted_index.the | 33, 49, 69, 109 |
| abstract_inverted_index.two | 59 |
| abstract_inverted_index.via | 64, 78, 99 |
| abstract_inverted_index.(DP) | 43 |
| abstract_inverted_index.(RL) | 14 |
| abstract_inverted_index.(ii) | 76 |
| abstract_inverted_index.LLMs | 39 |
| abstract_inverted_index.from | 80 |
| abstract_inverted_index.give | 91 |
| abstract_inverted_index.loop | 70 |
| abstract_inverted_index.such | 25 |
| abstract_inverted_index.this | 29 |
| abstract_inverted_index.user | 4 |
| abstract_inverted_index.with | 46 |
| abstract_inverted_index.work | 51 |
| abstract_inverted_index.human | 67, 81 |
| abstract_inverted_index.large | 7 |
| abstract_inverted_index.prove | 102 |
| abstract_inverted_index.study | 34, 58 |
| abstract_inverted_index.way). | 89 |
| abstract_inverted_index.while | 117 |
| abstract_inverted_index.work, | 30 |
| abstract_inverted_index.(LLMs) | 10 |
| abstract_inverted_index.(RLHF) | 83 |
| abstract_inverted_index.(e.g., | 71, 84 |
| abstract_inverted_index.models | 9 |
| abstract_inverted_index.review | 73 |
| abstract_inverted_index.strong | 119 |
| abstract_inverted_index.(2020), | 56 |
| abstract_inverted_index.Privacy | 42 |
| abstract_inverted_index.Ziegler | 53 |
| abstract_inverted_index.achieve | 97 |
| abstract_inverted_index.between | 1 |
| abstract_inverted_index.emerged | 16 |
| abstract_inverted_index.privacy | 120 |
| abstract_inverted_index.results | 107 |
| abstract_inverted_index.through | 11, 40 |
| abstract_inverted_index.utility | 116 |
| abstract_inverted_index.without | 66 |
| abstract_inverted_index.ChatGPT. | 27 |
| abstract_inverted_index.aligning | 6 |
| abstract_inverted_index.dominant | 60 |
| abstract_inverted_index.ensuring | 118 |
| abstract_inverted_index.feedback | 82 |
| abstract_inverted_index.initiate | 32 |
| abstract_inverted_index.language | 8 |
| abstract_inverted_index.learning | 13 |
| abstract_inverted_index.offering | 114 |
| abstract_inverted_index.positive | 72 |
| abstract_inverted_index.strategy | 20 |
| abstract_inverted_index.training | 22 |
| abstract_inverted_index.validate | 108 |
| abstract_inverted_index.Following | 48 |
| abstract_inverted_index.alignment | 37, 63, 77, 98 |
| abstract_inverted_index.approach, | 113 |
| abstract_inverted_index.framework | 95 |
| abstract_inverted_index.Positioned | 0 |
| abstract_inverted_index.paradigms: | 61 |
| abstract_inverted_index.prevailing | 19 |
| abstract_inverted_index.competitive | 115 |
| abstract_inverted_index.conjunction | 45 |
| abstract_inverted_index.deployment, | 5 |
| abstract_inverted_index.generation) | 74 |
| abstract_inverted_index.influential | 50 |
| abstract_inverted_index.instruction | 23 |
| abstract_inverted_index.Differential | 41 |
| abstract_inverted_index.correctness. | 104 |
| abstract_inverted_index.experimental | 106 |
| abstract_inverted_index.pre-training | 2 |
| abstract_inverted_index.protections. | 121 |
| abstract_inverted_index.effectiveness | 110 |
| abstract_inverted_index.reinforcement | 12 |
| abstract_inverted_index.summarization | 85 |
| abstract_inverted_index.human-preferred | 88 |
| abstract_inverted_index.following-models | 24 |
| abstract_inverted_index.privacy-preserving | 36 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 6 |
| citation_normalized_percentile |