Token-level Accept or Reject: A Micro Alignment Approach for Large Language Models Article Swipe

PDF

Yang Zhang , Yu Yu , Bo Tang , Limin Zhu , Chuxiong Sun , Wenqiang Wei , Jie Hu , Zheng Xie , Zhiyu Li , Feiyu Xiong , Edward Chung ·

YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2505.19743

With the rapid development of Large Language Models (LLMs), aligning these models with human preferences and values is critical to ensuring ethical and safe applications. However, existing alignment techniques such as RLHF or DPO often require direct fine-tuning on LLMs with billions of parameters, resulting in substantial computational costs and inefficiencies. To address this, we propose Micro token-level Accept-Reject Aligning (MARA) approach designed to operate independently of the language models. MARA simplifies the alignment process by decomposing sentence-level preference learning into token-level binary classification, where a compact three-layer fully-connected network determines whether candidate tokens are "Accepted" or "Rejected" as part of the response. Extensive experiments across seven different LLMs and three open-source datasets show that MARA achieves significant improvements in alignment performance while reducing computational costs. The source code and implementation details are publicly available at https://github.com/IAAR-Shanghai/MARA, and the trained models are released at https://huggingface.co/IAAR-Shanghai/MARA_AGENTS.

Related Topics

Truth And Reconciliation Commission Of Canada

2025 Nba Draft

28 Years Later

Reich Ministry Of Public Enlightenment And Propaganda

Concepts

No concepts available.

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2505.19743
PDF: https://arxiv.org/pdf/2505.19743
OA Status: green
OpenAlex ID: https://openalex.org/W4415035553

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4415035553

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2505.19743

Digital Object Identifier
Title: Token-level Accept or Reject: A Micro Alignment Approach for Large Language Models

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2025

Year of publication
Publication date: 2025-05-26

Full publication date if available
Authors: Yang Zhang, Yu Yu, Bo Tang, Limin Zhu, Chuxiong Sun, Wenqiang Wei, Jie Hu, Zheng Xie, Zhiyu Li, Feiyu Xiong, Edward Chung

List of authors in order
Landing page: https://arxiv.org/abs/2505.19743

Publisher landing page
PDF URL: https://arxiv.org/pdf/2505.19743

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2505.19743

Direct OA link when available
Cited by: 0

Total citation count in OpenAlex

Full payload

id	https://openalex.org/W4415035553
doi	https://doi.org/10.48550/arxiv.2505.19743
ids.doi	https://doi.org/10.48550/arxiv.2505.19743
ids.openalex	https://openalex.org/W4415035553
fwci
type	preprint
title	Token-level Accept or Reject: A Micro Alignment Approach for Large Language Models
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T10028
topics[0].field.id	https://openalex.org/fields/17
topics[0].field.display_name	Computer Science
topics[0].score	0.9634000062942505
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/1702
topics[0].subfield.display_name	Artificial Intelligence
topics[0].display_name	Topic Modeling
topics[1].id	https://openalex.org/T10181
topics[1].field.id	https://openalex.org/fields/17
topics[1].field.display_name	Computer Science
topics[1].score	0.9610999822616577
topics[1].domain.id	https://openalex.org/domains/3
topics[1].domain.display_name	Physical Sciences
topics[1].subfield.id	https://openalex.org/subfields/1702
topics[1].subfield.display_name	Artificial Intelligence
topics[1].display_name	Natural Language Processing Techniques
is_xpac	False
apc_list
apc_paid
language	en
locations[0].id	pmh:oai:arXiv.org:2505.19743
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2505.19743
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2505.19743
locations[1].id	doi:10.48550/arxiv.2505.19743
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2505.19743
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5100354819
authorships[0].author.orcid	https://orcid.org/0000-0003-4658-565X
authorships[0].author.display_name	Yang Zhang
authorships[0].author_position	first
authorships[0].raw_author_name	Zhang, Yang
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5110299446
authorships[1].author.orcid
authorships[1].author.display_name	Yu Yu
authorships[1].author_position	middle
authorships[1].raw_author_name	Yu, Yu
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5057816960
authorships[2].author.orcid	https://orcid.org/0000-0001-7129-0250
authorships[2].author.display_name	Bo Tang
authorships[2].author_position	middle
authorships[2].raw_author_name	Tang, Bo
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5103205239
authorships[3].author.orcid	https://orcid.org/0000-0003-3194-6731
authorships[3].author.display_name	Limin Zhu
authorships[3].author_position	middle
authorships[3].raw_author_name	Zhu, Yu
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5057734833
authorships[4].author.orcid
authorships[4].author.display_name	Chuxiong Sun
authorships[4].author_position	middle
authorships[4].raw_author_name	Sun, Chuxiong
authorships[4].is_corresponding	False
authorships[5].author.id	https://openalex.org/A5052103501
authorships[5].author.orcid	https://orcid.org/0000-0003-2078-9056
authorships[5].author.display_name	Wenqiang Wei
authorships[5].author_position	middle
authorships[5].raw_author_name	Wei, Wenqiang
authorships[5].is_corresponding	False
authorships[6].author.id	https://openalex.org/A5066864248
authorships[6].author.orcid
authorships[6].author.display_name	Jie Hu
authorships[6].author_position	middle
authorships[6].raw_author_name	Hu, Jie
authorships[6].is_corresponding	False
authorships[7].author.id	https://openalex.org/A5044995870
authorships[7].author.orcid	https://orcid.org/0000-0001-8260-3259
authorships[7].author.display_name	Zheng Xie
authorships[7].author_position	middle
authorships[7].raw_author_name	Xie, Zipeng
authorships[7].is_corresponding	False
authorships[8].author.id	https://openalex.org/A5113013369
authorships[8].author.orcid	https://orcid.org/0009-0002-8787-8192
authorships[8].author.display_name	Zhiyu Li
authorships[8].author_position	middle
authorships[8].raw_author_name	Li, Zhiyu
authorships[8].is_corresponding	False
authorships[9].author.id	https://openalex.org/A5090294171
authorships[9].author.orcid	https://orcid.org/0000-0002-1456-2202
authorships[9].author.display_name	Feiyu Xiong
authorships[9].author_position	middle
authorships[9].raw_author_name	Xiong, Feiyu
authorships[9].is_corresponding	False
authorships[10].author.id	https://openalex.org/A5044341587
authorships[10].author.orcid	https://orcid.org/0000-0001-6969-7764
authorships[10].author.display_name	Edward Chung
authorships[10].author_position	last
authorships[10].raw_author_name	Chung, Edward
authorships[10].is_corresponding	False
has_content.pdf	True
has_content.grobid_xml	True
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2505.19743
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	Token-level Accept or Reject: A Micro Alignment Approach for Large Language Models
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T10028
primary_topic.field.id	https://openalex.org/fields/17
primary_topic.field.display_name	Computer Science
primary_topic.score	0.9634000062942505
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1702
primary_topic.subfield.display_name	Artificial Intelligence
primary_topic.display_name	Topic Modeling
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2505.19743
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2505.19743
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2505.19743
primary_location.id	pmh:oai:arXiv.org:2505.19743
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2505.19743
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2505.19743
publication_date	2025-05-26
publication_year	2025
referenced_works_count	0
abstract_inverted_index.a	85
abstract_inverted_index.To	51
abstract_inverted_index.as	30, 98
abstract_inverted_index.at	135, 143
abstract_inverted_index.by	75
abstract_inverted_index.in	45, 119
abstract_inverted_index.is	17
abstract_inverted_index.of	4, 42, 66, 100
abstract_inverted_index.on	38
abstract_inverted_index.or	32, 96
abstract_inverted_index.to	19, 63
abstract_inverted_index.we	54
abstract_inverted_index.DPO	33
abstract_inverted_index.The	126
abstract_inverted_index.and	15, 22, 49, 109, 129, 137
abstract_inverted_index.are	94, 132, 141
abstract_inverted_index.the	1, 67, 72, 101, 138
abstract_inverted_index.LLMs	39, 108
abstract_inverted_index.MARA	70, 115
abstract_inverted_index.RLHF	31
abstract_inverted_index.With	0
abstract_inverted_index.code	128
abstract_inverted_index.into	80
abstract_inverted_index.part	99
abstract_inverted_index.safe	23
abstract_inverted_index.show	113
abstract_inverted_index.such	29
abstract_inverted_index.that	114
abstract_inverted_index.with	12, 40
abstract_inverted_index.Large	5
abstract_inverted_index.Micro	56
abstract_inverted_index.costs	48
abstract_inverted_index.human	13
abstract_inverted_index.often	34
abstract_inverted_index.rapid	2
abstract_inverted_index.seven	106
abstract_inverted_index.these	10
abstract_inverted_index.this,	53
abstract_inverted_index.three	110
abstract_inverted_index.where	84
abstract_inverted_index.while	122
abstract_inverted_index.(MARA)	60
abstract_inverted_index.Models	7
abstract_inverted_index.across	105
abstract_inverted_index.binary	82
abstract_inverted_index.costs.	125
abstract_inverted_index.direct	36
abstract_inverted_index.models	11, 140
abstract_inverted_index.source	127
abstract_inverted_index.tokens	93
abstract_inverted_index.values	16
abstract_inverted_index.(LLMs),	8
abstract_inverted_index.address	52
abstract_inverted_index.compact	86
abstract_inverted_index.details	131
abstract_inverted_index.ethical	21
abstract_inverted_index.models.	69
abstract_inverted_index.network	89
abstract_inverted_index.operate	64
abstract_inverted_index.process	74
abstract_inverted_index.propose	55
abstract_inverted_index.require	35
abstract_inverted_index.trained	139
abstract_inverted_index.whether	91
abstract_inverted_index.Aligning	59
abstract_inverted_index.However,	25
abstract_inverted_index.Language	6
abstract_inverted_index.achieves	116
abstract_inverted_index.aligning	9
abstract_inverted_index.approach	61
abstract_inverted_index.billions	41
abstract_inverted_index.critical	18
abstract_inverted_index.datasets	112
abstract_inverted_index.designed	62
abstract_inverted_index.ensuring	20
abstract_inverted_index.existing	26
abstract_inverted_index.language	68
abstract_inverted_index.learning	79
abstract_inverted_index.publicly	133
abstract_inverted_index.reducing	123
abstract_inverted_index.released	142
abstract_inverted_index.Extensive	103
abstract_inverted_index.alignment	27, 73, 120
abstract_inverted_index.available	134
abstract_inverted_index.candidate	92
abstract_inverted_index.different	107
abstract_inverted_index.response.	102
abstract_inverted_index.resulting	44
abstract_inverted_index."Accepted"	95
abstract_inverted_index."Rejected"	97
abstract_inverted_index.determines	90
abstract_inverted_index.preference	78
abstract_inverted_index.simplifies	71
abstract_inverted_index.techniques	28
abstract_inverted_index.decomposing	76
abstract_inverted_index.development	3
abstract_inverted_index.experiments	104
abstract_inverted_index.fine-tuning	37
abstract_inverted_index.open-source	111
abstract_inverted_index.parameters,	43
abstract_inverted_index.performance	121
abstract_inverted_index.preferences	14
abstract_inverted_index.significant	117
abstract_inverted_index.substantial	46
abstract_inverted_index.three-layer	87
abstract_inverted_index.token-level	57, 81
abstract_inverted_index.improvements	118
abstract_inverted_index.Accept-Reject	58
abstract_inverted_index.applications.	24
abstract_inverted_index.computational	47, 124
abstract_inverted_index.independently	65
abstract_inverted_index.implementation	130
abstract_inverted_index.sentence-level	77
abstract_inverted_index.classification,	83
abstract_inverted_index.fully-connected	88
abstract_inverted_index.inefficiencies.	50
abstract_inverted_index.https://github.com/IAAR-Shanghai/MARA,	136
abstract_inverted_index.https://huggingface.co/IAAR-Shanghai/MARA_AGENTS.	144
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	11
citation_normalized_percentile