Reasoning with Reinforced Functional Token Tuning Article Swipe

PDF

K. Zhang , Qi Yao , Baisheng Lai , Jiaxing Huang , Wenkai Fang , Dacheng Tao , Mingli Song , Shunyu Liu ·

YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2502.13389

In this work, we propose Reinforced Functional Token Tuning (RFTT), a novel reinforced fine-tuning framework that empowers Large Language Models (LLMs) with self-play learn-to-reason capabilities. Unlike prior prompt-driven reasoning efforts, RFTT embeds a rich set of learnable functional tokens (e.g., , , ) directly into the model vocabulary, enabling chain-of-thought construction with diverse human-like reasoning behaviors. Specifically, RFTT comprises two phases: (1) supervised fine-tuning performs prompt-driven tree search to obtain self-generated training data annotated with functional tokens, which warms up the model to learn these tokens for reasoning; and (2) online reinforcement learning further allows the model to explore different reasoning pathways through functional token sampling without relying on prompts, thereby facilitating effective self-improvement for functional reasoning. Extensive experiments demonstrate the superiority of the proposed RFTT on mathematical benchmarks, significantly boosting Qwen-2.5-7B-Instruct (70.6% to 79.8%) and LLaMA-3.1-8B-Instruct (32.2% to 60.2%) on the MATH dataset. Moreover, the performance of RFTT consistently improves with more search rollouts at inference time. Our code is available at https://github.com/sastpg/RFTT.

Related Topics

Security Token

Computer Science

Computer Security

Concepts

Security token Computer science Computer security

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2502.13389
PDF: https://arxiv.org/pdf/2502.13389
OA Status: green
Related Works: 10
OpenAlex ID: https://openalex.org/W4407764166

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4407764166

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2502.13389

Digital Object Identifier
Title: Reasoning with Reinforced Functional Token Tuning

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2025

Year of publication
Publication date: 2025-02-19

Full publication date if available
Authors: K. Zhang, Qi Yao, Baisheng Lai, Jiaxing Huang, Wenkai Fang, Dacheng Tao, Mingli Song, Shunyu Liu

List of authors in order
Landing page: https://arxiv.org/abs/2502.13389

Publisher landing page
PDF URL: https://arxiv.org/pdf/2502.13389

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2502.13389

Direct OA link when available
Concepts: Security token, Computer science, Computer security

Top concepts (fields/topics) attached by OpenAlex
Cited by: 0

Total citation count in OpenAlex
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4407764166
doi	https://doi.org/10.48550/arxiv.2502.13389
ids.doi	https://doi.org/10.48550/arxiv.2502.13389
ids.openalex	https://openalex.org/W4407764166
fwci
type	preprint
title	Reasoning with Reinforced Functional Token Tuning
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T10215
topics[0].field.id	https://openalex.org/fields/17
topics[0].field.display_name	Computer Science
topics[0].score	0.968999981880188
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/1702
topics[0].subfield.display_name	Artificial Intelligence
topics[0].display_name	Semantic Web and Ontologies
topics[1].id	https://openalex.org/T10181
topics[1].field.id	https://openalex.org/fields/17
topics[1].field.display_name	Computer Science
topics[1].score	0.9300000071525574
topics[1].domain.id	https://openalex.org/domains/3
topics[1].domain.display_name	Physical Sciences
topics[1].subfield.id	https://openalex.org/subfields/1702
topics[1].subfield.display_name	Artificial Intelligence
topics[1].display_name	Natural Language Processing Techniques
topics[2].id	https://openalex.org/T11010
topics[2].field.id	https://openalex.org/fields/17
topics[2].field.display_name	Computer Science
topics[2].score	0.9182000160217285
topics[2].domain.id	https://openalex.org/domains/3
topics[2].domain.display_name	Physical Sciences
topics[2].subfield.id	https://openalex.org/subfields/1702
topics[2].subfield.display_name	Artificial Intelligence
topics[2].display_name	Logic, Reasoning, and Knowledge
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C48145219
concepts[0].level	2
concepts[0].score	0.8233257532119751
concepts[0].wikidata	https://www.wikidata.org/wiki/Q1335365
concepts[0].display_name	Security token
concepts[1].id	https://openalex.org/C41008148
concepts[1].level	0
concepts[1].score	0.5094476342201233
concepts[1].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[1].display_name	Computer science
concepts[2].id	https://openalex.org/C38652104
concepts[2].level	1
concepts[2].score	0.17612555623054504
concepts[2].wikidata	https://www.wikidata.org/wiki/Q3510521
concepts[2].display_name	Computer security
keywords[0].id	https://openalex.org/keywords/security-token
keywords[0].score	0.8233257532119751
keywords[0].display_name	Security token
keywords[1].id	https://openalex.org/keywords/computer-science
keywords[1].score	0.5094476342201233
keywords[1].display_name	Computer science
keywords[2].id	https://openalex.org/keywords/computer-security
keywords[2].score	0.17612555623054504
keywords[2].display_name	Computer security
language	en
locations[0].id	pmh:oai:arXiv.org:2502.13389
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2502.13389
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2502.13389
locations[1].id	doi:10.48550/arxiv.2502.13389
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2502.13389
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5115603666
authorships[0].author.orcid	https://orcid.org/0000-0002-7138-9477
authorships[0].author.display_name	K. Zhang
authorships[0].author_position	first
authorships[0].raw_author_name	Zhang, Kongcheng
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5101598160
authorships[1].author.orcid	https://orcid.org/0000-0002-1332-7323
authorships[1].author.display_name	Qi Yao
authorships[1].author_position	middle
authorships[1].raw_author_name	Yao, Qi
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5057196915
authorships[2].author.orcid	https://orcid.org/0000-0001-8939-041X
authorships[2].author.display_name	Baisheng Lai
authorships[2].author_position	middle
authorships[2].raw_author_name	Lai, Baisheng
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5108055047
authorships[3].author.orcid	https://orcid.org/0000-0002-5353-7843
authorships[3].author.display_name	Jiaxing Huang
authorships[3].author_position	middle
authorships[3].raw_author_name	Huang, Jiaxing
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5111361888
authorships[4].author.orcid
authorships[4].author.display_name	Wenkai Fang
authorships[4].author_position	middle
authorships[4].raw_author_name	Fang, Wenkai
authorships[4].is_corresponding	False
authorships[5].author.id	https://openalex.org/A5074103823
authorships[5].author.orcid	https://orcid.org/0000-0001-7225-5449
authorships[5].author.display_name	Dacheng Tao
authorships[5].author_position	middle
authorships[5].raw_author_name	Tao, Dacheng
authorships[5].is_corresponding	False
authorships[6].author.id	https://openalex.org/A5026532752
authorships[6].author.orcid	https://orcid.org/0000-0003-2621-6048
authorships[6].author.display_name	Mingli Song
authorships[6].author_position	middle
authorships[6].raw_author_name	Song, Mingli
authorships[6].is_corresponding	False
authorships[7].author.id	https://openalex.org/A5090854835
authorships[7].author.orcid	https://orcid.org/0000-0003-0584-9129
authorships[7].author.display_name	Shunyu Liu
authorships[7].author_position	last
authorships[7].raw_author_name	Liu, Shunyu
authorships[7].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2502.13389
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	Reasoning with Reinforced Functional Token Tuning
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T10215
primary_topic.field.id	https://openalex.org/fields/17
primary_topic.field.display_name	Computer Science
primary_topic.score	0.968999981880188
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1702
primary_topic.subfield.display_name	Artificial Intelligence
primary_topic.display_name	Semantic Web and Ontologies
related_works	https://openalex.org/W4391375266, https://openalex.org/W2899084033, https://openalex.org/W2748952813, https://openalex.org/W4388335561, https://openalex.org/W2970530566, https://openalex.org/W4288261899, https://openalex.org/W4307309205, https://openalex.org/W2967478618, https://openalex.org/W4385009901, https://openalex.org/W4385572700
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2502.13389
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2502.13389
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2502.13389
primary_location.id	pmh:oai:arXiv.org:2502.13389
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2502.13389
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2502.13389
publication_date	2025-02-19
publication_year	2025
referenced_works_count	0
abstract_inverted_index.)	42
abstract_inverted_index.,	40, 41
abstract_inverted_index.a	10, 32
abstract_inverted_index.In	0
abstract_inverted_index.at	155, 162
abstract_inverted_index.is	160
abstract_inverted_index.of	35, 122, 147
abstract_inverted_index.on	108, 126, 140
abstract_inverted_index.to	68, 82, 97, 133, 138
abstract_inverted_index.up	79
abstract_inverted_index.we	3
abstract_inverted_index.(1)	61
abstract_inverted_index.(2)	89
abstract_inverted_index.Our	158
abstract_inverted_index.and	88, 135
abstract_inverted_index.for	86, 114
abstract_inverted_index.set	34
abstract_inverted_index.the	45, 80, 95, 120, 123, 141, 145
abstract_inverted_index.two	59
abstract_inverted_index.MATH	142
abstract_inverted_index.RFTT	30, 57, 125, 148
abstract_inverted_index.code	159
abstract_inverted_index.data	72
abstract_inverted_index.into	44
abstract_inverted_index.more	152
abstract_inverted_index.rich	33
abstract_inverted_index.that	15
abstract_inverted_index.this	1
abstract_inverted_index.tree	66
abstract_inverted_index.with	21, 51, 74, 151
abstract_inverted_index.Large	17
abstract_inverted_index.Token	7
abstract_inverted_index.learn	83
abstract_inverted_index.model	46, 81, 96
abstract_inverted_index.novel	11
abstract_inverted_index.prior	26
abstract_inverted_index.these	84
abstract_inverted_index.time.	157
abstract_inverted_index.token	104
abstract_inverted_index.warms	78
abstract_inverted_index.which	77
abstract_inverted_index.work,	2
abstract_inverted_index.(32.2%	137
abstract_inverted_index.(70.6%	132
abstract_inverted_index.(LLMs)	20
abstract_inverted_index.(e.g.,	39
abstract_inverted_index.60.2%)	139
abstract_inverted_index.79.8%)	134
abstract_inverted_index.Models	19
abstract_inverted_index.Tuning	8
abstract_inverted_index.Unlike	25
abstract_inverted_index.allows	94
abstract_inverted_index.embeds	31
abstract_inverted_index.obtain	69
abstract_inverted_index.online	90
abstract_inverted_index.search	67, 153
abstract_inverted_index.tokens	38, 85
abstract_inverted_index.(RFTT),	9
abstract_inverted_index.diverse	52
abstract_inverted_index.explore	98
abstract_inverted_index.further	93
abstract_inverted_index.phases:	60
abstract_inverted_index.propose	4
abstract_inverted_index.relying	107
abstract_inverted_index.thereby	110
abstract_inverted_index.through	102
abstract_inverted_index.tokens,	76
abstract_inverted_index.without	106
abstract_inverted_index.Language	18
abstract_inverted_index.boosting	130
abstract_inverted_index.dataset.	143
abstract_inverted_index.directly	43
abstract_inverted_index.efforts,	29
abstract_inverted_index.empowers	16
abstract_inverted_index.enabling	48
abstract_inverted_index.improves	150
abstract_inverted_index.learning	92
abstract_inverted_index.pathways	101
abstract_inverted_index.performs	64
abstract_inverted_index.prompts,	109
abstract_inverted_index.proposed	124
abstract_inverted_index.rollouts	154
abstract_inverted_index.sampling	105
abstract_inverted_index.training	71
abstract_inverted_index.Extensive	117
abstract_inverted_index.Moreover,	144
abstract_inverted_index.annotated	73
abstract_inverted_index.available	161
abstract_inverted_index.comprises	58
abstract_inverted_index.different	99
abstract_inverted_index.effective	112
abstract_inverted_index.framework	14
abstract_inverted_index.inference	156
abstract_inverted_index.learnable	36
abstract_inverted_index.reasoning	28, 54, 100
abstract_inverted_index.self-play	22
abstract_inverted_index.Functional	6
abstract_inverted_index.Reinforced	5
abstract_inverted_index.behaviors.	55
abstract_inverted_index.functional	37, 75, 103, 115
abstract_inverted_index.human-like	53
abstract_inverted_index.reasoning.	116
abstract_inverted_index.reasoning;	87
abstract_inverted_index.reinforced	12
abstract_inverted_index.supervised	62
abstract_inverted_index.benchmarks,	128
abstract_inverted_index.demonstrate	119
abstract_inverted_index.experiments	118
abstract_inverted_index.fine-tuning	13, 63
abstract_inverted_index.performance	146
abstract_inverted_index.superiority	121
abstract_inverted_index.vocabulary,	47
abstract_inverted_index.consistently	149
abstract_inverted_index.construction	50
abstract_inverted_index.facilitating	111
abstract_inverted_index.mathematical	127
abstract_inverted_index.Specifically,	56
abstract_inverted_index.capabilities.	24
abstract_inverted_index.prompt-driven	27, 65
abstract_inverted_index.reinforcement	91
abstract_inverted_index.significantly	129
abstract_inverted_index.self-generated	70
abstract_inverted_index.learn-to-reason	23
abstract_inverted_index.chain-of-thought	49
abstract_inverted_index.self-improvement	113
abstract_inverted_index.Qwen-2.5-7B-Instruct	131
abstract_inverted_index.LLaMA-3.1-8B-Instruct	136
abstract_inverted_index.https://github.com/sastpg/RFTT.	163
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	8
citation_normalized_percentile