Detecting Distillation Data from Reasoning Models Article Swipe
H. Zhang
,
Heekyu Choi
,
Shaoping Li
,
Haotong Wei
·
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2510.04850
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2510.04850
Reasoning distillation has emerged as an efficient and powerful paradigm for enhancing the reasoning capabilities of large language models. However, reasoning distillation may inadvertently cause benchmark contamination, where evaluation data included in distillation datasets can inflate performance metrics of distilled models. In this work, we formally define the task of distillation data detection, which is uniquely challenging due to the partial availability of distillation data. Then, we propose a novel and effective method Token Probability Deviation (TBD), which leverages the probability patterns of the generated output tokens. Our method is motivated by the analysis that distilled models tend to generate near-deterministic tokens for seen questions, while producing more low-probability tokens for unseen questions. Our key idea behind TBD is to quantify how far the generated tokens' probabilities deviate from a high reference probability. In effect, our method achieves competitive detection performance by producing lower scores for seen questions than for unseen questions. Extensive experiments demonstrate the effectiveness of our method, achieving an AUC of 0.918 and a TPR@1% FPR of 0.470 on the S1 dataset.
Related Topics
Concepts
No concepts available.
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2510.04850
- https://arxiv.org/pdf/2510.04850
- OA Status
- green
- OpenAlex ID
- https://openalex.org/W4416374253
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4416374253Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2510.04850Digital Object Identifier
- Title
-
Detecting Distillation Data from Reasoning ModelsWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2025Year of publication
- Publication date
-
2025-10-06Full publication date if available
- Authors
-
H. Zhang, Heekyu Choi, Shaoping Li, Haotong WeiList of authors in order
- Landing page
-
https://arxiv.org/abs/2510.04850Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2510.04850Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2510.04850Direct OA link when available
- Cited by
-
0Total citation count in OpenAlex
Full payload
| id | https://openalex.org/W4416374253 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2510.04850 |
| ids.doi | https://doi.org/10.48550/arxiv.2510.04850 |
| ids.openalex | https://openalex.org/W4416374253 |
| fwci | |
| type | preprint |
| title | Detecting Distillation Data from Reasoning Models |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2510.04850 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2510.04850 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2510.04850 |
| locations[1].id | doi:10.48550/arxiv.2510.04850 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2510.04850 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5034533594 |
| authorships[0].author.orcid | https://orcid.org/0000-0002-9843-2877 |
| authorships[0].author.display_name | H. Zhang |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Zhang, Hengxiang |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5101404463 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-1978-3039 |
| authorships[1].author.display_name | Heekyu Choi |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Choi, Hyeong Kyu |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5107072741 |
| authorships[2].author.orcid | https://orcid.org/0000-0003-4255-6229 |
| authorships[2].author.display_name | Shaoping Li |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Li, Sharon |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5031354322 |
| authorships[3].author.orcid | https://orcid.org/0000-0002-7273-6768 |
| authorships[3].author.display_name | Haotong Wei |
| authorships[3].author_position | last |
| authorships[3].raw_author_name | Wei, Hongxin |
| authorships[3].is_corresponding | False |
| has_content.pdf | True |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2510.04850 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Detecting Distillation Data from Reasoning Models |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-28T12:29:45.881840 |
| primary_topic | |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2510.04850 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2510.04850 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2510.04850 |
| primary_location.id | pmh:oai:arXiv.org:2510.04850 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2510.04850 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2510.04850 |
| publication_date | 2025-10-06 |
| publication_year | 2025 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 68, 129, 166 |
| abstract_inverted_index.In | 41, 133 |
| abstract_inverted_index.S1 | 173 |
| abstract_inverted_index.an | 5, 161 |
| abstract_inverted_index.as | 4 |
| abstract_inverted_index.by | 91, 141 |
| abstract_inverted_index.in | 31 |
| abstract_inverted_index.is | 54, 89, 118 |
| abstract_inverted_index.of | 15, 38, 49, 62, 82, 157, 163, 169 |
| abstract_inverted_index.on | 171 |
| abstract_inverted_index.to | 58, 98, 119 |
| abstract_inverted_index.we | 44, 66 |
| abstract_inverted_index.AUC | 162 |
| abstract_inverted_index.FPR | 168 |
| abstract_inverted_index.Our | 87, 113 |
| abstract_inverted_index.TBD | 117 |
| abstract_inverted_index.and | 7, 70, 165 |
| abstract_inverted_index.can | 34 |
| abstract_inverted_index.due | 57 |
| abstract_inverted_index.far | 122 |
| abstract_inverted_index.for | 10, 102, 110, 145, 149 |
| abstract_inverted_index.has | 2 |
| abstract_inverted_index.how | 121 |
| abstract_inverted_index.key | 114 |
| abstract_inverted_index.may | 22 |
| abstract_inverted_index.our | 135, 158 |
| abstract_inverted_index.the | 12, 47, 59, 79, 83, 92, 123, 155, 172 |
| abstract_inverted_index.data | 29, 51 |
| abstract_inverted_index.from | 128 |
| abstract_inverted_index.high | 130 |
| abstract_inverted_index.idea | 115 |
| abstract_inverted_index.more | 107 |
| abstract_inverted_index.seen | 103, 146 |
| abstract_inverted_index.task | 48 |
| abstract_inverted_index.tend | 97 |
| abstract_inverted_index.than | 148 |
| abstract_inverted_index.that | 94 |
| abstract_inverted_index.this | 42 |
| abstract_inverted_index.0.470 | 170 |
| abstract_inverted_index.0.918 | 164 |
| abstract_inverted_index.Then, | 65 |
| abstract_inverted_index.Token | 73 |
| abstract_inverted_index.cause | 24 |
| abstract_inverted_index.data. | 64 |
| abstract_inverted_index.large | 16 |
| abstract_inverted_index.lower | 143 |
| abstract_inverted_index.novel | 69 |
| abstract_inverted_index.where | 27 |
| abstract_inverted_index.which | 53, 77 |
| abstract_inverted_index.while | 105 |
| abstract_inverted_index.work, | 43 |
| abstract_inverted_index.(TBD), | 76 |
| abstract_inverted_index.TPR@1% | 167 |
| abstract_inverted_index.behind | 116 |
| abstract_inverted_index.define | 46 |
| abstract_inverted_index.method | 72, 88, 136 |
| abstract_inverted_index.models | 96 |
| abstract_inverted_index.output | 85 |
| abstract_inverted_index.scores | 144 |
| abstract_inverted_index.tokens | 101, 109 |
| abstract_inverted_index.unseen | 111, 150 |
| abstract_inverted_index.deviate | 127 |
| abstract_inverted_index.effect, | 134 |
| abstract_inverted_index.emerged | 3 |
| abstract_inverted_index.inflate | 35 |
| abstract_inverted_index.method, | 159 |
| abstract_inverted_index.metrics | 37 |
| abstract_inverted_index.models. | 18, 40 |
| abstract_inverted_index.partial | 60 |
| abstract_inverted_index.propose | 67 |
| abstract_inverted_index.tokens' | 125 |
| abstract_inverted_index.tokens. | 86 |
| abstract_inverted_index.However, | 19 |
| abstract_inverted_index.achieves | 137 |
| abstract_inverted_index.analysis | 93 |
| abstract_inverted_index.dataset. | 174 |
| abstract_inverted_index.datasets | 33 |
| abstract_inverted_index.formally | 45 |
| abstract_inverted_index.generate | 99 |
| abstract_inverted_index.included | 30 |
| abstract_inverted_index.language | 17 |
| abstract_inverted_index.paradigm | 9 |
| abstract_inverted_index.patterns | 81 |
| abstract_inverted_index.powerful | 8 |
| abstract_inverted_index.quantify | 120 |
| abstract_inverted_index.uniquely | 55 |
| abstract_inverted_index.Deviation | 75 |
| abstract_inverted_index.Extensive | 152 |
| abstract_inverted_index.Reasoning | 0 |
| abstract_inverted_index.achieving | 160 |
| abstract_inverted_index.benchmark | 25 |
| abstract_inverted_index.detection | 139 |
| abstract_inverted_index.distilled | 39, 95 |
| abstract_inverted_index.effective | 71 |
| abstract_inverted_index.efficient | 6 |
| abstract_inverted_index.enhancing | 11 |
| abstract_inverted_index.generated | 84, 124 |
| abstract_inverted_index.leverages | 78 |
| abstract_inverted_index.motivated | 90 |
| abstract_inverted_index.producing | 106, 142 |
| abstract_inverted_index.questions | 147 |
| abstract_inverted_index.reasoning | 13, 20 |
| abstract_inverted_index.reference | 131 |
| abstract_inverted_index.detection, | 52 |
| abstract_inverted_index.evaluation | 28 |
| abstract_inverted_index.questions, | 104 |
| abstract_inverted_index.questions. | 112, 151 |
| abstract_inverted_index.Probability | 74 |
| abstract_inverted_index.challenging | 56 |
| abstract_inverted_index.competitive | 138 |
| abstract_inverted_index.demonstrate | 154 |
| abstract_inverted_index.experiments | 153 |
| abstract_inverted_index.performance | 36, 140 |
| abstract_inverted_index.probability | 80 |
| abstract_inverted_index.availability | 61 |
| abstract_inverted_index.capabilities | 14 |
| abstract_inverted_index.distillation | 1, 21, 32, 50, 63 |
| abstract_inverted_index.probability. | 132 |
| abstract_inverted_index.effectiveness | 156 |
| abstract_inverted_index.inadvertently | 23 |
| abstract_inverted_index.probabilities | 126 |
| abstract_inverted_index.contamination, | 26 |
| abstract_inverted_index.low-probability | 108 |
| abstract_inverted_index.near-deterministic | 100 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 4 |
| citation_normalized_percentile |