A Two-Stage Framework with Self-Supervised Distillation For Cross-Domain Text Classification Article Swipe

PDF

Yun‐Long Feng , Bohan Li , Libo Qin , Xu Xiao , Wanxiang Che ·

YOU? · · 2023 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2304.09820

Cross-domain text classification aims to adapt models to a target domain that lacks labeled data. It leverages or reuses rich labeled data from the different but related source domain(s) and unlabeled data from the target domain. To this end, previous work focuses on either extracting domain-invariant features or task-agnostic features, ignoring domain-aware features that may be present in the target domain and could be useful for the downstream task. In this paper, we propose a two-stage framework for cross-domain text classification. In the first stage, we finetune the model with mask language modeling (MLM) and labeled data from the source domain. In the second stage, we further fine-tune the model with self-supervised distillation (SSD) and unlabeled data from the target domain. We evaluate its performance on a public cross-domain text classification benchmark and the experiment results show that our method achieves new state-of-the-art results for both single-source domain adaptations (94.17% $\uparrow$1.03%) and multi-source domain adaptations (95.09% $\uparrow$1.34%).

Related Topics

Computer Science

Benchmark (Surveying)

Artificial Intelligence

Mathematical Analysis

Systems Engineering

Geodesy

Organic Chemistry

Concepts

Computer science Domain (mathematical analysis) Benchmark (surveying) Artificial intelligence Labeled data Distillation Task (project management) Invariant (physics) Natural language processing Machine learning Pattern recognition (psychology) Mathematics Engineering Chemistry Mathematical physics Geography Mathematical analysis Systems engineering Geodesy Organic chemistry

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2304.09820
PDF: https://arxiv.org/pdf/2304.09820
OA Status: green
Cited By: 1
Related Works: 10
OpenAlex ID: https://openalex.org/W4366566347

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4366566347

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2304.09820

Digital Object Identifier
Title: A Two-Stage Framework with Self-Supervised Distillation For Cross-Domain Text Classification

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2023

Year of publication
Publication date: 2023-04-18

Full publication date if available
Authors: Yun‐Long Feng, Bohan Li, Libo Qin, Xu Xiao, Wanxiang Che

List of authors in order
Landing page: https://arxiv.org/abs/2304.09820

Publisher landing page
PDF URL: https://arxiv.org/pdf/2304.09820

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2304.09820

Direct OA link when available
Concepts: Computer science, Domain (mathematical analysis), Benchmark (surveying), Artificial intelligence, Labeled data, Distillation, Task (project management), Invariant (physics), Natural language processing, Machine learning, Pattern recognition (psychology), Mathematics, Engineering, Chemistry, Mathematical physics, Geography, Mathematical analysis, Systems engineering, Geodesy, Organic chemistry

Top concepts (fields/topics) attached by OpenAlex
Cited by: 1

Total citation count in OpenAlex
Citations by year (recent): 2024: 1

Per-year citation counts (last 5 years)
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4366566347
doi	https://doi.org/10.48550/arxiv.2304.09820
ids.doi	https://doi.org/10.48550/arxiv.2304.09820
ids.openalex	https://openalex.org/W4366566347
fwci
type	preprint
title	A Two-Stage Framework with Self-Supervised Distillation For Cross-Domain Text Classification
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T11550
topics[0].field.id	https://openalex.org/fields/17
topics[0].field.display_name	Computer Science
topics[0].score	0.9373999834060669
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/1702
topics[0].subfield.display_name	Artificial Intelligence
topics[0].display_name	Text and Document Classification Technologies
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C41008148
concepts[0].level	0
concepts[0].score	0.7867938280105591
concepts[0].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[0].display_name	Computer science
concepts[1].id	https://openalex.org/C36503486
concepts[1].level	2
concepts[1].score	0.7227110862731934
concepts[1].wikidata	https://www.wikidata.org/wiki/Q11235244
concepts[1].display_name	Domain (mathematical analysis)
concepts[2].id	https://openalex.org/C185798385
concepts[2].level	2
concepts[2].score	0.6810322999954224
concepts[2].wikidata	https://www.wikidata.org/wiki/Q1161707
concepts[2].display_name	Benchmark (surveying)
concepts[3].id	https://openalex.org/C154945302
concepts[3].level	1
concepts[3].score	0.6253871917724609
concepts[3].wikidata	https://www.wikidata.org/wiki/Q11660
concepts[3].display_name	Artificial intelligence
concepts[4].id	https://openalex.org/C2776145971
concepts[4].level	2
concepts[4].score	0.5589127540588379
concepts[4].wikidata	https://www.wikidata.org/wiki/Q30673951
concepts[4].display_name	Labeled data
concepts[5].id	https://openalex.org/C204030448
concepts[5].level	2
concepts[5].score	0.5206105709075928
concepts[5].wikidata	https://www.wikidata.org/wiki/Q101017
concepts[5].display_name	Distillation
concepts[6].id	https://openalex.org/C2780451532
concepts[6].level	2
concepts[6].score	0.4620686173439026
concepts[6].wikidata	https://www.wikidata.org/wiki/Q759676
concepts[6].display_name	Task (project management)
concepts[7].id	https://openalex.org/C190470478
concepts[7].level	2
concepts[7].score	0.4485359489917755
concepts[7].wikidata	https://www.wikidata.org/wiki/Q2370229
concepts[7].display_name	Invariant (physics)
concepts[8].id	https://openalex.org/C204321447
concepts[8].level	1
concepts[8].score	0.421691358089447
concepts[8].wikidata	https://www.wikidata.org/wiki/Q30642
concepts[8].display_name	Natural language processing
concepts[9].id	https://openalex.org/C119857082
concepts[9].level	1
concepts[9].score	0.39534276723861694
concepts[9].wikidata	https://www.wikidata.org/wiki/Q2539
concepts[9].display_name	Machine learning
concepts[10].id	https://openalex.org/C153180895
concepts[10].level	2
concepts[10].score	0.38602137565612793
concepts[10].wikidata	https://www.wikidata.org/wiki/Q7148389
concepts[10].display_name	Pattern recognition (psychology)
concepts[11].id	https://openalex.org/C33923547
concepts[11].level	0
concepts[11].score	0.09499740600585938
concepts[11].wikidata	https://www.wikidata.org/wiki/Q395
concepts[11].display_name	Mathematics
concepts[12].id	https://openalex.org/C127413603
concepts[12].level	0
concepts[12].score	0.061667799949645996
concepts[12].wikidata	https://www.wikidata.org/wiki/Q11023
concepts[12].display_name	Engineering
concepts[13].id	https://openalex.org/C185592680
concepts[13].level	0
concepts[13].score	0.06025362014770508
concepts[13].wikidata	https://www.wikidata.org/wiki/Q2329
concepts[13].display_name	Chemistry
concepts[14].id	https://openalex.org/C37914503
concepts[14].level	1
concepts[14].score	0.0
concepts[14].wikidata	https://www.wikidata.org/wiki/Q156495
concepts[14].display_name	Mathematical physics
concepts[15].id	https://openalex.org/C205649164
concepts[15].level	0
concepts[15].score	0.0
concepts[15].wikidata	https://www.wikidata.org/wiki/Q1071
concepts[15].display_name	Geography
concepts[16].id	https://openalex.org/C134306372
concepts[16].level	1
concepts[16].score	0.0
concepts[16].wikidata	https://www.wikidata.org/wiki/Q7754
concepts[16].display_name	Mathematical analysis
concepts[17].id	https://openalex.org/C201995342
concepts[17].level	1
concepts[17].score	0.0
concepts[17].wikidata	https://www.wikidata.org/wiki/Q682496
concepts[17].display_name	Systems engineering
concepts[18].id	https://openalex.org/C13280743
concepts[18].level	1
concepts[18].score	0.0
concepts[18].wikidata	https://www.wikidata.org/wiki/Q131089
concepts[18].display_name	Geodesy
concepts[19].id	https://openalex.org/C178790620
concepts[19].level	1
concepts[19].score	0.0
concepts[19].wikidata	https://www.wikidata.org/wiki/Q11351
concepts[19].display_name	Organic chemistry
keywords[0].id	https://openalex.org/keywords/computer-science
keywords[0].score	0.7867938280105591
keywords[0].display_name	Computer science
keywords[1].id	https://openalex.org/keywords/domain
keywords[1].score	0.7227110862731934
keywords[1].display_name	Domain (mathematical analysis)
keywords[2].id	https://openalex.org/keywords/benchmark
keywords[2].score	0.6810322999954224
keywords[2].display_name	Benchmark (surveying)
keywords[3].id	https://openalex.org/keywords/artificial-intelligence
keywords[3].score	0.6253871917724609
keywords[3].display_name	Artificial intelligence
keywords[4].id	https://openalex.org/keywords/labeled-data
keywords[4].score	0.5589127540588379
keywords[4].display_name	Labeled data
keywords[5].id	https://openalex.org/keywords/distillation
keywords[5].score	0.5206105709075928
keywords[5].display_name	Distillation
keywords[6].id	https://openalex.org/keywords/task
keywords[6].score	0.4620686173439026
keywords[6].display_name	Task (project management)
keywords[7].id	https://openalex.org/keywords/invariant
keywords[7].score	0.4485359489917755
keywords[7].display_name	Invariant (physics)
keywords[8].id	https://openalex.org/keywords/natural-language-processing
keywords[8].score	0.421691358089447
keywords[8].display_name	Natural language processing
keywords[9].id	https://openalex.org/keywords/machine-learning
keywords[9].score	0.39534276723861694
keywords[9].display_name	Machine learning
keywords[10].id	https://openalex.org/keywords/pattern-recognition
keywords[10].score	0.38602137565612793
keywords[10].display_name	Pattern recognition (psychology)
keywords[11].id	https://openalex.org/keywords/mathematics
keywords[11].score	0.09499740600585938
keywords[11].display_name	Mathematics
keywords[12].id	https://openalex.org/keywords/engineering
keywords[12].score	0.061667799949645996
keywords[12].display_name	Engineering
keywords[13].id	https://openalex.org/keywords/chemistry
keywords[13].score	0.06025362014770508
keywords[13].display_name	Chemistry
language	en
locations[0].id	pmh:oai:arXiv.org:2304.09820
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2304.09820
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2304.09820
locations[1].id	doi:10.48550/arxiv.2304.09820
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2304.09820
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5100295583
authorships[0].author.orcid	https://orcid.org/0000-0002-0561-8841
authorships[0].author.display_name	Yun‐Long Feng
authorships[0].author_position	first
authorships[0].raw_author_name	Feng, Yunlong
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5100443974
authorships[1].author.orcid	https://orcid.org/0000-0002-3408-9037
authorships[1].author.display_name	Bohan Li
authorships[1].author_position	middle
authorships[1].raw_author_name	Li, Bohan
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5029082837
authorships[2].author.orcid	https://orcid.org/0000-0002-3619-675X
authorships[2].author.display_name	Libo Qin
authorships[2].author_position	middle
authorships[2].raw_author_name	Qin, Libo
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5100319784
authorships[3].author.orcid	https://orcid.org/0000-0001-5774-1094
authorships[3].author.display_name	Xu Xiao
authorships[3].author_position	middle
authorships[3].raw_author_name	Xu, Xiao
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5019108029
authorships[4].author.orcid	https://orcid.org/0000-0002-3907-0335
authorships[4].author.display_name	Wanxiang Che
authorships[4].author_position	last
authorships[4].raw_author_name	Che, Wanxiang
authorships[4].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2304.09820
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2023-04-22T00:00:00
display_name	A Two-Stage Framework with Self-Supervised Distillation For Cross-Domain Text Classification
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T11550
primary_topic.field.id	https://openalex.org/fields/17
primary_topic.field.display_name	Computer Science
primary_topic.score	0.9373999834060669
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1702
primary_topic.subfield.display_name	Artificial Intelligence
primary_topic.display_name	Text and Document Classification Technologies
related_works	https://openalex.org/W2378211422, https://openalex.org/W2745001401, https://openalex.org/W4321353415, https://openalex.org/W2130974462, https://openalex.org/W2028665553, https://openalex.org/W2086519370, https://openalex.org/W972276598, https://openalex.org/W2087343574, https://openalex.org/W4246352526, https://openalex.org/W3144173820
cited_by_count	1
counts_by_year[0].year	2024
counts_by_year[0].cited_by_count	1
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2304.09820
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2304.09820
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2304.09820
primary_location.id	pmh:oai:arXiv.org:2304.09820
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2304.09820
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2304.09820
publication_date	2023-04-18
publication_year	2023
referenced_works_count	0
abstract_inverted_index.a	8, 74, 126
abstract_inverted_index.In	69, 81, 101
abstract_inverted_index.It	15
abstract_inverted_index.To	36
abstract_inverted_index.We	121
abstract_inverted_index.be	55, 63
abstract_inverted_index.in	57
abstract_inverted_index.on	42, 125
abstract_inverted_index.or	17, 47
abstract_inverted_index.to	4, 7
abstract_inverted_index.we	72, 85, 105
abstract_inverted_index.and	29, 61, 94, 114, 132, 151
abstract_inverted_index.but	25
abstract_inverted_index.for	65, 77, 144
abstract_inverted_index.its	123
abstract_inverted_index.may	54
abstract_inverted_index.new	141
abstract_inverted_index.our	138
abstract_inverted_index.the	23, 33, 58, 66, 82, 87, 98, 102, 108, 118, 133
abstract_inverted_index.aims	3
abstract_inverted_index.both	145
abstract_inverted_index.data	21, 31, 96, 116
abstract_inverted_index.end,	38
abstract_inverted_index.from	22, 32, 97, 117
abstract_inverted_index.mask	90
abstract_inverted_index.rich	19
abstract_inverted_index.show	136
abstract_inverted_index.text	1, 79, 129
abstract_inverted_index.that	11, 53, 137
abstract_inverted_index.this	37, 70
abstract_inverted_index.with	89, 110
abstract_inverted_index.work	40
abstract_inverted_index.(MLM)	93
abstract_inverted_index.(SSD)	113
abstract_inverted_index.adapt	5
abstract_inverted_index.could	62
abstract_inverted_index.data.	14
abstract_inverted_index.first	83
abstract_inverted_index.lacks	12
abstract_inverted_index.model	88, 109
abstract_inverted_index.task.	68
abstract_inverted_index.domain	10, 60, 147, 153
abstract_inverted_index.either	43
abstract_inverted_index.method	139
abstract_inverted_index.models	6
abstract_inverted_index.paper,	71
abstract_inverted_index.public	127
abstract_inverted_index.reuses	18
abstract_inverted_index.second	103
abstract_inverted_index.source	27, 99
abstract_inverted_index.stage,	84, 104
abstract_inverted_index.target	9, 34, 59, 119
abstract_inverted_index.useful	64
abstract_inverted_index.(94.17%	149
abstract_inverted_index.(95.09%	155
abstract_inverted_index.domain.	35, 100, 120
abstract_inverted_index.focuses	41
abstract_inverted_index.further	106
abstract_inverted_index.labeled	13, 20, 95
abstract_inverted_index.present	56
abstract_inverted_index.propose	73
abstract_inverted_index.related	26
abstract_inverted_index.results	135, 143
abstract_inverted_index.achieves	140
abstract_inverted_index.evaluate	122
abstract_inverted_index.features	46, 52
abstract_inverted_index.finetune	86
abstract_inverted_index.ignoring	50
abstract_inverted_index.language	91
abstract_inverted_index.modeling	92
abstract_inverted_index.previous	39
abstract_inverted_index.benchmark	131
abstract_inverted_index.different	24
abstract_inverted_index.domain(s)	28
abstract_inverted_index.features,	49
abstract_inverted_index.fine-tune	107
abstract_inverted_index.framework	76
abstract_inverted_index.leverages	16
abstract_inverted_index.two-stage	75
abstract_inverted_index.unlabeled	30, 115
abstract_inverted_index.downstream	67
abstract_inverted_index.experiment	134
abstract_inverted_index.extracting	44
abstract_inverted_index.adaptations	148, 154
abstract_inverted_index.performance	124
abstract_inverted_index.Cross-domain	0
abstract_inverted_index.cross-domain	78, 128
abstract_inverted_index.distillation	112
abstract_inverted_index.domain-aware	51
abstract_inverted_index.multi-source	152
abstract_inverted_index.single-source	146
abstract_inverted_index.task-agnostic	48
abstract_inverted_index.classification	2, 130
abstract_inverted_index.classification.	80
abstract_inverted_index.self-supervised	111
abstract_inverted_index.$\uparrow$1.03%)	150
abstract_inverted_index.domain-invariant	45
abstract_inverted_index.state-of-the-art	142
abstract_inverted_index.$\uparrow$1.34%).	156
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	5
citation_normalized_percentile