Adapting Pretrained Text-to-Text Models for Long Text Sequences Article Swipe

PDF

Wenhan Xiong , Anchit Gupta , Shubham Toshniwal , Yashar Mehdad , Scott Yih ·

YOU? · · 2023 · Open Access · · DOI: https://doi.org/10.18653/v1/2023.findings-emnlp.370

We present an empirical study of adapting an existing pretrained text-to-text model for long-sequence inputs. Through a comprehensive study along three axes of the pretraining pipeline – model architecture, optimization objective, and pretraining corpus, we propose an effective recipe to build long-context models from existing short-context models. Specifically, we replace the full attention in transformers with pooling-augmented blockwise attention, and pretrain the model with a masked-span prediction task with spans of varying lengths. In terms of the pretraining corpus, we find that using randomly concatenated short-documents from a large open-domain corpus results in better performance than using existing long document corpora, which are typically limited in their domain coverage. With these findings, we build a long-context model that achieves competitive performance on long-text QA tasks and establishes the new state of the art on five long-text summarization datasets, often outperforming previous methods with larger model sizes.

Related Topics

Computer Science

Artificial Intelligence

Concepts

Computer science Automatic summarization Artificial intelligence Natural language processing Pipeline (software) Transformer Pooling Context (archaeology) Language model Text generation Context model Speech recognition Task (project management) Information retrieval Management Programming language Biology Paleontology Quantum mechanics Voltage Economics Object (grammar) Physics

Metadata

Type: article
Language: en
Landing Page: https://doi.org/10.18653/v1/2023.findings-emnlp.370
PDF: https://aclanthology.org/2023.findings-emnlp.370.pdf
OA Status: gold
Cited By: 15
References: 48
Related Works: 10
OpenAlex ID: https://openalex.org/W4389520146

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4389520146

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.18653/v1/2023.findings-emnlp.370

Digital Object Identifier
Title: Adapting Pretrained Text-to-Text Models for Long Text Sequences

Work title
Type: article

OpenAlex work type
Language: en

Primary language
Publication year: 2023

Year of publication
Publication date: 2023-01-01

Full publication date if available
Authors: Wenhan Xiong, Anchit Gupta, Shubham Toshniwal, Yashar Mehdad, Scott Yih

List of authors in order
Landing page: https://doi.org/10.18653/v1/2023.findings-emnlp.370

Publisher landing page
PDF URL: https://aclanthology.org/2023.findings-emnlp.370.pdf

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: gold

Open access status per OpenAlex
OA URL: https://aclanthology.org/2023.findings-emnlp.370.pdf

Direct OA link when available
Concepts: Computer science, Automatic summarization, Artificial intelligence, Natural language processing, Pipeline (software), Transformer, Pooling, Context (archaeology), Language model, Text generation, Context model, Speech recognition, Task (project management), Information retrieval, Management, Programming language, Biology, Paleontology, Quantum mechanics, Voltage, Economics, Object (grammar), Physics

Top concepts (fields/topics) attached by OpenAlex
Cited by: 15

Total citation count in OpenAlex
Citations by year (recent): 2025: 2, 2024: 5, 2023: 8

Per-year citation counts (last 5 years)
References (count): 48

Number of works referenced by this work
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4389520146
doi	https://doi.org/10.18653/v1/2023.findings-emnlp.370
ids.doi	https://doi.org/10.18653/v1/2023.findings-emnlp.370
ids.openalex	https://openalex.org/W4389520146
fwci	3.83164331
type	article
title	Adapting Pretrained Text-to-Text Models for Long Text Sequences
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T10028
topics[0].field.id	https://openalex.org/fields/17
topics[0].field.display_name	Computer Science
topics[0].score	0.9998999834060669
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/1702
topics[0].subfield.display_name	Artificial Intelligence
topics[0].display_name	Topic Modeling
topics[1].id	https://openalex.org/T10181
topics[1].field.id	https://openalex.org/fields/17
topics[1].field.display_name	Computer Science
topics[1].score	0.9995999932289124
topics[1].domain.id	https://openalex.org/domains/3
topics[1].domain.display_name	Physical Sciences
topics[1].subfield.id	https://openalex.org/subfields/1702
topics[1].subfield.display_name	Artificial Intelligence
topics[1].display_name	Natural Language Processing Techniques
topics[2].id	https://openalex.org/T13083
topics[2].field.id	https://openalex.org/fields/17
topics[2].field.display_name	Computer Science
topics[2].score	0.9872999787330627
topics[2].domain.id	https://openalex.org/domains/3
topics[2].domain.display_name	Physical Sciences
topics[2].subfield.id	https://openalex.org/subfields/1702
topics[2].subfield.display_name	Artificial Intelligence
topics[2].display_name	Advanced Text Analysis Techniques
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C41008148
concepts[0].level	0
concepts[0].score	0.8717612624168396
concepts[0].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[0].display_name	Computer science
concepts[1].id	https://openalex.org/C170858558
concepts[1].level	2
concepts[1].score	0.7828789949417114
concepts[1].wikidata	https://www.wikidata.org/wiki/Q1394144
concepts[1].display_name	Automatic summarization
concepts[2].id	https://openalex.org/C154945302
concepts[2].level	1
concepts[2].score	0.6535340547561646
concepts[2].wikidata	https://www.wikidata.org/wiki/Q11660
concepts[2].display_name	Artificial intelligence
concepts[3].id	https://openalex.org/C204321447
concepts[3].level	1
concepts[3].score	0.6370206475257874
concepts[3].wikidata	https://www.wikidata.org/wiki/Q30642
concepts[3].display_name	Natural language processing
concepts[4].id	https://openalex.org/C43521106
concepts[4].level	2
concepts[4].score	0.6334695816040039
concepts[4].wikidata	https://www.wikidata.org/wiki/Q2165493
concepts[4].display_name	Pipeline (software)
concepts[5].id	https://openalex.org/C66322947
concepts[5].level	3
concepts[5].score	0.5695276856422424
concepts[5].wikidata	https://www.wikidata.org/wiki/Q11658
concepts[5].display_name	Transformer
concepts[6].id	https://openalex.org/C70437156
concepts[6].level	2
concepts[6].score	0.5188575983047485
concepts[6].wikidata	https://www.wikidata.org/wiki/Q7228652
concepts[6].display_name	Pooling
concepts[7].id	https://openalex.org/C2779343474
concepts[7].level	2
concepts[7].score	0.5138416886329651
concepts[7].wikidata	https://www.wikidata.org/wiki/Q3109175
concepts[7].display_name	Context (archaeology)
concepts[8].id	https://openalex.org/C137293760
concepts[8].level	2
concepts[8].score	0.5131335854530334
concepts[8].wikidata	https://www.wikidata.org/wiki/Q3621696
concepts[8].display_name	Language model
concepts[9].id	https://openalex.org/C2985684807
concepts[9].level	2
concepts[9].score	0.4683645963668823
concepts[9].wikidata	https://www.wikidata.org/wiki/Q1513879
concepts[9].display_name	Text generation
concepts[10].id	https://openalex.org/C183322885
concepts[10].level	3
concepts[10].score	0.44815951585769653
concepts[10].wikidata	https://www.wikidata.org/wiki/Q17007702
concepts[10].display_name	Context model
concepts[11].id	https://openalex.org/C28490314
concepts[11].level	1
concepts[11].score	0.4223138391971588
concepts[11].wikidata	https://www.wikidata.org/wiki/Q189436
concepts[11].display_name	Speech recognition
concepts[12].id	https://openalex.org/C2780451532
concepts[12].level	2
concepts[12].score	0.4110347032546997
concepts[12].wikidata	https://www.wikidata.org/wiki/Q759676
concepts[12].display_name	Task (project management)
concepts[13].id	https://openalex.org/C23123220
concepts[13].level	1
concepts[13].score	0.33246681094169617
concepts[13].wikidata	https://www.wikidata.org/wiki/Q816826
concepts[13].display_name	Information retrieval
concepts[14].id	https://openalex.org/C187736073
concepts[14].level	1
concepts[14].score	0.0
concepts[14].wikidata	https://www.wikidata.org/wiki/Q2920921
concepts[14].display_name	Management
concepts[15].id	https://openalex.org/C199360897
concepts[15].level	1
concepts[15].score	0.0
concepts[15].wikidata	https://www.wikidata.org/wiki/Q9143
concepts[15].display_name	Programming language
concepts[16].id	https://openalex.org/C86803240
concepts[16].level	0
concepts[16].score	0.0
concepts[16].wikidata	https://www.wikidata.org/wiki/Q420
concepts[16].display_name	Biology
concepts[17].id	https://openalex.org/C151730666
concepts[17].level	1
concepts[17].score	0.0
concepts[17].wikidata	https://www.wikidata.org/wiki/Q7205
concepts[17].display_name	Paleontology
concepts[18].id	https://openalex.org/C62520636
concepts[18].level	1
concepts[18].score	0.0
concepts[18].wikidata	https://www.wikidata.org/wiki/Q944
concepts[18].display_name	Quantum mechanics
concepts[19].id	https://openalex.org/C165801399
concepts[19].level	2
concepts[19].score	0.0
concepts[19].wikidata	https://www.wikidata.org/wiki/Q25428
concepts[19].display_name	Voltage
concepts[20].id	https://openalex.org/C162324750
concepts[20].level	0
concepts[20].score	0.0
concepts[20].wikidata	https://www.wikidata.org/wiki/Q8134
concepts[20].display_name	Economics
concepts[21].id	https://openalex.org/C2781238097
concepts[21].level	2
concepts[21].score	0.0
concepts[21].wikidata	https://www.wikidata.org/wiki/Q175026
concepts[21].display_name	Object (grammar)
concepts[22].id	https://openalex.org/C121332964
concepts[22].level	0
concepts[22].score	0.0
concepts[22].wikidata	https://www.wikidata.org/wiki/Q413
concepts[22].display_name	Physics
keywords[0].id	https://openalex.org/keywords/computer-science
keywords[0].score	0.8717612624168396
keywords[0].display_name	Computer science
keywords[1].id	https://openalex.org/keywords/automatic-summarization
keywords[1].score	0.7828789949417114
keywords[1].display_name	Automatic summarization
keywords[2].id	https://openalex.org/keywords/artificial-intelligence
keywords[2].score	0.6535340547561646
keywords[2].display_name	Artificial intelligence
keywords[3].id	https://openalex.org/keywords/natural-language-processing
keywords[3].score	0.6370206475257874
keywords[3].display_name	Natural language processing
keywords[4].id	https://openalex.org/keywords/pipeline
keywords[4].score	0.6334695816040039
keywords[4].display_name	Pipeline (software)
keywords[5].id	https://openalex.org/keywords/transformer
keywords[5].score	0.5695276856422424
keywords[5].display_name	Transformer
keywords[6].id	https://openalex.org/keywords/pooling
keywords[6].score	0.5188575983047485
keywords[6].display_name	Pooling
keywords[7].id	https://openalex.org/keywords/context
keywords[7].score	0.5138416886329651
keywords[7].display_name	Context (archaeology)
keywords[8].id	https://openalex.org/keywords/language-model
keywords[8].score	0.5131335854530334
keywords[8].display_name	Language model
keywords[9].id	https://openalex.org/keywords/text-generation
keywords[9].score	0.4683645963668823
keywords[9].display_name	Text generation
keywords[10].id	https://openalex.org/keywords/context-model
keywords[10].score	0.44815951585769653
keywords[10].display_name	Context model
keywords[11].id	https://openalex.org/keywords/speech-recognition
keywords[11].score	0.4223138391971588
keywords[11].display_name	Speech recognition
keywords[12].id	https://openalex.org/keywords/task
keywords[12].score	0.4110347032546997
keywords[12].display_name	Task (project management)
keywords[13].id	https://openalex.org/keywords/information-retrieval
keywords[13].score	0.33246681094169617
keywords[13].display_name	Information retrieval
language	en
locations[0].id	doi:10.18653/v1/2023.findings-emnlp.370
locations[0].is_oa	True
locations[0].source
locations[0].license	cc-by
locations[0].pdf_url	https://aclanthology.org/2023.findings-emnlp.370.pdf
locations[0].version	publishedVersion
locations[0].raw_type	proceedings-article
locations[0].license_id	https://openalex.org/licenses/cc-by
locations[0].is_accepted	True
locations[0].is_published	True
locations[0].raw_source_name	Findings of the Association for Computational Linguistics: EMNLP 2023
locations[0].landing_page_url	https://doi.org/10.18653/v1/2023.findings-emnlp.370
indexed_in	crossref
authorships[0].author.id	https://openalex.org/A5110635444
authorships[0].author.orcid
authorships[0].author.display_name	Wenhan Xiong
authorships[0].author_position	first
authorships[0].raw_author_name	Wenhan Xiong
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5081592288
authorships[1].author.orcid
authorships[1].author.display_name	Anchit Gupta
authorships[1].author_position	middle
authorships[1].raw_author_name	Anchit Gupta
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5073890145
authorships[2].author.orcid
authorships[2].author.display_name	Shubham Toshniwal
authorships[2].author_position	middle
authorships[2].raw_author_name	Shubham Toshniwal
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5067997502
authorships[3].author.orcid
authorships[3].author.display_name	Yashar Mehdad
authorships[3].author_position	middle
authorships[3].raw_author_name	Yashar Mehdad
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5027596907
authorships[4].author.orcid
authorships[4].author.display_name	Scott Yih
authorships[4].author_position	last
authorships[4].raw_author_name	Scott Yih
authorships[4].is_corresponding	False
has_content.pdf	True
has_content.grobid_xml	True
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://aclanthology.org/2023.findings-emnlp.370.pdf
open_access.oa_status	gold
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	Adapting Pretrained Text-to-Text Models for Long Text Sequences
has_fulltext	True
is_retracted	False
updated_date	2025-11-06T03:46:38.306776
primary_topic.id	https://openalex.org/T10028
primary_topic.field.id	https://openalex.org/fields/17
primary_topic.field.display_name	Computer Science
primary_topic.score	0.9998999834060669
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1702
primary_topic.subfield.display_name	Artificial Intelligence
primary_topic.display_name	Topic Modeling
related_works	https://openalex.org/W2366403280, https://openalex.org/W1495108544, https://openalex.org/W3010337296, https://openalex.org/W4206715480, https://openalex.org/W2988673764, https://openalex.org/W2996388263, https://openalex.org/W4385890381, https://openalex.org/W2402231715, https://openalex.org/W2986367395, https://openalex.org/W4388685619
cited_by_count	15
counts_by_year[0].year	2025
counts_by_year[0].cited_by_count	2
counts_by_year[1].year	2024
counts_by_year[1].cited_by_count	5
counts_by_year[2].year	2023
counts_by_year[2].cited_by_count	8
locations_count	1
best_oa_location.id	doi:10.18653/v1/2023.findings-emnlp.370
best_oa_location.is_oa	True
best_oa_location.source
best_oa_location.license	cc-by
best_oa_location.pdf_url	https://aclanthology.org/2023.findings-emnlp.370.pdf
best_oa_location.version	publishedVersion
best_oa_location.raw_type	proceedings-article
best_oa_location.license_id	https://openalex.org/licenses/cc-by
best_oa_location.is_accepted	True
best_oa_location.is_published	True
best_oa_location.raw_source_name	Findings of the Association for Computational Linguistics: EMNLP 2023
best_oa_location.landing_page_url	https://doi.org/10.18653/v1/2023.findings-emnlp.370
primary_location.id	doi:10.18653/v1/2023.findings-emnlp.370
primary_location.is_oa	True
primary_location.source
primary_location.license	cc-by
primary_location.pdf_url	https://aclanthology.org/2023.findings-emnlp.370.pdf
primary_location.version	publishedVersion
primary_location.raw_type	proceedings-article
primary_location.license_id	https://openalex.org/licenses/cc-by
primary_location.is_accepted	True
primary_location.is_published	True
primary_location.raw_source_name	Findings of the Association for Computational Linguistics: EMNLP 2023
primary_location.landing_page_url	https://doi.org/10.18653/v1/2023.findings-emnlp.370
publication_date	2023-01-01
publication_year	2023
referenced_works	https://openalex.org/W3209374680, https://openalex.org/W3034999214, https://openalex.org/W2923014074, https://openalex.org/W4287667694, https://openalex.org/W3207546842, https://openalex.org/W3169942382, https://openalex.org/W3033529678, https://openalex.org/W2971034336, https://openalex.org/W3105238007, https://openalex.org/W4287692509, https://openalex.org/W4385245566, https://openalex.org/W4225727438, https://openalex.org/W2996264288, https://openalex.org/W3170490008, https://openalex.org/W3131922516, https://openalex.org/W4299567010, https://openalex.org/W3161820423, https://openalex.org/W4287824654, https://openalex.org/W4287323699, https://openalex.org/W2578338321, https://openalex.org/W3159259047, https://openalex.org/W4286750685, https://openalex.org/W3156789018, https://openalex.org/W4386566496, https://openalex.org/W2963341956, https://openalex.org/W3118781290, https://openalex.org/W3015468748, https://openalex.org/W2963748441, https://openalex.org/W4385573994, https://openalex.org/W4295838474, https://openalex.org/W2963963993, https://openalex.org/W3123615524, https://openalex.org/W4309793872, https://openalex.org/W4321340837, https://openalex.org/W3171639395, https://openalex.org/W4288089799, https://openalex.org/W4225644300, https://openalex.org/W3153603860, https://openalex.org/W3163073193, https://openalex.org/W4306672396, https://openalex.org/W4287888039, https://openalex.org/W4287704453, https://openalex.org/W4385573804, https://openalex.org/W4225958595, https://openalex.org/W4389518794, https://openalex.org/W2963926728, https://openalex.org/W4287183331, https://openalex.org/W2805206884
referenced_works_count	48
abstract_inverted_index.a	16, 64, 87, 114
abstract_inverted_index.In	73
abstract_inverted_index.QA	123
abstract_inverted_index.We	0
abstract_inverted_index.an	2, 7, 36
abstract_inverted_index.in	53, 92, 105
abstract_inverted_index.of	5, 22, 70, 75, 130
abstract_inverted_index.on	121, 133
abstract_inverted_index.to	39
abstract_inverted_index.we	34, 48, 79, 112
abstract_inverted_index.and	31, 59, 125
abstract_inverted_index.are	102
abstract_inverted_index.art	132
abstract_inverted_index.for	12
abstract_inverted_index.new	128
abstract_inverted_index.the	23, 50, 61, 76, 127, 131
abstract_inverted_index.–	26
abstract_inverted_index.With	109
abstract_inverted_index.axes	21
abstract_inverted_index.find	80
abstract_inverted_index.five	134
abstract_inverted_index.from	43, 86
abstract_inverted_index.full	51
abstract_inverted_index.long	98
abstract_inverted_index.task	67
abstract_inverted_index.than	95
abstract_inverted_index.that	81, 117
abstract_inverted_index.with	55, 63, 68, 142
abstract_inverted_index.along	19
abstract_inverted_index.build	40, 113
abstract_inverted_index.large	88
abstract_inverted_index.model	11, 27, 62, 116, 144
abstract_inverted_index.often	138
abstract_inverted_index.spans	69
abstract_inverted_index.state	129
abstract_inverted_index.study	4, 18
abstract_inverted_index.tasks	124
abstract_inverted_index.terms	74
abstract_inverted_index.their	106
abstract_inverted_index.these	110
abstract_inverted_index.three	20
abstract_inverted_index.using	82, 96
abstract_inverted_index.which	101
abstract_inverted_index.better	93
abstract_inverted_index.corpus	90
abstract_inverted_index.domain	107
abstract_inverted_index.larger	143
abstract_inverted_index.models	42
abstract_inverted_index.recipe	38
abstract_inverted_index.sizes.	145
abstract_inverted_index.Through	15
abstract_inverted_index.corpus,	33, 78
abstract_inverted_index.inputs.	14
abstract_inverted_index.limited	104
abstract_inverted_index.methods	141
abstract_inverted_index.models.	46
abstract_inverted_index.present	1
abstract_inverted_index.propose	35
abstract_inverted_index.replace	49
abstract_inverted_index.results	91
abstract_inverted_index.varying	71
abstract_inverted_index.achieves	118
abstract_inverted_index.adapting	6
abstract_inverted_index.corpora,	100
abstract_inverted_index.document	99
abstract_inverted_index.existing	8, 44, 97
abstract_inverted_index.lengths.	72
abstract_inverted_index.pipeline	25
abstract_inverted_index.pretrain	60
abstract_inverted_index.previous	140
abstract_inverted_index.randomly	83
abstract_inverted_index.attention	52
abstract_inverted_index.blockwise	57
abstract_inverted_index.coverage.	108
abstract_inverted_index.datasets,	137
abstract_inverted_index.effective	37
abstract_inverted_index.empirical	3
abstract_inverted_index.findings,	111
abstract_inverted_index.long-text	122, 135
abstract_inverted_index.typically	103
abstract_inverted_index.attention,	58
abstract_inverted_index.objective,	30
abstract_inverted_index.prediction	66
abstract_inverted_index.pretrained	9
abstract_inverted_index.competitive	119
abstract_inverted_index.establishes	126
abstract_inverted_index.masked-span	65
abstract_inverted_index.open-domain	89
abstract_inverted_index.performance	94, 120
abstract_inverted_index.pretraining	24, 32, 77
abstract_inverted_index.concatenated	84
abstract_inverted_index.long-context	41, 115
abstract_inverted_index.optimization	29
abstract_inverted_index.text-to-text	10
abstract_inverted_index.transformers	54
abstract_inverted_index.Specifically,	47
abstract_inverted_index.architecture,	28
abstract_inverted_index.comprehensive	17
abstract_inverted_index.long-sequence	13
abstract_inverted_index.outperforming	139
abstract_inverted_index.short-context	45
abstract_inverted_index.summarization	136
abstract_inverted_index.short-documents	85
abstract_inverted_index.pooling-augmented	56
cited_by_percentile_year.max	99
cited_by_percentile_year.min	95
countries_distinct_count	0
institutions_distinct_count	5
citation_normalized_percentile.value	0.93330596
citation_normalized_percentile.is_in_top_1_percent	False
citation_normalized_percentile.is_in_top_10_percent	True