Thus Spake Long-Context Large Language Model Article Swipe
Xiaoran Liu
,
Ruixiao Li
,
Mianqiu Huang
,
Zhigeng Liu
,
Yuerong Song
,
Qipeng Guo
,
S. He
,
Qiqi Wang
,
Linlin Li
,
Qun Liu
,
Yaqian Zhou
,
Xuanjing Huang
,
Xipeng Qiu
·
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2502.17129
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2502.17129
Long context is an important topic in Natural Language Processing (NLP), running through the development of NLP architectures, and offers immense opportunities for Large Language Models (LLMs), giving LLMs the lifelong learning potential akin to humans. Unfortunately, the pursuit of a long context is accompanied by numerous obstacles. Nevertheless, long context remains a core competitive advantage for LLMs. In the past two years, the context length of LLMs has achieved a breakthrough extension to millions of tokens. Moreover, research on long-context LLMs has expanded beyond length extrapolation to a comprehensive focus on architecture, infrastructure, training, and evaluation technologies. Inspired by the symphonic poem, Thus Spake Zarathustra, we draw an analogy between the journey of extending the context of LLM and the attempts of humans to transcend their mortality. In this survey, we will illustrate how LLM struggles between the tremendous need for a longer context and its equal need to accept the fact that it is ultimately finite. To achieve this, we give a global picture of the lifecycle of long-context LLMs from four perspectives: architecture, infrastructure, training, and evaluation, showcasing the full spectrum of long-context technologies. At the end of this survey, we will present 10 unanswered questions currently faced by long-context LLMs. We hope this survey can serve as a systematic introduction to research on long-context LLMs. Video: https://www.bilibili.com/video/BV11h9AYoEYj. Github: https://github.com/OpenMOSS/Thus-Spake-Long-Context-LLM.
Related Topics
Concepts
No concepts available.
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2502.17129
- https://arxiv.org/pdf/2502.17129
- OA Status
- green
- OpenAlex ID
- https://openalex.org/W4414848605
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4414848605Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2502.17129Digital Object Identifier
- Title
-
Thus Spake Long-Context Large Language ModelWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2025Year of publication
- Publication date
-
2025-02-24Full publication date if available
- Authors
-
Xiaoran Liu, Ruixiao Li, Mianqiu Huang, Zhigeng Liu, Yuerong Song, Qipeng Guo, S. He, Qiqi Wang, Linlin Li, Qun Liu, Yaqian Zhou, Xuanjing Huang, Xipeng QiuList of authors in order
- Landing page
-
https://arxiv.org/abs/2502.17129Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2502.17129Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2502.17129Direct OA link when available
- Cited by
-
0Total citation count in OpenAlex
Full payload
| id | https://openalex.org/W4414848605 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2502.17129 |
| ids.doi | https://doi.org/10.48550/arxiv.2502.17129 |
| ids.openalex | https://openalex.org/W4414848605 |
| fwci | |
| type | preprint |
| title | Thus Spake Long-Context Large Language Model |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10028 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.993399977684021 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Topic Modeling |
| topics[1].id | https://openalex.org/T10181 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9666000008583069 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Natural Language Processing Techniques |
| topics[2].id | https://openalex.org/T12016 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9345999956130981 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1710 |
| topics[2].subfield.display_name | Information Systems |
| topics[2].display_name | Web Data Mining and Analysis |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2502.17129 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2502.17129 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2502.17129 |
| locations[1].id | doi:10.48550/arxiv.2502.17129 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2502.17129 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5104033093 |
| authorships[0].author.orcid | |
| authorships[0].author.display_name | Xiaoran Liu |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Liu, Xiaoran |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5101479594 |
| authorships[1].author.orcid | https://orcid.org/0009-0001-4804-4649 |
| authorships[1].author.display_name | Ruixiao Li |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Li, Ruixiao |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5102565005 |
| authorships[2].author.orcid | |
| authorships[2].author.display_name | Mianqiu Huang |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Huang, Mianqiu |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5113402612 |
| authorships[3].author.orcid | |
| authorships[3].author.display_name | Zhigeng Liu |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Liu, Zhigeng |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5111341600 |
| authorships[4].author.orcid | |
| authorships[4].author.display_name | Yuerong Song |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Song, Yuerong |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5030290474 |
| authorships[5].author.orcid | https://orcid.org/0000-0001-7113-651X |
| authorships[5].author.display_name | Qipeng Guo |
| authorships[5].author_position | middle |
| authorships[5].raw_author_name | Guo, Qipeng |
| authorships[5].is_corresponding | False |
| authorships[6].author.id | https://openalex.org/A5076721719 |
| authorships[6].author.orcid | https://orcid.org/0000-0002-2747-9264 |
| authorships[6].author.display_name | S. He |
| authorships[6].author_position | middle |
| authorships[6].raw_author_name | He, Siyang |
| authorships[6].is_corresponding | False |
| authorships[7].author.id | https://openalex.org/A5101519070 |
| authorships[7].author.orcid | https://orcid.org/0000-0002-7181-148X |
| authorships[7].author.display_name | Qiqi Wang |
| authorships[7].author_position | middle |
| authorships[7].raw_author_name | Wang, Qiqi |
| authorships[7].is_corresponding | False |
| authorships[8].author.id | https://openalex.org/A5100318906 |
| authorships[8].author.orcid | https://orcid.org/0000-0001-7076-2594 |
| authorships[8].author.display_name | Linlin Li |
| authorships[8].author_position | middle |
| authorships[8].raw_author_name | Li, Linlin |
| authorships[8].is_corresponding | False |
| authorships[9].author.id | https://openalex.org/A5100426149 |
| authorships[9].author.orcid | https://orcid.org/0000-0001-6997-3239 |
| authorships[9].author.display_name | Qun Liu |
| authorships[9].author_position | middle |
| authorships[9].raw_author_name | Liu, Qun |
| authorships[9].is_corresponding | False |
| authorships[10].author.id | https://openalex.org/A5100559702 |
| authorships[10].author.orcid | |
| authorships[10].author.display_name | Yaqian Zhou |
| authorships[10].author_position | middle |
| authorships[10].raw_author_name | Zhou, Yaqian |
| authorships[10].is_corresponding | False |
| authorships[11].author.id | https://openalex.org/A5088834359 |
| authorships[11].author.orcid | https://orcid.org/0000-0001-9197-9426 |
| authorships[11].author.display_name | Xuanjing Huang |
| authorships[11].author_position | middle |
| authorships[11].raw_author_name | Huang, Xuanjing |
| authorships[11].is_corresponding | False |
| authorships[12].author.id | https://openalex.org/A5044665993 |
| authorships[12].author.orcid | https://orcid.org/0000-0001-7163-5247 |
| authorships[12].author.display_name | Xipeng Qiu |
| authorships[12].author_position | last |
| authorships[12].raw_author_name | Qiu, Xipeng |
| authorships[12].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2502.17129 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Thus Spake Long-Context Large Language Model |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10028 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.993399977684021 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Topic Modeling |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2502.17129 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2502.17129 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2502.17129 |
| primary_location.id | pmh:oai:arXiv.org:2502.17129 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2502.17129 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2502.17129 |
| publication_date | 2025-02-24 |
| publication_year | 2025 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 40, 52, 70, 88, 142, 163, 211 |
| abstract_inverted_index.10 | 196 |
| abstract_inverted_index.At | 187 |
| abstract_inverted_index.In | 58, 128 |
| abstract_inverted_index.To | 158 |
| abstract_inverted_index.We | 204 |
| abstract_inverted_index.an | 3, 108 |
| abstract_inverted_index.as | 210 |
| abstract_inverted_index.by | 45, 99, 201 |
| abstract_inverted_index.in | 6 |
| abstract_inverted_index.is | 2, 43, 155 |
| abstract_inverted_index.it | 154 |
| abstract_inverted_index.of | 15, 39, 66, 75, 113, 117, 122, 166, 169, 184, 190 |
| abstract_inverted_index.on | 79, 91, 216 |
| abstract_inverted_index.to | 34, 73, 87, 124, 149, 214 |
| abstract_inverted_index.we | 106, 131, 161, 193 |
| abstract_inverted_index.LLM | 118, 135 |
| abstract_inverted_index.NLP | 16 |
| abstract_inverted_index.and | 18, 95, 119, 145, 178 |
| abstract_inverted_index.can | 208 |
| abstract_inverted_index.end | 189 |
| abstract_inverted_index.for | 22, 56, 141 |
| abstract_inverted_index.has | 68, 82 |
| abstract_inverted_index.how | 134 |
| abstract_inverted_index.its | 146 |
| abstract_inverted_index.the | 13, 29, 37, 59, 63, 100, 111, 115, 120, 138, 151, 167, 181, 188 |
| abstract_inverted_index.two | 61 |
| abstract_inverted_index.LLMs | 28, 67, 81, 171 |
| abstract_inverted_index.Long | 0 |
| abstract_inverted_index.Thus | 103 |
| abstract_inverted_index.akin | 33 |
| abstract_inverted_index.core | 53 |
| abstract_inverted_index.draw | 107 |
| abstract_inverted_index.fact | 152 |
| abstract_inverted_index.four | 173 |
| abstract_inverted_index.from | 172 |
| abstract_inverted_index.full | 182 |
| abstract_inverted_index.give | 162 |
| abstract_inverted_index.hope | 205 |
| abstract_inverted_index.long | 41, 49 |
| abstract_inverted_index.need | 140, 148 |
| abstract_inverted_index.past | 60 |
| abstract_inverted_index.that | 153 |
| abstract_inverted_index.this | 129, 191, 206 |
| abstract_inverted_index.will | 132, 194 |
| abstract_inverted_index.LLMs. | 57, 203, 218 |
| abstract_inverted_index.Large | 23 |
| abstract_inverted_index.Spake | 104 |
| abstract_inverted_index.equal | 147 |
| abstract_inverted_index.faced | 200 |
| abstract_inverted_index.focus | 90 |
| abstract_inverted_index.poem, | 102 |
| abstract_inverted_index.serve | 209 |
| abstract_inverted_index.their | 126 |
| abstract_inverted_index.this, | 160 |
| abstract_inverted_index.topic | 5 |
| abstract_inverted_index.(NLP), | 10 |
| abstract_inverted_index.Models | 25 |
| abstract_inverted_index.Video: | 219 |
| abstract_inverted_index.accept | 150 |
| abstract_inverted_index.beyond | 84 |
| abstract_inverted_index.giving | 27 |
| abstract_inverted_index.global | 164 |
| abstract_inverted_index.humans | 123 |
| abstract_inverted_index.length | 65, 85 |
| abstract_inverted_index.longer | 143 |
| abstract_inverted_index.offers | 19 |
| abstract_inverted_index.survey | 207 |
| abstract_inverted_index.years, | 62 |
| abstract_inverted_index.(LLMs), | 26 |
| abstract_inverted_index.Github: | 221 |
| abstract_inverted_index.Natural | 7 |
| abstract_inverted_index.achieve | 159 |
| abstract_inverted_index.analogy | 109 |
| abstract_inverted_index.between | 110, 137 |
| abstract_inverted_index.context | 1, 42, 50, 64, 116, 144 |
| abstract_inverted_index.finite. | 157 |
| abstract_inverted_index.humans. | 35 |
| abstract_inverted_index.immense | 20 |
| abstract_inverted_index.journey | 112 |
| abstract_inverted_index.picture | 165 |
| abstract_inverted_index.present | 195 |
| abstract_inverted_index.pursuit | 38 |
| abstract_inverted_index.remains | 51 |
| abstract_inverted_index.running | 11 |
| abstract_inverted_index.survey, | 130, 192 |
| abstract_inverted_index.through | 12 |
| abstract_inverted_index.tokens. | 76 |
| abstract_inverted_index.Inspired | 98 |
| abstract_inverted_index.Language | 8, 24 |
| abstract_inverted_index.achieved | 69 |
| abstract_inverted_index.attempts | 121 |
| abstract_inverted_index.expanded | 83 |
| abstract_inverted_index.learning | 31 |
| abstract_inverted_index.lifelong | 30 |
| abstract_inverted_index.millions | 74 |
| abstract_inverted_index.numerous | 46 |
| abstract_inverted_index.research | 78, 215 |
| abstract_inverted_index.spectrum | 183 |
| abstract_inverted_index.Moreover, | 77 |
| abstract_inverted_index.advantage | 55 |
| abstract_inverted_index.currently | 199 |
| abstract_inverted_index.extending | 114 |
| abstract_inverted_index.extension | 72 |
| abstract_inverted_index.important | 4 |
| abstract_inverted_index.lifecycle | 168 |
| abstract_inverted_index.potential | 32 |
| abstract_inverted_index.questions | 198 |
| abstract_inverted_index.struggles | 136 |
| abstract_inverted_index.symphonic | 101 |
| abstract_inverted_index.training, | 94, 177 |
| abstract_inverted_index.transcend | 125 |
| abstract_inverted_index.Processing | 9 |
| abstract_inverted_index.evaluation | 96 |
| abstract_inverted_index.illustrate | 133 |
| abstract_inverted_index.mortality. | 127 |
| abstract_inverted_index.obstacles. | 47 |
| abstract_inverted_index.showcasing | 180 |
| abstract_inverted_index.systematic | 212 |
| abstract_inverted_index.tremendous | 139 |
| abstract_inverted_index.ultimately | 156 |
| abstract_inverted_index.unanswered | 197 |
| abstract_inverted_index.accompanied | 44 |
| abstract_inverted_index.competitive | 54 |
| abstract_inverted_index.development | 14 |
| abstract_inverted_index.evaluation, | 179 |
| abstract_inverted_index.Zarathustra, | 105 |
| abstract_inverted_index.breakthrough | 71 |
| abstract_inverted_index.introduction | 213 |
| abstract_inverted_index.long-context | 80, 170, 185, 202, 217 |
| abstract_inverted_index.Nevertheless, | 48 |
| abstract_inverted_index.architecture, | 92, 175 |
| abstract_inverted_index.comprehensive | 89 |
| abstract_inverted_index.extrapolation | 86 |
| abstract_inverted_index.opportunities | 21 |
| abstract_inverted_index.perspectives: | 174 |
| abstract_inverted_index.technologies. | 97, 186 |
| abstract_inverted_index.Unfortunately, | 36 |
| abstract_inverted_index.architectures, | 17 |
| abstract_inverted_index.infrastructure, | 93, 176 |
| abstract_inverted_index.https://www.bilibili.com/video/BV11h9AYoEYj. | 220 |
| abstract_inverted_index.https://github.com/OpenMOSS/Thus-Spake-Long-Context-LLM. | 222 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 13 |
| citation_normalized_percentile |