DoubleDipper: Improving Long-Context LLMs via Context Recycling Article Swipe
Arie Cattan
,
Alon Jacovi
,
Alex Fabrikant
,
Jonathan Herzig
,
Roee Aharoni
,
Hannah Rashkin
,
Dror Marcus
,
Avinatan Hassidim
,
Yossi Matias
,
Idan Szpektor
,
Avi Caciularu
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2406.13632
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2406.13632
Despite recent advancements in Large Language Models (LLMs), their performance on tasks involving long contexts remains sub-optimal. In this work, we propose DoubleDipper, a novel In-Context-Learning method that automatically generates few-shot examples for long context QA tasks by recycling contexts. Specifically, given a long input context (1-3k tokens) and a query, we generate additional query-output pairs from the given context as few-shot examples, while introducing the context only once. This ensures that the demonstrations are leveraging the same context as the target query while only adding a small number of tokens to the prompt. We further enhance each demonstration by instructing the model to explicitly identify the relevant paragraphs before the answer, which improves performance while providing fine-grained attribution to the answer source. We apply our method on multiple LLMs and obtain substantial improvements (+16 absolute points on average across models) on various QA datasets with long context. Surprisingly, despite introducing only single-hop ICL examples, LLMs successfully generalize to multi-hop long-context QA using our approach.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2406.13632
- https://arxiv.org/pdf/2406.13632
- OA Status
- green
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4399913016
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4399913016Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2406.13632Digital Object Identifier
- Title
-
DoubleDipper: Improving Long-Context LLMs via Context RecyclingWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-06-19Full publication date if available
- Authors
-
Arie Cattan, Alon Jacovi, Alex Fabrikant, Jonathan Herzig, Roee Aharoni, Hannah Rashkin, Dror Marcus, Avinatan Hassidim, Yossi Matias, Idan Szpektor, Avi CaciularuList of authors in order
- Landing page
-
https://arxiv.org/abs/2406.13632Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2406.13632Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2406.13632Direct OA link when available
- Concepts
-
Context (archaeology), Shot (pellet), Work (physics), One shot, Computer science, Environmental science, Engineering, Materials science, Geography, Mechanical engineering, Archaeology, MetallurgyTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4399913016 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2406.13632 |
| ids.doi | https://doi.org/10.48550/arxiv.2406.13632 |
| ids.openalex | https://openalex.org/W4399913016 |
| fwci | |
| type | preprint |
| title | DoubleDipper: Improving Long-Context LLMs via Context Recycling |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T12978 |
| topics[0].field.id | https://openalex.org/fields/33 |
| topics[0].field.display_name | Social Sciences |
| topics[0].score | 0.16590000689029694 |
| topics[0].domain.id | https://openalex.org/domains/2 |
| topics[0].domain.display_name | Social Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/3304 |
| topics[0].subfield.display_name | Education |
| topics[0].display_name | Innovative Teaching Methodologies in Social Sciences |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C2779343474 |
| concepts[0].level | 2 |
| concepts[0].score | 0.7761603593826294 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q3109175 |
| concepts[0].display_name | Context (archaeology) |
| concepts[1].id | https://openalex.org/C2778344882 |
| concepts[1].level | 2 |
| concepts[1].score | 0.733826756477356 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q278938 |
| concepts[1].display_name | Shot (pellet) |
| concepts[2].id | https://openalex.org/C18762648 |
| concepts[2].level | 2 |
| concepts[2].score | 0.5782828330993652 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q42213 |
| concepts[2].display_name | Work (physics) |
| concepts[3].id | https://openalex.org/C2992734406 |
| concepts[3].level | 2 |
| concepts[3].score | 0.5552899241447449 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q413267 |
| concepts[3].display_name | One shot |
| concepts[4].id | https://openalex.org/C41008148 |
| concepts[4].level | 0 |
| concepts[4].score | 0.3673015236854553 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[4].display_name | Computer science |
| concepts[5].id | https://openalex.org/C39432304 |
| concepts[5].level | 0 |
| concepts[5].score | 0.3232317566871643 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q188847 |
| concepts[5].display_name | Environmental science |
| concepts[6].id | https://openalex.org/C127413603 |
| concepts[6].level | 0 |
| concepts[6].score | 0.2672475576400757 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q11023 |
| concepts[6].display_name | Engineering |
| concepts[7].id | https://openalex.org/C192562407 |
| concepts[7].level | 0 |
| concepts[7].score | 0.15418538451194763 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q228736 |
| concepts[7].display_name | Materials science |
| concepts[8].id | https://openalex.org/C205649164 |
| concepts[8].level | 0 |
| concepts[8].score | 0.1392614245414734 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q1071 |
| concepts[8].display_name | Geography |
| concepts[9].id | https://openalex.org/C78519656 |
| concepts[9].level | 1 |
| concepts[9].score | 0.09954506158828735 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q101333 |
| concepts[9].display_name | Mechanical engineering |
| concepts[10].id | https://openalex.org/C166957645 |
| concepts[10].level | 1 |
| concepts[10].score | 0.0 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q23498 |
| concepts[10].display_name | Archaeology |
| concepts[11].id | https://openalex.org/C191897082 |
| concepts[11].level | 1 |
| concepts[11].score | 0.0 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q11467 |
| concepts[11].display_name | Metallurgy |
| keywords[0].id | https://openalex.org/keywords/context |
| keywords[0].score | 0.7761603593826294 |
| keywords[0].display_name | Context (archaeology) |
| keywords[1].id | https://openalex.org/keywords/shot |
| keywords[1].score | 0.733826756477356 |
| keywords[1].display_name | Shot (pellet) |
| keywords[2].id | https://openalex.org/keywords/work |
| keywords[2].score | 0.5782828330993652 |
| keywords[2].display_name | Work (physics) |
| keywords[3].id | https://openalex.org/keywords/one-shot |
| keywords[3].score | 0.5552899241447449 |
| keywords[3].display_name | One shot |
| keywords[4].id | https://openalex.org/keywords/computer-science |
| keywords[4].score | 0.3673015236854553 |
| keywords[4].display_name | Computer science |
| keywords[5].id | https://openalex.org/keywords/environmental-science |
| keywords[5].score | 0.3232317566871643 |
| keywords[5].display_name | Environmental science |
| keywords[6].id | https://openalex.org/keywords/engineering |
| keywords[6].score | 0.2672475576400757 |
| keywords[6].display_name | Engineering |
| keywords[7].id | https://openalex.org/keywords/materials-science |
| keywords[7].score | 0.15418538451194763 |
| keywords[7].display_name | Materials science |
| keywords[8].id | https://openalex.org/keywords/geography |
| keywords[8].score | 0.1392614245414734 |
| keywords[8].display_name | Geography |
| keywords[9].id | https://openalex.org/keywords/mechanical-engineering |
| keywords[9].score | 0.09954506158828735 |
| keywords[9].display_name | Mechanical engineering |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2406.13632 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2406.13632 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2406.13632 |
| locations[1].id | doi:10.48550/arxiv.2406.13632 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | cc-by |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | https://openalex.org/licenses/cc-by |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2406.13632 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5047085655 |
| authorships[0].author.orcid | |
| authorships[0].author.display_name | Arie Cattan |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Cattan, Arie |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5045156755 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-7263-2061 |
| authorships[1].author.display_name | Alon Jacovi |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Jacovi, Alon |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5035688187 |
| authorships[2].author.orcid | |
| authorships[2].author.display_name | Alex Fabrikant |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Fabrikant, Alex |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5071893787 |
| authorships[3].author.orcid | https://orcid.org/0009-0000-7227-6557 |
| authorships[3].author.display_name | Jonathan Herzig |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Herzig, Jonathan |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5027223654 |
| authorships[4].author.orcid | |
| authorships[4].author.display_name | Roee Aharoni |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Aharoni, Roee |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5076412680 |
| authorships[5].author.orcid | |
| authorships[5].author.display_name | Hannah Rashkin |
| authorships[5].author_position | middle |
| authorships[5].raw_author_name | Rashkin, Hannah |
| authorships[5].is_corresponding | False |
| authorships[6].author.id | https://openalex.org/A5110310019 |
| authorships[6].author.orcid | |
| authorships[6].author.display_name | Dror Marcus |
| authorships[6].author_position | middle |
| authorships[6].raw_author_name | Marcus, Dror |
| authorships[6].is_corresponding | False |
| authorships[7].author.id | https://openalex.org/A5089579452 |
| authorships[7].author.orcid | https://orcid.org/0000-0002-3855-344X |
| authorships[7].author.display_name | Avinatan Hassidim |
| authorships[7].author_position | middle |
| authorships[7].raw_author_name | Hassidim, Avinatan |
| authorships[7].is_corresponding | False |
| authorships[8].author.id | https://openalex.org/A5065128060 |
| authorships[8].author.orcid | https://orcid.org/0000-0003-3960-6002 |
| authorships[8].author.display_name | Yossi Matias |
| authorships[8].author_position | middle |
| authorships[8].raw_author_name | Matias, Yossi |
| authorships[8].is_corresponding | False |
| authorships[9].author.id | https://openalex.org/A5026091724 |
| authorships[9].author.orcid | |
| authorships[9].author.display_name | Idan Szpektor |
| authorships[9].author_position | middle |
| authorships[9].raw_author_name | Szpektor, Idan |
| authorships[9].is_corresponding | False |
| authorships[10].author.id | https://openalex.org/A5077903013 |
| authorships[10].author.orcid | https://orcid.org/0000-0003-0573-1075 |
| authorships[10].author.display_name | Avi Caciularu |
| authorships[10].author_position | last |
| authorships[10].raw_author_name | Caciularu, Avi |
| authorships[10].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2406.13632 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2024-06-22T00:00:00 |
| display_name | DoubleDipper: Improving Long-Context LLMs via Context Recycling |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T12978 |
| primary_topic.field.id | https://openalex.org/fields/33 |
| primary_topic.field.display_name | Social Sciences |
| primary_topic.score | 0.16590000689029694 |
| primary_topic.domain.id | https://openalex.org/domains/2 |
| primary_topic.domain.display_name | Social Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/3304 |
| primary_topic.subfield.display_name | Education |
| primary_topic.display_name | Innovative Teaching Methodologies in Social Sciences |
| related_works | https://openalex.org/W2497720472, https://openalex.org/W4292659306, https://openalex.org/W3044321615, https://openalex.org/W4294892107, https://openalex.org/W2806221744, https://openalex.org/W2326937258, https://openalex.org/W394267150, https://openalex.org/W2773965352, https://openalex.org/W2357748469, https://openalex.org/W2392917037 |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2406.13632 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2406.13632 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2406.13632 |
| primary_location.id | pmh:oai:arXiv.org:2406.13632 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2406.13632 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2406.13632 |
| publication_date | 2024-06-19 |
| publication_year | 2024 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 23, 42, 49, 86 |
| abstract_inverted_index.In | 17 |
| abstract_inverted_index.QA | 35, 143, 161 |
| abstract_inverted_index.We | 94, 123 |
| abstract_inverted_index.as | 60, 79 |
| abstract_inverted_index.by | 37, 99 |
| abstract_inverted_index.in | 3 |
| abstract_inverted_index.of | 89 |
| abstract_inverted_index.on | 10, 127, 137, 141 |
| abstract_inverted_index.to | 91, 103, 119, 158 |
| abstract_inverted_index.we | 20, 51 |
| abstract_inverted_index.ICL | 153 |
| abstract_inverted_index.and | 48, 130 |
| abstract_inverted_index.are | 74 |
| abstract_inverted_index.for | 32 |
| abstract_inverted_index.our | 125, 163 |
| abstract_inverted_index.the | 57, 65, 72, 76, 80, 92, 101, 106, 110, 120 |
| abstract_inverted_index.(+16 | 134 |
| abstract_inverted_index.LLMs | 129, 155 |
| abstract_inverted_index.This | 69 |
| abstract_inverted_index.each | 97 |
| abstract_inverted_index.from | 56 |
| abstract_inverted_index.long | 13, 33, 43, 146 |
| abstract_inverted_index.only | 67, 84, 151 |
| abstract_inverted_index.same | 77 |
| abstract_inverted_index.that | 27, 71 |
| abstract_inverted_index.this | 18 |
| abstract_inverted_index.with | 145 |
| abstract_inverted_index.(1-3k | 46 |
| abstract_inverted_index.Large | 4 |
| abstract_inverted_index.apply | 124 |
| abstract_inverted_index.given | 41, 58 |
| abstract_inverted_index.input | 44 |
| abstract_inverted_index.model | 102 |
| abstract_inverted_index.novel | 24 |
| abstract_inverted_index.once. | 68 |
| abstract_inverted_index.pairs | 55 |
| abstract_inverted_index.query | 82 |
| abstract_inverted_index.small | 87 |
| abstract_inverted_index.tasks | 11, 36 |
| abstract_inverted_index.their | 8 |
| abstract_inverted_index.using | 162 |
| abstract_inverted_index.which | 112 |
| abstract_inverted_index.while | 63, 83, 115 |
| abstract_inverted_index.work, | 19 |
| abstract_inverted_index.Models | 6 |
| abstract_inverted_index.across | 139 |
| abstract_inverted_index.adding | 85 |
| abstract_inverted_index.answer | 121 |
| abstract_inverted_index.before | 109 |
| abstract_inverted_index.method | 26, 126 |
| abstract_inverted_index.number | 88 |
| abstract_inverted_index.obtain | 131 |
| abstract_inverted_index.points | 136 |
| abstract_inverted_index.query, | 50 |
| abstract_inverted_index.recent | 1 |
| abstract_inverted_index.target | 81 |
| abstract_inverted_index.tokens | 90 |
| abstract_inverted_index.(LLMs), | 7 |
| abstract_inverted_index.Despite | 0 |
| abstract_inverted_index.answer, | 111 |
| abstract_inverted_index.average | 138 |
| abstract_inverted_index.context | 34, 45, 59, 66, 78 |
| abstract_inverted_index.despite | 149 |
| abstract_inverted_index.enhance | 96 |
| abstract_inverted_index.ensures | 70 |
| abstract_inverted_index.further | 95 |
| abstract_inverted_index.models) | 140 |
| abstract_inverted_index.prompt. | 93 |
| abstract_inverted_index.propose | 21 |
| abstract_inverted_index.remains | 15 |
| abstract_inverted_index.source. | 122 |
| abstract_inverted_index.tokens) | 47 |
| abstract_inverted_index.various | 142 |
| abstract_inverted_index.Language | 5 |
| abstract_inverted_index.absolute | 135 |
| abstract_inverted_index.context. | 147 |
| abstract_inverted_index.contexts | 14 |
| abstract_inverted_index.datasets | 144 |
| abstract_inverted_index.examples | 31 |
| abstract_inverted_index.few-shot | 30, 61 |
| abstract_inverted_index.generate | 52 |
| abstract_inverted_index.identify | 105 |
| abstract_inverted_index.improves | 113 |
| abstract_inverted_index.multiple | 128 |
| abstract_inverted_index.relevant | 107 |
| abstract_inverted_index.approach. | 164 |
| abstract_inverted_index.contexts. | 39 |
| abstract_inverted_index.examples, | 62, 154 |
| abstract_inverted_index.generates | 29 |
| abstract_inverted_index.involving | 12 |
| abstract_inverted_index.multi-hop | 159 |
| abstract_inverted_index.providing | 116 |
| abstract_inverted_index.recycling | 38 |
| abstract_inverted_index.additional | 53 |
| abstract_inverted_index.explicitly | 104 |
| abstract_inverted_index.generalize | 157 |
| abstract_inverted_index.leveraging | 75 |
| abstract_inverted_index.paragraphs | 108 |
| abstract_inverted_index.single-hop | 152 |
| abstract_inverted_index.attribution | 118 |
| abstract_inverted_index.instructing | 100 |
| abstract_inverted_index.introducing | 64, 150 |
| abstract_inverted_index.performance | 9, 114 |
| abstract_inverted_index.substantial | 132 |
| abstract_inverted_index.advancements | 2 |
| abstract_inverted_index.fine-grained | 117 |
| abstract_inverted_index.improvements | 133 |
| abstract_inverted_index.long-context | 160 |
| abstract_inverted_index.query-output | 54 |
| abstract_inverted_index.sub-optimal. | 16 |
| abstract_inverted_index.successfully | 156 |
| abstract_inverted_index.DoubleDipper, | 22 |
| abstract_inverted_index.Specifically, | 40 |
| abstract_inverted_index.Surprisingly, | 148 |
| abstract_inverted_index.automatically | 28 |
| abstract_inverted_index.demonstration | 98 |
| abstract_inverted_index.demonstrations | 73 |
| abstract_inverted_index.In-Context-Learning | 25 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 11 |
| citation_normalized_percentile |