Evaluating the External and Parametric Knowledge Fusion of Large Language Models Article Swipe
Hao Zhang
,
Yuyang Zhang
,
Xiaoguang Li
,
Wenxuan Shi
,
Haonan Xu
,
Huanshuo Liu
,
Yasheng Wang
,
Lifeng Shang
,
Qun Liu
,
Yong Liu
,
Ruiming Tang
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2405.19010
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2405.19010
Integrating external knowledge into large language models (LLMs) presents a promising solution to overcome the limitations imposed by their antiquated and static parametric memory. Prior studies, however, have tended to over-reliance on external knowledge, underestimating the valuable contributions of an LLMs' intrinsic parametric knowledge. The efficacy of LLMs in blending external and parametric knowledge remains largely unexplored, especially in cases where external knowledge is incomplete and necessitates supplementation by their parametric knowledge. We propose to deconstruct knowledge fusion into four distinct scenarios, offering the first thorough investigation of LLM behavior across each. We develop a systematic pipeline for data construction and knowledge infusion to simulate these fusion scenarios, facilitating a series of controlled experiments. Our investigation reveals that enhancing parametric knowledge within LLMs can significantly bolster their capability for knowledge integration. Nonetheless, we identify persistent challenges in memorizing and eliciting parametric knowledge, and determining parametric knowledge boundaries. Our findings aim to steer future explorations on harmonizing external and parametric knowledge within LLMs.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2405.19010
- https://arxiv.org/pdf/2405.19010
- OA Status
- green
- Cited By
- 1
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4399198436
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4399198436Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2405.19010Digital Object Identifier
- Title
-
Evaluating the External and Parametric Knowledge Fusion of Large Language ModelsWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-05-29Full publication date if available
- Authors
-
Hao Zhang, Yuyang Zhang, Xiaoguang Li, Wenxuan Shi, Haonan Xu, Huanshuo Liu, Yasheng Wang, Lifeng Shang, Qun Liu, Yong Liu, Ruiming TangList of authors in order
- Landing page
-
https://arxiv.org/abs/2405.19010Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2405.19010Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2405.19010Direct OA link when available
- Concepts
-
Computer science, Fusion, Parametric statistics, Parametric model, Natural language processing, Artificial intelligence, Linguistics, Mathematics, Statistics, PhilosophyTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
1Total citation count in OpenAlex
- Citations by year (recent)
-
2021: 1Per-year citation counts (last 5 years)
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4399198436 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2405.19010 |
| ids.doi | https://doi.org/10.48550/arxiv.2405.19010 |
| ids.openalex | https://openalex.org/W4399198436 |
| fwci | |
| type | preprint |
| title | Evaluating the External and Parametric Knowledge Fusion of Large Language Models |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10028 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9401999711990356 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Topic Modeling |
| topics[1].id | https://openalex.org/T10181 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9373999834060669 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Natural Language Processing Techniques |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C41008148 |
| concepts[0].level | 0 |
| concepts[0].score | 0.576141893863678 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[0].display_name | Computer science |
| concepts[1].id | https://openalex.org/C158525013 |
| concepts[1].level | 2 |
| concepts[1].score | 0.5266323685646057 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q2593739 |
| concepts[1].display_name | Fusion |
| concepts[2].id | https://openalex.org/C117251300 |
| concepts[2].level | 2 |
| concepts[2].score | 0.5142683386802673 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q1849855 |
| concepts[2].display_name | Parametric statistics |
| concepts[3].id | https://openalex.org/C24574437 |
| concepts[3].level | 3 |
| concepts[3].score | 0.43271195888519287 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q7135228 |
| concepts[3].display_name | Parametric model |
| concepts[4].id | https://openalex.org/C204321447 |
| concepts[4].level | 1 |
| concepts[4].score | 0.3891737759113312 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q30642 |
| concepts[4].display_name | Natural language processing |
| concepts[5].id | https://openalex.org/C154945302 |
| concepts[5].level | 1 |
| concepts[5].score | 0.36408531665802 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[5].display_name | Artificial intelligence |
| concepts[6].id | https://openalex.org/C41895202 |
| concepts[6].level | 1 |
| concepts[6].score | 0.2395096719264984 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q8162 |
| concepts[6].display_name | Linguistics |
| concepts[7].id | https://openalex.org/C33923547 |
| concepts[7].level | 0 |
| concepts[7].score | 0.155437171459198 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q395 |
| concepts[7].display_name | Mathematics |
| concepts[8].id | https://openalex.org/C105795698 |
| concepts[8].level | 1 |
| concepts[8].score | 0.10012856125831604 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q12483 |
| concepts[8].display_name | Statistics |
| concepts[9].id | https://openalex.org/C138885662 |
| concepts[9].level | 0 |
| concepts[9].score | 0.07892945408821106 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q5891 |
| concepts[9].display_name | Philosophy |
| keywords[0].id | https://openalex.org/keywords/computer-science |
| keywords[0].score | 0.576141893863678 |
| keywords[0].display_name | Computer science |
| keywords[1].id | https://openalex.org/keywords/fusion |
| keywords[1].score | 0.5266323685646057 |
| keywords[1].display_name | Fusion |
| keywords[2].id | https://openalex.org/keywords/parametric-statistics |
| keywords[2].score | 0.5142683386802673 |
| keywords[2].display_name | Parametric statistics |
| keywords[3].id | https://openalex.org/keywords/parametric-model |
| keywords[3].score | 0.43271195888519287 |
| keywords[3].display_name | Parametric model |
| keywords[4].id | https://openalex.org/keywords/natural-language-processing |
| keywords[4].score | 0.3891737759113312 |
| keywords[4].display_name | Natural language processing |
| keywords[5].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[5].score | 0.36408531665802 |
| keywords[5].display_name | Artificial intelligence |
| keywords[6].id | https://openalex.org/keywords/linguistics |
| keywords[6].score | 0.2395096719264984 |
| keywords[6].display_name | Linguistics |
| keywords[7].id | https://openalex.org/keywords/mathematics |
| keywords[7].score | 0.155437171459198 |
| keywords[7].display_name | Mathematics |
| keywords[8].id | https://openalex.org/keywords/statistics |
| keywords[8].score | 0.10012856125831604 |
| keywords[8].display_name | Statistics |
| keywords[9].id | https://openalex.org/keywords/philosophy |
| keywords[9].score | 0.07892945408821106 |
| keywords[9].display_name | Philosophy |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2405.19010 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2405.19010 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2405.19010 |
| locations[1].id | doi:10.48550/arxiv.2405.19010 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2405.19010 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5100396844 |
| authorships[0].author.orcid | https://orcid.org/0000-0002-0404-6941 |
| authorships[0].author.display_name | Hao Zhang |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Zhang, Hao |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5100409569 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-2244-0125 |
| authorships[1].author.display_name | Yuyang Zhang |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Zhang, Yuyang |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5100373865 |
| authorships[2].author.orcid | https://orcid.org/0000-0002-3345-1313 |
| authorships[2].author.display_name | Xiaoguang Li |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Li, Xiaoguang |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5103110802 |
| authorships[3].author.orcid | https://orcid.org/0000-0003-3217-3646 |
| authorships[3].author.display_name | Wenxuan Shi |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Shi, Wenxuan |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5101432723 |
| authorships[4].author.orcid | https://orcid.org/0009-0003-9664-0378 |
| authorships[4].author.display_name | Haonan Xu |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Xu, Haonan |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5104316983 |
| authorships[5].author.orcid | |
| authorships[5].author.display_name | Huanshuo Liu |
| authorships[5].author_position | middle |
| authorships[5].raw_author_name | Liu, Huanshuo |
| authorships[5].is_corresponding | False |
| authorships[6].author.id | https://openalex.org/A5115592503 |
| authorships[6].author.orcid | |
| authorships[6].author.display_name | Yasheng Wang |
| authorships[6].author_position | middle |
| authorships[6].raw_author_name | Wang, Yasheng |
| authorships[6].is_corresponding | False |
| authorships[7].author.id | https://openalex.org/A5046228314 |
| authorships[7].author.orcid | |
| authorships[7].author.display_name | Lifeng Shang |
| authorships[7].author_position | middle |
| authorships[7].raw_author_name | Shang, Lifeng |
| authorships[7].is_corresponding | False |
| authorships[8].author.id | https://openalex.org/A5100426149 |
| authorships[8].author.orcid | https://orcid.org/0000-0001-6997-3239 |
| authorships[8].author.display_name | Qun Liu |
| authorships[8].author_position | middle |
| authorships[8].raw_author_name | Liu, Qun |
| authorships[8].is_corresponding | False |
| authorships[9].author.id | https://openalex.org/A5028071674 |
| authorships[9].author.orcid | https://orcid.org/0000-0002-6739-621X |
| authorships[9].author.display_name | Yong Liu |
| authorships[9].author_position | middle |
| authorships[9].raw_author_name | Liu, Yong |
| authorships[9].is_corresponding | False |
| authorships[10].author.id | https://openalex.org/A5054330014 |
| authorships[10].author.orcid | https://orcid.org/0000-0002-9224-2431 |
| authorships[10].author.display_name | Ruiming Tang |
| authorships[10].author_position | last |
| authorships[10].raw_author_name | Tang, Ruiming |
| authorships[10].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2405.19010 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Evaluating the External and Parametric Knowledge Fusion of Large Language Models |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10028 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9401999711990356 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Topic Modeling |
| related_works | https://openalex.org/W2289718384, https://openalex.org/W3204019825, https://openalex.org/W1995675544, https://openalex.org/W2509524819, https://openalex.org/W2012121796, https://openalex.org/W2068427817, https://openalex.org/W2952090425, https://openalex.org/W2538333368, https://openalex.org/W3127866798, https://openalex.org/W4294845631 |
| cited_by_count | 1 |
| counts_by_year[0].year | 2021 |
| counts_by_year[0].cited_by_count | 1 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2405.19010 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2405.19010 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2405.19010 |
| primary_location.id | pmh:oai:arXiv.org:2405.19010 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2405.19010 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2405.19010 |
| publication_date | 2024-05-29 |
| publication_year | 2024 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 9, 94, 109 |
| abstract_inverted_index.We | 72, 92 |
| abstract_inverted_index.an | 39 |
| abstract_inverted_index.by | 17, 68 |
| abstract_inverted_index.in | 48, 58, 136 |
| abstract_inverted_index.is | 63 |
| abstract_inverted_index.of | 38, 46, 87, 111 |
| abstract_inverted_index.on | 31, 154 |
| abstract_inverted_index.to | 12, 29, 74, 103, 150 |
| abstract_inverted_index.we | 132 |
| abstract_inverted_index.LLM | 88 |
| abstract_inverted_index.Our | 114, 147 |
| abstract_inverted_index.The | 44 |
| abstract_inverted_index.aim | 149 |
| abstract_inverted_index.and | 20, 51, 65, 100, 138, 142, 157 |
| abstract_inverted_index.can | 123 |
| abstract_inverted_index.for | 97, 128 |
| abstract_inverted_index.the | 14, 35, 83 |
| abstract_inverted_index.LLMs | 47, 122 |
| abstract_inverted_index.data | 98 |
| abstract_inverted_index.four | 79 |
| abstract_inverted_index.have | 27 |
| abstract_inverted_index.into | 3, 78 |
| abstract_inverted_index.that | 117 |
| abstract_inverted_index.LLMs' | 40 |
| abstract_inverted_index.LLMs. | 161 |
| abstract_inverted_index.Prior | 24 |
| abstract_inverted_index.cases | 59 |
| abstract_inverted_index.each. | 91 |
| abstract_inverted_index.first | 84 |
| abstract_inverted_index.large | 4 |
| abstract_inverted_index.steer | 151 |
| abstract_inverted_index.their | 18, 69, 126 |
| abstract_inverted_index.these | 105 |
| abstract_inverted_index.where | 60 |
| abstract_inverted_index.(LLMs) | 7 |
| abstract_inverted_index.across | 90 |
| abstract_inverted_index.fusion | 77, 106 |
| abstract_inverted_index.future | 152 |
| abstract_inverted_index.models | 6 |
| abstract_inverted_index.series | 110 |
| abstract_inverted_index.static | 21 |
| abstract_inverted_index.tended | 28 |
| abstract_inverted_index.within | 121, 160 |
| abstract_inverted_index.bolster | 125 |
| abstract_inverted_index.develop | 93 |
| abstract_inverted_index.imposed | 16 |
| abstract_inverted_index.largely | 55 |
| abstract_inverted_index.memory. | 23 |
| abstract_inverted_index.propose | 73 |
| abstract_inverted_index.remains | 54 |
| abstract_inverted_index.reveals | 116 |
| abstract_inverted_index.behavior | 89 |
| abstract_inverted_index.blending | 49 |
| abstract_inverted_index.distinct | 80 |
| abstract_inverted_index.efficacy | 45 |
| abstract_inverted_index.external | 1, 32, 50, 61, 156 |
| abstract_inverted_index.findings | 148 |
| abstract_inverted_index.however, | 26 |
| abstract_inverted_index.identify | 133 |
| abstract_inverted_index.infusion | 102 |
| abstract_inverted_index.language | 5 |
| abstract_inverted_index.offering | 82 |
| abstract_inverted_index.overcome | 13 |
| abstract_inverted_index.pipeline | 96 |
| abstract_inverted_index.presents | 8 |
| abstract_inverted_index.simulate | 104 |
| abstract_inverted_index.solution | 11 |
| abstract_inverted_index.studies, | 25 |
| abstract_inverted_index.thorough | 85 |
| abstract_inverted_index.valuable | 36 |
| abstract_inverted_index.eliciting | 139 |
| abstract_inverted_index.enhancing | 118 |
| abstract_inverted_index.intrinsic | 41 |
| abstract_inverted_index.knowledge | 2, 53, 62, 76, 101, 120, 129, 145, 159 |
| abstract_inverted_index.promising | 10 |
| abstract_inverted_index.antiquated | 19 |
| abstract_inverted_index.capability | 127 |
| abstract_inverted_index.challenges | 135 |
| abstract_inverted_index.controlled | 112 |
| abstract_inverted_index.especially | 57 |
| abstract_inverted_index.incomplete | 64 |
| abstract_inverted_index.knowledge, | 33, 141 |
| abstract_inverted_index.knowledge. | 43, 71 |
| abstract_inverted_index.memorizing | 137 |
| abstract_inverted_index.parametric | 22, 42, 52, 70, 119, 140, 144, 158 |
| abstract_inverted_index.persistent | 134 |
| abstract_inverted_index.scenarios, | 81, 107 |
| abstract_inverted_index.systematic | 95 |
| abstract_inverted_index.Integrating | 0 |
| abstract_inverted_index.boundaries. | 146 |
| abstract_inverted_index.deconstruct | 75 |
| abstract_inverted_index.determining | 143 |
| abstract_inverted_index.harmonizing | 155 |
| abstract_inverted_index.limitations | 15 |
| abstract_inverted_index.unexplored, | 56 |
| abstract_inverted_index.Nonetheless, | 131 |
| abstract_inverted_index.construction | 99 |
| abstract_inverted_index.experiments. | 113 |
| abstract_inverted_index.explorations | 153 |
| abstract_inverted_index.facilitating | 108 |
| abstract_inverted_index.integration. | 130 |
| abstract_inverted_index.necessitates | 66 |
| abstract_inverted_index.contributions | 37 |
| abstract_inverted_index.investigation | 86, 115 |
| abstract_inverted_index.over-reliance | 30 |
| abstract_inverted_index.significantly | 124 |
| abstract_inverted_index.supplementation | 67 |
| abstract_inverted_index.underestimating | 34 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 11 |
| citation_normalized_percentile |