Tracing and Mitigating Hallucinations in Multimodal LLMs via Dynamic Attention Localization Article Swipe
Jiaye Lin
,
Guimin Hu
,
Li Hu
·
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2509.07864
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2509.07864
Multimodal Large Language Models (MLLMs) achieve strong performance on tasks like image captioning and visual question answering, but remain prone to hallucinations, where generated text conflicts with the visual input. Prior work links this partly to insufficient visual attention, but existing attention-based detectors and mitigation typically apply uniform adjustments across layers and heads, obscuring where errors originate. In this paper, we first show these methods fail to accurately localize problematic layers. Then, we introduce two diagnostics: Layer Image Attention Entropy (LIAE) which flags anomalous layers, and Image Attention Focus (IAF) which scores attention heads within those layers. Analysis shows that LIAE pinpoints faulty layers and IAF reliably ranks heads that warrant correction. Guided by these signals, we propose Dynamic Layer-wise Entropy and Attention Fusion (D-LEAF), a task-agnostic, attention-guided method that dynamically localizes and corrects errors during inference with negligible overhead. Furthermore, by establishing a connection between D-LEAF and DPO, we provide theoretical justification for the effectiveness of D-LEAF. Results show our D-LEAF delivers a 53\% relative improvement on standard captioning benchmarks, and on VQA both accuracy and F1-score improve by approximately 4\%, substantially suppressing hallucinations while preserving efficiency.
Related Topics
Concepts
No concepts available.
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2509.07864
- https://arxiv.org/pdf/2509.07864
- OA Status
- green
- OpenAlex ID
- https://openalex.org/W4416358420
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4416358420Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2509.07864Digital Object Identifier
- Title
-
Tracing and Mitigating Hallucinations in Multimodal LLMs via Dynamic Attention LocalizationWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2025Year of publication
- Publication date
-
2025-09-09Full publication date if available
- Authors
-
Jiaye Lin, Guimin Hu, Li HuList of authors in order
- Landing page
-
https://arxiv.org/abs/2509.07864Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2509.07864Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2509.07864Direct OA link when available
- Cited by
-
0Total citation count in OpenAlex
Full payload
| id | https://openalex.org/W4416358420 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2509.07864 |
| ids.doi | https://doi.org/10.48550/arxiv.2509.07864 |
| ids.openalex | https://openalex.org/W4416358420 |
| fwci | |
| type | preprint |
| title | Tracing and Mitigating Hallucinations in Multimodal LLMs via Dynamic Attention Localization |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2509.07864 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2509.07864 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2509.07864 |
| locations[1].id | doi:10.48550/arxiv.2509.07864 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | cc-by |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | https://openalex.org/licenses/cc-by |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2509.07864 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5059445819 |
| authorships[0].author.orcid | https://orcid.org/0009-0002-4485-6612 |
| authorships[0].author.display_name | Jiaye Lin |
| authorships[0].author_position | last |
| authorships[0].raw_author_name | Lin, Jiaye |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5061615741 |
| authorships[1].author.orcid | https://orcid.org/0000-0001-8364-3076 |
| authorships[1].author.display_name | Guimin Hu |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Hu, Guimin |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5011519506 |
| authorships[2].author.orcid | https://orcid.org/0000-0001-7003-2903 |
| authorships[2].author.display_name | Li Hu |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Hu, Lijie |
| authorships[2].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2509.07864 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Tracing and Mitigating Hallucinations in Multimodal LLMs via Dynamic Attention Localization |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-28T12:05:10.739060 |
| primary_topic | |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2509.07864 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2509.07864 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2509.07864 |
| primary_location.id | pmh:oai:arXiv.org:2509.07864 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2509.07864 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2509.07864 |
| publication_date | 2025-09-09 |
| publication_year | 2025 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 125, 143, 163 |
| abstract_inverted_index.In | 57 |
| abstract_inverted_index.by | 113, 141, 179 |
| abstract_inverted_index.of | 156 |
| abstract_inverted_index.on | 8, 167, 172 |
| abstract_inverted_index.to | 20, 35, 66 |
| abstract_inverted_index.we | 60, 72, 116, 149 |
| abstract_inverted_index.IAF | 105 |
| abstract_inverted_index.VQA | 173 |
| abstract_inverted_index.and | 13, 43, 51, 85, 104, 121, 132, 147, 171, 176 |
| abstract_inverted_index.but | 17, 39 |
| abstract_inverted_index.for | 153 |
| abstract_inverted_index.our | 160 |
| abstract_inverted_index.the | 27, 154 |
| abstract_inverted_index.two | 74 |
| abstract_inverted_index.4\%, | 181 |
| abstract_inverted_index.53\% | 164 |
| abstract_inverted_index.DPO, | 148 |
| abstract_inverted_index.LIAE | 100 |
| abstract_inverted_index.both | 174 |
| abstract_inverted_index.fail | 65 |
| abstract_inverted_index.like | 10 |
| abstract_inverted_index.show | 62, 159 |
| abstract_inverted_index.text | 24 |
| abstract_inverted_index.that | 99, 109, 129 |
| abstract_inverted_index.this | 33, 58 |
| abstract_inverted_index.with | 26, 137 |
| abstract_inverted_index.work | 31 |
| abstract_inverted_index.(IAF) | 89 |
| abstract_inverted_index.Focus | 88 |
| abstract_inverted_index.Image | 77, 86 |
| abstract_inverted_index.Large | 1 |
| abstract_inverted_index.Layer | 76 |
| abstract_inverted_index.Prior | 30 |
| abstract_inverted_index.Then, | 71 |
| abstract_inverted_index.apply | 46 |
| abstract_inverted_index.first | 61 |
| abstract_inverted_index.flags | 82 |
| abstract_inverted_index.heads | 93, 108 |
| abstract_inverted_index.image | 11 |
| abstract_inverted_index.links | 32 |
| abstract_inverted_index.prone | 19 |
| abstract_inverted_index.ranks | 107 |
| abstract_inverted_index.shows | 98 |
| abstract_inverted_index.tasks | 9 |
| abstract_inverted_index.these | 63, 114 |
| abstract_inverted_index.those | 95 |
| abstract_inverted_index.where | 22, 54 |
| abstract_inverted_index.which | 81, 90 |
| abstract_inverted_index.while | 185 |
| abstract_inverted_index.(LIAE) | 80 |
| abstract_inverted_index.D-LEAF | 146, 161 |
| abstract_inverted_index.Fusion | 123 |
| abstract_inverted_index.Guided | 112 |
| abstract_inverted_index.Models | 3 |
| abstract_inverted_index.across | 49 |
| abstract_inverted_index.during | 135 |
| abstract_inverted_index.errors | 55, 134 |
| abstract_inverted_index.faulty | 102 |
| abstract_inverted_index.heads, | 52 |
| abstract_inverted_index.input. | 29 |
| abstract_inverted_index.layers | 50, 103 |
| abstract_inverted_index.method | 128 |
| abstract_inverted_index.paper, | 59 |
| abstract_inverted_index.partly | 34 |
| abstract_inverted_index.remain | 18 |
| abstract_inverted_index.scores | 91 |
| abstract_inverted_index.strong | 6 |
| abstract_inverted_index.visual | 14, 28, 37 |
| abstract_inverted_index.within | 94 |
| abstract_inverted_index.(MLLMs) | 4 |
| abstract_inverted_index.D-LEAF. | 157 |
| abstract_inverted_index.Dynamic | 118 |
| abstract_inverted_index.Entropy | 79, 120 |
| abstract_inverted_index.Results | 158 |
| abstract_inverted_index.achieve | 5 |
| abstract_inverted_index.between | 145 |
| abstract_inverted_index.improve | 178 |
| abstract_inverted_index.layers, | 84 |
| abstract_inverted_index.layers. | 70, 96 |
| abstract_inverted_index.methods | 64 |
| abstract_inverted_index.propose | 117 |
| abstract_inverted_index.provide | 150 |
| abstract_inverted_index.uniform | 47 |
| abstract_inverted_index.warrant | 110 |
| abstract_inverted_index.Analysis | 97 |
| abstract_inverted_index.F1-score | 177 |
| abstract_inverted_index.Language | 2 |
| abstract_inverted_index.accuracy | 175 |
| abstract_inverted_index.corrects | 133 |
| abstract_inverted_index.delivers | 162 |
| abstract_inverted_index.existing | 40 |
| abstract_inverted_index.localize | 68 |
| abstract_inverted_index.question | 15 |
| abstract_inverted_index.relative | 165 |
| abstract_inverted_index.reliably | 106 |
| abstract_inverted_index.signals, | 115 |
| abstract_inverted_index.standard | 168 |
| abstract_inverted_index.(D-LEAF), | 124 |
| abstract_inverted_index.Attention | 78, 87, 122 |
| abstract_inverted_index.anomalous | 83 |
| abstract_inverted_index.attention | 92 |
| abstract_inverted_index.conflicts | 25 |
| abstract_inverted_index.detectors | 42 |
| abstract_inverted_index.generated | 23 |
| abstract_inverted_index.inference | 136 |
| abstract_inverted_index.introduce | 73 |
| abstract_inverted_index.localizes | 131 |
| abstract_inverted_index.obscuring | 53 |
| abstract_inverted_index.overhead. | 139 |
| abstract_inverted_index.pinpoints | 101 |
| abstract_inverted_index.typically | 45 |
| abstract_inverted_index.Layer-wise | 119 |
| abstract_inverted_index.Multimodal | 0 |
| abstract_inverted_index.accurately | 67 |
| abstract_inverted_index.answering, | 16 |
| abstract_inverted_index.attention, | 38 |
| abstract_inverted_index.captioning | 12, 169 |
| abstract_inverted_index.connection | 144 |
| abstract_inverted_index.mitigation | 44 |
| abstract_inverted_index.negligible | 138 |
| abstract_inverted_index.originate. | 56 |
| abstract_inverted_index.preserving | 186 |
| abstract_inverted_index.adjustments | 48 |
| abstract_inverted_index.benchmarks, | 170 |
| abstract_inverted_index.correction. | 111 |
| abstract_inverted_index.dynamically | 130 |
| abstract_inverted_index.efficiency. | 187 |
| abstract_inverted_index.improvement | 166 |
| abstract_inverted_index.performance | 7 |
| abstract_inverted_index.problematic | 69 |
| abstract_inverted_index.suppressing | 183 |
| abstract_inverted_index.theoretical | 151 |
| abstract_inverted_index.Furthermore, | 140 |
| abstract_inverted_index.diagnostics: | 75 |
| abstract_inverted_index.establishing | 142 |
| abstract_inverted_index.insufficient | 36 |
| abstract_inverted_index.approximately | 180 |
| abstract_inverted_index.effectiveness | 155 |
| abstract_inverted_index.justification | 152 |
| abstract_inverted_index.substantially | 182 |
| abstract_inverted_index.hallucinations | 184 |
| abstract_inverted_index.task-agnostic, | 126 |
| abstract_inverted_index.attention-based | 41 |
| abstract_inverted_index.hallucinations, | 21 |
| abstract_inverted_index.attention-guided | 127 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 3 |
| citation_normalized_percentile |