Beyond Anti-Forgetting: Multimodal Continual Instruction Tuning with Positive Forward Transfer Article Swipe
Junhao Zheng
,
Qianli Ma
,
Zhen Liu
,
Binquan Wu
,
Huawen Feng
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2401.09181
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2401.09181
Multimodal Continual Instruction Tuning (MCIT) enables Multimodal Large Language Models (MLLMs) to meet continuously emerging requirements without expensive retraining. MCIT faces two major obstacles: catastrophic forgetting (where old knowledge is forgotten) and negative forward transfer (where the performance of future tasks is degraded). Although existing methods have greatly alleviated catastrophic forgetting, they still suffer from negative forward transfer. We discover a large discrepancy in different input embeddings by performing singular value decomposition (SVD) on input embeddings. This discrepancy results in the model learning irrelevant information for old and pre-trained tasks, leading to catastrophic forgetting and negative forward transfer. To address these issues, we propose Prompt Tuning with Positive Forward Transfer (Fwd-Prompt), a prompt-based method that projects the prompt gradient to the residual space to minimize interference between tasks and to the pre-trained subspace for reusing pre-trained knowledge. Our experiments demonstrate that Fwd-Prompt achieves state-of-the-art performance while updating fewer parameters and requiring no old samples. Our research illuminates the potential of continuously adapting MLLMs to new tasks under the instruction tuning paradigm and encourages future studies to explore MCIT.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2401.09181
- https://arxiv.org/pdf/2401.09181
- OA Status
- green
- Cited By
- 2
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4391013432
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4391013432Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2401.09181Digital Object Identifier
- Title
-
Beyond Anti-Forgetting: Multimodal Continual Instruction Tuning with Positive Forward TransferWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-01-17Full publication date if available
- Authors
-
Junhao Zheng, Qianli Ma, Zhen Liu, Binquan Wu, Huawen FengList of authors in order
- Landing page
-
https://arxiv.org/abs/2401.09181Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2401.09181Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2401.09181Direct OA link when available
- Concepts
-
Forgetting, Computer science, Retraining, Artificial intelligence, Subspace topology, Transfer of learning, Machine learning, Cognitive psychology, Psychology, International trade, BusinessTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
2Total citation count in OpenAlex
- Citations by year (recent)
-
2025: 1, 2024: 1Per-year citation counts (last 5 years)
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4391013432 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2401.09181 |
| ids.doi | https://doi.org/10.48550/arxiv.2401.09181 |
| ids.openalex | https://openalex.org/W4391013432 |
| fwci | |
| type | preprint |
| title | Beyond Anti-Forgetting: Multimodal Continual Instruction Tuning with Positive Forward Transfer |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T11714 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9940999746322632 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1707 |
| topics[0].subfield.display_name | Computer Vision and Pattern Recognition |
| topics[0].display_name | Multimodal Machine Learning Applications |
| topics[1].id | https://openalex.org/T10028 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9840999841690063 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Topic Modeling |
| topics[2].id | https://openalex.org/T11307 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9828000068664551 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1702 |
| topics[2].subfield.display_name | Artificial Intelligence |
| topics[2].display_name | Domain Adaptation and Few-Shot Learning |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C7149132 |
| concepts[0].level | 2 |
| concepts[0].score | 0.8524227142333984 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q1377840 |
| concepts[0].display_name | Forgetting |
| concepts[1].id | https://openalex.org/C41008148 |
| concepts[1].level | 0 |
| concepts[1].score | 0.7356899976730347 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[1].display_name | Computer science |
| concepts[2].id | https://openalex.org/C2778712577 |
| concepts[2].level | 2 |
| concepts[2].score | 0.5452386736869812 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q3505966 |
| concepts[2].display_name | Retraining |
| concepts[3].id | https://openalex.org/C154945302 |
| concepts[3].level | 1 |
| concepts[3].score | 0.46845221519470215 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[3].display_name | Artificial intelligence |
| concepts[4].id | https://openalex.org/C32834561 |
| concepts[4].level | 2 |
| concepts[4].score | 0.4558213949203491 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q660730 |
| concepts[4].display_name | Subspace topology |
| concepts[5].id | https://openalex.org/C150899416 |
| concepts[5].level | 2 |
| concepts[5].score | 0.43466565012931824 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q1820378 |
| concepts[5].display_name | Transfer of learning |
| concepts[6].id | https://openalex.org/C119857082 |
| concepts[6].level | 1 |
| concepts[6].score | 0.35135066509246826 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q2539 |
| concepts[6].display_name | Machine learning |
| concepts[7].id | https://openalex.org/C180747234 |
| concepts[7].level | 1 |
| concepts[7].score | 0.11682116985321045 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q23373 |
| concepts[7].display_name | Cognitive psychology |
| concepts[8].id | https://openalex.org/C15744967 |
| concepts[8].level | 0 |
| concepts[8].score | 0.06581801176071167 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q9418 |
| concepts[8].display_name | Psychology |
| concepts[9].id | https://openalex.org/C155202549 |
| concepts[9].level | 1 |
| concepts[9].score | 0.0 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q178803 |
| concepts[9].display_name | International trade |
| concepts[10].id | https://openalex.org/C144133560 |
| concepts[10].level | 0 |
| concepts[10].score | 0.0 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q4830453 |
| concepts[10].display_name | Business |
| keywords[0].id | https://openalex.org/keywords/forgetting |
| keywords[0].score | 0.8524227142333984 |
| keywords[0].display_name | Forgetting |
| keywords[1].id | https://openalex.org/keywords/computer-science |
| keywords[1].score | 0.7356899976730347 |
| keywords[1].display_name | Computer science |
| keywords[2].id | https://openalex.org/keywords/retraining |
| keywords[2].score | 0.5452386736869812 |
| keywords[2].display_name | Retraining |
| keywords[3].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[3].score | 0.46845221519470215 |
| keywords[3].display_name | Artificial intelligence |
| keywords[4].id | https://openalex.org/keywords/subspace-topology |
| keywords[4].score | 0.4558213949203491 |
| keywords[4].display_name | Subspace topology |
| keywords[5].id | https://openalex.org/keywords/transfer-of-learning |
| keywords[5].score | 0.43466565012931824 |
| keywords[5].display_name | Transfer of learning |
| keywords[6].id | https://openalex.org/keywords/machine-learning |
| keywords[6].score | 0.35135066509246826 |
| keywords[6].display_name | Machine learning |
| keywords[7].id | https://openalex.org/keywords/cognitive-psychology |
| keywords[7].score | 0.11682116985321045 |
| keywords[7].display_name | Cognitive psychology |
| keywords[8].id | https://openalex.org/keywords/psychology |
| keywords[8].score | 0.06581801176071167 |
| keywords[8].display_name | Psychology |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2401.09181 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2401.09181 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2401.09181 |
| locations[1].id | doi:10.48550/arxiv.2401.09181 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | cc-by |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | https://openalex.org/licenses/cc-by |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2401.09181 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5101773557 |
| authorships[0].author.orcid | https://orcid.org/0000-0001-9124-2467 |
| authorships[0].author.display_name | Junhao Zheng |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Zheng, Junhao |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5076609933 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-9356-2883 |
| authorships[1].author.display_name | Qianli Ma |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Ma, Qianli |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5003356618 |
| authorships[2].author.orcid | https://orcid.org/0000-0002-8107-0929 |
| authorships[2].author.display_name | Zhen Liu |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Liu, Zhen |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5079609030 |
| authorships[3].author.orcid | https://orcid.org/0009-0008-5031-8339 |
| authorships[3].author.display_name | Binquan Wu |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Wu, Binquan |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5104245194 |
| authorships[4].author.orcid | |
| authorships[4].author.display_name | Huawen Feng |
| authorships[4].author_position | last |
| authorships[4].raw_author_name | Feng, Huawen |
| authorships[4].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2401.09181 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2024-01-19T00:00:00 |
| display_name | Beyond Anti-Forgetting: Multimodal Continual Instruction Tuning with Positive Forward Transfer |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T11714 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9940999746322632 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1707 |
| primary_topic.subfield.display_name | Computer Vision and Pattern Recognition |
| primary_topic.display_name | Multimodal Machine Learning Applications |
| related_works | https://openalex.org/W4289718052, https://openalex.org/W2164121020, https://openalex.org/W2081982437, https://openalex.org/W2145559838, https://openalex.org/W2027050655, https://openalex.org/W2905319430, https://openalex.org/W3116498279, https://openalex.org/W3028244590, https://openalex.org/W4254349500, https://openalex.org/W4287549553 |
| cited_by_count | 2 |
| counts_by_year[0].year | 2025 |
| counts_by_year[0].cited_by_count | 1 |
| counts_by_year[1].year | 2024 |
| counts_by_year[1].cited_by_count | 1 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2401.09181 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2401.09181 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2401.09181 |
| primary_location.id | pmh:oai:arXiv.org:2401.09181 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2401.09181 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2401.09181 |
| publication_date | 2024-01-17 |
| publication_year | 2024 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 60, 111 |
| abstract_inverted_index.To | 98 |
| abstract_inverted_index.We | 58 |
| abstract_inverted_index.by | 67 |
| abstract_inverted_index.in | 63, 79 |
| abstract_inverted_index.is | 29, 41 |
| abstract_inverted_index.no | 151 |
| abstract_inverted_index.of | 38, 159 |
| abstract_inverted_index.on | 73 |
| abstract_inverted_index.to | 11, 91, 119, 123, 129, 163, 175 |
| abstract_inverted_index.we | 102 |
| abstract_inverted_index.Our | 137, 154 |
| abstract_inverted_index.and | 31, 87, 94, 128, 149, 171 |
| abstract_inverted_index.for | 85, 133 |
| abstract_inverted_index.new | 164 |
| abstract_inverted_index.old | 27, 86, 152 |
| abstract_inverted_index.the | 36, 80, 116, 120, 130, 157, 167 |
| abstract_inverted_index.two | 21 |
| abstract_inverted_index.MCIT | 19 |
| abstract_inverted_index.This | 76 |
| abstract_inverted_index.from | 54 |
| abstract_inverted_index.have | 46 |
| abstract_inverted_index.meet | 12 |
| abstract_inverted_index.that | 114, 140 |
| abstract_inverted_index.they | 51 |
| abstract_inverted_index.with | 106 |
| abstract_inverted_index.(SVD) | 72 |
| abstract_inverted_index.Large | 7 |
| abstract_inverted_index.MCIT. | 177 |
| abstract_inverted_index.MLLMs | 162 |
| abstract_inverted_index.faces | 20 |
| abstract_inverted_index.fewer | 147 |
| abstract_inverted_index.input | 65, 74 |
| abstract_inverted_index.large | 61 |
| abstract_inverted_index.major | 22 |
| abstract_inverted_index.model | 81 |
| abstract_inverted_index.space | 122 |
| abstract_inverted_index.still | 52 |
| abstract_inverted_index.tasks | 40, 127, 165 |
| abstract_inverted_index.these | 100 |
| abstract_inverted_index.under | 166 |
| abstract_inverted_index.value | 70 |
| abstract_inverted_index.while | 145 |
| abstract_inverted_index.(MCIT) | 4 |
| abstract_inverted_index.(where | 26, 35 |
| abstract_inverted_index.Models | 9 |
| abstract_inverted_index.Prompt | 104 |
| abstract_inverted_index.Tuning | 3, 105 |
| abstract_inverted_index.future | 39, 173 |
| abstract_inverted_index.method | 113 |
| abstract_inverted_index.prompt | 117 |
| abstract_inverted_index.suffer | 53 |
| abstract_inverted_index.tasks, | 89 |
| abstract_inverted_index.tuning | 169 |
| abstract_inverted_index.(MLLMs) | 10 |
| abstract_inverted_index.Forward | 108 |
| abstract_inverted_index.address | 99 |
| abstract_inverted_index.between | 126 |
| abstract_inverted_index.enables | 5 |
| abstract_inverted_index.explore | 176 |
| abstract_inverted_index.forward | 33, 56, 96 |
| abstract_inverted_index.greatly | 47 |
| abstract_inverted_index.issues, | 101 |
| abstract_inverted_index.leading | 90 |
| abstract_inverted_index.methods | 45 |
| abstract_inverted_index.propose | 103 |
| abstract_inverted_index.results | 78 |
| abstract_inverted_index.reusing | 134 |
| abstract_inverted_index.studies | 174 |
| abstract_inverted_index.without | 16 |
| abstract_inverted_index.Although | 43 |
| abstract_inverted_index.Language | 8 |
| abstract_inverted_index.Positive | 107 |
| abstract_inverted_index.Transfer | 109 |
| abstract_inverted_index.achieves | 142 |
| abstract_inverted_index.adapting | 161 |
| abstract_inverted_index.discover | 59 |
| abstract_inverted_index.emerging | 14 |
| abstract_inverted_index.existing | 44 |
| abstract_inverted_index.gradient | 118 |
| abstract_inverted_index.learning | 82 |
| abstract_inverted_index.minimize | 124 |
| abstract_inverted_index.negative | 32, 55, 95 |
| abstract_inverted_index.paradigm | 170 |
| abstract_inverted_index.projects | 115 |
| abstract_inverted_index.research | 155 |
| abstract_inverted_index.residual | 121 |
| abstract_inverted_index.samples. | 153 |
| abstract_inverted_index.singular | 69 |
| abstract_inverted_index.subspace | 132 |
| abstract_inverted_index.transfer | 34 |
| abstract_inverted_index.updating | 146 |
| abstract_inverted_index.Continual | 1 |
| abstract_inverted_index.different | 64 |
| abstract_inverted_index.expensive | 17 |
| abstract_inverted_index.knowledge | 28 |
| abstract_inverted_index.potential | 158 |
| abstract_inverted_index.requiring | 150 |
| abstract_inverted_index.transfer. | 57, 97 |
| abstract_inverted_index.Fwd-Prompt | 141 |
| abstract_inverted_index.Multimodal | 0, 6 |
| abstract_inverted_index.alleviated | 48 |
| abstract_inverted_index.degraded). | 42 |
| abstract_inverted_index.embeddings | 66 |
| abstract_inverted_index.encourages | 172 |
| abstract_inverted_index.forgetting | 25, 93 |
| abstract_inverted_index.forgotten) | 30 |
| abstract_inverted_index.irrelevant | 83 |
| abstract_inverted_index.knowledge. | 136 |
| abstract_inverted_index.obstacles: | 23 |
| abstract_inverted_index.parameters | 148 |
| abstract_inverted_index.performing | 68 |
| abstract_inverted_index.Instruction | 2 |
| abstract_inverted_index.demonstrate | 139 |
| abstract_inverted_index.discrepancy | 62, 77 |
| abstract_inverted_index.embeddings. | 75 |
| abstract_inverted_index.experiments | 138 |
| abstract_inverted_index.forgetting, | 50 |
| abstract_inverted_index.illuminates | 156 |
| abstract_inverted_index.information | 84 |
| abstract_inverted_index.instruction | 168 |
| abstract_inverted_index.performance | 37, 144 |
| abstract_inverted_index.pre-trained | 88, 131, 135 |
| abstract_inverted_index.retraining. | 18 |
| abstract_inverted_index.catastrophic | 24, 49, 92 |
| abstract_inverted_index.continuously | 13, 160 |
| abstract_inverted_index.interference | 125 |
| abstract_inverted_index.prompt-based | 112 |
| abstract_inverted_index.requirements | 15 |
| abstract_inverted_index.(Fwd-Prompt), | 110 |
| abstract_inverted_index.decomposition | 71 |
| abstract_inverted_index.state-of-the-art | 143 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 5 |
| citation_normalized_percentile |