IPA: An Information-Reconstructive Input Projection Framework for Efficient Foundation Model Adaptation Article Swipe
Tuan-Hung Vu
,
Andrei Bursuc
,
Matthieu Cord
·
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2509.04398
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2509.04398
Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, reduce adaptation cost by injecting low-rank updates into pretrained weights. However, LoRA's down-projection is randomly initialized and data-agnostic, discarding potentially useful information. Prior analyses show that this projection changes little during training, while the up-projection carries most of the adaptation, making the random input compression a performance bottleneck. We propose IPA, a feature-aware projection framework that explicitly aims to reconstruct the original input within a reduced hidden space. In the linear case, we instantiate IPA with algorithms approximating top principal components, enabling efficient projector pretraining with negligible inference overhead. Across language and vision benchmarks, IPA consistently improves over LoRA and DoRA, achieving on average 1.5 points higher accuracy on commonsense reasoning and 2.3 points on VTAB-1k, while matching full LoRA performance with roughly half the trainable parameters when the projection is frozen. Code available at https://github.com/valeoai/peft-ipa .
Related Topics
Concepts
No concepts available.
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2509.04398
- https://arxiv.org/pdf/2509.04398
- OA Status
- green
- OpenAlex ID
- https://openalex.org/W4417145600
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4417145600Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2509.04398Digital Object Identifier
- Title
-
IPA: An Information-Reconstructive Input Projection Framework for Efficient Foundation Model AdaptationWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2025Year of publication
- Publication date
-
2025-09-04Full publication date if available
- Authors
-
Tuan-Hung Vu, Andrei Bursuc, Matthieu CordList of authors in order
- Landing page
-
https://arxiv.org/abs/2509.04398Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2509.04398Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2509.04398Direct OA link when available
- Cited by
-
0Total citation count in OpenAlex
Full payload
| id | https://openalex.org/W4417145600 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2509.04398 |
| ids.doi | https://doi.org/10.48550/arxiv.2509.04398 |
| ids.openalex | https://openalex.org/W4417145600 |
| fwci | |
| type | preprint |
| title | IPA: An Information-Reconstructive Input Projection Framework for Efficient Foundation Model Adaptation |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2509.04398 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2509.04398 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2509.04398 |
| locations[1].id | doi:10.48550/arxiv.2509.04398 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2509.04398 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5102828738 |
| authorships[0].author.orcid | https://orcid.org/0000-0002-9765-8233 |
| authorships[0].author.display_name | Tuan-Hung Vu |
| authorships[0].author_position | last |
| authorships[0].raw_author_name | Vu, Tuan-Hung |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5003783466 |
| authorships[1].author.orcid | https://orcid.org/0000-0003-2485-9402 |
| authorships[1].author.display_name | Andrei Bursuc |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Bursuc, Andrei |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5108118084 |
| authorships[2].author.orcid | https://orcid.org/0000-0002-0627-5844 |
| authorships[2].author.display_name | Matthieu Cord |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Cord, Matthieu |
| authorships[2].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2509.04398 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | IPA: An Information-Reconstructive Input Projection Framework for Efficient Foundation Model Adaptation |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-12-10T02:49:46.989445 |
| primary_topic | |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2509.04398 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2509.04398 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2509.04398 |
| primary_location.id | pmh:oai:arXiv.org:2509.04398 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2509.04398 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2509.04398 |
| publication_date | 2025-09-04 |
| publication_year | 2025 |
| referenced_works_count | 0 |
| abstract_inverted_index.. | 143 |
| abstract_inverted_index.a | 52, 58, 71 |
| abstract_inverted_index.In | 75 |
| abstract_inverted_index.We | 55 |
| abstract_inverted_index.as | 5 |
| abstract_inverted_index.at | 141 |
| abstract_inverted_index.by | 10 |
| abstract_inverted_index.is | 20, 137 |
| abstract_inverted_index.of | 44 |
| abstract_inverted_index.on | 109, 115, 121 |
| abstract_inverted_index.to | 65 |
| abstract_inverted_index.we | 79 |
| abstract_inverted_index.1.5 | 111 |
| abstract_inverted_index.2.3 | 119 |
| abstract_inverted_index.IPA | 81, 101 |
| abstract_inverted_index.and | 23, 98, 106, 118 |
| abstract_inverted_index.the | 40, 45, 48, 67, 76, 131, 135 |
| abstract_inverted_index.top | 85 |
| abstract_inverted_index.Code | 139 |
| abstract_inverted_index.IPA, | 57 |
| abstract_inverted_index.LoRA | 105, 126 |
| abstract_inverted_index.aims | 64 |
| abstract_inverted_index.cost | 9 |
| abstract_inverted_index.full | 125 |
| abstract_inverted_index.half | 130 |
| abstract_inverted_index.into | 14 |
| abstract_inverted_index.most | 43 |
| abstract_inverted_index.over | 104 |
| abstract_inverted_index.show | 31 |
| abstract_inverted_index.such | 4 |
| abstract_inverted_index.that | 32, 62 |
| abstract_inverted_index.this | 33 |
| abstract_inverted_index.when | 134 |
| abstract_inverted_index.with | 82, 92, 128 |
| abstract_inverted_index.DoRA, | 107 |
| abstract_inverted_index.LoRA, | 6 |
| abstract_inverted_index.Prior | 29 |
| abstract_inverted_index.case, | 78 |
| abstract_inverted_index.input | 50, 69 |
| abstract_inverted_index.while | 39, 123 |
| abstract_inverted_index.(PEFT) | 2 |
| abstract_inverted_index.Across | 96 |
| abstract_inverted_index.LoRA's | 18 |
| abstract_inverted_index.during | 37 |
| abstract_inverted_index.hidden | 73 |
| abstract_inverted_index.higher | 113 |
| abstract_inverted_index.linear | 77 |
| abstract_inverted_index.little | 36 |
| abstract_inverted_index.making | 47 |
| abstract_inverted_index.points | 112, 120 |
| abstract_inverted_index.random | 49 |
| abstract_inverted_index.reduce | 7 |
| abstract_inverted_index.space. | 74 |
| abstract_inverted_index.useful | 27 |
| abstract_inverted_index.vision | 99 |
| abstract_inverted_index.within | 70 |
| abstract_inverted_index.average | 110 |
| abstract_inverted_index.carries | 42 |
| abstract_inverted_index.changes | 35 |
| abstract_inverted_index.frozen. | 138 |
| abstract_inverted_index.propose | 56 |
| abstract_inverted_index.reduced | 72 |
| abstract_inverted_index.roughly | 129 |
| abstract_inverted_index.updates | 13 |
| abstract_inverted_index.However, | 17 |
| abstract_inverted_index.VTAB-1k, | 122 |
| abstract_inverted_index.accuracy | 114 |
| abstract_inverted_index.analyses | 30 |
| abstract_inverted_index.enabling | 88 |
| abstract_inverted_index.improves | 103 |
| abstract_inverted_index.language | 97 |
| abstract_inverted_index.low-rank | 12 |
| abstract_inverted_index.matching | 124 |
| abstract_inverted_index.methods, | 3 |
| abstract_inverted_index.original | 68 |
| abstract_inverted_index.randomly | 21 |
| abstract_inverted_index.weights. | 16 |
| abstract_inverted_index.achieving | 108 |
| abstract_inverted_index.available | 140 |
| abstract_inverted_index.efficient | 89 |
| abstract_inverted_index.framework | 61 |
| abstract_inverted_index.inference | 94 |
| abstract_inverted_index.injecting | 11 |
| abstract_inverted_index.overhead. | 95 |
| abstract_inverted_index.principal | 86 |
| abstract_inverted_index.projector | 90 |
| abstract_inverted_index.reasoning | 117 |
| abstract_inverted_index.trainable | 132 |
| abstract_inverted_index.training, | 38 |
| abstract_inverted_index.adaptation | 8 |
| abstract_inverted_index.algorithms | 83 |
| abstract_inverted_index.discarding | 25 |
| abstract_inverted_index.explicitly | 63 |
| abstract_inverted_index.negligible | 93 |
| abstract_inverted_index.parameters | 133 |
| abstract_inverted_index.pretrained | 15 |
| abstract_inverted_index.projection | 34, 60, 136 |
| abstract_inverted_index.adaptation, | 46 |
| abstract_inverted_index.benchmarks, | 100 |
| abstract_inverted_index.bottleneck. | 54 |
| abstract_inverted_index.commonsense | 116 |
| abstract_inverted_index.components, | 87 |
| abstract_inverted_index.compression | 51 |
| abstract_inverted_index.fine-tuning | 1 |
| abstract_inverted_index.initialized | 22 |
| abstract_inverted_index.instantiate | 80 |
| abstract_inverted_index.performance | 53, 127 |
| abstract_inverted_index.potentially | 26 |
| abstract_inverted_index.pretraining | 91 |
| abstract_inverted_index.reconstruct | 66 |
| abstract_inverted_index.consistently | 102 |
| abstract_inverted_index.information. | 28 |
| abstract_inverted_index.approximating | 84 |
| abstract_inverted_index.feature-aware | 59 |
| abstract_inverted_index.up-projection | 41 |
| abstract_inverted_index.data-agnostic, | 24 |
| abstract_inverted_index.down-projection | 19 |
| abstract_inverted_index.Parameter-efficient | 0 |
| abstract_inverted_index.https://github.com/valeoai/peft-ipa | 142 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 3 |
| citation_normalized_percentile |