Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization Article Swipe
Alexandra Chronopoulou
,
Jonas Pfeiffer
,
Joshua Maynez
,
Xinyi Wang
,
Sebastian Ruder
,
Priyanka Agrawal
·
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2311.09344
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2311.09344
Parameter-efficient fine-tuning (PEFT) using labeled task data can significantly improve the performance of large language models (LLMs) on the downstream task. However, there are 7000 languages in the world and many of these languages lack labeled data for real-world language generation tasks. In this paper, we propose to improve zero-shot cross-lingual transfer by composing language or task specialized parameters. Our method composes language and task PEFT modules via element-wise arithmetic operations to leverage unlabeled data and English labeled data. We extend our approach to cases where labeled data from more languages is available and propose to arithmetically compose PEFT modules trained on languages related to the target. Empirical results on summarization demonstrate that our method is an effective strategy that obtains consistent gains using minimal training of PEFT modules.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2311.09344
- https://arxiv.org/pdf/2311.09344
- OA Status
- green
- Cited By
- 1
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4388787436
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4388787436Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2311.09344Digital Object Identifier
- Title
-
Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot SummarizationWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2023Year of publication
- Publication date
-
2023-11-15Full publication date if available
- Authors
-
Alexandra Chronopoulou, Jonas Pfeiffer, Joshua Maynez, Xinyi Wang, Sebastian Ruder, Priyanka AgrawalList of authors in order
- Landing page
-
https://arxiv.org/abs/2311.09344Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2311.09344Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2311.09344Direct OA link when available
- Concepts
-
Automatic summarization, Leverage (statistics), Computer science, Task (project management), Zero (linguistics), Natural language processing, Labeled data, Artificial intelligence, Arithmetic, Mathematics, Linguistics, Engineering, Systems engineering, PhilosophyTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
1Total citation count in OpenAlex
- Citations by year (recent)
-
2024: 1Per-year citation counts (last 5 years)
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4388787436 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2311.09344 |
| ids.doi | https://doi.org/10.48550/arxiv.2311.09344 |
| ids.openalex | https://openalex.org/W4388787436 |
| fwci | |
| type | preprint |
| title | Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10028 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9991000294685364 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Topic Modeling |
| topics[1].id | https://openalex.org/T10181 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9980999827384949 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Natural Language Processing Techniques |
| topics[2].id | https://openalex.org/T11714 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9901000261306763 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1707 |
| topics[2].subfield.display_name | Computer Vision and Pattern Recognition |
| topics[2].display_name | Multimodal Machine Learning Applications |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C170858558 |
| concepts[0].level | 2 |
| concepts[0].score | 0.8570469617843628 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q1394144 |
| concepts[0].display_name | Automatic summarization |
| concepts[1].id | https://openalex.org/C153083717 |
| concepts[1].level | 2 |
| concepts[1].score | 0.7869893312454224 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q6535263 |
| concepts[1].display_name | Leverage (statistics) |
| concepts[2].id | https://openalex.org/C41008148 |
| concepts[2].level | 0 |
| concepts[2].score | 0.773032009601593 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[2].display_name | Computer science |
| concepts[3].id | https://openalex.org/C2780451532 |
| concepts[3].level | 2 |
| concepts[3].score | 0.7098715305328369 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q759676 |
| concepts[3].display_name | Task (project management) |
| concepts[4].id | https://openalex.org/C2780813799 |
| concepts[4].level | 2 |
| concepts[4].score | 0.6019411683082581 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q3274237 |
| concepts[4].display_name | Zero (linguistics) |
| concepts[5].id | https://openalex.org/C204321447 |
| concepts[5].level | 1 |
| concepts[5].score | 0.5861400365829468 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q30642 |
| concepts[5].display_name | Natural language processing |
| concepts[6].id | https://openalex.org/C2776145971 |
| concepts[6].level | 2 |
| concepts[6].score | 0.5421491861343384 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q30673951 |
| concepts[6].display_name | Labeled data |
| concepts[7].id | https://openalex.org/C154945302 |
| concepts[7].level | 1 |
| concepts[7].score | 0.5180649757385254 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[7].display_name | Artificial intelligence |
| concepts[8].id | https://openalex.org/C94375191 |
| concepts[8].level | 1 |
| concepts[8].score | 0.35677117109298706 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q11205 |
| concepts[8].display_name | Arithmetic |
| concepts[9].id | https://openalex.org/C33923547 |
| concepts[9].level | 0 |
| concepts[9].score | 0.13296720385551453 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q395 |
| concepts[9].display_name | Mathematics |
| concepts[10].id | https://openalex.org/C41895202 |
| concepts[10].level | 1 |
| concepts[10].score | 0.11421522498130798 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q8162 |
| concepts[10].display_name | Linguistics |
| concepts[11].id | https://openalex.org/C127413603 |
| concepts[11].level | 0 |
| concepts[11].score | 0.06350848078727722 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q11023 |
| concepts[11].display_name | Engineering |
| concepts[12].id | https://openalex.org/C201995342 |
| concepts[12].level | 1 |
| concepts[12].score | 0.0 |
| concepts[12].wikidata | https://www.wikidata.org/wiki/Q682496 |
| concepts[12].display_name | Systems engineering |
| concepts[13].id | https://openalex.org/C138885662 |
| concepts[13].level | 0 |
| concepts[13].score | 0.0 |
| concepts[13].wikidata | https://www.wikidata.org/wiki/Q5891 |
| concepts[13].display_name | Philosophy |
| keywords[0].id | https://openalex.org/keywords/automatic-summarization |
| keywords[0].score | 0.8570469617843628 |
| keywords[0].display_name | Automatic summarization |
| keywords[1].id | https://openalex.org/keywords/leverage |
| keywords[1].score | 0.7869893312454224 |
| keywords[1].display_name | Leverage (statistics) |
| keywords[2].id | https://openalex.org/keywords/computer-science |
| keywords[2].score | 0.773032009601593 |
| keywords[2].display_name | Computer science |
| keywords[3].id | https://openalex.org/keywords/task |
| keywords[3].score | 0.7098715305328369 |
| keywords[3].display_name | Task (project management) |
| keywords[4].id | https://openalex.org/keywords/zero |
| keywords[4].score | 0.6019411683082581 |
| keywords[4].display_name | Zero (linguistics) |
| keywords[5].id | https://openalex.org/keywords/natural-language-processing |
| keywords[5].score | 0.5861400365829468 |
| keywords[5].display_name | Natural language processing |
| keywords[6].id | https://openalex.org/keywords/labeled-data |
| keywords[6].score | 0.5421491861343384 |
| keywords[6].display_name | Labeled data |
| keywords[7].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[7].score | 0.5180649757385254 |
| keywords[7].display_name | Artificial intelligence |
| keywords[8].id | https://openalex.org/keywords/arithmetic |
| keywords[8].score | 0.35677117109298706 |
| keywords[8].display_name | Arithmetic |
| keywords[9].id | https://openalex.org/keywords/mathematics |
| keywords[9].score | 0.13296720385551453 |
| keywords[9].display_name | Mathematics |
| keywords[10].id | https://openalex.org/keywords/linguistics |
| keywords[10].score | 0.11421522498130798 |
| keywords[10].display_name | Linguistics |
| keywords[11].id | https://openalex.org/keywords/engineering |
| keywords[11].score | 0.06350848078727722 |
| keywords[11].display_name | Engineering |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2311.09344 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2311.09344 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2311.09344 |
| locations[1].id | doi:10.48550/arxiv.2311.09344 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2311.09344 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5067446532 |
| authorships[0].author.orcid | |
| authorships[0].author.display_name | Alexandra Chronopoulou |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Chronopoulou, Alexandra |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5024983536 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-8634-6170 |
| authorships[1].author.display_name | Jonas Pfeiffer |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Pfeiffer, Jonas |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5021334672 |
| authorships[2].author.orcid | https://orcid.org/0000-0003-4948-2875 |
| authorships[2].author.display_name | Joshua Maynez |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Maynez, Joshua |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5100382950 |
| authorships[3].author.orcid | https://orcid.org/0000-0002-6280-1300 |
| authorships[3].author.display_name | Xinyi Wang |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Wang, Xinyi |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5037310413 |
| authorships[4].author.orcid | |
| authorships[4].author.display_name | Sebastian Ruder |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Ruder, Sebastian |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5047179830 |
| authorships[5].author.orcid | https://orcid.org/0000-0001-9894-9625 |
| authorships[5].author.display_name | Priyanka Agrawal |
| authorships[5].author_position | last |
| authorships[5].raw_author_name | Agrawal, Priyanka |
| authorships[5].is_corresponding | False |
| has_content.pdf | True |
| has_content.grobid_xml | True |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2311.09344 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization |
| has_fulltext | True |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10028 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9991000294685364 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Topic Modeling |
| related_works | https://openalex.org/W2366403280, https://openalex.org/W1495108544, https://openalex.org/W4289143854, https://openalex.org/W2129767422, https://openalex.org/W3210196349, https://openalex.org/W4214728004, https://openalex.org/W2950181282, https://openalex.org/W2963261224, https://openalex.org/W2798287483, https://openalex.org/W2913410650 |
| cited_by_count | 1 |
| counts_by_year[0].year | 2024 |
| counts_by_year[0].cited_by_count | 1 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2311.09344 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2311.09344 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2311.09344 |
| primary_location.id | pmh:oai:arXiv.org:2311.09344 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2311.09344 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2311.09344 |
| publication_date | 2023-11-15 |
| publication_year | 2023 |
| referenced_works_count | 0 |
| abstract_inverted_index.In | 42 |
| abstract_inverted_index.We | 79 |
| abstract_inverted_index.an | 116 |
| abstract_inverted_index.by | 52 |
| abstract_inverted_index.in | 26 |
| abstract_inverted_index.is | 91, 115 |
| abstract_inverted_index.of | 12, 31, 126 |
| abstract_inverted_index.on | 17, 101, 109 |
| abstract_inverted_index.or | 55 |
| abstract_inverted_index.to | 47, 71, 83, 95, 104 |
| abstract_inverted_index.we | 45 |
| abstract_inverted_index.Our | 59 |
| abstract_inverted_index.and | 29, 63, 75, 93 |
| abstract_inverted_index.are | 23 |
| abstract_inverted_index.can | 7 |
| abstract_inverted_index.for | 37 |
| abstract_inverted_index.our | 81, 113 |
| abstract_inverted_index.the | 10, 18, 27, 105 |
| abstract_inverted_index.via | 67 |
| abstract_inverted_index.7000 | 24 |
| abstract_inverted_index.PEFT | 65, 98, 127 |
| abstract_inverted_index.data | 6, 36, 74, 87 |
| abstract_inverted_index.from | 88 |
| abstract_inverted_index.lack | 34 |
| abstract_inverted_index.many | 30 |
| abstract_inverted_index.more | 89 |
| abstract_inverted_index.task | 5, 56, 64 |
| abstract_inverted_index.that | 112, 119 |
| abstract_inverted_index.this | 43 |
| abstract_inverted_index.cases | 84 |
| abstract_inverted_index.data. | 78 |
| abstract_inverted_index.gains | 122 |
| abstract_inverted_index.large | 13 |
| abstract_inverted_index.task. | 20 |
| abstract_inverted_index.there | 22 |
| abstract_inverted_index.these | 32 |
| abstract_inverted_index.using | 3, 123 |
| abstract_inverted_index.where | 85 |
| abstract_inverted_index.world | 28 |
| abstract_inverted_index.(LLMs) | 16 |
| abstract_inverted_index.(PEFT) | 2 |
| abstract_inverted_index.extend | 80 |
| abstract_inverted_index.method | 60, 114 |
| abstract_inverted_index.models | 15 |
| abstract_inverted_index.paper, | 44 |
| abstract_inverted_index.tasks. | 41 |
| abstract_inverted_index.English | 76 |
| abstract_inverted_index.compose | 97 |
| abstract_inverted_index.improve | 9, 48 |
| abstract_inverted_index.labeled | 4, 35, 77, 86 |
| abstract_inverted_index.minimal | 124 |
| abstract_inverted_index.modules | 66, 99 |
| abstract_inverted_index.obtains | 120 |
| abstract_inverted_index.propose | 46, 94 |
| abstract_inverted_index.related | 103 |
| abstract_inverted_index.results | 108 |
| abstract_inverted_index.target. | 106 |
| abstract_inverted_index.trained | 100 |
| abstract_inverted_index.However, | 21 |
| abstract_inverted_index.approach | 82 |
| abstract_inverted_index.composes | 61 |
| abstract_inverted_index.language | 14, 39, 54, 62 |
| abstract_inverted_index.leverage | 72 |
| abstract_inverted_index.modules. | 128 |
| abstract_inverted_index.strategy | 118 |
| abstract_inverted_index.training | 125 |
| abstract_inverted_index.transfer | 51 |
| abstract_inverted_index.Empirical | 107 |
| abstract_inverted_index.available | 92 |
| abstract_inverted_index.composing | 53 |
| abstract_inverted_index.effective | 117 |
| abstract_inverted_index.languages | 25, 33, 90, 102 |
| abstract_inverted_index.unlabeled | 73 |
| abstract_inverted_index.zero-shot | 49 |
| abstract_inverted_index.arithmetic | 69 |
| abstract_inverted_index.consistent | 121 |
| abstract_inverted_index.downstream | 19 |
| abstract_inverted_index.generation | 40 |
| abstract_inverted_index.operations | 70 |
| abstract_inverted_index.real-world | 38 |
| abstract_inverted_index.demonstrate | 111 |
| abstract_inverted_index.fine-tuning | 1 |
| abstract_inverted_index.parameters. | 58 |
| abstract_inverted_index.performance | 11 |
| abstract_inverted_index.specialized | 57 |
| abstract_inverted_index.element-wise | 68 |
| abstract_inverted_index.cross-lingual | 50 |
| abstract_inverted_index.significantly | 8 |
| abstract_inverted_index.summarization | 110 |
| abstract_inverted_index.arithmetically | 96 |
| abstract_inverted_index.Parameter-efficient | 0 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 6 |
| sustainable_development_goals[0].id | https://metadata.un.org/sdg/4 |
| sustainable_development_goals[0].score | 0.7300000190734863 |
| sustainable_development_goals[0].display_name | Quality Education |
| citation_normalized_percentile |