Low Resource Summarization using Pre-trained Language Models Article Swipe

MAKE IMPACT

Mubashir Munaf , Hammad Afzal , Naima Iltaf , Khawir Mahmood · YOU? · · 2023 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2310.02790

With the advent of Deep Learning based Artificial Neural Networks models, Natural Language Processing (NLP) has witnessed significant improvements in textual data processing in terms of its efficiency and accuracy. However, the research is mostly restricted to high-resource languages such as English and low-resource languages still suffer from a lack of available resources in terms of training datasets as well as models with even baseline evaluation results. Considering the limited availability of resources for low-resource languages, we propose a methodology for adapting self-attentive transformer-based architecture models (mBERT, mT5) for low-resource summarization, supplemented by the construction of a new baseline dataset (76.5k article, summary pairs) in a low-resource language Urdu. Choosing news (a publicly available source) as the application domain has the potential to make the proposed methodology useful for reproducing in other languages with limited resources. Our adapted summarization model \textit{urT5} with up to 44.78\% reduction in size as compared to \textit{mT5} can capture contextual information of low resource language effectively with evaluation score (up to 46.35 ROUGE-1, 77 BERTScore) at par with state-of-the-art models in high resource language English \textit{(PEGASUS: 47.21, BART: 45.14 on XSUM Dataset)}. The proposed method provided a baseline approach towards extractive as well as abstractive summarization with competitive evaluation results in a limited resource setup.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2310.02790

PDF

https://arxiv.org/pdf/2310.02790

OA Status

green

Cited By

1

Related Works

10

OpenAlex ID

https://openalex.org/W4387390155

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4387390155

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2310.02790

Digital Object Identifier

Title

Low Resource Summarization using Pre-trained Language Models

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2023

Year of publication

Publication date

2023-10-04

Full publication date if available

Authors

Mubashir Munaf, Hammad Afzal, Naima Iltaf, Khawir Mahmood

List of authors in order

Landing page

https://arxiv.org/abs/2310.02790

Publisher landing page

PDF URL

https://arxiv.org/pdf/2310.02790

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2310.02790

Direct OA link when available

Concepts

Automatic summarization, Computer science, Baseline (sea), Natural language processing, Artificial intelligence, Transformer, Resource (disambiguation), Architecture, Machine learning, Geography, Engineering, Geology, Oceanography, Voltage, Computer network, Archaeology, Electrical engineering

Top concepts (fields/topics) attached by OpenAlex

Cited by

1

Total citation count in OpenAlex

Citations by year (recent)

2025: 1

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload