On Robustness of Finetuned Transformer-based NLP Models Article Swipe

MAKE IMPACT

Pavan Kalyan Reddy Neerudu , Subba Reddy Oota , Mounika Marreddy , Venkateswara Rao Kagita , Manish Gupta · YOU? · · 2023 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2305.14453

Transformer-based pretrained models like BERT, GPT-2 and T5 have been finetuned for a large number of natural language processing (NLP) tasks, and have been shown to be very effective. However, while finetuning, what changes across layers in these models with respect to pretrained checkpoints is under-studied. Further, how robust are these models to perturbations in input text? Does the robustness vary depending on the NLP task for which the models have been finetuned? While there exists some work on studying the robustness of BERT finetuned for a few NLP tasks, there is no rigorous study that compares this robustness across encoder only, decoder only and encoder-decoder models. In this paper, we characterize changes between pretrained and finetuned language model representations across layers using two metrics: CKA and STIR. Further, we study the robustness of three language models (BERT, GPT-2 and T5) with eight different text perturbations on classification tasks from the General Language Understanding Evaluation (GLUE) benchmark, and generation tasks like summarization, free-form generation and question generation. GPT-2 representations are more robust than BERT and T5 across multiple types of input perturbation. Although models exhibit good robustness broadly, dropping nouns, verbs or changing characters are the most impactful. Overall, this study provides valuable insights into perturbation-specific weaknesses of popular Transformer-based models, which should be kept in mind when passing inputs. We make the code and models publicly available [https://github.com/PavanNeerudu/Robustness-of-Transformers-models].

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2305.14453

PDF

https://arxiv.org/pdf/2305.14453

OA Status

green

Cited By

3

Related Works

10

OpenAlex ID

https://openalex.org/W4378469154

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4378469154

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2305.14453

Digital Object Identifier

Title

On Robustness of Finetuned Transformer-based NLP Models

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2023

Year of publication

Publication date

2023-05-23

Full publication date if available

Authors

Pavan Kalyan Reddy Neerudu, Subba Reddy Oota, Mounika Marreddy, Venkateswara Rao Kagita, Manish Gupta

List of authors in order

Landing page

https://arxiv.org/abs/2305.14453

Publisher landing page

PDF URL

https://arxiv.org/pdf/2305.14453

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2305.14453

Direct OA link when available

Concepts

Computer science, Robustness (evolution), Encoder, Transformer, Artificial intelligence, Language model, Natural language processing, Automatic summarization, Machine learning, Quantum mechanics, Physics, Voltage, Biochemistry, Operating system, Chemistry, Gene

Top concepts (fields/topics) attached by OpenAlex

Cited by

3

Total citation count in OpenAlex

Citations by year (recent)

2025: 1, 2024: 2

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload