BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models Article Swipe

PDF

Related Concepts

Computer science Transformer Language model Artificial intelligence Simple (philosophy) Process (computing) Training set Machine learning Task (project management) Fine-tuning Natural language processing Voltage Management Epistemology Philosophy Quantum mechanics Economics Physics Operating system

Elad Ben Zaken , Yoav Goldberg , Shauli Ravfogel ·

YOU? · · 2022 · Open Access · · DOI: https://doi.org/10.18653/v1/2022.acl-short.1 · OA: W3176828726

We introduce BitFit, a sparse-finetuning method where only the bias-terms of the model (or a subset of them) are being modified. We show that with small-to-medium training data, applying BitFit on pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model. For larger data, the method is competitive with other sparse fine-tuning methods.Besides their practical utility, these findings are relevant for the question of understanding the commonly-used process of finetuning: they support the hypothesis that finetuning is mainly about exposing knowledge induced by language-modeling training, rather than learning new task-specific linguistic knowledge.

BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models Article Swipe

Related Topics