Scalable Bayesian Low-Rank Adaptation of Large Language Models via Stochastic Variational Subspace Inference Article Swipe

MAKE IMPACT

Colin Samplawski , Adam D. Cobb , Manoj Acharya , Ramneet Kaur , Susmit Jha · YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2506.21408

Despite their widespread use, large language models (LLMs) are known to hallucinate incorrect information and be poorly calibrated. This makes the uncertainty quantification of these models of critical importance, especially in high-stakes domains, such as autonomy and healthcare. Prior work has made Bayesian deep learning-based approaches to this problem more tractable by performing inference over the low-rank adaptation (LoRA) parameters of a fine-tuned model. While effective, these approaches struggle to scale to larger LLMs due to requiring further additional parameters compared to LoRA. In this work we present $\textbf{Scala}$ble $\textbf{B}$ayesian $\textbf{L}$ow-Rank Adaptation via Stochastic Variational Subspace Inference (ScalaBL). We perform Bayesian inference in an $r$-dimensional subspace, for LoRA rank $r$. By repurposing the LoRA parameters as projection matrices, we are able to map samples from this subspace into the full weight space of the LLM. This allows us to learn all the parameters of our approach using stochastic variational inference. Despite the low dimensionality of our subspace, we are able to achieve competitive performance with state-of-the-art approaches while only requiring ${\sim}1000$ additional parameters. Furthermore, it allows us to scale up to the largest Bayesian LLM to date, with four times as a many base parameters as prior work.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2506.21408

PDF

https://arxiv.org/pdf/2506.21408

OA Status

green

OpenAlex ID

https://openalex.org/W4415183342

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4415183342

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2506.21408

Digital Object Identifier

Title

Scalable Bayesian Low-Rank Adaptation of Large Language Models via Stochastic Variational Subspace Inference

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2025

Year of publication

Publication date

2025-06-26

Full publication date if available

Authors

Colin Samplawski, Adam D. Cobb, Manoj Acharya, Ramneet Kaur, Susmit Jha

List of authors in order

Landing page

https://arxiv.org/abs/2506.21408

Publisher landing page

PDF URL

https://arxiv.org/pdf/2506.21408

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2506.21408

Direct OA link when available

Cited by

0

Total citation count in OpenAlex

Full payload