Prompting Large Language Models with Speech Recognition Abilities Article Swipe

MAKE IMPACT

Yassir Fathullah , Chunyang Wu , Egor Lakomkin , Junteng Jia , Yuan Shangguan , Ke Li , Jinxi Guo , Wenhan Xiong , Jay Mahadeokar , Ozlem Kalinli , Christian Fuegen , Mike Seltzer · YOU? · · 2023 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2307.11795

Large language models have proven themselves highly flexible, able to solve a wide range of generative tasks, such as abstractive summarization and open-ended question answering. In this paper we extend the capabilities of LLMs by directly attaching a small audio encoder allowing it to perform speech recognition. By directly prepending a sequence of audial embeddings to the text token embeddings, the LLM can be converted to an automatic speech recognition (ASR) system, and be used in the exact same manner as its textual counterpart. Experiments on Multilingual LibriSpeech (MLS) show that incorporating a conformer encoder into the open sourced LLaMA-7B allows it to outperform monolingual baselines by 18% and perform multilingual speech recognition despite LLaMA being trained overwhelmingly on English text. Furthermore, we perform ablation studies to investigate whether the LLM can be completely frozen during training to maintain its original capabilities, scaling up the audio encoder, and increasing the audio encoder striding to generate fewer embeddings. The results from these studies show that multilingual ASR is possible even when the LLM is frozen or when strides of almost 1 second are used in the audio encoder opening up the possibility for LLMs to operate on long-form audio.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2307.11795

PDF

https://arxiv.org/pdf/2307.11795

OA Status

green

Cited By

2

Related Works

10

OpenAlex ID

https://openalex.org/W4385260920

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4385260920

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2307.11795

Digital Object Identifier

Title

Prompting Large Language Models with Speech Recognition Abilities

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2023

Year of publication

Publication date

2023-07-21

Full publication date if available

Authors

Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Junteng Jia, Yuan Shangguan, Ke Li, Jinxi Guo, Wenhan Xiong, Jay Mahadeokar, Ozlem Kalinli, Christian Fuegen, Mike Seltzer

List of authors in order

Landing page

https://arxiv.org/abs/2307.11795

Publisher landing page

PDF URL

https://arxiv.org/pdf/2307.11795

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2307.11795

Direct OA link when available

Concepts

Computer science, Encoder, Speech recognition, Automatic summarization, Security token, Language model, Natural language processing, Acoustic model, Artificial intelligence, Speech processing, Operating system, Computer security

Top concepts (fields/topics) attached by OpenAlex

Cited by

2

Total citation count in OpenAlex

Citations by year (recent)

2025: 1, 2024: 1

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload