GEM: Empowering LLM for both Embedding Generation and Language Understanding Article Swipe

MAKE IMPACT

C. Zhang , Qiang Zhang , Ke Li , Sai Vidyaranya Nuthalapati , Benyu Zhang , Lizhu Zhang , Xiangjun Fan · YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2506.04344

Large decoder-only language models (LLMs) have achieved remarkable success in generation and reasoning tasks, where they generate text responses given instructions. However, many applications, e.g., retrieval augmented generation (RAG), still rely on separate embedding models to generate text embeddings, which can complicate the system and introduce discrepancies in understanding of the query between the embedding model and LLMs. To address this limitation, we propose a simple self-supervised approach, Generative Embedding large language Model (GEM), that enables any large decoder-only LLM to generate high-quality text embeddings while maintaining its original text generation and reasoning capabilities. Our method inserts new special token(s) into a text body, and generates summarization embedding of the text by manipulating the attention mask. This method could be easily integrated into post-training or fine tuning stages of any existing LLMs. We demonstrate the effectiveness of our approach by applying it to two popular LLM families, ranging from 1B to 8B parameters, and evaluating the transformed models on both text embedding benchmarks (MTEB) and NLP benchmarks (MMLU). The results show that our proposed method significantly improves the original LLMs on MTEB while having a minimal impact on MMLU. Our strong results indicate that our approach can empower LLMs with state-of-the-art text embedding capabilities while maintaining their original NLP performance

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2506.04344

PDF

https://arxiv.org/pdf/2506.04344

OA Status

green

OpenAlex ID

https://openalex.org/W4416131795

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4416131795

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2506.04344

Digital Object Identifier

Title

GEM: Empowering LLM for both Embedding Generation and Language Understanding

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2025

Year of publication

Publication date

2025-06-04

Full publication date if available

Authors

C. Zhang, Qiang Zhang, Ke Li, Sai Vidyaranya Nuthalapati, Benyu Zhang, Lizhu Zhang, Xiangjun Fan

List of authors in order

Landing page

https://arxiv.org/abs/2506.04344

Publisher landing page

PDF URL

https://arxiv.org/pdf/2506.04344

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2506.04344

Direct OA link when available

Cited by

0

Total citation count in OpenAlex

Full payload