From Dense to Dynamic: Token-Difficulty Driven MoEfication of Pre-Trained LLMs Article Swipe

MAKE IMPACT

Kumari Nishu , Sachin Mehta , Samira Abnar , Mehrdad Farajtabar , Maxwell Horton , Mahyar Najibi , Moin Nabi , Minsik Cho , Devang Naik · YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2502.12325

Training large language models (LLMs) for different inference constraints is computationally expensive, limiting control over efficiency-accuracy trade-offs. Moreover, once trained, these models typically process tokens uniformly, regardless of their complexity, leading to static and inflexible behavior. In this paper, we introduce a post-training optimization framework, DynaMoE, that adapts a pre-trained dense LLM to a token-difficulty-driven Mixture-of-Experts model with minimal fine-tuning cost. This adaptation makes the model dynamic, with sensitivity control to customize the balance between efficiency and accuracy. DynaMoE features a token-difficulty-aware router that predicts the difficulty of tokens and directs them to the appropriate sub-networks or experts, enabling larger experts to handle more complex tokens and smaller experts to process simpler ones. Our experiments demonstrate that DynaMoE can generate a range of adaptive model variants of the existing trained LLM with a single fine-tuning step, utilizing only $10B$ tokens, a minimal cost compared to the base model's training. Each variant offers distinct trade-offs between accuracy and performance. Compared to the baseline post-training optimization framework, Flextron, our method achieves similar aggregated accuracy across downstream tasks, despite using only $\frac{1}{9}\text{th}$ of their fine-tuning cost.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2502.12325

PDF

https://arxiv.org/pdf/2502.12325

OA Status

green

Related Works

10

OpenAlex ID

https://openalex.org/W4407759271

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4407759271

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2502.12325

Digital Object Identifier

Title

From Dense to Dynamic: Token-Difficulty Driven MoEfication of Pre-Trained LLMs

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2025

Year of publication

Publication date

2025-02-17

Full publication date if available

Authors

Kumari Nishu, Sachin Mehta, Samira Abnar, Mehrdad Farajtabar, Maxwell Horton, Mahyar Najibi, Moin Nabi, Minsik Cho, Devang Naik

List of authors in order

Landing page

https://arxiv.org/abs/2502.12325

Publisher landing page

PDF URL

https://arxiv.org/pdf/2502.12325

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2502.12325

Direct OA link when available

Concepts

Security token, Computer science, Computer security

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload