ERMoE: Eigen-Reparameterized Mixture-of-Experts for Stable Routing and Interpretable Specialization Article Swipe

MAKE IMPACT

Shukai Duan , Shixuan Li , Chenzhong Yin , Mingxi Cheng , Heng Ping , Tamoghna Chattopadhyay , Sophia I. Thomopoulos , Shahin Nazarian , Paul Thompson , Paul Bogdan · YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2511.10971

Mixture-of-Experts (MoE) architectures expand model capacity by sparsely activating experts but face two core challenges: misalignment between router logits and each expert's internal structure leads to unstable routing and expert underutilization, and load imbalances create straggler bottlenecks. Standard solutions, such as auxiliary load-balancing losses, can reduce load disparities but often weaken expert specialization and hurt downstream performance. To address these issues, we propose ERMoE, a sparse MoE transformer that reparameterizes each expert in a learned orthonormal eigenbasis and replaces learned gating logits with an "Eigenbasis Score", defined as the cosine similarity between input features and an expert's basis. This content-aware routing ties token assignments directly to experts' representation spaces, stabilizing utilization and promoting interpretable specialization without sacrificing sparsity. Crucially, ERMoE removes the need for explicit balancing losses and avoids the interfering gradients they introduce. We show that ERMoE achieves state-of-the-art accuracy on ImageNet classification and cross-modal image-text retrieval benchmarks (e.g., COCO, Flickr30K), while naturally producing flatter expert load distributions. Moreover, a 3D MRI variant (ERMoE-ba) improves brain age prediction accuracy by more than 7\% and yields anatomically interpretable expert specializations. ERMoE thus introduces a new architectural principle for sparse expert models that directly addresses routing instabilities and enables improved performance with scalable, interpretable specialization.

Type

preprint

Landing Page

http://arxiv.org/abs/2511.10971

PDF

https://arxiv.org/pdf/2511.10971

OA Status

green

OpenAlex ID

https://openalex.org/W4416341092

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4416341092

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2511.10971

Digital Object Identifier

Title

ERMoE: Eigen-Reparameterized Mixture-of-Experts for Stable Routing and Interpretable Specialization

Work title

Type

preprint

OpenAlex work type

Publication year

2025

Year of publication

Publication date

2025-11-14

Full publication date if available

Authors

Shukai Duan, Shixuan Li, Chenzhong Yin, Mingxi Cheng, Heng Ping, Tamoghna Chattopadhyay, Sophia I. Thomopoulos, Shahin Nazarian, Paul Thompson, Paul Bogdan

List of authors in order

Landing page

https://arxiv.org/abs/2511.10971

Publisher landing page

PDF URL

https://arxiv.org/pdf/2511.10971

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2511.10971

Direct OA link when available

Cited by

0

Total citation count in OpenAlex

Full payload