OpenELM: An Efficient Language Model Family with Open Training and Inference Framework Article Swipe

MAKE IMPACT

Sachin Mehta , Mohammad Hossein Sekhavat , Qingqing Cao , Maxwell Horton , Yanzi Jin , Chenfan Sun , Iman Mirzadeh , Mahyar Najibi , Dmitry Belenko , Peter Zatloukal , Mohammad Rastegari · YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2404.14619

The reproducibility and transparency of large language models are crucial for advancing open research, ensuring the trustworthiness of results, and enabling investigations into data and model biases, as well as potential risks. To this end, we release OpenELM, a state-of-the-art open language model. OpenELM uses a layer-wise scaling strategy to efficiently allocate parameters within each layer of the transformer model, leading to enhanced accuracy. For example, with a parameter budget of approximately one billion parameters, OpenELM exhibits a 2.36% improvement in accuracy compared to OLMo while requiring $2\times$ fewer pre-training tokens. Diverging from prior practices that only provide model weights and inference code, and pre-train on private datasets, our release includes the complete framework for training and evaluation of the language model on publicly available datasets, including training logs, multiple checkpoints, and pre-training configurations. We also release code to convert models to MLX library for inference and fine-tuning on Apple devices. This comprehensive release aims to empower and strengthen the open research community, paving the way for future open research endeavors. Our source code along with pre-trained model weights and training recipes is available at \url{https://github.com/apple/corenet}. Additionally, \model models can be found on HuggingFace at: \url{https://huggingface.co/apple/OpenELM}.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2404.14619

PDF

https://arxiv.org/pdf/2404.14619

OA Status

green

Cited By

2

Related Works

10

OpenAlex ID

https://openalex.org/W4395443230

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4395443230

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2404.14619

Digital Object Identifier

Title

OpenELM: An Efficient Language Model Family with Open Training and Inference Framework

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2024

Year of publication

Publication date

2024-04-22

Full publication date if available

Authors

Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, Mohammad Rastegari

List of authors in order

Landing page

https://arxiv.org/abs/2404.14619

Publisher landing page

PDF URL

https://arxiv.org/pdf/2404.14619

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2404.14619

Direct OA link when available

Concepts

Computer science, Inference, Open source, Training (meteorology), Natural language processing, Artificial intelligence, Programming language, Geography, Software, Meteorology

Top concepts (fields/topics) attached by OpenAlex

Cited by

2

Total citation count in OpenAlex

Citations by year (recent)

2025: 2

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload