FastVLM: Efficient Vision Encoding for Vision Language Models Article Swipe

MAKE IMPACT

Pavan Kumar Anasosalu Vasu , Fartash Faghri , Chunliang Li , Cem Koc , Nate True , Albert Antony , G. Santhanam , James Gabriel , Peter Grasch , Oncel Tuzel , Hadi Pouransari · YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2412.13303

Scaling the input image resolution is essential for enhancing the performance of Vision Language Models (VLMs), particularly in text-rich image understanding tasks. However, popular visual encoders such as ViTs become inefficient at high resolutions due to the large number of tokens and high encoding latency caused by stacked self-attention layers. At different operational resolutions, the vision encoder of a VLM can be optimized along two axes: reducing encoding latency and minimizing the number of visual tokens passed to the LLM, thereby lowering overall latency. Based on a comprehensive efficiency analysis of the interplay between image resolution, vision latency, token count, and LLM size, we introduce FastVLM, a model that achieves an optimized trade-off between latency, model size and accuracy. FastVLM incorporates FastViTHD, a novel hybrid vision encoder designed to output fewer tokens and significantly reduce encoding time for high-resolution images. Unlike previous methods, FastVLM achieves the optimal balance between visual token count and image resolution solely by scaling the input image, eliminating the need for additional token pruning and simplifying the model design. In the LLaVA-1.5 setup, FastVLM achieves 3.2$\times$ improvement in time-to-first-token (TTFT) while maintaining similar performance on VLM benchmarks compared to prior works. Compared to LLaVa-OneVision at the highest resolution (1152$\times$1152), FastVLM achieves better performance on key benchmarks like SeedBench, MMMU and DocVQA, using the same 0.5B LLM, but with 85$\times$ faster TTFT and a vision encoder that is 3.4$\times$ smaller. Code and models are available at https://github.com/apple/ml-fastvlm.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2412.13303

PDF

https://arxiv.org/pdf/2412.13303

OA Status

green

Cited By

1

Related Works

10

OpenAlex ID

https://openalex.org/W4405875854

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4405875854

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2412.13303

Digital Object Identifier

Title

FastVLM: Efficient Vision Encoding for Vision Language Models

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2024

Year of publication

Publication date

2024-12-17

Full publication date if available

Authors

Pavan Kumar Anasosalu Vasu, Fartash Faghri, Chunliang Li, Cem Koc, Nate True, Albert Antony, G. Santhanam, James Gabriel, Peter Grasch, Oncel Tuzel, Hadi Pouransari

List of authors in order

Landing page

https://arxiv.org/abs/2412.13303

Publisher landing page

PDF URL

https://arxiv.org/pdf/2412.13303

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2412.13303

Direct OA link when available

Concepts

Encoding (memory), Computer science, Artificial intelligence, Computer vision

Top concepts (fields/topics) attached by OpenAlex

Cited by

1

Total citation count in OpenAlex

Citations by year (recent)

2025: 1

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload