Vec-LUT: Vector Table Lookup for Parallel Ultra-Low-Bit LLM Inference on Edge Devices Article Swipe

MAKE IMPACT

Li Xiangyu , Yin Cheng-yu , Wang Wei-jun , Wei Jianyu , Cao, Ting , Liu Yunxin · YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2512.06443

Large language models (LLMs) are increasingly deployed on edge devices. To meet strict resource constraints, real-world deployment has pushed LLM quantization from 8-bit to 4-bit, 2-bit, and now 1.58-bit. Combined with lookup table (LUT)-based inference, CPUs run these ultra-low-bit LLMs even faster than NPUs, opening new opportunities for ubiquitous on-device intelligence. However, this paper identifies that LUT-based inference underutilizes memory bandwidth during parallel inference, which is required for prefilling, test-time scaling, and other multi-token scenarios. The root cause is the scalar LUT paradigm, which performs repetitive and non-contiguous memory accesses for each token. To solve the issue, we propose vector LUT, a new lookup paradigm that constructs a unified LUT across parallel tokens, and performs a single $1 \rightarrow N$ lookup per index. To realize it efficiently, we further introduce (1) Vector LUT-Centric Tensor Layout, and (2) Cache-Aware Streamed Lookup techniques. Evaluations on 5 edge devices across 3 LLMs show that Vec-LUT outperforms state-of-the-art baselines by up to $4.2\times$. Our implementation is integrated into llama.cpp. The code is available at https://github.com/Cipherxzc/vlut.cpp.

Type

preprint

Landing Page

https://doi.org/10.48550/arxiv.2512.06443

OA Status

green

OpenAlex ID

https://openalex.org/W7110787080

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W7110787080

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2512.06443

Digital Object Identifier

Title

Vec-LUT: Vector Table Lookup for Parallel Ultra-Low-Bit LLM Inference on Edge Devices

Work title

Type

preprint

OpenAlex work type

Publication year

2025

Year of publication

Publication date

2025-12-06

Full publication date if available

Authors

Li Xiangyu, Yin Cheng-yu, Wang Wei-jun, Wei Jianyu, Cao, Ting, Liu Yunxin

List of authors in order

Landing page

https://doi.org/10.48550/arxiv.2512.06443

Publisher landing page

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://doi.org/10.48550/arxiv.2512.06443

Direct OA link when available

Concepts

Lookup table, Computer science, Table (database), Parallel computing, Bandwidth (computing), Software deployment, Enhanced Data Rates for GSM Evolution, Inference, Memory bandwidth, Edge device, Computer engineering, Algorithm, Computer hardware, Computer architecture, Quantization (signal processing), Vector quantization, Key (lock), Code (set theory), Scalar (mathematics), High memory, Resource (disambiguation), Theoretical computer science, Memory management

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Full payload