Automatically Improving LLM-based Verilog Generation using EDA Tool Feedback Article Swipe

MAKE IMPACT

Jason Blocklove , Shailja Thakur , Benjamin Tan , Hammond Pearce , Siddharth Garg , Ramesh Karri · YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2411.11856

Traditionally, digital hardware designs are written in the Verilog hardware description language (HDL) and debugged manually by engineers. This can be time-consuming and error-prone for complex designs. Large Language Models (LLMs) are emerging as a potential tool to help generate fully functioning HDL code, but most works have focused on generation in the single-shot capacity: i.e., run and evaluate, a process that does not leverage debugging and, as such, does not adequately reflect a realistic development process. In this work, we evaluate the ability of LLMs to leverage feedback from electronic design automation (EDA) tools to fix mistakes in their own generated Verilog. To accomplish this, we present an open-source, highly customizable framework, AutoChip, which combines conversational LLMs with the output from Verilog compilers and simulations to iteratively generate and repair Verilog. To determine the success of these LLMs we leverage the VerilogEval benchmark set. We evaluate four state-of-the-art conversational LLMs, focusing on readily accessible commercial models. EDA tool feedback proved to be consistently more effective than zero-shot prompting only with GPT-4o, the most computationally complex model we evaluated. In the best case, we observed a 5.8% increase in the number of successful designs with a 34.2% decrease in cost over the best zero-shot results. Mixing smaller models with this larger model at the end of the feedback iterations resulted in equally as much success as with GPT-4o using feedback, but incurred 41.9% lower cost (corresponding to an overall decrease in cost over zero-shot by 89.6%).

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2411.11856

PDF

https://arxiv.org/pdf/2411.11856

OA Status

green

Related Works

10

OpenAlex ID

https://openalex.org/W4404573312

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4404573312

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2411.11856

Digital Object Identifier

Title

Automatically Improving LLM-based Verilog Generation using EDA Tool Feedback

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2024

Year of publication

Publication date

2024-11-01

Full publication date if available

Authors

Jason Blocklove, Shailja Thakur, Benjamin Tan, Hammond Pearce, Siddharth Garg, Ramesh Karri

List of authors in order

Landing page

https://arxiv.org/abs/2411.11856

Publisher landing page

PDF URL

https://arxiv.org/pdf/2411.11856

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2411.11856

Direct OA link when available

Concepts

Verilog, Computer science, Computer architecture, Embedded system, Field-programmable gate array

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload