TarGEN: Targeted Data Generation with Large Language Models Article Swipe

MAKE IMPACT

Himanshu Gupta , Kevin Scaria , Ujjwala Anantheswaran , Shreyas Verma , Mihir Parmar , Saurabh Arjun Sawant , Swaroop Mishra , Chitta Baral · YOU? · · 2023 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2310.17876

The rapid advancement of large language models (LLMs) has sparked interest in data synthesis techniques, aiming to generate diverse and high-quality synthetic datasets. However, these synthetic datasets often suffer from a lack of diversity and added noise. In this paper, we present TarGEN, a multi-step prompting strategy for generating high-quality synthetic datasets utilizing a LLM. An advantage of TarGEN is its seedless nature; it does not require specific task instances, broadening its applicability beyond task replication. We augment TarGEN with a method known as self-correction empowering LLMs to rectify inaccurately labeled instances during dataset creation, ensuring reliable labels. To assess our technique's effectiveness, we emulate 8 tasks from the SuperGLUE benchmark and finetune various language models, including encoder-only, encoder-decoder, and decoder-only models on both synthetic and original training sets. Evaluation on the original test set reveals that models trained on datasets generated by TarGEN perform approximately 1-2% points better than those trained on original datasets (82.84% via syn. vs. 81.12% on og. using Flan-T5). When incorporating instruction tuning, the performance increases to 84.54% on synthetic data vs. 81.49% on original data by Flan-T5. A comprehensive analysis of the synthetic dataset compared to the original dataset reveals that the synthetic dataset demonstrates similar or higher levels of dataset complexity and diversity. Furthermore, the synthetic dataset displays a bias level that aligns closely with the original dataset. Finally, when pre-finetuned on our synthetic SuperGLUE dataset, T5-3B yields impressive results on the OpenLLM leaderboard, surpassing the model trained on the Self-Instruct dataset by 4.14% points. We hope that TarGEN can be helpful for quality data generation and reducing the human efforts to create complex benchmarks.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2310.17876

PDF

https://arxiv.org/pdf/2310.17876

OA Status

green

Cited By

3

Related Works

10

OpenAlex ID

https://openalex.org/W4388032353

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4388032353

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2310.17876

Digital Object Identifier

Title

TarGEN: Targeted Data Generation with Large Language Models

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2023

Year of publication

Publication date

2023-10-27

Full publication date if available

Authors

Himanshu Gupta, Kevin Scaria, Ujjwala Anantheswaran, Shreyas Verma, Mihir Parmar, Saurabh Arjun Sawant, Swaroop Mishra, Chitta Baral

List of authors in order

Landing page

https://arxiv.org/abs/2310.17876

Publisher landing page

PDF URL

https://arxiv.org/pdf/2310.17876

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2310.17876

Direct OA link when available

Concepts

Computer science, Synthetic data, Benchmark (surveying), Task (project management), Set (abstract data type), Machine learning, Encoder, Artificial intelligence, Language model, Quality (philosophy), Noise (video), Data mining, Epistemology, Geography, Economics, Image (mathematics), Management, Geodesy, Programming language, Operating system, Philosophy

Top concepts (fields/topics) attached by OpenAlex

Cited by

3

Total citation count in OpenAlex

Citations by year (recent)

2025: 2, 2024: 1

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload