Cookbook: A framework for improving LLM generative abilities via programmatic data generating templates Article Swipe
Avanika Narayan
,
Mayee F. Chen
,
Kush Bhatia
,
Christopher Ré
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2410.05224
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2410.05224
Fine-tuning large language models (LLMs) on instruction datasets is a common way to improve their generative capabilities. However, instruction datasets can be expensive and time-consuming to manually curate, and while LLM-generated data is less labor-intensive, it may violate user privacy agreements or terms of service of LLM providers. Therefore, we seek a way of constructing instruction datasets with samples that are not generated by humans or LLMs but still improve LLM generative capabilities. In this work, we introduce Cookbook, a framework that programmatically generates training data consisting of simple patterns over random tokens, resulting in a scalable, cost-effective approach that avoids legal and privacy issues. First, Cookbook uses a template -- a data generating Python function -- to produce training data that encourages the model to learn an explicit pattern-based rule that corresponds to a desired task. We find that fine-tuning on Cookbook-generated data is able to improve performance on its corresponding task by up to 52.7 accuracy points. Second, since instruction datasets improve performance on multiple downstream tasks simultaneously, Cookbook algorithmically learns how to mix data from various templates to optimize performance on multiple tasks. On the standard multi-task GPT4ALL evaluation suite, Mistral-7B fine-tuned using a Cookbook-generated dataset attains the best accuracy on average compared to other 7B parameter instruction-tuned models and is the best performing model on 3 out of 8 tasks. Finally, we analyze when and why Cookbook improves performance and present a metric that allows us to verify that the improvement is largely explained by the model's generations adhering better to template rules.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2410.05224
- https://arxiv.org/pdf/2410.05224
- OA Status
- green
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4403924772
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4403924772Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2410.05224Digital Object Identifier
- Title
-
Cookbook: A framework for improving LLM generative abilities via programmatic data generating templatesWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-10-07Full publication date if available
- Authors
-
Avanika Narayan, Mayee F. Chen, Kush Bhatia, Christopher RéList of authors in order
- Landing page
-
https://arxiv.org/abs/2410.05224Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2410.05224Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2410.05224Direct OA link when available
- Concepts
-
Template, Generative grammar, Computer science, Generative model, Data science, Artificial intelligence, Programming languageTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4403924772 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2410.05224 |
| ids.doi | https://doi.org/10.48550/arxiv.2410.05224 |
| ids.openalex | https://openalex.org/W4403924772 |
| fwci | |
| type | preprint |
| title | Cookbook: A framework for improving LLM generative abilities via programmatic data generating templates |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T12171 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9089999794960022 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1706 |
| topics[0].subfield.display_name | Computer Science Applications |
| topics[0].display_name | Open Education and E-Learning |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C82714645 |
| concepts[0].level | 2 |
| concepts[0].score | 0.9277133941650391 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q438331 |
| concepts[0].display_name | Template |
| concepts[1].id | https://openalex.org/C39890363 |
| concepts[1].level | 2 |
| concepts[1].score | 0.6731879115104675 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q36108 |
| concepts[1].display_name | Generative grammar |
| concepts[2].id | https://openalex.org/C41008148 |
| concepts[2].level | 0 |
| concepts[2].score | 0.5795628428459167 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[2].display_name | Computer science |
| concepts[3].id | https://openalex.org/C167966045 |
| concepts[3].level | 3 |
| concepts[3].score | 0.4755430221557617 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q5532625 |
| concepts[3].display_name | Generative model |
| concepts[4].id | https://openalex.org/C2522767166 |
| concepts[4].level | 1 |
| concepts[4].score | 0.3220102787017822 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q2374463 |
| concepts[4].display_name | Data science |
| concepts[5].id | https://openalex.org/C154945302 |
| concepts[5].level | 1 |
| concepts[5].score | 0.288845419883728 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[5].display_name | Artificial intelligence |
| concepts[6].id | https://openalex.org/C199360897 |
| concepts[6].level | 1 |
| concepts[6].score | 0.19135025143623352 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q9143 |
| concepts[6].display_name | Programming language |
| keywords[0].id | https://openalex.org/keywords/template |
| keywords[0].score | 0.9277133941650391 |
| keywords[0].display_name | Template |
| keywords[1].id | https://openalex.org/keywords/generative-grammar |
| keywords[1].score | 0.6731879115104675 |
| keywords[1].display_name | Generative grammar |
| keywords[2].id | https://openalex.org/keywords/computer-science |
| keywords[2].score | 0.5795628428459167 |
| keywords[2].display_name | Computer science |
| keywords[3].id | https://openalex.org/keywords/generative-model |
| keywords[3].score | 0.4755430221557617 |
| keywords[3].display_name | Generative model |
| keywords[4].id | https://openalex.org/keywords/data-science |
| keywords[4].score | 0.3220102787017822 |
| keywords[4].display_name | Data science |
| keywords[5].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[5].score | 0.288845419883728 |
| keywords[5].display_name | Artificial intelligence |
| keywords[6].id | https://openalex.org/keywords/programming-language |
| keywords[6].score | 0.19135025143623352 |
| keywords[6].display_name | Programming language |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2410.05224 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2410.05224 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2410.05224 |
| locations[1].id | doi:10.48550/arxiv.2410.05224 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | cc-by |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | https://openalex.org/licenses/cc-by |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2410.05224 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5039749428 |
| authorships[0].author.orcid | |
| authorships[0].author.display_name | Avanika Narayan |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Narayan, Avanika |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5076868148 |
| authorships[1].author.orcid | |
| authorships[1].author.display_name | Mayee F. Chen |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Chen, Mayee F. |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5046329704 |
| authorships[2].author.orcid | |
| authorships[2].author.display_name | Kush Bhatia |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Bhatia, Kush |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5103852640 |
| authorships[3].author.orcid | |
| authorships[3].author.display_name | Christopher Ré |
| authorships[3].author_position | last |
| authorships[3].raw_author_name | Ré, Christopher |
| authorships[3].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2410.05224 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Cookbook: A framework for improving LLM generative abilities via programmatic data generating templates |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T12171 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9089999794960022 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1706 |
| primary_topic.subfield.display_name | Computer Science Applications |
| primary_topic.display_name | Open Education and E-Learning |
| related_works | https://openalex.org/W4365211920, https://openalex.org/W3014948380, https://openalex.org/W4380551139, https://openalex.org/W4317695495, https://openalex.org/W4287117424, https://openalex.org/W4387506531, https://openalex.org/W2087346071, https://openalex.org/W2967848559, https://openalex.org/W4299831724, https://openalex.org/W4283803360 |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2410.05224 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2410.05224 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2410.05224 |
| primary_location.id | pmh:oai:arXiv.org:2410.05224 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2410.05224 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2410.05224 |
| publication_date | 2024-10-07 |
| publication_year | 2024 |
| referenced_works_count | 0 |
| abstract_inverted_index.3 | 219 |
| abstract_inverted_index.8 | 222 |
| abstract_inverted_index.a | 9, 51, 79, 95, 108, 111, 134, 196, 235 |
| abstract_inverted_index.-- | 110, 116 |
| abstract_inverted_index.7B | 208 |
| abstract_inverted_index.In | 73 |
| abstract_inverted_index.On | 186 |
| abstract_inverted_index.We | 137 |
| abstract_inverted_index.an | 127 |
| abstract_inverted_index.be | 21 |
| abstract_inverted_index.by | 63, 153, 248 |
| abstract_inverted_index.in | 94 |
| abstract_inverted_index.is | 8, 32, 144, 213, 245 |
| abstract_inverted_index.it | 35 |
| abstract_inverted_index.of | 43, 45, 53, 87, 221 |
| abstract_inverted_index.on | 5, 141, 149, 165, 183, 203, 218 |
| abstract_inverted_index.or | 41, 65 |
| abstract_inverted_index.to | 12, 25, 117, 125, 133, 146, 155, 174, 180, 206, 240, 254 |
| abstract_inverted_index.up | 154 |
| abstract_inverted_index.us | 239 |
| abstract_inverted_index.we | 49, 76, 225 |
| abstract_inverted_index.LLM | 46, 70 |
| abstract_inverted_index.and | 23, 28, 102, 212, 228, 233 |
| abstract_inverted_index.are | 60 |
| abstract_inverted_index.but | 67 |
| abstract_inverted_index.can | 20 |
| abstract_inverted_index.how | 173 |
| abstract_inverted_index.its | 150 |
| abstract_inverted_index.may | 36 |
| abstract_inverted_index.mix | 175 |
| abstract_inverted_index.not | 61 |
| abstract_inverted_index.out | 220 |
| abstract_inverted_index.the | 123, 187, 200, 214, 243, 249 |
| abstract_inverted_index.way | 11, 52 |
| abstract_inverted_index.why | 229 |
| abstract_inverted_index.52.7 | 156 |
| abstract_inverted_index.LLMs | 66 |
| abstract_inverted_index.able | 145 |
| abstract_inverted_index.best | 201, 215 |
| abstract_inverted_index.data | 31, 85, 112, 120, 143, 176 |
| abstract_inverted_index.find | 138 |
| abstract_inverted_index.from | 177 |
| abstract_inverted_index.less | 33 |
| abstract_inverted_index.over | 90 |
| abstract_inverted_index.rule | 130 |
| abstract_inverted_index.seek | 50 |
| abstract_inverted_index.task | 152 |
| abstract_inverted_index.that | 59, 81, 99, 121, 131, 139, 237, 242 |
| abstract_inverted_index.this | 74 |
| abstract_inverted_index.user | 38 |
| abstract_inverted_index.uses | 107 |
| abstract_inverted_index.when | 227 |
| abstract_inverted_index.with | 57 |
| abstract_inverted_index.large | 1 |
| abstract_inverted_index.learn | 126 |
| abstract_inverted_index.legal | 101 |
| abstract_inverted_index.model | 124, 217 |
| abstract_inverted_index.other | 207 |
| abstract_inverted_index.since | 160 |
| abstract_inverted_index.still | 68 |
| abstract_inverted_index.task. | 136 |
| abstract_inverted_index.tasks | 168 |
| abstract_inverted_index.terms | 42 |
| abstract_inverted_index.their | 14 |
| abstract_inverted_index.using | 195 |
| abstract_inverted_index.while | 29 |
| abstract_inverted_index.work, | 75 |
| abstract_inverted_index.(LLMs) | 4 |
| abstract_inverted_index.First, | 105 |
| abstract_inverted_index.Python | 114 |
| abstract_inverted_index.allows | 238 |
| abstract_inverted_index.avoids | 100 |
| abstract_inverted_index.better | 253 |
| abstract_inverted_index.common | 10 |
| abstract_inverted_index.humans | 64 |
| abstract_inverted_index.learns | 172 |
| abstract_inverted_index.metric | 236 |
| abstract_inverted_index.models | 3, 211 |
| abstract_inverted_index.random | 91 |
| abstract_inverted_index.rules. | 256 |
| abstract_inverted_index.simple | 88 |
| abstract_inverted_index.suite, | 192 |
| abstract_inverted_index.tasks. | 185, 223 |
| abstract_inverted_index.verify | 241 |
| abstract_inverted_index.GPT4ALL | 190 |
| abstract_inverted_index.Second, | 159 |
| abstract_inverted_index.analyze | 226 |
| abstract_inverted_index.attains | 199 |
| abstract_inverted_index.average | 204 |
| abstract_inverted_index.curate, | 27 |
| abstract_inverted_index.dataset | 198 |
| abstract_inverted_index.desired | 135 |
| abstract_inverted_index.improve | 13, 69, 147, 163 |
| abstract_inverted_index.issues. | 104 |
| abstract_inverted_index.largely | 246 |
| abstract_inverted_index.model's | 250 |
| abstract_inverted_index.points. | 158 |
| abstract_inverted_index.present | 234 |
| abstract_inverted_index.privacy | 39, 103 |
| abstract_inverted_index.produce | 118 |
| abstract_inverted_index.samples | 58 |
| abstract_inverted_index.service | 44 |
| abstract_inverted_index.tokens, | 92 |
| abstract_inverted_index.various | 178 |
| abstract_inverted_index.violate | 37 |
| abstract_inverted_index.Cookbook | 106, 170, 230 |
| abstract_inverted_index.Finally, | 224 |
| abstract_inverted_index.However, | 17 |
| abstract_inverted_index.accuracy | 157, 202 |
| abstract_inverted_index.adhering | 252 |
| abstract_inverted_index.approach | 98 |
| abstract_inverted_index.compared | 205 |
| abstract_inverted_index.datasets | 7, 19, 56, 162 |
| abstract_inverted_index.explicit | 128 |
| abstract_inverted_index.function | 115 |
| abstract_inverted_index.improves | 231 |
| abstract_inverted_index.language | 2 |
| abstract_inverted_index.manually | 26 |
| abstract_inverted_index.multiple | 166, 184 |
| abstract_inverted_index.optimize | 181 |
| abstract_inverted_index.patterns | 89 |
| abstract_inverted_index.standard | 188 |
| abstract_inverted_index.template | 109, 255 |
| abstract_inverted_index.training | 84, 119 |
| abstract_inverted_index.Cookbook, | 78 |
| abstract_inverted_index.expensive | 22 |
| abstract_inverted_index.explained | 247 |
| abstract_inverted_index.framework | 80 |
| abstract_inverted_index.generated | 62 |
| abstract_inverted_index.generates | 83 |
| abstract_inverted_index.introduce | 77 |
| abstract_inverted_index.parameter | 209 |
| abstract_inverted_index.resulting | 93 |
| abstract_inverted_index.scalable, | 96 |
| abstract_inverted_index.templates | 179 |
| abstract_inverted_index.Mistral-7B | 193 |
| abstract_inverted_index.Therefore, | 48 |
| abstract_inverted_index.agreements | 40 |
| abstract_inverted_index.consisting | 86 |
| abstract_inverted_index.downstream | 167 |
| abstract_inverted_index.encourages | 122 |
| abstract_inverted_index.evaluation | 191 |
| abstract_inverted_index.fine-tuned | 194 |
| abstract_inverted_index.generating | 113 |
| abstract_inverted_index.generative | 15, 71 |
| abstract_inverted_index.multi-task | 189 |
| abstract_inverted_index.performing | 216 |
| abstract_inverted_index.providers. | 47 |
| abstract_inverted_index.Fine-tuning | 0 |
| abstract_inverted_index.corresponds | 132 |
| abstract_inverted_index.fine-tuning | 140 |
| abstract_inverted_index.generations | 251 |
| abstract_inverted_index.improvement | 244 |
| abstract_inverted_index.instruction | 6, 18, 55, 161 |
| abstract_inverted_index.performance | 148, 164, 182, 232 |
| abstract_inverted_index.constructing | 54 |
| abstract_inverted_index.LLM-generated | 30 |
| abstract_inverted_index.capabilities. | 16, 72 |
| abstract_inverted_index.corresponding | 151 |
| abstract_inverted_index.pattern-based | 129 |
| abstract_inverted_index.cost-effective | 97 |
| abstract_inverted_index.time-consuming | 24 |
| abstract_inverted_index.algorithmically | 171 |
| abstract_inverted_index.simultaneously, | 169 |
| abstract_inverted_index.labor-intensive, | 34 |
| abstract_inverted_index.programmatically | 82 |
| abstract_inverted_index.instruction-tuned | 210 |
| abstract_inverted_index.Cookbook-generated | 142, 197 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 4 |
| citation_normalized_percentile |