Scaling Up Models and Data with $\texttt{t5x}$ and $\texttt{seqio}$ Article Swipe
Adam Roberts
,
Hyung Won Chung
,
Anselm Levskaya
,
Gaurav Mishra
,
James T. Bradbury
,
Daniel Andor
,
Sharan Narang
,
Brian Lester
,
Colin Gaffney
,
Afroz Mohiuddin
,
Curtis Hawthorne
,
Aitor Lewkowycz
,
Alex Salcianu
,
Marc van Zee
,
Jacob Austin
,
Sebastian Goodman
,
Livio Baldini Soares
,
Haitang Hu
,
Sasha Tsvyashchenko
,
Aakanksha Chowdhery
,
Jasmijn Bastings
,
Jannis Bulian
,
Xavier García
,
Jianmo Ni
,
Andrew Chen
,
Kathleen Kenealy
,
Jonathan H. Clark
,
Stephan Lee
,
Dan Garrette
,
James Lee-Thorp
,
Colin Raffel
,
Noam Shazeer
,
Marvin Ritter
,
Maarten Bosma
,
A. M. A. dos Passos
,
Jeremy Maitin-Shepard
,
Noah Fiedel
,
Mark Omernick
,
Brennan Saeta
,
Ryan Sepassi
,
Alexander Spiridonov
,
Joshua Newlan
,
Andréa Gesmundo
·
YOU?
·
· 2022
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2203.17189
YOU?
·
· 2022
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2203.17189
Recent neural network-based language models have benefited greatly from scaling up the size of training datasets and the number of parameters in the models themselves. Scaling can be complicated due to various factors including the need to distribute computation on supercomputer clusters (e.g., TPUs), prevent bottlenecks when infeeding data, and ensure reproducible results. In this work, we present two software libraries that ease these issues: $\texttt{t5x}$ simplifies the process of building and training large language models at scale while maintaining ease of use, and $\texttt{seqio}$ provides a task-based API for simple creation of fast and reproducible training data and evaluation pipelines. These open-source libraries have been used to train models with hundreds of billions of parameters on datasets with multiple terabytes of training data. Along with the libraries, we release configurations and instructions for T5-like encoder-decoder models as well as GPT-like decoder-only architectures. $\texttt{t5x}$ and $\texttt{seqio}$ are open source and available at https://github.com/google-research/t5x and https://github.com/google/seqio, respectively.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2203.17189
- https://arxiv.org/pdf/2203.17189
- OA Status
- green
- Cited By
- 47
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4224442590
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4224442590Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2203.17189Digital Object Identifier
- Title
-
Scaling Up Models and Data with $\texttt{t5x}$ and $\texttt{seqio}$Work title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2022Year of publication
- Publication date
-
2022-03-31Full publication date if available
- Authors
-
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James T. Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bulian, Xavier García, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, A. M. A. dos Passos, Jeremy Maitin-Shepard, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, Andréa GesmundoList of authors in order
- Landing page
-
https://arxiv.org/abs/2203.17189Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2203.17189Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2203.17189Direct OA link when available
- Concepts
-
Terabyte, Computer science, Supercomputer, Scaling, Process (computing), Encoder, Computation, Computational science, Software, Parallel computing, Operating system, Programming language, Geometry, MathematicsTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
47Total citation count in OpenAlex
- Citations by year (recent)
-
2024: 2, 2023: 39, 2022: 6Per-year citation counts (last 5 years)
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4224442590 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2203.17189 |
| ids.doi | https://doi.org/10.48550/arxiv.2203.17189 |
| ids.openalex | https://openalex.org/W4224442590 |
| fwci | |
| type | preprint |
| title | Scaling Up Models and Data with $\texttt{t5x}$ and $\texttt{seqio}$ |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10028 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9983999729156494 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Topic Modeling |
| topics[1].id | https://openalex.org/T10181 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9832000136375427 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Natural Language Processing Techniques |
| topics[2].id | https://openalex.org/T10036 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9768000245094299 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1707 |
| topics[2].subfield.display_name | Computer Vision and Pattern Recognition |
| topics[2].display_name | Advanced Neural Network Applications |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C199683683 |
| concepts[0].level | 2 |
| concepts[0].score | 0.8486843109130859 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q8799 |
| concepts[0].display_name | Terabyte |
| concepts[1].id | https://openalex.org/C41008148 |
| concepts[1].level | 0 |
| concepts[1].score | 0.8265692591667175 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[1].display_name | Computer science |
| concepts[2].id | https://openalex.org/C83283714 |
| concepts[2].level | 2 |
| concepts[2].score | 0.6572855710983276 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q121117 |
| concepts[2].display_name | Supercomputer |
| concepts[3].id | https://openalex.org/C99844830 |
| concepts[3].level | 2 |
| concepts[3].score | 0.6150655150413513 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q102441924 |
| concepts[3].display_name | Scaling |
| concepts[4].id | https://openalex.org/C98045186 |
| concepts[4].level | 2 |
| concepts[4].score | 0.5396497249603271 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q205663 |
| concepts[4].display_name | Process (computing) |
| concepts[5].id | https://openalex.org/C118505674 |
| concepts[5].level | 2 |
| concepts[5].score | 0.5292741060256958 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q42586063 |
| concepts[5].display_name | Encoder |
| concepts[6].id | https://openalex.org/C45374587 |
| concepts[6].level | 2 |
| concepts[6].score | 0.4676589071750641 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q12525525 |
| concepts[6].display_name | Computation |
| concepts[7].id | https://openalex.org/C459310 |
| concepts[7].level | 1 |
| concepts[7].score | 0.4542534649372101 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q117801 |
| concepts[7].display_name | Computational science |
| concepts[8].id | https://openalex.org/C2777904410 |
| concepts[8].level | 2 |
| concepts[8].score | 0.42669832706451416 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q7397 |
| concepts[8].display_name | Software |
| concepts[9].id | https://openalex.org/C173608175 |
| concepts[9].level | 1 |
| concepts[9].score | 0.3080093264579773 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q232661 |
| concepts[9].display_name | Parallel computing |
| concepts[10].id | https://openalex.org/C111919701 |
| concepts[10].level | 1 |
| concepts[10].score | 0.25890618562698364 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q9135 |
| concepts[10].display_name | Operating system |
| concepts[11].id | https://openalex.org/C199360897 |
| concepts[11].level | 1 |
| concepts[11].score | 0.23612865805625916 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q9143 |
| concepts[11].display_name | Programming language |
| concepts[12].id | https://openalex.org/C2524010 |
| concepts[12].level | 1 |
| concepts[12].score | 0.0 |
| concepts[12].wikidata | https://www.wikidata.org/wiki/Q8087 |
| concepts[12].display_name | Geometry |
| concepts[13].id | https://openalex.org/C33923547 |
| concepts[13].level | 0 |
| concepts[13].score | 0.0 |
| concepts[13].wikidata | https://www.wikidata.org/wiki/Q395 |
| concepts[13].display_name | Mathematics |
| keywords[0].id | https://openalex.org/keywords/terabyte |
| keywords[0].score | 0.8486843109130859 |
| keywords[0].display_name | Terabyte |
| keywords[1].id | https://openalex.org/keywords/computer-science |
| keywords[1].score | 0.8265692591667175 |
| keywords[1].display_name | Computer science |
| keywords[2].id | https://openalex.org/keywords/supercomputer |
| keywords[2].score | 0.6572855710983276 |
| keywords[2].display_name | Supercomputer |
| keywords[3].id | https://openalex.org/keywords/scaling |
| keywords[3].score | 0.6150655150413513 |
| keywords[3].display_name | Scaling |
| keywords[4].id | https://openalex.org/keywords/process |
| keywords[4].score | 0.5396497249603271 |
| keywords[4].display_name | Process (computing) |
| keywords[5].id | https://openalex.org/keywords/encoder |
| keywords[5].score | 0.5292741060256958 |
| keywords[5].display_name | Encoder |
| keywords[6].id | https://openalex.org/keywords/computation |
| keywords[6].score | 0.4676589071750641 |
| keywords[6].display_name | Computation |
| keywords[7].id | https://openalex.org/keywords/computational-science |
| keywords[7].score | 0.4542534649372101 |
| keywords[7].display_name | Computational science |
| keywords[8].id | https://openalex.org/keywords/software |
| keywords[8].score | 0.42669832706451416 |
| keywords[8].display_name | Software |
| keywords[9].id | https://openalex.org/keywords/parallel-computing |
| keywords[9].score | 0.3080093264579773 |
| keywords[9].display_name | Parallel computing |
| keywords[10].id | https://openalex.org/keywords/operating-system |
| keywords[10].score | 0.25890618562698364 |
| keywords[10].display_name | Operating system |
| keywords[11].id | https://openalex.org/keywords/programming-language |
| keywords[11].score | 0.23612865805625916 |
| keywords[11].display_name | Programming language |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2203.17189 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2203.17189 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2203.17189 |
| locations[1].id | doi:10.48550/arxiv.2203.17189 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2203.17189 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5052454696 |
| authorships[0].author.orcid | https://orcid.org/0000-0003-1621-1964 |
| authorships[0].author.display_name | Adam Roberts |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Roberts, Adam |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5051828575 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-1280-9953 |
| authorships[1].author.display_name | Hyung Won Chung |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Chung, Hyung Won |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5069544528 |
| authorships[2].author.orcid | |
| authorships[2].author.display_name | Anselm Levskaya |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Levskaya, Anselm |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5085443636 |
| authorships[3].author.orcid | https://orcid.org/0000-0002-3254-8797 |
| authorships[3].author.display_name | Gaurav Mishra |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Mishra, Gaurav |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5052263308 |
| authorships[4].author.orcid | |
| authorships[4].author.display_name | James T. Bradbury |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Bradbury, James |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5026230181 |
| authorships[5].author.orcid | |
| authorships[5].author.display_name | Daniel Andor |
| authorships[5].author_position | middle |
| authorships[5].raw_author_name | Andor, Daniel |
| authorships[5].is_corresponding | False |
| authorships[6].author.id | https://openalex.org/A5079540764 |
| authorships[6].author.orcid | |
| authorships[6].author.display_name | Sharan Narang |
| authorships[6].author_position | middle |
| authorships[6].raw_author_name | Narang, Sharan |
| authorships[6].is_corresponding | False |
| authorships[7].author.id | https://openalex.org/A5021911747 |
| authorships[7].author.orcid | https://orcid.org/0000-0002-4297-2508 |
| authorships[7].author.display_name | Brian Lester |
| authorships[7].author_position | middle |
| authorships[7].raw_author_name | Lester, Brian |
| authorships[7].is_corresponding | False |
| authorships[8].author.id | https://openalex.org/A5034219676 |
| authorships[8].author.orcid | |
| authorships[8].author.display_name | Colin Gaffney |
| authorships[8].author_position | middle |
| authorships[8].raw_author_name | Gaffney, Colin |
| authorships[8].is_corresponding | False |
| authorships[9].author.id | https://openalex.org/A5081121649 |
| authorships[9].author.orcid | https://orcid.org/0000-0002-6310-7660 |
| authorships[9].author.display_name | Afroz Mohiuddin |
| authorships[9].author_position | middle |
| authorships[9].raw_author_name | Mohiuddin, Afroz |
| authorships[9].is_corresponding | False |
| authorships[10].author.id | https://openalex.org/A5083209727 |
| authorships[10].author.orcid | |
| authorships[10].author.display_name | Curtis Hawthorne |
| authorships[10].author_position | middle |
| authorships[10].raw_author_name | Hawthorne, Curtis |
| authorships[10].is_corresponding | False |
| authorships[11].author.id | https://openalex.org/A5019210292 |
| authorships[11].author.orcid | |
| authorships[11].author.display_name | Aitor Lewkowycz |
| authorships[11].author_position | middle |
| authorships[11].raw_author_name | Lewkowycz, Aitor |
| authorships[11].is_corresponding | False |
| authorships[12].author.id | https://openalex.org/A5007249066 |
| authorships[12].author.orcid | |
| authorships[12].author.display_name | Alex Salcianu |
| authorships[12].author_position | middle |
| authorships[12].raw_author_name | Salcianu, Alex |
| authorships[12].is_corresponding | False |
| authorships[13].author.id | https://openalex.org/A5021557939 |
| authorships[13].author.orcid | https://orcid.org/0000-0003-2413-8074 |
| authorships[13].author.display_name | Marc van Zee |
| authorships[13].author_position | middle |
| authorships[13].raw_author_name | van Zee, Marc |
| authorships[13].is_corresponding | False |
| authorships[14].author.id | https://openalex.org/A5014400463 |
| authorships[14].author.orcid | https://orcid.org/0009-0001-2589-2805 |
| authorships[14].author.display_name | Jacob Austin |
| authorships[14].author_position | middle |
| authorships[14].raw_author_name | Austin, Jacob |
| authorships[14].is_corresponding | False |
| authorships[15].author.id | https://openalex.org/A5090365098 |
| authorships[15].author.orcid | |
| authorships[15].author.display_name | Sebastian Goodman |
| authorships[15].author_position | middle |
| authorships[15].raw_author_name | Goodman, Sebastian |
| authorships[15].is_corresponding | False |
| authorships[16].author.id | https://openalex.org/A5076699933 |
| authorships[16].author.orcid | |
| authorships[16].author.display_name | Livio Baldini Soares |
| authorships[16].author_position | middle |
| authorships[16].raw_author_name | Soares, Livio Baldini |
| authorships[16].is_corresponding | False |
| authorships[17].author.id | https://openalex.org/A5101268821 |
| authorships[17].author.orcid | https://orcid.org/0009-0002-8557-1445 |
| authorships[17].author.display_name | Haitang Hu |
| authorships[17].author_position | middle |
| authorships[17].raw_author_name | Hu, Haitang |
| authorships[17].is_corresponding | False |
| authorships[18].author.id | https://openalex.org/A5027632940 |
| authorships[18].author.orcid | |
| authorships[18].author.display_name | Sasha Tsvyashchenko |
| authorships[18].author_position | middle |
| authorships[18].raw_author_name | Tsvyashchenko, Sasha |
| authorships[18].is_corresponding | False |
| authorships[19].author.id | https://openalex.org/A5055969617 |
| authorships[19].author.orcid | https://orcid.org/0000-0002-0628-5225 |
| authorships[19].author.display_name | Aakanksha Chowdhery |
| authorships[19].author_position | middle |
| authorships[19].raw_author_name | Chowdhery, Aakanksha |
| authorships[19].is_corresponding | False |
| authorships[20].author.id | https://openalex.org/A5077391471 |
| authorships[20].author.orcid | https://orcid.org/0000-0002-5445-4417 |
| authorships[20].author.display_name | Jasmijn Bastings |
| authorships[20].author_position | middle |
| authorships[20].raw_author_name | Bastings, Jasmijn |
| authorships[20].is_corresponding | False |
| authorships[21].author.id | https://openalex.org/A5089402969 |
| authorships[21].author.orcid | |
| authorships[21].author.display_name | Jannis Bulian |
| authorships[21].author_position | middle |
| authorships[21].raw_author_name | Bulian, Jannis |
| authorships[21].is_corresponding | False |
| authorships[22].author.id | https://openalex.org/A5082383881 |
| authorships[22].author.orcid | https://orcid.org/0000-0002-8500-4224 |
| authorships[22].author.display_name | Xavier García |
| authorships[22].author_position | middle |
| authorships[22].raw_author_name | Garcia, Xavier |
| authorships[22].is_corresponding | False |
| authorships[23].author.id | https://openalex.org/A5077817759 |
| authorships[23].author.orcid | https://orcid.org/0000-0002-6863-8073 |
| authorships[23].author.display_name | Jianmo Ni |
| authorships[23].author_position | middle |
| authorships[23].raw_author_name | Ni, Jianmo |
| authorships[23].is_corresponding | False |
| authorships[24].author.id | https://openalex.org/A5100703651 |
| authorships[24].author.orcid | https://orcid.org/0000-0002-6239-8443 |
| authorships[24].author.display_name | Andrew Chen |
| authorships[24].author_position | middle |
| authorships[24].raw_author_name | Chen, Andrew |
| authorships[24].is_corresponding | False |
| authorships[25].author.id | https://openalex.org/A5009962014 |
| authorships[25].author.orcid | |
| authorships[25].author.display_name | Kathleen Kenealy |
| authorships[25].author_position | middle |
| authorships[25].raw_author_name | Kenealy, Kathleen |
| authorships[25].is_corresponding | False |
| authorships[26].author.id | https://openalex.org/A5112226713 |
| authorships[26].author.orcid | |
| authorships[26].author.display_name | Jonathan H. Clark |
| authorships[26].author_position | middle |
| authorships[26].raw_author_name | Clark, Jonathan H. |
| authorships[26].is_corresponding | False |
| authorships[27].author.id | https://openalex.org/A5064864500 |
| authorships[27].author.orcid | |
| authorships[27].author.display_name | Stephan Lee |
| authorships[27].author_position | middle |
| authorships[27].raw_author_name | Lee, Stephan |
| authorships[27].is_corresponding | False |
| authorships[28].author.id | https://openalex.org/A5087636608 |
| authorships[28].author.orcid | |
| authorships[28].author.display_name | Dan Garrette |
| authorships[28].author_position | middle |
| authorships[28].raw_author_name | Garrette, Dan |
| authorships[28].is_corresponding | False |
| authorships[29].author.id | https://openalex.org/A5018339854 |
| authorships[29].author.orcid | https://orcid.org/0000-0001-6445-7155 |
| authorships[29].author.display_name | James Lee-Thorp |
| authorships[29].author_position | middle |
| authorships[29].raw_author_name | Lee-Thorp, James |
| authorships[29].is_corresponding | False |
| authorships[30].author.id | https://openalex.org/A5045077843 |
| authorships[30].author.orcid | |
| authorships[30].author.display_name | Colin Raffel |
| authorships[30].author_position | middle |
| authorships[30].raw_author_name | Raffel, Colin |
| authorships[30].is_corresponding | False |
| authorships[31].author.id | https://openalex.org/A5021878400 |
| authorships[31].author.orcid | |
| authorships[31].author.display_name | Noam Shazeer |
| authorships[31].author_position | middle |
| authorships[31].raw_author_name | Shazeer, Noam |
| authorships[31].is_corresponding | False |
| authorships[32].author.id | https://openalex.org/A5068127101 |
| authorships[32].author.orcid | |
| authorships[32].author.display_name | Marvin Ritter |
| authorships[32].author_position | middle |
| authorships[32].raw_author_name | Ritter, Marvin |
| authorships[32].is_corresponding | False |
| authorships[33].author.id | https://openalex.org/A5074322007 |
| authorships[33].author.orcid | |
| authorships[33].author.display_name | Maarten Bosma |
| authorships[33].author_position | middle |
| authorships[33].raw_author_name | Bosma, Maarten |
| authorships[33].is_corresponding | False |
| authorships[34].author.id | https://openalex.org/A5018424374 |
| authorships[34].author.orcid | https://orcid.org/0000-0002-9917-0688 |
| authorships[34].author.display_name | A. M. A. dos Passos |
| authorships[34].author_position | middle |
| authorships[34].raw_author_name | Passos, Alexandre |
| authorships[34].is_corresponding | False |
| authorships[35].author.id | https://openalex.org/A5011172118 |
| authorships[35].author.orcid | https://orcid.org/0000-0001-8453-7961 |
| authorships[35].author.display_name | Jeremy Maitin-Shepard |
| authorships[35].author_position | middle |
| authorships[35].raw_author_name | Maitin-Shepard, Jeremy |
| authorships[35].is_corresponding | False |
| authorships[36].author.id | https://openalex.org/A5087591662 |
| authorships[36].author.orcid | |
| authorships[36].author.display_name | Noah Fiedel |
| authorships[36].author_position | middle |
| authorships[36].raw_author_name | Fiedel, Noah |
| authorships[36].is_corresponding | False |
| authorships[37].author.id | https://openalex.org/A5070268107 |
| authorships[37].author.orcid | |
| authorships[37].author.display_name | Mark Omernick |
| authorships[37].author_position | middle |
| authorships[37].raw_author_name | Omernick, Mark |
| authorships[37].is_corresponding | False |
| authorships[38].author.id | https://openalex.org/A5067187716 |
| authorships[38].author.orcid | |
| authorships[38].author.display_name | Brennan Saeta |
| authorships[38].author_position | middle |
| authorships[38].raw_author_name | Saeta, Brennan |
| authorships[38].is_corresponding | False |
| authorships[39].author.id | https://openalex.org/A5065754485 |
| authorships[39].author.orcid | |
| authorships[39].author.display_name | Ryan Sepassi |
| authorships[39].author_position | middle |
| authorships[39].raw_author_name | Sepassi, Ryan |
| authorships[39].is_corresponding | False |
| authorships[40].author.id | https://openalex.org/A5107896946 |
| authorships[40].author.orcid | https://orcid.org/0000-0002-0935-3764 |
| authorships[40].author.display_name | Alexander Spiridonov |
| authorships[40].author_position | middle |
| authorships[40].raw_author_name | Spiridonov, Alexander |
| authorships[40].is_corresponding | False |
| authorships[41].author.id | https://openalex.org/A5039246529 |
| authorships[41].author.orcid | |
| authorships[41].author.display_name | Joshua Newlan |
| authorships[41].author_position | middle |
| authorships[41].raw_author_name | Newlan, Joshua |
| authorships[41].is_corresponding | False |
| authorships[42].author.id | https://openalex.org/A5069086621 |
| authorships[42].author.orcid | |
| authorships[42].author.display_name | Andréa Gesmundo |
| authorships[42].author_position | last |
| authorships[42].raw_author_name | Gesmundo, Andrea |
| authorships[42].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2203.17189 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Scaling Up Models and Data with $\texttt{t5x}$ and $\texttt{seqio}$ |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10028 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9983999729156494 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Topic Modeling |
| related_works | https://openalex.org/W2066858118, https://openalex.org/W2134017072, https://openalex.org/W1976914335, https://openalex.org/W2915208987, https://openalex.org/W2152256925, https://openalex.org/W1940452713, https://openalex.org/W2018090346, https://openalex.org/W1582436825, https://openalex.org/W1996803181, https://openalex.org/W2979588510 |
| cited_by_count | 47 |
| counts_by_year[0].year | 2024 |
| counts_by_year[0].cited_by_count | 2 |
| counts_by_year[1].year | 2023 |
| counts_by_year[1].cited_by_count | 39 |
| counts_by_year[2].year | 2022 |
| counts_by_year[2].cited_by_count | 6 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2203.17189 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2203.17189 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2203.17189 |
| primary_location.id | pmh:oai:arXiv.org:2203.17189 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2203.17189 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2203.17189 |
| publication_date | 2022-03-31 |
| publication_year | 2022 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 86 |
| abstract_inverted_index.In | 53 |
| abstract_inverted_index.as | 137, 139 |
| abstract_inverted_index.at | 76, 151 |
| abstract_inverted_index.be | 27 |
| abstract_inverted_index.in | 21 |
| abstract_inverted_index.of | 13, 19, 69, 81, 92, 112, 114, 121 |
| abstract_inverted_index.on | 39, 116 |
| abstract_inverted_index.to | 30, 36, 107 |
| abstract_inverted_index.up | 10 |
| abstract_inverted_index.we | 56, 128 |
| abstract_inverted_index.API | 88 |
| abstract_inverted_index.and | 16, 49, 71, 83, 94, 98, 131, 144, 149, 153 |
| abstract_inverted_index.are | 146 |
| abstract_inverted_index.can | 26 |
| abstract_inverted_index.due | 29 |
| abstract_inverted_index.for | 89, 133 |
| abstract_inverted_index.the | 11, 17, 22, 34, 67, 126 |
| abstract_inverted_index.two | 58 |
| abstract_inverted_index.been | 105 |
| abstract_inverted_index.data | 97 |
| abstract_inverted_index.ease | 62, 80 |
| abstract_inverted_index.fast | 93 |
| abstract_inverted_index.from | 8 |
| abstract_inverted_index.have | 5, 104 |
| abstract_inverted_index.need | 35 |
| abstract_inverted_index.open | 147 |
| abstract_inverted_index.size | 12 |
| abstract_inverted_index.that | 61 |
| abstract_inverted_index.this | 54 |
| abstract_inverted_index.use, | 82 |
| abstract_inverted_index.used | 106 |
| abstract_inverted_index.well | 138 |
| abstract_inverted_index.when | 46 |
| abstract_inverted_index.with | 110, 118, 125 |
| abstract_inverted_index.Along | 124 |
| abstract_inverted_index.These | 101 |
| abstract_inverted_index.data, | 48 |
| abstract_inverted_index.data. | 123 |
| abstract_inverted_index.large | 73 |
| abstract_inverted_index.scale | 77 |
| abstract_inverted_index.these | 63 |
| abstract_inverted_index.train | 108 |
| abstract_inverted_index.while | 78 |
| abstract_inverted_index.work, | 55 |
| abstract_inverted_index.(e.g., | 42 |
| abstract_inverted_index.Recent | 0 |
| abstract_inverted_index.TPUs), | 43 |
| abstract_inverted_index.ensure | 50 |
| abstract_inverted_index.models | 4, 23, 75, 109, 136 |
| abstract_inverted_index.neural | 1 |
| abstract_inverted_index.number | 18 |
| abstract_inverted_index.simple | 90 |
| abstract_inverted_index.source | 148 |
| abstract_inverted_index.Scaling | 25 |
| abstract_inverted_index.T5-like | 134 |
| abstract_inverted_index.factors | 32 |
| abstract_inverted_index.greatly | 7 |
| abstract_inverted_index.issues: | 64 |
| abstract_inverted_index.present | 57 |
| abstract_inverted_index.prevent | 44 |
| abstract_inverted_index.process | 68 |
| abstract_inverted_index.release | 129 |
| abstract_inverted_index.scaling | 9 |
| abstract_inverted_index.various | 31 |
| abstract_inverted_index.GPT-like | 140 |
| abstract_inverted_index.billions | 113 |
| abstract_inverted_index.building | 70 |
| abstract_inverted_index.clusters | 41 |
| abstract_inverted_index.creation | 91 |
| abstract_inverted_index.datasets | 15, 117 |
| abstract_inverted_index.hundreds | 111 |
| abstract_inverted_index.language | 3, 74 |
| abstract_inverted_index.multiple | 119 |
| abstract_inverted_index.provides | 85 |
| abstract_inverted_index.results. | 52 |
| abstract_inverted_index.software | 59 |
| abstract_inverted_index.training | 14, 72, 96, 122 |
| abstract_inverted_index.available | 150 |
| abstract_inverted_index.benefited | 6 |
| abstract_inverted_index.including | 33 |
| abstract_inverted_index.infeeding | 47 |
| abstract_inverted_index.libraries | 60, 103 |
| abstract_inverted_index.terabytes | 120 |
| abstract_inverted_index.distribute | 37 |
| abstract_inverted_index.evaluation | 99 |
| abstract_inverted_index.libraries, | 127 |
| abstract_inverted_index.parameters | 20, 115 |
| abstract_inverted_index.pipelines. | 100 |
| abstract_inverted_index.simplifies | 66 |
| abstract_inverted_index.task-based | 87 |
| abstract_inverted_index.bottlenecks | 45 |
| abstract_inverted_index.complicated | 28 |
| abstract_inverted_index.computation | 38 |
| abstract_inverted_index.maintaining | 79 |
| abstract_inverted_index.open-source | 102 |
| abstract_inverted_index.themselves. | 24 |
| abstract_inverted_index.decoder-only | 141 |
| abstract_inverted_index.instructions | 132 |
| abstract_inverted_index.reproducible | 51, 95 |
| abstract_inverted_index.network-based | 2 |
| abstract_inverted_index.respectively. | 155 |
| abstract_inverted_index.supercomputer | 40 |
| abstract_inverted_index.$\texttt{t5x}$ | 65, 143 |
| abstract_inverted_index.architectures. | 142 |
| abstract_inverted_index.configurations | 130 |
| abstract_inverted_index.encoder-decoder | 135 |
| abstract_inverted_index.$\texttt{seqio}$ | 84, 145 |
| abstract_inverted_index.https://github.com/google/seqio, | 154 |
| abstract_inverted_index.https://github.com/google-research/t5x | 152 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 43 |
| sustainable_development_goals[0].id | https://metadata.un.org/sdg/9 |
| sustainable_development_goals[0].score | 0.4099999964237213 |
| sustainable_development_goals[0].display_name | Industry, innovation and infrastructure |
| citation_normalized_percentile |