GLaM: Efficient Scaling of Language Models with Mixture-of-Experts Article Swipe
Nan Du
,
Yanping Huang
,
Andrew M. Dai
,
Simon Tong
,
Dmitry Lepikhin
,
Yuanzhong Xu
,
Maxim Krikun
,
Yanqi Zhou
,
Adams Wei Yu
,
Orhan Fırat
,
Barret Zoph
,
Liam Fedus
,
Maarten Bosma
,
Zongwei Zhou
,
Tao Wang
,
Yu Emma Wang
,
Kellie Webster
,
Marie Pellat
,
Kevin Robinson
,
Kathy Meier-Hellstern
,
Toju Duke
,
Lucas Dixon
,
Kun Zhang
,
Quoc V. Le
,
Yonghui Wu
,
Zhifeng Chen
,
Claire Cui
·
YOU?
·
· 2021
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2112.06905
YOU?
·
· 2021
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2112.06905
Scaling language models with more data, compute and parameters has driven significant progress in natural language processing. For example, thanks to scaling, GPT-3 was able to achieve strong results on in-context learning tasks. However, training these large dense models requires significant amounts of computing resources. In this paper, we propose and develop a family of language models named GLaM (Generalist Language Model), which uses a sparsely activated mixture-of-experts architecture to scale the model capacity while also incurring substantially less training cost compared to dense variants. The largest GLaM has 1.2 trillion parameters, which is approximately 7x larger than GPT-3. It consumes only 1/3 of the energy used to train GPT-3 and requires half of the computation flops for inference, while still achieving better overall zero-shot and one-shot performance across 29 NLP tasks.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2112.06905
- https://arxiv.org/pdf/2112.06905
- OA Status
- green
- Cited By
- 167
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4200634402
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4200634402Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2112.06905Digital Object Identifier
- Title
-
GLaM: Efficient Scaling of Language Models with Mixture-of-ExpertsWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2021Year of publication
- Publication date
-
2021-12-13Full publication date if available
- Authors
-
Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Fırat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V. Le, Yonghui Wu, Zhifeng Chen, Claire CuiList of authors in order
- Landing page
-
https://arxiv.org/abs/2112.06905Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2112.06905Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2112.06905Direct OA link when available
- Concepts
-
Computer science, Scaling, Inference, Language model, Computation, Context (archaeology), Artificial intelligence, Natural language processing, Scale (ratio), Machine learning, Algorithm, Mathematics, Physics, Paleontology, Biology, Quantum mechanics, GeometryTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
167Total citation count in OpenAlex
- Citations by year (recent)
-
2025: 32, 2024: 46, 2023: 69, 2022: 19, 2021: 1Per-year citation counts (last 5 years)
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4200634402 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2112.06905 |
| ids.doi | https://doi.org/10.48550/arxiv.2112.06905 |
| ids.openalex | https://openalex.org/W4200634402 |
| fwci | |
| type | preprint |
| title | GLaM: Efficient Scaling of Language Models with Mixture-of-Experts |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10028 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9993000030517578 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Topic Modeling |
| topics[1].id | https://openalex.org/T10181 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9962999820709229 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Natural Language Processing Techniques |
| topics[2].id | https://openalex.org/T11714 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9850000143051147 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1707 |
| topics[2].subfield.display_name | Computer Vision and Pattern Recognition |
| topics[2].display_name | Multimodal Machine Learning Applications |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C41008148 |
| concepts[0].level | 0 |
| concepts[0].score | 0.7689605951309204 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[0].display_name | Computer science |
| concepts[1].id | https://openalex.org/C99844830 |
| concepts[1].level | 2 |
| concepts[1].score | 0.7518633008003235 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q102441924 |
| concepts[1].display_name | Scaling |
| concepts[2].id | https://openalex.org/C2776214188 |
| concepts[2].level | 2 |
| concepts[2].score | 0.7222841382026672 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q408386 |
| concepts[2].display_name | Inference |
| concepts[3].id | https://openalex.org/C137293760 |
| concepts[3].level | 2 |
| concepts[3].score | 0.6706558465957642 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q3621696 |
| concepts[3].display_name | Language model |
| concepts[4].id | https://openalex.org/C45374587 |
| concepts[4].level | 2 |
| concepts[4].score | 0.5918241143226624 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q12525525 |
| concepts[4].display_name | Computation |
| concepts[5].id | https://openalex.org/C2779343474 |
| concepts[5].level | 2 |
| concepts[5].score | 0.5844715237617493 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q3109175 |
| concepts[5].display_name | Context (archaeology) |
| concepts[6].id | https://openalex.org/C154945302 |
| concepts[6].level | 1 |
| concepts[6].score | 0.5299583077430725 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[6].display_name | Artificial intelligence |
| concepts[7].id | https://openalex.org/C204321447 |
| concepts[7].level | 1 |
| concepts[7].score | 0.4932956099510193 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q30642 |
| concepts[7].display_name | Natural language processing |
| concepts[8].id | https://openalex.org/C2778755073 |
| concepts[8].level | 2 |
| concepts[8].score | 0.4125417172908783 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q10858537 |
| concepts[8].display_name | Scale (ratio) |
| concepts[9].id | https://openalex.org/C119857082 |
| concepts[9].level | 1 |
| concepts[9].score | 0.357910692691803 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q2539 |
| concepts[9].display_name | Machine learning |
| concepts[10].id | https://openalex.org/C11413529 |
| concepts[10].level | 1 |
| concepts[10].score | 0.21114492416381836 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q8366 |
| concepts[10].display_name | Algorithm |
| concepts[11].id | https://openalex.org/C33923547 |
| concepts[11].level | 0 |
| concepts[11].score | 0.09706851840019226 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q395 |
| concepts[11].display_name | Mathematics |
| concepts[12].id | https://openalex.org/C121332964 |
| concepts[12].level | 0 |
| concepts[12].score | 0.0 |
| concepts[12].wikidata | https://www.wikidata.org/wiki/Q413 |
| concepts[12].display_name | Physics |
| concepts[13].id | https://openalex.org/C151730666 |
| concepts[13].level | 1 |
| concepts[13].score | 0.0 |
| concepts[13].wikidata | https://www.wikidata.org/wiki/Q7205 |
| concepts[13].display_name | Paleontology |
| concepts[14].id | https://openalex.org/C86803240 |
| concepts[14].level | 0 |
| concepts[14].score | 0.0 |
| concepts[14].wikidata | https://www.wikidata.org/wiki/Q420 |
| concepts[14].display_name | Biology |
| concepts[15].id | https://openalex.org/C62520636 |
| concepts[15].level | 1 |
| concepts[15].score | 0.0 |
| concepts[15].wikidata | https://www.wikidata.org/wiki/Q944 |
| concepts[15].display_name | Quantum mechanics |
| concepts[16].id | https://openalex.org/C2524010 |
| concepts[16].level | 1 |
| concepts[16].score | 0.0 |
| concepts[16].wikidata | https://www.wikidata.org/wiki/Q8087 |
| concepts[16].display_name | Geometry |
| keywords[0].id | https://openalex.org/keywords/computer-science |
| keywords[0].score | 0.7689605951309204 |
| keywords[0].display_name | Computer science |
| keywords[1].id | https://openalex.org/keywords/scaling |
| keywords[1].score | 0.7518633008003235 |
| keywords[1].display_name | Scaling |
| keywords[2].id | https://openalex.org/keywords/inference |
| keywords[2].score | 0.7222841382026672 |
| keywords[2].display_name | Inference |
| keywords[3].id | https://openalex.org/keywords/language-model |
| keywords[3].score | 0.6706558465957642 |
| keywords[3].display_name | Language model |
| keywords[4].id | https://openalex.org/keywords/computation |
| keywords[4].score | 0.5918241143226624 |
| keywords[4].display_name | Computation |
| keywords[5].id | https://openalex.org/keywords/context |
| keywords[5].score | 0.5844715237617493 |
| keywords[5].display_name | Context (archaeology) |
| keywords[6].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[6].score | 0.5299583077430725 |
| keywords[6].display_name | Artificial intelligence |
| keywords[7].id | https://openalex.org/keywords/natural-language-processing |
| keywords[7].score | 0.4932956099510193 |
| keywords[7].display_name | Natural language processing |
| keywords[8].id | https://openalex.org/keywords/scale |
| keywords[8].score | 0.4125417172908783 |
| keywords[8].display_name | Scale (ratio) |
| keywords[9].id | https://openalex.org/keywords/machine-learning |
| keywords[9].score | 0.357910692691803 |
| keywords[9].display_name | Machine learning |
| keywords[10].id | https://openalex.org/keywords/algorithm |
| keywords[10].score | 0.21114492416381836 |
| keywords[10].display_name | Algorithm |
| keywords[11].id | https://openalex.org/keywords/mathematics |
| keywords[11].score | 0.09706851840019226 |
| keywords[11].display_name | Mathematics |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2112.06905 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2112.06905 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2112.06905 |
| locations[1].id | doi:10.48550/arxiv.2112.06905 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | cc-by |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | https://openalex.org/licenses/cc-by |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2112.06905 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5100721790 |
| authorships[0].author.orcid | https://orcid.org/0000-0003-2855-7452 |
| authorships[0].author.display_name | Nan Du |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Du, Nan |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5033809109 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-4430-3965 |
| authorships[1].author.display_name | Yanping Huang |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Huang, Yanping |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5101597225 |
| authorships[2].author.orcid | https://orcid.org/0009-0007-9200-8577 |
| authorships[2].author.display_name | Andrew M. Dai |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Dai, Andrew M. |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5063584465 |
| authorships[3].author.orcid | |
| authorships[3].author.display_name | Simon Tong |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Tong, Simon |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5049203448 |
| authorships[4].author.orcid | |
| authorships[4].author.display_name | Dmitry Lepikhin |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Lepikhin, Dmitry |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5081428852 |
| authorships[5].author.orcid | |
| authorships[5].author.display_name | Yuanzhong Xu |
| authorships[5].author_position | middle |
| authorships[5].raw_author_name | Xu, Yuanzhong |
| authorships[5].is_corresponding | False |
| authorships[6].author.id | https://openalex.org/A5014055962 |
| authorships[6].author.orcid | |
| authorships[6].author.display_name | Maxim Krikun |
| authorships[6].author_position | middle |
| authorships[6].raw_author_name | Krikun, Maxim |
| authorships[6].is_corresponding | False |
| authorships[7].author.id | https://openalex.org/A5081567725 |
| authorships[7].author.orcid | https://orcid.org/0000-0003-2051-7616 |
| authorships[7].author.display_name | Yanqi Zhou |
| authorships[7].author_position | middle |
| authorships[7].raw_author_name | Zhou, Yanqi |
| authorships[7].is_corresponding | False |
| authorships[8].author.id | https://openalex.org/A5045729519 |
| authorships[8].author.orcid | |
| authorships[8].author.display_name | Adams Wei Yu |
| authorships[8].author_position | middle |
| authorships[8].raw_author_name | Yu, Adams Wei |
| authorships[8].is_corresponding | False |
| authorships[9].author.id | https://openalex.org/A5035914396 |
| authorships[9].author.orcid | https://orcid.org/0000-0001-5775-2420 |
| authorships[9].author.display_name | Orhan Fırat |
| authorships[9].author_position | middle |
| authorships[9].raw_author_name | Firat, Orhan |
| authorships[9].is_corresponding | False |
| authorships[10].author.id | https://openalex.org/A5026064427 |
| authorships[10].author.orcid | |
| authorships[10].author.display_name | Barret Zoph |
| authorships[10].author_position | middle |
| authorships[10].raw_author_name | Zoph, Barret |
| authorships[10].is_corresponding | False |
| authorships[11].author.id | https://openalex.org/A5029678131 |
| authorships[11].author.orcid | |
| authorships[11].author.display_name | Liam Fedus |
| authorships[11].author_position | middle |
| authorships[11].raw_author_name | Fedus, Liam |
| authorships[11].is_corresponding | False |
| authorships[12].author.id | https://openalex.org/A5074322007 |
| authorships[12].author.orcid | |
| authorships[12].author.display_name | Maarten Bosma |
| authorships[12].author_position | middle |
| authorships[12].raw_author_name | Bosma, Maarten |
| authorships[12].is_corresponding | False |
| authorships[13].author.id | https://openalex.org/A5084104975 |
| authorships[13].author.orcid | https://orcid.org/0000-0002-3154-9851 |
| authorships[13].author.display_name | Zongwei Zhou |
| authorships[13].author_position | middle |
| authorships[13].raw_author_name | Zhou, Zongwei |
| authorships[13].is_corresponding | False |
| authorships[14].author.id | https://openalex.org/A5101549822 |
| authorships[14].author.orcid | https://orcid.org/0000-0002-7268-8671 |
| authorships[14].author.display_name | Tao Wang |
| authorships[14].author_position | middle |
| authorships[14].raw_author_name | Wang, Tao |
| authorships[14].is_corresponding | False |
| authorships[15].author.id | https://openalex.org/A5024099060 |
| authorships[15].author.orcid | |
| authorships[15].author.display_name | Yu Emma Wang |
| authorships[15].author_position | middle |
| authorships[15].raw_author_name | Wang, Yu Emma |
| authorships[15].is_corresponding | False |
| authorships[16].author.id | https://openalex.org/A5088842522 |
| authorships[16].author.orcid | |
| authorships[16].author.display_name | Kellie Webster |
| authorships[16].author_position | middle |
| authorships[16].raw_author_name | Webster, Kellie |
| authorships[16].is_corresponding | False |
| authorships[17].author.id | https://openalex.org/A5059267294 |
| authorships[17].author.orcid | |
| authorships[17].author.display_name | Marie Pellat |
| authorships[17].author_position | middle |
| authorships[17].raw_author_name | Pellat, Marie |
| authorships[17].is_corresponding | False |
| authorships[18].author.id | https://openalex.org/A5036699396 |
| authorships[18].author.orcid | |
| authorships[18].author.display_name | Kevin Robinson |
| authorships[18].author_position | middle |
| authorships[18].raw_author_name | Robinson, Kevin |
| authorships[18].is_corresponding | False |
| authorships[19].author.id | https://openalex.org/A5010819457 |
| authorships[19].author.orcid | |
| authorships[19].author.display_name | Kathy Meier-Hellstern |
| authorships[19].author_position | middle |
| authorships[19].raw_author_name | Meier-Hellstern, Kathy |
| authorships[19].is_corresponding | False |
| authorships[20].author.id | https://openalex.org/A5041537705 |
| authorships[20].author.orcid | |
| authorships[20].author.display_name | Toju Duke |
| authorships[20].author_position | middle |
| authorships[20].raw_author_name | Duke, Toju |
| authorships[20].is_corresponding | False |
| authorships[21].author.id | https://openalex.org/A5103159534 |
| authorships[21].author.orcid | https://orcid.org/0000-0003-1094-1675 |
| authorships[21].author.display_name | Lucas Dixon |
| authorships[21].author_position | middle |
| authorships[21].raw_author_name | Dixon, Lucas |
| authorships[21].is_corresponding | False |
| authorships[22].author.id | https://openalex.org/A5100342310 |
| authorships[22].author.orcid | https://orcid.org/0000-0002-0743-9003 |
| authorships[22].author.display_name | Kun Zhang |
| authorships[22].author_position | middle |
| authorships[22].raw_author_name | Zhang, Kun |
| authorships[22].is_corresponding | False |
| authorships[23].author.id | https://openalex.org/A5088551093 |
| authorships[23].author.orcid | https://orcid.org/0000-0002-1087-2844 |
| authorships[23].author.display_name | Quoc V. Le |
| authorships[23].author_position | middle |
| authorships[23].raw_author_name | Le, Quoc V |
| authorships[23].is_corresponding | False |
| authorships[24].author.id | https://openalex.org/A5102302031 |
| authorships[24].author.orcid | |
| authorships[24].author.display_name | Yonghui Wu |
| authorships[24].author_position | middle |
| authorships[24].raw_author_name | Wu, Yonghui |
| authorships[24].is_corresponding | False |
| authorships[25].author.id | https://openalex.org/A5100715425 |
| authorships[25].author.orcid | https://orcid.org/0000-0001-8901-1002 |
| authorships[25].author.display_name | Zhifeng Chen |
| authorships[25].author_position | middle |
| authorships[25].raw_author_name | Chen, Zhifeng |
| authorships[25].is_corresponding | False |
| authorships[26].author.id | https://openalex.org/A5077034022 |
| authorships[26].author.orcid | https://orcid.org/0000-0001-5422-6468 |
| authorships[26].author.display_name | Claire Cui |
| authorships[26].author_position | last |
| authorships[26].raw_author_name | Cui, Claire |
| authorships[26].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2112.06905 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | GLaM: Efficient Scaling of Language Models with Mixture-of-Experts |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10028 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9993000030517578 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Topic Modeling |
| related_works | https://openalex.org/W141820298, https://openalex.org/W4378770497, https://openalex.org/W2049584446, https://openalex.org/W2079781215, https://openalex.org/W4308245303, https://openalex.org/W2014033564, https://openalex.org/W2055243143, https://openalex.org/W2910573937, https://openalex.org/W4385571583, https://openalex.org/W4389519396 |
| cited_by_count | 167 |
| counts_by_year[0].year | 2025 |
| counts_by_year[0].cited_by_count | 32 |
| counts_by_year[1].year | 2024 |
| counts_by_year[1].cited_by_count | 46 |
| counts_by_year[2].year | 2023 |
| counts_by_year[2].cited_by_count | 69 |
| counts_by_year[3].year | 2022 |
| counts_by_year[3].cited_by_count | 19 |
| counts_by_year[4].year | 2021 |
| counts_by_year[4].cited_by_count | 1 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2112.06905 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2112.06905 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2112.06905 |
| primary_location.id | pmh:oai:arXiv.org:2112.06905 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2112.06905 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2112.06905 |
| publication_date | 2021-12-13 |
| publication_year | 2021 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 52, 64 |
| abstract_inverted_index.29 | 129 |
| abstract_inverted_index.7x | 95 |
| abstract_inverted_index.In | 45 |
| abstract_inverted_index.It | 99 |
| abstract_inverted_index.in | 13 |
| abstract_inverted_index.is | 93 |
| abstract_inverted_index.of | 42, 54, 103, 113 |
| abstract_inverted_index.on | 29 |
| abstract_inverted_index.to | 20, 25, 69, 82, 107 |
| abstract_inverted_index.we | 48 |
| abstract_inverted_index.1.2 | 89 |
| abstract_inverted_index.1/3 | 102 |
| abstract_inverted_index.For | 17 |
| abstract_inverted_index.NLP | 130 |
| abstract_inverted_index.The | 85 |
| abstract_inverted_index.and | 7, 50, 110, 125 |
| abstract_inverted_index.for | 117 |
| abstract_inverted_index.has | 9, 88 |
| abstract_inverted_index.the | 71, 104, 114 |
| abstract_inverted_index.was | 23 |
| abstract_inverted_index.GLaM | 58, 87 |
| abstract_inverted_index.able | 24 |
| abstract_inverted_index.also | 75 |
| abstract_inverted_index.cost | 80 |
| abstract_inverted_index.half | 112 |
| abstract_inverted_index.less | 78 |
| abstract_inverted_index.more | 4 |
| abstract_inverted_index.only | 101 |
| abstract_inverted_index.than | 97 |
| abstract_inverted_index.this | 46 |
| abstract_inverted_index.used | 106 |
| abstract_inverted_index.uses | 63 |
| abstract_inverted_index.with | 3 |
| abstract_inverted_index.GPT-3 | 22, 109 |
| abstract_inverted_index.data, | 5 |
| abstract_inverted_index.dense | 37, 83 |
| abstract_inverted_index.flops | 116 |
| abstract_inverted_index.large | 36 |
| abstract_inverted_index.model | 72 |
| abstract_inverted_index.named | 57 |
| abstract_inverted_index.scale | 70 |
| abstract_inverted_index.still | 120 |
| abstract_inverted_index.these | 35 |
| abstract_inverted_index.train | 108 |
| abstract_inverted_index.which | 62, 92 |
| abstract_inverted_index.while | 74, 119 |
| abstract_inverted_index.GPT-3. | 98 |
| abstract_inverted_index.across | 128 |
| abstract_inverted_index.better | 122 |
| abstract_inverted_index.driven | 10 |
| abstract_inverted_index.energy | 105 |
| abstract_inverted_index.family | 53 |
| abstract_inverted_index.larger | 96 |
| abstract_inverted_index.models | 2, 38, 56 |
| abstract_inverted_index.paper, | 47 |
| abstract_inverted_index.strong | 27 |
| abstract_inverted_index.tasks. | 32, 131 |
| abstract_inverted_index.thanks | 19 |
| abstract_inverted_index.Model), | 61 |
| abstract_inverted_index.Scaling | 0 |
| abstract_inverted_index.achieve | 26 |
| abstract_inverted_index.amounts | 41 |
| abstract_inverted_index.compute | 6 |
| abstract_inverted_index.develop | 51 |
| abstract_inverted_index.largest | 86 |
| abstract_inverted_index.natural | 14 |
| abstract_inverted_index.overall | 123 |
| abstract_inverted_index.propose | 49 |
| abstract_inverted_index.results | 28 |
| abstract_inverted_index.However, | 33 |
| abstract_inverted_index.Language | 60 |
| abstract_inverted_index.capacity | 73 |
| abstract_inverted_index.compared | 81 |
| abstract_inverted_index.consumes | 100 |
| abstract_inverted_index.example, | 18 |
| abstract_inverted_index.language | 1, 15, 55 |
| abstract_inverted_index.learning | 31 |
| abstract_inverted_index.one-shot | 126 |
| abstract_inverted_index.progress | 12 |
| abstract_inverted_index.requires | 39, 111 |
| abstract_inverted_index.scaling, | 21 |
| abstract_inverted_index.sparsely | 65 |
| abstract_inverted_index.training | 34, 79 |
| abstract_inverted_index.trillion | 90 |
| abstract_inverted_index.achieving | 121 |
| abstract_inverted_index.activated | 66 |
| abstract_inverted_index.computing | 43 |
| abstract_inverted_index.incurring | 76 |
| abstract_inverted_index.variants. | 84 |
| abstract_inverted_index.zero-shot | 124 |
| abstract_inverted_index.in-context | 30 |
| abstract_inverted_index.inference, | 118 |
| abstract_inverted_index.parameters | 8 |
| abstract_inverted_index.resources. | 44 |
| abstract_inverted_index.(Generalist | 59 |
| abstract_inverted_index.computation | 115 |
| abstract_inverted_index.parameters, | 91 |
| abstract_inverted_index.performance | 127 |
| abstract_inverted_index.processing. | 16 |
| abstract_inverted_index.significant | 11, 40 |
| abstract_inverted_index.architecture | 68 |
| abstract_inverted_index.approximately | 94 |
| abstract_inverted_index.substantially | 77 |
| abstract_inverted_index.mixture-of-experts | 67 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 27 |
| sustainable_development_goals[0].id | https://metadata.un.org/sdg/4 |
| sustainable_development_goals[0].score | 0.699999988079071 |
| sustainable_development_goals[0].display_name | Quality Education |
| citation_normalized_percentile |