Exploring foci of:
Findings of the Association for Computational Linguistics: NAACL 2022
LongT5: Efficient Text-To-Text Transformer for Long Sequences
January 2022 • Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontañón, Jianmo Ni, Yun-Hsuan Sung, Yinfei Yang
Recent work has shown that either (1) increasing the input length or (2) increasing model size can improve the performance of Transformer-based neural models. In this paper, we present LongT5, a new model that explores the effects of scaling both the input length and model size at the same time. Specifically, we integrate attention ideas from long-input transformers (ETC), and adopt pre-training strategies from summarization pre-training (PEGASUS) into the scalable T5 architecture. The result is a new attention me…
Open Article Page
Computer Science
Transformer
Architecture
Artificial Intelligence
Machine Learning
Database
Engineering
Art
Visual Arts
Open Article
Electrical Engineering
Voltage
Open Article