Natural language understanding ≈ Natural language understanding
View article
Exploring the Limits of Transfer Learning with a Unified Text-to-Text\n Transformer Open
Transfer learning, where a model is first pre-trained on a data-rich task\nbefore being fine-tuned on a downstream task, has emerged as a powerful\ntechnique in natural language processing (NLP). The effectiveness of transfer\nlearning has…
View article
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Open
Human ability to understand language is general, flexible, and robust. In contrast, most NLU models above the word level are designed for a specific task and struggle with out-of-domain data. If we aspire to develop models with understandi…
View article
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Open
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has gi…
View article
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing Open
This article surveys and organizes research works in a new paradigm in natural language processing, which we dub “prompt-based learning.” Unlike traditional supervised learning, which trains a model to take in an input x and predict an out…
View article
PaLM: Scaling Language Modeling with Pathways Open
Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically reduces the number of task-specific training examples needed to adapt the model t…
View article
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks Open
We present ViLBERT (short for Vision-and-Language BERT), a model for learning task-agnostic joint representations of image content and natural language. We extend the popular BERT architecture to a multi-modal two-stream model, pro-cessing…
View article
TinyBERT: Distilling BERT for Natural Language Understanding Open
Language model pre-training, such as BERT, has significantly improved the performances of many natural language processing tasks. However, pre-trained language models are usually computationally expensive, so it is difficult to efficiently…
View article
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts Open
The remarkable success of pretrained language models has motivated the study of what kinds of knowledge these models learn during pretraining. Reformulating tasks as fill-in-the-blanks problems (e.g., cloze tests) is a natural approach for…
View article
Multi-Task Deep Neural Networks for Natural Language Understanding Open
In this paper, we present a Multi-Task Deep Neural Network (MT-DNN) for learning representations across multiple natural language understanding (NLU) tasks. MT-DNN not only leverages large amounts of cross-task data, but also benefits from…
View article
Unified Language Model Pre-training for Natural Language Understanding and Generation Open
This paper presents a new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks. The model is pre-trained using three types of language modeling tasks: unidirectional…
View article
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data Open
The success of the large neural language models on many NLP tasks is exciting. However, we find that these successes sometimes lead to hype in which these models are being described as “understanding” language or capturing “meaning”. In th…
View article
COMET: Commonsense Transformers for Automatic Knowledge Graph Construction Open
We present the first comprehensive study on automatic knowledge base construction for two prevalent commonsense knowledge graphs: ATOMIC (Sap et al., 2019) and ConceptNet (Speer et al., 2017). Contrary to many conventional KBs that store k…
View article
ChatGPT and Open-AI Models: A Preliminary Review Open
According to numerous reports, ChatGPT represents a significant breakthrough in the field of artificial intelligence. ChatGPT is a pre-trained AI model designed to engage in natural language conversations, utilizing sophisticated technique…
View article
Attention in Natural Language Processing Open
Attention is an increasingly popular mechanism used in a wide range of neural architectures. The mechanism itself has been realized in a variety of formats. However, because of the fast-paced advances in this domain, a systematic overview …
View article
Multitask Prompted Training Enables Zero-Shot Task Generalization Open
Large language models have recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks (Brown et al., 2020). It has been hypothesized that this is a consequence of implicit multitask learning in language mod…
View article
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Open
For natural language understanding (NLU) technology to be maximally useful, both practically and as a scientific object of study, it must be general: it must be able to process language in a way that is not exclusively tailored to any one …
View article
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing Open
This paper surveys and organizes research works in a new paradigm in natural language processing, which we dub "prompt-based learning". Unlike traditional supervised learning, which trains a model to take in an input x and predict an outpu…
View article
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances Open
Large language models can encode a wealth of semantic knowledge about the world. Such knowledge could be extremely useful to robots aiming to act upon high-level, temporally extended instructions expressed in natural language. However, a s…
View article
AllenNLP: A Deep Semantic Natural Language Processing Platform Open
This paper describes AllenNLP, a platform for research on deep learning methods in natural language understanding. AllenNLP is designed to support researchers who want to build novel language understanding models quickly and easily. It is …
View article
GPT (Generative Pre-Trained Transformer)— A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions Open
The Generative Pre-trained Transformer (GPT) represents a notable breakthrough in the domain of natural language processing, which is propelling us toward the development of machines that can understand and communicate using language in a …
View article
Is ChatGPT a General-Purpose Natural Language Processing Task Solver? Open
Spurred by advancements in scale, large language models (LLMs) have demonstrated the ability to perform a variety of natural language processing (NLP) tasks zero-shot—i.e., without adaptation on downstream data. Recently, the debut of Chat…
View article
DeBERTa: Decoding-enhanced BERT with Disentangled Attention Open
Recent progress in pre-trained neural language models has significantly improved the performance of many natural language processing (NLP) tasks. In this paper we propose a new model architecture DeBERTa (Decoding-enhanced BERT with disent…
View article
SenticNet 5: Discovering Conceptual Primitives for Sentiment Analysis by Means of Context Embeddings Open
With the recent development of deep learning, research in AI has gained new vigor and prominence. While machine learning has succeeded in revitalizing many research fields, such as computer vision, speech recognition, and medical diagnosis…
View article
Shared computational principles for language processing in humans and deep language models Open
Departing from traditional linguistic models, advances in deep learning have resulted in a new type of predictive (autoregressive) deep language models (DLMs). Using a self-supervised next-word prediction task, these models generate approp…
View article
Argument Mining: A Survey Open
Argument mining is the automatic identification and extraction of the structure of inference and reasoning expressed as arguments presented in natural language. Understanding argumentative structure makes it possible to determine not only …
View article
Snips Voice Platform: an embedded Spoken Language Understanding system\n for private-by-design voice interfaces Open
This paper presents the machine learning architecture of the Snips Voice\nPlatform, a software solution to perform Spoken Language Understanding on\nmicroprocessors typical of IoT devices. The embedded inference is fast and\naccurate while…
View article
Semantics-Aware BERT for Language Understanding Open
The latest work on language representations carefully integrates contextualized features into language model training, which enables a series of success especially in various machine reading comprehension and natural language inference tas…
View article
Distilling Task-Specific Knowledge from BERT into Simple Neural Networks Open
In the natural language processing literature, neural networks are becoming increasingly deeper and complex. The recent poster child of this trend is the deep language representation model, which includes BERT, ELMo, and GPT. These develop…
View article
Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison Open
Word Sense Disambiguation is a longstanding task in Natural Language Processing, lying at the core of human language understanding. However, the evaluation of automatic systems has been problematic, mainly due to the lack of a reliable eva…
View article
Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces Open
This paper presents the machine learning architecture of the Snips Voice Platform, a software solution to perform Spoken Language Understanding on microprocessors typical of IoT devices. The embedded inference is fast and accurate while en…