Stephen Roller
YOU?
Author Swipe
View article: Enhancing Performance on Seen and Unseen Dialogue Scenarios using Retrieval-Augmented End-to-End Task-Oriented System
Enhancing Performance on Seen and Unseen Dialogue Scenarios using Retrieval-Augmented End-to-End Task-Oriented System Open
End-to-end task-oriented dialogue (TOD) systems have achieved promising performance by leveraging sophisticated natural language understanding and natural language generation capabilities of pre-trained models. This work enables the TOD sy…
View article: Leveraging Implicit Feedback from Deployment Data in Dialogue
Leveraging Implicit Feedback from Deployment Data in Dialogue Open
We study improving social conversational agents by learning from natural dialogue between users and a deployed model, without extra annotations. To implicitly measure the quality of a machine-generated utterance, we leverage signals like u…
View article: A Theory on Adam Instability in Large-Scale Machine Learning
A Theory on Adam Instability in Large-Scale Machine Learning Open
We present a theory for the previously unexplained divergent behavior noticed in the training of large language models. We argue that the phenomenon is an artifact of the dominant optimization algorithm used for training, called Adam. We o…
View article: Scaling Laws for Generative Mixed-Modal Language Models
Scaling Laws for Generative Mixed-Modal Language Models Open
Generative language models define distributions over sequences of tokens that can represent essentially any combination of data modalities (e.g., any permutation of image tokens from VQ-VAEs, speech tokens from HuBERT, BPE tokens for langu…
View article: Enhancing Performance on Seen and Unseen Dialogue Scenarios using Retrieval-Augmented End-to-End Task-Oriented System
Enhancing Performance on Seen and Unseen Dialogue Scenarios using Retrieval-Augmented End-to-End Task-Oriented System Open
Jianguo Zhang, Stephen Roller, Kun Qian, Zhiwei Liu, Rui Meng, Shelby Heinecke, Huan Wang, Silvio Savarese, Caiming Xiong. Proceedings of the 24th Meeting of the Special Interest Group on Discourse and Dialogue. 2023.
View article: Human-Level Play in the Game of Diplomacy by Combining Language Models with Strategic Reasoning
Human-Level Play in the Game of Diplomacy by Combining Language Models with Strategic Reasoning Open
Despite much progress in training AI systems to imitate human language, building agents that use language to communicate intentionally with humans in interactive environments remains a major challenge. We introduce CICERO, the first AI age…
View article: BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage Open
We present BlenderBot 3, a 175B parameter dialogue model capable of open-domain conversation with access to the internet and a long-term memory, and having been trained on a large number of user defined tasks. We release both the model wei…
View article: Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion
Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion Open
Language models (LMs) have recently been shown to generate more factual responses by employing modularity (Zhou et al., 2021) in combination with retrieval (Adolphs et al., 2021). We extend the recent approach of Adolphs et al. (2021) to i…
View article: Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents
Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents Open
At the heart of improving conversational AI is the open problem of how to evaluate conversations. Issues with automatic metrics are well known (Liu et al., 2016, arXiv:1603.08023), with human evaluations still considered the gold standard.…
View article: Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents
Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents Open
At the heart of improving conversational AI is the open problem of how to evaluate conversations. Issues with automatic metrics are well known (Liu et al., 2016), with human evaluations still considered the gold standard. Unfortunately, ho…
View article: Analysing Off-The-Shelf Options for Question Answering with Portuguese FAQs
Analysing Off-The-Shelf Options for Question Answering with Portuguese FAQs Open
Following the current interest in developing automatic question answering systems, we analyse alternative approaches for finding suitable answers from a list of Frequently Asked Questions (FAQs), in Portuguese. These rely on different tech…
View article: Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion
Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion Open
Language models (LMs) have recently been shown to generate more factual responses by employing modularity (Zhou et al., 2022) in combination with retrieval (Adolphs et al., 2021). We extend the recent approach of Adolphs et al. (2021) to i…
View article: Teaching Models new APIs: Domain-Agnostic Simulators for Task Oriented Dialogue
Teaching Models new APIs: Domain-Agnostic Simulators for Task Oriented Dialogue Open
We demonstrate that large language models are able to simulate Task Oriented Dialogues in novel domains, provided only with an API implementation and a list of goals. We show these simulations can formulate online, automatic metrics that c…
View article: Hash Layers For Large Sparse Models
Hash Layers For Large Sparse Models Open
We investigate the training of sparse layers that use different parameters for different inputs based on hashing in large Transformer models. Specifically, we modify the feedforward layer to hash to different sets of weights depending on t…
View article: Staircase Attention for Recurrent Processing of Sequences
Staircase Attention for Recurrent Processing of Sequences Open
Attention mechanisms have become a standard tool for sequence modeling tasks, in particular by stacking self-attention layers over the entire input sequence as in the Transformer architecture. In this work we introduce a novel attention pr…
View article: Not All Memories are Created Equal: Learning to Forget by Expiring
Not All Memories are Created Equal: Learning to Forget by Expiring Open
Attention mechanisms have shown promising results in sequence modeling tasks that require long-term memory. Recent work investigated mechanisms to reduce the computational cost of preserving and storing memories. However, not all content i…
View article: Adding Chit-Chat to Enhance Task-Oriented Dialogues
Adding Chit-Chat to Enhance Task-Oriented Dialogues Open
Kai Sun, Seungwhan Moon, Paul Crook, Stephen Roller, Becka Silvert, Bing Liu, Zhiguang Wang, Honglei Liu, Eunjoon Cho, Claire Cardie. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Lin…
View article: Recipes for Building an Open-Domain Chatbot
Recipes for Building an Open-Domain Chatbot Open
Building open-domain chatbots is a challenging area for machine learning research. While prior work has shown that scaling neural models in the number of parameters and the size of the data they are trained on gives improved results, we sh…
View article: Adding Chit-Chats to Enhance Task-Oriented Dialogues
Adding Chit-Chats to Enhance Task-Oriented Dialogues Open
Existing dialogue corpora and models are typically designed under two disjoint motives: while task-oriented systems focus on achieving functional goals (e.g., booking hotels), open-domain chatbots aim at making socially engaging conversati…
View article: Adding Chit-Chat to Enhance Task-Oriented Dialogues
Adding Chit-Chat to Enhance Task-Oriented Dialogues Open
Existing dialogue corpora and models are typically designed under two disjoint motives: while task-oriented systems focus on achieving functional goals (e.g., booking hotels), open-domain chatbots aim at making socially engaging conversati…
View article: Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions Open
We present our view of what is necessary to build an engaging open-domain conversational agent: covering the qualities of such an agent, the pieces of the puzzle that have been built so far, and the gaping holes we have not filled yet. We …
View article: The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents
The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents Open
We introduce dodecaDialogue: a set of 12 tasks that measures if a conversational agent can communicate engagingly with personality and empathy, ask questions, answer questions by utilizing knowledge resources, discuss topics and situations…
View article: Don’t Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training
Don’t Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training Open
Generative dialogue models currently suffer from a number of problems which standard maximum likelihood training does not address. They tend to produce generations that (i) rely too much on copying from the context, (ii) contain repetition…
View article: Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training
Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training Open
Generative dialogue models currently suffer from a number of problems which standard maximum likelihood training does not address. They tend to produce generations that (i) rely too much on copying from the context, (ii) contain repetition…
View article: The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded\n Conversational Agents
The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded\n Conversational Agents Open
We introduce dodecaDialogue: a set of 12 tasks that measures if a\nconversational agent can communicate engagingly with personality and empathy,\nask questions, answer questions by utilizing knowledge resources, discuss\ntopics and situati…
View article: ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons
ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons Open
While dialogue remains an important end-goal of natural language research, the difficulty of evaluation is an oft-quoted reason why it remains troublesome to make real progress towards its solution. Evaluation difficulties are actually two…
View article: Neural Text Generation with Unlikelihood Training
Neural Text Generation with Unlikelihood Training Open
Neural text generation is a key tool in natural language applications, but it is well known there are major problems at its core. In particular, standard likelihood training and decoding leads to dull and repetitive outputs. While some pos…
View article: What makes a good conversation? How controllable attributes affect human judgments
What makes a good conversation? How controllable attributes affect human judgments Open
A good conversation requires balance -- between simplicity and detail; staying on topic and changing it; asking questions and answering them. Although dialogue agents are commonly evaluated via human judgments of overall quality, the relat…
View article: Inferring Concept Hierarchies from Text Corpora via Hyperbolic Embeddings
Inferring Concept Hierarchies from Text Corpora via Hyperbolic Embeddings Open
We consider the task of inferring “is-a” relationships from large text corpora. For this purpose, we propose a new method combining hyperbolic embeddings and Hearst patterns. This approach allows us to set appropriate constraints for infer…