Relational Memory Augmented Language Models Article Swipe

PDF

Related Concepts

Perplexity Computer science Language model Graph Artificial intelligence Autoregressive model Natural language processing Coherence (philosophical gambling strategy) Theoretical computer science Character (mathematics) Mathematics Geometry Econometrics Statistics

Qi Liu , Dani Yogatama , Phil Blunsom ·

YOU? · · 2022 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2201.09680 · OA: W4285661761

We present a memory-augmented approach to condition an autoregressive language model on a knowledge graph. We represent the graph as a collection of relation triples and retrieve relevant relations for a given context to improve text generation. Experiments on WikiText-103, WMT19, and enwik8 English datasets demonstrate that our approach produces a better language model in terms of perplexity and bits per character. We also show that relational memory improves coherence, is complementary to token-based memory, and enables causal interventions. Our model provides a simple yet effective way to combine an autoregressive language model with a knowledge graph for a more coherent and logical generation.