Complex Momentum for Optimization in Games Article Swipe

PDF

Related Concepts

Momentum (technical analysis) Differentiable function Convergence (economics) Bilinear interpolation Generative grammar Generalization Adversarial system Computer science Mathematical optimization Perspective (graphical) Gradient descent Applied mathematics Mathematics Algorithm Artificial intelligence Pure mathematics Artificial neural network Mathematical analysis Economics Computer vision Economic growth Finance

Jonathan Lorraine , David Acuna , Paul Vicol , David Duvenaud ·

YOU? · · 2021 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2102.08431 · OA: W3166610402

We generalize gradient descent with momentum for optimization in differentiable games to have complex-valued momentum. We give theoretical motivation for our method by proving convergence on bilinear zero-sum games for simultaneous and alternating updates. Our method gives real-valued parameter updates, making it a drop-in replacement for standard optimizers. We empirically demonstrate that complex-valued momentum can improve convergence in realistic adversarial games - like generative adversarial networks - by showing we can find better solutions with an almost identical computational cost. We also show a practical generalization to a complex-valued Adam variant, which we use to train BigGAN to better inception scores on CIFAR-10.