Towards Stability of Autoregressive Neural Operators Article Swipe

MAKE IMPACT

Michael T. McCabe , Peter de B. Harrington , Shashank Subramanian , Jed Brown · YOU? · · 2023 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2306.10619

Neural operators have proven to be a promising approach for modeling spatiotemporal systems in the physical sciences. However, training these models for large systems can be quite challenging as they incur significant computational and memory expense -- these systems are often forced to rely on autoregressive time-stepping of the neural network to predict future temporal states. While this is effective in managing costs, it can lead to uncontrolled error growth over time and eventual instability. We analyze the sources of this autoregressive error growth using prototypical neural operator models for physical systems and explore ways to mitigate it. We introduce architectural and application-specific improvements that allow for careful control of instability-inducing operations within these models without inflating the compute/memory expense. We present results on several scientific systems that include Navier-Stokes fluid flow, rotating shallow water, and a high-resolution global weather forecasting system. We demonstrate that applying our design principles to neural operators leads to significantly lower errors for long-term forecasts as well as longer time horizons without qualitative signs of divergence compared to the original models for these systems. We open-source our \href{https://github.com/mikemccabe210/stabilizing_neural_operators}{code} for reproducibility.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2306.10619

PDF

https://arxiv.org/pdf/2306.10619

OA Status

green

Cited By

2

Related Works

10

OpenAlex ID

https://openalex.org/W4381564169

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4381564169

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2306.10619

Digital Object Identifier

Title

Towards Stability of Autoregressive Neural Operators

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2023

Year of publication

Publication date

2023-06-18

Full publication date if available

Authors

Michael T. McCabe, Peter de B. Harrington, Shashank Subramanian, Jed Brown

List of authors in order

Landing page

https://arxiv.org/abs/2306.10619

Publisher landing page

PDF URL

https://arxiv.org/pdf/2306.10619

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2306.10619

Direct OA link when available

Concepts

Autoregressive model, Computer science, Artificial neural network, Divergence (linguistics), Stability (learning theory), Operator (biology), Recurrent neural network, Artificial intelligence, Machine learning, Mathematics, Econometrics, Philosophy, Gene, Chemistry, Repressor, Biochemistry, Linguistics, Transcription factor

Top concepts (fields/topics) attached by OpenAlex

Cited by

2

Total citation count in OpenAlex

Citations by year (recent)

2025: 2

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload