SLE-FNO: Single-Layer Extensions for Task-Agnostic Continual Learning in Fourier Neural Operators
arXiv:2603.20410v1 Announce Type: new Abstract: Scientific machine learning is increasingly used to build surrogate models, yet most models are trained under a restrictive assumption in which future data follow the same distribution as the training set. In practice, new experimental conditions or simulation regimes may differ significantly, requiring extrapolation and model updates without re-access to prior data. This creates a need for continual learning (CL) frameworks that can adapt to distribution shifts while preventing catastrophic forgetting. Such challenges are pronounced in fluid dynamics, where changes in geometry, boundary conditions, or flow regimes induce non-trivial changes to the solution. Here, we introduce a new architecture-based approach (SLE-FNO) combining a Single-Layer Extension (SLE) with the Fourier Neural Operator (FNO) to support efficient CL. SLE-FNO was compared with a range of established CL methods, including Elastic Weight Consolidation (EWC), Learning
arXiv:2603.20410v1 Announce Type: new Abstract: Scientific machine learning is increasingly used to build surrogate models, yet most models are trained under a restrictive assumption in which future data follow the same distribution as the training set. In practice, new experimental conditions or simulation regimes may differ significantly, requiring extrapolation and model updates without re-access to prior data. This creates a need for continual learning (CL) frameworks that can adapt to distribution shifts while preventing catastrophic forgetting. Such challenges are pronounced in fluid dynamics, where changes in geometry, boundary conditions, or flow regimes induce non-trivial changes to the solution. Here, we introduce a new architecture-based approach (SLE-FNO) combining a Single-Layer Extension (SLE) with the Fourier Neural Operator (FNO) to support efficient CL. SLE-FNO was compared with a range of established CL methods, including Elastic Weight Consolidation (EWC), Learning without Forgetting (LwF), replay-based approaches, Orthogonal Gradient Descent (OGD), Gradient Episodic Memory (GEM), PiggyBack, and Low-Rank Approximation (LoRA), within an image-to-image regression setting. The models were trained to map transient concentration fields to time-averaged wall shear stress (TAWSS) in pulsatile aneurysmal blood flow. Tasks were derived from 230 computational fluid dynamics simulations grouped into four sequential and out-of-distribution configurations. Results show that replay-based methods and architecture-based approaches (PiggyBack, LoRA, and SLE-FNO) achieve the best retention, with SLE-FNO providing the strongest overall balance between plasticity and stability, achieving accuracy with zero forgetting and minimal additional parameters. Our findings highlight key differences between CL algorithms and introduce SLE-FNO as a promising strategy for adapting baseline models when extrapolation is required.
Executive Summary
This article introduces a novel architecture-based approach, SLE-FNO, for task-agnostic continual learning in Fourier Neural Operators. SLE-FNO combines a Single-Layer Extension with the Fourier Neural Operator to efficiently adapt to distribution shifts in fluid dynamics simulations. The authors compare SLE-FNO with established CL methods and demonstrate its superiority in terms of plasticity and stability. The findings suggest that replay-based methods and architecture-based approaches are effective in retaining knowledge, with SLE-FNO providing the best balance between accuracy and parameter efficiency. The article highlights the need for adaptable CL frameworks in scientific machine learning and introduces SLE-FNO as a promising strategy for extrapolation and model updates. The study's results have significant implications for the development of robust and efficient CL methods in various domains.
Key Points
- ▸ SLE-FNO is a novel architecture-based approach for task-agnostic continual learning in Fourier Neural Operators.
- ▸ SLE-FNO combines a Single-Layer Extension with the Fourier Neural Operator to efficiently adapt to distribution shifts.
- ▸ The approach demonstrates superiority in terms of plasticity and stability compared to established CL methods.
Merits
Strength in Adaptability
SLE-FNO's ability to adapt to distribution shifts without catastrophic forgetting makes it a strong contender for CL applications in fluid dynamics and beyond.
Efficient Parameterization
The architecture-based approach enables SLE-FNO to achieve accuracy with minimal additional parameters, making it an attractive option for resource-constrained environments.
Demerits
Limited Generalizability
The study's results may not be directly applicable to other domains or applications, as the approach is specifically designed for task-agnostic CL in Fourier Neural Operators.
High Computational Requirements
The Single-Layer Extension and Fourier Neural Operator may require significant computational resources, limiting the approach's scalability and applicability to resource-constrained environments.
Expert Commentary
The article presents a novel and promising approach for task-agnostic continual learning in Fourier Neural Operators. However, its limitations and potential drawbacks should be carefully considered. The study's findings highlight the need for further research into adaptable CL frameworks, particularly in domains with high distribution shifts. Moreover, the approach's high computational requirements and potential for limited generalizability should be addressed in future studies. Overall, the article makes a significant contribution to the field of CL and scientific machine learning, and its results have important implications for the development of more accurate and robust models in various domains.
Recommendations
- ✓ Future studies should explore the generalizability of SLE-FNO to other domains and applications.
- ✓ The approach's computational requirements should be addressed through optimization techniques or more efficient implementation methods.
Sources
Original: arXiv - cs.LG
