Academic

Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning

arXiv:2603.00191v1 Announce Type: new Abstract: Continual Learning (CL) requires models to sequentially adapt to new tasks without forgetting old knowledge. Recently, Low-Rank Adaptation (LoRA), a representative Parameter-Efficient Fine-Tuning (PEFT) method, has gained increasing attention in CL. Several LoRA-based CL methods reduce interference across tasks by separating their update spaces, typically building the new space from the estimated null space of past tasks. However, they (i) overlook task-shared directions, which suppresses knowledge transfer, and (ii) fail to capture truly effective task-specific directions since these ``null bases" of old tasks can remain nearly inactive for new task under correlated tasks. To address this, we study LoRA learning capability from a projection energy perspective, and propose Low-rank Decomposition and Adaptation (LoDA). It performs a task-driven decomposition to build general and truly task-specific LoRA subspaces by solving two energy-bas

L
Lingfeng He, De Cheng, Huaijie Wang, Xi Yang, Nannan Wang, Xinbo Gao · · 1 min read · 18 views
#cs.LG #cs.CV

arXiv:2603.00191v1 Announce Type: new Abstract: Continual Learning (CL) requires models to sequentially adapt to new tasks without forgetting old knowledge. Recently, Low-Rank Adaptation (LoRA), a representative Parameter-Efficient Fine-Tuning (PEFT) method, has gained increasing attention in CL. Several LoRA-based CL methods reduce interference across tasks by separating their update spaces, typically building the new space from the estimated null space of past tasks. However, they (i) overlook task-shared directions, which suppresses knowledge transfer, and (ii) fail to capture truly effective task-specific directions since these ``null bases" of old tasks can remain nearly inactive for new task under correlated tasks. To address this, we study LoRA learning capability from a projection energy perspective, and propose Low-rank Decomposition and Adaptation (LoDA). It performs a task-driven decomposition to build general and truly task-specific LoRA subspaces by solving two energy-based objectives, decoupling directions for knowledge sharing and isolation. LoDA fixes LoRA down-projections on two subspaces and learns robust up-projections via a Gradient-Aligned Optimization (GAO) approach. After each task, before integrating the LoRA updates into the backbone, LoDA derives a closed-form recalibration for the general update, approximating a feature-level joint optimum along this task-shared direction. Experiments indicate that LoDA outperforms existing CL methods.

Executive Summary

This article proposes a novel approach to Low-Rank Adaptation (LoRA) in Continual Learning (CL), addressing the limitations of existing methods. The proposed Low-rank Decomposition and Adaptation (LoDA) framework performs a task-driven decomposition to build general and task-specific LoRA subspaces. LoDA achieves this by solving two energy-based objectives, decoupling directions for knowledge sharing and isolation. The framework also learns robust up-projections via a Gradient-Aligned Optimization (GAO) approach. The authors demonstrate the effectiveness of LoDA through experiments, showcasing its ability to outperform existing CL methods. This work has significant implications for the development of efficient and effective CL systems, particularly in scenarios where knowledge sharing and isolation are crucial.

Key Points

  • ▸ Task-driven subspace decomposition for LoRA-based CL
  • ▸ Decoupling directions for knowledge sharing and isolation
  • ▸ Gradient-Aligned Optimization (GAO) approach for robust up-projections

Merits

Strength in addressing existing limitations

The proposed LoDA framework effectively addresses the limitations of existing LoRA-based CL methods, such as overlooking task-shared directions and failing to capture truly effective task-specific directions.

Improved performance through task-driven decomposition

The task-driven decomposition in LoDA enables the model to learn general and task-specific LoRA subspaces, leading to improved performance in CL tasks.

Demerits

Potential computational complexity

The proposed LoDA framework may incur additional computational complexity due to the need to solve two energy-based objectives and learn robust up-projections via the GAO approach.

Dependence on task similarity

The effectiveness of LoDA may depend on the similarity between tasks, and its performance may degrade in scenarios where tasks are highly dissimilar.

Expert Commentary

This article makes a significant contribution to the field of Continual Learning, proposing a novel approach to Low-Rank Adaptation (LoRA) that addresses the limitations of existing methods. The proposed LoDA framework demonstrates its effectiveness through experiments, showcasing its ability to outperform existing CL methods. However, the framework's potential computational complexity and dependence on task similarity are notable limitations that need to be addressed in future work. The implications of this research are far-reaching, with potential applications in real-world scenarios where CL is crucial. The framework's ability to decouple directions for knowledge sharing and isolation can also inform the development of more effective optimization techniques in CL.

Recommendations

  • ✓ Recommendation 1: Future work should focus on addressing the potential computational complexity of the proposed LoDA framework.
  • ✓ Recommendation 2: The framework's performance in scenarios with highly dissimilar tasks should be thoroughly evaluated to assess its robustness.
  • ✓ Recommendation 3: The proposed LoDA framework should be applied in real-world scenarios to demonstrate its practical effectiveness.
  • ✓ Recommendation 4: The framework's ability to learn robust up-projections via the GAO approach should be further investigated to inform the development of more effective optimization techniques in CL.

Sources

Related Articles