Bridging Domains through Subspace-Aware Model Merging
arXiv:2603.05768v1 Announce Type: new Abstract: Model merging integrates multiple task-specific models into a single consolidated one. Recent research has made progress in improving merging performance for in-distribution or multi-task scenarios, but domain generalization in model merging remains underexplored. We investigate how merging models fine-tuned on distinct domains affects generalization to unseen domains. Through an analysis of parameter competition in the task matrix using singular value decomposition, we show that merging models trained under different distribution shifts induces stronger conflicts between their subspaces compared to traditional multi-task settings. To mitigate this issue, we propose SCORE (Subspace COnflict-Resolving mErging), a method designed to alleviate such singular subspace conflicts. SCORE finds a shared orthogonal basis by computing the principal components of the concatenated leading singular vectors of all models. It then projects each task mat
arXiv:2603.05768v1 Announce Type: new Abstract: Model merging integrates multiple task-specific models into a single consolidated one. Recent research has made progress in improving merging performance for in-distribution or multi-task scenarios, but domain generalization in model merging remains underexplored. We investigate how merging models fine-tuned on distinct domains affects generalization to unseen domains. Through an analysis of parameter competition in the task matrix using singular value decomposition, we show that merging models trained under different distribution shifts induces stronger conflicts between their subspaces compared to traditional multi-task settings. To mitigate this issue, we propose SCORE (Subspace COnflict-Resolving mErging), a method designed to alleviate such singular subspace conflicts. SCORE finds a shared orthogonal basis by computing the principal components of the concatenated leading singular vectors of all models. It then projects each task matrix into the shared basis, pruning off-diagonal components to remove conflicting singular directions. SCORE consistently outperforms, on average, existing model merging approaches in domain generalization settings across a variety of architectures and model scales, demonstrating its effectiveness and scalability.
Executive Summary
This study proposes SCORE, a novel model merging method designed to alleviate subspace conflicts in domain generalization settings. By analyzing parameter competition in the task matrix using singular value decomposition, the authors demonstrate that merging models trained on distinct domains induces stronger conflicts between their subspaces. SCORE consistently outperforms existing model merging approaches across various architectures and model scales, showcasing its effectiveness and scalability. The study contributes to the underexplored area of domain generalization in model merging and has significant implications for the development of robust and adaptable AI models.
Key Points
- ▸ SCORE is a novel model merging method that alleviates subspace conflicts in domain generalization settings.
- ▸ Merging models trained on distinct domains induces stronger conflicts between their subspaces.
- ▸ SCORE consistently outperforms existing model merging approaches across various architectures and model scales.
Merits
Strength
SCORE's ability to alleviate subspace conflicts improves model generalizability and adaptability in domain generalization settings.
Methodological Innovation
The study introduces a novel analysis of parameter competition in the task matrix using singular value decomposition, providing new insights into model merging.
Demerits
Limitation
The study assumes a fixed number of domains, which may not be realistic in real-world scenarios where domain shifts are unpredictable and dynamic.
Expert Commentary
This study makes a significant contribution to the field of model merging by introducing a novel analysis of parameter competition in the task matrix. SCORE's ability to alleviate subspace conflicts improves model generalizability and adaptability in domain generalization settings, making it a valuable tool for AI researchers and practitioners. However, the study's assumption of a fixed number of domains may limit its applicability in real-world scenarios. Future research should explore the extension of SCORE to dynamic domain shifts and investigate its performance in more complex scenarios.
Recommendations
- ✓ Future research should focus on extending SCORE to dynamic domain shifts and exploring its performance in more complex scenarios.
- ✓ The development of AI policies that prioritize model generalizability and adaptability should be accelerated to address the growing need for robust and adaptable AI models.