Differentially Private Non-convex Distributionally Robust Optimization
arXiv:2602.16155v1 Announce Type: new Abstract: Real-world deployments routinely face distribution shifts, group imbalances, and adversarial perturbations, under which the traditional Empirical Risk Minimization (ERM) framework can degrade severely. Distributionally Robust Optimization (DRO) addresses this issue by optimizing the worst-case expected loss over an uncertainty set of distributions, offering a principled approach to robustness. Meanwhile, as training data in DRO always involves sensitive information, safeguarding it against leakage under Differential Privacy (DP) is essential. In contrast to classical DP-ERM, DP-DRO has received much less attention due to its minimax optimization structure with uncertainty constraint. To bridge the gap, we provide a comprehensive study of DP-(finite-sum)-DRO with $\psi$-divergence and non-convex loss. First, we study DRO with general $\psi$-divergence by reformulating it as a minimization problem, and develop a novel $(\varepsil
arXiv:2602.16155v1 Announce Type: new Abstract: Real-world deployments routinely face distribution shifts, group imbalances, and adversarial perturbations, under which the traditional Empirical Risk Minimization (ERM) framework can degrade severely. Distributionally Robust Optimization (DRO) addresses this issue by optimizing the worst-case expected loss over an uncertainty set of distributions, offering a principled approach to robustness. Meanwhile, as training data in DRO always involves sensitive information, safeguarding it against leakage under Differential Privacy (DP) is essential. In contrast to classical DP-ERM, DP-DRO has received much less attention due to its minimax optimization structure with uncertainty constraint. To bridge the gap, we provide a comprehensive study of DP-(finite-sum)-DRO with $\psi$-divergence and non-convex loss. First, we study DRO with general $\psi$-divergence by reformulating it as a minimization problem, and develop a novel $(\varepsilon, \delta)$-DP optimization method, called DP Double-Spider, tailored to this structure. Under mild assumptions, we show that it achieves a utility bound of $\mathcal{O}(\frac{1}{\sqrt{n}}+ (\frac{\sqrt{d \log (1/\delta)}}{n \varepsilon})^{2/3})$ in terms of the gradient norm, where $n$ denotes the data size and $d$ denotes the model dimension. We further improve the utility rate for specific divergences. In particular, for DP-DRO with KL-divergence, by transforming the problem into a compositional finite-sum optimization problem, we develop a DP Recursive-Spider method and show that it achieves a utility bound of $\mathcal{O}((\frac{\sqrt{d \log(1/\delta)}}{n\varepsilon})^{2/3} )$, matching the best-known result for non-convex DP-ERM. Experimentally, we demonstrate that our proposed methods outperform existing approaches for DP minimax optimization.
Executive Summary
This article presents a comprehensive study on Differentially Private Non-convex Distributionally Robust Optimization (DP-DRO), addressing the gap between classical DP-Empirical Risk Minimization (DP-ERM) and DP-DRO. The authors propose novel optimization methods, DP Double-Spider and DP Recursive-Spider, tailored to the minimax optimization structure of DRO. The methods achieve improved utility bounds for various divergences, including KL-divergence, and outperform existing approaches in experiments. This work contributes to the development of robust and private machine learning models, essential for real-world applications.
Key Points
- ▸ Differentially Private Non-convex Distributionally Robust Optimization (DP-DRO) addresses distribution shifts and adversarial perturbations in machine learning.
- ▸ DP-DRO has received less attention than classical DP-Empirical Risk Minimization (DP-ERM) due to its minimax optimization structure.
- ▸ The authors propose novel optimization methods, DP Double-Spider and DP Recursive-Spider, tailored to the DRO structure.
Merits
Strength
The authors provide a comprehensive study on DP-DRO, addressing the gap between classical DP-ERM and DP-DRO. The proposed methods achieve improved utility bounds and outperform existing approaches in experiments.
Demerits
Limitation
The proposed methods assume a specific form of the uncertainty set, which may not be applicable to all real-world scenarios.
Expert Commentary
This article presents a significant contribution to the field of robust and private machine learning, addressing the gap between classical DP-ERM and DP-DRO. The proposed methods demonstrate improved utility bounds and outperform existing approaches in experiments. The work has practical implications for real-world applications, where robustness and privacy are essential. However, the assumption of a specific form of the uncertainty set is a limitation. Further research is needed to generalize these results to more complex scenarios.
Recommendations
- ✓ Future research should focus on generalizing the proposed methods to more complex uncertainty sets.
- ✓ The results should be applied to real-world machine learning models to demonstrate their practical impact.
