Testing For Distribution Shifts with Conditional Conformal Test Martingales
arXiv:2602.13848v1 Announce Type: new Abstract: We propose a sequential test for detecting arbitrary distribution shifts that allows conformal test martingales (CTMs) to work under a fixed, reference-conditional setting. Existing CTM detectors construct test martingales by continually growing a reference set with each incoming sample, using it to assess how atypical the new sample is relative to past observations. While this design yields anytime-valid type-I error control, it suffers from test-time contamination: after a change, post-shift observations enter the reference set and dilute the evidence for distribution shift, increasing detection delay and reducing power. In contrast, our method avoids contamination by design by comparing each new sample to a fixed null reference dataset. Our main technical contribution is a robust martingale construction that remains valid conditional on the null reference data, achieved by explicitly accounting for the estimation error in the refere
arXiv:2602.13848v1 Announce Type: new Abstract: We propose a sequential test for detecting arbitrary distribution shifts that allows conformal test martingales (CTMs) to work under a fixed, reference-conditional setting. Existing CTM detectors construct test martingales by continually growing a reference set with each incoming sample, using it to assess how atypical the new sample is relative to past observations. While this design yields anytime-valid type-I error control, it suffers from test-time contamination: after a change, post-shift observations enter the reference set and dilute the evidence for distribution shift, increasing detection delay and reducing power. In contrast, our method avoids contamination by design by comparing each new sample to a fixed null reference dataset. Our main technical contribution is a robust martingale construction that remains valid conditional on the null reference data, achieved by explicitly accounting for the estimation error in the reference distribution induced by the finite reference set. This yields anytime-valid type-I error control together with guarantees of asymptotic power one and bounded expected detection delay. Empirically, our method detects shifts faster than standard CTMs, providing a powerful and reliable distribution-shift detector.
Executive Summary
The article introduces a novel sequential test for detecting arbitrary distribution shifts using conformal test martingales (CTMs) in a fixed, reference-conditional setting. Unlike existing CTM detectors that continually update the reference set with incoming samples, the proposed method employs a fixed null reference dataset to avoid test-time contamination. This approach ensures anytime-valid type-I error control, asymptotic power one, and bounded expected detection delay. Empirical results demonstrate faster shift detection compared to standard CTMs, making it a robust and reliable tool for distribution-shift detection.
Key Points
- ▸ Proposes a sequential test for detecting arbitrary distribution shifts using CTMs in a fixed, reference-conditional setting.
- ▸ Avoids test-time contamination by using a fixed null reference dataset.
- ▸ Achieves anytime-valid type-I error control, asymptotic power one, and bounded expected detection delay.
- ▸ Empirical results show faster detection of shifts compared to standard CTMs.
Merits
Robustness
The method's robustness is enhanced by using a fixed null reference dataset, which prevents post-shift observations from diluting the evidence for distribution shifts.
Theoretical Guarantees
The article provides theoretical guarantees of anytime-valid type-I error control, asymptotic power one, and bounded expected detection delay, which are crucial for reliable detection.
Empirical Performance
Empirical results demonstrate that the proposed method detects shifts faster than standard CTMs, making it a practical and effective tool.
Demerits
Complexity
The method's reliance on a fixed null reference dataset may introduce complexity in implementation and require careful selection of the reference dataset.
Generalizability
The effectiveness of the method may vary across different types of distribution shifts and datasets, requiring further validation in diverse scenarios.
Expert Commentary
The article presents a significant advancement in the field of distribution shift detection by addressing the critical issue of test-time contamination. The proposed method's use of a fixed null reference dataset is a novel approach that ensures robust and reliable detection. The theoretical guarantees provided, including anytime-valid type-I error control and asymptotic power one, are particularly noteworthy. These guarantees are essential for practical applications where the consequences of false positives or delayed detection can be significant. The empirical results further validate the method's effectiveness, demonstrating faster detection of shifts compared to standard CTMs. However, the complexity introduced by the fixed null reference dataset and the need for careful selection of the reference dataset are potential limitations that should be considered. Overall, the article makes a valuable contribution to the field and provides a powerful tool for detecting distribution shifts in various applications.
Recommendations
- ✓ Further validation of the method in diverse scenarios and datasets to assess its generalizability.
- ✓ Exploration of methods to simplify the implementation and reduce the complexity of selecting the fixed null reference dataset.
