Dissecting Chronos: Sparse Autoencoders Reveal Causal Feature Hierarchies in Time Series Foundation Models
arXiv:2603.10071v1 Announce Type: new Abstract: Time series foundation models (TSFMs) are increasingly deployed in high-stakes domains, yet their internal representations remain opaque. We present the first application of sparse autoencoders (SAEs) to a TSFM, training TopK SAEs on activations of Chronos-T5-Large (710M parameters) across six layers. Through 392 single-feature ablation experiments, we establish that every ablated feature produces a positive CRPS degradation, confirming causal relevance. Our analysis reveals a depth-dependent hierarchy: early encoder layers encode low-level frequency features, the mid-encoder concentrates causally critical change-detection features, and the final encoder compresses a rich but less causally important taxonomy of temporal concepts. The most critical features reside in the mid-encoder (max single-feature Delta CRPS = 38.61), not in the semantically richest final encoder layer, where progressive ablation paradoxically improves forecast quali
arXiv:2603.10071v1 Announce Type: new Abstract: Time series foundation models (TSFMs) are increasingly deployed in high-stakes domains, yet their internal representations remain opaque. We present the first application of sparse autoencoders (SAEs) to a TSFM, training TopK SAEs on activations of Chronos-T5-Large (710M parameters) across six layers. Through 392 single-feature ablation experiments, we establish that every ablated feature produces a positive CRPS degradation, confirming causal relevance. Our analysis reveals a depth-dependent hierarchy: early encoder layers encode low-level frequency features, the mid-encoder concentrates causally critical change-detection features, and the final encoder compresses a rich but less causally important taxonomy of temporal concepts. The most critical features reside in the mid-encoder (max single-feature Delta CRPS = 38.61), not in the semantically richest final encoder layer, where progressive ablation paradoxically improves forecast quality. These findings demonstrate that mechanistic interpretability transfers effectively to TSFMs and that Chronos-T5 relies on abrupt-dynamics detection rather than periodic pattern recognition.
Executive Summary
The article presents a novel approach to understanding the internal representations of time series foundation models (TSFMs) using sparse autoencoders (SAEs). The study focuses on the Chronos-T5-Large model, revealing a depth-dependent hierarchy of causal feature hierarchies. The findings indicate that early encoder layers encode low-level frequency features, while mid-encoder layers concentrate on causally critical change-detection features. The analysis demonstrates the effectiveness of mechanistic interpretability in TSFMs and highlights the importance of abrupt-dynamics detection in the Chronos-T5 model.
Key Points
- ▸ Application of sparse autoencoders to time series foundation models
- ▸ Depth-dependent hierarchy of causal feature hierarchies in Chronos-T5-Large
- ▸ Mid-encoder layers contain the most critical features for forecast quality
Merits
Methodological Innovation
The study introduces a novel method for analyzing TSFMs, providing new insights into their internal representations
Empirical Rigor
The analysis is based on 392 single-feature ablation experiments, ensuring robust and reliable results
Demerits
Model-Specific Findings
The study focuses on a single model (Chronos-T5-Large), which may limit the generalizability of the results to other TSFMs
Lack of Theoretical Framework
The article does not provide a comprehensive theoretical framework for understanding the causal feature hierarchies in TSFMs
Expert Commentary
The article presents a significant contribution to the field of time series analysis and explainability in AI. The use of sparse autoencoders to analyze the internal representations of TSFMs provides a novel and effective approach to understanding these complex models. The findings have important implications for the deployment of TSFMs in high-stakes domains and highlight the need for more transparent and explainable AI models. However, further research is necessary to generalize the results to other models and to develop a comprehensive theoretical framework for understanding causal feature hierarchies in TSFMs.
Recommendations
- ✓ Future studies should investigate the applicability of SAEs to other TSFMs and complex models
- ✓ The development of a theoretical framework for understanding causal feature hierarchies in TSFMs is essential for advancing the field