Mechanistic Interpretability of Cognitive Complexity in LLMs via Linear Probing using Bloom's Taxonomy
arXiv:2602.17229v1 Announce Type: new Abstract: The black-box nature of Large Language Models necessitates novel evaluation frameworks that transcend surface-level performance metrics. This study investigates the internal neural representations of cognitive complexity using Bloom's Taxonomy as a hierarchical lens. By analyzing high-dimensional activation vectors from different LLMs, we probe whether different cognitive levels, ranging from basic recall (Remember) to abstract synthesis (Create), are linearly separable within the model's residual streams. Our results demonstrate that linear classifiers achieve approximately 95% mean accuracy across all Bloom levels, providing strong evidence that cognitive level is encoded in a linearly accessible subspace of the model's representations. These findings provide evidence that the model resolves the cognitive difficulty of a prompt early in the forward pass, with representations becoming increasingly separable across layers.
arXiv:2602.17229v1 Announce Type: new Abstract: The black-box nature of Large Language Models necessitates novel evaluation frameworks that transcend surface-level performance metrics. This study investigates the internal neural representations of cognitive complexity using Bloom's Taxonomy as a hierarchical lens. By analyzing high-dimensional activation vectors from different LLMs, we probe whether different cognitive levels, ranging from basic recall (Remember) to abstract synthesis (Create), are linearly separable within the model's residual streams. Our results demonstrate that linear classifiers achieve approximately 95% mean accuracy across all Bloom levels, providing strong evidence that cognitive level is encoded in a linearly accessible subspace of the model's representations. These findings provide evidence that the model resolves the cognitive difficulty of a prompt early in the forward pass, with representations becoming increasingly separable across layers.
Executive Summary
This study explores the internal neural representations of cognitive complexity in Large Language Models (LLMs) using Bloom's Taxonomy. The researchers analyze high-dimensional activation vectors from different LLMs and find that linear classifiers can achieve approximately 95% mean accuracy across all Bloom levels, indicating that cognitive level is encoded in a linearly accessible subspace of the model's representations. The findings suggest that LLMs resolve cognitive difficulty early in the forward pass, with representations becoming increasingly separable across layers.
Key Points
- ▸ The study investigates internal neural representations of cognitive complexity in LLMs using Bloom's Taxonomy
- ▸ Linear classifiers achieve approximately 95% mean accuracy across all Bloom levels
- ▸ Cognitive level is encoded in a linearly accessible subspace of the model's representations
Merits
Novel Evaluation Framework
The study provides a novel evaluation framework that transcends surface-level performance metrics, offering a more nuanced understanding of LLMs' cognitive complexity
Demerits
Limited Generalizability
The study's findings may not generalize to other types of language models or tasks, limiting the scope of the research
Expert Commentary
The study's use of Bloom's Taxonomy as a hierarchical lens to investigate cognitive complexity in LLMs is a notable strength, offering a more nuanced understanding of the models' internal neural representations. The findings have significant implications for the development of more effective and efficient LLMs, as well as the broader issue of explainability in AI. However, the study's limited generalizability and potential biases in the data and methodology must be carefully considered in future research.
Recommendations
- ✓ Future research should aim to replicate and extend the study's findings to other types of language models and tasks
- ✓ The development of more transparent and accountable AI systems should be prioritized, with a focus on explainability and interpretability
