Modality Collapse as Mismatched Decoding: Information-Theoretic Limits of Multimodal LLMs
arXiv:2602.23136v1 Announce Type: new Abstract: Multimodal LLMs can process speech and images, but they cannot hear a speaker's voice or see an object's texture. We show this is not a failure of encoding: speaker identity, emotion, and visual attributes survive through every LLM layer (3--55$\times$ above chance in linear probes), yet removing 64--71% of modality-specific variance improves decoder loss. The decoder has no learned use for these directions; their presence is noise. We formalize this as a mismatched decoder problem: a decoder trained on text can only extract information along text-aligned directions. Accessible information is bounded by the Generalized Mutual Information (GMI), with degradation scaling with distributional distance and decoder sensitivity. The bound is a property of the decoder's scoring rule, not of any particular architecture; it applies whether non-text inputs arrive through a learned projection, a discrete codebook, or no explicit adapter at all. We
arXiv:2602.23136v1 Announce Type: new Abstract: Multimodal LLMs can process speech and images, but they cannot hear a speaker's voice or see an object's texture. We show this is not a failure of encoding: speaker identity, emotion, and visual attributes survive through every LLM layer (3--55$\times$ above chance in linear probes), yet removing 64--71% of modality-specific variance improves decoder loss. The decoder has no learned use for these directions; their presence is noise. We formalize this as a mismatched decoder problem: a decoder trained on text can only extract information along text-aligned directions. Accessible information is bounded by the Generalized Mutual Information (GMI), with degradation scaling with distributional distance and decoder sensitivity. The bound is a property of the decoder's scoring rule, not of any particular architecture; it applies whether non-text inputs arrive through a learned projection, a discrete codebook, or no explicit adapter at all. We validate this across five models spanning speech and vision. A controlled experiment (two Prismatic VLMs differing only in encoder text-alignment) confirms the bottleneck is the decoder's scoring rule, not the encoder or projection. A LoRA intervention demonstrates the fix: training with an emotion objective improves emotion accessibility ($+$7.5%) without affecting other attributes, confirming that the training objective determines what becomes accessible.
Executive Summary
The article 'Modality Collapse as Mismatched Decoding: Information-Theoretic Limits of Multimodal LLMs' explores the limitations of multimodal large language models (LLMs) in processing non-textual inputs such as speech and images. The study demonstrates that while these models can encode modality-specific information like speaker identity and visual attributes, the decoder's inability to utilize this information results in modality collapse. The authors formalize this as a mismatched decoder problem, where the decoder, trained on text, can only extract information aligned with text directions. They validate their findings across five models and propose a solution involving training with specific objectives to improve accessibility of non-text attributes.
Key Points
- ▸ Multimodal LLMs can encode modality-specific information but fail to utilize it effectively due to decoder limitations.
- ▸ The mismatched decoder problem is formalized using Generalized Mutual Information (GMI), highlighting the decoder's scoring rule as the bottleneck.
- ▸ Controlled experiments and interventions confirm that the training objective determines the accessibility of non-text attributes.
Merits
Comprehensive Analysis
The article provides a thorough analysis of the information-theoretic limits of multimodal LLMs, supported by rigorous experimental validation across multiple models.
Innovative Formalization
The formalization of the mismatched decoder problem using GMI offers a novel perspective on the limitations of multimodal LLMs.
Practical Solution
The proposed LoRA intervention demonstrates a practical approach to improving the accessibility of non-text attributes, confirming the impact of training objectives.
Demerits
Limited Scope
The study focuses primarily on speech and vision modalities, which may not fully capture the complexities of other multimodal interactions.
Generalizability
While the findings are validated across five models, the generalizability to other architectures and applications remains to be fully explored.
Expert Commentary
The article 'Modality Collapse as Mismatched Decoding: Information-Theoretic Limits of Multimodal LLMs' presents a significant advancement in understanding the limitations of multimodal large language models. By formalizing the mismatched decoder problem and validating it through extensive experiments, the authors provide a compelling argument for the role of the decoder's scoring rule as a critical bottleneck. The study's innovative use of Generalized Mutual Information (GMI) offers a fresh perspective on the information-theoretic constraints of these models. The practical implications are substantial, as the proposed LoRA intervention demonstrates a viable path to improving the accessibility of non-text attributes. However, the study's focus on speech and vision modalities leaves room for further exploration into other multimodal interactions. Overall, this work contributes valuable insights to the field of multimodal machine learning and underscores the importance of information-theoretic principles in AI research.
Recommendations
- ✓ Further research should explore the generalizability of these findings to other modalities and architectures to ensure comprehensive understanding.
- ✓ Policymakers and practitioners should consider the implications of these findings in developing more robust and fair multimodal AI systems.
