Information-Theoretic Storage Cost in Sentence Comprehension
arXiv:2602.18217v1 Announce Type: new Abstract: Real-time sentence comprehension imposes a significant load on working memory, as comprehenders must maintain contextual information to anticipate future input. While measures of such load have played an important role in psycholinguistic theories, they have been formalized, largely, using symbolic grammars, which assign discrete, uniform costs to syntactic predictions. This study proposes a measure of processing storage cost based on an information-theoretic formalization, as the amount of information previous words carry about future context, under uncertainty. Unlike previous discrete, grammar-based metrics, this measure is continuous, theory-neutral, and can be estimated from pre-trained neural language models. The validity of this approach is demonstrated through three analyses in English: our measure (i) recovers well-known processing asymmetries in center embeddings and relative clauses, (ii) correlates with a grammar-based storag
arXiv:2602.18217v1 Announce Type: new Abstract: Real-time sentence comprehension imposes a significant load on working memory, as comprehenders must maintain contextual information to anticipate future input. While measures of such load have played an important role in psycholinguistic theories, they have been formalized, largely, using symbolic grammars, which assign discrete, uniform costs to syntactic predictions. This study proposes a measure of processing storage cost based on an information-theoretic formalization, as the amount of information previous words carry about future context, under uncertainty. Unlike previous discrete, grammar-based metrics, this measure is continuous, theory-neutral, and can be estimated from pre-trained neural language models. The validity of this approach is demonstrated through three analyses in English: our measure (i) recovers well-known processing asymmetries in center embeddings and relative clauses, (ii) correlates with a grammar-based storage cost in a syntactically-annotated corpus, and (iii) predicts reading-time variance in two large-scale naturalistic datasets over and above baseline models with traditional information-based predictors.
Executive Summary
The article 'Information-Theoretic Storage Cost in Sentence Comprehension' introduces a novel approach to measuring the cognitive load associated with sentence comprehension by leveraging information-theoretic principles. Unlike traditional symbolic grammar-based methods, this study proposes a continuous, theory-neutral measure of processing storage cost, which quantifies the amount of information previous words carry about future context under uncertainty. The validity of this measure is demonstrated through three analyses in English, showing its ability to recover known processing asymmetries, correlate with grammar-based storage costs, and predict reading-time variance in large-scale datasets beyond traditional models.
Key Points
- ▸ Introduction of an information-theoretic measure of processing storage cost in sentence comprehension.
- ▸ The measure is continuous, theory-neutral, and can be estimated from pre-trained neural language models.
- ▸ Validation through three analyses: recovering processing asymmetries, correlating with grammar-based storage costs, and predicting reading-time variance.
Merits
Innovative Approach
The study introduces a novel, information-theoretic measure that moves beyond discrete, grammar-based metrics, offering a more nuanced and continuous assessment of cognitive load in sentence comprehension.
Empirical Validation
The measure is rigorously validated through multiple analyses, demonstrating its effectiveness in recovering known linguistic phenomena and predicting reading-time variance.
Theory-Neutral
The measure is theory-neutral, making it applicable across different linguistic theories and models, enhancing its utility and broad applicability.
Demerits
Limited Scope
The study focuses primarily on English, which may limit the generalizability of the findings to other languages with different syntactic structures.
Dependence on Neural Language Models
The measure relies on pre-trained neural language models, which may introduce biases or limitations inherent in these models.
Complexity of Interpretation
The information-theoretic measure, while innovative, may be complex to interpret and apply in practical settings, requiring further simplification for broader adoption.
Expert Commentary
The article presents a significant advancement in the field of psycholinguistics by introducing an information-theoretic measure of processing storage cost in sentence comprehension. This approach is particularly noteworthy for its departure from traditional symbolic grammar-based metrics, offering a more nuanced and continuous assessment of cognitive load. The study's empirical validation through multiple analyses strengthens its claims and demonstrates the measure's effectiveness in recovering known linguistic phenomena and predicting reading-time variance. However, the focus on English and the reliance on neural language models present certain limitations that need to be addressed in future research. The theory-neutral nature of the measure is a considerable strength, allowing for its application across different linguistic theories and models. This study not only contributes to our understanding of cognitive load in language processing but also highlights the potential of machine learning tools in cognitive science. The practical and policy implications are substantial, offering insights that can inform educational practices and the development of more effective language learning tools. Overall, the article is a valuable contribution to the field, with the potential to influence both theoretical and applied research in psycholinguistics.
Recommendations
- ✓ Future research should explore the applicability of this measure to other languages to enhance its generalizability.
- ✓ Further studies should investigate the robustness of the measure across different neural language models to address potential biases and limitations.
