Measuring Social Integration Through Participation: Categorizing Organizations and Leisure Activities in the Displaced Karelians Interview Archive using LLMs
arXiv:2602.15436v1 Announce Type: new Abstract: Digitized historical archives make it possible to study everyday social life on a large scale, but the information extracted directly from text often does not directly allow one to answer the research questions posed by historians or sociologists in a quantitative manner. We address this problem in a large collection of Finnish World War II Karelian evacuee family interviews. Prior work extracted more than 350K mentions of leisure time activities and organizational memberships from these interviews, yielding 71K unique activity and organization names -- far too many to analyze directly. We develop a categorization framework that captures key aspects of participation (the kind of activity/organization, how social it typically is, how regularly it happens, and how physically demanding it is). We annotate a gold-standard set to allow for a reliable evaluation, and then test whether large language models can apply the same schema at scale.
arXiv:2602.15436v1 Announce Type: new Abstract: Digitized historical archives make it possible to study everyday social life on a large scale, but the information extracted directly from text often does not directly allow one to answer the research questions posed by historians or sociologists in a quantitative manner. We address this problem in a large collection of Finnish World War II Karelian evacuee family interviews. Prior work extracted more than 350K mentions of leisure time activities and organizational memberships from these interviews, yielding 71K unique activity and organization names -- far too many to analyze directly. We develop a categorization framework that captures key aspects of participation (the kind of activity/organization, how social it typically is, how regularly it happens, and how physically demanding it is). We annotate a gold-standard set to allow for a reliable evaluation, and then test whether large language models can apply the same schema at scale. Using a simple voting approach across multiple model runs, we find that an open-weight LLM can closely match expert judgments. Finally, we apply the method to label the 350K entities, producing a structured resource for downstream studies of social integration and related outcomes.
Executive Summary
This article presents a novel approach to measuring social integration through participation by developing a categorization framework for leisure activities and organizational memberships in a large collection of Finnish World War II Karelian evacuee family interviews. Using large language models, the authors demonstrate their framework's ability to categorize over 350,000 entities, creating a structured resource for downstream studies of social integration and related outcomes. This research contributes to the field by providing a scalable method for analyzing historical archives and offers insights into the social dynamics of displacement and migration. The findings have implications for historians, sociologists, and policymakers seeking to understand the impact of displacement on social integration and community building.
Key Points
- ▸ Develops a categorization framework for leisure activities and organizational memberships to measure social integration through participation
- ▸ Applies large language models to categorize over 350,000 entities in a large collection of Finnish World War II Karelian evacuee family interviews
- ▸ Creates a structured resource for downstream studies of social integration and related outcomes
Merits
Strength in Scalability
The authors' ability to apply their framework to analyze a large collection of historical archives, demonstrating its scalability and potential for broader applications.
Methodological Innovation
The use of large language models to categorize entities in historical archives represents a novel approach, offering a promising solution for analyzing large amounts of unstructured data.
Potential for Generalizability
The framework's potential to be applied to other historical archives and research contexts, making it a valuable tool for historians, sociologists, and policymakers.
Demerits
Limited Contextualization
The article primarily focuses on the categorization framework and its application, with limited discussion of the historical context and social dynamics of the Karelian evacuee community.
Dependence on Large Language Models
The framework's reliance on large language models raises concerns about its reliability and potential for bias, particularly in situations where models may perform poorly.
Lack of Comparative Analysis
The article does not engage in comparative analysis with existing methods or frameworks for measuring social integration, making it difficult to assess the framework's relative strengths and weaknesses.
Expert Commentary
The article presents a promising approach to measuring social integration through participation by developing a categorization framework for leisure activities and organizational memberships. While the framework shows promise, its reliance on large language models raises concerns about reliability and potential bias. The article's focus on scalability and potential for generalizability are significant strengths, but the limited contextualization and lack of comparative analysis are notable weaknesses. To build on this research, future studies should engage in comparative analysis with existing methods and frameworks, explore the framework's limitations and potential biases, and consider the historical context and social dynamics of the Karelian evacuee community.
Recommendations
- ✓ Future studies should engage in comparative analysis with existing methods and frameworks for measuring social integration to assess the framework's relative strengths and weaknesses.
- ✓ Researchers should explore ways to mitigate the potential biases and limitations of large language models in categorizing entities in historical archives.
- ✓ The framework's scalability and potential for generalizability should be further tested in various research contexts and historical archives to assess its broader applicability.
