Academic

FCN-LLM: Empower LLM for Brain Functional Connectivity Network Understanding via Graph-level Multi-task Instruction Tuning

arXiv:2603.01135v1 Announce Type: new Abstract: Large Language Models have achieved remarkable success in language understanding and reasoning, and their multimodal extensions enable comprehension of images, video, and audio. Inspired by this, foundation models for brain functional connectivity networks derived from resting-state fMRI have shown promise in clinical tasks. However, existing methods do not align FCNs with the text modality, limiting the ability of LLMs to directly understand FCNs. To address this, we propose FCN-LLM, a framework that enables LLMs to understand FCNs through graph-level, multi-task instruction tuning. Our approach employs a multi-scale FCN encoder capturing brain-region, functional subnetwork, and whole-brain features, projecting them into the semantic space of LLM. We design multi-paradigm instruction tasks covering 19 subject-specific attributes across demographics, phenotypes, and psychiatric conditions. A multi-stage learning strategy first aligns FCN

X
Xingcan Hu, Wei Wang, Li Xiao · · 1 min read · 18 views
#cs.AI

arXiv:2603.01135v1 Announce Type: new Abstract: Large Language Models have achieved remarkable success in language understanding and reasoning, and their multimodal extensions enable comprehension of images, video, and audio. Inspired by this, foundation models for brain functional connectivity networks derived from resting-state fMRI have shown promise in clinical tasks. However, existing methods do not align FCNs with the text modality, limiting the ability of LLMs to directly understand FCNs. To address this, we propose FCN-LLM, a framework that enables LLMs to understand FCNs through graph-level, multi-task instruction tuning. Our approach employs a multi-scale FCN encoder capturing brain-region, functional subnetwork, and whole-brain features, projecting them into the semantic space of LLM. We design multi-paradigm instruction tasks covering 19 subject-specific attributes across demographics, phenotypes, and psychiatric conditions. A multi-stage learning strategy first aligns FCN embeddings with the LLM and then jointly fine-tunes the entire model to capture high-level semantic information. Experiments on a large-scale, multi-site FCN database show that FCN-LLM achieves strong zero-shot generalization on unseen datasets, outperforming conventional supervised and foundation models. This work introduces a new paradigm for integrating brain functional networks with LLMs, offering a flexible and interpretable framework for neuroscience.

Executive Summary

The proposed FCN-LLM framework enables Large Language Models (LLMs) to understand brain functional connectivity networks (FCNs) through graph-level, multi-task instruction tuning. This approach projects FCN features into the semantic space of LLMs, allowing for zero-shot generalization on unseen datasets. The framework achieves strong results, outperforming conventional supervised and foundation models, and introduces a new paradigm for integrating brain functional networks with LLMs.

Key Points

  • ▸ Introduction of FCN-LLM framework for integrating brain functional networks with LLMs
  • ▸ Use of graph-level, multi-task instruction tuning for aligning FCNs with the text modality
  • ▸ Achievement of strong zero-shot generalization on unseen datasets

Merits

Improved Generalization

The FCN-LLM framework demonstrates strong zero-shot generalization on unseen datasets, indicating its potential for real-world applications.

Interpretability

The framework offers a flexible and interpretable approach for neuroscience, allowing for a deeper understanding of brain functional networks.

Demerits

Complexity

The proposed framework may be computationally expensive and require significant resources, which could limit its adoption.

Limited Domain Knowledge

The framework's performance may be limited by the quality and availability of domain-specific knowledge and data.

Expert Commentary

The FCN-LLM framework represents a significant advancement in the integration of brain functional networks with LLMs. By leveraging graph-level, multi-task instruction tuning, the framework demonstrates strong zero-shot generalization and offers a flexible and interpretable approach for neuroscience. However, the complexity of the framework and limited domain knowledge may pose challenges for its adoption. Further research is needed to fully explore the potential of this framework and address the ethical and regulatory implications of its use.

Recommendations

  • ✓ Further evaluation of the framework's performance on diverse datasets and domains
  • ✓ Investigation of the framework's potential applications in clinical diagnosis and treatment
  • ✓ Development of regulatory frameworks and ethical guidelines for the use of AI in neuroscience

Sources

Related Articles