Size Transferability of Graph Transformers with Convolutional Positional Encodings
arXiv:2602.15239v1 Announce Type: new Abstract: Transformers have achieved remarkable success across domains, motivating the rise of Graph Transformers (GTs) as attention-based architectures for graph-structured data. A key design choice in GTs is the use of Graph Neural Network (GNN)-based positional encodings to incorporate structural information. In this work, we study GTs through the lens of manifold limit models for graph sequences and establish a theoretical connection between GTs with GNN positional encodings and Manifold Neural Networks (MNNs). Building on transferability results for GNNs under manifold convergence, we show that GTs inherit transferability guarantees from their positional encodings. In particular, GTs trained on small graphs provably generalize to larger graphs under mild assumptions. We complement our theory with extensive experiments on standard graph benchmarks, demonstrating that GTs exhibit scalable behavior on par with GNNs. To further show the efficienc
arXiv:2602.15239v1 Announce Type: new Abstract: Transformers have achieved remarkable success across domains, motivating the rise of Graph Transformers (GTs) as attention-based architectures for graph-structured data. A key design choice in GTs is the use of Graph Neural Network (GNN)-based positional encodings to incorporate structural information. In this work, we study GTs through the lens of manifold limit models for graph sequences and establish a theoretical connection between GTs with GNN positional encodings and Manifold Neural Networks (MNNs). Building on transferability results for GNNs under manifold convergence, we show that GTs inherit transferability guarantees from their positional encodings. In particular, GTs trained on small graphs provably generalize to larger graphs under mild assumptions. We complement our theory with extensive experiments on standard graph benchmarks, demonstrating that GTs exhibit scalable behavior on par with GNNs. To further show the efficiency in a real-world scenario, we implement GTs for shortest path distance estimation over terrains to better illustrate the efficiency of the transferable GTs. Our results provide new insights into the understanding of GTs and suggest practical directions for efficient training of GTs in large-scale settings.
Executive Summary
This article examines the size transferability of Graph Transformers (GTs) with Convolutional Positional Encodings, establishing a theoretical connection between GTs and Manifold Neural Networks (MNNs). The authors demonstrate that GTs trained on small graphs provably generalize to larger graphs under mild assumptions, leveraging transferability results for Graph Neural Networks (GNNs). The study is complemented by extensive experiments on standard graph benchmarks, showcasing the scalable behavior of GTs comparable to GNNs. To further illustrate the efficiency of transferable GTs, the authors implement GTs for shortest path distance estimation over terrains. The results provide new insights into GTs and suggest practical directions for efficient training in large-scale settings.
Key Points
- ▸ Establishes a theoretical connection between GTs and MNNs
- ▸ Demonstrates provable generalization of GTs to larger graphs
- ▸ Showcases scalable behavior of GTs comparable to GNNs
- ▸ Illustrates efficiency of transferable GTs through real-world application
Merits
Theoretical Foundation
The article provides a solid theoretical foundation for understanding the behavior of GTs, leveraging established results from GNNs and MNNs.
Empirical Validation
The study is complemented by extensive experiments on standard graph benchmarks, providing empirical validation for the theoretical results.
Practical Implications
The results suggest practical directions for efficient training of GTs in large-scale settings, making the study relevant to real-world applications.
Demerits
Assumptions Limit Generalizability
The provable generalization of GTs is contingent upon mild assumptions, which may not always hold in real-world scenarios.
Limited Scope
The study focuses primarily on GTs with Convolutional Positional Encodings, limiting the scope of the findings to this specific variant of GTs.
Lack of Comparative Analysis
The article does not provide a comprehensive comparison of GTs with other graph neural network architectures, which may be necessary for a complete understanding of their performance.
Expert Commentary
The article provides a significant contribution to the understanding of Graph Transformers, establishing a theoretical connection between GTs and MNNs. The study's findings are supported by extensive empirical validation, demonstrating the scalable behavior of GTs comparable to GNNs. However, the assumptions underlying the provable generalization of GTs may limit the generalizability of the results. Furthermore, the study's focus on GTs with Convolutional Positional Encodings may not fully capture the performance of other variants of GTs.
Recommendations
- ✓ Future studies should investigate the performance of GTs in more diverse graph benchmarks and compare their results with other graph neural network architectures.
- ✓ The authors should explore ways to relax the assumptions underlying the provable generalization of GTs, enabling their application in a broader range of scenarios.
