Graph Attention Networks

Graph attention networks weight each neighbor by a learned relevance score, letting a node listen more to some neighbors than others.

Attention on graphs

A basic graph convolution treats all neighbors equally or weights them by a fixed function of node degree. A graph attention network (GAT) instead learns how much attention each node should pay to each neighbor. For every edge it computes an attention coefficient from the two nodes' features, normalizes the coefficients across a node's neighbors with softmax, and aggregates neighbor features using those weights.

The attention coefficient

For a node i and neighbor j, GAT computes a raw score by applying a small shared network to the concatenation of their transformed features, then passes it through a LeakyReLU. Softmax over all of i's neighbors turns the raw scores into weights that sum to one. The node's new representation is the weighted sum of its neighbors' transformed features. Multiple attention heads run in parallel and their outputs are combined, mirroring multi-head attention in transformers.

python

# GAT edge score, then neighbor-normalized softmax
# e_ij = LeakyReLU( a^T [ W h_i || W h_j ] )
# alpha_ij = softmax_j( e_ij )   over neighbors of i
# h_i' = sum_j alpha_ij * (W h_j)

Advantages

Because weights are computed from features rather than from graph structure alone, GAT adapts to which neighbors matter for a given node and task. It does not need the full adjacency structure precomputed and generalizes to nodes and graphs not seen in training. This makes it well suited to inductive settings where new nodes arrive after training.

Relation to transformers

A transformer is essentially attention over a fully connected graph of tokens, and a GAT is attention restricted to a graph's actual edges. This connection runs deep: graph transformers apply full self-attention to graph nodes, sometimes adding structural or positional encodings so the model knows the graph topology. GAT can be seen as a sparse, locality-respecting cousin of transformer attention.

Trade-offs

Attention adds parameters and computation per edge relative to a plain graph convolution, and on some benchmarks the gain over simpler aggregation is modest. Still, the ability to weight neighbors is valuable when a node's neighbors vary widely in relevance, such as heterogeneous graphs with many edge types. As always, the simplest model that performs well is the right default.

• Learns per-neighbor attention weights from features.
• Softmax normalizes weights over each node's neighbors.
• Multi-head attention, as in transformers.
• Attention over the graph's edges rather than a full grid.