Graph Attention Networks
Graph attention networks weight the contribution of each neighbor with a learned attention coefficient, letting a node emphasize its most relevant connections.
Attention on graphs
Graph attention networks (GATs) operate on data structured as nodes and edges. Each node updates its representation by aggregating information from its neighbors, but rather than treating all neighbors equally, a GAT computes a learned attention coefficient for each edge. A node can then attend more strongly to the neighbors most relevant to it, which is valuable when connections carry different importance, as in citation graphs, molecules, or social networks.
Computing attention coefficients
For a node i and each neighbor j, the layer transforms both features by a shared weight matrix, concatenates them, applies a small learned vector and a LeakyReLU to produce a raw score, then normalizes the scores across all neighbors of i with a softmax. The new representation of i is the attention-weighted sum of its neighbors' transformed features, passed through a nonlinearity. Attention depends only on the two endpoint features, so the same parameters apply to any graph regardless of size.
# per edge (i, j): raw score, then softmax over neighbors of i
e_ij = leaky_relu(a @ torch.cat([W @ h_i, W @ h_j]))
alpha_ij = softmax_over_neighbors(e_ij)
h_i_new = elu(sum(alpha_ij * (W @ h_j) for j in neighbors(i)))Multi-head attention
As in transformers, GATs use multiple attention heads to stabilize learning and capture different relationship types. Intermediate layers concatenate the heads' outputs; the final layer averages them. Multiple heads let the model attend to distinct neighborhood aspects in parallel.
- No expensive graph-wide operations; each node only looks at its neighbors
- Handles graphs of varying size and works in an inductive setting on unseen nodes
- Attention weights offer a degree of interpretability about which edges mattered
- Stacking layers widens the receptive field to multi-hop neighborhoods
Relation to other graph methods
GAT is one instance of the broader message-passing framework, where nodes exchange and aggregate messages along edges. It differs from graph convolutional networks by using learned, data-dependent edge weights instead of weights fixed by the graph structure. The attention mechanism itself is the same idea that powers transformers, adapted to irregular graph neighborhoods.