Partitioning and Sharding
Partitioning splits a dataset into pieces along a key so that queries touch less data and work spreads across machines.
Dividing data to scale it
As datasets grow, keeping everything in one undivided store becomes slow and hard to manage. Partitioning splits data into pieces along a chosen key, and sharding distributes those pieces across multiple machines. Done well, a query reads only the relevant partitions and parallel work spreads evenly. Done poorly, it creates hotspots and skew.
Partitioning strategies
- Range partitioning: split by contiguous ranges of a key, such as time windows.
- Hash partitioning: assign by a hash of the key, spreading load evenly.
- List partitioning: group by discrete key values, such as diagnostic type.
- Composite: combine, for example hash within time ranges.
Partition pruning
The main payoff is pruning: a query filtering on the partition key can skip partitions that cannot match. Time-partitioned data lets a query for one shot or one day read only that partition. This is why choosing a partition key that matches common query filters matters so much.
Skew and hotspots
If one partition holds far more data or receives far more traffic than others, it becomes a bottleneck, a hotspot. Range partitioning by time can hotspot on the newest partition during ingestion; hash partitioning avoids this but loses range-pruning. The right choice balances even distribution against query patterns.
Cardinality and small files
Over-partitioning creates many tiny partitions, which is inefficient because each carries overhead. Under-partitioning leaves partitions too large to prune usefully. For time-series and simulation-case data in a research program, partitioning by time or by run identifier commonly matches how the data is queried, keeping scans focused while avoiding the small-file problem discussed in object storage.