Columnar Storage: Parquet and Arrow
Columnar formats store tables by column rather than by row, making analytical scans and compression far more efficient.
By column, not by row
A table can be laid out two ways: row by row, keeping each record's fields together, or column by column, keeping all values of a field together. Row storage suits transactions that touch whole records. Columnar storage suits analytics that scan a few columns across many rows, because it reads only the columns a query needs.
Why columns compress better
Values within a column share a type and often a narrow range, so they compress far better than mixed row data. A column of timestamps, temperatures, or category codes has strong local structure that encoders exploit with dictionary, run-length, and delta methods. Column layout and compression reinforce each other.
Parquet
- On-disk columnar format for analytical tables.
- Stores per-column statistics (min, max) so queries skip irrelevant chunks.
- Splits data into row groups for parallel and partial reads.
- Widely supported across data tools and languages.
Arrow
- In-memory columnar format, a standard layout for analytics.
- Enables zero-copy data sharing between tools and languages.
- Complements Parquet: Arrow for compute, Parquet for storage.
- Reduces serialization overhead when moving data between systems.
Predicate pushdown
Because Parquet stores column statistics per row group, a query filtering on a range can skip entire groups whose min-max cannot match, reading far less data. This predicate pushdown, combined with reading only needed columns, is why columnar formats power analytical warehouses and lakehouses. For a research program, columnar storage suits large tables of derived scalar results across many simulation cases, where analysis scans a few fields at a time.