Columnar Storage and Parquet
Columnar formats store each column contiguously, giving analytical queries far less I/O and far better compression than row-oriented storage.
Rows versus columns on disk
A table can be laid out on disk two ways. Row-oriented storage keeps all fields of a record together, which suits reading and writing whole records. Column-oriented storage keeps all values of one column together. Analytical queries typically touch a few columns across many rows, so a columnar layout reads only the needed columns and skips the rest, cutting I/O dramatically.
Why columns compress better
Values within a column share a type and often a narrow range, so they compress far better than a mixed row. A column of timestamps, a column of status codes, and a column of measurements each admit specialized encodings: dictionary encoding for low-cardinality strings, run-length encoding for repeated values, and delta encoding for sorted numerics. Compression both saves space and reduces the bytes a query must read.
Parquet structure
- A file splits into row groups, each a horizontal slice of the table
- Within a row group each column is stored as a chunk
- Column chunks divide into pages, the unit of encoding and compression
- Footer metadata records per-chunk min and max statistics
Predicate pushdown
The min/max statistics in the footer enable predicate pushdown: a query filtering for values above a threshold can skip any row group whose maximum is below it, without reading the data. Combined with reading only the referenced columns (projection pushdown), a well-organized Parquet dataset lets a selective query touch a small fraction of the file. Sorting data by a common filter column before writing multiplies this benefit.
When columnar is the wrong choice
Columnar layouts pay a cost on writes and on point lookups of whole records, because assembling one record means gathering from many column chunks. Transactional workloads that read and write individual records favor row storage; analytical workloads that scan and aggregate favor columnar. This is the same divide as OLTP versus OLAP. See Apache Arrow and OLAP vs OLTP.