Time-Series Compression
Time-series compression exploits the regularity of timestamped measurements to store them in a small fraction of their raw size, often losslessly.
Why time series compress well
Time-series data has structure that general compression cannot fully exploit but specialized encodings can. Timestamps arrive at near-regular intervals, and consecutive measurements change little. Time-series compression encodes these regularities directly, shrinking storage and the bytes a query must read, frequently by an order of magnitude and often without losing any information.
Delta and delta-of-delta
Timestamps are rarely stored as absolute values. Delta encoding stores the difference from the previous timestamp; for a steady interval these deltas are nearly constant. Delta-of-delta stores the change in the delta, which for a perfectly regular series is zero and compresses to almost nothing. A one-second cadence becomes a long run of zeros after this transform, which downstream entropy coding crushes.
Encoding the values
- XOR encoding: store the bitwise difference between consecutive floats
- When values change slowly, most XORed bits are zero and pack tightly
- Run-length encoding: collapse repeated identical readings
- Quantization: round to a tolerance when exactness is not required (lossy)
Lossless versus lossy
Lossless compression reconstructs the original values exactly and is mandatory when every sample may matter, as in a diagnostic record that could later be re-examined. Lossy compression discards detail within a stated tolerance for far higher ratios, appropriate for long-term trend storage where fine noise is irrelevant. The choice is a policy decision: what fidelity must survive, and for how long. A common design keeps recent data lossless and downsamples older data, since old data is queried at coarser resolution.
Compression and query cost
Compression is not only about storage. Because queries are frequently I/O-bound, reading fewer bytes means faster scans, so good compression speeds up analytics as a side effect. The trade-off is decompression CPU, but lightweight schemes like delta and XOR decode fast enough that the reduced I/O wins overall. This is why time-series databases build these encodings into their storage engines. See time-series databases, columnar storage, and edge data collection.