fluxcode timestamps: the time axis

← codec comparison (index.html) · SPEC §3: time axis

fluxcode can store each sample's timestamp exactly next to its values (by default; the last section lets them move by a share of an interval). This page covers only that time axis: what the timestamps add to a one-minute block group (60 blocks of 1,000 samples, nominally 1 kHz, int64 ns ticks). Every number here is the block group with times minus the same block group without them. For scale, that block group's values alone take 17,976 bytes (a 4.12 Hz sine at the default settings), 311 µs to encode and 100 µs to decode. Regenerate: cd experimental && uv run python make_time_report.py.

How timestamps are stored

• Per block: a start, a step and a reference. The step is the GCD of the block's intervals, and each interval divided by it is a quotient: the reference plus a residual. A block whose 999 intervals are all equal is regular: every quotient equals the reference, so it stores nothing else (24 bytes before compression, a few after). Any other block is irregular and stores the residuals, zigzagged, as 32 bit planes (64 in the rare long block, with a residual of 2³² or more: in ns ticks, a gap of seconds). Most of those planes are zero, and zstd removes them.
• The reference is the rounded mean or the minimum, chosen per block. The mean suits jitter around a center; the minimum suits skewed intervals (a gap, events, deadband logging), whose mean sits away from most of them. Without a reference, intervals near 1000 flip several bit planes together as they wobble, which costs about a bit per sample.
• A gap costs only its own block. The blocks before and after it stay regular.
• The resolution is free. The GCD takes out the grid, so a millisecond grid costs the same in ns, µs or ms ticks.
• Block starts are stored as increases over the previous start, so a regular block group's 60 starts compress to a few bytes.

The common shapes

A perfect grid, and a perfect grid with a few gaps, are by far the most common timestamps in practice. A noisy clock (host timestamps on arrival) is the third common shape, and the only one of the three where the timestamps cost more than the values (here about two and a half times as much).

The example is the median-size block group of 64 seeds per shape. Times are single thread, best of 60, Apple M3 on AC power; each added time's percentage is of the same block group's values-only encode (311 µs) or decode (100 µs). vs values is the timestamps' bytes as a share of the values' 17,976.

• Perfect grid: 66 bytes for 60,000 timestamps, whatever the start, step or tick unit. Decoding writes the 60,000 int64 ticks, and that is most of its cost.
• A few gaps: each gap makes one block irregular. Its intervals are all 1 except one, so the block's planes are nearly all zero and it costs about 20 to 60 bytes (next section).
• Noisy clock: the jitter is real entropy, and no lossless coder can remove it: σ = 20 µs at µs resolution carries about 6.4 bits per sample, and the block group spends 7.1. The rest comes from storing intervals (each the difference of two jitters) with each bit plane coded on its own. If the jitter is measurement noise rather than information, let fluxcode round it away: with Params(time_error=0.1) the same clock costs 65 bytes (last section).

Cost per gap

A grid with k gaps per minute, each 5 ms to 3 s; median of 16 block groups per row.

Harder shapes

For comparison, shapes with many more irregular blocks (from bench/time_axis.py). Only the cases with random intervals cost more than the values. Percentages are of the values-only encode and decode, as above.

1% dropped and a drifting clock stay cheap: their intervals take two or three values. Jitter, Poisson events and deadband logging have random intervals, and ns resolution costs more than µs when the jitter really is at ns resolution.

Rounding timestamps: Params.time_error

When the jitter is measurement noise, Params(time_error=e) lets fluxcode round the timestamps (0 to 0.5; the default 0 keeps them exact). Each block gets a coarse consistent step at which 95% of its times keep within e of their own interval, and none moves more than e of the block's mean interval. The values are kept as measured, and the format doesn't change: the decoder sees ordinary times.

• The check. No time moves more than e mean intervals (the block's span over its intervals, 2% slack), and at most one in 20 moves more than e times its own interval, the smaller of the two either side of it. Only unusually close times may be off by more than e of their own interval, and by no more than the others.
• Candidates, the first that passes is taken: the lattice continuing the block before, when it is regular on the same clock; then the block's own 1-2-5 grid (1, 2, 5, 10, 20, … ticks): the coarsest step of at most 2e mean intervals if the check passes there (clocks do), else from the step of a short interval (the 3rd smallest of 15 sampled) down until it passes, at the clock's own phase (a multiple of a tenth of the step, kept from the previous block unless another fits clearly better), so a free-running or drifting clock rounds onto its own grid, not the epoch's; then, for a clock no 1-2-5 step divides (1024 Hz, 60 Hz), whose rounded intervals alternate, a lattice of its own (its typical interval as the period).
• A jittered clock becomes regular once e is about 5 times its jitter (σ over the interval): every time stays on its own grid point, and the block stores no residuals. Below that the size falls in steps, one at each 1-2-5 step, and each step pays off on its own: there is no threshold below which rounding stops helping.
• Regular blocks are left alone, and times already on a grid don't change.
• Order is kept across blocks and block groups, and in time blocks every time stays in its block (a time that rounds up to the block's end stays at the end). Times are rounded once, as they are encoded: update_time_blocks stores new times exactly, and replace_time_blocks re-encodes whole blocks from the samples given.

Each cell: bytes the timestamps add · irregular blocks of 60, then the encode time they add (percentage of the values-only encode, 311 µs) and the largest move in mean intervals. Single thread, best of 60, Apple M3 on AC power. The chart samples e every 0.0025.

• Jittered clocks drop from about 120 KB to 63–66 bytes once e reaches about 5σ (0.05 here), on round times or free-running. With drift as well, the phase follows the clock block by block: 210 bytes at 0.05 and 111 at 0.1. Once regular they also encode faster: a regular block skips the residuals.
• Clocks no 1-2-5 step divides go onto lattices of their own once e covers the jitter (about 5σ): 1024 Hz clock: 112,735 → 131 bytes at 0.05 and 100 at 0.1; 60 Hz clock: 144,005 → 216 bytes at 0.05 and 125 at 0.1.
• Times already on a grid don't change, but checking them costs time: +16 µs (+5%) per block group on a perfect grid (its blocks are regular and aren't rounded, but the ticks are checked for order and regularity) and +1 µs (+0%) with a few gaps (rounding the irregular blocks).
• The noisy clock (σ = 20 µs, 2%) needs e = 0.1 to land on its grid; at 0.05 about 1% of its times fall to a neighbouring grid point.
• Events and deadband logging have no grid to land on. Rounding still takes out the low bits (events get 2.2 times smaller at 0.05). Events have many close pairs, and 95% of times must keep within e of their own interval, so their step is finer than a clock's at the same e. Deadband logging's times are already on a 1 ms grid and mostly a few ms apart: any coarser step moves too many of them by more than e of their own interval, so it gains nothing below e = 0.5, and the attempt costs +96 µs (+31%) per block group at 0.05.

Measurements behind the choices (1-2-5 steps over powers of two, no interpolation of values, no resampling, per-block phases): docs/TUNING.md, Time error.