Rate control: quantize → predict → entropy code

← codec comparison (index.html)

Window = one block group: a minute of 60 blocks, 5 consecutive seconds of each of the 12 signal kinds, every block rescaled to [0, 1], standing in for one sensor passing through different regimes. 5 groups (one per report seed); bits/sample are whole block groups, every byte counted. Errors are absolute, which equals % of each block's full scale; they are per block, summarized over all blocks (or a kind's blocks). Regenerate: cd experimental && uv run python make_rate_report.py.

Method

• Pipeline per block: q = round(x / Δ_b) (the only lossy step, so |err| ≤ Δ_b/2 is guaranteed), residual = diff(q, order) with order ∈ (0, 1, 2, 3) chosen per block, then an entropy coder. Everything but the Rice/FLAC bit-packing is vectorized numpy.
• window: one Δ for the whole block group, bisected so the block group averages 4 bits/sample including all headers. A block that would exceed 8 bits/sample on its own gets its own coarser step Δ·2^e/8 (1-byte e per block). With uniform quantization, one shared step is the MSE-optimal way to spend a total budget.
• fixed: every block individually driven to 4 bits/sample coded on its own (same format), for comparison; the block group then costs what it costs.
• Framing: the rice coder stores its block payloads back to back with a length each; zstd and deflate make one compressor call per block group (flags, then start values, then byte planes); flac is one stream per block group, one frame per block; best: … stores a coder id per block, then each coder's block group of the blocks it won.
• Coders: delta0123-rice — FLAC-style partitioned Rice (2^p partitions, own k each; order/p/k chosen by exact cost); delta0123-zstd (level 9) / delta0123-deflate — zigzag residuals, narrowest byte width, byte-stream-split, then compress; flac — real libFLAC level 8 (LPC up to order 12) on the same integers, frame bytes only; best: … — per block, keep the coder whose standalone encoding is smallest (+1 byte coder id); 0-order entropy — zero-order entropy of the best residual + headers. A reference, not a codec and not a true floor: partitioned Rice, LPC and LZ matching adapt within a block and can beat it.
• delta0123-zstd-B: no search — step = block range/(2^B−1), order = argmin var(diff(q, k)), one zstd (level 3) call per block group; lowers B if a block's estimated size would exceed 8 bits/sample. delta12-zstd-8 only considers orders 1 and 2 (performance option).
• All real coders are round-trip verified per block group.

Summary

One-shot rows include each block's min and max, stored in the block group. Encode time is the full rate-control search for window/fixed rows (Python, single thread), per block; one-shot is a single pass. Every block spans [0, 1], so one-shot's per-block step and the shared window step mean the same thing.

Bits and error per signal kind

Per signal kind: delta0123-zstd-10 / one-shot

Median over the kind's 25 blocks; bits/sample is the block coded on its own (inside the block group it only has a share of the total).

Per signal kind: delta0123-zstd / window

Block group Δ = 0.000679, 0.000635, 0.00063, 0.00063, 0.000655

Median over the kind's 25 blocks; bits/sample is the block coded on its own (inside the block group it only has a share of the total).

Per signal kind: best: rice|zstd|flac / window

Block group Δ = 0.000278, 0.000275, 0.000266, 0.000261, 0.000245

Median over the kind's 25 blocks; bits/sample is the block coded on its own (inside the block group it only has a share of the total).

Per signal kind: best: rice|zstd|flac / fixed

Block group Δ = 9.31e-10, 9.31e-10, 9.31e-10, 9.31e-10, 9.31e-10

Median over the kind's 25 blocks; bits/sample is the block coded on its own (inside the block group it only has a share of the total).