fluxcode on swinging-door historian data

← codec comparison (index.html) · the time axis (time.html)

Process historians that compress with swinging-door trending (SDT; PI's compression, and many others) scan each tag at a fixed rate, drop scans that are within an exception deadband (ExcDev) of the last reported one, and archive only the points that swinging door keeps: actual scans, at irregular times, such that straight lines between them stay close to every snapshot (nominally within CompDev; in practice up to twice that). That is what an export of the archive gives you, and what this page encodes: one block group per tag and day, values and exact timestamps. The data are six simulated tags (5 days each, 1 s scans) chosen to cover what SDT archives look like: from 1 point in 2,000 scans (a valve) to nearly every scan (a noisy tag with a tight CompDev).

Store the archived points themselves. A historian can also export a regular series (interpolated, or held as a forward fill gives), but that throws the points away and costs more: resampled exports.

Regenerate: cd experimental && uv run python make_sdt_report.py (a few minutes; sizes and errors only, no timings). The simulator is tslab/common/historian.py.

Findings

• The noise floor applies only where the historian kept nearly every scan. Swinging door's points look like white noise without being noise, so a block needs 90% of its intervals one scan. At each tag's own CompDev, no archive gets it: the densest, motor-current, has 80%–84% of intervals one scan, and the decimal tags decode exactly as with the floor off. In the CompDev sweep it applies to motor-current at 0.25× its CompDev (94%–96% one scan): -41% in size, max error 0.6 of its own CompDevs; motor-current at 0.5× its CompDev (89%–92% one scan): -33% in size, max error 0.6 of its own CompDevs: CompDev is set far below the noise there (noise floor). At the defaults the float32 tag is within 0.0008 CompDevs at 2.47 bytes per point.
• Store the archived points, not a resampled export. Read back at the 1 s scan rate, the same archive takes 0.6–7.8× the bytes interpolated and 0.6–1.9× held (resampled exports), and fluxcode can't recover the points from either.
• Against lossless coders, on the dense tags fluxcode is 1.7–2.3× smaller than zstd of the columns and 1.7–6.1× smaller than Gorilla. On the sparse tags (tank-level, valve: about 50 points a day) a day costs a few hundred bytes whatever the coder, and block overhead decides: hourly blocks cost 2.2–2.5× what one block per day does (layout). Fixed time buckets keep a store simple, and the inflation only hits data that is tiny anyway.
• Timestamps are cheap when scan-aligned: 0.13–0.70 bytes per point on the dense tags, since every interval is a whole number of scans. Device timestamps at ms resolution add 1.11–1.31 bytes per point.
• fluxcode adds almost nothing to SDT's error: at the defaults, the trend through the decoded points has the historian's own max error (accuracy), which is about 2 CompDevs from swinging door itself.
• SDT doesn't always pay. At the same max error, fluxcode on every scan is smaller than SDT + fluxcode on flow, ph-float32, motor-current (SDT or every scan): on noisy tags SDT keeps most scans anyway, and a regular grid's times cost almost nothing. On smooth or flat tags SDT wins by a wide margin.

The tags

Each tag is scanned every second for a day, then filtered by exception (ExcDev = CompDev / 2, the usual rule; the scan before each exception is reported too) and by swinging door (CompMax 8 h). Times are whole seconds, as a scan class gives. Errors are in CompDevs, CompDev being the historian's nominal tolerance. The historian's trend (straight lines between archived points) strays from the scans by up to about 2.5 CompDevs: swinging door archives the real scan where the doors close, and the line to it is only guaranteed within CompDev at that scan, so earlier snapshots can be up to 2 CompDevs off it (about 1.9 here); the exception filter adds the rest.

Points per day: mean over the days. Trend vs scans: the worst over the days, and the mean RMS, in CompDevs, before any fluxcode.

Size

Bytes per archived point over all 5 days, everything a decoder needs included. Raw is an int64 time and a float64 value (16 bytes). zstd and Gorilla are lossless: zstd-3 of the time deltas and the values as two int64/float64 columns; Gorilla with its own delta-of-delta timestamps (in seconds) and XOR values. fluxcode is one block group per tag-day in 1 h blocks (layout), with exact times; its values and times columns split its bytes, the times' being those of the same blocks with the noise floor off minus the same blocks without times.

Max error: the worst |decoded − archived| over the days, in CompDevs. Exact: the share of points that decode bit-exact. Values and times: the times' bytes are those of the same blocks with the noise floor off minus the same without times. The noise floor section explains where the defaults and the noise floor off differ.

Accuracy against the scans

What matters downstream is the trend: straight lines between the decoded points, compared with the scans the historian saw. SDT alone is the historian's own error; fluxcode's is added on top. In CompDevs, worst over the 5 days.

Noise floor

fluxcode's noise floor (default 0.25σ) coarsens the step of blocks whose samples look like white measurement noise. An SDT archive's sparse points look like that without being noise: swinging door keeps exactly the points a straight line can't predict. So a block with irregular times gets a floor only if it has a cadence: at least 90% of its intervals one scan (between half and 1.5 times the median of 15 evenly spaced intervals). Then the noise is measured on consecutive scans only. An archive that kept nearly every scan is raw sensor data, and gets the floor; any sparser one gets none and keeps its values. Below, each tag's archive at five CompDevs (ExcDev half of it), 1 h blocks, all 5 days, against the same with the noise floor off.

One-scan share: the range over the 1 h blocks of at least 256 points (smaller blocks get no noise floor). Blocks floored: those that decode differently with the noise floor off, of the blocks of at least 256 points. Max error: the worst |decoded − archived| at the defaults, in CompDevs of the tag's own (the first table), so that the rows of a tag compare. Exact: the share of points that decode bit-exact at the defaults.

Size vs accuracy

fluxcode's B (max_quantize_bits) from 4 to 16 with the noise floor off, and the defaults (diamond: B = 16 and the noise floor). The error is the worst |decoded − archived| over the 5 days, in CompDevs; exact decodes sit on the bottom line. B sets the step from each block's range, so the same B gives a different error on each tag: there is no B that means "a tenth of CompDev".

Each cell: bytes per archived point (values + times), and below it the max error in CompDevs.

Block layout

An archive has few points per hour, so how a day is cut into blocks matters more than for 1 kHz data: every block stores its own anchors, sizes and time reference. Bytes per point at the defaults, all 5 days.

Time blocks come from encode_time_blocks (empty hours cost almost nothing); point blocks from encode_group(block_len=…). One block per day is 65,535 points at most, so the densest tag gets two.

Timestamps

Scan-aligned times are whole seconds; with source timestamps each point keeps the device's time, a scan delayed by 0–200 ms at ms resolution (as OPC timestamps often are). The values are the same. Bytes per point for the times only (fluxcode defaults, the block group minus the same blocks without times), and Gorilla's whole size for comparison.

Resampled exports

A historian can also be read at a fixed rate: interpolated (straight lines between the archived points) or held (each scan takes the last archived value, as a forward fill gives). If that is all you have, fluxcode stores the regular series (defaults, 1000-scan blocks, its 1 s times included), but the archived points are cheaper and keep the information: an interpolated series is off the decimal grid, and its rounded ramps leave residuals of a step or so. kB per day, mean over the 5 days; errors are max |decoded − read-back| in CompDevs.

SDT or every scan?

If the scans are still available (at the source, or before the historian compresses), fluxcode can store every one of them instead. Here every scan of the day is one block group (1000-scan blocks, the 1 s times included): at the defaults, and at the coarsest B (noise floor off) whose max error is within a budget: one CompDev (SDT's nominal tolerance), or the error SDT actually made that day (the same max error, a fair comparison). The SDT route is the archive with fluxcode (defaults), its error that of the trend. kB per day, mean over the 5 days; errors are the worst max |decoded − scan| in CompDevs.

The same comparison as CompDev changes (ExcDev stays half of it), mean kB per day: