Testing and fuzzing

For anyone changing hipoq. Everything here runs on stable Rust with cargo test — there is no separate campaign to remember to start.

cargo test                    # everything, including the fuzz targets
cargo fmt --all --check
cargo clippy --all-targets --all-features -- -D warnings

Where the tests live

	what
src/** unit tests	the arithmetic: expression evaluation, topology matching, column arrangement, status decoding, CSV quoting, the table layout, doctor's recovery, the TUI's focus and theme
src/fuzz.rs	property tests over the parsers, and the commands against corrupt files
tests/output_format.rs	runs the shipped binary and parses its output with something else
examples/	fixture generators — gen_clas12, gen_mc, gen_truth, gen_badpindex, gen_extra, plus strip_trailer for doctor

Seven TUI tests are #[ignore]d because they need a real HIPO file. CI generates one and runs them with --include-ignored, then asserts nothing was skipped — a test gaining #[ignore] later is the usual response to flakiness, and it silently shrinks coverage while the suite still reads green.

Fuzzing

src/fuzz.rs uses proptest. The property is almost always the only one worth asserting: Err, never a panic. A CLI that diagnoses malformed input is behaving; one that panics loses the work and takes the exit code with it.

Two kinds of target, and the prefix is part of the contract:

• parse_* — the hand-written parsers: the --where grammar, --topology, --events A..B, hist --range, the --events-from index list, status over every i32. ~0.3 ms per case, so this is where a raised case count goes. Each has a second generator biased toward strings that look like the grammar, because uniform random text fails at the first character and never reaches the operator precedence or the in (...) lists.

• file_* — the commands themselves, against a real fixture with bytes flipped, against truncations, and against random bytes. Each case writes a file and runs one of the 22 read commands through the real argument parser. On the 712-byte fixture that is ~0.7 ms at the soak's own case count — but the cost does not scale linearly, which is the whole reason for the split. See the warning below.

  doctor has its own file_ target rather than riding the shared list, because "did not panic" is far too weak a property for a repair tool: it asserts that the output opens, and that repairing the output again loses nothing. That target found a genuine hang — see below.

PROPTEST_CASES overrides with_cases

Raising it multiplies every target, including the file_* ones, each of which writes a file and runs a command. Raise the two groups separately:

PROPTEST_CASES=20000 cargo test fuzz::parse_    # 8 targets, ~46 s
PROPTEST_CASES=400   cargo test fuzz::file_     # 4 targets, ~1 s

Those two lines are exactly what fuzz:soak runs — about 50 s of test time on a laptop, so the job's 30-minute budget is almost entirely the build.

Do not raise it globally. PROPTEST_CASES=20000 cargo test fuzz:: puts the file_* targets at 20,000 too, and that run did not finish inside ten minutes locally — against the ~1 minute that scaling the 400-case run linearly would predict. The per-case cost climbs with the case count, so the blowup is worse than the 50× ratio suggests, and a file_ target misnamed as parse_ inherits that by its name alone.

Two things the harness has to do that are easy to miss:

• It silences the commands' stdout. They write through io::stdout() directly — that is what makes them fast — and cargo's harness only captures println!. Every case otherwise leaks its full output to the real stdout: 1.9 KB per case measured, over 100 MB of CI log at soak scale, which once took a job to the runner's one-hour limit while the tests themselves ran in under a minute.
• run(Cli) -> Result<i32> exists so main can own the exit. The diagnostics used to call std::process::exit(1) inline, which inside a test kills the runner.

When proptest finds a failing case it writes a minimised reproducer to a .proptest-regressions file. Commit it — that is a permanent regression test, and CI fails the soak if one appears.

Benchmarking a change

scripts/bench-where.sh measures the --where engine against a real file. It exists because the synthetic fixtures got three things wrong, and only running it on run 22083 revealed them:

scripts/bench-where.sh --file "/path/run*.hipo" --baseline HEAD~8 --threads 32

Six sections, each re-measuring one specific claim: cost per distinct column reference, cost per comparison as its control, whether count(...) aggregates short-circuit, the three cross-bank join modes, whether a bank-presence pushdown would pay, and end-to-end throughput.

--baseline REF builds that ref in a git worktree and reports both — but only after checking the two binaries select the same events for seven queries. An A/B whose faster side is wrong is worse than no measurement, so it exits rather than print timings.

What it corrected, and what to expect if you run it:

• The engine is ~6% of a filtered scan at -j 1 — 7.43 µs/event with no engine at all, 7.86 with a four-column cut. Decompression is the rest, so the ceiling on optimising the engine is low.
• Per-reference and per-comparison costs are 2× apart, not 17×. The fixture that said otherwise set pid to 11 on only the last row, so a chain of && short-circuited at the first comparison — it was timing a short-circuit, not eight comparisons.
• A bank-presence pushdown pays nothing, even at 0.2% bank occupancy.

Block-per-binary timing drifts as much as the effect

Timing each binary in a block and comparing blocks gave 3–5% run-to-run drift — the same size as the ~7% steps being bisected, which is why one step first read as +4% and then as +7%. Round-robin (time every binary once per repetition, then take best-of) is what made the result reproducible.

What CI runs

job	stage	what
rust:check	test	fmt, clippy, build --locked, the whole suite with --include-ignored, then every command that writes a file plus a read-back
fuzz:soak	soak	the same properties at 20,000 cases. allow_failure — a new finding is to triage, and rust:check is the gate
docs:build	build	the site on merge requests; a broken internal link fails it
pages	deploy	publishes the site from the default branch

fuzz:soak is in its own stage after test so it pulls the target/ cache rust:check pushes. In the same stage they run in parallel and it pays for a cold build of every dependency.

Conventions worth keeping

Check that a fuzz target reaches its assertions. The doctor target looked fine and was mostly idling: measured, only 42 of 300 cases got far enough for it to assert anything, because the shared fixture holds every event in one record, so truncating it is all-or-nothing. A fixture with many small records — what a killed writer actually leaves — took that to 237 of 300. A target that never reaches its property is a slower way of asserting nothing.

Mutation-test a new test. Break the thing it covers and check it fails. Several tests in this repo passed against a deliberately broken implementation until the fixture was fixed: a tiling test used the real 65,536-event window against a 2,500-event fixture, so it only ever took one window; a thread-cap test could not distinguish a cap from no cap on a machine with fewer cores than the cap; a banks test could not tell "extract events 3, 7, 11" from "take the first three" because every event in the fixture was identical.

Check output against something other than a previous build. A byte-comparison catches regressions and cannot catch a format that was always wrong — that is what tests/output_format.rs is for.

Verify perf claims on a real file. Synthetic fixtures are too small and too uniform: they have one detector bank where a DST has 47 pointing at the same REC::Particle, and no T#N array columns at all.

.gitlab-ci.yml script lines are YAML strings. A line containing ": " while unquoted parses as a mapping, GitLab rejects the config, and the pipeline fails having created zero jobs — which reads as infrastructure trouble, not a typo. It disabled all of CI for thirteen commits once. util::ci_config_tests asserts it, because an invalid config is exactly the case where no CI job can catch it.