Commands overview

Every command follows the same shape:

hipoq <command> <FILE> [options]

The command comes first, then the input, then flags. FILE may be a single file, a directory, or a glob like "run/*.hipo" — quote globs so your shell passes them through literally.

All commands

Command	What it does
info	events / records / schemas, per-file sizes
dict	schema dictionary, or one bank's columns
banks	which banks carry data, with event/row counts
head	peek at the first N events
tail	the last N events, by seeking rather than reading
records	per-record map: position, events, inflated size
status	decode a REC::Particle.status value
scan	project bank columns across an event range
dump	whole events as JSON, one object per line (NDJSON)
count	count events, optionally under a cut
stats	min/max/mean/std/count of a scalar column
hist	ASCII histogram of a scalar column
quality	occupancy, particle rates, multiplicity, health flags
check	readability, truncation, duplicate events
diff	compare two files: schema + event/cell differences
mc	REC↔MC truth match: efficiency/purity + resolution
dist	compare two files by a column's distribution
skim	write a new file with only the events passing a cut
sample	write a subset (fraction / stride / first-N)
split	split into contiguous chunk files
merge	event-mixing overlay (background onto signal)
doctor	repair a malformed file
convert	re-encode into another codec, parts written in parallel
bench	measure read throughput here and name the right -j
index	per-record index sidecar, and what a cut could skip
tui	interactive terminal event explorer
completions	shell completion script
man	roff man page

Global flags

These may appear anywhere on the command line (before or after the positional arguments) and apply across commands.

--tag, --tag-any, --record-tag

HIPO carries a tag in each event header and each record header. These select on them, and apply to every command that reads events. Event tags covers the whole subject — finding out what a file uses, seeing the tag, and how it survives the commands that write files.

hipoq dump run.hipo --tag 3              # events whose tag is exactly 3
hipoq scan run.hipo --bank REC::Particle --tag-any Dvcs
hipoq count run.hipo --record-tag 7

flag	keeps	cost
--tag N	events whose tag equals one of these	reads the event header; inflates nothing
--tag-any NAME|MASK	events with any of those bits set	same
--record-tag N	records whose tag matches	skips the record without decompressing it

--tag is for a tag used as one value; --tag-any for a tag used as a set of flags, where it accepts either a name the file declares or a mask (0x6, 12, 0b110), with repeats ORed together.

Names come from the file's own tag registry, so info lists them — without that the flag is unguessable, since nothing else reveals a file has named tags. An unknown name is an error listing the known ones: a silently-zero mask would select nothing and read as missing data.

These compose with --where, and are much cheaper

A --where cut has to inflate the banks it references. A tag test reads a header. Put the tag first when both apply and the tag does most of the narrowing.

-r, --require BANK

Keep only events that contain all of the named banks. Repeatable and ANDed (-r REC::Particle -r REC::Event).

This is a selection cut, not a projection — events missing any required bank are dropped, so count, scan, skim, and the rest all see fewer events. It is also fast: non-matching events are skipped by peeking the record's bank-presence table, without inflating any bank stream.

hipoq count "skim/*.hipo" -r REC::Particle -r REC::Event

--topology SPEC

Select events by final state, in the notation you would write in a talk:

hipoq count run.hipo --topology "e- pi+ pi- p"    # exactly this, nothing else
hipoq count run.hipo --topology "e- p X"          # at least this (X = anything else)
hipoq count run.hipo --topology "e- 2pi+ pi-"     # a count prefix repeats

Exclusive by default: the event must hold exactly the listed particles and nothing more. X makes it inclusive.

This is not expressible with --where, which compares one value at a time and so can neither count the particles in an event nor forbid a fourth one. Applies wherever --where does — count, scan, dump, skim, sample, the TUI.

Every other command refuses it rather than accepting it and reporting on the whole file: quality --topology "e-" used to echo the topology line and then measure every event anyway. To narrow those, select first and run them on the result:

hipoq skim run.hipo dvcs.hipo --topology "e- p g"
hipoq quality dvcs.hipo

--explain is the exception — it describes a selection without running one, so it still explains a topology on any command.

Names accept the spellings people type (e-, e, electron, E-), commas or spaces as separators, and a bare PID for anything not in the table. Light nuclei are the Geant3 codes CLAS12 writes (d=45, t=46, he4=47, he3=49), not the PDG 10LZZZAAAI form — a topology written the PDG way matches nothing.

The parsed topology is echoed to stderr before any work, so a mangled shell quote shows up instead of quietly selecting something else:

topology: e- g p  (exclusive — nothing else)

note

-r narrows the event set globally, but it does not narrow the tui explorer, which navigates by raw event index. Use tui --where and m/M to hop between matching events there.

-j, --threads N

Worker threads for the parallel sweeps: count, stats, hist, banks, dist and index. 0 = all cores, 1 = sequential. Default: 0.

It is also the memory dial — each worker holds its own record decompression buffer, so peak RSS scales with the thread count rather than with the file. On a 256-core node, stats over an 8.5 GB / 598,738-event DST held 61 MB at -j 1, 341 MB at -j 16 and 1157 MB at -j 64. Under a batch memory limit, pass -j 8.

--events A..B

Restrict a command to a half-open span of events. 500..600, 500.. (to the end) or ..600 (from the start); a span past the end is clamped.

hipoq count run.hipo --events 1000..2000
hipoq stats run.hipo --col REC::Particle.px --events ..50000
hipoq skim  run.hipo slice.hipo --events 1000..2000
hipoq head  run.hipo -n 5 --events 1000..2000   # the slice's first five

Every command that reads events takes it, so count --events 1000..2000 answers for exactly the events dump --events 1000..2000 prints. Only the records the span touches are read, so the cost follows the slice rather than the file.

The commands that read the whole file by construction — info, dict, records, doctor, convert, merge, bench, index, tui — refuse it rather than accept it and quietly ignore it. To convert or repair a slice, cut it out with skim --events first.

Which events A..B names

File event indices, unless one of the chain-level filters is in play — --require, --tag, --tag-any, --record-tag. Those are applied when the file is opened, so --events then counts surviving events, the same convention dump and scan use for the event numbers they print.

--topology and --where are not in that list: they are evaluated per event as it is read, so the window still names file indices and the cut is applied inside it. On a 200-event file where --topology "e-" keeps 20, count --events 0..10 --topology "e-" reports 1 — the matches among the first ten events — while count --events 0..10 --tag 1 reports 10, the first ten tagged events.

stats, hist, dist and banks read columns by file index and cannot express the surviving-event space, so they refuse that combination; skim the selection to its own file first, then measure it.

--pdg

Decode pid columns to particle symbols (e-, p, pi+, …) in the human-facing output: scan, head, tui, and dump --format table. The data encodings (csv, ndjson, json) deliberately stay numeric so downstream tools get clean numbers — including dump's default NDJSON stream, which never decodes.

--explain

Print how each --where was compiled — the driver bank, and how every cross-bank reference reaches it — then exit without reading any event. A cross-bank reference is summed, gathered or broadcast depending on which banks carry a pindex, and those give materially different answers from the same text, so this shows which one a given cut actually got.

hipoq count run.hipo --where "REC::Particle.pid == 11" --explain

Being global, it also explains --topology on commands that have no --where at all, such as stats, hist and quality. Given nothing to explain it says so and exits 2:

$ hipoq info run.hipo --explain
--explain: this command has no --where or --topology to explain

Exit status and errors

Commands exit non-zero on error. A misspelled bank or column in a --where expression is a hard error, not a silent zero — so a cut that references REC::Partical (typo) fails loudly rather than quietly matching nothing.

bench — what -j should I use?

hipoq bench run.hipo --max-threads 32

-j 0 (all cores) is the default, and on a farm node it is the wrong answer. Measured on ifarm2402 against a 9.1 GB DST, a full read scaled at 96% efficiency to 4 threads, 65% at 16 and 46% at 24 — and past the knee more threads made it slower. Only a measurement on the machine in front of you can say where that knee is, so bench sweeps the thread ladder and names it, alongside the file's layout (events, records, inflated bytes per record) which costs nothing to read from the trailer.

It also times a one-column read against reading every column of every bank, which is a rough guide to how much a split codec would buy you.

What it deliberately does not tell you

It does not claim to identify the file's codec. Three attempts to turn that ratio into a whole-record-or-not verdict each misclassified one of the two real files available to check against, because both sides of the comparison pay to extract columns as well as to inflate streams. It reports the ratio with two calibration points instead — the same 150,000 events measured 5.6× as stock Lz4 and 17.4× as lz4-per-column — and leaves the conclusion to you.

Two further guards worth knowing: the sweep is round-robin, because timing all repetitions of one thread count before moving on let a warming page cache flatter the high counts (the first version recommended the largest count off an edge that a second pass did not reproduce); and it refuses a file too small to time, below a 100 ms single-threaded read, rather than reporting confident noise.

completions and man

hipoq completions bash > ~/.local/share/bash-completion/completions/hipoq
hipoq completions zsh  > "${fpath[1]}/_hipoq"
hipoq completions fish > ~/.config/fish/completions/hipoq.fish
hipoq man > ~/.local/share/man/man1/hipoq.1

Both are generated from the same argument definitions the binary parses with, so neither can drift from the real flags. Neither opens a file — which matters for completions, since the point is to run it in a fresh shell where there is no file yet.

index — can a cut skip records?

hipoq index run.hipo --col REC::Particle.pid --for "REC::Particle.pid == -321"

A record is the unit of decompression, so reading less means skipping whole records, and that is possible only when the cut is false for every event in the record. index writes a per-record summary sidecar (FILE.hipoidx) and reports whether that condition is ever met on your data.

Usually it is not. On a stock CLAS12 DST (~307 events per record) the median record contains all twelve distinct pids in the file:

pid	skippable at 307/rec	at 50/rec	at 25/rec
K−	1.5%	85.1%	92.1%
e+	0.2%	29.1%	54.1%
e−, p, π±	0.1%	0%	0%

It works for rare values on a finely-recorded file and never for common particles at any granularity. Record size is the lever, so convert takes --max-record-events: at 50 events per record the file grows ~14% and scans faster (0.038 s against 0.051 s at 16 threads — more records means more parallel work).

It does not make reads faster yet

Wiring record-skipping into count --where gave the right answer and was 4.5× slower on a real file — 0.59 s against 0.13 s — while correctly skipping 85% of events. Reading a subset of records means seeking per event, which runs at 55 kev/s against the streaming scan's 6,403; breaking even would need >99% of records skipped, which nothing realistic reaches. That path was removed rather than shipped behind a flag. Exploiting the index needs a ranged streaming read in oxihipo; --for exists to tell you whether it would be worth building.

Skipping is deliberately conservative — only a top-level BANK.COL == k or in (...) prunes a record. A !, an || spanning two columns, or an inequality contributes nothing: in each case a record containing none of the values may still hold matching events. A stale sidecar is detected by event count and file lengths, and ignored with a warning rather than trusted.