Data quality & integrity

Two commands answer "is this file healthy?" — quality looks at the physics content, check looks at the bytes.

`quality`

A one-pass data-quality scan reporting:

bank occupancy — what fraction of events contain each bank, and its average row count;
particle rates & multiplicity — how often each pid appears, and the per-event multiplicity distribution;
health flags — NaN/Inf values, zero-momentum particles, and negative-beta entries.

hipoq quality rec.hipo
hipoq quality rec.hipo --max 20000   # sample the first 20000 events

Flag	Meaning
`--max N`	stop after scanning `N` events (`0` = whole file)

The health flags catch the classic reconstruction pathologies: a particle with p == 0, a beta < 0, or a stray NaN in a covariance — the kind of thing that quietly poisons a downstream histogram. quality tells you how many and where; use scan --where to pull the offending events.

# after quality flags them, look at the zero-momentum particles
hipoq scan rec.hipo --bank REC::Particle --cols pid,px,py,pz \
  --where "p() == 0"

`check`

Integrity diagnostics at the file level:

readability — does every record and event decode?
truncation — does the data end where the trailer says it should?
duplicates — are there byte-identical repeated events?
detector links — does every pindex point at a particle that exists?

hipoq check rec.hipo

files:   1
records: 326
trailer event count: 100000

Readability
  events read OK: 100000
  ✓ all events decoded

Truncation
  ✓ read count matches the trailer index (100000)

Duplicates (byte-identical events)
  ✓ none

Detector links (pindex in range)
  ✓ every pindex points at a particle that exists

OK — no integrity issues found

The pindex check is the one no other diagnostic gives you. A detector row whose pindex falls outside its event's REC::Particle points at a particle that is not there — nothing about the file is malformed, it reads and it decodes, so a join on it either drops the row or credits the wrong particle. That is the corruption that changes physics without announcing itself.

The exit status is the verdict: 0 when clean, 1 when anything above is ✗, so hipoq check f.hipo && process f.hipo does the right thing.

check diagnoses; it never modifies the file. When it finds a problem, the fix is doctor, which salvages the readable events into a fresh file.

check vs doctor

Think of check as a health report and doctor as the repair. check streams and stops at the first unreadable record (telling you where it broke); doctor reads by random access so a bad record only costs its own events.

quality​

check​

`quality`

`check`