Comparing files & MC truth

diff compares two files against each other; mc compares reconstruction against Monte-Carlo truth within a file.

diff

Compare this file (A) against another (B) by event index: first a schema diff, then per-event / per-cell numeric differences under a relative tolerance.

hipoq diff a.hipo b.hipo
hipoq diff a.hipo b.hipo --bank REC::Particle    # focus one bank, show first differing cells

Flag	Meaning
--bank B	show a table of the first differing cells for bank B
--tolerance T	relative tolerance for numeric equality (default 1e-9)
--max N	compare at most N events (0 = min of the two counts)
--key BANK.COLUMN	pair events by a column value instead of by index
--samples N	how many differing events to list (default 12)
--events A..B	compare only this half-open range of A's indices

Two numbers are considered equal when |a - b| <= T * max(1, |a|, |b|), so the default 1e-9 ignores floating-point noise while catching real changes. Use a looser --tolerance to diff files processed by different code paths, or a tighter one to demand bit-for-bit agreement.

schema: identical (18 banks)
events: A=100000  B=100000  comparing 100000

REC::Particle differs in 37 events
  first diff — event 12, row 0, column px:  A=0.5123  B=0.5108  (Δrel 2.9e-3)

diff is the tool for validating a reprocessing: run the old and new code, then diff the outputs to see exactly which banks, events, and columns moved.

Pairing by key, not by position

By default event i of A is compared with event i of B, which assumes the two files are in the same order. --key pairs them by a column value instead:

hipoq diff a.hipo b.hipo --key RUN::config.event

Use it whenever "event 500" does not mean the same event on both sides — after a skim, or a re-cook that reordered. Index pairing turns a single inserted event at the front into "every event after it differs". On a real skim comparison, index pairing reported 99 of 100 pairs differing; by key, every matched pair was identical and the events that exist only in A were reported as such.

Keys must be an integer column: pairing is exact-match, so a float cannot identify an event. Keys are read columnar, so aligning a 600k-event file does not first decode every event.

An event number alone may not be unique

A DST that concatenates sub-files restarts RUN::config.event, so several events share a key. A repeated value identifies no single event, so those events are not paired at all and are reported as ambiguous — pairing them against an arbitrary occurrence manufactured seven differences on a real file before this was the behaviour.

Where the divergence starts

diff reports the shape of the difference, not just the count — a change that alters everything from one event onward and a sporadic float difference produce the same "N events differ" line and mean completely different things:

divergence: identical through 1 pair(s), then every one of the remaining 99 differs
divergence: scattered — first at A event 0 / B event 2, then 7 of the following 105 (6.7%)

Banks that only one file's dictionary declares are not compared per event — the schema diff already says so once, and repeating it per event added nothing but noise. The count of such banks is noted so their absence is not mistaken for agreement.

dist

Compare two files by the distribution of a column rather than cell by cell.

hipoq dist a.hipo b.hipo --col REC::Particle.px
hipoq dist a.hipo b.hipo --col REC::Particle.px --bins 100 --tolerance 0.02

Flag	Meaning
--col C	(required) the scalar column, BANK.COLUMN or a bare name
--bank B	the bank — needed only when --col is a bare name
--bins N	bins for the shape comparison (default 50)
--tolerance T	tolerance on the moments and the shape distance (default 0.01)

diff answers "same events, same numbers" — right for a bit-for-bit regression test, wrong for did the physics change? Once reconstruction changes, event N is no longer comparable to event N cell by cell (track counts move, particles appear and vanish), so diff calls everything different while saying nothing about whether the spectrum moved. Two files holding the same events in a different order are identical under dist.

It reports the moments side by side, then a shape comparison on a binning common to both files, then the bins carrying the disagreement — so the output says where the distribution moved:

  total variation distance: 0.000864
  = 0.086% of the probability mass falls in different bins
  largest bin differences:
    [   1659.0000,    2212.0000)  A     16093  B      8122   -0.086%

The distance is total variation — half the summed absolute difference of the two normalised histograms — so it lands in [0, 1] and reads directly as "this fraction of the probability mass sits in different bins".

Two defaults differ from diff's on purpose:

• Tolerance is 1%, not 1e-9. This is a statistical comparison, where two honest samples of one distribution differ by order 1/sqrt(N); a float-equality tolerance makes every real comparison report "differs" and the exit code useless.
• The count is reported but excluded from the verdict. Half a file is the same distribution with half the statistics, and that is the common case.

Bounded memory, like stats: the two bin arrays and two accumulators, never the column.

mc

Reconstruction-vs-truth comparison: per-PID efficiency, PID purity and momentum/angular resolution.

hipoq mc rec.hipo
hipoq mc rec.hipo --min-quality 0.98      # accept only high-confidence matches
hipoq mc rec.hipo --max 100000            # first 100k events

Flag	Meaning
--min-quality Q	minimum MC::GenMatch.quality (0..1) to accept a match. Default 0 — accept all
--max N	stop after scanning N events (0 = whole file)

The match comes from the simulation, not from geometry

mc reads the MC↔REC association out of MC::GenMatch, the bank coatjava's truth-matching service writes. That service compares the detector hits a generated and a reconstructed particle actually share, and records a matched pindex with a quality in 0..1 — coatjava calls 0.98 "very good" for a charged track.

There is deliberately no geometric fallback. Pairing each MC::Particle with the nearest REC::Particle by opening angle looks reasonable and fails exactly where it matters: two particles emitted close together have indistinguishable angles, so the pairing comes out confident and wrong, and the efficiency built on it looks entirely plausible. A number you cannot trust is worse than no number.

So a file whose simulation ran without that service gets an error, not a table of zeros — "nothing to report" is not the same as "the reconstruction found nothing":

N event(s) carry MC::Particle but no MC::GenMatch, so no MC↔REC link exists to report. […] Re-run the simulation with that service enabled (it writes MC::GenMatch and MC::RecMatch).

If only some events lack it, those are skipped and counted in a warning rather than quietly folded into the denominator — an efficiency is only as meaningful as the set it was computed over. Matches dropped by --min-quality are reported the same way.

What it prints

Efficiency  (matched by: MC::GenMatch truth matching)
┌──────┬──────┬─────┬───────┬───────┬─────────┐
│ pid  ┆ name ┆ mc  ┆ found ┆ eff%  ┆ pid_ok% │
╞══════╪══════╪═════╪═══════╪═══════╪═════════╡
│ 11   ┆ e-   ┆ 200 ┆ 200   ┆ 100.0 ┆ 90.0    │
│ 2212 ┆ p    ┆ 200 ┆ 160   ┆ 80.0  ┆ 100.0   │
└──────┴──────┴─────┴───────┴───────┴─────────┘

Resolution (correctly-identified matches, n = 340)
  Δp/p   mean +0.0006   std 0.0101
  Δθ     mean +0.0000°  std 0.1651°
  Δφ     mean +0.0000°  std 0.0000°

• mc — generated particles of that PID; found — those the truth matcher linked to a reconstructed one.
• eff% = found / mc: was the particle reconstructed at all?
• pid_ok% — of those found, the fraction whose reconstructed pid is right. A track can be found and misidentified, which is why these are two columns rather than a single "purity".
• Resolution covers only correctly-identified matches, so a PID mix-up cannot widen the momentum spread. Angles are in degrees, and mean sits next to std because a non-zero mean is a calibration bias rather than resolution.

The header line always states how the matching was done, and names the threshold when --min-quality is set.

This needs a file carrying MC::Particle, REC::Particle and MC::GenMatch. Both the gen_mc and gen_truth example generators write one — see installation. gen_truth is the interesting one: its two truth particles are close enough that geometry pairs them the wrong way round while MC::GenMatch pairs them correctly, which is the case this command refuses to guess at.