Writing & transforming files

Six commands write new HIPO files. skim selects by content, sample and split select by position, merge overlays several inputs into one event stream, convert re-encodes into another codec, and doctor repairs.

Most copy events verbatim — same dictionary, same per-event tags, every bank preserved. Two rebuild an event instead of re-emitting its bytes: skim --banks, which drops banks, and merge, which concatenates rows from several inputs. Both carry the event tag across (merge takes the signal's, input 0), but neither is a byte-for-byte copy.

All six refuse to write onto one of their own input files.

Each takes a --compress codec (default lz4-per-bank).

skim

Write a new file containing only the events that pass a --where cut — the natural "extract my electrons" operation.

# keep only events with an electron
hipoq skim run.hipo electrons.hipo --where "REC::Particle.pid == 11"

# per-column compression for a scan-heavy output
hipoq skim run.hipo electrons.hipo --where "REC::Particle.pid == 11" --compress lz4-per-column

Flag	Meaning
--where E	keep only events passing the cut (repeatable, ANDed)
--compress C	output codec (default lz4-per-bank)
--banks A,B	keep only these banks — a lean DST
--events-from FILE	write only the events listed in FILE (- for stdin)

With no --where, skim copies every event — so it doubles as a re-compress / merge:

# merge a run's files into one, recompressed
hipoq skim "run/*.hipo" merged.hipo --compress lz4-per-bank

A global -r BANK narrows the skim to events that carry BANK (it's an event cut), and the summary reports kept/total so any drop is visible.

Bank projection — a lean DST

hipoq skim run.hipo lean.hipo --banks REC::Particle,REC::Calorimeter

On 5000 events of a CLAS12 run: 274 banks to 2, 83.1 MiB to 3.0 MiB, with diff reporting the kept banks identical to the source.

This is the one operation that cannot copy events byte-for-byte — a raw event blob cannot have a bank removed — so it rebuilds each event and is slower. Without --banks the fast copy path is untouched.

Dropping the bank a pindex points at

Keep a detector bank without REC::Particle and every surviving pindex refers to something the file no longer contains. It still reads, it still decodes, and every join on it is wrong. skim names the dangling columns and tells you which bank to add. A --banks entry the file does not declare is an error, not a silent no-op.

Reusing a search — --events-from

Finding the interesting events is the expensive pass. Spend it once:

hipoq count run.hipo --where "REC::Particle.pid == 11" --list > electrons.txt
hipoq skim run.hipo electrons.hipo --events-from electrons.txt

The file is one global event index per line; # starts a comment and - reads stdin. Lists are sorted and deduplicated on read, since the consumer cannot honour either the original order or a repeat.

Indices are global — the space dump --events and tail use, not the filtered position. --events-from seeks to them rather than streaming, so extracting 1151 events from a 5000-event file reads the records holding them instead of all of them, and the gap widens with file size. An index past the end is reported, not clamped — clamping would silently duplicate the last event.

merge

Event-mixing overlay: event i of every input becomes one output event carrying all of their banks. This is background overlay — what you do to make a simulation look like beam-on data.

hipoq merge signal.hipo background.hipo -o overlaid.hipo

Flag	Meaning
-o FILE	output file
--compress C	output codec

It is not concatenation — skim with a glob already appends files end to end. Here the events are combined, not appended.

Because two inputs both carrying REC::Particle end up with their rows concatenated, every pindex in a later input has to move by the number of particles the earlier ones contributed. Getting that wrong is silent: the file reads, the banks decode, and every detector hit from the background is credited to the wrong particle. So banks are rebuilt row by row with the row references shifted explicitly, and negative indices — CLAS12's "no associated row" sentinel — are left alone.

Runs to the shortest input and says so. Inputs that disagree about a bank's layout are refused rather than decoded against each other's schema, which is what happens if you try to overlay two different cooks.

sample

Write a subset of events chosen by position — a quick test file, or a downsample. Exactly one selection mode is required.

hipoq sample run.hipo small.hipo --fraction 0.01   # ~1% pseudo-random subsample
hipoq sample run.hipo small.hipo --every 100       # every 100th event (stride)
hipoq sample run.hipo small.hipo --count 500       # the first 500 events

Flag	Meaning
--fraction F	keep a pseudo-random fraction in (0, 1]
--every N	keep every N-th event
--count K	keep the first K events
--seed S	seed for --fraction (default 0)
--where E	only sample among events passing the cut
--compress C	output codec

--fraction is deterministic: the same file, fraction, and --seed always produce the same subset — so re-runs are reproducible, and a different --seed reshuffles. (Internally the keep/drop decision is a hash of the event index and seed, so no two runs diverge.)

A --where cut composes with any mode — --count 500 --where "REC::Particle.pid==11" is "the first 500 electron events":

hipoq sample run.hipo first500_e.hipo --count 500 --where "REC::Particle.pid == 11"

split

Cut a file into contiguous chunks — events 0..M in the first file, M..2M in the next, and so on — for fanning a file out across parallel batch jobs. Because chunks are contiguous, stitching them back together reproduces the original event stream.

hipoq split run.hipo out/ --chunks 8              # 8 roughly-equal files, for 8 workers
hipoq split run.hipo out/ --events-per-file 50000 # fixed-size pieces

Flag	Meaning
--chunks N	produce N contiguous files
--events-per-file M	produce files of M events each
--prefix P	output filename prefix (default part)
--compress C	output codec

Files are named PREFIX_0000.hipo, PREFIX_0001.hipo, … under OUTDIR, which is created if it doesn't exist. Each chunk is a complete, independently readable HIPO file (its own dictionary and trailer).

Exactly one of --chunks / --events-per-file is required. --chunks N sizes each file at ceil(total / N) events, so the last file may be shorter.

convert

Re-encode a file into another codec, writing the parts concurrently.

hipoq convert run.hipo converted/ --compress lz4-per-column --parts 16
hipoq count "converted/*.hipo"

Flag	Meaning
--compress C	target codec (default lz4-per-column)
--parts N	output files, written in parallel (default 8)
--prefix P	filename prefix (default part)
--max-record-events N	cap events per output record

Converting to a split codec is the largest read win available — around 3× on a warm selective read — and until this existed it was too slow to bother with. sample --compress lz4-per-column took 274 s for 150,000 events against 12 s for plain lz4, because HC-compressing every bank stream is expensive and the write path used one core. On ifarm2402, over those same events:

--parts	wall	vs serial
1	243.4 s	—
8	30.7 s	7.9×
16	15.8 s	15.4×

That takes a 598,738-event DST from about eighteen minutes to about one. Verified with diff: all 150,000 pairs identical to the input. convert also refuses to report success if the number of events written does not match the number read — a short conversion yields a file that opens and reads cleanly, which is exactly the failure a human would not notice.

The output is a directory, not a file

Compression happens inside the writer, which cannot be handed an already-compressed record, so one output file is inherently one thread. What parallelises is several files over disjoint event ranges — and a chain is a first-class input everywhere in this tool, which is why the result is read with a glob. A cooked run already looks like this (rec_clas_022083.evio.00000-00009.hipo is one of several). Use --parts 1 for a single file.

Record size is a knob too

--max-record-events N caps events per output record. That is the granularity at which a reader can skip anything — see index — and smaller records also scan faster: at 50 events per record a full read took 0.038 s against 0.051 s at the stock size (16 threads), because more records means more parallel work. The cost is ~14% more file and slower column extraction (0.419 s against 0.325 s).

Two costs before you convert a shared sample

Split codecs are oxihipo extensions (wire tags 6 and 7) and cannot be read by C++ hipo4 — so not by coatjava or clas12root either. Convert a working copy and keep the original. Ranges are also record-aligned, so an input holding every event in one record converts single-threaded whatever --parts says; the command tells you when that happens rather than letting it look like slowness.

doctor

Salvage what it can from a corrupt or truncated file and write a fresh, clean copy.

hipoq doctor broken.hipo fixed.hipo            # recover every readable event
hipoq doctor broken.hipo fixed.hipo --dedup    # also drop byte-identical duplicates

Flag	Meaning
--dedup	drop byte-identical duplicate events while rebuilding
--compress C	output codec

Unlike skim (which streams and stops at the first bad record), doctor reads each event by random access, so a corrupt or truncated record only costs its own events — the rest are recovered. It reports recovered/total and how many events were unreadable.

Missing or corrupt trailer

The classic crashed-writer file — records intact, but the trailer index (written last) never made it to disk — is handled transparently. The reader rebuilds the index by scanning the records, so doctor recovers every complete record and writes a fresh file with a valid trailer.

A destroyed file header

This used to be the end of the line: the reader parses the 56-byte file header before anything else, so a file missing it could not be opened at all.

doctor now falls back to finding the records itself. The header is only bookkeeping — magic, version, counts, where the dictionary and trailer sit — and every record carries its own header and magic, so the records can be located by scanning:

$ hipoq doctor nohdr.hipo fixed.hipo
this file's header is unusable, so `doctor` located the records by scanning instead:
  60 event(s) in 6 record(s)
  the dictionary survived: 1 schema(s)
doctored fixed.hipo: recovered 60/60 events

The repaired file opens normally, and diff reports its events identical to the original's.

The dictionary may not survive

It lives in the record immediately after the header, so damage that took one often took the other. doctor says which happened. Without it the events are still recovered and still copied verbatim — but their banks have no names or column types, because those appear nowhere else in the file, and the output has zero schemas. Recovering the data is then only half the job; you would need the dictionary from a sibling file of the same cook.

What it still cannot recover:

• the events inside a record that is itself truncated, and
• a file with no readable record header anywhere, which is refused rather than turned into an empty output that reads like a successful repair.

Exit status

0 when at least one event was recovered, 1 when none was — the same convention check and diff use for "ran fine, found a problem".

The second case is worth knowing about, because it looks benign:

$ hipoq doctor wrecked.hipo out.hipo
doctored out.hipo: recovered 0/0 events  (0 records, 416 B)
nothing was recovered from this file — no event could be read.

It happens when the file header survives — so the file opens — but no record header does, which is what a truncation inside the first record leaves behind.

A file that cannot be opened at all is a different case, and says so:

$ hipoq doctor junk.hipo out.hipo
Error: `doctor` could not open this file at all, and scanning it for records found
none either, so there is nothing to salvage.
Recovery works by walking records; a file with no readable record header has no
events left to find. If this is not a HIPO file, or was truncated at the very
front, rewrite it from whatever produced it.

Caused by:
    0: file "junk.hipo": invalid HIPO magic at offset 0x0: got 0x33a97ae4, expected 0x4f504948
    1: invalid HIPO magic at offset 0x0: got 0x33a97ae4, expected 0x4f504948

Note "and scanning it for records found none either": a broken file header alone is no longer fatal, since doctor falls back to locating the records by scanning. This message means both routes failed, which is what a file that is not HIPO at all looks like. Every event is gone, and doctor used to report that with a success line and exit 0; in a script it was indistinguishable from a repair that worked.

Truncation anywhere else is much kinder. Records are independent, so cutting a file short loses the tail records and nothing else: what comes back is a contiguous prefix of the original events, byte-for-byte unchanged.

Because it re-reads a record per event, doctor is slower than skim — reach for it to repair, not to recompress a healthy file. Pair it with check, which diagnoses what doctor fixes.