Compression codecs

The write commands — skim, sample, split, doctor, and merge — take a --compress codec for the output file. These are the codecs oxihipo supports.

Default: lz4-per-bank.

Codec	Wire tag	Notes
none	0	no compression — largest, fastest to write, useful for debugging
lz4	1	plain LZ4 over the whole record — fast
lz4-best	2	LZ4 high-compression over the whole record — slower to write
gzip	3	gzip over the whole record — slower both ways
lz4-per-bank	6	(default) LZ4-HC, one stream per bank
lz4-per-column	7	LZ4-HC, one stream per column

hipoq skim run.hipo out.hipo --where "REC::Particle.pid==11" --compress lz4-per-column
hipoq skim "run/*.hipo" merged.hipo --compress lz4-per-bank
hipoq split run.hipo chunks/ --chunks 8 --compress lz4-best

Re-encoding an existing file? Use convert

Every command above writes with one thread, and HC-compressing each bank stream is expensive: re-encoding 150,000 events to lz4-per-column through skim / sample takes ~274 s, which is around eighteen minutes for one DST. convert does the same work across several output files at once — 15.8 s for those events at --parts 16 — so converting a sample is a coffee-break job rather than an afternoon.

hipoq convert run.hipo converted/ --compress lz4-per-column --parts 16
hipoq count "converted/*.hipo"

Why the layout matters more than the ratio

The first four codecs compress a whole record as one blob: to read any bank of any event, the reader inflates all of it. lz4-per-bank and lz4-per-column split the record into independently compressed streams, so a read can inflate only what it asked for.

That is what makes the layout choice matter:

• -r/--require gets cheaper. A bank-presence check reads the record's offset tables and never touches a bank stream.
• Narrow reads get cheaper. scan --cols pid,px on a per-column file inflates two streams, not the whole record. On a per-bank file it inflates one bank instead of all of them.
• banks becomes nearly free, because occupancy is in the offset tables rather than in the data.

The split layouts are also not the size penalty you might expect for that. Both use LZ4-HC, and grouping like with like compresses better than a record's interleaved bytes — per-column especially, since a column of px floats is far more uniform than a row of mixed types. What they cost is write time, from the HC compressor.

Choosing a codec

• Anything you will read repeatedly? Keep the default lz4-per-bank, or use lz4-per-column if your reads are usually a few columns of a wide bank.
• Fastest possible write, or debugging a file by hand? none or plain lz4.
• Size matters most? Measure — see below. Which codec wins depends on the data, and the differences between the LZ4 variants are usually small next to the choice of layout.

Measure it on your own file

Ratios depend on what is in the banks, so the honest answer is a one-liner rather than a table. skim with no cut is a straight re-compress:

for c in none lz4 lz4-best gzip lz4-per-bank lz4-per-column; do
  hipoq skim run.hipo "out-$c.hipo" --compress "$c" >/dev/null
  printf '%-16s %s\n' "$c" "$(wc -c < "out-$c.hipo")"
done

Two things worth knowing before reading the result. lz4 and lz4-best can come out byte-identical when the data is repetitive enough that HC finds nothing extra. And gzip is not reliably the smallest — on highly repetitive records it can be more than double plain LZ4, because LZ4's long-match encoding suits that shape better than DEFLATE's window. Neither is a bug; both are reasons to measure rather than assume.

The two split codecs are not readable by C++ hipo4

lz4-per-bank (the default) and lz4-per-column are oxihipo formats. A file written with either cannot be read by the C++ hipo4 reader, and so not by coatjava or anything built on it.

That is fine for a skim you will analyse with hipoq or the oxihipo Python bindings, and wrong for a file you intend to hand to someone else's CLAS12 workflow. For that, write lz4 or gzip:

hipoq skim run.hipo for-coatjava.hipo --where "REC::Particle.pid==11" --compress lz4

Split-codec files and other implementations

The two split codecs carry an on-disk format version — 2 for lz4-per-bank, 1 for lz4-per-column. Those numbers are a cross-implementation contract: the feature/bybank-bycolumn-compression branches of hipo-cpp and hipo-java implement exactly these two versions. Verified on a JLab farm node: hipoq, C++ and Java read the same split-codec files and produce byte-identical checksums, array (T#N) columns included.

Do not use 0.2.2 for split-codec files

0.2.2 wrote these files with the wrong format version (it inherited oxihipo 0.7.0's bump to 3 and 2). Neither other implementation can read them — hipo-cpp segfaults and hipo-java throws failed to decode ByBank record section — and neither can hipoq 0.2.1.

0.2.3 writes the correct versions again. If you produced lz4-per-bank or lz4-per-column files with 0.2.2, re-convert them with 0.2.3. hipoq reads both, so re-converting is lossless.

Compatibility as of 0.2.3, measured by cross-reading:

written by	0.2.1	0.2.2	0.2.3	C++ / Java
stock codecs, any version	✅	✅	✅	✅
split codecs @ 0.2.1	✅	✅	✅	✅
split codecs @ 0.2.2	❌	✅	✅	❌ broken
split codecs @ 0.2.3	✅	✅	✅	✅

Files written with the four stock codecs are byte-identical across all three versions.

Composite banks

A handful of CLAS12 structures (RUN::scaler and friends) carry an inline format string instead of a schema. Only one byte of the bank's header marks it as composite, and the split codecs discard those headers when they take a record apart — so before 0.2.2 a composite bank in a lz4-per-bank or lz4-per-column file came back looking like an ordinary one and could not be decoded as a composite at all.

0.2.2 fixed that and 0.2.3 keeps the fix, now without breaking the other implementations. (They do not carry the extra table, so a composite bank in a split-codec file is still invisible to them — a gap in the shared format rather than something a reader can work around.)

This cannot be repaired after the fact

If you converted a file containing composite banks with hipoq 0.2.1 or earlier, the marker was never written to disk. Re-reading with a current version does not recover it. Re-convert from the original file.

Ordinary reconstruction output is not affected — neither an 8.5 GB CLAS12 DST nor a simulation file (71 and 106 distinct banks) carries a composite bank.

Two codecs are gone, not renamed

Wire tags 4 (lz4-chunked) and 5 (lz4-by-bank) were removed from the HIPO format. oxihipo does not read them: a file carrying either is rejected at header parse, not read slowly or partially.

This matters because tag 5 was once a default, so files written with it exist. Reading one fails with unknown compression type: 5, raised while parsing the record header.

There is no recovery path inside hipoq. doctor rebuilds a file whose records are damaged or whose trailer is missing, and it gets no further here than any other command: the tag is rejected before a record is decoded at all. Such a file has to be rewritten with a build of HIPO that still understands the tag, then read here.

The old spelling lz4-by-bank-v2 is still accepted, because it was tag 6 — the same codec now called lz4-per-bank. It is kept only as an alias so existing scripts do not break; new ones should use the real name.

hipoq skim run.hipo out.hipo --compress lz4-by-bank-v2   # works: alias of lz4-per-bank
hipoq skim run.hipo out.hipo --compress lz4-by-bank      # error: no such codec