Output formats

Two commands take --format: scan and dump. Other commands render a table and have no choice.

Shape and encoding are two different things

--format picks the encoding. The shape — what one record is — comes from the command, and this catches people out because the same --format word means a different top-level structure in each:

	scan — the row shape	dump — the event shape
one record is	one bank row, flat	one event, with its banks nested
table	one table; several banks are stacked into it with a union of their columns, blank where a bank lacks one	one table per bank per event, each with its own columns, nothing blank
ndjson	{"event":0,"row":0,"pid":11,…}	{"event":0,"REC::Particle":[…],…}
json	one array of row objects	one array of event objects
csv	one header over the rows	refused — an event has no single header

Read it as: the verb chooses the shape, the flag chooses the encoding. If you want a flat column across many events, that is scan whatever the encoding; if you want everything about an event, that is dump.

The encodings themselves, independent of shape:

Format	Encoding	Best for
table	aligned columns, human-readable	reading in the terminal
csv	comma-separated, one header row	spreadsheets, ROOT, numpy
ndjson	one JSON object per line	jq, pandas, DuckDB, streaming
json	a single JSON array	tools that want one document

csv is the one combination that does not exist: it is a single header over uniform rows, and an event is several banks with different row counts and different columns. dump --format csv says so and points at scan.

hipoq scan rec.hipo --bank REC::Particle --cols pid,px,py,pz --format table   # default
hipoq scan rec.hipo --bank REC::Particle --cols pid,px,py,pz --format csv
hipoq scan rec.hipo --bank REC::Particle --cols pid,px,py,pz --format ndjson | jq .
hipoq scan rec.hipo --bank REC::Particle --cols pid,px,py,pz --format json

ndjson vs json

• ndjson (newline-delimited) emits one object per line. It streams — downstream tools can process rows as they arrive — and is the right choice for large outputs and pipelines.
• json emits one array containing every row. It's a single valid document, convenient for tools that JSON.parse the whole thing, but it must be buffered.

Prefer ndjson unless a consumer specifically needs a single array.

Array columns

A column that is a fixed-length vector per row (e.g. a covariance cov) comes out as an array in every format:

• in ndjson / json it's a real nested JSON array — "cov": [0.0, 0.001, …];
• in csv / table it's rendered as a bracketed list in the cell.

To reduce one to a number — for a cut, a histogram or a moment — address a single element with [N]: see --where and stats/hist.

Numbers stay numbers

Data formats (csv, ndjson, json) are kept numeric on purpose:

• pid is not decoded to a symbol even with global --pdg — that decoding happens only in the human table (and in head / tui). This keeps piped output clean for parsing.
• Floating-point values use a shortest round-trip representation, so a stored 0.1_f32 prints as 0.1, not 0.10000000149….
• Non-finite values (NaN, Inf) — which JSON cannot represent — are emitted as null.

Cost of each format

Measured on 465,286 rows of a wide raw bank from CLAS12 run 22083 (scan --bank AHDC::adc --limit 2000000), best of three:

format	time	throughput	peak memory	output
csv	0.54 s	57 MB/s	85 MB	31 MB
ndjson	0.58 s	183 MB/s	83 MB	106 MB
json	0.82 s	184 MB/s	85 MB	151 MB
table	1.54 s	—	601 MB	327 MB

Two things worth reading off that table.

csv is the fastest per row, not the slowest. Its lower MB/s is compactness — it repeats no keys, so the same rows are 31 MB against ndjson's 106 MB. Per row it is 1.16 µs against ndjson's 1.25.

table is the one format whose memory grows with the export. The data formats stream: a row is written and dropped, so peak memory is flat no matter how many rows you ask for. A table cannot, because a column is as wide as its widest cell — the last row can change the first line — so every row is held until the end. Above 100,000 rows scan says so on stderr and names the cheaper formats.

For scale, dump writing every bank of an event as NDJSON runs at 247 MB/s, or 660 µs per event on that run — a full run file is about 6.6 minutes and roughly 95 GB of JSON.

These are not the numbers to design around

They are one bank on one machine with a warm page cache. Use them for the relative picture — which format to pick, and which one will not fit in your memory limit — rather than as a throughput budget.