Programmatic access

hipoq is a CLI, but its output formats are designed to feed other tools — shell pipelines, dataframes, SQL, and batch schedulers. This guide collects the patterns.

The two JSON shapes

There are two ways to get JSON out, and they nest differently:

Command	Shape	Use when
scan --format ndjson	one object per bank row (flat)	you want a single bank as a flat table
dump	one object per event (banks nested)	you want whole events, all banks at once

Both are NDJSON (one JSON object per line), which streams and is understood by jq, pandas, DuckDB, and most log tooling.

Piping

Three cases, and they behave differently.

Into other tools — the normal one

hipoq scan rec.hipo --bank REC::Particle --format ndjson | jq .
hipoq dump rec.hipo --events 0..10 | head -3

stdout carries only data. The data formats emit nothing else at all — a two-row --format ndjson writes two lines to stdout and zero to stderr. Where a command does print a summary, it is on stderr: --format table puts its 2 rows there, and count --list its 100 / 100 events matched. So a pipe stays machine-readable without 2>/dev/null; redirect stderr only to keep the summaries off your terminal.

Closing the pipe early is safe. hipoq restores the default SIGPIPE disposition at startup, so | head kills it quietly rather than making it print a broken-pipe error over your output — see the exit-code note for the one place that matters.

hipoq to hipoq on a file — not possible

$ hipoq count - < rec.hipo
Error: io error: no such file or directory: -

No command reads a HIPO file from stdin, and that is the format rather than an omission: HIPO v6 keeps its record index in a trailer at the end of the file, and every command seeks there before reading anything. A stream has no end to seek to. Use a file between the two stages:

hipoq skim rec.hipo /tmp/sel.hipo --where "REC::Particle.pid == 11"
hipoq scan /tmp/sel.hipo --bank REC::Particle --format csv

hipoq to hipoq through an index list — the one that works

The payload here is text, so it pipes:

hipoq count rec.hipo --where "REC::Particle.pid == 11" --list \
  | hipoq skim rec.hipo electrons.hipo --events-from -

count --list prints one global event index per line and skim --events-from reads them, with - meaning stdin. Finding the events is the expensive pass; this is how you spend it once. Both sides use global indices — the space dump --events and tail address — so a list produced under one --require and consumed under another would name different events.

set -o pipefail and exit 141

pipeline	exit
hipoq dump big.hipo | head -3	0
the same under set -o pipefail	141

141 is 128 + 13: the shell reporting that hipoq was killed by SIGPIPE when head stopped reading. That is correct Unix behaviour and the reason output does not get a broken-pipe error splattered through it — but under pipefail, which CI scripts and Makefiles commonly set, it fails the pipeline. Guard it where you deliberately truncate:

set -o pipefail
{ hipoq dump big.hipo || [ $? -eq 141 ]; } | head -3   # exits 0

Nothing is needed when the consumer reads to the end — jq, pandas, duckdb, awk, a file redirect. Only early-exit consumers such as head, grep -m, or quitting a pager are affected.

With jq

Flat rows from scan are the easiest to slice:

# electron transverse momenta, one number per line
hipoq scan rec.hipo --bank REC::Particle --cols px,py --where "REC::Particle.pid==11" --format ndjson \
  | jq '.px, .py'

# whole events, then pull the particle array
hipoq dump rec.hipo --events 0..10 | jq '.["REC::Particle"]'

Into pandas

Flat, one-row-per-particle — the shape most analyses want — comes from scan:

hipoq scan rec.hipo --bank REC::Particle --cols pid,px,py,pz \
  --expr p --expr theta_deg --format ndjson > particles.ndjson

import pandas as pd
df = pd.read_json("particles.ndjson", lines=True)
df[df.pid == 11]["p"].hist()

Whole events come from dump; each bank is a list of row-dicts you can normalize:

import pandas as pd
events = pd.read_json("events.ndjson", lines=True)   # from: hipoq dump rec.hipo > events.ndjson
particles = pd.json_normalize(
    events.to_dict("records"),
    record_path="REC::Particle",
    meta="event",
)

Into DuckDB

DuckDB reads NDJSON directly and gives you SQL over it:

hipoq scan rec.hipo --bank REC::Particle --cols pid,px,py,pz --format ndjson > particles.ndjson

-- in the duckdb shell
SELECT pid, count(*), avg(px)
FROM read_json_auto('particles.ndjson')
GROUP BY pid ORDER BY 2 DESC;

CSV for spreadsheets and ROOT

hipoq scan rec.hipo --bank REC::Particle --cols pid,px,py,pz --format csv > particles.csv

csv is the format for spreadsheets, TTree::ReadStream, numpy.genfromtxt, and anything else that speaks comma-separated values.

Parallel batch jobs

For embarrassingly-parallel processing, split a file into per-worker chunks, process them independently, and merge the results back with skim:

# 1. fan out into 16 contiguous chunks
hipoq split run.hipo chunks/ --chunks 16

# 2. process each chunk in parallel (your analysis here)
ls chunks/*.hipo | xargs -P 16 -I{} my_analysis {}

# 3. (optionally) merge selected events back into one file
hipoq skim "chunks/*.hipo" merged.hipo --where "REC::Particle.pid == 11"

Because split chunks are contiguous and events are copied verbatim, the pieces reassemble losslessly — nothing is dropped or reordered.

Quick, reproducible test files

When iterating on analysis code, work against a tiny deterministic subsample so runs are fast and repeatable:

hipoq sample run.hipo dev.hipo --fraction 0.001 --seed 42

The same --seed always selects the same events, so a bug you hit on dev.hipo reproduces every time.

Want a native library or dataframe export?

Today the programmatic surface is the CLI plus these text/JSON formats. A typed columnar export (Parquet/Arrow) or language bindings would be the next step if the JSON path becomes a bottleneck — see the project's roadmap.