Skip to main content

hipoq

A fast command-line toolkit for exploring and manipulating HIPO v6 files — the binary event format used by the Jefferson Lab CLAS12 experiment. It is built on the oxihipo Rust reader/writer.

Where the graphical HIPO browsers are made for clicking through one file, hipoq is made for the shell: it streams, filters, projects, and rewrites HIPO files at batch speed, and it speaks a CLAS12-aware expression language so cuts read the way you think about the physics (REC::Particle.pid == 11 && p() > 1.5).

# how many events have at least two electrons?
hipoq count run.hipo --where "count(REC::Particle.pid == 11) >= 2"

# project momentum and polar angle for the first 100 events
hipoq scan run.hipo --bank REC::Particle --cols pid --expr p --expr theta_deg --events 0..100

# extract just the electron events into a new file
hipoq skim run.hipo electrons.hipo --where "REC::Particle.pid == 11"

What it does​

  • Inspect a file's structure — events, records, schemas, which banks carry data — with info, dict, banks, head.
  • Project & export bank columns as a table, CSV, or JSON with scan, or whole events as NDJSON with dump (the bridge to jq, pandas, DuckDB, …).
  • Summarize with count, stats, hist, including multiplicity cuts and ASCII histograms.
  • Query with a real expression language: &&/||, arithmetic, functions, ranges, membership, cross-bank pindex joins, physics helpers (p(), theta_deg(), e()), and computed columns.
  • Assess quality — occupancy, particle rates, NaN/Inf and physics-sanity flags, truncation and duplicate detection — with quality and check.
  • Compare two files cell-by-cell with diff, or reconstruction against Monte-Carlo truth with mc.
  • Transform — filter, subsample, split, repair, and re-encode — with skim, sample, split, doctor, convert. convert writes its parts in parallel, which is what makes switching a file to a per-column codec (~3x on selective reads) a minute's work rather than an afternoon's.
  • Browse interactively in a terminal UI with tui: paging, bookmarks, jump history, live filtering, and a pindex cross-reference panel.
  • Tune the read — bench sweeps thread counts on your machine and names the -j to use, because the -j 0 default is right on a laptop and wrong on a farm node.

Where to start​

  1. Install it — a prebuilt Linux binary is one curl away; macOS builds from source.
  2. Work through the quick start.
  3. Skim the data model once — especially the pindex link, which is what makes cross-bank cuts work.
  4. Reach for the command reference and the --where language as you go.