Record

Records are the fundamental I/O unit of HIPO files. They group multiple events into a single compressed block, amortizing LZ4 compression and disk I/O over thousands of events.

hipo::recordHeader_t

Parsed representation of the 56-byte record header.

typedef struct recordHeader_t {
    int signatureString{};              // Magic: 0xc0da0100
    int recordLength{};                 // Total length in 32-bit words
    int recordDataLength{};             // Uncompressed data length
    int recordDataLengthCompressed{};   // Compressed data length
    int numberOfEvents{};               // Events in this record
    int headerLength{};                 // Header length in words (always 14)
    int indexDataLength{};              // Index array length in bytes
    int userHeaderLength{};             // Optional user header length
    int userHeaderLengthPadding{};      // Padding after user header
    int bitInfo{};                      // Packed version + padding
    int compressionType{};              // 0=none, 1=LZ4
    int compressedLengthPadding{};      // Padding after compressed data
    int dataEndianness{};               // 0=little, 1=big
} recordHeader_t;

hipo::record

Handles reading and decompression of records.

Reading

void readRecord(std::ifstream& stream, long position, int dataOffset);
bool loadRecord(record& rec, int index);  // load by record index

The readRecord() pipeline:

1. Read 80 bytes of header from disk
2. Detect endianness via magic number
3. Parse header fields
4. Read compressed payload
5. Decompress with LZ4
6. Convert event index array from sizes to cumulative positions

Event Extraction

void readHipoEvent(std::vector<char>& buffer, int index);
int  getEventCount();

Bulk Column Extraction

Extract an entire column across all events in the record:

std::vector<int>   getColumn(bank& b, const char *columnName);
std::vector<float> getColumnFloat(bank& b, const char *columnName);

Benchmarks

benchmark& getReadBenchmark();     // disk read time
benchmark& getUnzipBenchmark();    // LZ4 decompression time
benchmark& getIndexBenchmark();    // index building time

Header Access

recordHeader_t& getHeader();

hipo::recordbuilder

Constructs records for writing. Accumulates events, then compresses and serializes.

Construction

recordbuilder();
recordbuilder(int maxEvents, int maxSize);

Default limits: 100,000 events or 8 MB.

Adding Events

bool addEvent(std::vector<char>& buffer);

Returns false when the buffer is full -- the caller should build(), write, and reset().

Building

void build();    // compress and finalize
void reset();    // clear for next record

Info

int getRecordSize();   // built record size in bytes
int getEntries();      // number of events added

User Words

int  getUserWordOne();
int  getUserWordTwo();
void setUserWordOne(int val);
void setUserWordTwo(int val);

Internal Buffers

record (reading)

Buffer	Purpose
bufferCompressed	Raw compressed data from disk (grows by 500 KB)
bufferDecompressed	Decompressed record data (grows by 20%)

Both buffers are reused across readRecord() calls (never shrink).

recordbuilder (writing)

Buffer	Purpose
bufferIndex	Event size index (4 bytes per event)
bufferEvents	Raw concatenated event data
bufferData	Combined index + user header + events
bufferRecord	Final record: header + compressed data