NPS REPLAY with apptainer

To run NPS replay we have a NPSlib container. NPSlib is a extention on hcana for NPS experiment. Github link here

The container for NPSlib are at dockerHUB : https://hub.docker.com/r/jeffersonlab/hallc-npslib/tags

Building the NPSlib image

$ apptainer build nps.sif docker://jeffersonlab/hallc-npslib:v0.1.0

Here we are using the v0.1.0 version of the NPSlib which is also version of container. I have build my image with the name at “nps.sif

Lets write a bash script to run on ifarm. Filename for this eaxmple is run_nps_replay_apptainer.sh . To make it work with you, please change the paths in “modify as you need” part of code. Following asummes that your replay takes the env var argument as for raw data , rootfile and report output as shwown in here

#!/usr/bin/bash

spec="nps"
SPEC="NPS"

# Two input args
runNum=$1    # RUN Number
nEvent=$2    # Number of events in that run

# Modify as you need
#--------------------------------------------------
SCRDIR="/path/to/nps_replay"                  # my replay directory
SCRIPT="SCRIPTS/${SPEC}/eel108_replay.C"      # replay script to run, relative to nps_replay directory
APPTAINER_IMAGE="/path/to/apptainer/nps.sif"  # apptainer image location
# for the following make sure your replay script take following as env varibales.
# e.g. in https://github.com/JeffersonLab/nps_replay/pull/37/files

DATA_DIR=/cache/hallc/c-nps/raw               # raw data directory.
LOG_DIR=/path/to/REPORT_OUTPUT/dir        # REPORT output directory
OUT_DIR=/path/to/output/root/dir              # OUTPUT rootfiles
#--------------------------------------------------

# making sure the directory exists.
mkdir -f ${OUT_DIR}
mkdir -f ${LOG_DIR}

cd $SCRDIR

# Check if apptainer is available
if command -v apptainer > /dev/null 2>&1; then
    echo "apptainer is already available."
else
    # Load apptainer if not available
    echo "Loading apptainer..."
    eval module load apptainer
fi
# Set the output filename
output_file="output.env"

# Write each environment variable name and value to the output file
echo "DATA_DIR=${DATA_DIR}" >> "${output_file}"
echo "LOG_DIR=${LOG_DIR}" >> "${output_file}"
echo "OUT_DIR=${OUT_DIR}" >> "${output_file}"

echo "Environment variables written to ${output_file}"

echo
echo "---------------------------------------------------------------------------------------------"
echo "NPS SINGLE REPLAY for ${runNum}. NEvent=${nEvent} using NPSlib container=${APPTAINER_IMAGE}"
echo "----------------------------------------------------------------------------------------------"
echo

runStr="apptainer exec  --env-file output.env --bind ${DATA_DIR} --bind ${OUT_DIR} --bind ${LOG_DIR} --bind ${SCRDIR}  ${APPTAINER_IMAGE} bash -c \"hcana -q ${SCRIPT}\(${runNum},${nEvent}\)\""
eval ${runStr}

if your replay doesn’t take env variable for raw data, output rootfile and report . Make sure you have corrresponding link directory in the replay directory. Use following as run_nps_replay_apptainer.sh for link directory structure in replay:

#!/usr/bin/bash

spec="nps"
SPEC="NPS"

# Two input args
runNum=$1    # RUN Number
nEvent=$2    # Number of events in that run

# Modify as you need
#--------------------------------------------------
SCRDIR="/path/to/nps_replay"                  # my replay directory
SCRIPT="SCRIPTS/${SPEC}/eel108_replay.C"      # replay script to run, relative to nps_replay directory
APPTAINER_IMAGE="/path/to/apptainer/nps.sif"  # apptainer image location
DATA_DIR=/path/to/RAWDIR
OUT_DIR=/path/to/output/root/dir
LOG_DIR=/path/to/REPORT_OUTPUT/dir
#--------------------------------------------------

mkdir -f ${OUT_DIR}
mkdir -f ${LOG_DIR}

cd $SCRDIR

# Check if apptainer is available
if command -v apptainer > /dev/null 2>&1; then
    echo "apptainer is already available."
else
    # Load apptainer if not available
    echo "Loading apptainer..."
    eval module load apptainer
fi

echo
echo "---------------------------------------------------------------------------------------------"
echo "NPS SINGLE REPLAY for ${runNum}. NEvent=${nEvent} using NPSlib container=${APPTAINER_IMAGE}"
echo "----------------------------------------------------------------------------------------------"
echo

runStr="apptainer exec --bind ${DATA_DIR} --bind ${OUT_DIR} --bind ${LOG_DIR} --bind ${SCRDIR}  ${APPTAINER_IMAGE} bash -c \"hcana -q ${SCRIPT}\(${runNum},${nEvent}\)\""
eval ${runStr}

Make the file executables as:

$ chmod +x run_nps_replay_apptainer.sh

Now in ifarm you can just to following to run a single jobs. Add corresponding run_num and num_events that you want to analyze.

$ bash run_nps_replay_apptainer.sh <run_num> <num_events>

Debug using interactive shell

To debug anything or execute command manually inside the image you envoke inetractive shell using the following.

$ apptainer shell --env-file output.env --bind ${DATA_DIR} --bind ${OUT_DIR} --bind ${LOG_DIR} --bind ${SCRDIR}  ${APPTAINER_IMAGE} bash

swif2 jobs

To use the above script with swif jobs. Please replace the DATA_DIR path to following.

DATA_DIR=${PWD}

How to use swif is explained in this . Input file is specified in swif as :

-input <filename> mss:/mss/hallc/c-nps/raw/<filename>

Here , replace <filename> with actual filename. For example nps_345.dat.0. This will then will automatically show up in the working directory. On the script make sure to have DATA_DIR specifies as $PWD before moving into any other direrectory. Example swif command line to submit a jobs is:

$ swif2 add-job myworkflow -account hallc -partition production -shell /bin/bash -input nps_345.dat.0 mss:/mss/hallc/c-nps/raw/nps_345.dat.0 -cores 1 /path/to/run_nps_replay_apptainer.sh 345 100

Your std out and err of the job will writeen to /farm_out/$USER/<foo>.{out,err} by default.

hcswif

Comming soon