User Guide

Tip

If any terminology in this guide is unfamiliar to you, see the glossary.

Available Container Images

The following info-boxes contain information for each available image; this first box describes the images' info-box contents:

Image Name

Description	Describes the purpose of this image
Included Software	Lists included software. This excludes Linux package manager packages; for that, see: Image Specification Image Dependency Graph
URI	This is the URI for the latest tag of the image; for example, use it as apptainer pull [URI]
Links	Link to the Registry, which lists all other tags for this image; note that you must prefix any URI with docker:// Link to the Dockerfile that was used to build this image

base

Description	base image with software commonly used by all images
Included Software	ccdb, cmake, hipo, qadb, rcdb
URI	docker://codecr.jlab.org/hallb/clas12/container-forge/base:latest
Links	Registry Dockerfile

base_root

Description	same as base image, but with ROOT
Included Software	ccdb, cmake, hipo, qadb, rcdb, root
URI	docker://codecr.jlab.org/hallb/clas12/container-forge/base_root:latest
Links	Registry Dockerfile

recon

Description	software for reconstruction
Included Software	asprof, ccdb, clara, clas12-config, cmake, coatjava, denoise, hipo, qadb, rcdb
URI	docker://codecr.jlab.org/hallb/clas12/container-forge/recon:latest
Links	Registry Dockerfile

analysis

Description	software for analysis
Included Software	ccdb, clas12root, cmake, hipo, iguana, qadb, rcdb, root
URI	docker://codecr.jlab.org/hallb/clas12/container-forge/analysis:latest
Links	Registry Dockerfile

Image Tags

We use the following convention for image tag names:

• the latest tag is the latest version of the image, and may sometimes be unstable
• tags which start with v are versioned images, and are supposed to be stable, but their software is older than latest
• tags which start with MR are temporary images, built by developers of this repository; these images are auto-deleted, so do not use them unless you have a reason to

Container Information

Software that we build and install for CLAS12 may be found in a few locations inside a running container:

File Path	Description
/usr/local	Common installation location for software that produces directories such as bin, include, lib
/opt	All other software packages are installed in /opt/<package>

The environment is configured to point to these installations; run env in a container to see.

Use container_info to list the installed software packages and their versions:

container_info          # list all packages
container_info --name   # get the name of the image for this container
container_info --help   # more usage guidance

How to Use Containers

Apptainer is a program that can download images and run containers. See the sections below for an introduction, and refer to Apptainer documentation for more.

Alternatively, you may use Docker or Podman, but we have not yet written documentation here for these.

Apptainer Setup for ifarm

If you are on ifarm, you should adjust some of Apptainer's default directories, to avoid consuming available disk space in certain directories, such as your home directory; you may do so with environment variables:

Environment Variable	Description	Default
APPTAINER_CACHEDIR	image cache	$HOME/.apptainer/cache
APPTAINER_TMPDIR	temporary folder for Apptainer	/tmp

Unfortunately, we do not yet have a good choice for what these variables should be set to on ifarm, but SciComp is working on it. For now, consider using either /volatile:

export APPTAINER_CACHEDIR=/volatile/clas12/users/$LOGNAME/apptainer/cache
export APPTAINER_TMPDIR=/volatile/clas12/users/$LOGNAME/apptainer/tmp

or /scratch (which differs for each computing node, e.g., ifarm2401 or ifarm2402):

export APPTAINER_CACHEDIR=/scratch/$LOGNAME/apptainer/cache
export APPTAINER_TMPDIR=/scratch/$LOGNAME/apptainer/tmp

You should also make sure these directories exist:

mkdir -p $APPTAINER_CACHEDIR $APPTAINER_TMPDIR

Warning

Both /volatile and /scratch auto-remove files after some time, which can cause weird issues, so you may need to wipe these directories if you need to pull an image again; this issue is the reason why these directories are not ideal for this use case.

Good News: Once you have pulled an image and created a .sif file, you do not need either of these directories, since the entire image will be in a single file, which you may store anywhere, e.g, on /work.

Note

You may also need to customize these directories on your personal computer, depending on how your disk is partitioned (e.g., /tmp may be too small).

Pulling an Image

To download an image, use apptainer pull, which will download ("pull") an image and store it in a large (~few GB) .sif file, from which you may then run a container.

For example, let's pull the analysis image, using its latest URI given in the table above:

apptainer pull docker://codecr.jlab.org/hallb/clas12/container-forge/analysis:latest

This will create the file analysis_latest.sif in your current directory, and you may move it wherever you want. All of the software in this image is included within this file.

Running a Container

Running a container may be done with apptainer run:

apptainer run [IMAGE]

where [IMAGE] is the .sif file you have downloaded; alternatively, you can use a URI, which will pull the image and run a container.

Doing this will open a shell in the container for interactive use.

If you want to just use the container to run a program within, pass additional arguments; for example, to run qadb-info (which is found in any container image which has the QADB installed), run:

apptainer run [IMAGE] qadb-info --help

In other words, with apptainer run, you may either:

1. start an interactive shell inside the container
2. run container commands from your host computer's shell

Note

If you are submitting ifarm jobs, you'll need to use (2), since the computing nodes are the host computers. Alternatively, use apptainer exec.

Tip

There are two other useful commands to run containers:

apptainer shell [IMAGE]              # run an interactive shell in a container
apptainer exec [IMAGE] [COMMAND]...  # run a command within a container

For further documentation on any apptainer command, use the --help option, or man:

man apptainer-run
man apptainer-shell
man apptainer-exec

Here are some useful options for apptainer run (and for apptainer shell and apptainer exec):

Option	Effect
--bind	be able to access a host filesystem path from within the container, such as /cache
--cleanenv	clean the environment before running container
--fakeroot	be root user inside the container
--writable-tmpfs	makes the container's filesystem (from /) writeable, but do not persist the changes

Example 1

Run a container interactively, and have access to files on /cache:

apptainer run --bind /cache [IMAGE]

Example 2

If you want to test installing additional packages using pacman (the container's Linux distribution's package manager):

### start a container
apptainer run --cleanenv --fakeroot --writable-tmpfs [IMAGE]
### inside container shell
pacman -Syu         # update packages
pacman -S crystal   # install support for the Crysal programming language

Once you leave the container, your changes will be destroyed, however.

Tip

If there is some common software you would like available in the image(s), ask the maintainers or open a merge request.

Customizing the Interactive Container Shell

When using apptainer run, the default shell for interactive use is bash, for which we apply the following settings:

bashrc

# aliases
alias ls='ls --color=auto'
alias grep='grep --color=auto'

If you want to use an interactive shell without these settings applied:

apptainer shell [IMAGE]

Note

Different shells are available, such as tcsh, zsh, and fish, which you may either run within an interactive container shell, or start with preferred shell using exec:

apptainer exec [IMAGE] zsh

Danger

Be careful, since by default, these shells may use configuration files from your host home directory, potentially overwriting certain environment variables set in the container image.

See below for more details and how to work around this.

If you prefer your own settings, you may either:

1. Write your own configuration (rc) file, and use that; for example, if its name is my_rc_file:

   apptainer exec [IMAGE] bash --rcfile my_rc_file
   

2. Use your host computer's shell settings, which makes the container interactive shell feel just like your host computer's shell; just pass your shell name, e.g., bash:

   apptainer exec [IMAGE] bash
   

   Danger

   This can be very convenient, but you need to be careful: if your host shell's configuration sets environment variables, such as common paths like $LD_LIBRARY_PATH or software-specific variables like $HIPO, doing this will modify the values of these variables in the container using the values from the host.

   Example

   $HIPO in a container points to the image's build of hipo, but if you set $HIPO in your shell's config file to point to your host computer's hipo build, the container will use your host's version of hipo.

   One way to work around this is to start a shell without sourcing configuration:

   bash --norc
   tcsh -f
   zsh --no-rcs
   fish --no-config
   

   So for example, to start zsh:

   apptainer exec [IMAGE] zsh --no-rcs
   

   This approach does not use your shell's custom settings, however; if you really want to use your custom settings, you'll need to go through your configuration files and make use of the environment variable $APPTAINER_CONTAINER, which should only be set to a non-empty value if you are in a container shell. For example, in your .bashrc:

   if [ -z "$APPTAINER_CONTAINER" ]; then
     export HIPO=/path/to/host/hipo
   fi
   

Tip

The default prompt in a container is Apptainer>. You can change it by setting $APPTAINERENV_PS1 on the host computer. For example:

export APPTAINERENV_PS1="$(tput bold)<<<<<<<< \w\n:        [\D{%a} \t] \u@\H\n>>> $(tput sgr0)"

Then,

apptainer shell [IMAGE]

has the following prompt:

<<<<<<<< /path/to/current/directory
:        [Fri 19:15:38] user_name@host_name
>>>

which is designed to be easier to see the separation between subsequent command outputs.