Skip to main content

Container Storage for Shared Environments

Running containers in shared HPC/HTC environments like Jefferson Lab requires careful configuration of storage settings to avoid disk quota issues and ensure compatibility with user-specific paths. This document explains the graphRoot and runRoot paths set up in storage.conf.

What is storage.conf?​

The STORAGE configuration file specifies all of the available container storage options for tools using shared container storage. A TOML format is used so the specification can be easily formatted and versioned. The format and specifications are available in the containers/storage upstream of podman.

ℹ️ The default /etc/containers/storage.conf file is configured to work as-is for rootless users. You do not need to create your own storage.conf file. Adding your own storage.conf file is an advanced use-case when you are a part of multiple groups, or perform development on many projects.

GraphRoot​

All of the associated container storage is stored in a graph. This includes the container overlays, volumes, and the SQL database used by podman to manage the container engine. This location stores alot of data like container images. By default, on JLab farm / ifarm nodes, graphRoot is set to /scratch/$USER/.local/share/containers/storage directory.

/scratch/$USER/.local/share/containers/storage/
├── db.sql
├── defaultNetworkBackend
├── libpod
├── networks
├── overlay
├── overlay-containers
├── overlay-images
├── overlay-layers
├── secrets
├── storage.lock
├── userns.lock
└── volumes
  • overlay/ – The union filesystem directories used for each container
  • overlay-containers/ – Metadata per container instance
  • overlay-images/ – Metadata for container images
  • volumes/ – Named volumes created via podman volume create
  • tmp/, libpod/, mounts/ – Additional state used by the container engine

In HPC/HTC environments, the container engine and container runtime should be viewed as an ephemeral tool used to run your analysis. While many parts of /scratch are automatically cleared with a strict deletion policy, the /scratch/$USER/.local/share/containers directory is explicitly excluded from these deletion policies.

:information_source : Please be mindful of the data you store in this area, and clean it up regularly.

  • Remove unused containers from podman container ls -a
  • Save tagged container images in codecr.jlab.org. Remove older unused tagged images.
  • Opt to store data in host level locations. See the volumes section for more details

⚠️ Users should NOT configure the GraphRoot to be on the /u/home/$USER directory. This is the quickest way to fill up your CUE_home directory.

RunRoot​

This is the default directory podman uses to store all temporary writable content created by container storage engines. When a container is run, anything that is needed to support the running container is saved in runroot. All the volumes and images are saved to the graphRoot while volatile supporting tasks are started in runRoot. The ideal location for storing volatile /run data is in /run/user/$UID/containers. This is the default location for rootless podman. You can run podman info to show your graphRoot and runRoot in your environment.

podman info --format 'Graph Root: {{.Store.GraphRoot}}, RunRoot: {{.Store.RunRoot}}'

ℹ️ If your graphRoot is set to /scratch/$USER/.local/share/containers/storage, and runRoot is /run/user/$UID/containers you should be set up to use podman.

Use the default Configuration​

By default, Podman uses /etc/containers/storage.conf unless a $HOME/containers/storage.conf exists. Both of these paths are overridden if $XDG_CONFIG_HOME is set. In the default /etc/containers/storage.conf the root runRoot is set to /run/containers/storage. This is not the case for rootless environments. It is recommended to use the default environment for user level configurations. If you must set a storage.conf, make sure to do the following:

  • Create the default config and store it in $HOME/containers/storage.conf
  • Set the run dirtectory /run/user/$UID/containers for the specific user id.
  • Ensure the rootless_container_storage is set to a large disk space.

⚠️ Do not set the rootless_container_storage to any small file system such as $HOME. This can fill up quickly.

This file overrides system-wide defaults and lets you define where container images, layers, and runtime data should go. This is not typically needed unless you participate in many groups simultaneously and regularly build new containers.

Why is this Needed for Shared Systems?​

On JLab interactive systems (ifarm/farm), shared home directories (/u/home/$USER) have limited space and are not meant for container layers. The /scratch filesystem is intended for larger data loads and is faster and more robust for ephemeral container use.

  • ✅ Avoids filling up user home directory
  • ✅ Prevents permission issues between batch and interactive jobs
  • ✅ Keeps container state isolated and reproducible

Final Notes​

It is important to understand the data that resides under graphRoot and runRoot. Users should make sure any images are pushed to a container registry and that any important data is saved on host-level disks or backed up to the mass storage system. If user-level configurations are needed, check the specification of the storage.conf file on containers github.

  • Run podman info to verify your configuration.
  • This setup is optimized for JLab infrastructure but also works on other shared scientific clusters.