Skip to main content

Frequently Asked Questions

Common Problems Encountered by New Users​

Containers are powerful tools for reproducibility and portability, but new users often encounter subtle issues due to how containers isolate environments and interact with the host. Below is a curated list of common pitfalls and their explanations, tailored for JLab users but broadly applicable to scientific workflows.

I’m root in the container — why can’t I access everything?​

Answer: Even though you're root inside the container, you're not root on the host. Most systems at JLab use rootless Podman, where your container runs with your actual user permissions. This means host protections like file permissions, SELinux labels, and kernel namespaces still apply. You do not get full host access.

I am able to mount a directory, but I am getting permission denied errors. I can access the directories on the host machine, why can't I access the directory in the container?​

Answer: This often happens when:

  • The host path doesn’t exist or isn’t accessible.
  • You're using a host with SELinux, which may require the --security-opt label=disable option.
  • You forgot to use --userns=keep-id, which preserves file ownership.

Try:

podman run --rm -it --userns=keep-id --security-opt label=disable -v /my/data:/data jlab/base:alma9.5

When I mount /work, I cannot access my experiments full path.​

Answer: The /work directory is composed of several different locations symlinked together. /work itself is a symlink to /w/work. The separation of ZFS pools exposed as NFS allows for pools to be expanded and shrunk by administrators, and is common in HPC environments. To mount your /work directory it is best practice to mount the full path instead of all of /work.

-v /work/path/to/your/area:/my-work

ℹ️ If you must mount several areas, you resolve the paths by mounting both /work and /w. This is not recommended because it is too broad of a scope. Please use a specific area under your personal experiment and work directory.

I tried to use module load inside my container but got "command not found".​

Answer: The module command is a shell function loaded via environment modules. Vendor-supplied containers do not have the environment modules package installed. Check whether modules.sh exists in /etc/profile.d, and source it if available:

source /etc/profile.d/modules.sh
which module

My container ran fine interactively, but fails in batch mode (Slurm).​

Answer: In batch environments:

  • Environment variables might not be set the same as in an interactive shell.
  • Volumes may be missing if you didn’t include all the needed mounts.
  • You may be missing --userns=keep-id which leads to file ownership issues.

My data disappeared when I stopped the container. Where did it go?​

Answer: Containers are ephemeral. Any data not saved to a mounted volume will be lost when the container is removed.

Solution: Mount data directories explicitly:

-v /volatile/myjob:/data

ℹ️ You may be able to find the data in a container if it was not removed after execution:

docker container ls -a # Find and start the container used to run analysis

I built a custom image, but others can't use it. Why?​

Answer: Images must be pushed to a registry to be shared. Use:

podman login codecr.jlab.org
podman push codecr.jlab.org/myimage:latest

ℹ️ Make sure collaborators also mount any needed directories (e.g., /cvmfs, /work, /volatile).

I’m using docker, but it behaves differently than expected.​

Answer: At JLab, docker is aliased to podman. Most commands work the same, but there are differences:

  • Use podman --version to confirm your environment. Look up differences between podman and docker for your version if unexpected behavior occurs.

Why is my container image huge?​

Answer: Layered images can grow quickly. Common causes:

  • Installing large packages without cleaning up.
  • Copying source code or data directly into the image instead of mounting. Tips:
  • Use multistage builds.
  • Group RUN commands to reduce intermediate layers.
  • Mount input/output data at runtime instead of baking it in.

My files show up with a username of nobody and a group of nogroup! What is going on?​

Answer: Nothing is wrong, this is common when groups and users are distributed in HPC clusters from an LDAP source. Instead of users and groups being distirbuted under /etc/passwd or /etc/group, they are determined by sssd. The files you create on disk will have the username and group of the user that started the container. Always prefer to use the --userns=keep-id and avoid using root in the container for unexpected behavior.

Can I mount my full home directory to bring in config files?​

Answer: Not recommended. Your .bashrc, .cshrc, or .profile may conflict with the container's intended environment.

Better approach: Write a minimal .bashrc specific to the container and mount it explicitly if needed.

Why do I see strange path errors using CVMFS?​

General Tips​

  • Always test containers interactively before running them in Slurm.
  • Separate code, data, and configuration — mount only what you need.
  • Push your custom containers to codecr.jlab.org for reuse across systems.
  • Never assume a path exists in the container — bind it explicitly.
  • Prefer --mount over -v for clarity and reproducibility in scripts.
--mount type=bind,source=/work,target=/work