Developer Guide

This repository builds container images using Continuous Integration (CI) and publishes them to a Container Registry (CR).

Repository Layout

Directory	Description
`docker/`	`Dockerfile` templates
`context/`	Additional files used for Docker build contexts
`lib/`	Source code for `Forge`, a Ruby library which does most of the hard work
`bin/`	Executable scripts, e.g., for Dockerfile generation; depends on `Forge`
`doc/`	Files for generating this documentation

Building The Images

If you want to modify one of the existing images, or propose a new image, open a Merge Request (MR), which will trigger the CI, build all the images, and push them to the CR.

The images that are built from an MR have a tag starting with MR-, followed by an ID number corresponding to the MR. The CR has a cleanup policy such that these tags will eventually be auto-deleted, so don't rely on them persisting; instead, request your MR to be reviewed and merged.

Image build jobs are also triggered when:

an MR is merged to the main branch, tagging them as latest
a git tag is created for this repository, tagging them with that git tag, so this fixed version of each image is preserved

Neither the latest nor the git tagged versions are auto-deleted; these are the tags that users should use.

The following sections explain more how the image building works.

Image Specification

The images.yaml file is the main file used for specifying the contents of the images. Its top-level keys are the names of each of the images, and their values specify the image contents. For a diagrammatic view of this file, see the Image Dependency Graph.

Here is an example image specification, with comments explaining further details:

example YAML image specification

example_image:  # this is the name of the image

  # the image from which `example_image` is based; if not included,
  # then the Dockerfile template must explicitly specify it
  from: base_image

  # specify the Dockerfile template to use; in this case,
  # `docker/Dockerfile.example_template` (which does not exist)
  template: example_template

  # software to install from the Linux distribution's repository
  upstream:
    - clang
    - jdk-openjdk
    - maven
    - meson
    - python
    - ruby

  # list of software to build and install, all of which are likely
  # CLAS12-specific; the key is the name and the value is the version;
  # Dockerfile templates in `docker/builds/` are used for these
  builds:
    coatjava: 11.1.0
    iguana: 0.8.0

  # if true, copy this YAML file to the generated Dockerfile context
  # (default value is false)
  include_this_yaml: false

Dockerfile Generation

Dockerfile templates, found in docker/, along with contexts from context/, are used to generate full docker build contexts using the script bin/generate_dockerfiles (and source code in lib/). The CI runs this script and stores the generated context directories as an artifact; it then proceeds to build the images and publish them to the CR. You may also run this script locally, if you want to see the generated Dockerfiles, or view them here.

The context/ directory contains files meant to be included in each image's Dockerfile context; for example, a COPY instruction may be used to install a file in an image.

The docker/ directory contains Dockerfile templates:

the top-level files are the templates referred to by the template YAML node in images.yaml
docker/builds contains Dockerfile instructions for building software, such as coatjava
docker/upstream contains Dockerfile instructions for installing packages from the upstream Linux repositories
docker/misc contains miscellaneous instructions

These Dockerfile templates contain strings that are surrounded by @ symbols, such as @image_name@. The generator script bin/generate_dockerfiles will replace these strings, each of which has a certain meaning / usage:

String	Effect
`@base_name@`	Replaced by the image name specified in `from:` in the YAML image specification
`@dump_runner_specs@`	Replaced by a `RUN` instruction to dump the hardware specifications of the CI runner, along with the constrained specifications in the CI job.
`@git_clone@`	Replaced by a `git clone` command, using the version specified in the YAML image specification; note the version can be a branch or tag
`@image_name@`	Replaced by the image name in the YAML image specification
`@install_builds@`	Replaced with the build instructions from `docker/builds/`, for each software package listed in the YAML image specification under `builds:`.
`@install_upstream_pacman_packages@`	Replaced with a `pacman` command to install all of the packages listed in the YAML image specification under `upstream:`; if no packages are listed, this line is simply deleted.
`@upstream_packages@`	Replaced by a list of packages listed in the YAML image specification under `upstream`; this is used by other substitutions, such as `@install_upstream_pacman_packages@`.
`@version@`	Replaced by the package version

How to add a new software package

Add package build instructions in a file within docker/builds/
Add the package to one or more images in images.yaml

Note

Avoid using WORKDIR in package build instructions, since that can cause a change in the current working directory for subsequent package build instructions; instead use cd in a RUN instruction, so that directory changes are limited to that RUN, and therefore have no impact on subsequent RUN instructions.

Documentation Generation

This documentation is also generated by the CI, via bin/generate_documentation, and deployed as a GitLab Page. If you open an MR, you can check the documentation generation artifacts to preview the documentation webpage associated to that MR.

To run documentation generation locally, first you'll need some dependencies; call these from the top-level repository directory:

gem install bundler  # install Ruby's `bundler` (if you don't have it)
bundle install       # install Ruby gems
python -m pip install -r doc/requirements.txt   # install Python dependencies

Then generate dockerfiles and documentation build files, outputting to a custom directory:

bin/generate_dockerfiles [DOCKERFILE_DIR]
bin/generate_documentation [BUILD_DIR] [DOCKERFILE_DIR]

Then run the suggested mkdocs command, and serve it:

mkdocs build --config-file [BUILD_DIR]/mkdocs.yaml
mkdocs serve --config-file [BUILD_DIR]/mkdocs.yaml

Interactive Ruby Shell

For convenience, you may use bin/console for an interactive Ruby shell with the Forge library loaded.