Chef

This page documents standard tools for all CLAS12 chefs for data processing, including decoding, reconstruction, and trains. These tools leverage supported JLab Scicomp utilities, such as SWIF workflows and the SWIF and SLURM databases, for controlling and monitoring large groups of heterogenous jobs.

The goal is standardizing and optimizing the way run group chefs deal with their jobs on the JLab batch farm. This includes avoiding the need for writing any specialized scripts, e.g. generating file lists or submission files, job bookkeeping or monitoring and cleanup tools, as well as minimizing room for human error and creating a consistent naming scheme for file and sub-directory structures.

This system also includes rigorous error checking and corresponding exit code reporting within the jobs. Integrity checks are performed on every input and output file, before releasing outputs to their final destination.

Environment Setup¶

To get access to these tools, use the usual environment [[CLAS12 Software Environment @ JLab|modules for CLAS12 software]], and then load the workflow module, e.g.:

  module use /scigroup/cvmfs/hallb/clas12/sw/modulefiles
  module load workflow

That will put a set of frontend scripts, discussed and used on the other tabs on this wiki page, into your $PATH for easy access. ''Note, all commands have the -h option to provide full usage information, and also give feedback on improper usage whenever possible.''

Workflows¶

Overview¶

All jobs are grouped into SWIF "workflows". Our user interface to workflow generation is the command:

clas12-workflow

Its command-line options are currently not modular, i.e. not all options are required or honored for all types of workflows, but it provides clear and consistent feedback on what options are missing such that a workflow can be built interactively and progressively.

Checking the validity of user arguments is done whenever and as early as possible, with feedback to the user upon issues.

Its output is a JSON file in the current working directory, which contains the full description of the SWIF workflow. Note, no jobs will be submitted unless the --submit option is used, or the JSON file is manually uploaded to SWIF, otherwise one can freely play around with zero risk.

A configuration file option is available for commonly repeated arguments, but initial use should start with only command-line arguments to interactively build a workflow with minimal options. Be weary of configurations inherited from others unless you understand all their parameters.

Key Parameters¶

These are the key configuration parameters that should be understood first:

• --model

This parameter controls which types of jobs will be included in the workflow. These components are decoding (dec) and merging (mrg), reconstruction (rec), trains (ana), histogramming for quality timelines (qtl), and each are implemented in their own, separate jobs. Some are single-threaded, others are multi-threaded to various degrees, and we leverage SWIF to coordinate these heterogeneous jobs into a workflow. Multi-component workflows utilize SWIF antecedents to ensure that a given job is not submitted until its dependencies are satisfied, e.g. a single train job runs on many reconstructed files and will not be submitted until those reconstruction jobs are succesfull.

The primary point of merging decoded files is to produce file sizes appropriate for few-hour, single-file reconstruction jobs, and avoid tiny files on tape, while the trains are for producing multiple, simultaneous output skims for dedicated purposes. ''Note, merging should normally only be used for full processing of an entire data set.''

''Note, regarding "timeline" production, there remain 2 interactive steps that must be executed after the SWIF workflow, see the [https://github.com/JeffersonLab/clas12-timeline/blob/main/doc/chef_guide.md minimal (for chef workflows)] or [https://github.com/JeffersonLab/clas12-timeline/blob/main/README.md full] timeline documentation for details.''

[[File:wokflow.png|center|frame|Diagram of an example workflow for one run number. Each rectangle represents a job, and arrows represent data files passed between jobs. The number and relationship between jobs depends on the model(s) and other workflow configuration parameters. The black jobs are included in all and only workflows that involve trains.]]

• --config

This option reads a configuration file for parameters common to most workflows, the idea being to reuse it many times and shorten the command line or document full passes (''not make a new config file for every little workflow''). Command-line arguments override those in the configuration file. To generate an example config file, see the --show option.

Output Locations¶

The logic here probably seems unnecessarily complex, but it accommodates flexibility that was needed in the past. We could just require all 4 options be defined even if they're not used, but that could lead to even more confusion and would sacrifice usability and promotion of a common directory substructure.

• --outDir This is the standard one for "everyday" workflows and applies to all workflow components. If outDir is all that's specified, the final output structure will look like this (directories only):

  outDir
  ├── decoded
  │   ├── 010001
  │   └── 010002
  └── schemaName
      ├── recon
      │   ├── 010001
      │   └── 010002
      └── train
          ├── wagonName1
          └── wagonName2

However, these directories are also used for temporary, intermediate files during the workflow: decoded and train files before merging. Train files are always merged into one per run per wagon. Decoded files are merged if requested.

• --workDir

This option sends temporary, intermediate files somewhere besides outDir. Primary use case is when outDir starts with /cache or /mss.

• --decDir

This redirects decoded files to somewhere other than outDir. This option came about for economy, to decode only once (and send it to tape) but write recon/train outputs to outDir.

• --trainDir

This redirects train files to somewhere other than outDir. This option came about for similar reasons, e.g. during a full reconstruction pass we may want to run trains to a non-tape location with the idea that they will be rerun before becoming permanent.

Post-processing Options¶

Post-processing operates single-threaded on the output of reconstruction, with the primary purpose to transfer tag-1 bank information into REC::Event.beamCharge/helicity and optionally rebuild some of the tag-1 banks, see below. * --nopostproc Thioption disables post-processing and should generally not be needed. * --recharge This option rebuilds the RUN::scaler bank from the RAW::scaler bank and is only applicable for cases where decoding was done previously with different CCDB parameters for Faraday cup calibrations. * --helflip This option inverts the helicity during post-processing and was never utilized and has been disabled. * --noheldel This option disables delayed helicity correction during post-processing.

--phaseSize¶

This parameter controls the number of jobs in each phase in the workflow, where SWIF enforces the constraint that jobs in phase i+1 are not submitted until all jobs in phase i are successfull. The value of --phaseSize is the threshold, in terms of number of jobs (runs) if greater (less) than 100, for incrementing successive jobs to the next phase, with the constraint that single runs are never split across multiple phases. This is similar to generating multiple smaller workflows and launching them in series, but phases are launched automatically. Note, this configuration is done during workflow generation and cannot be modified later.

The default is zero, which means no phasing. This gives maximum throughput and should always be used for smaller workflows, e.g. calibration iterations, a few runs in full, or a "pass0" of 10 files from every run.

Non-zero phase sizes are useful for creating run-ordering on tape, promoting full run completion for analysis, keeping a fixed footprint for temporary disk space, and avoiding large queues preventing other run groups from reasonable turnaround on their smaller workflows.

All Options¶

All chef command-line tools have full help information, accessible with the -h option. Here's its output for the workflow generator (note, this printout may be out of date):

ifarm9> clas12-workflow -h

  usage: clas12-workflow [-h] [...]

  Generate a CLAS12 SWIF workflow.

  optional arguments:
  -h, --help            show this help message and exit
  --runGroup NAME       (*) run group name
  --tag NAME            (*) e.g. pass1v0, automatically prefixed with runGroup
                        and suffixed by model to define workflow name
  --model NAME          (*) workflow model (dec/decmrg/rec/ana/decrec/decmrgre
                        c/recana/decrecana/decmrgrecana)
  --inputs PATH         (*) name of file containing a list of input files, or
                        a directory to be searched recursively for input
                        files, or a (quoted) shell glob of either. This option
                        is repeatable.
  --runs RUN/PATH       (*) run numbers (e.g. "4013" or "4013,4015" or
                        "3980,4000-4999"), or a file containing a list of run
                        numbers. This option is repeatable.
  --outDir PATH         final data location
  --decDir PATH         overrides outDir for decoding
  --trainDir PATH       overrides outDir for trains
  --workDir PATH        temporary data location for single train files before merging
  --coatjava VERSION/PATH
                        coatjava version number (or install location)
  --clara PATH          clara install location (unnecessary if coatjava is
                        specified as a VERSION)
  --threads #           number of Clara threads
  --reconYaml PATH      absolute path to recon yaml file (stock options = )
  --trainYaml PATH      absolute path to train yaml file (stock options =
                        calib/rf/trigger)
  --phaseSize #         number of files (or runs if less than 100) per phase,
                        while negative is unphased
  --mergeSize #         number of decoded files per merge
  --trainSize #         number of files per train job
  --reconSize #         number of files per recon job
  --postproc            enable post-processing of helicity and beam charge
  --recharge            rebuild RUN::scaler during post-processing
  --helflip             flip offline helicity
  --ccdbsqlite PATH     path to CCDB sqlite file (default = mysql database)
  --torus #.#           override RCDB torus scale
  --solenoid #.#        override RCDB solenoid scale
  --fileRegex REGEX     input filename format for matching run and file
                        numbers, default=".*clas[_A-Za-z]*_(\d+)\.evio\.(\d+)"
  --lowpriority         run with non-priority fairshare
  --config PATH         load config file (overriden by command line arguments)
  --defaults            print default config file and exit
  --show                print config file and exit
  --submit              submit and run jobs immediately

  (*) = required option for all models, from command-line or config file

Monitoring¶

Workflows¶

Monitoring workflows leverages multiple utilities. One is JLab scicomp's website:

  https://scicomp.jlab.org

  swif2

  swif-status.py

We also have a simple GUI that allows doing all the most common operations needed by chefs with only mouse clicks:

  swif-gui

Individual Jobs¶

Currently broken, looks like an API change ... The above swif-status.py does provide some job-level details, but some types of information are more easily interpreted from the SLURM database of finished jobs. For this, we have a convenience wrapper script, which includes summary statistics and histogramming of wall/cpu times and memory request/usage:

There is also scicomp's website, where the "Job Query" feature uses the same database as the previously mentioned script:

https://scicomp.jlab.org/scicomp/#/farmJobs/activeJob

Benchmarks¶

Reconstruction processing rates can also be extracted from log files and viewed with:

  clara-log-ana.py

cron¶

We use cron jobs primarily to automatically issue SWIF retries. The crontabs should be configured and installed for the chefs in their priority accounts by the software/computing coordinators. Locks are used to avoid pileup of simultaneous, duplicate tasks in case of system issues. Since CLAS12 has many simultaneously operating run groups, and cronjobs can lead to issues if not properly managed, it is important that cronjobs are not modified without approval.

These cron jobs must be configured with email addresses for the relevant parties, and it is important that these emails are not completely ignored. The overall, time-averaged job failure rate due to system issues is around few-%. Many days there will be zero cron emails, some days there can be many, depending on system performance, but abnormal failures can be indicative of misconfigured jobs and a workflow that will never work and needs to be deleted and fixed.

clas12mon¶

The cron jobs above can also be configured to send periodic snapshots of SWIF statuses to [http://clas12mon.jlab.org clas12mon]. This provides plotting of workflow progression over time and can be useful for a quick status check or catching system issues. This requires a certificate installed in the home directory for the given account. Contact the software/computing coordinators for more information.

Problems¶

SWIF classifies job problems and reports them with the various monitoring tools mentioned elsewhere on this wiki. Understanding what the problem types mean is important in determining the correct course of action. Some problem types are relatively common and due to system problems that will recover with a job retry. Others are generally unrecoverable without manual intervention and retrying them is a waste of time and resources.

Benign System Errors¶

These are indicative of system problems, nothing specific to our jobs, and can just be blindly retried. But, if they become much larger than ~1%, please report them. * SWIF_SYSTEM_ERROR - A problem with the SWIF management system. * SLURM_NODE_FAIL - The node the job was running on had issues. (See below for when the problem becomes common.) * SLURM_CANCELLED - SLURM cancelled the job for some reason. * SITE_LAUNCH_FAIL - The attempt to submit the job to SLURM failed. One way this happens is if the job submission is invalid, e.g., invalid node specification or account, but that cannot happen when using our chef workflow tools.

Critical Errors¶

These are indicative of something seriously wrong with the job itself and should never happen. In all cases you should see swif-status.py --problems PROBLEM_TYPE to get more details about the particular job's problems. * SWIF_INPUT_FAIL - The job specified an input file but SWIF could not retrieve it. This can happen if the input file was specified to be on disk and was deleted before the job ran. Or, if the file was specified to be from tape and its tape has problems. * SITE_REAP_FAIL - The job specified an output file, but that file was not found. This requires expert investigation and should be impossible with our standard jobs. * SITE_PREP_FAIL - This usually means the job's inputs are larger than the requested disk space. For rare cases, the job can just be modified and retried (see swif2 modify-jobs or swif-gui), otherwise it should be reported for investigation. * SLURM_OUT_OF_MEMORY - The job used more than the requested memory. This should never happen. For rare cases, the job can be modified and retried, otherwise it should be reported for investigation.

Internal Job Errors¶

• SLURM_FAILED This one ''usually'' means our job itself detected a problem and delivered a non-zero exit code, and determining what actually happened requires inspecting log files (see swif-status.py --problemLogs or swif-gui).

These are the two common causes, whose jobs can be retried: * CLARA stalls and eventually kills itself to avoid wasting resources * This is indicated by log files terminating with "watchdog killed" messages. * CCDB access failed * This is indicated by log files terminating during database server access, and can happen when many jobs are simultaneously accessing the server. The number of server connections [https://cc.jlab.org/systems/ganglia2/graph_all_periods.php?c=Database%20Cluster&h=clasdb-a.jlab.org&r=hour&z=small&jr=&js=&st=1598734889&v=7232&m=num_conn&ti=Number&z=large is visible here]. * For our largest-scale processing we should always just use SQLite to avoid this problem. See the [[CLAS12 Software Center]]'s "Reconstruction" tab for CCDB documentation and making an SQLite snapshot.

This one is uncommon and unpleasant to deal with, requiring manual intervention to recover it: * An input file is corrupted. See the next section.

File Corruption¶

Occasionally we find corrupted input files. If the source file is really corrupted, recovering them or abandoning their associated jobs, to allow a workflow to continue successfully, is a manual procedure:

• If the file in question is on /cache disk, it can be compared to its tape partner with: check-cached-tape.py [--crc32] [--md5] /cache/clas12/path/to/file/or/dir
• If the corrupted file appears to be on tape, the first step is to use the jget command (not jcache get) to retrieve the file from tape to somewhere other than the /cache, and then see if that disk copy is also corrupted. If that new copy is good, we can ask scicomp to delete the cached file and then the corresponding workflow job should recover automatically.
• If the file in question is EVIO, and the copy on tape is corrupted, we can try processing it with eviocure and see if the resulting file is fixed by running decoder on it manually. If that resolves it, we can ask scicomp to replace the EVIO file on tape with the fixed version (and delete the corresponding file on /cache), and then the job should recover automatically.
• If a decoded HIPO file is corrupted, the first option is to try to manually recreate it.
• In all cases, a fallback option is to use swif-status.py --abandonRun to abandon all jobs associated with a particular run number and let the workflow continue, and then revisit the run with the corrupted file(s) later.

Note, the log message for a corrupted EVIO file would normally contain: org.jlab.coda.jevio.EvioException: magic number is bad, 0x32333531

Refreshing Inputs¶

'''Starting in 2022''', SWIF preserves a cache of jobs' input files and reuses it on a retry, to avoid unnecessary transfers. But if that input file was corrupt, then retries will just keep failing on the cached copy, even after fixing the source. The error state should be SLURM_FAILED. To recover from this: * Recreate the corrupt file, confirm it's good, and replace the corrupt one. * Use a new (and currently undocumented) option to clear the cache and retry the job:

swif2 retry-jobs WORKFLOWNAME -refresh-input /path/to/previously/corrupt/file SWIF_JOB_ID

  swif-status.py --refresh --workflow WORKFLOWNAME

Resuming Jobs¶

Occaisonally SWIF database or Lustre issues cause a job to fail during the stages external to our job itself. These can result in SITE_REAP_FAIL,SWIF_INPUT_FAIL,SWIF_OUTPUT_FAIL, SWIF_MISSING_OUTPUT, and can sometimes be recovered without retrying the full job by a (currently undocument) SWIF command "resume-jobs", or, in our wrapper:

  swif-status.py --resume --workflow WORKFLOWNAME

Stuck Jobs¶

In the past jobs were occasionally found eternally stuck in a pending state. The first option is to get the SLURM ID for the job and try to "release" it:

  scontrol release SLURM_ID

  scancel SLURM_ID

Problem Nodes¶

Occasionally there are issues on particular nodes, e.g. very slow, and jobs that land there will never succeed. This will show the distribution of failures across nodes:

  swif-status.py --problemNodes

  scancel -U $USER -w 'farm[1909-1912]'

To also modify all pending SLURM jobs to exclude the same nodes:

  slurm-exclude-nodes.sh 'farm[1909-1912]'

Tips¶

Input Data Specification¶

The --inputs option accepts directories or files. Files are assumed to contain a list of full paths of EVIO/HIPO files, and directories are searched recursively for all EVIO/HIPO files. The --runs argument then specifies which runs to include from those files.

For a large run group, you don't want to be frequently crawling the same, unchanged filesystem every time you generate a workflow. Instead, for example, do find /mss/clas12/rg-X/data > raw.txt and then use raw.txt as the argument of the --inputs option.

Stock Bank Schema¶

The selection of banks output during reconstruction is limited to those in the schema_dir defined in CLARA's YAML file. It must be specified as an absolute filesystem path. There are a few relevant preset or "stock" schema delivered in COATJAVA installations, listed below and ordered by size, and it is important to choose the correct one for a particular workflow. ''The analysis coordinator for your run group should know which schema to use. If unsure, contact the computing/software coordinators.'' * dst * dsthb * calib * mon * full Note, the mon and full schema generate a very large volume of data and should only be used for small projects, e.g. a pass0 on 10 files per run.

Stock Trains¶

During normal calibration phases, "stock" train configurations are used from the workflow generator, without any need for a custom YAML file. These are currently the rf, calib, and trigger arguments to the --trainYaml option, which are shorthand for [https://github.com/baltzell/clas12-workflow/tree/master/yamls these YAMLs] instead of specifying the path to a file. They include predefined skims with bank schema suitable for all run groups' calibrations.

Common Mistakes¶

YAML¶

Currently the most common mistakes are in the YAML configuration files for CLARA reconstruction and trains, e.g. incorrect or even invalid CCDB timestamp or variation and missing class names or service sections. Standard YAML files are [https://github.com/JeffersonLab/clas12-config/tree/main/coatjava maintained here], but production workflows should always specify a CCDB timestamp and variation. A simple, example train YAML is [https://github.com/baltzell/clas12-workflow/blob/master/yamls/train_rf.yaml train_rf.yaml].

All possible YAML integrity checks are run automatically during workflow generation and will abort upon error, e.g. YAML syntax, no hidden/special characters, valid CCDB timestamps and variation names and Java classes, and consistent service and configuration sections. These checks can also be run manually with:

  clara-check-yaml.py

Note: * The required format for CCDB timestamps is MM/DD/YYYY-HH:MM:SS, where hours are 24-hour time, each component must be 2 digits (except 4 for the year). Yes, the date format is unfortunately the silly US convention: month first, then day, they year! * Some COATJAVA versions add new CCDB tables and will not work with a timestamp prior to the release, and that should be mentioned in the release notes. * Modifying a YAML file after workflow generation is usually bad idea, and the workflow generator even removes the write bit to avoid accidents.

Trigger Bit Filtering¶

Trigger bit filtering causes only events that satisfy a trigger bit mask to be processed, and all others ignored and dropped from the output data. This can be useful for cases where only certain triggers are of interest, and the others are a significant waste of CPU cycles and disk space. However, the trigger bit definitions change over time, and choosing incorrectly can result in not-so-useful output data, so care should be taken.

Schema + Trains Combos¶

The full, mon, and calib schema should generally never be used with physics trains.' Those schema are very large, and physics trains generally have no bank filtering, so locally staged data on the batch node gets huge, the run-merged files are huge and take a long time to merge, and the end result is not ideal for any common analysis. However, it is common to use the rf or calib trains with those large schema, as they have bank filtering for each wagon.

Trains¶

The maximum number of wagons is 32, and the corresponding ids are limited to integers between 1 and 32, inclusive. If not using the workflow tools, like when interactively testing a train configuration, numbers outside that range will not report error but modulate or rollover back to the valid range with unintended consenquences.

Other¶

Care should be taken with the --helflip option, which applies an overall sign flip to the offline helicity during post-processing in reconstruction jobs. One use case is data decoded prior to COATJAVA 6.5.11, which had the wrong offline helicity sign. Plus, in general, the helicity hardware signal from the accelerator can have an overall sign flip relative to the actual electrons on the Hall B target, and that depends on the accelerator configuration and thus the run period in Hall B.

YAML Tricks¶

Global Section¶

For settings common to all services, e.g. CCDB variation and timestamp, a global section in the severices section is honored (and overriden by service-specific ones of the same name). For example:

configuration:
  global:
    variation: default
    timestamp: MM/DD/YYYY-HH:MM:SS
    beamEnergy: 11.023

Custom Train Names¶

For trains, the I/O service always outputs files with prefixes named according to their skim ''number''. However, the workflow can be configured to rename the final, run-merged trains with a customizable prefix for easier human consumption. This is based on an additional configuration subsection in the train YAML file, for example:

configuration:
  custom-names:
    1: jpsitcs
    2: pentaquark
    3: hyperon
    4: semi-inclusive

Fileservers¶

Quotas on all JLab's scicomp filesystems are available in the left sidebar at [https://scicomp.jlab.org https://scicomp.jlab.org], and usage for large workflows should be discussed with the Hall B and CLAS collaboration computing/software coordinators in advance.

Note, large sets of workflows on a full data set, e.g. a "pass1", must be configured to read their inputs from tape /mss even if those inputs are already on the /cache disk. This is, in part, to avoid any risk of auto-deletion of inputs between workflow generation and when the jobs actually run. Note, SWIF will only trigger a (re)read of a file specified as /mss/... from tape if it's not already cached on disk, and it will keep that file pinned on cache until the job finishes.

• /work

The /work fileserver should be avoided for all workflow jobs by CLAS12 run groups chefs until it is upgraded to more I/O-performant hardware. Meanwhile, only tape and Lustre should be used for inputs and outputs, i.e. paths starting with one of /mss, /cache, or /volatile. The /work filesystem also requires manual management to keep it from going over quota.

• /mss (tape) Sometimes a ls or find can be slow due to the large number of files. A much faster and otherwise equivalent option to list everything on tape is:

    jasmine list-files -under /mss/clas12/rg-a/data | awk '{print$1}'
  

  A common optimization topic is tape access, although our current reconstruction speeds are far from a tape bottleneck under normal system performance. Leveraging pinning of files on /cache serves to avoid unnecessarily rereading large volumes of data from tape, e.g. data that is known will be reused in the near future. Also, pre-caching a subset of data from tape, in advance of a large-scale processing, is an associated strategy. These should be discussed with the software/computing coordinators in advance, in order to properly manage our pin quota and cache usage across all run groups.

• /farm_out

Log files are written to /farm_out/$USER/$WORKFLOW_NAME. There are quotas and an auto-deletion algorithm in place that are usually sufficient. However, our software can get misconfigured (e.g. YAMLs above), or input data corrupted, and result in generating humongous log files. If you see jobs erring with quota issues, check your log usage, e.g.

du -sh /farm_out/$USER

Auto-deletion¶

The /volatile and /cache filesystems are managed by auto-deletion algorithms. Files written to /cache are guaranteed to be written to tape before auto-deletion kicks in. Current usage and auto-deletion queues of CLAS12 filesystems are available at [[CLAS12 Software Center#FAQ|FAQ #5]] on the sowftware wiki and updated weekly, and the usual lifetime is a few months.

Deleting from Tape¶

November 2024: This section is presumably out of date and requires updating for read-only /cache.

Note, deleting files from tape should rarely be necessary and discussed first.

• If the files in question have never been on the /cache disk (which is generally not the case in our workflows that use write-through cache)
• Then jremove is all that is necessary to remove them from tape.
• Otherwise:
• First the files will need to be currently present on /cache, e.g. by using jcache get
• Then, to delete the files from tape, use jcache tapeRemove /mss/..
• ''At this point, the files will still be disk resident on /cache. If the desire is really to delete these copies too, just use the standard rm command. In case they were previously retrieved from tape, their permissions will be such that we have to ask scicomp to delete them.''

Pinning on /cache¶

Sometimes we want to preserve data on the /cache disk to avoid unnecessarily reretrieving it from tape in future months. This should be discussed in advance in the chef meetings to help manage our pin quota. Here "pinning" essentially means moving things to the bottom of /cache's autodeletion queue. This example would pin all raw EVIO files in the 40s on RG-C's /cache for 60 days:

  jcache pin -D 60 /cache/clas12/rg-c/data/*/*0004?

Examples¶

Workflow Generation¶

First, a configuration file named ./cfg.json containing:

  {
    "runGroup" : "rgb",
    "coatjava" : "6.5.6",
    "mergeSize" : 5,
    "phaseSize" : 5000,
    "reconYaml" : "/path/to/recon-custom.yaml",
    "trainYaml" : "/path/to/train-custom.yaml",
    "inputs" : "/mss/clas12/rg-b/data",
    "runs" : "/path/to/goldenruns.txt"
  }

<source lang="bash">
  tape=/mss/clas12/rg-b
  cache=/cache/clas12/rg-b/production
  volatile=/volatile/clas12/rg-b/production
</source>

  clas12-workflow --config cfg.json --tag pass1v0 --model decmrg --runs 10200,10201,10203 --decDir $cache/decoded/6.5.6

  clas12-workflow --config cfg.json --tag v25.1 --model decrec --runs 10200-10300 --inputs "$tape/data/clas_*/*.0004*" --outDir $volatile/recon/pass0/v25.1 --decDir $volatile/decoded/6.5.6 --postproc --recharge

  --inputs $tape/data --fileRegex '.*clas[_A-Za-z]*_(\d+)\.evio\.(0004\d)'

Run only trains with the stock train "calib" YAML:

  clas12-workflow --config cfg.json --tag v25.1 --model ana --inputs "$cache/pass1/recon/dst/" --outDir $volatile/recon/pass0/v25.1 --trainYaml calib

  clas12-workflow --config cfg.json --tag pass1v0 --model decmrgrecana --inputs $tape/data --decDir $cache/decoded/6.5.8 --outDir $cache/recon/spring2019/torus-1/pass1/v0 --workDir $volatile/foo/bar --trainDir $volatile/.../pass1/v0 --postproc --recharge

Monitoring¶

clas12-1@ifarm1901:~> swif-status.py --workflow rgb-ra-pass1v1-11420x60 --stats

  ------------------ rgb-ra-pass1v1-11420x60 -------------------
       total:      6455 /    15567 = 41.4659%
       recon:      6455 /    12494 = 51.6648%
         ana:         0 /     2524 = 0.0000%
      anamrg:         0 /       60 = 0.0000%
    anaclean:         0 /       60 = 0.0000%
        jput:         0 /      429 = 0.0000%

clas12-1@ifarm1901:~> swif-status.py --workflow rgb-ra-pass1v1-11420x60 --runStats

  ------------------ rgb-ra-pass1v1-11420x60 -------------------
       total:      6468 /    15567 = 41.5494%
      011420:       265 /      325 = 81.5385%
      011421:        18 /       24 = 75.0000%
      011422:       293 /      368 = 79.6196%
      011423:       231 /      288 = 80.2083%
      011429:       109 /      133 = 81.9549%
      011430:       265 /      330 = 80.3030%

clas12-5@ifarm1901> swif-status.py --workflow rgm-dra-production-15053x2 --problemNodes
  Problem summary for rgm-dra-production-15053x2:
        Node      AUGER-CANCELLED         AUGER-FAILED     AUGER-INPUT-FAIL         AUGER-OUTPUT    AUGER-OUTPUT-FAIL         AUGER-SUBMIT        AUGER-TIMEOUT  SWIF-MISSING-OUTPUT    SWIF-SYSTEM-ERROR   SWIF-USER-NON-ZERO
  farm180220                    0                    0                    0                    0                    0                    0                    0                    0                    0                    1
    farm1909                    0                    4                    0                    0                    0                    0                    0                    0                    0                    0
    farm1910                    0                    2                    0                    0                    0                    0                    0                    0                    0                    0
    farm1911                    0                    1                    0                    0                    0                    0                    0                    0                    0                    0
        AUGER-FAILED : recon:7
  SWIF-USER-NON-ZERO : ana:1

Disk Management¶

Filesystems¶

Note, quotas and usages for all filesystems can be accessed at [https://scicomp.jlab.org JLab's scientific computing website], and the [https://clasweb.jlab.org/clas12offline/disk/ auto-deletion queue for Lustre filesystems] is updated daily.

• /volatile/clas12

''This filesystem is where all transient data goes, like for testing and calibrations.''

This is a Lustre filesystem for data, large I/O, with auto-deletion and no backups. The system has a "guarantee" quota, below which data will not be deleted, and a "high" quota above which deletion is guaranteed.

• /cache/clas12

This filesystem is where all full data sets should be written.

This is also a Lustre filesystem and provides access to the JLab tape library. It is where data is staged to disk from tape, e.g. by the jcache get command. Additionally, new files larger than 1 MB written directly to /cache will be automatically transferred to tape, generally after they're a week or two old but can be forced to go to tape now with jcache put. The quota system and auto-deletion queue is similar to /volatile, but it isn't executed until the data is already on tape. So, with the "1 MB" caveat, this filesystem is automatically backed up to tape.

• /mss/clas12 If reproducibility or having a precise file list is important, workflows's should specify their inputs to be from /mss for data on tape, even if it's already on the /cache disk.

This filesystem is just metadata files mirroring the directory structure and contents of the tape silo. That metadata contains the usual file names and timestamps, plus checksums, tape identifiers, etc. SWIF inputs on tape are read and copied to /cache automatically (but only if necessary).

• /mss/hallb/scratch This filesystem is appropriate for large but partial data processings to be used for physics analysis, where /volatile is undesireable due to its shorter and more uncertain lifetime.

This is a special tape volume for longer-term (think ~1 year) but non-permanent storage. The general policy is that at some point the tapes will need to be repurposed, and then we'll be given a few months to copy any data requiring permanent storage. It is parterned with the /cache/hallb/scratch disk filesystem.

• /work/clas12 This is a ZFS filesystem with a fixed, hard quota, no auto-deletion, and automatic snapshot backups accessible at /work/clas12/.zfs. It is much less performant at large I/O than Lustre, so we cannot use /work/clas12 for anything in clas12 workflows except SQLite snapshots and software.

Maintanence¶

Currently this mainly applies to /volatile, as that is the dumping ground for a lot of routine data processing by chefs in support of calibrations and benefits from the auto-deletion algorithm on that filesystem. However, the "write it and forget it" mentality is common and decreases the lifetime of everyone's data on /volatile.

Monitoring¶

Various tools exist to assess disk usage at JLab:

• The "File System" section in the left sidebar at [https://scicomp.jlab.org/scicomp/home https://scicomp.jlab.org].
• This is more generic but does support filtering for clas12 and sorting by usage
• Some clas12-specific tallies and our auto-deletion queue on /volatile, linked from [https://clasweb.jlab.org/wiki/index.php/CLAS12_Software_Center#tab=FAQ the FAQs].
• Standard old linux utilities, for example:
• To see the total disk usage of a given directory or all its subdirectories:
  • du -sh /volatile/clas12/rg-m/pass0v2
  • du -sh /volatile/clas12/rg-m/pass0v2/*

The find command can do all kind of fancy things, but the syntax can be a bid hard to remember. To get a list of all empty directories:

find /volatile/clas12/rg-m -type d -empty

find /volatile/clas12/rg-m -type f -size +10G -mtime +7

Cleanup¶

To increase the lifetime of useful data, it would be ideal if we could more proactively delete no-longer-needed data on /volatile. This requires agreement between the run group's analysis coordinator and CALCOM, and execution by the run group's chef.

It is also necessary to occasionally get rid of all the old empty directories, as they are not deleted by the automated system but still put overhead on the Lustre metadata server.

Some tools to facilitate cleanup are:

• Standard old linux find utility, for example: ** To delete everything older than 20 days in a particular directory: *** find /volatile/clas12/rg-m/ -mtime 20 -delete
• A dedicated python script, with more user-friendly command-line options and help (and doesn't modify times on directories, unlike find -delete): ** /cvmfs/oasis.opensciencegrid.org/jlab/hallb/clas12/sw/noarch/clas12-utilities/dev/osg/bin/disk-cleanup.py ** For example, this would delete all empty directories older than 20 days and all files older than 30 days: *** disk-cleanup.py -path /volatile/clas12/rg-m -empty 20 -delete 30