Introduction to CERN GitLab CI/CD

  • GitLab CI/CD automates repetitive tasks and helps ensure code quality.
  • Pipelines are defined in .gitlab-ci.yml and consist of stages and jobs that run automatically on each push.

A simple CMSSW example

  • CMSSW analysis code must be placed inside the src directory of your CMSSW work area to be compiled.
  • Always add required CMSSW packages before copying or compiling your analysis code.
  • Running and testing your analysis locally helps catch issues before automating with CI.
  • Comparing results after code or selection changes is essential for reliable and reproducible analyses.

Setting up CMSSW in GitLab CI/CD

  • GitLab CI runs on remote runners; capture all environment setup in .gitlab-ci.yml and do not rely on local machine files.
  • Use a CVMFS-enabled Kubernetes runner tag (e.g., k8s-cvmfs); the image: keyword is only honored on Kubernetes runners; verify /cvmfs/cms.cern.ch is accessible.
  • Set up CMSSW with cmsset_default.sh, SCRAM_ARCH, cmsrel, and cmsenv, then validate with cmsRun --help.
  • Temporarily relax shell strict mode (set +u … set -u) around setup to avoid failures from unset variables or non-standard exit codes; inspect job logs to diagnose issues.

Running a CMSSW analysis in GitLab CI/CD

  • GitLab CI runs on remote runners; every dependency and setup step must be in .gitlab-ci.yml.
  • Use CVMFS-enabled Kubernetes runners (k8s-cvmfs); the image: keyword is honored only there.
  • Split the pipeline into stages and pass the built CMSSW area via artifacts to avoid rebuilding in each job.
  • Validate outputs in CI (e.g., myZPeak.root, event counts) to catch issues early; inspect job logs for failures.
  • Artifacts are write-protected by default; downstream jobs must copy them to a writable directory using mkdir, cp -r, and chmod -R +w.
  • Define artifacts with untracked: true to capture build outputs, and set expire_in to control automatic cleanup (e.g., 1 hour for testing, 1 week for production).
  • Use the needs keyword for faster pipelines—jobs start immediately when dependencies complete, rather than waiting for entire stages to finish. dependencies is an alternative that strictly respects stage order.

Securely adding passwords and files to GitLab

  • Never store grid certificates or passwords in version control; they remain in commit history and violate grid policy.
  • Encode grid certificates and passwords with base64 and store them as protected CI/CD variables in GitLab.
  • Set variables to Protected (available only on protected branches) and Masked (hidden in logs) for maximum security.
  • Restore grid certificate files and obtain a VOMS proxy in CI jobs using base64 -d and voms-proxy-init --pwstdin.

CAT services for GitLab CI

  • CAT services simplify CMS resource access in GitLab CI by eliminating manual certificate and credential management.
  • CAT services only work in the cms-analysis namespace for security; move your project there via Settings > General > Advanced > Transfer project.
  • The CAT EOS file service provides JWT-authenticated access to datasets in /eos/cms/store/group/cat via the cmscat service account.
  • The CAT VOMS proxy service provides short-lived grid proxies without storing personal certificates; proxies are returned as base64-encoded strings.
  • Both services require configuring id_tokens in .gitlab-ci.yml and querying CAT endpoints to retrieve authentication tokens.

Putting It All Together: Final Run with CAT Services

EXTRA: Building a Container Image in GitLab CI/CD