CustomDriver
Idea
Use a GitLab CI Costum Runner to handle HPC resource access for CI Pipelines.
Stages
Config
- Store which type of Runner shall be used (shell, Slurm, Singularity, others will follow)
- Log GitLab user info
- Store CI trigger info for accountability
- Define batch parameters
Prepare
- Get the cluster username
- Log CI access for user
- Downscope to user
- Startup the Runner environment (e.g. get an interactive batch job, submit job to batch etc.)
- Create reasonable directories
Run
- Run the generated scripts in the changed environment
Cleanup
- End batch session
- Clear tmp files
Wished Funktionality
- (Singularity) container spawning
- Shell jobs should be executed on batch nodes as batch jobs
- Slurm jobs should directly submit jobs to the queue and wait for completion
- Canceling a pipeline should cancel slurm jobs (not applicable for shell runner)
- User logs and admin logs should be generated and identifyable in case of problems
- Admin logs should contain infos on the repo, user, script etc. that is to be executed