gitlab-runner.md

name	GitLab Runner Codebase Deep Dive
description	Conversation notes on what GitLab Runner does and how it executes CI jobs — entrypoint, build flow, and executor internals
type	project
originSessionId	33921352-0ffd-4509-a7a3-4d2b4dfff777

GitLab Runner is the official CI/CD job executor for GitLab, written in Go. It picks up CI/CD jobs from a GitLab instance, executes them, and streams results back.

---

Job Request Flow

When the runner receives a job request from GitLab server, the call chain is:

processRunners          (commands/multi.go:840)
  → processRunner       (commands/multi.go:898)
    → processBuildOnRunner  (commands/multi.go:945)
      → requestJob      (commands/multi.go:1082)
        → network.RequestJob(...)    ← HTTP POST /api/v4/jobs/request
        → network.ProcessJob(...)    ← creates trace/log stream
      → build.Run(config, trace)     ← execution starts here (common/build.go:1151)

---

build.Run — Top-Level Execution (common/build.go:1151)

Build.Run
  ├── executeBuildSection → executor.Prepare()   (spin up container/VM/shell)
  └── build.run (goroutine)
      └── executeScript
          ├── executePrepareScripts
          │   ├── executeStage(prepare)           → GenerateShellScript → executor.Run()
          │   ├── executeStage(get_sources)        → git clone/fetch
          │   ├── executeStage(restore_cache)
          │   └── executeStage(download_artifacts)
          ├── executeUserScripts
          │   └── for each Step (before_script, script, …)
          │       └── executeStage(step)           → GenerateShellScript → executor.Run()
          ├── executeArchiveCache
          └── executeUploadArtifacts

1. Expand inputs / resolve secrets — interpolates CI variables and fetches vault secrets
2. executeBuildSection (common/build.go:1205) — prepares the executor:
   • Calls retryCreateExecutor → provider.Create() then executor.Prepare(options)
   • This is where Docker containers are spun up, SSH connections made, Kubernetes pods created, etc.
3. build.run(ctx, trace, executor) (common/build.go:1217) — runs the actual job in a goroutine, then selects on: timeout / interrupt / finish / panic

---

build.run — Goroutine (common/build.go:875)

Spawns a goroutine calling executeScript, then waits for it to finish or be interrupted/cancelled.

---

executeScript — Stage Orchestration (common/build.go:574)

Runs the job in this order:

1. executePrepareScripts — runner-controlled setup stages:
   • prepare — shell profile init
   • get_sources — git clone/fetch (with retries)
   • restore_cache
   • download_artifacts
2. executeUserScripts (common/build.go:645) — iterates over b.Steps (the user-defined before_script, script, etc.) and calls executeStage for each
3. executeArchiveCache — uploads cache
4. executeUploadArtifacts — uploads artifacts
5. executeUploadReferees — uploads metric reports

---

executeStage — Per-Stage Runner (common/build.go:440)

The core per-stage runner:

1. Calls GenerateShellScript(ctx, buildStage, *shell) — generates the shell script for this stage
2. Wraps it in an ExecutorCommand{Script, Stage, ...}
3. Calls executor.Run(cmd) — hands it off to the executor (Docker, Shell, K8s, etc.) to actually execute the script

---

How Executors Execute Scripts

The Executor.Run(cmd ExecutorCommand) interface is defined in common/executor.go:69. Each executor receives a generated shell script string in cmd.Script and runs it differently.

executeStage(buildStage)
  └── GenerateShellScript()     → produces shell script string
      └── executor.Run(cmd)
            Shell    → os/exec (bash/sh/powershell)
            Docker   → Docker API attach (stdin = script)
            K8s      → kubectl exec or emptyDir script file + attach
            SSH      → SSH session + script over stdin

The key insight: the script is always passed as a string (cmd.Script). Each executor decides whether to pipe it as stdin, write it to a temp file, or copy it to a volume — but they all execute the same generated shell script.

---

Shell Executor (executors/shell/shell.go:70)

The simplest: runs the script as a local OS process.

Run(cmd)
├── Build stdout/stderr loggers
├── shellScriptArgs(cmd)
│   ├── PassFile=false → pass script as stdin to the shell process
│   └── PassFile=true  → write script to a temp file, pass as argument
├── newCommander(BuildShell.Command, args, opts)   ← e.g. "bash", "/bin/sh", "powershell"
├── c.Start()                                       ← os/exec spawns the process
└── select { waitCh | cmd.Context.Done() → KillAndWait }

---

Docker Executor (executors/docker/docker_command.go:87)

Runs the script inside a Docker container.

Run(cmd)
└── runContainer(containerType, cmd)
    ├── requestContainer()       ← get/create the pre-pulled build or predefined (helper) container
    ├── startAndWatchContainer(ctx, containerID, bytes.NewBufferString(cmd.Script))
    │   ├── exec.NewDocker(...)  ← wraps Docker API client
    │   ├── build stdout/stderr loggers as IOStreams
    │   └── dockerExec.Exec(ctx, containerID, streams, gracefulExitFn)
    │       └── Docker ContainerAttach API → streams stdin (script) → waits for exit
    └── return err

Two container types:

• predefined → helper container (gitlab-runner-helper image) — used for runner-controlled stages like get_sources, restore_cache
• build → user's image — used for before_script, script, after_script

---

Kubernetes Executor (executors/kubernetes/kubernetes.go:565)

Runs the script in a pod container. Two strategies:

Legacy (runWithExecLegacy):

setupPodLegacy()               ← create K8s Pod if not exists
runInContainerWithExec(...)    ← kubectl exec equivalent, passes cmd.Script via stdin

Attach mode (runWithAttach, default):

ensurePodsConfigured()
saveScriptOnEmptyDir(ctx, stageName, containerName, cmd.Script)
  └── writes script to an emptyDir volume mounted in the pod
→ attaches to container and watches for exit

---

Key Files

File	Purpose
commands/multi.go	Job polling, worker dispatch, requestJob, processRunner
common/build.go	Build.Run, executeScript, executeStage — the core execution orchestration
common/executor.go	Executor interface definition
network/gitlab.go	RequestJob (HTTP poll) and ProcessJob (trace stream)
executors/shell/shell.go	Shell executor — runs scripts via os/exec
executors/docker/docker_command.go	Docker executor — runs scripts via Docker ContainerAttach API
executors/kubernetes/kubernetes.go	Kubernetes executor — runs scripts via kubectl exec or emptyDir + attach