GitLab 08 🦊 Introduction to GitLab CI/CD and Pipeline Architecture - KandZ

⏱️ Estimated read: 10 min | |

A pipeline is the engine of GitLab’s DevSecOps platform. It is not a single script; it is a directed acyclic graph of jobs, organized into stages, executed by runners, and triggered by events. The .gitlab-ci.yml file is the definition, but the architecture—the relationship between stages, jobs, and dependencies—determines whether the pipeline is a linear sequence that waits unnecessarily or a parallel graph that finishes as fast as the dependencies allow. The exam tests the basic structure: stages define order, jobs define work, runners execute the work, and needs breaks the stage barrier when a job can start before the previous stage completes.

Key point: A basic pipeline runs stages sequentially and jobs within a stage in parallel. The needs keyword bypasses the stage ordering by declaring explicit job dependencies. A pipeline is triggered by an event (push, merge request, schedule, manual) and executed by a runner. The .gitlab-ci.yml file must be at the repository root and is case-sensitive.

Why pipeline architecture matters

The stage-barrier problem. In a basic pipeline, all jobs in a stage must complete before any job in the next stage begins. If build_a finishes in 10 seconds and build_b takes 10 minutes, the test stage cannot start until build_b finishes—even if the test job only depends on build_a. This is the stage barrier: it guarantees ordering but not efficiency. The needs keyword removes the barrier by declaring that a job depends on specific jobs, not on an entire stage. The dependent job starts as soon as its declared dependencies finish .

The runner problem. A pipeline does not execute itself. A runner is the agent that picks up jobs and runs them. Runners can be shared across a GitLab instance, specific to a group, or specific to a project. The runner loads the container image specified in the job, clones the repository, and executes the script. Without a runner, the pipeline stays in the pending state indefinitely.

The trigger problem. A pipeline is created by an event. The most common triggers are a push to a branch, a merge request event, a schedule (cron), or a manual action in the UI. The rules keyword controls which jobs run for which triggers, allowing a pipeline to behave differently on a feature branch than on the default branch .

The configuration problem. The .gitlab-ci.yml file is parsed before the pipeline starts. A syntax error prevents the pipeline from being created. The file can include other files (include), use hidden jobs (.job_name) for shared configuration, and use YAML anchors and map merges to reduce repetition. The pipeline editor in the GitLab UI validates the syntax before the commit is made.

The trade-off. A basic pipeline is simple to read and maintain. A needs-based pipeline is more complex but faster. The correct choice depends on the size of the project. For a small project with three stages, the basic pipeline is sufficient. For a monorepo with fifty jobs, the needs pipeline is the only way to keep the feedback loop short.

a. The basic pipeline: stages and jobs

The .gitlab-ci.yml file defines the pipeline. The stages keyword lists the stages in order. Each job declares which stage it belongs to and what it runs.

# .gitlab-ci.yml
stages:
  - build
  - test
  - deploy

build-job:
  stage: build
  script:
    - echo "Compiling the code..."
    - make build

test-job:
  stage: test
  script:
    - echo "Running tests..."
    - make test

deploy-job:
  stage: deploy
  script:
    - echo "Deploying..."
    - make deploy

The pipeline runs the build stage first. When all build jobs complete, the test stage runs. When all test jobs complete, the deploy stage runs. If any job fails, the pipeline stops and the subsequent stages do not run . The script keyword is a list of shell commands. A failure in any command fails the job.

Jobs can declare an image to specify the container environment. The runner pulls the image and executes the script inside it.

build-job:
  stage: build
  image: node:20
  script:
    - npm ci
    - npm run build

The image keyword is optional. If omitted, the runner uses its default environment. The script is mandatory; a job without a script is invalid.

b. The `needs` pipeline: breaking the stage barrier

The needs keyword declares explicit dependencies between jobs. A job with needs starts as soon as its dependencies finish, regardless of the stage it belongs to. This allows a deploy job to run immediately after the build job it depends on, without waiting for unrelated test jobs in the same stage .

stages:
  - build
  - test
  - deploy

build_a:
  stage: build
  script: echo "Fast build"

build_b:
  stage: build
  script: sleep 60; echo "Slow build"

test_a:
  stage: test
  needs: [build_a]
  script: echo "Test A runs after build_a"

test_b:
  stage: test
  needs: [build_b]
  script: echo "Test B runs after build_b"

deploy_a:
  stage: deploy
  needs: [test_a]
  script: echo "Deploy A runs after test_a"

In this example, test_a starts as soon as build_a finishes. It does not wait for build_b, which is still running in the build stage. The deploy_a job starts after test_a completes. The pipeline graph is a dependency graph, not a stage sequence. The needs: [] syntax makes a job start immediately, with no dependencies .

The needs keyword is the primary optimization for large pipelines. It reduces the total pipeline duration by allowing independent paths to run in parallel.

c. Runners, variables, and triggers

A runner executes the pipeline. GitLab.com provides shared runners for Linux, Windows, and macOS. A self-managed instance requires runners to be registered. The runner is tagged with labels (docker, shell, macos, etc.), and a job can specify tags to select a runner that matches.

job:
  tags:
    - docker
  script: echo "Runs on a docker runner"

CI/CD variables are key-value pairs that customize jobs. They can be defined in the .gitlab-ci.yml file, in the project settings, or in the group settings. The predefined variables (CI_COMMIT_SHA, CI_JOB_ID, CI_COMMIT_BRANCH) provide context about the current job and pipeline .

variables:
  NODE_ENV: "production"

job:
  script:
    - echo "Branch: $CI_COMMIT_BRANCH"
    - echo "Environment: $NODE_ENV"

Variables can be protected (only available on protected branches) or masked (hidden in job logs). Both settings are configured in the project settings, not in the YAML file.

The pipeline is triggered by an event. The rules keyword controls which jobs run for which events. A job with rules: if: $CI_COMMIT_BRANCH == "main" only runs on the main branch.

deploy:
  stage: deploy
  script: echo "Deploying"
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

Complete Example Session

# ============================================
# PART 1: CREATE THE BASIC PIPELINE
# ============================================

cat > .gitlab-ci.yml <<'EOF'
stages:
  - build
  - test

build-job:
  stage: build
  script:
    - echo "Building..."

test-job:
  stage: test
  script:
    - echo "Testing..."
EOF

# ============================================
# PART 2: COMMIT AND TRIGGER
# ============================================

git add .gitlab-ci.yml
git commit -m "Add CI pipeline"
git push origin main
# GitLab creates a pipeline and assigns it to a runner.

# ============================================
# PART 3: VIEW THE PIPELINE
# ============================================

# Build → Pipelines → click the pipeline
# The graph shows build-job, then test-job.

# ============================================
# PART 4: ADD A NEEDS DEPENDENCY
# ============================================

cat > .gitlab-ci.yml <<'EOF'
stages:
  - build
  - test

build_a:
  stage: build
  script: echo "Fast"

build_b:
  stage: build
  script: sleep 60; echo "Slow"

test_a:
  stage: test
  needs: [build_a]
  script: echo "After build_a"
EOF

# ============================================
# PART 5: OBSERVE THE GRAPH
# ============================================

# The pipeline graph shows test_a starting
# while build_b is still running.

# ============================================
# PART 6: ADD A VARIABLE
# ============================================

variables:
  APP_ENV: "staging"

job:
  script:
    - echo "Environment: $APP_ENV"

# ============================================
# PART 7: ADD A RULE
# ============================================

deploy:
  stage: deploy
  script: echo "Deploying"
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

# ============================================
# PART 8: RUN ON A TAGGED RUNNER
# ============================================

job:
  tags:
    - docker
  script: echo "Docker runner"

# ============================================
# PART 9: CHECK PIPELINE STATUS
# ============================================

# Build → Pipelines
# Green: passed
# Red: failed
# Gray: pending (no runner available)

The nine parts trace the pipeline lifecycle: basic configuration, commit, view, needs dependency, graph observation, variable, rule, runner tag, and status check.

Quick Reference

Pipeline Components

Component	Purpose
`stages`	Defines execution order
`job`	A unit of work
`script`	Commands to execute
`image`	Container environment
`tags`	Runner selection
`needs`	Explicit job dependency
`rules`	Conditional execution
`variables`	Key-value configuration

Trigger Events

Event	Description
Push	Commit to a branch
Merge request	MR opened or updated
Schedule	Cron-based
Manual	Run from UI
API	External trigger

Predefined Variables

Variable	Value
`CI_COMMIT_SHA`	Commit hash
`CI_COMMIT_BRANCH`	Branch name
`CI_JOB_ID`	Job identifier
`CI_PIPELINE_ID`	Pipeline identifier

`needs` Syntax

Syntax	Meaning
`needs: [job_a]`	Depends on job_a
`needs: []`	No dependencies, starts immediately
`needs: [job_a, job_b]`	Depends on both

Best Practices

✅ Do This:

test:
  needs: [build]
  script: make test

✅ Use needs for fast pipelines

deploy:
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

✅ Gate deployment by branch

job:
  tags:
    - docker

✅ Select the correct runner

❌ Don’t Do This:

# Rely on stage order for every job
<!-- Use needs to break the barrier -->              # ❌

# Hardcode secrets in .gitlab-ci.yml
<!-- Use protected/masked variables -->              # ❌

# Assume a runner is always available
<!-- Check the pipeline status for pending -->       # ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
Pipeline pending forever	No runner available	Register a runner or add tags
Job not running	`rules` excludes it	Check the condition
`needs` job not found	Wrong job name	Verify the job name in YAML
Variable empty	Not defined or protected	Check project/group settings
Pipeline not triggered	`.gitlab-ci.yml` not at root	Move to root

Real-World Examples

1. Basic Pipeline

stages:
  - build
  - test
build:
  stage: build
  script: make build
test:
  stage: test
  script: make test

2. Needs Pipeline

test:
  needs: [build]
  script: make test

3. Image

job:
  image: node:20
  script: npm test

4. Variable

variables:
  NODE_ENV: production

5. Rule

deploy:
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

6. Runner Tag

job:
  tags: [docker]

7. Predefined Variable

script:
  - echo $CI_COMMIT_SHA

8. Manual Trigger

job:
  when: manual

9. Schedule

CI/CD → Schedules → New schedule

10. Pipeline Graph

Build → Pipelines → click pipeline

Visual

Basic Pipeline (Stage Barrier)

┌──────────────────────────────────────────────┐
│  Stage: build                                │
│  ├── build_a                                 │
│  └── build_b                                 │
└──────────────────┬───────────────────────────┘
                   │ ALL must finish
                   ▼
┌──────────────────────────────────────────────┐
│  Stage: test                                 │
│  ├── test_a                                  │
│  └── test_b                                  │
└──────────────────┬───────────────────────────┘
                   │ ALL must finish
                   ▼
┌──────────────────────────────────────────────┐
│  Stage: deploy                               │
│  └── deploy                                  │
└──────────────────────────────────────────────┘

Needs Pipeline (Dependency Graph)

┌──────────────────────────────────────────────┐
│  build_a (fast)                              │
└──────────────────┬───────────────────────────┘
                   │ needs
                   ▼
┌──────────────────────────────────────────────┐
│  test_a                                      │
└──────────────────┬───────────────────────────┘
                   │ needs
                   ▼
┌──────────────────────────────────────────────┐
│  deploy_a                                    │
└──────────────────────────────────────────────┘

┌──────────────────────────────────────────────┐
│  build_b (slow)                              │
└──────────────────┬───────────────────────────┘
                   │ needs
                   ▼
┌──────────────────────────────────────────────┐
│  test_b                                      │
└──────────────────────────────────────────────┘

The two paths run in parallel. The fast path does not wait for the slow path.

Summary

Item	Value
Configuration file	`.gitlab-ci.yml` at repository root
Stage purpose	Execution order
Job purpose	Unit of work
Runner purpose	Executes jobs
`needs`	Explicit job dependency, bypasses stage order
`script`	Commands to run
`image`	Container environment
`rules`	Conditional execution
Predefined variables	`CI_COMMIT_SHA`, `CI_COMMIT_BRANCH`

Key takeaways:

A pipeline is stages and jobs. Stages define order. Jobs define work. The basic pipeline waits for every job in a stage to finish before starting the next stage .
needs breaks the stage barrier. A job with needs starts as soon as its declared dependencies finish, regardless of the stage. This is the primary optimization for pipeline speed .
Runners execute the jobs. Without a runner, the pipeline stays pending. The tags keyword selects a runner that matches the tag.
Variables customize jobs. They can be defined in YAML, project settings, or group settings. Predefined variables provide context about the pipeline and commit .
Rules control execution. The rules keyword determines which jobs run for which events. A deploy job can be gated to the main branch only.

Remember: The pipeline is the engine. The .gitlab-ci.yml is the fuel. The runner is the driver. The needs keyword is the accelerator. A basic pipeline is a train: it stops at every station. A needs pipeline is a graph: it goes where it needs to go, as fast as it can.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!