GitLab 10 🦊 Stages, Jobs, and Execution Flow in GitLab CI/CD - KandZ

⏱️ Estimated read: 9 min | |

The previous chapter covered the syntax of the .gitlab-ci.yml file. This chapter focuses on the execution model: how GitLab translates the YAML into a pipeline, how the runner picks up jobs, and how the stages, jobs, and needs relationships produce the final execution graph. The exam tests the difference between a stage-bound pipeline and a Directed Acyclic Graph (DAG) pipeline using needs, the behavior of allow_failure, and the mechanics of parallel execution with a matrix. The critical distinction is between the pipeline’s visual representation (the stages left-to-right) and its actual execution order (determined by dependencies and parallelism).

Key point: Stages are execution barriers. Every job in a stage must finish before any job in the next stage starts. The needs keyword breaks the barrier by declaring direct dependencies. A job with needs: [] starts immediately. A job with a parallel:matrix spawns multiple identical jobs with different variables.

Why the execution flow matters

The stage barrier cost. In a standard pipeline, a slow job blocks the entire stage. If test_unit takes 5 minutes and test_integration takes 50 minutes, the deploy stage waits for 50 minutes, even if deploy only depends on test_unit. The needs keyword eliminates this wait. It tells GitLab that deploy requires test_unit, not the entire test stage. The deployment can proceed as soon as test_unit finishes, while test_integration continues in parallel .

The fan-out problem. A test suite that runs 50 integration tests sequentially takes 50 times longer than one that runs them in parallel. The parallel keyword allows a job to be spawned multiple times. The parallel: N syntax spawns N instances of the job. The parallel:matrix syntax spawns instances with different variable combinations, which is the mechanism for testing across multiple Node.js versions or multiple database backends in a single job definition .

The failure propagation problem. When a job fails, the pipeline stops by default. The subsequent stages do not run. The allow_failure: true setting tells GitLab to continue even if the job fails. This is used for non-blocking jobs like linting or optional security scans. The pipeline status remains “passed with warnings” rather than “failed” .

The artifact flow problem. Artifacts are passed from one job to another. Without needs or dependencies, a job downloads artifacts from all jobs in the previous stage. With needs, it downloads only the artifacts from the specified jobs. This reduces download time and avoids conflicts when multiple jobs produce files with the same name.

The trade-off. A stage-bound pipeline is simple to reason about and easy to visualize. A DAG pipeline is faster but harder to debug when a job fails unexpectedly. The correct choice depends on the size of the pipeline and the number of independent paths.

a. Stage-bound execution and failure behavior

In a stage-bound pipeline, the stages run in the order defined by the stages keyword. Every job in a stage runs in parallel, and the stage is complete only when all jobs finish. If a job fails and allow_failure is not set, the pipeline stops.

stages:
  - build
  - test
  - deploy

build:
  stage: build
  script: make build

test_unit:
  stage: test
  script: make test-unit

test_integration:
  stage: test
  script: make test-integration

deploy:
  stage: deploy
  script: make deploy

The test_unit and test_integration jobs run in parallel. The deploy job waits for both. If test_integration fails, the deploy job does not run, and the pipeline status is failed.

The allow_failure: true setting changes this behavior. The job is allowed to fail without blocking the pipeline.

lint:
  stage: test
  script: make lint
  allow_failure: true

If lint fails, the pipeline continues. The pipeline status becomes passed with warnings. The allow_failure setting is the mechanism for non-blocking checks.

b. DAG execution with `needs`

The needs keyword declares direct dependencies. A job with needs starts as soon as its dependencies finish, regardless of the stage. The needs: [] syntax makes the job start immediately.

stages:
  - build
  - test
  - deploy

build:
  stage: build
  script: make build

test_unit:
  stage: test
  needs: [build]
  script: make test-unit

test_integration:
  stage: test
  needs: [build]
  script: make test-integration

deploy:
  stage: deploy
  needs: [test_unit]
  script: make deploy

The deploy job depends on test_unit only. It starts as soon as test_unit finishes, even if test_integration is still running. The pipeline graph shows two parallel paths: build → test_unit → deploy and build → test_integration.

The needs keyword also controls artifact passing. Without it, a job downloads artifacts from all jobs in the previous stage. With needs, it downloads only the specified jobs’ artifacts.

test_unit:
  needs:
    - job: build
      artifacts: true

The artifacts: true is the default. The artifacts: false syntax is used when the job needs the dependency for ordering but not for the artifacts.

c. Parallel jobs and matrix execution

The parallel keyword spawns multiple instances of a job. The parallel: N syntax spawns N instances. The instances are identified by CI_NODE_INDEX and CI_NODE_TOTAL.

test:
  stage: test
  parallel: 4
  script:
    - echo "Instance $CI_NODE_INDEX of $CI_NODE_TOTAL"
    - ./run-tests.sh --shard $CI_NODE_INDEX

The parallel:matrix syntax spawns instances with different variable combinations. The matrix is a list of variable maps. Each map is a combination.

test:
  stage: test
  parallel:
    matrix:
      - NODE_VERSION: ["18", "20", "22"]
        DATABASE: ["postgres", "mysql"]
  script:
    - echo "Node $NODE_VERSION with $DATABASE"
    - ./run-tests.sh

This spawns six jobs: three Node.js versions times two databases. Each job receives its own value of NODE_VERSION and DATABASE. The matrix is the mechanism for testing multiple environments without duplicating the job definition .

Complete Example Session

# ============================================
# PART 1: STAGE-BOUND PIPELINE
# ============================================

cat > .gitlab-ci.yml <<'EOF'
stages:
  - build
  - test
  - deploy

build:
  stage: build
  script: echo "Building"

test_unit:
  stage: test
  script: echo "Unit tests"

test_integration:
  stage: test
  script: echo "Integration tests"

deploy:
  stage: deploy
  script: echo "Deploying"
EOF

# ============================================
# PART 2: OBSERVE THE PIPELINE
# ============================================

# Build → Pipelines
# build → (test_unit, test_integration) → deploy
# Deploy waits for both test jobs.

# ============================================
# PART 3: CONVERT TO DAG
# ============================================

cat > .gitlab-ci.yml <<'EOF'
stages:
  - build
  - test
  - deploy

build:
  stage: build
  script: echo "Building"

test_unit:
  stage: test
  needs: [build]
  script: echo "Unit tests"

test_integration:
  stage: test
  needs: [build]
  script: echo "Integration tests"

deploy:
  stage: deploy
  needs: [test_unit]
  script: echo "Deploying"
EOF

# ============================================
# PART 4: OBSERVE THE NEW GRAPH
# ============================================

# build → test_unit → deploy
# build → test_integration
# deploy starts when test_unit finishes.

# ============================================
# PART 5: ADD ALLOW_FAILURE
# ============================================

lint:
  stage: test
  script: echo "Linting"
  allow_failure: true

# ============================================
# PART 6: ADD PARALLEL
# ============================================

test_sharded:
  stage: test
  parallel: 4
  script:
    - echo "Shard $CI_NODE_INDEX of $CI_NODE_TOTAL"

# ============================================
# PART 7: ADD MATRIX
# ============================================

test_matrix:
  stage: test
  parallel:
    matrix:
      - NODE_VERSION: ["18", "20"]
        DATABASE: ["postgres", "mysql"]
  script:
    - echo "Node $NODE_VERSION, DB $DATABASE"

# ============================================
# PART 8: VERIFY IN THE PIPELINE EDITOR
# ============================================

# Build → Pipeline editor → Visualize
# The graph shows parallel matrix jobs.

# ============================================
# PART 9: CHECK FAILURE BEHAVIOR
# ============================================

# If a matrix job fails, the pipeline fails.
# If lint fails, the pipeline passes with warnings.

The nine parts trace the execution flow: stage-bound pipeline, observation, DAG conversion, graph verification, allow_failure, parallel sharding, matrix expansion, editor visualization, and failure behavior.

Quick Reference

Execution Keywords

Keyword	Effect
`stages`	Define order
`stage`	Assign job to stage
`needs`	Explicit dependency
`needs: []`	Start immediately
`allow_failure`	Continue on failure
`parallel: N`	Spawn N instances
`parallel:matrix`	Spawn variable combinations

Parallel Variables

Variable	Value
`CI_NODE_INDEX`	Instance number (1-based)
`CI_NODE_TOTAL`	Total instances

Failure States

State	Meaning
`passed`	All jobs passed
`failed`	At least one job failed
`passed with warnings`	`allow_failure` job failed
`manual`	Blocked on manual action
`canceled`	Pipeline canceled

Best Practices

✅ Do This:

deploy:
  needs: [test_unit]

✅ Use needs for fast deployment

lint:
  allow_failure: true

✅ Use allow_failure for non-blocking checks

parallel:
  matrix:
    - NODE_VERSION: ["18", "20"]

✅ Use matrix for multi-environment testing

❌ Don’t Do This:

# Rely on stage order for deployment
<!-- Use needs for faster feedback -->               # ❌

# Set allow_failure: true on critical jobs
<!-- The pipeline would pass with broken code -->    # ❌

# Duplicate job definitions for each matrix variation
<!-- Use parallel:matrix instead -->                 # ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
Deploy waits unnecessarily	Stage barrier	Use `needs`
Pipeline passes with broken code	`allow_failure` on critical job	Remove or restrict
Matrix job variables empty	Matrix syntax error	Verify indentation
Needs job not found	Typo in job name	Check the YAML
Artifacts not passed	`needs` without `artifacts`	Use `needs: [{ job, artifacts }]`

Real-World Examples

1. Stage-Bound Pipeline

stages:
  - build
  - test
deploy:
  stage: deploy
  script: make deploy

2. DAG Pipeline

deploy:
  needs: [test_unit]

3. No Dependencies

job:
  needs: []

4. Allow Failure

lint:
  allow_failure: true

5. Parallel Instances

test:
  parallel: 4

6. Matrix

test:
  parallel:
    matrix:
      - NODE: ["18", "20"]

7. Artifact Passing

test:
  needs:
    - job: build
      artifacts: true

8. Disable Artifact Passing

test:
  needs:
    - job: build
      artifacts: false

9. Node Index

script:
  - echo $CI_NODE_INDEX of $CI_NODE_TOTAL

10. Check Pipeline Status

Build → Pipelines → click pipeline

Visual

Stage-Bound Pipeline

┌──────────────────────────────────────────────┐
│  build                                       │
└──────────────────┬───────────────────────────┘
                   │ all
                   ▼
┌──────────────────────────────────────────────┐
│  test_unit | test_integration (parallel)     │
└──────────────────┬───────────────────────────┘
                   │ all
                   ▼
┌──────────────────────────────────────────────┐
│  deploy                                      │
└──────────────────────────────────────────────┘

DAG Pipeline

┌──────────────────────────────────────────────┐
│  build                                       │
└──────────────────┬───────────────────────────┘
                   │ needs
        ┌──────────┴──────────┐
        │                     │
        ▼                     ▼
┌───────────────┐     ┌───────────────┐
│  test_unit    │     │test_integration│
└───────┬───────┘     └───────────────┘
        │ needs
        ▼
┌───────────────┐
│  deploy       │
│  (does not    │
│   wait for    │
│   integration)│
└───────────────┘

Matrix Expansion

┌──────────────────────────────────────────────┐
│  parallel:matrix                             │
│  NODE_VERSION: [18, 20]                      │
│  DATABASE: [postgres, mysql]                 │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────┐
│  Four jobs spawned:                          │
│  ├── Node 18 + postgres                      │
│  ├── Node 18 + mysql                         │
│  ├── Node 20 + postgres                      │
│  └── Node 20 + mysql                         │
└──────────────────────────────────────────────┘

Summary

Item	Value
Stage barrier	All jobs in a stage must finish
`needs`	Break the barrier with dependencies
`needs: []`	Start immediately
`allow_failure`	Continue on failure
`parallel: N`	Spawn N instances
`parallel:matrix`	Spawn variable combinations
Instance variables	`CI_NODE_INDEX`, `CI_NODE_TOTAL`
Pipeline states	passed, failed, passed with warnings

Key takeaways:

Stages are barriers. The basic pipeline waits for every job in a stage before starting the next. The needs keyword breaks the barrier by declaring direct dependencies .
needs is the optimization. A job with needs starts as soon as its dependencies finish, regardless of the stage. The needs: [] syntax starts the job immediately.
allow_failure is for non-blocking checks. A job with allow_failure: true does not fail the pipeline. The pipeline status becomes passed with warnings.
parallel is for speed. The parallel: N syntax spawns N instances. The parallel:matrix syntax spawns instances with different variable combinations, enabling multi-environment testing in a single job definition .
The pipeline editor is the validation tool. The graph view shows the actual execution order, not the stage order. Use it to verify that needs is doing what is intended.

Remember: The pipeline is a graph, not a list. The stages define the default order, but the needs keyword defines the actual order. The parallel keyword multiplies the jobs. The allow_failure keyword changes the failure behavior. The pipeline editor visualizes the graph. The YAML is the source. The graph is the execution.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!