GitLab 12 🦊 Using needs for Directed Acyclic Graph (DAG) Pipelines - KandZ

⏱️ Estimated read: 9 min | |

The previous chapters established the stage-bound pipeline and the basic needs syntax. This chapter focuses on the DAG model as a whole: how GitLab builds the execution graph, how needs interacts with parallel:matrix, how the pipeline visualization reflects the actual dependencies, and how to debug a pipeline that fails to start because of a cycle or a missing dependency. The exam tests the syntax of needs, the effect on artifact downloads, and the behavior of the pipeline when a dependency is skipped due to rules or when: manual. The critical distinction is between a job that depends on a job that will run (guaranteed to start) and a job that depends on a job that might be skipped (the pipeline fails with “needs job X not found”).

Key point: needs declares explicit dependencies. A job starts as soon as all its needs dependencies finish, regardless of the stage. The needs:project and needs:pipeline syntax can pull artifacts from another pipeline. The needs:parallel:matrix syntax declares a dependency on a specific matrix instance. The dependency graph must be acyclic; a cycle causes the pipeline to fail before any job runs.

Why DAG pipelines require explicit dependency management

The stage barrier cost. In a stage-bound pipeline, the deploy stage waits for every job in the test stage, even if the deploy only needs the results of one test job. In a large pipeline, this wait can dominate the total duration. The needs keyword eliminates the barrier by declaring the specific dependency. The deploy job starts as soon as its dependency finishes, in parallel with the unrelated test jobs .

The artifact download problem. Without needs, a job downloads artifacts from every job in the previous stages. With needs, it downloads only the artifacts from the declared dependencies. This reduces download time and avoids file name collisions when multiple jobs produce files with the same name . The artifacts: false setting downloads nothing from the dependency.

The skipped dependency problem. A job with needs: [build] assumes that build will run. If build is skipped because of a rules condition or when: never, the dependent job fails with “needs job build not found.” The optional: true setting allows the dependency to be skipped without failing the pipeline, which is useful when the dependency is conditional .

The cycle problem. A DAG is a directed acyclic graph. A cycle—job_a needs job_b and job_b needs job_a—makes the pipeline impossible to execute. GitLab detects the cycle and refuses to create the pipeline. The error message identifies the cycle, and the fix is to remove one of the needs declarations.

The trade-off. A DAG pipeline is faster but harder to visualize. The needs keyword replaces the implicit stage order with explicit dependencies. The pipeline graph shows the actual execution order, not the stage order. For large pipelines, the graph is the only way to understand the flow.

a. The basic `needs` syntax and the pipeline graph

The needs keyword accepts a list of job names. The job starts when all the listed jobs finish, regardless of the stage.

stages:
  - build
  - test
  - deploy

build:
  stage: build
  script: make build
  artifacts:
    paths:
      - dist/

test_unit:
  stage: test
  needs: [build]
  script: make test-unit

test_integration:
  stage: test
  needs: [build]
  script: make test-integration

deploy:
  stage: deploy
  needs: [test_unit]
  script: ./deploy.sh dist/

The deploy job depends on test_unit, not on the entire test stage. It starts as soon as test_unit finishes, even if test_integration is still running. The pipeline graph shows two parallel paths: build → test_unit → deploy and build → test_integration.

The needs: [] syntax makes a job start immediately, with no dependencies. This is used for jobs that do not depend on any other job, such as a linting job that checks the source code directly.

lint:
  stage: test
  needs: []
  script: make lint

The needs keyword is not compatible with dependencies. If both are specified, needs takes precedence and dependencies is ignored.

b. Optional dependencies, cross-project dependencies, and parallel matrix

The optional: true setting allows a dependency to be skipped without failing the pipeline. This is useful when the dependency has a rules condition that may exclude it from some pipelines.

test:
  needs:
    - job: build
      optional: true
  script: make test

If build is skipped, the test job still runs. Without optional: true, the pipeline fails with “needs job build not found.”

The needs:project syntax pulls artifacts from a job in another project. The syntax is project, job, ref, and optionally artifacts. This is the mechanism for cross-project pipelines.

test:
  needs:
    - project: my-group/shared-lib
      job: build
      ref: main
      artifacts: true

The needs:pipeline syntax pulls artifacts from a specific pipeline ID. The needs:pipeline:job syntax pulls artifacts from a specific job in a specific pipeline.

test:
  needs:
    - pipeline: $PARENT_PIPELINE_ID
      job: build

The needs:parallel:matrix syntax declares a dependency on a specific instance of a parallel job. The matrix section specifies the variable values that identify the instance.

test:
  needs:
    - job: build
      parallel:
        matrix:
          - NODE_VERSION: "20"
            DATABASE: "postgres"

This test job depends on the build instance with NODE_VERSION=20 and DATABASE=postgres. The other instances of build do not affect it.

c. Debugging the DAG: cycles, skipped jobs, and the graph view

The pipeline editor in GitLab visualizes the DAG. The “Visualize” tab shows the jobs and their dependencies as a graph. The graph shows which jobs run in parallel and which wait for others. The “Lint” tab validates the syntax and the dependency graph before the commit.

A cycle in the needs declarations prevents the pipeline from being created. The error message identifies the cycle.

Error: The pipeline contains a cycle:
  job_a needs job_b
  job_b needs job_a

The fix is to remove one of the needs declarations. A DAG cannot contain a cycle.

A skipped dependency causes the dependent job to fail unless optional: true is set. The error message is “needs job build not found.” The optional: true setting is the correct fix when the dependency is conditional.

The needs keyword also affects the pipeline status. If a job with needs fails, the pipeline fails, but the jobs that do not depend on it continue. The pipeline status becomes “failed” only when all jobs finish. The graph shows the failed job and the jobs that were blocked by it.

Complete Example Session

# ============================================
# PART 1: BASIC DAG PIPELINE
# ============================================

stages:
  - build
  - test
  - deploy

build:
  stage: build
  script: make build
  artifacts:
    paths:
      - dist/

test_unit:
  stage: test
  needs: [build]
  script: make test-unit

test_integration:
  stage: test
  needs: [build]
  script: make test-integration

deploy:
  stage: deploy
  needs: [test_unit]
  script: ./deploy.sh

# ============================================
# PART 2: OBSERVE THE GRAPH
# ============================================

# Build → Pipelines → click the pipeline
# The graph shows:
# build → test_unit → deploy
# build → test_integration

# ============================================
# PART 3: ADD AN OPTIONAL DEPENDENCY
# ============================================

test:
  needs:
    - job: build
      optional: true
  script: make test

# ============================================
# PART 4: ADD A CROSS-PROJECT DEPENDENCY
# ============================================

test:
  needs:
    - project: my-group/shared-lib
      job: build
      ref: main

# ============================================
# PART 5: ADD A PARALLEL MATRIX DEPENDENCY
# ============================================

test:
  needs:
    - job: build
      parallel:
        matrix:
          - NODE_VERSION: "20"
            DATABASE: "postgres"

# ============================================
# PART 6: CAUSE A CYCLE
# ============================================

job_a:
  needs: [job_b]
  script: echo "a"

job_b:
  needs: [job_a]
  script: echo "b"

# ============================================
# PART 7: OBSERVE THE CYCLE ERROR
# ============================================

# Pipeline editor → Lint
# Error: The pipeline contains a cycle.

# ============================================
# PART 8: FIX THE CYCLE
# ============================================

job_a:
  needs: []
  script: echo "a"

job_b:
  needs: [job_a]
  script: echo "b"

# ============================================
# PART 9: VERIFY THE GRAPH
# ============================================

# Build → Pipeline editor → Visualize
# The graph shows job_a → job_b.

The nine parts trace the DAG workflow: basic pipeline, graph observation, optional dependency, cross-project dependency, parallel matrix dependency, cycle introduction, cycle error, cycle fix, and graph verification.

Quick Reference

`needs` Syntax

Syntax	Purpose
`needs: [job_a]`	Depend on job_a
`needs: []`	No dependencies
`needs: [{ job: job_a, optional: true }]`	Optional dependency
`needs: [{ project, job, ref }]`	Cross-project
`needs: [{ pipeline, job }]`	Specific pipeline
`needs: [{ job, parallel: { matrix } }]`	Matrix instance

`needs` vs `dependencies`

Aspect	`needs`	`dependencies`
Execution order	Starts when deps finish	Respects stages
Artifact download	From specified jobs	From specified jobs
Default download	Only `needs` jobs	All previous stage jobs
DAG support	Yes	No

Pipeline States

State	Meaning
`created`	Pipeline created but not started
`pending`	Waiting for runner
`running`	At least one job running
`passed`	All jobs passed
`failed`	At least one job failed
`blocked`	Waiting for manual action

Best Practices

✅ Do This:

deploy:
  needs: [test_unit]

✅ Use needs for fast deployment

needs:
  - job: build
    optional: true

✅ Use optional for conditional dependencies

needs:
  - job: build
    parallel:
      matrix:
        - NODE_VERSION: "20"

✅ Depend on a specific matrix instance

❌ Don’t Do This:

# Create a cycle
<!-- job_a needs job_b, job_b needs job_a -->        # ❌

# Use needs without optional for conditional jobs
<!-- Pipeline fails if the dependency is skipped -->  # ❌

# Combine needs and dependencies
<!-- needs takes precedence; dependencies is ignored --> # ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
Pipeline fails with “needs job not found”	Dependency skipped	Add `optional: true`
Cycle error	Mutual dependencies	Remove one `needs`
Artifacts not downloaded	`artifacts: false`	Set to `true`
Graph does not match expected order	`needs` overrides stages	Verify the graph in the editor
Cross-project dependency fails	No access to project	Verify project and ref

Real-World Examples

1. Basic `needs`

deploy:
  needs: [build]

2. No Dependencies

lint:
  needs: []

3. Optional Dependency

needs:
  - job: build
    optional: true

4. Cross-Project

needs:
  - project: group/shared
    job: build
    ref: main

5. Matrix Dependency

needs:
  - job: build
    parallel:
      matrix:
        - NODE: ["20"]

6. Artifacts from Dependency

needs:
  - job: build
    artifacts: true

7. No Artifacts

needs:
  - job: build
    artifacts: false

8. Pipeline ID

needs:
  - pipeline: $PARENT_PIPELINE_ID
    job: build

9. Fix Cycle

job_a:
  needs: []

10. Verify Graph

Build → Pipeline editor → Visualize

Visual

DAG Pipeline

┌──────────────────────────────────────────────┐
│  build                                       │
└──────────────────┬───────────────────────────┘
                   │
        ┌──────────┴──────────┐
        │                     │
        ▼                     ▼
┌───────────────┐     ┌───────────────┐
│  test_unit    │     │test_integration│
└───────┬───────┘     └───────────────┘
        │ needs
        ▼
┌───────────────┐
│  deploy       │
└───────────────┘

Cycle Error

┌──────────────────────────────────────────────┐
│  job_a ── needs ──▶ job_b                    │
└──────────────────┬───────────────────────────┘
                   │
                   │ needs
                   ▼
┌──────────────────────────────────────────────┐
│  job_b ── needs ──▶ job_a                    │
│  (cycle, pipeline fails)                     │
└──────────────────────────────────────────────┘

Summary

Item	Value
`needs` purpose	Explicit job dependency
`needs: []`	No dependencies
`optional: true`	Allow skipped dependency
Cross-project	`needs:project`
Matrix instance	`needs:parallel:matrix`
Cycle	Pipeline fails
Graph view	Pipeline editor → Visualize

Key takeaways:

needs declares explicit dependencies. A job starts when its dependencies finish, regardless of the stage. This breaks the stage barrier and reduces pipeline duration .
needs controls artifact downloads. A job with needs downloads only the artifacts from its declared dependencies. The artifacts: false setting disables the download .
optional: true handles conditional dependencies. A job that depends on a job that may be skipped fails unless the dependency is marked optional .
The pipeline must be acyclic. A cycle in the needs declarations prevents the pipeline from being created. The error message identifies the cycle.
The graph view is the verification tool. The pipeline editor’s “Visualize” tab shows the actual execution order, not the stage order. Use it to confirm the DAG is correct.

Remember: needs is the DAG. The stages define the default order. The needs keyword defines the actual order. The graph view shows the result. The optional flag handles the conditional dependencies. The cycle is the error. The syntax is the tool.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!