GitLab 11 🦊 Caching and Artifacts: Passing Data Between Pipeline Jobs - KandZ

⏱️ Estimated read: 10 min | |

A GitLab pipeline is a sequence of isolated jobs. Each job runs in a fresh environment, often on a different runner, with a clean checkout of the repository. Files created in one job are not visible to another job unless they are explicitly passed. GitLab provides two mechanisms for this: cache and artifacts. They look similar in YAML, but they serve fundamentally different purposes. The exam tests this distinction. A cache is a best-effort optimization to speed up jobs by reusing downloaded dependencies. An artifact is a guaranteed data transfer from one job to another, with a defined expiry and a defined path. Confusing the two produces pipelines that either download everything unnecessarily or fail because the expected file is missing.

Key point: cache is for dependencies (node_modules, .m2, vendor). It is stored on the runner, scoped by a key, and not guaranteed to be available. artifacts is for build output (dist/, test reports, binaries). It is uploaded to GitLab, passed to downstream jobs by default, and available for download from the UI. The dependencies or needs keyword controls which artifacts a job downloads.

Why cache and artifacts exist separately

The dependency problem. A Node.js project runs npm ci in every job. Without caching, every job downloads the full dependency tree from the registry. The cache stores the downloaded packages (~/.npm or node_modules) and restores them in subsequent jobs with the same cache key. The key is usually the lockfile hash or the branch slug. If the cache is missing, the job still runs—it just takes longer. The cache is an optimization, not a dependency .

The build output problem. A build job produces a dist/ directory. The test job needs that directory to run integration tests. The deploy job needs it to push to production. The cache is the wrong mechanism here: it is keyed for dependency reuse, it is not guaranteed to be available, and it is not designed for passing a specific build to a specific downstream job. The artifacts keyword is the correct mechanism. It archives the dist/ directory after the build job succeeds, and makes it available to downstream jobs .

The default artifact download problem. By default, every job downloads artifacts from all jobs in previous stages. In a pipeline with ten jobs, the last job may download ten sets of artifacts, most of which it does not need. The dependencies: [] setting disables artifact downloads for a job that does not need them. The dependencies: [build-job] setting limits the download to a specific job. The needs keyword provides the same control in a DAG pipeline .

The cache key security problem. GitLab 15.0 introduced cache key suffixes. A cache key like $CI_COMMIT_REF_SLUG on a protected branch receives the -protected suffix, and on an unprotected branch receives the -non_protected suffix. This prevents an unprotected branch from poisoning the cache used by a protected branch. A user with Developer role on an unprotected branch cannot write to the protected cache. The suffix is automatic and is a security feature, not a bug .

The trade-off. Cache is faster when it works, but it is not guaranteed. Artifacts are guaranteed but consume storage and download time. A pipeline that uses artifacts for everything is slow and expensive. A pipeline that uses cache for build outputs is fragile. The correct pattern is: cache dependencies, pass build outputs as artifacts.

a. Configuring cache for dependencies

The cache keyword is defined at the global level or per job. The key determines which cache is restored. The paths lists the directories to cache. The policy controls whether the job pulls, pushes, or both.

# Global cache for Node.js dependencies
default:
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - .npm/
    policy: pull-push

install:
  script:
    - npm ci --cache .npm --prefer-offline

The key:files form computes the cache key from the hash of the listed files. When package-lock.json changes, the cache key changes, and the old cache is not restored. This is the correct behavior: new dependencies require a fresh download. The key:files accepts a maximum of two files .

The policy setting is important for performance. An install job that only writes the cache uses policy: push. A test job that only reads the cache uses policy: pull. The default is pull-push, which downloads the cache before the job and uploads it after. For jobs that do not modify the cache, pull is faster .

test:
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - .npm/
    policy: pull
  script:
    - npm ci --cache .npm --prefer-offline
    - npm test

To disable the cache entirely for a job, use an empty array: cache: [] .

b. Configuring artifacts for build output

The artifacts keyword archives files after a job succeeds. The paths lists the files or directories to archive. The expire_in sets the retention period. The default retention is 30 days, but the instance setting may override it.

build:
  stage: build
  script:
    - npm run build
  artifacts:
    paths:
      - dist/
    expire_in: 1 week

The exclude option removes specific files from the archive. The exclude is not recursive by default. To exclude a directory and its contents, use dist/tmp/**/* .

artifacts:
  paths:
    - dist/
  exclude:
    - dist/tmp/**/*

The when option controls when the artifacts are uploaded. The default is on_success. The on_failure value uploads artifacts when the job fails, which is useful for test reports and crash logs. The always value uploads regardless of the job status .

test:
  script:
    - npm test
  artifacts:
    when: always
    paths:
      - coverage/
    reports:
      junit: junit.xml

The reports section is a special artifact type. The junit report is parsed by GitLab and displayed in the merge request. The coverage report extracts the coverage percentage. The dotenv report passes environment variables to downstream jobs .

c. Controlling artifact downloads with dependencies and needs

By default, a job downloads artifacts from every job in the previous stages. The dependencies keyword limits the download to specific jobs. The needs keyword does the same in a DAG pipeline, and also starts the job earlier.

test:
  stage: test
  dependencies:
    - build
  script:
    - ./run-tests.sh dist/

The test job downloads only the artifacts from build. It does not download artifacts from any other job in the build stage. To disable artifact downloads entirely, use dependencies: [] .

lint:
  stage: test
  dependencies: []
  script:
    - ./lint.sh

The needs keyword is the DAG equivalent. A job with needs starts as soon as its dependencies finish, and by default downloads their artifacts.

deploy:
  stage: deploy
  needs:
    - job: build
      artifacts: true
  script:
    - ./deploy.sh dist/

The artifacts: true is the default. The artifacts: false setting downloads nothing from the dependency, which is useful when the job needs the ordering but not the files .

Complete Example Session

# ============================================
# PART 1: GLOBAL CACHE FOR DEPENDENCIES
# ============================================

default:
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - .npm/
    policy: pull-push

# ============================================
# PART 2: INSTALL JOB (CACHE PUSH)
# ============================================

install:
  stage: build
  script:
    - npm ci --cache .npm --prefer-offline
  cache:
    policy: push

# ============================================
# PART 3: BUILD JOB WITH ARTIFACTS
# ============================================

build:
  stage: build
  needs: [install]
  script:
    - npm run build
  artifacts:
    paths:
      - dist/
    expire_in: 1 week

# ============================================
# PART 4: TEST JOB (CACHE PULL, ARTIFACT DOWNLOAD)
# ============================================

test:
  stage: test
  needs: [build]
  cache:
    policy: pull
  script:
    - npm ci --cache .npm --prefer-offline
    - npm test

# ============================================
# PART 5: LINT JOB (NO ARTIFACTS NEEDED)
# ============================================

lint:
  stage: test
  dependencies: []
  script:
    - npm run lint

# ============================================
# PART 6: DEPLOY JOB (ARTIFACTS ONLY)
# ============================================

deploy:
  stage: deploy
  needs:
    - job: build
      artifacts: true
  script:
    - ./deploy.sh dist/

# ============================================
# PART 7: DOTENV REPORT
# ============================================

version:
  stage: build
  script:
    - echo "APP_VERSION=1.2.3" >> build.env
  artifacts:
    reports:
      dotenv: build.env

deploy:
  needs: [version]
  script:
    - echo "Deploying $APP_VERSION"

# ============================================
# PART 8: VERIFY ARTIFACT DOWNLOAD
# ============================================

# In the job log, search for "Downloading artifacts"
# The log shows which jobs' artifacts were downloaded.

# ============================================
# PART 9: CACHE KEY INVALIDATION
# ============================================

# When package-lock.json changes:
# - The cache key changes.
# - The old cache is not restored.
# - The install job downloads fresh dependencies.

The nine parts trace the pipeline data flow: global cache, install job, build with artifacts, test with cache pull and artifact download, lint with no artifacts, deploy with artifacts, dotenv report, log verification, and cache invalidation.

Quick Reference

Cache vs Artifacts

Aspect	Cache	Artifacts
Purpose	Speed up jobs	Pass data between jobs
Guaranteed	No	Yes
Storage	Runner	GitLab
Scope	Key-based	Pipeline
Default download	Restored by key	All previous stages
Expiry	No automatic	`expire_in`

Cache Keywords

Keyword	Purpose
`key`	Cache identifier
`key:files`	Hash of lockfiles
`paths`	Directories to cache
`policy`	`pull`, `push`, `pull-push`
`fallback_keys`	Backup keys

Artifact Keywords

Keyword	Purpose
`paths`	Files to archive
`exclude`	Files to omit
`expire_in`	Retention period
`when`	`on_success`, `on_failure`, `always`
`reports`	Special report types

Download Control

Keyword	Effect
`dependencies: []`	No artifacts
`dependencies: [job]`	Only from job
`needs: [job]`	Only from job (DAG)
`needs: [{ job, artifacts: false }]`	No artifacts from job

Best Practices

✅ Do This:

cache:
  key:
    files:
      - package-lock.json
<!-- Use lockfile hash as cache key -->                # ✅

artifacts:
  paths:
    - dist/
  expire_in: 1 week
<!-- Set expiry for artifacts -->                     # ✅

lint:
  dependencies: []
<!-- Disable artifacts when not needed -->            # ✅

❌ Don’t Do This:

# Use cache for build output
<!-- Cache is not guaranteed; use artifacts -->       # ❌

# Use artifacts for node_modules
<!-- Artifacts are slow and storage-heavy -->         # ❌

# Omit expire_in on large artifacts
<!-- Storage accumulates -->                          # ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
Artifact not found	Wrong job name in `dependencies`	Verify job name
Cache not restored	Key mismatch	Check `key:files`
Slow pipeline	All jobs download all artifacts	`dependencies: []`
Cache overwritten	Same key, different paths	Use `key:prefix`
Artifacts expire early	`expire_in` too short	Increase or remove

Real-World Examples

1. Node.js Cache

cache:
  key:
    files:
      - package-lock.json
  paths:
    - .npm/

2. Build Artifacts

artifacts:
  paths:
    - dist/
  expire_in: 1 week

3. No Artifacts

dependencies: []

4. Specific Artifacts

dependencies:
  - build

5. Needs with Artifacts

needs:
  - job: build
    artifacts: true

6. Needs without Artifacts

needs:
  - job: build
    artifacts: false

7. Cache Policy

cache:
  policy: pull

8. Dotenv Report

artifacts:
  reports:
    dotenv: build.env

9. JUnit Report

artifacts:
  reports:
    junit: junit.xml

10. Exclude Files

artifacts:
  exclude:
    - dist/tmp/**/*

Visual

Cache vs Artifacts

┌──────────────────────────────────────────────┐
│  CACHE                                       │
│  ├── Dependencies (node_modules, .npm)       │
│  ├── Stored on runner                        │
│  ├── Key-based                               │
│  └── Not guaranteed                          │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────┐
│  ARTIFACTS                                   │
│  ├── Build output (dist/, reports)           │
│  ├── Stored on GitLab                        │
│  ├── Pipeline-scoped                         │
│  └── Guaranteed                              │
└──────────────────────────────────────────────┘

Artifact Flow

┌──────────────────────────────────────────────┐
│  build job                                   │
│  Produces dist/                              │
│  artifacts: paths: [dist/]                   │
└──────────────────┬───────────────────────────┘
                   │
                   │ uploaded to GitLab
                   ▼
┌──────────────────────────────────────────────┐
│  test job                                    │
│  dependencies: [build]                       │
│  Downloads dist/                             │
└──────────────────┬───────────────────────────┘
                   │
                   │
                   ▼
┌──────────────────────────────────────────────┐
│  deploy job                                  │
│  needs: [{ job: build, artifacts: true }]    │
│  Downloads dist/                             │
└──────────────────────────────────────────────┘

Summary

Item	Value
Cache purpose	Speed up dependency install
Artifact purpose	Pass build output between jobs
Cache key	`key:files` or `$CI_COMMIT_REF_SLUG`
Cache policy	`pull`, `push`, `pull-push`
Artifact expiry	`expire_in` (default 30 days)
Artifact download control	`dependencies` or `needs`
Disable artifacts	`dependencies: []`
Dotenv report	`reports: dotenv`

Key takeaways:

Cache is for dependencies, artifacts are for output. The cache speeds up npm ci or bundle install. The artifacts pass dist/ to the next job. Using cache for build output is fragile; using artifacts for dependencies is slow .
The cache key determines invalidation. The key:files form hashes the lockfile. When the lockfile changes, the cache key changes, and the old cache is not restored. This is the correct behavior .
Artifacts are downloaded by default. Every job downloads artifacts from all previous stages unless dependencies or needs is used to limit the download. The dependencies: [] setting is the most effective optimization for jobs that do not need files .
The needs keyword is the DAG equivalent of dependencies. It limits artifact downloads and starts the job earlier. The artifacts: false setting downloads nothing from the dependency.
Cache key suffixes are a security feature. Protected branches and Maintainer-role pipelines use a -protected suffix. Unprotected branches use -non_protected. This prevents cache poisoning .

Remember: The pipeline is a sequence of isolated jobs. The cache is the optimization that makes dependency installation fast. The artifacts are the contract that moves build output from one job to the next. The dependencies or needs keyword controls which artifacts are downloaded. The expire_in controls how long they are kept. The exam tests the distinction and the download control.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!