GitLab 17 🦊 Automated Testing, Code Quality, and Coverage Reports - KandZ

⏱️ Estimated read: 10 min | |

A pipeline that builds and deploys without testing is a pipeline that ships bugs. GitLab provides three distinct mechanisms for quality assurance: the test job itself (which runs the suite and fails the pipeline on failure), the code quality report (which surfaces static analysis findings in the merge request), and the coverage report (which displays the percentage of code exercised by tests). The exam tests the configuration of each, and the critical distinction between the two coverage mechanisms: the coverage keyword extracts a percentage from the job log using a regex, while the artifacts:reports:coverage_report keyword parses a Cobertura or JaCoCo XML file to annotate lines in the merge request diff. Configuring one does not enable the other.

Key point: The coverage keyword uses a regular expression to extract a percentage from the job log output. The artifacts:reports:coverage_report keyword uploads an XML file that GitLab parses for line-by-line annotations. The artifacts:reports:codequality keyword uploads a JSON file that GitLab parses for code quality findings. These are three separate configurations with three separate inputs and outputs .

Why automated testing and quality reports require separate configuration

The test job problem. A test job runs the suite. If the suite fails, the job fails, and the pipeline stops. The test job is the gate. It does not, by itself, produce a coverage percentage or a code quality report. Those are separate artifacts that the job must generate and upload .

The coverage percentage problem. The coverage keyword is a regular expression. GitLab scans the job log for the pattern, extracts the first matching number, and displays it in the merge request widget and the pipeline list. The regex must match the tool’s output format. A Jest report prints All files | 85.5 | ..., and the regex /All files[^|]*\|[^|]*\s+([\d\.]+)/ extracts 85.5 . A pytest report prints TOTAL 85%, and the regex /TOTAL.+ ([0-9]{1,3}%)/ extracts 85 . The regex is tool-specific.

The line-by-line coverage problem. The percentage is a single number. It does not tell the reviewer which lines are covered. The artifacts:reports:coverage_report keyword uploads an XML file in Cobertura or JaCoCo format. GitLab parses the file and annotates the lines in the merge request diff: green for covered, red for uncovered, orange for loaded but not executed . The annotations appear only on files changed in the merge request. A file not in the diff is not annotated, even if the report includes coverage data for it .

The code quality problem. Static analysis tools produce findings: unused variables, complexity violations, style issues. The artifacts:reports:codequality keyword uploads a JSON file that GitLab parses. The format is a subset of the CodeClimate specification. Each finding has a description, a check_name, a fingerprint, a location.path, a location.lines.begin, and a severity . The findings appear in the merge request widget as a comparison between the source branch and the target branch. If a finding is new, it appears as a degradation. If a finding was fixed, it appears as an improvement.

The trade-off. The test job is mandatory. The coverage percentage is optional but useful for tracking trends. The line-by-line coverage is optional but useful for reviewing the diff. The code quality report is optional but useful for catching issues before review. Each adds a configuration step and a parsing step. The exam tests the configuration and the distinction between the mechanisms.

a. Configuring the test job and coverage percentage

The test job runs the suite. The coverage keyword extracts the percentage from the log.

test:
  stage: test
  image: node:20
  script:
    - npm ci
    - npm test -- --coverage
  coverage: '/All files[^|]*\|[^|]*\s+([\d\.]+)/'
  artifacts:
    when: always
    reports:
      junit: junit.xml
    paths:
      - coverage/

The coverage keyword uses a regex. The artifacts:reports:junit uploads the JUnit XML file, which GitLab parses to display individual test results in the merge request. The paths uploads the coverage directory as a regular artifact for download .

The allow_failure setting is used for non-blocking tests. A job with allow_failure: true does not fail the pipeline if the test fails. This is used for experimental tests or for coverage analysis that should not block deployment .

coverage-analysis:
  stage: test
  script:
    - npm run coverage
  allow_failure: true
  coverage: '/All files[^|]*\|[^|]*\s+([\d\.]+)/'

The allow_failure job runs, produces the coverage percentage, and does not block the pipeline. The percentage is still recorded and displayed.

b. Configuring code quality reports

The code quality report is a JSON file. The tool must produce the JSON in the expected format. Many tools have a built-in formatter or a plugin. ESLint has a eslint-formatter-gitlab plugin. markdownlint-cli2 has a markdownlint-cli2-formatter-codequality formatter . The job runs the tool and uploads the report as an artifact.

codequality:
  stage: test
  image: node:20
  script:
    - npm ci
    - npx eslint . --format gitlab --output-file gl-code-quality-report.json
  artifacts:
    reports:
      codequality: gl-code-quality-report.json

The artifacts:reports:codequality keyword tells GitLab to parse the file. The file must be a JSON array. Each object must have the required fields. A UTF-8 byte order mark at the beginning of the file causes the parser to fail. Some Windows tools add it by default. The sed command can remove it .

The merge request widget shows the findings compared to the target branch. A new finding is a degradation. A fixed finding is an improvement. If the report has nothing to compare to (the first time the job runs), the widget is empty .

c. Configuring line-by-line coverage

The line-by-line coverage requires an XML file in Cobertura or JaCoCo format. The test tool must generate the file. Jest generates Cobertura with --coverageReporters=cobertura. pytest-cov generates Cobertura with --cov-report=xml. The job uploads the file as an artifact.

test:
  stage: test
  image: node:20
  script:
    - npm ci
    - npm test -- --coverage --coverageReporters=cobertura
  coverage: '/All files[^|]*\|[^|]*\s+([\d\.]+)/'
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

The coverage keyword extracts the percentage. The coverage_report artifact parses the XML for annotations. Both are configured on the same job. The percentage appears in the widget; the annotations appear in the diff .

The path resolution is automatic. GitLab matches the relative paths in the report to the absolute paths in the repository. If the match fails, the annotation is silently skipped. The Cobertura <sources> element is used to reconstruct the path. The automatic correction works when the <source> path follows the format <CI_BUILDS_DIR>/<PROJECT_FULL_PATH>/... .

Complete Example Session

# ============================================
# PART 1: TEST JOB WITH COVERAGE
# ============================================

test:
  stage: test
  image: node:20
  script:
    - npm ci
    - npm test -- --coverage
  coverage: '/All files[^|]*\|[^|]*\s+([\d\.]+)/'
  artifacts:
    when: always
    reports:
      junit: junit.xml
    paths:
      - coverage/

# ============================================
# PART 2: COVERAGE PERCENTAGE
# ============================================

# The coverage keyword extracts the percentage.
# It appears in the MR widget.
# The regex matches the "All files" line in the Jest output.

# ============================================
# PART 3: LINE-BY-LINE COVERAGE
# ============================================

test:
  stage: test
  script:
    - npm test -- --coverage --coverageReporters=cobertura
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

# ============================================
# PART 4: OBSERVE THE ANNOTATIONS
# ============================================

# Open the MR → Changes tab.
# Lines covered by tests are green.
# Lines not covered are red.

# ============================================
# PART 5: CODE QUALITY JOB
# ============================================

codequality:
  stage: test
  script:
    - npx eslint . --format gitlab --output-file gl-code-quality-report.json
  artifacts:
    reports:
      codequality: gl-code-quality-report.json

# ============================================
# PART 6: OBSERVE THE CODE QUALITY WIDGET
# ============================================

# Open the MR.
# The widget shows the number of new findings vs fixed findings.

# ============================================
# PART 7: ALLOW FAILURE FOR COVERAGE
# ============================================

coverage-analysis:
  stage: test
  script:
    - npm run coverage
  allow_failure: true
  coverage: '/All files[^|]*\|[^|]*\s+([\d\.]+)/'

# ============================================
# PART 8: COMBINE REPORTS IN ONE JOB
# ============================================

test:
  stage: test
  script:
    - npm test -- --coverage --coverageReporters=cobertura
  coverage: '/All files[^|]*\|[^|]*\s+([\d\.]+)/'
  artifacts:
    reports:
      junit: junit.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

# ============================================
# PART 9: VERIFY THE REPORTS
# ============================================

# MR widget: coverage percentage
# MR diff: line annotations
# MR widget: code quality findings

The nine parts trace the configuration: test job with coverage, coverage percentage extraction, line-by-line coverage, annotation observation, code quality job, widget observation, allow failure, combined reports, and verification.

Quick Reference

Coverage Mechanisms

Mechanism	Keyword	Input	Output
Coverage percentage	`coverage`	Regex on job log	Percentage in widget
Line annotations	`artifacts:reports:coverage_report`	Cobertura/JaCoCo XML	Green/red lines in diff

Code Quality

Keyword	Input	Output
`artifacts:reports:codequality`	JSON array	Findings in widget

Common Coverage Regex

Tool	Regex
Jest	`/All files[^
pytest-cov	`/TOTAL.+ ([0-9]{1,3}%)/`
go test	`/coverage: \d+.\d+% of statements/`
JaCoCo	`/Total.*?([0-9]{1,3})%/`

Code Quality Fields

Field	Required
`description`	Yes
`check_name`	Yes
`fingerprint`	Yes
`location.path`	Yes
`location.lines.begin`	Yes
`severity`	Yes

Best Practices

✅ Do This:

coverage: '/All files[^|]*\|[^|]*\s+([\d\.]+)/'
<!-- Use the correct regex for the tool -->           # ✅

artifacts:
  reports:
    coverage_report:
      coverage_format: cobertura
      path: coverage/cobertura-coverage.xml
<!-- Upload the XML for line annotations -->          # ✅

allow_failure: true
<!-- Use for non-blocking coverage analysis -->       # ✅

❌ Don’t Do This:

# Assume coverage keyword enables annotations
<!-- It only extracts a percentage -->               # ❌

# Upload a report with a UTF-8 BOM
<!-- The parser rejects it -->                       # ❌

# Configure multiple code quality jobs
<!-- Only the last job's report is used -->          # ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
Coverage percentage missing	Regex does not match	Test regex against job log
Line annotations missing	Path mismatch	Check Cobertura `<sources>`
Code quality widget empty	No target branch report	Run job on target branch first
Parser rejects report	UTF-8 BOM	Remove BOM with `sed`
Duplicate code quality reports	Multiple jobs	Use one job

Real-World Examples

1. Jest Coverage

coverage: '/All files[^|]*\|[^|]*\s+([\d\.]+)/'

2. Cobertura Report

artifacts:
  reports:
    coverage_report:
      coverage_format: cobertura
      path: coverage/cobertura-coverage.xml

3. ESLint Code Quality

script:
  - npx eslint . --format gitlab --output-file gl-code-quality-report.json
artifacts:
  reports:
    codequality: gl-code-quality-report.json

4. markdownlint Code Quality

script:
  - markdownlint-cli2 "**/*.md"
artifacts:
  reports:
    codequality: markdownlint-cli2-codequality.json

5. JUnit Report

artifacts:
  reports:
    junit: junit.xml

6. Allow Failure

allow_failure: true

7. Combined Reports

artifacts:
  reports:
    junit: junit.xml
    coverage_report:
      coverage_format: cobertura
      path: coverage/cobertura-coverage.xml

8. pytest Coverage

coverage: '/TOTAL.+ ([0-9]{1,3}%)/'

9. Go Coverage

coverage: '/coverage: \d+.\d+% of statements/'

10. BOM Removal

sed -i '1s/^\xEF\xBB\xBF//' report.json

Visual

Coverage Mechanisms

┌──────────────────────────────────────────────┐
│  coverage keyword                            │
│  ├── Regex on job log                        │
│  ├── Extracts percentage                     │
│  └── Displays in MR widget                   │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────┐
│  coverage_report artifact                    │
│  ├── Cobertura/JaCoCo XML                    │
│  ├── Parses line data                        │
│  └── Annotates MR diff                       │
└──────────────────────────────────────────────┘

Report Types

┌──────────────────────────────────────────────┐
│  junit                                       │
│  → Test results in MR                        │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────┐
│  codequality                                 │
│  → Static analysis findings in MR            │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────┐
│  coverage_report                             │
│  → Line annotations in MR diff               │
└──────────────────────────────────────────────┘

Summary

Item	Value
Coverage percentage	`coverage` keyword (regex)
Line annotations	`artifacts:reports:coverage_report` (XML)
Code quality	`artifacts:reports:codequality` (JSON)
Test results	`artifacts:reports:junit` (XML)
Coverage formats	Cobertura, JaCoCo
Code quality format	CodeClimate subset
Non-blocking tests	`allow_failure: true`
Code quality BOM	Parser rejects it

Key takeaways:

The coverage keyword extracts a percentage from the job log. It uses a regex to match the tool’s output. The percentage appears in the merge request widget and the pipeline list. It does not produce line-by-line annotations .
The coverage_report artifact parses an XML file for annotations. It requires a Cobertura or JaCoCo XML file. The annotations appear only on files changed in the merge request. The path resolution is automatic but can fail silently if the paths do not match .
The codequality artifact parses a JSON file for findings. It requires a JSON array in the CodeClimate subset format. The findings appear in the merge request widget as a comparison between the source and target branches .
The allow_failure setting makes a job non-blocking. A job with allow_failure: true does not fail the pipeline. It is used for coverage analysis that should not prevent deployment .
The two coverage mechanisms are independent. Configuring coverage does not enable annotations. Configuring coverage_report does not enable the percentage. Both must be configured for both outputs .

Remember: The test job is the gate. The coverage percentage is the metric. The line annotations are the diff view. The code quality report is the static analysis. Each is a separate configuration with a separate input. The coverage keyword uses a regex. The coverage_report artifact uses XML. The codequality artifact uses JSON. The exam tests the distinction and the configuration.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!