Running in CI

In CI, one command scans, compares the results with the baseline, decides the gate and writes the files CI needs:

npx cora scan

Its exit code fails the job when the gate fails, when a target could not be scanned, or when an error stops it. The CLI reference lists each code, and how a script tells an unscanned target from an error.

Your account

cora scan, cora report merge and cora baseline update need your CORA account. In CI, they sign in with a CI token in CORA_TOKEN. Without one, they fail with account.missing.

1. Create a CI token on the CORA site, as CI tokens describes.
2. Add it to your CI system’s secrets as CORA_TOKEN.
3. Pass the secret to each job that runs cora, as the workflows below do.

Caution

Never commit a CI token, or write it into a config or a workflow file. Keep it in the CI system’s secrets.

Some runs get no secrets, such as a GitHub Actions run for a pull request from a fork. Skip the scan there with a notice, rather than fail the job. The GitHub Actions workflow does so with its if conditions, and the Azure Pipelines one with its conditions.

Serving the app

cora scan needs the app answering at its targets. The config’s server key starts the app’s server before the scan, waits until its URL answers, and stops it after. The app’s server shows how to set it.

The starter config cora init writes runs your dev script, which works in CI as it does on your machine:

• The job installs the project’s dependencies first, as the dev script needs them.
• Nothing else may answer at server.url, or the scan fails with server.already-running.
• A dev server that compiles a page on its first request can take longer than server.timeout on a slow runner: raise it there with CORA_SERVER_TIMEOUT.
• A job whose environment sets PORT moves a dev server that reads it, such as Next.js’s, Nuxt’s or Parcel’s, off the address in the config: set CORA_SERVER_URL and CORA_BASE_URL to the server’s address there.

To scan the production build in CI instead, as the workflows below do, set server to serve it and baseUrl to its address, since the crawl starts from baseUrl. Or set CORA_SERVER_COMMAND, CORA_SERVER_URL and CORA_BASE_URL in the job.

The workflows below build the app in a step of their own, and leave serving the build to server:

cora.config.json

{
  "baseUrl": "http://localhost:4173",
  "targets": ["/", "/orders"],
  "server": {
    "command": "npm run preview",
    "url": "http://localhost:4173"
  }
}

A step that starts the server in the background, as npm run preview &, races the scan instead. The scan can start before the server listens, meet a refused connection, and exit with code 2.

The build step sets CORA_SOURCE_STAMPS, so an app built with CORA’s build plugin carries source stamps. An Angular app carries its source locations in the builds whose configuration names the tsconfig that turns them on, whatever the variable says, so for a production build add the configuration Angular shows and build with ng build --configuration production,cora. An issue with a source location then names its file and line, in its annotation, its code scanning alert and the summary.

• sourceStamps set to true stamps every build instead, a production build’s too.
• A server.command that also builds, such as npm run build && npm run preview, needs no build step and stamps its build. server.timeout then bounds the build too.

The report directory

In CI, cora scan also writes the report directory, report/ in the state directory, so .cora/report/ by default, in the project or, outside one, in the folder the job runs cora in:

File	Holds
results.json	the results compared with the baseline, which cora baseline update also accepts
results.sarif	the results as SARIF 2.1.0, for GitHub code scanning and other SARIF viewers
summary.md	a Markdown summary: the gate, the new issues, which targets were not scanned and why, and which were scanned before their page settled

cora knows it runs in CI from the CI environment variable, which most CI systems set, or from a CI provider’s own variables. npx cora info shows the CI provider it detected. Detection never changes the verdict or the exit code, so the same command fails the same way on a laptop.

• --report <dir> writes the directory somewhere else, relative to the working directory, and outside CI writes it at all.
• The directory ignores itself in git, wherever it is.
• The summary’s headline counts the new issues at or above policy.failOn, and its New issues column every new issue. With failOn above minor, the headline can count fewer.

results.json is written last, and an error stops before it is written. Into a directory an earlier run left, cora first deletes only that run’s results.json. So a write that fails partway never leaves one beside a mix of both runs’ files.

Note

.cora/ is hidden, since its name starts with .. actions/upload-artifact skips hidden files unless its include-hidden-files is true, as the workflow below sets it. github/codeql-action/upload-sarif reads the file its sarif_file names, so it needs no such input.

GitHub Actions

Create the workflow at .github/workflows/accessibility.yml:

.github/workflows/accessibility.yml

name: Accessibility

on:
  push:
    branches: [main]
  pull_request:

jobs:
  accessibility:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      security-events: write # uploading SARIF
    env:
      CORA_TOKEN: ${{ secrets.CORA_TOKEN }} # empty in a run that gets no secrets
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 22
      - run: npm ci
      - run: npm run build
        env:
          CORA_SOURCE_STAMPS: 1 # stamp source locations, with CORA's build plugin registered
      - run: npx cora scan # the config's server key serves the build
        if: env.CORA_TOKEN != ''
      - run: echo "::notice::No CORA_TOKEN reached this run, so the accessibility scan was skipped."
        if: env.CORA_TOKEN == ''
      - uses: github/codeql-action/upload-sarif@v4
        if: ${{ !cancelled() && env.CORA_TOKEN != '' }}
        with:
          sarif_file: .cora/report/results.sarif
          category: accessibility
      - uses: actions/upload-artifact@v7
        if: ${{ !cancelled() && env.CORA_TOKEN != '' }}
        with:
          name: accessibility-report
          path: .cora/report/
          include-hidden-files: true # .cora/ is hidden, which the upload skips by default

• It runs on pushes to main and on pull requests. Code scanning then compares a pull request’s alerts with its base branch’s, and annotates the lines it changes.
• The upload steps run when the scan fails too, so a failed gate still reaches code scanning and the artifact. They skip only when the run is cancelled or got no token.
• A Dependabot run gets no Actions secrets. Add CORA_TOKEN to Dependabot’s secrets too, or the scan skips there.
• The Ubuntu runner image has Google Chrome, which cora scans with.
• Each action is named by its major version, so it takes that version’s fixes. A repository that pins its actions to commit SHAs, as GitHub’s security guidance recommends, pins each to the newest release of that major version, with the version in a comment.

Annotations and the job summary

Under GitHub Actions, cora scan and cora report merge also report to the run itself, with no step of their own:

• The job summary. They append summary.md to the step’s job summary, the file GITHUB_STEP_SUMMARY names. Its tables list issues by source file and position, then by page. An issue’s Where column gives its page, or, for an element of an app built with source stamps, its file, line and column instead, such as src/routes/orders/[id].tsx:42:9. Its Element column gives the element of an issue found on a page either way. An issue found in a source file itself has a file, line and column but no page or element, so its Element column is empty.
• Annotations, which GitHub shows on the run’s summary page and on a pull request’s changed files. The first gives the gate’s verdict. A notice then names any target scanned before its page settled. The rest are the issues the summary lists, in its order: new issues, issues that need review, issues of rules set to report, and issues in page states an agent reached, each kind by source file and position, then by page. Each title names the issue’s severity, its kind and its rule, such as CORA critical issue, new: axe/image-alt, and the message of an issue found on a page names the page and element.
• Source lines. An issue with a source location is annotated at that line in its source file, the element’s own tag or its call site’s, and an issue found in a source file at its line there.
• Levels. As in code scanning, a critical or serious violation is an error, and a moderate one a warning. A minor one is a notice, and so are an issue to review and an issue of a rule set to report.

GitHub shows at most 10 annotations of each level from a step, so cora keeps the first of each level, in the order above, and the verdict’s annotation says how many issues it left out. Every issue is still in results.json. results.sarif holds up to the 25,000 results GitHub code scanning accepts, the first by level, and code scanning shows them once the SARIF log is uploaded.

With --out -, stdout carries the results, so cora scan prints no annotations; it still appends the summary. Under any other CI provider, cora writes the report directory alone.

Code scanning

GitHub code scanning shows each issue as an alert. It closes an alert when a later upload no longer reports its issue.

• Issues the baseline accepts stay open alerts, while the gate fails only on new ones.
• An alert’s rule help links CORA’s guidance page for the rule where it has one, and otherwise its engine’s help page, as the rules reference does. A rule with neither has no link.
• An issue with a source location is located at that line in its source file: the element’s own tag, or its call site’s. So is an issue found in a source file itself. An accepted issue that the run did not verify is located at its source file, with no line, since the baseline stores none. Any other issue has no source file, so its alert is located at the repository root. The message of an issue found on a page names the page and element.
• For each issue at a source line, upload-sarif logs a warning: “Calculated fingerprint of … but found existing inconsistent fingerprint value …”. It is harmless. CORA gives each result its issue’s fingerprint, which the action keeps, so an alert stays one alert when the code around it moves.
• Whether code scanning is available depends on the repository’s visibility and GitHub plan, as GitHub’s docs set out.
• A project that uploads several cora logs, such as one per app, gives each its own category.

Azure Pipelines

cora detects Azure Pipelines by the variables it sets, such as SYSTEM_TEAMFOUNDATIONCOLLECTIONURI. Add CORA_TOKEN as a secret variable of the pipeline:

azure-pipelines.yml

trigger:
  - main

pool:
  vmImage: ubuntu-latest

steps:
  - task: UseNode@1
    inputs:
      version: '22.x'
  - script: npm ci
  - script: npm run build
    env:
      CORA_SOURCE_STAMPS: 1
  - script: npx cora scan # the config's server key serves the build
    condition: and(succeeded(), ne(variables['System.PullRequest.IsFork'], 'True'))
    env:
      CORA_TOKEN: $(CORA_TOKEN) # a secret variable reaches a script only when mapped
  - script: echo "##vso[task.logissue type=warning]A pull request from a fork gets no secrets, so the accessibility scan was skipped."
    condition: eq(variables['System.PullRequest.IsFork'], 'True')
  - publish: .cora/report
    artifact: accessibility
    condition: and(succeededOrFailed(), ne(variables['System.PullRequest.IsFork'], 'True'))

• Pull requests. In Azure Repos, a branch policy’s build validation runs the pipeline on pull requests (PR triggers). A GitHub repository takes a pr: trigger in the file.
• Forks. A build of a pull request from a fork gets no secrets, unless the pipeline’s Make secrets available to builds of forks setting is on. Azure Pipelines then leaves $(CORA_TOKEN) as written, so cora would get that text as its token. The pipeline skips the scan on such a pull request by System.PullRequest.IsFork instead, with a warning, since Azure Pipelines has no notice level.
• A pipeline that turns that setting on leaves out its System.PullRequest.IsFork checks.

Sharding

A large target list can be split across CI jobs. Each job scans its share and uploads its results file. One more job merges them, compares the merged results with the baseline, and decides the gate:

.github/workflows/accessibility.yml

jobs:
  scan:
    strategy:
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      # ...checkout, install and build as above; each scan serves the build
      - run: npx cora scan --shard ${{ matrix.shard }}/4 --out results-${{ matrix.shard }}.json
      - uses: actions/upload-artifact@v7
        with:
          name: results-${{ matrix.shard }}
          path: results-${{ matrix.shard }}.json
  report:
    needs: scan
    steps:
      # ...checkout and install, for the config and the baseline
      - uses: actions/download-artifact@v8
        with:
          pattern: results-*
          path: results
          merge-multiple: true
      - run: npx cora report merge results

• Every job that runs cora scan or cora report merge needs CORA_TOKEN, as in the GitHub Actions workflow.
• A sharded cora scan writes its results file and exits 0. Only the merge can compare the whole run with the baseline, so --baseline and --report belong on cora report merge, and are an error on a sharded scan.
• cora report merge reads every .json file directly in a directory, so merge-multiple: true puts the shards in one.
• Every shard must be merged: a missing or repeated shard stops the merge with exit code 2.
• The shards of one run may differ only in keys outside the config’s hash, which the config reference marks on each key as In configHash.
• cora report merge always writes the report directory: .cora/report/ in the project, or --report <dir>.

Combining with other runs

cora report merge also combines runs of different kinds into one gate, such as a scan beside another run’s results. Give such a scan --no-gate, so it writes its results file without failing the job first:

npx cora scan --no-gate --out cli.json
npx cora report merge cli.json other-results.json

Merge each run once. Runs of an older commit keep their findings, and cora warns when the runs it merges come from more than one commit.