Skip to content

Latest commit

 

History

History
328 lines (258 loc) · 9.48 KB

results-api.mdx

File metadata and controls

328 lines (258 loc) · 9.48 KB
titledescriptionsidebar_labelsidebar_position
API for test results | Cypress UI Coverage
Programmatically fetch UI Coverage results in a CI environment and fail the build if the test coverage is not acceptable.
Results API
100

Results API

The @cypress/extract-cloud-results module provides the getUICoverageResults utility to programmatically fetch UI Coverage results for a run in a CI environment. This allows you to determine if test coverage meets your requirements before merging code changes.

Supported CI Providers

The utility supports the following CI providers. Refer to the linked guides for setup details:

For other CI providers, contact Cypress Support to request support.

Installation

Install the @cypress/extract-cloud-results module in your install step in CI.

npm install --force https://cdn.cypress.io/extract-cloud-results/v1/extract-cloud-results.tgz

:::caution

Do not check this module in as a dependency. We recommend you install it separately outside of your normal module installation. Use --force to get the latest version.

If you check this in as a dependency, your installation will fail when we update the package.

:::

Usage

1. Get the UI Coverage Results

Write a script to fetch UI Coverage results and assert test coverage criteria. This script will be executed in CI.

const{ getUICoverageResults }=require('@cypress/extract-cloud-results')getUICoverageResults({projectId: process.env.CYPRESS_PROJECT_ID,// Optional if set from envrecordKey: process.env.CYPRESS_RECORD_KEY,// Optional if set from envrunTags: [process.env.RUN_TAGS],// Required if recording multiple runs}).then((results)=>{const{ runNumber, uiCoverageReportUrl, summary, views }=resultsconsole.log(`Received ${summary.isPartialReport ? 'partial' : ''} results for run #${runNumber}.`)console.log(`See full report at ${uiCoverageReportUrl}.`)// Verify overall coverageif(summary.coverage<80){thrownewError(`Project coverage is ${summary.coverage}, below the minimum threshold of 80%.`)}constcriticalViews=[/login/,/checkout/]// Verify critical view coverageviews.forEach((view)=>{const{ displayName, coverage, uiCoverageReportUrl }=viewif(criticalViews.some((pattern)=>pattern.test(displayName))&&coverage<95){thrownewError(`Critical view "${displayName}" coverage is ${coverage}%, below the required 95%. See: ${uiCoverageReportUrl}`)}})console.log('UI Coverage is above minimum thresholds.')})

getUICoverageResults arguments

getUICoverageResults accpets the following arguments:

getUICoverageResults({// The Cypress project ID.// Optional if the CYPRESS_PROJECT_ID env is setprojectId: string// The project's record key.// Optional if the CYPRESS_RECORD_KEY env is setrecordKey: string// The run tags associated with the run.// Required IF you are recording multiple Cypress runs from a single CI build.// Pass the run tags you used when recording in each runrunTags: string[]})

Result Details

The getUICoverageResults utility returns the following data:

{// The run number of the identified build. runNumber: number// The run url for the identified build. runUrl: 'https://cloud.cypress.io/projects/:project_id/runs/:run_number'// The status of the identified build. runStatus: 'passed'|'failed'|'errored'|'timedOut'|'cancelled'|'noTests'// The url that links to UI Coverage report for the identified build. uiCoverageReportUrl: 'https://cloud.cypress.io/[...]' summary: {// Indicates whether a complete UI Coverage report was generated.// For example, if a run was cancelled and the report expected to run// for 20 specs, but only 10 ran, this would result in a partial report. isPartialReport: boolean// The report coverage from 0-100 with 2 decimal precision (e.g 92.45). coverage: number// The number of views tested and analyzed. viewCount: number// The number of interactive elements that were tested. testedElementsCount:number// The number of interactive elements that were not tested. untestedElementsCount: number}// The list of tested views and the coverage of each page. views: [{// The sanatized URL pattern shown in the report.displayName: string// The view coverage from 0-100 with 2 decimal precision (e.g 92.45).coverage: number// The number of interactive elements that were tested on this view.testedElementsCount:number// The number of interactive elements that were not tested on this view.untestedElementsCount: number// The url that links the report for this view.uiCoverageReportUrl: 'https://cloud.cypress.io/[...]'}]}

2. Add to CI Workflow

In your CI workflow that runs your Cypress tests,

  1. Update your install job to install the @cypress/extract-cloud-results module.
  2. Pass in the necessary arguments to getUICoverageResults.
  3. Add a new step to the job that runs your Cypress tests to verify the UI Coverage results.

:::info

If you record multiple runs in a single CI build, you must record these runs using the --tag parameter and then call getUICoverageResults with the runTags argument for each run.

This is necessary to identify each unique run and return a corresponding set of results. The tags are how each run is uniquely identified.

Example

  • Let's imagine that within a single CI build you call cypress run --record multiple times because you're running one set of tests against a staging environment, followed by a production environment.
  • In this scenario, you pass a different --tag to each cypress run
    • cypress run --record --tag staging
    • cypress run --record --tag production
  • When calling getUICoverageResults you would then pass these same tags to get the unique set of results for each run
    • getUICoverageResults({ runTags: ['staging']})
    • getUICoverageResults({ runTags: ['production']})

:::

Example Job Workflow Update:

name: My Workflow on: push env: CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }} jobs: run-cypress: runs-on: ubuntu-24.04 steps: - name: Checkout uses: actions/checkout@v4 - name: install run: npm install - name: Run run: npx cypress run --record + - name: Verify UI Coverage Results+ run: |+ npm install --force https://cdn.cypress.io/extract-cloud-results/v1/extract-cloud-results.tgz+ node ./scripts/verifyUICoverageResults.js
name: Run Cypress Tests image: node:latest stages: - test run-cypress: stage: test secrets: CYPRESS_RECORD_KEY: vault: vault/cypressRecordKey script: - npm install - npx cypress run --record + - npm install --force https://cdn.cypress.io/extract-cloud-results/v1/extract-cloud-results.tgz+ - node ./scripts/verifyUICoverageResults.js
pipeline { agent { docker { image 'cypress/base:22.12.0' } } environment { CYPRESS_PROJECT_ID: 'xxxx' CYPRESS_RECORD_KEY = credentials('cypress-record-key') } stages { stage('build and test') { steps { sh 'npm ci' sh 'npx cypress run --record' } } + stage('Verify UI Coverage Results') {+ steps {+ sh 'npm install --force https://cdn.cypress.io/extract-cloud-results/v1/extract-cloud-results.tgz'+ sh 'node ./scripts/verifyUICoverageResults.js'+ }+ } } }
jobs: - job: run_tests pool: vmImage: 'ubuntu-latest' steps: - task: NodeTool@0 inputs: versionSpec: '20.x' displayName: 'Install Node.js' - script: npm i displayName: 'Install npm dependencies' - script: npx cypress run --record displayName: 'Run Cypress tests' env: # avoid warnings about terminal TERM: xterm CYPRESS_RECORD_KEY: $(CYPRESS_RECORD_KEY) + - script: |+ npm install --force https://cdn.cypress.io/extract-cloud-results/v1/extract-cloud-results.tgz+ node ./scripts/verifyUICoverageResults.js+ displayName: 'Verify UI Coverage Results'+ env:+ CYPRESS_PROJECT_ID: $(CYPRESS_PROJECT_ID)+ CYPRESS_RECORD_KEY: $(CYPRESS_RECORD_KEY)
version: 2.1 jobs: linux-test: docker: - image: cypress/base:22.12.0 working_directory: ~/repo steps: - checkout - run: npm install - run: npx run cypress:run --record + - run: npm install --force https://cdn.cypress.io/extract-cloud-results/beta/v1/extract-cloud-results.tgz+ - run: node ./scripts/verifyUICoverageResults.js workflows: version: 2 tests: jobs: - run-cypress
close