Push JUnit Results from Jenkins, Azure DevOps, CircleCI or Bitbucket

For teams that already run tests in CI and want their history in Certyn, next to Certyn's own runs.

Your setup

• Jenkins, Azure DevOps, CircleCI or Bitbucket Pipelines.
• A suite that writes JUnit XML: JUnit, TestNG, pytest, Jest, Cypress, NUnit and most others can.
• You want pass rate and flaky tests per test, without moving the suite.

Say this to Certyn

Our Jenkins pipeline runs Maven tests and writes JUnit XML. Send those results to Certyn for the e2e suite.

What Certyn does

1. Opens the ci_setup form. Pick Other CI/CD. It creates an API key and shows it once. Save it as a CI secret named CERTYN_API_KEY (a Jenkins credential, an Azure DevOps secret variable, a CircleCI or Bitbucket secured variable).
2. Gives you one upload step for your CI, after the tests:

   curl -sS -X POST \
     "$CERTYN_API_URL/api/ci/results?projectSlug=my-app&suite=e2e&environmentKey=staging&format=junit&commitSha=$COMMIT_SHA&externalUrl=$BUILD_URL" \
     -H "X-API-Key: $CERTYN_API_KEY" \
     -H "Idempotency-Key: $BUILD_ID" \
     --data-binary @junit.xml
   

3. Tells you how to run it even when tests fail: Jenkins post { always { ... } }, Azure DevOps condition: always(), CircleCI when: always, Bitbucket after-script.

The first upload for a suite creates its process in Certyn. Later uploads add to its history.

What you get

• Each CI run in Certyn's run list, with the commit and a link back to the build.
• Pass rate, flaky rate and duration per test over time. Tests that pass and fail with no change show as flaky.
• These runs count in release readiness.

Limits

• An upload does not explain failures by itself. Ask Certyn in the chat about a failed run, or let Certyn run the suite itself to get each failure sorted by cause.
• Each upload is one run. If your suite writes many XML files, merge them into one file first.
• Flaky tests show only after several runs.
• One report can be up to 32 MB.

Related

• Bring Your Test Framework
• CI Runs API
• GitLab CI Calls Certyn After Deploy