Run your tests in the cloud

For teams with tests in a repo and an app on the internet, who want Certyn to run them and explain the failures. You host nothing, and your CI stays as it is.

Your setup

• Tests in a repo: Playwright, Cypress, Selenium (Python), WebdriverIO, a Postman collection, Hurl files, or similar.
• The app is on the internet, for example a staging URL.
• The code is on GitHub, or on another host with its token.
• You want to know what failed and why.

Say this to Certyn

Run the Playwright tests in github.com/acme/web against staging and tell me what failed and why.

Other examples:

• Run our pytest Selenium tests from github.com/acme/qa against staging.
• Run our WebdriverIO tests from acme/web the way our GitHub Actions workflow does, and use the JUnit report.
• Run our Postman collection (postman/api.json in acme/api) with Newman against staging, and file the real API bugs.
• Write Hurl checks for our orders API on staging: create, read, list, and a 404 for an unknown id. Put them in acme/api under api-checks/.

What Certyn does

1. Opens a form for repo access if it has none. For GitHub, make a fine-grained token for that one repo with Contents: Read. It is saved as the secret project variable GH_TOKEN. Add Pull requests: Read and write only if you want fixes as pull requests. Public code may need no token.
2. If the tests need secrets (an API key, a test account), opens a small form for each one. Secrets never go in the chat.
3. Reads your CI file, package files and test config: the runtime version, the install step, the test command and the variables the workflow sets. It uses the same on the Machine.
4. Passes the environment's URL to the tests. On the Machine it is $TARGET_BASE_URL; Certyn sets the option or variable your config reads.
5. Runs one test or one file first to check the setup, then saves the repo and test command as project variables (for example GITHUB_REPO) for later runs.
6. Starts a run in the background that clones, installs and runs the whole suite. A run has 60 minutes by default. For a long suite Certyn asks for more, up to 4 hours. It says the length before it starts.
7. Reads the report (JUnit XML, JSON or the console output) and decides the cause of each failure: product bug, broken test, flaky test, or setup problem.

Run commands

What a cloud Machine has, and the command Certyn usually runs. Certyn uses your CI's own command when it finds one.

Notes per framework:

• Selenium (Python): Certyn makes a virtual environment, installs from requirements.txt or pyproject.toml, and reads conftest.py to see how the driver starts and where the URL comes from.
• WebdriverIO: with @wdio/junit-reporter in the config, Certyn reads the JUnit XML for exact totals and the first error of each failed test. Without it, it reads the console output. It can add the reporter as a pull request.
• Newman: it runs one folder first (--folder), then the whole collection. For each failed assertion it decides: an API bug, an out-of-date assertion, or a setup problem such as a wrong token.
• Hurl: Certyn can also write the checks. It reads the OpenAPI spec if there is one, writes one .hurl file per flow, fixes a check that is wrong, and opens a PR. A real API bug stays failing and becomes a finding. Secrets go in from the shell, never in the file.

What you get

• One run report: the totals, and the failures grouped by cause.
• One finding per real product bug, with the failing tests as evidence. Ten failing tests on one broken page make one finding. A finding people dismissed before is not raised again.
• Broken and flaky tests in the report, not as findings.
• If you ask: a fix for a broken test as a pull request from a certyn/... branch. Certyn never merges.
• If you ask: the same run on a schedule or after each deploy.

Keep results from your own CI too

Your CI can also send its own test results to Certyn. You then get history per test: pass rate and flaky tests. The ci_setup form in the chat creates the API key; save it as a CI secret.

With the GitHub Action, upload a Playwright JSON report:

- name: Run tests
  id: tests
  run: npx playwright test --reporter=json > pw-report.json
  continue-on-error: true

- name: Upload results to Certyn
  if: always()
  uses: certyn-one/action@v1
  with:
    api_key: ${{ secrets.CERTYN_API_KEY }}
    project_slug: my-app
    environment_key: staging
    results_file: pw-report.json
    results_format: playwright
    suite: playwright-e2e

From any CI, the same upload is one call:

curl -sS -X POST \
  "https://api.certyn.io/api/ci/results?projectSlug=my-app&suite=playwright-e2e&environmentKey=staging&format=playwright" \
  -H "X-API-Key: $CERTYN_API_KEY" \
  --data-binary @pw-report.json

Use format=junit (or results_format: junit) for JUnit XML from Cypress, pytest, WebdriverIO, Newman and others. See Push JUnit results. Keep suite the same across runs, so the history stays in one place.

Results pushed from CI give history only. They do not explain failures by themselves: ask Certyn, or let it run the suite.

Limits

• A cloud Machine starts clean, so it installs your dependencies at every run. To keep them, use your own runner with a workspace folder.
• Machines run Linux. Safari is not available. Other browsers, another Python version, Google Chrome or a fixed driver path must be installed during the run. Certyn says what to change; it does not change your code unless you ask.
• Tests that sign in need a test account, saved through the sign-in form, never typed in the chat.
• A Postman collection that lives only in Postman's cloud: export it to your repo first. Keep secrets out of the exported environment; save them as Certyn variables, and restart the Machine after you save a new one.
• Tests that create or delete data belong on staging. On production, Certyn asks before it changes real data.
• Hurl blocks saved in a Certyn test case stop after 300 seconds, and only {{baseUrl}} is filled for you. Long or token-based Hurl suites run in the shell (30 minutes per command).
• An app that is not on the internet needs a runner inside your network.
• A run uses plan time for as long as it runs.

Related

• Run your tests on your runner
• Run your tests on a schedule
• Work in your repo
• Bring your test framework
• Test accounts and secrets