Skip to content
RecipesDoc

Fix a Flaky Test, Find the Cause, Open a PR

Certyn reruns a flaky test many times, finds why it fails sometimes, and fixes the test without making it weaker.

For teams with a test that passes and fails with no code change, and no time to find out why.

Your setup

  • A Playwright, Cypress, pytest or similar suite in a GitHub repo.
  • One test fails sometimes, in CI or in Certyn's run report.
  • A test environment the test can run against.

Say this to Certyn

The "apply coupon" test in acme/web is flaky. Find out why and fix it in a PR.

What Certyn does

  1. Opens the GH_TOKEN form if needed: Contents and Pull requests, Read and write.
  2. Clones the repo and runs the test many times in a row against the environment (for Playwright, --repeat-each), with the same settings as your CI.
  3. Compares passing and failing tries: traces, screenshots, timing, test data, test order.
  4. Names the cause, for example a click before the page is ready, data shared between tests, or a fixed wait.
  5. Makes the smallest fix on a certyn/... branch, such as waiting for the right element or giving the test its own data.
  6. Runs it many times again and opens a pull request with the cause, the fix, and the before and after pass counts.

What you get

  • One pull request with the fix and the proof.
  • If the app is at fault (for example a button that sometimes submits twice), a Certyn issue with evidence, not a test change.

Limits

  • Certyn never makes a test pass by weakening it: no removed checks, no retries added to hide the failure, no larger timeouts as the only fix.
  • Some flakes appear only under CI load. If Certyn cannot make the test fail in many tries, it says so and does not guess a fix. Share a CI trace or run it on your self-hosted runner.
  • One shell command on a Machine can run 30 minutes. Longer repeat loops run in the background, or as a run of up to 4 hours.