For teams with a test that passes and fails with no code change, and no time to find out why.
Your setup
- A Playwright, Cypress, pytest or similar suite in a GitHub repo.
- One test fails sometimes, in CI or in Certyn's run report.
- A test environment the test can run against.
Say this to Certyn
The "apply coupon" test in acme/web is flaky. Find out why and fix it in a PR.
What Certyn does
- Opens the
GH_TOKENform if needed: Contents and Pull requests, Read and write. - Clones the repo and runs the test many times in a row against the environment (for Playwright,
--repeat-each), with the same settings as your CI. - Compares passing and failing tries: traces, screenshots, timing, test data, test order.
- Names the cause, for example a click before the page is ready, data shared between tests, or a fixed wait.
- Makes the smallest fix on a
certyn/...branch, such as waiting for the right element or giving the test its own data. - Runs it many times again and opens a pull request with the cause, the fix, and the before and after pass counts.
What you get
- One pull request with the fix and the proof.
- If the app is at fault (for example a button that sometimes submits twice), a Certyn issue with evidence, not a test change.
Limits
- Certyn never makes a test pass by weakening it: no removed checks, no retries added to hide the failure, no larger timeouts as the only fix.
- Some flakes appear only under CI load. If Certyn cannot make the test fail in many tries, it says so and does not guess a fix. Share a CI trace or run it on your self-hosted runner.
- One shell command on a Machine can run 30 minutes. Longer repeat loops run in the background, or as a run of up to 4 hours.