For QA leads who need to say what changed in quality between two versions.
Your setup
- Production runs
2.5.0. Staging runs2.6.0. - Both versions had runs: Certyn suites, explore runs, maybe CI results you push.
- Your deploys set each environment's version, so runs and issues belong to a version.
- You want new problems, old ones still open, and likely regressions.
Say this to Certyn
Compare 2.6.0 with 2.5.0. What is new, what is still open from 2.5, how do the pass rates compare, and what looks like a regression?
What Certyn does
- Gets both release summaries (
app.release.get): the verdict, the issues found in each, the runs with results, and for 2.6.0 the open issues from earlier releases. - Splits the issues: new in 2.6.0, still open from 2.5.0 or before (not checked again on 2.6.0), and closed.
- Counts run results per version (
app.queryon executions, byversionandresult) and gives a pass rate for each. It names the test cases that ran on both versions, so the numbers compare the same tests. - Lists likely regressions: test cases that passed on 2.5.0 and failed on 2.6.0, and new 2.6.0 issues in areas that passed on 2.5.0.
- Offers next steps, such as retesting the old open issues on 2.6.0 or running a suite that ran on only one version. It asks before it starts a run.
What you get
- A side-by-side view: verdict, issues and pass rate per version.
- Likely regressions, each with links to the runs and issues.
- The gaps: what ran on one version only.
Limits
- Pass rate per version is counted from the runs. It is not a stored number: the pass-rate trend in metrics is per day, not per version.
- Only known runs count: Certyn runs, and CI results pushed while the environment had that version. If each version ran different tests, the rates do not compare well, and Certyn says so.
- A likely regression is a guess from results. A failed test can be a broken or flaky test. Certyn checks the cause in the run report before it calls it a product bug.
- Release names must match exactly. If your releases are
2.5.0and2.6.0, "2.5" finds nothing; Certyn lists the releases to find the right names.