Skip to content
certyn logoCertyn
Start Free
RecipesDoc

A Run Got It Wrong, Fix the Verdict, Run It Again, Teach Certyn

Correct a wrong pass or fail, close a wrong finding with a reason, run it again, and save the lesson so the same mistake does not come back.

For QA leads and developers who do not agree with a run's result.

Your setup

  • A run failed, but the app is fine. Or it passed, but the feature is broken.
  • Or a run filed a finding that is expected behavior.
  • You want the record right, and no repeat next week.

Say this to Certyn

This run failed because the discount banner is missing, but we removed it on purpose in 2.4. Fix the result and make sure Certyn does not flag it again. (paste the run's link)

What Certyn does

  1. Reads the run (app.execution.get): summary, screenshots, notes, linked issues.
  2. Says what went wrong: product bug, broken test, flaky step, or setup problem.
  3. Fixes the record, asking you before each change:
    • the run's verdict (app.execution.update): completed, failed, blocked or aborted;
    • in a suite run, one test case's result: passed, failed or blocked, with new result text;
    • a wrong finding (app.issue.update): closed as not_a_bug, duplicate or wont_fix, with a note.
  4. Saves the lesson where later runs read it: a testing rule in the wiki (The discount banner was removed in 2.4. Do not report it missing.), a memory (app.memory.create), or new steps in the test case (app.testcase.update).
  5. Runs it again if you want. It asks first:
    • retry: the same run again, for a failed, blocked or stopped run. The run page's Re-run button does this for a failed or stopped run.
    • retest: an issue after its fix. Needs a confirmed issue linked to a tracker issue or a test case.
    • run the suite again on the environment.

What you get

  • Correct results in reports and release checks.
  • A finding closed as not_a_bug, duplicate or wont_fix is not filed again when a later run finds the same thing.
  • A rule or memory that later runs and chats read.

Limits

  • A new verdict changes the record only. It does not tell later runs why. Save a rule, memory or test change for that.
  • Change a verdict in the chat. The run page has no button for it.
  • A retry runs the same thing. Save the rule or fix the test first, or it can fail the same way.
  • The repeat match is by similarity. A repeat written very differently can get through; close it again.
PreviousWatch Certyn Live, Step In for a Captcha, Code or SSO, and Stop a RunNextAttach a Screenshot, Log or HAR File in the Chat, Mention Issues with @, Use / Commands