Test Strategy & Coverage
Build practical coverage across smoke, regression, CI, environments, and risk areas.
A test strategy explains what evidence the team needs, where that evidence comes from, and how it affects release decisions.
Coverage is about risk
Coverage is not only a count of tests. Useful coverage connects tests to risk:
- Which user workflows are protected?
- Which roles, plans, environments, and browsers matter?
- Which areas changed recently?
- Which failures would block a release?
- Which known risks are accepted or deferred?
In Certyn, capture project-specific rules in the wiki so Ask Certyn and test authors can apply them consistently.
Suite types
| Suite | Purpose | Typical cadence |
|---|---|---|
| Smoke | Fast check that the environment is usable | After every deploy |
| Regression | Broad check that important workflows still work | Before release, nightly, or on schedule |
| Critical path | Must-pass workflows for release | Before release and after risky changes |
| Feature | Focused tests around a changed area | During feature development |
| Exploratory | Learn about uncertain or new behavior | Before major launch or after ambiguous failures |
CI cadence
Use automation where it creates fast, reliable feedback:
- Run smoke tests after deploys to staging and production-like environments.
- Run changed-area tests for pull requests or preview deployments when available.
- Run full regression before release or on a predictable schedule.
- Fail CI only on signals the team agrees are blockers.
- Route ambiguous results to review instead of pretending they are clean pass/fail decisions.
Environment strategy
Do not treat every environment as equivalent.
| Environment | Best use |
|---|---|
| Local or preview | Fast feedback on changed workflows |
| Staging | Release confidence and integration checks |
| Production | Lightweight smoke, monitoring, and critical path verification |
Use stable test data and dedicated test accounts. Keep secrets in environment variables, not in test text.
Coverage gaps
A coverage gap is an important risk without enough evidence.
Common gaps:
- no tests for a high-value workflow
- tests only cover the happy path
- no coverage for permissions or plan limits
- no recent run against the release candidate
- tests exist but are flaky or blocked
- assertions are too vague to prove behavior
When Ask Certyn identifies a coverage gap, turn it into one of:
- a new test case
- a wiki rule
- a process/schedule update
- an accepted risk with owner and expiry
Risk-based release thresholds
Default thresholds are useful, but project policy wins. Define release rules in the project wiki, such as:
- minimum pass rate
- required suites
- critical tests that must pass
- allowed flaky tests
- issue severities that block release
- maximum age of the latest regression run