Test Case Quality

A good Certyn test case is clear enough for an agent to execute and specific enough for a reviewer to trust the result.

Quality checklist

A strong test case has:

• one main workflow or decision
• a clear starting point
• stable test data or environment variables
• expected outcomes
• enough detail to avoid guessing
• no secrets in the test text
• assertions that can be observed in the browser

Strong pattern

Use this structure:

Goal: Verify that an owner can create an API key.

Steps:
1. Sign in using OWNER_EMAIL and OWNER_PASSWORD.
2. Open Settings.
3. Open API Keys.
4. Create a new API key named "Smoke Test Key".

Expected:
- The new key appears in the API Keys list.
- The key has an active status.
- No error banner is shown.

The goal explains intent. The steps tell the agent what to do. The expected section defines success.

What to avoid

Weak instruction	Better instruction
Verify it works	Verify the dashboard loads and shows the user menu
Click the blue button in the top right	Click the button labeled "Create API Key"
Use my password	Use OWNER_PASSWORD from environment variables
Check the table	Verify at least one row appears with status Active
Test checkout	Complete checkout for one in-stock item and verify confirmation

Stable references

Prefer:

• labels and visible text
• page titles
• URLs when they are stable
• roles and permissions
• named environment variables
• durable test data

Avoid:

• pixel positions
• colors unless color is the behavior under test
• implementation details hidden from users
• exact dynamic timestamps
• personal accounts or real customer data

Assertions

Every test should say what success means.

Good assertions:

• "Verify the URL contains /settings/api-keys."
• "Verify the error message says the email is already registered."
• "Verify the issue count increases by one."
• "Verify the checkout confirmation page includes an order number."

Weak assertions:

• "Looks good."
• "No problems."
• "Everything works."

Data and cleanup

Prefer test data that is safe to reuse:

• dedicated test accounts
• environment variables for credentials and IDs
• predictable names with timestamps when uniqueness is required
• cleanup steps when the product supports safe cleanup

If cleanup is risky, make the test data easy to identify and exclude it from reporting.

When to split tests

Split a test when:

• it verifies more than one user goal
• failure would be hard to diagnose
• setup dominates the useful behavior
• one branch is critical and another is optional

Keep end-to-end tests valuable, not tiny. A login test and a checkout test may share login setup, but checkout should still verify the business workflow, not only navigation.

Certyn-specific guidance

• Use processes to group related tests into smoke, regression, and critical-path suites.
• Use tags to make suite selection predictable.
• Put app-specific rules in the project wiki.
• Promote useful exploratory observations into test cases when they should become repeatable coverage.