Test Runs

Execute all published workflows as regression tests.

Test Runs

Test runs execute all your published workflows in a single batch, capturing pass/fail results for each workflow. They serve as regression tests to ensure your application continues working as expected after changes.

Starting a Test Run

  1. Go to Runs in the sidebar
  2. Click New Run → Test Run
  3. All published workflows will execute concurrently

Note: The "Test Run" option is only available if you have at least one published workflow.

What Happens During a Test Run

When you start a test run:

  1. Discovery - Canary finds all workflows with status "published"
  2. Execution - Each workflow runs concurrently in its own browser session
  3. Verification - Results are checked against expected outcomes
  4. Reporting - Pass/fail status is recorded for each workflow

Test runs use the same execution engine as individual workflow runs, including automatic retries for flaky failures.

Test Run Statuses

StatusMeaning
QueuedTest run is waiting to start
RunningWorkflows are executing
CompletedAll workflows finished successfully
Completed (Errors)One or more workflows failed

Workflow Outcomes

Each workflow in a test run has its own outcome:

OutcomeMeaning
SuccessWorkflow passed on first attempt
FailedWorkflow failed after all retries
Flaky SuccessWorkflow passed after one or more retries
WaitingWorkflow is paused at a Wait node

Auto-Verification and Issue Filing

When a workflow fails during a test run:

  1. The failure is analyzed to determine if it's a real bug or test instability
  2. If it's a real bug, an issue is automatically created with:
    • Screenshot at the moment of failure
    • Steps that led to the failure
    • Error message and context
  3. Issues are deduplicated - the same failure won't create multiple issues

Viewing Test Run Results

From the Runs page:

  1. Click the Test Runs tab
  2. Click any test run row to see details
  3. View per-workflow outcomes and drill into specific failures

CI/CD Integration

Test runs can be triggered from your CI/CD pipeline:

# Start a test run via API
curl -X POST https://api.trycanary.ai/workflows/test-runs \
  -H "Authorization: Bearer $CANARY_API_KEY"

See the CI/CD Integration guide for detailed setup instructions.

Best Practices

  1. Keep workflows published - Only published workflows are included in test runs
  2. Use descriptive names - Makes it easier to identify which test failed
  3. Add Wait nodes carefully - Workflows with Wait nodes take longer to complete
  4. Monitor flaky tests - Workflows that frequently show "Flaky Success" may need adjustment
  5. Run before deploys - Use test runs as a quality gate before production deployments