Running tests with the CLI

Install the Canary CLI, authenticate, and run your workflow tests from the terminal.

The Canary CLI lets you trigger test runs from your terminal and stream results in real-time. Use it locally during development or wire it into CI/CD to gate deployments.

Prerequisites

At least one published flow in your organization.
An API key or an interactive login session.

If you haven't built flows yet, start with Building your first smoke suite.

Install the CLI

# npm
npm install -g @canaryai/cli

# or with bun
bun add -g @canaryai/cli

Verify the install:

canary version

Authenticate

You have two options: interactive login (for local development) or an API key (for CI/CD and automation).

Option A: Interactive login

canary login

This opens your browser for a one-time device code flow. Once approved, the CLI stores a long-lived token at ~/.config/canary-cli/auth.json.

If your account belongs to multiple organizations, you'll be prompted to pick one. You can also pass --org <name> to skip the prompt.

Option B: API key

Create an API key in Settings > API Keys (requires admin access), then pass it to the CLI:

canary test --remote --token cnry_your_api_key

Or set it as an environment variable so you don't have to pass it every time:

export CANARY_API_TOKEN=cnry_your_api_key

See API Keys for details on creating and managing keys.

Run your tests

Trigger a test run across all published workflows:

canary test --remote

The CLI will:

POST to the Canary API to start a test suite
Subscribe to an SSE stream for real-time results
Print pass/fail status for each workflow as it finishes
Exit with code 0 if all workflows passed, or 1 if any failed

Example output:

Starting remote workflow tests...

  ✓ Sign in flow
  ✓ Create project
  ✗ Checkout flow
    Error: Expected "Order confirmed" but page showed "500 Internal Server"
  ✓ Search and filter

──────────────────────────────────────────────────
FAILED: 1 of 4 workflows failed (75% pass rate)

Filter which tests run

You don't always want to run everything. Use tags and name patterns to run a subset.

By tag

Tags are assigned to flows in the UI. Filter by tag to run a specific category:

canary test --remote --tag smoke

By name pattern

Match workflows by name (case-insensitive):

canary test --remote --name-pattern "checkout"

Combine filters

When both are provided, workflows must match both criteria:

canary test --remote --tag smoke --name-pattern "auth"

Verbose mode

Add --verbose (or -v) to see every SSE event as it streams, including suite metadata and timing:

canary test --remote --tag smoke --verbose

Exit codes

Code	Meaning
`0`	All workflows passed
`1`	One or more workflows failed

This makes the CLI a natural fit for CI/CD pipeline gates: if a workflow fails, the pipeline step fails.

Run in GitHub Actions

Add this workflow to .github/workflows/canary-tests.yml:

name: Canary Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  canary:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install Canary CLI
        run: npm install -g @canaryai/cli

      - name: Run smoke tests
        env:
          CANARY_API_TOKEN: ${{ secrets.CANARY_API_TOKEN }}
        run: canary test --remote --tag smoke

Setup steps

Create an API key in your Canary organization.
In your GitHub repo, go to Settings > Secrets and variables > Actions.
Add a secret named CANARY_API_TOKEN with the key value.
Commit the workflow file above.

Tips

Use the --tag flag to run only smoke tests in CI rather than the full suite. Save the full suite for nightly or staging deploys.
Pin to a specific CLI version if you want reproducible builds: npm install -g @canaryai/cli@0.1.7
Add the step after your deploy so you're testing the freshly deployed version.
Use --verbose in CI for better debugging when a failure occurs.

For more CI platforms (GitLab CI, CircleCI, Jenkins), see the CI/CD Integration guide.

Environment variables

Variable	Description	Default
`CANARY_API_TOKEN`	API key or login token	(none)
`CANARY_API_URL`	API endpoint	`https://api.trycanary.ai`

Troubleshooting

"No API token found"

Either set CANARY_API_TOKEN or run canary login first.

"Failed to start tests: 401"

Your token is invalid, expired, or revoked. Create a new API key or re-run canary login.

"No workflows found matching the filter criteria"

Check that your filters match at least one published workflow. Tags are exact matches; name patterns are case-insensitive substrings.

Tests pass in the UI but fail in CI

Confirm your flows target the correct environment (staging vs production).
Check that credentials are configured for the environment being tested.
Use --verbose to see detailed event data.

Next steps

Building your first smoke suite -- pick the right flows to test
API Keys -- manage keys for your team
CI/CD Integration -- examples for GitLab, CircleCI, and Jenkins
Test Runs -- understand test run lifecycle and auto-verification