CI/CD Integration

Run your workflow tests automatically in CI/CD pipelines using the Canary CLI.

Run your workflow tests automatically as part of your CI/CD pipeline. The Canary CLI triggers remote test runs, streams results in real-time, and exits with the appropriate code for your pipeline.

Prerequisites

  • You have created Flows that you want to run in CI/CD.
  • You have an API key (requires admin access).

New to the CLI? Start with Running tests with the CLI for a walkthrough.

1) Create an API Key

API keys authenticate your CI/CD pipeline with Canary. Unlike user sessions, they're designed for automated systems.

  1. Go to Settings → API Keys
  2. Click Create Key
  3. Give it a descriptive name (e.g., "GitHub Actions", "CircleCI Production")
  4. Copy the key immediately — it will only be shown once

The key format is cnry_... and grants full access to your organization's workflows.

Security tips:

  • Store the key as a secret in your CI/CD platform (never commit it to code)
  • Create separate keys for different pipelines so you can revoke them independently
  • Rotate keys periodically

2) Install the Canary CLI

Install the CLI globally or as a dev dependency:

# Global install
npm install -g @canaryai/cli

# Or as a dev dependency
npm install -D @canaryai/cli

3) Run remote tests

Use canary test --remote to trigger your workflow tests:

# Run all workflows
canary test --remote --token cnry_your_api_key

# Filter by tag
canary test --remote --token cnry_your_api_key --tag smoke

# Filter by name pattern
canary test --remote --token cnry_your_api_key --name-pattern "checkout"

# Combine filters
canary test --remote --token cnry_your_api_key --tag smoke --name-pattern "auth"

# Verbose mode (stream all events)
canary test --remote --token cnry_your_api_key --verbose

Exit codes

CodeMeaning
0All workflows passed
1One or more workflows failed

Environment variables

Instead of passing --token on every command, set the CANARY_API_TOKEN environment variable:

export CANARY_API_TOKEN=cnry_your_api_key
canary test --remote --tag smoke

You can also set CANARY_API_URL if you're using a self-hosted Canary instance:

export CANARY_API_URL=https://api.your-canary-instance.com

CI/CD Examples

GitHub Actions

name: Canary Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install Canary CLI
        run: npm install -g @canaryai/cli

      - name: Run smoke tests
        env:
          CANARY_API_TOKEN: ${{ secrets.CANARY_API_TOKEN }}
        run: canary test --remote --tag smoke

GitLab CI

canary-tests:
  image: node:20
  stage: test
  script:
    - npm install -g @canaryai/cli
    - canary test --remote --tag smoke
  variables:
    CANARY_API_TOKEN: $CANARY_API_TOKEN

CircleCI

version: 2.1

jobs:
  canary-tests:
    docker:
      - image: cimg/node:20.0
    steps:
      - checkout
      - run:
          name: Install Canary CLI
          command: npm install -g @canaryai/cli
      - run:
          name: Run smoke tests
          command: canary test --remote --tag smoke
          environment:
            CANARY_API_TOKEN: $CANARY_API_TOKEN

workflows:
  test:
    jobs:
      - canary-tests

Jenkins

pipeline {
    agent any

    environment {
        CANARY_API_TOKEN = credentials('canary-api-token')
    }

    stages {
        stage('Canary Tests') {
            steps {
                sh 'npm install -g @canaryai/cli'
                sh 'canary test --remote --tag smoke'
            }
        }
    }
}

Filtering workflows

Use tags and name patterns to run specific subsets of your workflows.

By tag

Tags let you categorize workflows (e.g., smoke, regression, checkout). Add tags when creating flows in the UI.

# Run only smoke tests
canary test --remote --tag smoke

# Run only checkout-related tests
canary test --remote --tag checkout

By name pattern

Filter workflows by name using case-insensitive matching:

# Run workflows with "login" in the name
canary test --remote --name-pattern login

# Run workflows with "payment" in the name
canary test --remote --name-pattern payment

Combining filters

When using both --tag and --name-pattern, workflows must match both criteria:

# Run smoke tests that also have "auth" in the name
canary test --remote --tag smoke --name-pattern auth

Verbose mode

Use --verbose (or -v) to see all events as they stream:

canary test --remote --tag smoke --verbose

This shows:

  • Suite start/completion events
  • Individual workflow status changes
  • Detailed timing information

Useful for debugging or when you want full visibility into test execution.

Troubleshooting

"No API token found"

Make sure you've either:

  • Set the CANARY_API_TOKEN environment variable, or
  • Passed --token cnry_... on the command line

"Failed to start tests: 401"

Your API key may be:

  • Invalid or mistyped
  • Revoked (check Settings → API Keys)
  • Expired

Create a new API key if needed.

"No workflows found matching the filter criteria"

Check that:

  • Your filter matches at least one workflow
  • The workflows have the tag you're filtering by
  • The name pattern matches workflow names (case-insensitive)

Tests pass locally but fail in CI

Common causes:

  • Different environment (staging vs production)
  • Timing issues (CI may be slower)
  • Missing credentials or configuration

Check the workflow run details in the Canary UI for screenshots and traces.

Next steps