Continuous Integration

Welcome to the Schemathesis CI guide! This document provides all the necessary information to integrate Schemathesis into your Continuous Integration workflows.

Quickstart

You can use these code samples to test your API in a pull request or run tests against a publicly resolvable API.

If you need to start your API server locally before testing, check out the Preparing your App section below.

GitHub Actions

Important

We have a native GitHub app that reports test results directly to your pull requests.

api-tests:
  runs-on: ubuntu-20.04
  steps:
    # Runs Schemathesis tests with all checks enabled
    - uses: schemathesis/action@v1
      with:
        # Your API schema location
        schema: 'http://localhost:5000/api/openapi.json'
        # OPTIONAL. Your Schemathesis.io token
        token: ${{ secrets.SCHEMATHESIS_TOKEN }}

For the fully working example, check .github/workflows/example-no-build.yml in the repository.

If you enabled PR comments via our GitHub app, you’ll see a test report once your pipeline is finished:

When interacting with APIs that require headers, use the -H option in the Schemathesis CLI. Ensure the entire header value is enclosed in quotes:

# Save access token to $GITHUB_ENV as ACCESS_TOKEN.
- name: Set access token
  run: echo "ACCESS_TOKEN=super-secret" >> $GITHUB_ENV

# Use the saved access token in the Schemathesis GitHub action.
- uses: schemathesis/action@v1
  with:
    schema: 'https://example.schemathesis.io/openapi.json'
    args: '-H "Authorization: Bearer ${{ env.ACCESS_TOKEN }}"'

• Make sure any variables or dynamic content in the header value, like ${{ env.ACCESS_TOKEN }}, are correctly resolved in your CI environment.

• Remember, the -H option allows you to pass other headers in a similar manner if needed.

GitLab CI

api-tests:
  stage: test
  image:
    name: schemathesis/schemathesis:stable
    entrypoint: [""]

  variables:
    # API Schema location
    API_SCHEMA: 'http://localhost:5000/api/openapi.json'
    # OPTIONAL. Your Schemathesis.io token
    SCHEMATHESIS_TOKEN: ${{ secrets.SCHEMATHESIS_TOKEN }}

  script:
    - st run $API_SCHEMA --checks=all --report

How does it work?

Schemathesis works over HTTP and expects that your application is reachable from the CI environment. You can prepare your application in the same CI as the previous step or run it against a staging environment.

Preparing your App

Start API before testing

It is common to have a test suite as a part of the application repo. For this scenario, you will need to build your app first.

The application could be built in any programming language, Schemathesis expects only its API schema.

Here is a GitHub Actions workflow for a sample Python app:

api-tests:
  runs-on: ubuntu-20.04
  steps:
    # Gets a copy of the source code in your repository before running API tests
    - uses: actions/checkout@v3.0.0

    - uses: actions/setup-python@v4
      with:
        python-version: '3.10'

    # Installs project's dependencies
    - run: pip install -r requirements.txt

    # Start the API in the background
    - run: python main.py &

    # Runs Schemathesis tests with all checks enabled
    - uses: schemathesis/action@v1
      with:
        # Your API schema location
        schema: 'http://localhost:5000/api/openapi.json'
        # OPTIONAL. Your Schemathesis.io token
        token: ${{ secrets.SCHEMATHESIS_TOKEN }}

Note

This example expects the API schema available at http://localhost:5000/api/openapi.json inside the CI environment.

For the fully working example, check .github/workflows/example-build.yml in the repository.

API schema in a file

If you store your API schema in a file, use its file path for the API_SCHEMA environment variable. Set your API base path to SCHEMATHESIS_BASE_URL:

api-tests:
  runs-on: ubuntu-20.04
  steps:
    # Runs positive Schemathesis tests
    - uses: schemathesis/action@v1
      with:
        # A local API schema location
        schema: './docs/openapi.json'
        # API base URL
        base-url: 'http://127.0.0.1:8080/api/v2/'
        # OPTIONAL. Your Schemathesis.io token
        token: ${{ secrets.SCHEMATHESIS_TOKEN }}