Command Line Interface

Installing Schemathesis installs the schemathesis script to your virtualenv, which you can use to test your APIs

Note

To see the full list of CLI options & commands use the --help option or check the Full list of CLI options.

Basic usage

To execute tests, use the schemathesis run command:

$ schemathesis run http://example.com/swagger.json

With this command, Schemathesis will load the schema from http://example.com/swagger.json and generate separate test sets for each operation in this schema. Each test set includes up to 100 test cases by default, depending on the operation definition.

For example, if your API schema has three operations, then you will see a similar output:

================ Schemathesis test session starts ===============
platform Linux -- Python 3.8.5, schemathesis-2.5.0, ...
rootdir: /
hypothesis profile 'default' -> ...
Schema location: http://127.0.0.1:8081/schema.yaml
Base URL: http://127.0.0.1:8081/api
Specification version: Swagger 2.0
Workers: 1
Collected API operations: 3

GET /api/path_variable/{key} .                             [ 33%]
GET /api/success .                                         [ 66%]
POST /api/users/ .                                         [100%]

============================ SUMMARY ============================

Performed checks:
    not_a_server_error              201 / 201 passed       PASSED

======================= 3 passed in 1.77s =======================

The output style is inspired by pytest and provides necessary information about the loaded API schema, processed operations, found errors, and used checks.

By default, Schemathesis refuses to work with schemas that do not conform to the Open API spec, but you can disable this behavior with --validate-schema=false.

Note

Schemathesis supports colorless output via the NO_COLOR <https://no-color.org/> environment variable or the --no-color CLI option.

Testing specific operations

By default, Schemathesis runs tests for all operations, but you can select specific operations with the following CLI options:

  • --endpoint / -E. Operation path;

  • --method / -M. HTTP method;

  • --tag / -T. Open API tag;

  • --operation-id / -O. operationId field value;

Each option accepts a case-insensitive regex string and could be used multiple times in a single command. For example, the following command will select all operations which paths start with /api/users:

$ schemathesis run -E ^/api/users http://api.com/swagger.json

Important

As filters are treated as regular expressions, ensure that they contain proper anchors. For example, /users/ will match /v1/users/orders/, but ^/users/$ will match only /users/.

If your API contains deprecated operations (that have deprecated: true in their definition), then you can skip them by passing --skip-deprecated-operations:

$ schemathesis run --skip-deprecated-operations ...

Tests configuration

Schemathesis is built on top of the Hypothesis library and allows you to configure testing process in the same way.

We support all configuration options accepted by the hypothesis.settings decorator. All of them are prefixed with --hypothesis- and underscores are replaced with dashes, for example:

  • --hypothesis-max-examples=1000. Generate up to 1000 test cases per API operation;

  • --hypothesis-phases=explicit. Run only examples, specified explicitly in the API schema;

  • --hypothesis-suppress-health-check=too_slow. Disables the too_slow health check and makes Schemathesis continue testing even if it is considered too slow.

See the whole list of available options via the schemathesis run --help command and in the Hypothesis documentation.

How are responses checked?

For each API response received during the test, Schemathesis runs several checks to verify response conformance. By default, it runs only one check that raises an error if the checked response has a 5xx HTTP status code.

There are four built-in checks you can use via the –checks / -c CLI option:

  • not_a_server_error. The response has 5xx HTTP status;

  • status_code_conformance. The response status is not defined in the API schema;

  • content_type_conformance. The response content type is not defined in the API schema;

  • response_schema_conformance. The response content does not conform to the schema defined for this specific response;

  • response_headers_conformance. The response headers does not contain all defined headers.

To make Schemathesis perform all built-in checks use --checks all CLI option:

$ schemathesis run --checks all http://api.com/swagger.json
================ Schemathesis test session starts ===============
platform Linux -- Python 3.8.5, schemathesis-2.5.0, ...
rootdir: /
hypothesis profile 'default' -> ...
Schema location: http://api.com/swagger.json
Base URL: http://api.com/
Specification version: Swagger 2.0
Workers: 1
Collected API operations: 3

GET /api/path_variable/{key} .                             [ 33%]
GET /api/success .                                         [ 66%]
POST /api/users/ .                                         [100%]

============================ SUMMARY ============================

Performed checks:
    not_a_server_error              201 / 201 passed       PASSED
    status_code_conformance         201 / 201 passed       PASSED
    content_type_conformance        201 / 201 passed       PASSED
    response_schema_conformance     201 / 201 passed       PASSED

======================= 3 passed in 1.69s =======================

Additionally, you can define the response time limit with --max-response-time. If any response will take longer than the provided value (in milliseconds) than it will indicate a failure:

$ schemathesis run --max-response-time=50 ...
================ Schemathesis test session starts ===============
platform Linux -- Python 3.8.5, schemathesis-2.5.0, ...
rootdir: /
hypothesis profile 'default' -> ...
Schema location: http://api.com/swagger.json
Base URL: http://api.com/
Specification version: Swagger 2.0
Workers: 1
Collected API operations: 1

GET /api/slow F                                            [100%]

============================ FAILURES ===========================
__________________________ GET /api/slow ________________________
1. Response time exceeded the limit of 50 ms

Run this Python code to reproduce this failure:

    requests.get('http://127.0.0.1:8081/api/slow')

Or add this option to your command line parameters:
    --hypothesis-seed=103697217851787640556597810346466192664
============================ SUMMARY ============================

Performed checks:
    not_a_server_error                  2 / 2 passed       PASSED
    max_response_time                   0 / 2 passed       FAILED

======================= 1 failed in 0.29s =======================

Concurrent testing

In some cases, you can speed up the testing process by distributing all tests among multiple threads via the -w / --workers option:

schemathesis run --workers 8 https://example.com/api/swagger.json

In the example above, all tests will be distributed among eight worker threads. Note that it is not guaranteed to improve performance because it depends on your application behavior.

Code samples style

To reproduce test failures Schemathesis generates code samples:

requests.get("http://127.0.0.1:8081/api/failure")

You can control these samples via the --code-sample-style CLI option. For example, passing curl will generate a cURL command like this:

curl -X GET http://127.0.0.1:8081/api/failure

ASGI / WSGI support

Schemathesis natively supports testing of ASGI and WSGI compatible apps (e.g., Flask or FastAPI), which is significantly faster since it doesn’t involve the network.

To test your app with this approach, you need to pass a special “path” to your application instance via the --app CLI option. This path consists of two parts, separated by :. The first one is an importable path to the module with your app. The second one is the variable name that points to your app. Example: --app=project.wsgi:app.

Then your schema location could be:

  • A full URL;

  • An existing filesystem path;

  • In-app path with the schema.

For example:

schemathesis run --app=src.wsgi:app /swagger.json

NOTE. Depending on your setup, you might need to run this command with a custom PYTHONPATH environment variable like this:

$ PYTHONPATH=$(pwd) schemathesis run --app=src.wsgi:app /swagger.json

Storing and replaying test cases

It can be useful for debugging purposes to store all requests generated by Schemathesis and all responses from the app into a separate file. Schemathesis allows you to do this with -store-network-log command-line option:

$ schemathesis run --store-network-log=cassette.yaml http://127.0.0.1/schema.yaml

This command will create a new YAML file that will network interactions in VCR format. It might look like this:

command: 'schemathesis run --store-network-log=cassette.yaml http://127.0.0.1/schema.yaml'
recorded_with: 'Schemathesis 1.2.0'
http_interactions:
- id: '0'
  status: 'FAILURE'
  seed: '1'
  elapsed: '0.00123'
  recorded_at: '2020-04-22T17:52:51.275318'
  checks:
    - name: 'not_a_server_error'
      status: 'FAILURE'
      message: 'Received a response with 5xx status code: 500'
  request:
    uri: 'http://127.0.0.1/api/failure'
    method: 'GET'
    headers:
      ...
    body:
      encoding: 'utf-8'
      base64_string: ''
  response:
    status:
      code: '500'
      message: 'Internal Server Error'
    headers:
      ...
    body:
      encoding: 'utf-8'
      base64_string: 'NTAwOiBJbnRlcm5hbCBTZXJ2ZXIgRXJyb3I='
    http_version: '1.1'

Schemathesis provides the following extra fields:

  • command. Full CLI command used to run Schemathesis.

  • http_interactions.id. A numeric interaction ID within the current cassette.

  • http_interactions.status. Type of test outcome is one of SUCCESS, FAILURE. The status value is calculated from individual checks statuses - if any check failed, then the final status is FAILURE.

  • http_interactions.seed. The Hypothesis seed used in that particular case could be used as an argument to --hypothesis-seed CLI option to reproduce this request.

  • http_interactions.elapsed. Time in seconds that a request took.

  • http_interactions.checks. A list of executed checks and and their status.

To work with the cassette, you could use yq or any similar tool. Show response body content of first failed interaction:

$ yq r foo.yaml 'http_interactions.(status==FAILURE).response.body.base64_string' | head -n 1 | base64 -d
500: Internal Server Error

Check payload in requests to /api/upload_file:

$ yq r foo.yaml 'http_interactions.(request.uri==http://127.0.0.1/api/upload_file).request.body.base64_string' | base64 -d
--7d4db38ad065994d913cb02b2982e3ba
Content-Disposition: form-data; name="data"; filename="data"


--7d4db38ad065994d913cb02b2982e3ba--

Saved cassettes can be replayed with schemathesis replay command. Additionally, you may filter what interactions to replay by these parameters:

  • id. Specific, unique ID;

  • status. Replay only interactions with this status (SUCCESS or FAILURE);

  • uri. A regular expression for request URI;

  • method. A regular expression for request method;

During replaying, Schemathesis will output interactions being replayed together with the response codes from the initial and current execution:

$ schemathesis replay foo.yaml --status=FAILURE
Replaying cassette: foo.yaml
Total interactions: 4005

  ID              : 0
  URI             : http://127.0.0.1:8081/api/failure
  Old status code : 500
  New status code : 500

  ID              : 1
  URI             : http://127.0.0.1:8081/api/failure
  Old status code : 500
  New status code : 500

JUnit support

It is possible to export test results to format, acceptable by such tools as Jenkins.

$ schemathesis run --junit-xml=/path/junit.xml http://127.0.0.1/schema.yaml

This command will create an XML at a given path, as in the example below.

<?xml version="1.0" ?>
<testsuites disabled="0" errors="1" failures="1" tests="3" time="0.10743043999536894">
    <testsuite disabled="0" errors="1" failures="1" name="schemathesis" skipped="0" tests="3" time="0.10743043999536894" hostname="bespin">
        <testcase name="GET /api/failure" time="0.089057">
            <failure type="failure" message="2. Received a response with 5xx status code: 500"/>
        </testcase>
        <testcase name="GET /api/malformed_json" time="0.011977">
            <error type="error" message="json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
">Traceback (most recent call last):
  File &quot;/home/user/work/schemathesis/src/schemathesis/runner/impl/core.py&quot;, line 87, in run_test
    test(checks, targets, result, **kwargs)
  File &quot;/home/user/work/schemathesis/src/schemathesis/runner/impl/core.py&quot;, line 150, in network_test
    case: Case,
  File &quot;/home/user/.pyenv/versions/3.8.0/envs/schemathesis/lib/python3.8/site-packages/hypothesis/core.py&quot;, line 1095, in wrapped_test
    raise the_error_hypothesis_found
  File &quot;/home/user/work/schemathesis/src/schemathesis/runner/impl/core.py&quot;, line 165, in network_test
    run_checks(case, checks, result, response)
  File &quot;/home/user/work/schemathesis/src/schemathesis/runner/impl/core.py&quot;, line 133, in run_checks
    check(response, case)
  File &quot;/home/user/work/schemathesis/src/schemathesis/checks.py&quot;, line 87, in response_schema_conformance
    data = response.json()
  File &quot;/home/user/.pyenv/versions/3.8.0/envs/schemathesis/lib/python3.8/site-packages/requests/models.py&quot;, line 889, in json
    return complexjson.loads(
  File &quot;/home/user/.pyenv/versions/3.8.0/lib/python3.8/json/__init__.py&quot;, line 357, in loads
    return _default_decoder.decode(s)
  File &quot;/home/user/.pyenv/versions/3.8.0/lib/python3.8/json/decoder.py&quot;, line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File &quot;/home/user/.pyenv/versions/3.8.0/lib/python3.8/json/decoder.py&quot;, line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
</error>
        </testcase>
        <testcase name="GET /api/success" time="0.006397"/>
    </testsuite>
</testsuites>

Base URL configuration

If your Open API schema defines servers (or basePath in Open API 2.0), these values will be used to construct a full operation URL during testing. In the case of Open API 3.0, the first value from servers will be used.

However, you may want to run tests against a different base URL. To do this, you need to pass the --base-url option in CLI or provide base_url argument to a loader/runner if you use Schemathesis in your code:

schemathesis run --base-url=http://127.0.0.1:8080/api/v2 http://production.com/api/openapi.json

And if your schema defines servers like this:

servers:
  - url: https://production.com/api/{basePath}
    variables:
      basePath:
        default: v1

Then the tests will be executed against /api/v2 base path.

The --base-url argument is also used if you wish to load the OpenAPI specification from a local file.

schemathesis run --base-url=http://127.0.0.1:8080/api/v1 path/to/openapi.json

Extending CLI

To fit Schemathesis to your workflows, you might want to extend it with your custom checks or setup environment before the test run. Schemathesis can load your Python code via the --pre-run option:

$ schemathesis --pre-run test.setup \
      run https://example.com/api/swagger.json

NOTE. This option should be passed before the run subcommand.

Also, depending on your setup, you might need to run this command with a custom PYTHONPATH environment variable like this:

$ PYTHONPATH=$(pwd) schemathesis --pre-run test.setup \
      run https://example.com/api/swagger.json

The passed value will be treated as an importable Python path and imported before the test run.

Registering custom checks

To use your custom checks with Schemathesis CLI, you need to register them via the register_check decorator:

import schemathesis


@schemathesis.register_check
def new_check(response, case):
    # some awesome assertions!
    pass

The registered check should accept a response with requests.Response / schemathesis.utils.WSGIResponse type and case with schemathesis.models.Case type. This code should be placed in the module you pass to the --pre-run option.

Then your checks will be available in Schemathesis CLI, and you can use them via the -c command-line option.

$ schemathesis --pre-run module.with.checks run \
      -c new_check https://example.com/api/swagger.json

Additionally, checks may return True to skip the check under certain conditions. For example, you may only want to run checks when the response code is 200.

import schemathesis


@schemathesis.register_check
def conditional_check(response, case):
    if response.status_code == 200:
        ...  # some awesome assertions!
    else:
        # check not relevant to this response, skip test
        return True

Skipped check calls will not be reported in the run summary.

Note

Learn more about writing custom checks here.

Debugging

If Schemathesis produces an internal error, its traceback is hidden. To show error tracebacks in the CLI output, use the --show-errors-tracebacks option.

Additionally you can dump all internal events to a JSON Lines file with the --debug-output-file CLI option.

Running CLI via Docker

Schemathesis CLI is also available as a Docker image:

docker run schemathesis/schemathesis:stable \
    run http://api.com/schema.json

To run it against the localhost server, add --network=host parameter:

docker run --network="host" schemathesis/schemathesis:stable \
    run http://127.0.0.1/schema.json

If your API spec is stored in a file, you could use it too by specifying a Docker volume:

docker run -v $(pwd):/mnt schemathesis/schemathesis:stable \
    run /mnt/spec.json

In the example above, the spec.json file from the current working directory is shared with the Schemathesis container. Note, that $(pwd) is shell-specific and works in sh / bash / zsh, but could be different in e.g. PowerShell.

Note

See Docker volumes documentation for more information.

Full list of CLI options

schemathesis

Command line tool for testing your web application built with Open API / GraphQL specifications.

schemathesis [OPTIONS] COMMAND [ARGS]...

Options

--pre-run <pre_run>

A module to execute before the running the tests.

--version

Show the version and exit.

run

Perform schemathesis test against an API specified by SCHEMA.

SCHEMA must be a valid URL or file path pointing to an Open API / GraphQL specification.

schemathesis run [OPTIONS] SCHEMA

Options

-c, --checks <checks>

List of checks to run.

Default

not_a_server_error

Options

not_a_server_error | status_code_conformance | content_type_conformance | response_headers_conformance | response_schema_conformance | all

-D, --data-generation-method <data_generation_methods>

Defines how Schemathesis generates data for tests.

Default

DataGenerationMethod.positive

Options

positive | negative

--max-response-time <max_response_time>

A custom check that will fail if the response time is greater than the specified one in milliseconds.

-t, --target <targets>

Targets for input generation.

Default

Options

response_time | all

-x, --exitfirst

Exit instantly on first error or failed test.

Default

False

--dry-run

Disable sending data to the application and checking responses. Helpful to verify whether data is generated at all.

-a, --auth <auth>

Server user and password. Example: USER:PASSWORD

-A, --auth-type <auth_type>

The authentication mechanism to be used. Defaults to ‘basic’.

Default

basic

Options

basic | digest

-H, --header <headers>

Custom header that will be used in all requests to the server. Example: Authorization: Bearer123

-E, --endpoint <endpoints>

Filter schemathesis tests by API operation path pattern. Example: users/d+

-M, --method <methods>

Filter schemathesis tests by HTTP method.

-T, --tag <tags>

Filter schemathesis tests by schema tag pattern.

-O, --operation-id <operation_ids>

Filter schemathesis tests by operationId pattern.

-w, --workers <workers_num>

Number of workers to run tests.

Default

1

Options

auto | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64

-b, --base-url <base_url>

Base URL address of the API, required for SCHEMA if specified by file.

--app <app>

WSGI/ASGI application to test.

--request-timeout <request_timeout>

Timeout in milliseconds for network requests during the test run.

--request-tls-verify <request_tls_verify>

Controls whether Schemathesis verifies the server’s TLS certificate. You can also pass the path to a CA_BUNDLE file for private certs.

Default

true

--validate-schema <validate_schema>

Enable or disable validation of input schema.

Default

True

--skip-deprecated-operations

Skip testing of deprecated API operations.

Default

False

--junit-xml <junit_xml>

Create junit-xml style report file at given path.

--debug-output-file <debug_output_file>

Save debug output as JSON lines in the given file.

--show-errors-tracebacks

Show full tracebacks for internal errors.

Default

False

--code-sample-style <code_sample_style>

Controls the style of code samples for failure reproduction.

Options

python | curl

--store-network-log <store_network_log>

Store requests and responses into a file.

--fixups <fixups>

Install specified compatibility fixups.

Options

fast_api | all

--stateful <stateful>

Utilize stateful testing capabilities.

Options

links

--stateful-recursion-limit <stateful_recursion_limit>

Limit recursion depth for stateful testing.

Default

5

--force-schema-version <force_schema_version>

Force Schemathesis to parse the input schema with the specified spec version.

Options

20 | 30

--hypothesis-deadline <hypothesis_deadline>

Duration in milliseconds that each individual example with a test is not allowed to exceed.

--hypothesis-derandomize

Use Hypothesis’s deterministic mode.

--hypothesis-max-examples <hypothesis_max_examples>

Maximum number of generated examples per each method/path combination.

--hypothesis-phases <hypothesis_phases>

Control which phases should be run.

Options

explicit | reuse | generate | target | shrink | explain

--hypothesis-report-multiple-bugs <hypothesis_report_multiple_bugs>

Raise only the exception with the smallest minimal example.

--hypothesis-seed <hypothesis_seed>

Set a seed to use for all Hypothesis tests.

--hypothesis-suppress-health-check <hypothesis_suppress_health_check>

Comma-separated list of health checks to disable.

Options

data_too_large | filter_too_much | too_slow | return_value | large_base_example | not_a_test_method | function_scoped_fixture

--hypothesis-verbosity <hypothesis_verbosity>

Verbosity level of Hypothesis messages.

Options

quiet | normal | verbose | debug

--no-color

Disable ANSI color escape codes.

--schemathesis-io-token <schemathesis_io_token>

Schemathesis.io authentication token. If present, test run results will be uploaded to Schemathesis.io

--schemathesis-io-url <schemathesis_io_url>

Schemathesis.io base URL.

-v, --verbosity

Reduce verbosity of error output.

Arguments

SCHEMA

Required argument