Command Line Interface
Installing Schemathesis installs the schemathesis
script to your virtualenv, which you can use to test your APIs
Note
To see the full list of CLI options & commands use the --help
option or check the Full list of CLI options.
Basic usage
To execute tests, use the st run
command:
$ st run https://example.schemathesis.io/openapi.json
With this command, Schemathesis will load the schema from https://example.schemathesis.io/openapi.json
and generate separate
test sets for each operation in this schema. Each test set includes up to 100 test cases by default, depending on the operation definition.
For example, if your API schema has three operations, then you will see a similar output:
================ Schemathesis test session starts ===============
Schema location: http://127.0.0.1:8081/schema.yaml
Base URL: http://127.0.0.1:8081/api
Specification version: Swagger 2.0
Workers: 1
Collected API operations: 3
GET /api/path_variable/{key} . [ 33%]
GET /api/success . [ 66%]
POST /api/users/ . [100%]
============================ SUMMARY ============================
Performed checks:
not_a_server_error 201 / 201 passed PASSED
======================= 3 passed in 1.77s =======================
The output style is inspired by pytest and provides necessary information about the loaded API schema, processed operations, found errors, and used checks.
By default, Schemathesis works with schemas that do not conform to the Open API spec, but you can enable schema validation with --validate-schema=true
.
Note
Schemathesis supports colorless output via the NO_COLOR <https://no-color.org/> environment variable or the --no-color
CLI option.
Testing specific operations
By default, Schemathesis runs tests for all operations, but you can select specific operations with the following CLI options:
--endpoint / -E
. Operation path;--method / -M
. HTTP method;--tag / -T
. Open API tag;--operation-id / -O
.operationId
field value;
Each option accepts a case-insensitive regex string and could be used multiple times in a single command.
For example, the following command will select all operations which paths start with /api/users
:
$ st run -E ^/api/users https://example.schemathesis.io/openapi.json
Important
As filters are treated as regular expressions, ensure that they contain proper anchors. For example, /users/ will match /v1/users/orders/, but ^/users/$ will match only /users/.
If your API contains deprecated operations (that have deprecated: true
in their definition),
then you can skip them by passing --skip-deprecated-operations
:
$ st run --skip-deprecated-operations ...
Overriding test data
You can set specific values for Open API parameters in test cases, such as query parameters, headers and cookies.
This is particularly useful for scenarios where specific parameter values are required for deeper testing. For instance, when dealing with values that represent data in a database, which Schemathesis might not automatically know or generate.
Each override follows the general form of --set-[part] name=value
.
For Open API, the [part]
corresponds to the in
value of a parameter which is query
, header
, cookie
, or path
.
You can specify multiple overrides in a single command and each of them will be applied only to API operations that use such a parameter.
For example, to override a query parameter and path:
$ st run --set-query apiKey=secret --set-path user_id=42 ...
This command overrides the apiKey
query parameter and user_id
path parameter, using secret
and 42
as their respective values in all applicable test cases.
Tests configuration
Schemathesis is built on top of the Hypothesis library and allows you to configure testing process in the same way.
We support all configuration options accepted by the hypothesis.settings
decorator.
All of them are prefixed with --hypothesis-
and underscores are replaced with dashes, for example:
--hypothesis-max-examples=1000
. Generate up to 1000 test cases per API operation;--hypothesis-phases=explicit
. Run only examples, specified explicitly in the API schema;--hypothesis-suppress-health-check=too_slow
. Disables thetoo_slow
health check and makes Schemathesis continue testing even if it is considered too slow.
See the whole list of available options via the st run --help
command and in the Hypothesis documentation.
How are responses checked?
For each API response received during the test, Schemathesis runs several checks to verify response conformance. By default, it runs only one check that raises an error if the checked response has a 5xx HTTP status code.
There are four built-in checks you can use via the –checks / -c CLI option:
not_a_server_error
. The response has 5xx HTTP status;status_code_conformance
. The response status is not defined in the API schema;content_type_conformance
. The response content type is not defined in the API schema;response_schema_conformance
. The response content does not conform to the schema defined for this specific response;response_headers_conformance
. The response headers does not contain all defined headers.
To make Schemathesis perform all built-in checks use --checks all
CLI option:
$ st run --checks all https://example.schemathesis.io/openapi.json
================ Schemathesis test session starts ===============
Schema location: https://example.schemathesis.io/openapi.json
Base URL: http://api.com/
Specification version: Swagger 2.0
Workers: 1
Collected API operations: 3
GET /api/path_variable/{key} . [ 33%]
GET /api/success . [ 66%]
POST /api/users/ . [100%]
============================ SUMMARY ============================
Performed checks:
not_a_server_error 201 / 201 passed PASSED
status_code_conformance 201 / 201 passed PASSED
content_type_conformance 201 / 201 passed PASSED
response_schema_conformance 201 / 201 passed PASSED
======================= 3 passed in 1.69s =======================
You can also define a list of checks to exclude using the --exclude-checks
CLI option:
$ st run --checks all --exclude-checks not_a_server_error https://example.schemathesis.io/openapi.json
================ Schemathesis test session starts ===============
Schema location: https://example.schemathesis.io/openapi.json
Base URL: http://api.com/
Specification version: Swagger 2.0
Workers: 1
Collected API operations: 3
GET /api/path_variable/{key} . [ 33%]
GET /api/success . [ 66%]
POST /api/users/ . [100%]
============================ SUMMARY ============================
Performed checks:
status_code_conformance 201 / 201 passed PASSED
content_type_conformance 201 / 201 passed PASSED
response_schema_conformance 201 / 201 passed PASSED
======================= 3 passed in 1.69s =======================
Additionally, you can define the response time limit with --max-response-time
.
If any response will take longer than the provided value (in milliseconds) than it will indicate a failure:
$ st run --max-response-time=50 ...
================ Schemathesis test session starts ===============
Schema location: https://example.schemathesis.io/openapi.json
Base URL: https://example.schemathesis.io/api
Specification version: Swagger 2.0
Workers: 1
Collected API operations: 1
GET /api/slow F [100%]
============================ FAILURES ===========================
__________________________ GET /api/slow ________________________
1. Test Case ID: 9Yjzd8
- Response time limit exceeded
Actual: 101.92ms
Limit: 50.00ms
[200] OK:
`{"success": true}`
Reproduce with:
curl -X GET http://127.0.0.1:8081/api/slow
Or add this option to your command line parameters:
--hypothesis-seed=103697217851787640556597810346466192664
============================ SUMMARY ============================
Performed checks:
not_a_server_error 2 / 2 passed PASSED
max_response_time 0 / 2 passed FAILED
======================= 1 failed in 0.29s =======================
Concurrent testing
In some cases, you can speed up the testing process by distributing all tests among multiple threads via the -w / --workers
option:
st run --workers 8 https://example.com/api/swagger.json
In the example above, all tests will be distributed among eight worker threads. Note that it is not guaranteed to improve performance because it depends on your application behavior.
Code samples style
To reproduce test failures Schemathesis generates code samples:
requests.get("http://127.0.0.1:8081/api/failure")
You can control these samples via the --code-sample-style
CLI option. For example, passing curl
will generate a cURL command like this:
curl -X GET http://127.0.0.1:8081/api/failure
ASGI / WSGI support
Schemathesis natively supports testing of ASGI and WSGI compatible apps (e.g., Flask or FastAPI), which is significantly faster since it doesn’t involve the network.
To test your app with this approach, you need to pass a special “path” to your application instance via the --app
CLI option. This path consists of two parts, separated by :
.
The first one is an importable path to the module with your app. The second one is the variable name that points to your app. Example: --app=project.wsgi:app
.
Then your schema location could be:
A full URL;
An existing filesystem path;
In-app path with the schema.
For example:
st run --app=src.wsgi:app /swagger.json
NOTE. Depending on your setup, you might need to run this command with a custom PYTHONPATH
environment variable like this:
$ PYTHONPATH=$(pwd) st run --app=src.wsgi:app /swagger.json
Storing and replaying test cases
It can be useful for debugging purposes to store all requests generated by Schemathesis and all responses from the app into a separate file.
Schemathesis allows you to do this with the --cassette-path
command-line option:
$ st run --cassette-path cassette.yaml http://127.0.0.1/schema.yaml
This command will create a new YAML file that will network interactions in VCR format. It might look like this:
command: 'st run --cassette-path=cassette.yaml http://127.0.0.1/schema.yaml'
recorded_with: 'Schemathesis 1.2.0'
http_interactions:
- id: '0'
status: 'FAILURE'
seed: '1'
elapsed: '0.00123'
recorded_at: '2020-04-22T17:52:51.275318'
checks:
- name: 'not_a_server_error'
status: 'FAILURE'
message: 'Received a response with 5xx status code: 500'
request:
uri: 'http://127.0.0.1/api/failure'
method: 'GET'
headers:
...
body:
encoding: 'utf-8'
string: ''
response:
status:
code: '500'
message: 'Internal Server Error'
headers:
...
body:
encoding: 'utf-8'
string: '500: Internal Server Error'
http_version: '1.1'
Schemathesis provides the following extra fields:
command
. Full CLI command used to run Schemathesis.http_interactions.id
. A numeric interaction ID within the current cassette.http_interactions.status
. Type of test outcome is one ofSUCCESS
,FAILURE
. The status value is calculated from individual checks statuses - if any check failed, then the final status isFAILURE
.http_interactions.seed
. The Hypothesis seed used in that particular case could be used as an argument to--hypothesis-seed
CLI option to reproduce this request.http_interactions.elapsed
. Time in seconds that a request took.http_interactions.checks
. A list of executed checks and and their status.http_interactions.data_generation_method
. The way test case was generated -positive
ornegative
.http_interactions.thread_id
. Unique integer that identifies the thread where a test case was used.http_interactions.correlation_id
. A unique ID which connects events that happen during testing of the same API operation
By default, payloads are converted to strings, but similar to the original Ruby’s VCR, Schemathesis supports preserving exact body bytes via the --cassette-preserve-exact-body-bytes
option.
To work with the cassette, you could use yq or any similar tool. Show response body content of first failed interaction:
$ yq '.http_interactions.[] | select(.status == "FAILURE") | .response.body.string' foo.yaml | head -n 1
500: Internal Server Error
Check payload in requests to /api/upload_file
:
$ yq '.http_interactions.[] | select(.request.uri == "http://127.0.0.1:8081/api/upload_file").request.body.string'
--7d4db38ad065994d913cb02b2982e3ba
Content-Disposition: form-data; name="data"; filename="data"
--7d4db38ad065994d913cb02b2982e3ba--
If you use --cassette-preserve-exact-body-bytes
then you need to look for the base64_string
field and decode it:
$ yq '.http_interactions.[] | select(.status == "FAILURE") | .response.body.base64_string' foo.yaml | head -n 1 | base64 -d
500: Internal Server Error
Saved cassettes can be replayed with st replay
command. Additionally, you may filter what interactions to
replay by these parameters:
id
. Specific, unique ID;status
. Replay only interactions with this status (SUCCESS
orFAILURE
);uri
. A regular expression for request URI;method
. A regular expression for request method;
During replaying, Schemathesis will output interactions being replayed together with the response codes from the initial and current execution:
$ st replay foo.yaml --status=FAILURE
Replaying cassette: foo.yaml
Total interactions: 4005
ID : 0
URI : http://127.0.0.1:8081/api/failure
Old status code : 500
New status code : 500
ID : 1
URI : http://127.0.0.1:8081/api/failure
Old status code : 500
New status code : 500
JUnit support
It is possible to export test results to format, acceptable by such tools as Jenkins.
$ st run --junit-xml=/path/junit.xml http://127.0.0.1/schema.yaml
This command will create an XML at a given path, as in the example below.
<?xml version="1.0" ?>
<testsuites disabled="0" errors="0" failures="4" tests="4" time="1.7481054730014876">
<testsuite disabled="0" errors="0" failures="4" name="schemathesis" skipped="0" tests="4" time="1.7481054730014876" hostname="midgard">
<testcase name="GET /response-conformance/missing-field" time="0.859204">
<failure type="failure" message="1. Test Case ID: JA63GZ
- Response violates schema
'age' is a required property
Schema:
{
"type": "object",
"properties": {
"id": {
"type": "string"
},
"name": {
"type": "string"
},
"age": {
"type": "integer"
}
},
"required": [
"id",
"name",
"age"
]
}
Value:
{
"id": "123",
"name": "Alice"
}
[200] OK:
`{"id":"123","name":"Alice"}`
Reproduce with:
curl -X GET https://example.schemathesis.io/response-conformance/missing-field"/>
</testcase>
<testcase name="GET /response-conformance/malformed-json" time="0.068179">
<failure type="failure" message="1. Test Case ID: Vn5hfI
- JSON deserialization error
Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
[200] OK:
`{success: true}`
Reproduce with:
curl -X GET https://example.schemathesis.io/response-conformance/malformed-json"/>
</testcase>
<testcase name="GET /response-conformance/undocumented-status-code" time="0.756355">
<failure type="failure" message="1. Test Case ID: jm2nOs
- Undocumented HTTP status code
Received: 404
Documented: 200, 400
[404] Not Found:
`{"error":"Not Found"}`
Reproduce with:
curl -X GET 'https://example.schemathesis.io/response-conformance/undocumented-status-code?id=1'"/>
</testcase>
<testcase name="GET /response-conformance/incorrect-content-type" time="0.064367">
<failure type="failure" message="1. Test Case ID: Sveexo
- Undocumented Content-Type
Received: text/plain
Documented: application/json
[200] OK:
`Success!`
Reproduce with:
curl -X GET https://example.schemathesis.io/response-conformance/incorrect-content-type"/>
</testcase>
</testsuite>
</testsuites>
Base URL configuration
If your Open API schema defines servers
(or basePath
in Open API 2.0), these values will be used to
construct a full operation URL during testing. In the case of Open API 3, the first value from servers
will be used.
However, you may want to run tests against a different base URL. To do this, you need to pass the --base-url
option in CLI
or provide base_url
argument to a loader/runner if you use Schemathesis in your code:
st run --base-url=http://127.0.0.1:8080/api/v2 http://production.com/api/openapi.json
And if your schema defines servers
like this:
servers:
- url: https://production.com/api/{basePath}
variables:
basePath:
default: v1
Then the tests will be executed against /api/v2
base path.
The --base-url
argument is also used if you wish to load the OpenAPI specification from a local file.
st run --base-url=http://127.0.0.1:8080/api/v1 path/to/openapi.json
Extending CLI
To fit Schemathesis to your workflows, you might want to extend it with your custom checks or setup environment before the test run.
Extensions should be placed in a separate Python module.
Then, Schemathesis should be informed about this module via the SCHEMATHESIS_HOOKS
environment variable:
export SCHEMATHESIS_HOOKS=myproject.tests.hooks
st run http://127.0.0.1/openapi.yaml
Also, depending on your setup, you might need to run this command with a custom PYTHONPATH
environment variable like this:
export PYTHONPATH=$(pwd)
export SCHEMATHESIS_HOOKS=myproject.tests.hooks
st run https://example.com/api/swagger.json
The passed value will be treated as an importable Python path and imported before the test run.
Note
You can find more details on how to extend Schemathesis in the Extending Schemathesis section.
Registering custom checks
To use your custom checks with Schemathesis CLI, you need to register them via the schemathesis.check
decorator:
import schemathesis
@schemathesis.check
def new_check(response, case):
# some awesome assertions!
pass
The registered check should accept a response
with requests.Response
/ schemathesis.utils.WSGIResponse
type and
case
with schemathesis.models.Case
type. This code should be placed in the module you pass to the SCHEMATHESIS_HOOKS
environment variable.
Then your checks will be available in Schemathesis CLI, and you can use them via the -c
command-line option.
$ SCHEMATHESIS_HOOKS=module.with.checks
$ st run -c new_check https://example.com/api/swagger.json
Additionally, checks may return True
to skip the check under certain conditions. For example, you may only want to run checks when the
response code is 200
.
import schemathesis
@schemathesis.check
def conditional_check(response, case):
if response.status_code == 200:
... # some awesome assertions!
else:
# check not relevant to this response, skip test
return True
Skipped check calls will not be reported in the run summary.
Note
Learn more about writing custom checks here.
Rate limiting
APIs implement rate limiting to prevent misuse of their resources.
Schemathesis CLI’s --rate-limit
option can be used to set the maximum number of requests per second, minute, hour, or day during testing to avoid hitting these limits.
# 3 requests per second
st run --rate-limit=3/s
# 100 requests per minute
st run --rate-limit=100/m
# 1000 requests per hour
st run --rate-limit=1000/h
# 10000 requests per day
st run --rate-limit=10000/d
Debugging
If Schemathesis produces an internal error, its traceback is hidden. To show error tracebacks in the CLI output, use
the --show-trace
option.
Additionally you can dump all internal events to a JSON Lines file with the --debug-output-file
CLI option.
Running CLI via Docker
Schemathesis CLI is also available as a Docker image:
docker run schemathesis/schemathesis:stable \
run http://api.com/schema.json
To run it against the localhost server, add --network=host
parameter:
docker run --network="host" schemathesis/schemathesis:stable \
run http://127.0.0.1/schema.json
If your API spec is stored in a file, you could use it too by specifying a Docker volume:
docker run -v $(pwd):/mnt schemathesis/schemathesis:stable \
run /mnt/spec.json
In the example above, the spec.json
file from the current working directory is shared with the Schemathesis container.
Note, that $(pwd)
is shell-specific and works in sh
/ bash
/ zsh
, but could be different in e.g. PowerShell
.
When running from Docker, by default color output is not present. You can use --force-color
if you know that the host’s terminal supports colors.
Note that --force-color
and --no-color
are not compatible with each other.
Note
See Docker volumes documentation for more information.
Docker on MacOS
Due to the networking behavior of Docker on MacOS, the containerized application cannot directly reach localhost
of the host machine.
To address this, MacOS users should use the special DNS name host.docker.internal
when referring to the host within Docker.
docker run schemathesis/schemathesis:stable \
run http://host.docker.internal:8080/swagger.json
Note
See Docker on MacOS documentation for more details
Full list of CLI options
schemathesis
Automated API testing employing fuzzing techniques for OpenAPI and GraphQL.
schemathesis [OPTIONS] COMMAND [ARGS]...
Options
- --version
Show the version and exit.
run
Run tests against an API using a specified SCHEMA.
[Required] SCHEMA: Path to an OpenAPI (.json, .yml) or GraphQL SDL file, or a URL pointing to such specifications.
[Optional] API_NAME: Identifier for uploading test data to Schemathesis.io.
schemathesis run [OPTIONS] SCHEMA [API_NAME]
Options
- -c, --checks <checks>
Specifies the validation checks to apply to API responses. Provide a comma-separated list of checks such as ‘not_a_server_error,status_code_conformance’, etc. Default is ‘not_a_server_error’.
- Default:
not_a_server_error
- Options:
not_a_server_error | status_code_conformance | content_type_conformance | response_headers_conformance | response_schema_conformance | all
- --exclude-checks <exclude_checks>
Specifies the validation checks to skip during testing. Provide a comma-separated list of checks you wish to bypass.
- Default:
- Options:
not_a_server_error | status_code_conformance | content_type_conformance | response_headers_conformance | response_schema_conformance
- -D, --data-generation-method <data_generation_methods>
Specifies the approach Schemathesis uses to generate test data. Use ‘positive’ for valid data, ‘negative’ for invalid data, or ‘all’ for both. Default is ‘positive’.
- Default:
positive
- Options:
positive | negative | all
- --max-response-time <max_response_time>
Sets a custom time limit for API response times. The test will fail if a response time exceeds this limit. Provide the time in milliseconds.
- -t, --target <targets>
Guides input generation to values more likely to expose bugs via targeted property-based testing.
- Default:
- Options:
response_time | all
- -x, --exitfirst
Terminates the test suite immediately upon the first failure or error encountered.
- Default:
False
- --max-failures <max_failures>
Terminates the test suite after reaching a specified number of failures or errors.
- --dry-run
Simulates test execution without making any actual requests, useful for validating data generation.
- -a, --auth <auth>
Provides the server authentication details in the ‘USER:PASSWORD’ format.
- -A, --auth-type <auth_type>
Specifies the authentication method. Default is ‘basic’.
- Default:
basic
- Options:
basic | digest
- --set-query <set_query>
OpenAPI: Override a specific query parameter by specifying ‘parameter=value’
- --set-header <set_header>
OpenAPI: Override a specific header parameter by specifying ‘parameter=value’
- --set-cookie <set_cookie>
OpenAPI: Override a specific cookie parameter by specifying ‘parameter=value’
- --set-path <set_path>
OpenAPI: Override a specific path parameter by specifying ‘parameter=value’
- -H, --header <headers>
Adds a custom HTTP header to all API requests. Format: ‘Header-Name: Value’.
- -E, --endpoint <endpoints>
API operation path pattern (e.g., users/d+).
- -M, --method <methods>
HTTP method (e.g., GET, POST).
- -T, --tag <tags>
Schema tag pattern.
- -O, --operation-id <operation_ids>
OpenAPI operationId pattern.
- -w, --workers <workers_num>
Sets the number of concurrent workers for testing. Auto-adjusts if ‘auto’ is specified.
- Default:
1
- Options:
auto | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64
- -b, --base-url <base_url>
Provides the base URL of the API, required when schema is provided as a file.
- --app <app>
Specifies the WSGI/ASGI application under test, provided as an importable Python path.
- --wait-for-schema <wait_for_schema>
Maximum duration, in seconds, to wait for the API schema to become available.
- --request-timeout <request_timeout>
Sets a timeout limit, in milliseconds, for each network request during tests.
- --request-proxy <request_proxy>
Set the proxy for all network requests.
- --request-tls-verify <request_tls_verify>
Configures TLS certificate verification for server requests. Can specify path to CA_BUNDLE for custom certs.
- Default:
true
- --request-cert <request_cert>
File path of unencrypted client certificate for authentication. The certificate can be bundled with a private key (e.g. PEM) or the private key can be provided with the –request-cert-key argument.
- --request-cert-key <request_cert_key>
Specifies the file path of the private key for the client certificate.
- --validate-schema <validate_schema>
Toggles validation of incoming payloads against the defined API schema. Set to ‘True’ to enable or ‘False’ to disable. Default is ‘False’.
- Default:
False
- --skip-deprecated-operations
Exclude deprecated API operations from testing.
- Default:
False
- --junit-xml <junit_xml>
Outputs a JUnit-XML style report at the specified file path.
- --report <report_value>
Specifies how the generated report should be handled. If used without an argument, the report data will automatically be uploaded to Schemathesis.io. If a file name is provided, the report will be stored in that file. The report data, consisting of a tar gz file with multiple JSON files, is subject to change.
- --debug-output-file <debug_output_file>
Saves debugging information in a JSONL format at the specified file path.
- --show-trace
Displays complete traceback information for internal errors.
- Default:
False
- --code-sample-style <code_sample_style>
Selects the code sample style for reproducing failures.
- Options:
python | curl
- --cassette-path <cassette_path>
Saves the test outcomes in a VCR-compatible format.
- --cassette-preserve-exact-body-bytes
Retains exact byte sequence of payloads in cassettes, encoded as base64.
- --fixups <fixups>
Applies compatibility adjustments like ‘fast_api’, ‘utf8_bom’.
- Options:
fast_api | utf8_bom | all
- --rate-limit <rate_limit>
Specifies a rate limit for test requests in ‘<limit>/<duration>’ format. Example - 100/m for 100 requests per minute.
- --stateful <stateful>
Enables or disables stateful testing features.
- Options:
none | links
- --force-schema-version <force_schema_version>
Forces the schema to be interpreted as a particular OpenAPI version.
- Options:
20 | 30
- --sanitize-output <sanitize_output>
Enable or disable automatic output sanitization to obscure sensitive data.
- Default:
True
- --contrib-unique-data
Forces the generation of unique test cases.
- Default:
False
- --contrib-openapi-formats-uuid
Enables support for the ‘uuid’ string format in OpenAPI.
- Default:
False
- --contrib-openapi-fill-missing-examples
Enables generation of random examples for API operations that do not have explicit examples defined.
- Default:
False
- --hypothesis-database <hypothesis_database>
Configures storage for examples discovered by Hypothesis. Use ‘none’ to disable, ‘:memory:’ for temporary storage, or specify a file path for persistent storage.
- --hypothesis-deadline <hypothesis_deadline>
Sets a time limit for each test case generated by Hypothesis, in milliseconds. Exceeding this limit will cause the test to fail.
- --hypothesis-derandomize
Enables deterministic mode in Hypothesis, which eliminates random variation between test runs.
- --hypothesis-max-examples <hypothesis_max_examples>
Sets the cap on the number of examples generated by Hypothesis for each API method/path pair.
- --hypothesis-phases <hypothesis_phases>
Specifies which testing phases to execute.
- Options:
explicit | reuse | generate | target | shrink
- --hypothesis-no-phases <hypothesis_no_phases>
Specifies which testing phases to exclude from execution.
- Options:
explicit | reuse | generate | target | shrink
- --hypothesis-report-multiple-bugs <hypothesis_report_multiple_bugs>
If set, only the most easily reproducible exception will be reported when multiple issues are found.
- --hypothesis-seed <hypothesis_seed>
Sets a seed value for Hypothesis, ensuring reproducibility across test runs.
- --hypothesis-suppress-health-check <hypothesis_suppress_health_check>
Disables specified health checks from Hypothesis like ‘data_too_large’, ‘filter_too_much’, etc. Provide a comma-separated list
- Options:
data_too_large | filter_too_much | too_slow | large_base_example
- --hypothesis-verbosity <hypothesis_verbosity>
Controls the verbosity level of Hypothesis output.
- Options:
quiet | normal | verbose | debug
- --no-color
Disable ANSI color escape codes.
- --force-color
Explicitly tells to enable ANSI color escape codes.
- --experimental <experiments>
Enable experimental support for specific features.
- Options:
openapi-3.1 | schema-analysis
- --generation-allow-x00 <generation_allow_x00>
Determines whether to allow the generation of `` bytes within strings.
- Default:
true
- --generation-codec <generation_codec>
Specifies the codec used for generating strings.
- --schemathesis-io-token <schemathesis_io_token>
Schemathesis.io authentication token.
- --schemathesis-io-url <schemathesis_io_url>
Schemathesis.io base URL.
- --schemathesis-io-telemetry <schemathesis_io_telemetry>
Controls whether you send anonymized CLI usage data to Schemathesis.io along with your report.
- Default:
true
- --hosts-file <hosts_file>
Path to a file to store the Schemathesis.io auth configuration.
- -v, --verbosity
Increase verbosity of the output.
Arguments
- SCHEMA
Required argument
- API_NAME
Optional argument
Environment variables
- SCHEMATHESIS_API_NAME
Provide a default for
API_NAME
- SCHEMATHESIS_BASE_URL
Provide a default for
--base-url
- SCHEMATHESIS_WAIT_FOR_SCHEMA
Provide a default for
--wait-for-schema
- SCHEMATHESIS_REPORT
Provide a default for
--report
- SCHEMATHESIS_TOKEN
Provide a default for
--schemathesis-io-token
- SCHEMATHESIS_URL
Provide a default for
--schemathesis-io-url
- SCHEMATHESIS_TELEMETRY
Provide a default for
--schemathesis-io-telemetry
- SCHEMATHESIS_HOSTS_PATH
Provide a default for
--hosts-file