Changelog

Unreleased - TBD

3.19.0 - 2023-03-22

Added

Schemathesis now supports custom authentication mechanisms from the requests library. You can use schemathesis.auth.set_from_requests to set up Schemathesis CLI with any third-party authentication implementation that works with requests. #1700

import schemathesis
from requests_ntlm import HttpNtlmAuth

schemathesis.auth.set_from_requests(HttpNtlmAuth("domain\\username", "password"))

Ability to apply authentication conditionally to specific API operations using a combination of @schemathesis.auth.apply_to() and @schemathesis.auth.skip_for() decorators.

import schemathesis


# Apply auth only for operations that path starts with `/users/` but not the `POST` method
@schemathesis.auth().apply_to(path_regex="^/users/").skip_for(method="POST")
class MyAuth:
    ...

Add a convenience mapping-like interface to OperationDefinition including indexing access, the get method, and “in” support.
Request throttling via the --rate-limit CLI option. #910

Changed

Unified Schemathesis custom authentication usage via the schema.auth decorator, replacing the previous schema.auth.register and schema.auth.apply methods:

Before:

import schemathesis

schema = schemathesis.from_uri("https://example.schemathesis.io/openapi.json")


# Schema-level auth
# Before: @schema.auth.register()
@schema.auth()
class MyAuth:
    ...


# Test-level auth
# Before: @schema.auth.apply(MyAuth)
@schema.auth(MyAuth)
@schema.parametrize()
def test_api(case):
    ...

Fixed

Handling of query parameters and cookies passed to case.call and query parameters passed to case.call_wsgi. The user-provided values are now merged with the data generated by Schemathesis, instead of overriding it completely. #1705
Parameter definition takes precedence over security schemes with the same name.
Unsatisfiable error when explicit header name passed via CLI clashes with the header parameter name. #1699
Not using the port keyword argument in schema loaders during API schema loading. #1721

3.18.5 - 2023-02-18

Added

Support for specifying the path to load hooks from via the SCHEMATHESIS_HOOKS environment variable. #1702.

Deprecated

Use of the --pre-run CLI option for loading hooks. Use the SCHEMATHESIS_HOOKS environment variable instead.

3.18.4 - 2023-02-16

Changed

Support any Werkzeug 2.x in order to allow mitigation of CVE-2023-25577. #1695

3.18.3 - 2023-02-12

Added

APIStateMachine.run method to simplify running stateful tests.

Changed

Improved quality of generated test sequences by updating state machines in Schemathesis to always run a minimum of two steps during testing. #1627 If you use hypothesis.stateful.run_state_machine_as_test to run your stateful tests, please use the run method on your state machine class instead. This change requires upgrading Hypothesis to at least version 6.68.1.

3.18.2 - 2023-02-08

Performance

Modify values in-place inside built-in map functions as there is no need to copy them.
Update hypothesis-jsonschema to 0.22.1 for up to 30% faster data generation in some workflows.

3.18.1 - 2023-02-06

Changed

Stateful testing: Only make stateful requests when stateful data is available from another operation. This change significantly reduces the number of API calls that likely will fail because of absense of stateful data. #1669

Performance

Do not merge component schemas into the currently tested schema if they are not referenced by it. Originally all schemas were merged to make them visible to hypothesis-jsonschema, but they imply significant overhead. #1180
Use a faster, specialized version of deepcopy.

3.18.0 - 2023-02-01

Added

Extra information to VCR cassettes.
The --contrib-unique-data CLI option that forces Schemathesis to generate unique test cases only. This feature is also available as a hook in schemathesis.contrib.unique_data.
A few decorators & functions that provide a simpler API to extend Schemathesis:
- schemathesis.auth() for authentication providers;
- schemathesis.check for checks;
- schemathesis.hook & BaseSchema.hook for hooks;
- schemathesis.serializer for serializers;
- schemathesis.target for targets;
- schemathesis.openapi.format for custom OpenAPI formats.
- schemathesis.graphql.scalar for GraphQL scalars.
Open API: UUID format generation via the schemathesis.contrib.openapi.formats.uuid extension You could enable it via the --contrib-openapi-formats-uuid CLI option.

Changed

Build: Switch the build backend to Hatch.
Relax requirements for attrs. #1643
Avoid occasional empty lines in cassettes.

Deprecated

schemathesis.register_check in favor of schemathesis.check.
schemathesis.register_target in favor of schemathesis.target.
schemathesis.register_string_format in favor of schemathesis.openapi.format.
schemathesis.graphql.register_scalar in favor of schemathesis.graphql.scalar.
schemathesis.auth.register in favor of schemathesis.auth.

Fixed

Remove recursive references from the last reference resolution level. It works on the best effort basis and does not cover all possible cases. #947
Invalid cassettes when headers contain characters with a special meaning in YAML.
Properly display flaky deadline errors.
Internal error when the utf8_bom fixup is used for WSGI apps.
Printing header that are set explicitly via get_call_kwargs in stateful testing. #828
Display all explicitly defined headers in the generated cURL command.
Replace starlette.testclient.TestClient with starlette_testclient.TestClient to keep compatibility with newer starlette versions. #1637

Performance

Running negative tests filters out less data.
Schema loading: Try a faster loader first if an HTTP response or a file is expected to be JSON.

3.17.5 - 2022-11-08

Added

Python 3.11 support. #1632

Fixed

Allow Werkzeug<=2.2.2. #1631

3.17.4 - 2022-10-19

Fixed

Appending an extra slash to the / path. #1625

3.17.3 - 2022-10-10

Fixed

Missing httpx dependency. #1614

3.17.2 - 2022-08-27

Fixed

Insufficient timeout for report uploads.

3.17.1 - 2022-08-19

Changed

Support requests==2.28.1.

3.17.0 - 2022-08-17

Added

Support for exception groups in newer Hypothesis versions. #1592
A way to generate negative and positive test cases within the same CLI run via -D all.

Fixed

Allow creating APIs in Schemathesis.io by name when the schema is passed as a file.
Properly trim tracebacks on Hypothesis>=6.54.0.
Skipping negative tests when they should not be skipped.

Changed

pytest: Generate positive & negative within the same test node.
CLI: Warning if there are too many HTTP 403 API responses.
Runner: BeforeExecution.data_generation_method and AfterExecution.data_generation_method changed to lists of DataGenerationMethod as the same test may contain data coming from different data generation methods.

3.16.5 - 2022-08-11

Fixed

CLI: Hanging on CTRL-C when --report is enabled.
Internal error when GraphQL schema has its root types renamed. #1591

3.16.4 - 2022-08-09

Changed

Suggest using --wait-for-schema if API schema is not available.

3.16.3 - 2022-08-08

Added

CLI: --max-failures=N option to exit after first N failures or errors. #1580
CLI: --wait-for-schema=N option to automatically retry schema loading for N seconds. #1582
CLI: Display old and new payloads in st replay when the -v option is passed. #1584

Fixed

Internal error on generating negative tests for query parameters with explode: true.

3.16.2 - 2022-08-05

Added

CLI: Warning if ALL API responses are HTTP 404.
The after_load_schema hook, which is designed for modifying the loaded API schema before running tests. For example, you can use it to add Open API links to your schema via schema.add_link.
New utf8_bom fixup. It helps to mitigate JSON decoding errors inside the response_schema_conformance check when payload contains BOM. #1563

Fixed

Description of -v or --verbosity option for CLI.

Changed

Execute before_call / after_call hooks inside the call_* methods. It makes them available for the pytest integration.

3.16.1 - 2022-07-29

Added

CLI: Warning if the API returns too many HTTP 401.
Add SCHEMATHESIS_BASE_URL environment variable for specifying --base-url in CLI.
Collect anonymyzed CLI usage telemetry when reports are uploaded. We do not collect any free-form values you use in your CLI, except for header names. Instead, we measure how many times you use each free-form option in this command. Additionally we count all non-default hook types only by hook name.

Important

You can disable usage this with the --schemathesis-io-telemetry=false CLI option or the SCHEMATHESIS_TELEMETRY=false environment variable.

3.16.0 - 2022-07-22

Added

Report uploading to Schemathesis.io via the --report CLI option.

Changed

Do not validate schemas by default in the pytest integration.
CLI: Display test run environment metadata only if -v is provided.
CLI: Do not display headers automatically added by requests in code samples.

Fixed

Do not report optional headers as missing.
Compatibility with hypothesis>=6.49. #1538
Handling of unittest.case.SkipTest emitted by newer Hypothesis versions.
Generating invalid headers when their schema has array or object types.

Removed

Previously, data was uploaded to Schemathesis.io when the proper credentials were specified. This release removes this behavior. From now on, every upload requires the explicit --report CLI option.
Textual representation of HTTP requests in CLI output in order to decrease verbosity and avoid showing the same data in multiple places.

3.15.6 - 2022-06-23

Fixed

Do not discard dots (.) in OpenAPI expressions during parsing.

3.15.5 - 2022-06-21

Fixed

TypeError when using --auth-type=digest in CLI.

3.15.4 - 2022-06-06

Added

Support generating data for Open API request payloads with wildcard media types. #1526

Changed

Mark tests as skipped if there are no explicit examples and --hypothesis-phases=explicit is used. #1323
Parse all YAML mapping keys as strings, ignoring the YAML grammar rules. For example, on: true will be parsed as {"on": True} instead of {True: True}. Even though YAML does not restrict keys to strings, in the Open API and JSON Schema context, this restriction is implied because the underlying data model comes from JSON.
INTERNAL: Improve flexibility of event serialization.
INTERNAL: Store request / response history in SerializedCheck.

3.15.3 - 2022-05-28

Fixed

Deduplication of failures caused by malformed JSON payload. #1518
Do not re-raise InvalidArgument exception as InvalidSchema in non-Schemathesis tests. #1514

3.15.2 - 2022-05-09

Fixed

Avoid generating negative query samples that requests will treat as an empty query.
Editable installation via pip.

3.15.1 - 2022-05-03

Added

OpenAPI: Expose APIOperation.get_security_requirements that returns a list of security requirements applied to the API operation
Attach originally failed checks to “grouped” exceptions.

Fixed

Internal error when Schemathesis doesn’t have permission to create its hosts.toml file.
Do not show internal Hypothesis warning multiple times when the Hypothesis database directory is not usable.
Do not print not relevant Hypothesis reports when run in CI.
Invalid verbose_name value in SerializedCase for GraphQL tests.

3.15.0 - 2022-05-01

Added

GraphQL: Mutations supports. Schemathesis will generate random mutations by default from now on.
GraphQL: Support for registering strategies to generate custom scalars.
Custom auth support for schemas created via from_pytest_fixture.

Changed

Do not encode payloads in cassettes as base64 by default. This change makes Schemathesis match the default Ruby’s VCR behavior and leads to more human-readable cassettes. Use --cassette-preserve-exact-body-bytes to restore the old behavior. #1413
Bump hypothesis-graphql to 0.9.0.
Avoid simultaneous authentication requests inside auth providers when caching is enabled.
Reduce the verbosity of pytest output. A few internal frames and the “Falsifying example” block are removed from the output.
Skip negative tests on API operations that are not possible to negate. #1463
Make it possible to generate negative tests if at least one parameter can be negated.
Treat flaky errors as failures and display full report about the failure. #1081
Do not duplicate failing explicit example in the HYPOTHESIS OUTPUT CLI output section. #881

Fixed

GraphQL: Semantically invalid queries without aliases.
GraphQL: Rare crashes on invalid schemas.
Internal error inside BaseOpenAPISchema.validate_response on requests>=2.27 when response body contains malformed JSON. #1485
schemathesis.from_pytest_fixture: Display each failure if Hypothesis found multiple of them.

Performance

GraphQL: Over 2x improvement from internal optimizations.

3.14.2 - 2022-04-21

Added

Support for auth customization & automatic refreshing. #966

3.14.1 - 2022-04-18

Fixed

Using @schema.parametrize with test methods on pytest>=7.0.

3.14.0 - 2022-04-17

Added

Open API link name customization via the name argument to schema.add_link.
st as an alias to the schemathesis command line entrypoint.
st auth login / st auth logout to authenticate with Schemathesis.io.
X-Schemathesis-TestCaseId header to help to distinguish test cases on the application side. #1303
Support for comma separated lists in the --checks CLI option. #1373
Hypothesis Database configuration for CLI via the --hypothesis-database option. #1326
Make the SCHEMA CLI argument accept API names from Schemathesis.io.

Changed

Enable Open API links traversal by default. To disable it, use --stateful=none.
Do not validate API schema by default. To enable it back, use --validate-schema=true.
Add the api_name CLI argument to upload data to Schemathesis.io.
Show response status code on failing checks output in CLI.
Improve error message on malformed Open API path templates (like /foo}/). #1372
Improve error message on malformed media types that appear in the schema or in response headers. #1382
Relax dependencies on pyyaml and click.
Add --cassette-path that is going to replace --store-network-log. The old option is deprecated and will be removed in Schemathesis 4.0

Fixed

Show the proper Hypothesis configuration in the CLI output. #1445
Missing source attribute in the Case.partial_deepcopy implementation. #1429
Duplicated failure message from content_type_conformance and response_schema_conformance checks when the checked response has no Content-Type header. #1394
Not copied case & response inside Case.validate_response.
Ignored pytest.mark decorators when they are applied before schema.parametrize if the schema is created via from_pytest_fixture. #1378

3.13.9 - 2022-04-14

Fixed

Compatibility with pytest-asyncio>=0.17.1. #1452

3.13.8 - 2022-04-05

Fixed

Missing media_type in the Case.partial_deepcopy implementation. It led to missing payload in failure reproduction code samples.

3.13.7 - 2022-04-02

Added

Support for Hypothesis>=6.41.0. #1425

3.13.6 - 2022-03-31

Changed

Deep-clone Response instances before passing to check functions.

3.13.5 - 2022-03-31

Changed

Deep-clone Case instances before passing to check functions.

3.13.4 - 2022-03-29

Added

Support for Werkzeug>=2.1.0. #1410

Changed

Validate requests kwargs to catch cases when the ASGI integration is used, but the proper ASGI client is not supplied. #1335

3.13.3 - 2022-02-20

Added

--request-tls-verify CLI option for the replay command. It controls whether Schemathesis verifies the server’s TLS certificate. You can also pass the path to a CA_BUNDLE file for private certs. #1395
Support for client certificate authentication with --request-cert and --request-cert-key arguments for the replay command.

3.13.2 - 2022-02-16

Changed

Use Schemathesis default User-Agent when communicating with SaaS.

Fixed

Use the same correlation_id in BeforeExecution and AfterExecution events if the API schema contains an error that causes an InvalidSchema exception during test execution.
Use full_path in error messages in recoverable schema-level errors. It makes events generated in such cases consistent with usual events.

3.13.1 - 2022-02-10

Added

APIOperation.iter_parameters helper to iterate over all parameters.

Fixed

Properly handle error if Open API parameter doesn’t have content or schema keywords.

3.13.0 - 2022-02-09

Changed

Update integration with Schemathesis.io.
Always show traceback for errors in Schemathesis.io integration.

3.12.3 - 2022-01-13

Fixed

Generating illegal unicode surrogates in queries. #1370

3.12.2 - 2022-01-12

Fixed

Not-escaped single quotes in generated Python code samples. #1359

3.12.1 - 2021-12-31

Fixed

Improper handling of base_url in call_asgi, when the base URL has a non-empty base path. #1366

3.12.0 - 2021-12-29

Changed

Upgrade typing-extensions to >=3.7,<5.
Upgrade jsonschema to ^4.3.2.
Upgrade hypothesis-jsonschema to >=0.22.0.

Fixed

Generating values not compliant with the ECMAScript regex syntax. #1350, #1241.

Removed

Support for Python 3.6.

3.11.7 - 2021-12-23

Added

Support for Python 3.10. #1292

3.11.6 - 2021-12-20

Added

Support for client certificate authentication with --request-cert and --request-cert-key arguments. #1173
Support for readOnly and writeOnly Open API keywords. #741

3.11.5 - 2021-12-04

Changed

Generate tests for API operations with the HTTP TRACE method on Open API 2.0.

3.11.4 - 2021-12-03

Changed

Add AfterExecution.data_generation_method.
Minor changes to the Schemathesis.io integration.

3.11.3 - 2021-12-02

Fixed

Silently failing to detect numeric status codes when the schema contains a shared parameters key. #1343
Not raising an error when tests generated by schemas loaded with from_pytest_fixture match no API operations. #1342

3.11.2 - 2021-11-30

Changed

Use name & data_generation_method parameters to subtest context instead of path & method. It allows the end-user to disambiguate among subtest reports.
Raise an error if a test function wrapped with schema.parametrize matches no API operations. #1336

Fixed

Handle KeyboardInterrupt that happens outside of the main test loop inside the runner. It makes interrupt handling consistent, independent at what point it happens. #1325
Respect the data_generation_methods config option defined on a schema instance when it is loaded via from_pytest_fixture. #1331
Ignored hooks defined on a schema instance when it is loaded via from_pytest_fixture. #1340

3.11.1 - 2021-11-20

Changed

Update click and PyYaml dependency versions. #1328

3.11.0 - 2021-11-03

Changed

Show cURL code samples by default instead of Python. #1269
Improve reporting of jsonschema errors which are caused by non-string object keys.
Store data_generation_method in BeforeExecution.
Use case-insensitive dictionary for Case.headers. #1280

Fixed

Pass data_generation_method to Case for GraphQL schemas.
Generation of invalid headers in some cases. #1142
Unescaped quotes in generated Python code samples on some schemas. #1030

Performance

Dramatically improve CLI startup performance for large API schemas.
Open API 3: Inline only components/schemas before passing schemas to hypothesis-jsonschema.
Generate tests on demand when multiple workers are used during CLI runs. #1287

3.10.1 - 2021-10-04

Added

DataGenerationMethod.all shortcut to get all possible enum variants.

Fixed

Unresolvable dependency due to incompatible changes in the new hypothesis-jsonschema release. #1290

3.10.0 - 2021-09-13

Added

Optional integration with Schemathesis.io.
New before_init_operation hook.
INTERNAL. description attribute for all parsed parameters inside APIOperation.
Timeouts when loading external schema components or external examples.

Changed

Pin werkzeug to >=0.16.0.
INTERNAL. OpenAPI20CompositeBody.definition type to List[OpenAPI20Parameter].
Open API schema loaders now also accept single DataGenerationMethod instances for the data_generation_methods argument. #1260
Improve error messages when the loaded API schema is not in JSON or YAML. #1262

Fixed

Internal error in make_case calls for GraphQL schemas.
TypeError on case.call with bytes data on GraphQL schemas.
Worker threads may not be immediately stopped on SIGINT. #1066
Re-used referenced objects during inlining. Now they are independent.
Rewrite not resolved remote references to local ones. #986
Stop worker threads on failures with exit_first enabled. #1204
Properly report all failures when custom checks are passed to case.validate_response.

Performance

Avoid using filters for header values when is not necessary.

3.9.7 - 2021-07-26

Added

New process_call_kwargs CLI hook. #1233

Changed

Check non-string response status codes when Open API links are collected. #1226

3.9.6 - 2021-07-15

Added

New before_call and after_call CLI hooks. #1224, #700

3.9.5 - 2021-07-14

Fixed

Preserve non-body parameter types in requests during Open API runtime expression evaluation.

3.9.4 - 2021-07-09

Fixed

KeyError when the response_schema_conformance check is executed against responses without schema definition. #1220
TypeError during negative testing on Open API schemas with parameters that have non-default style value. #1208

3.9.3 - 2021-06-22

Added

ExecutionEvent.is_terminal attribute that indicates whether an event is the last one in the stream.

Fixed

When EventStream.stop is called, the next event always is the last one.

3.9.2 - 2021-06-16

Changed

Return response from Case.call_and_validate.

Fixed

Incorrect deduplication applied to response schema conformance failures that happen to have the same failing validator but different input values. #907

3.9.1 - 2021-06-13

Changed

ExecutionEvent.asdict adds the event_type field which is the event class name.
Add API schema to the Initialized event.
Internal: Add SerializedCase.cookies
Convert all FailureContext class attributes to instance attributes. For simpler serialization via attrs.

3.9.0 - 2021-06-07

Added

GraphQL support in CLI. #746
A way to stop the Schemathesis runner’s event stream manually via events.stop() / events.finish() methods. #1202

Changed

Avoid pytest warnings when internal Schemathesis classes are in the test module scope.

3.8.0 - 2021-06-03

Added

Negative testing. #65
Case.data_generation_method attribute that provides the information of the underlying data generation method (e.g. positive or negative)

Changed

Raise UsageError if schema.parametrize or schema.given are applied to the same function more than once. #1194
Python values of True, False and None are converted to their JSON equivalents when generated for path parameters or query. #1166
Bump hypothesis-jsonschema version. It allows the end-user to override known string formats.
Bump hypothesis version.
APIOperation.make_case behavior. If no media_type is passed along with body, then it tries to infer the proper media type and raises an error if it is not possible. #1094

Fixed

Compatibility with hypothesis>=6.13.3.

3.7.8 - 2021-06-02

Fixed

Open API style & explode for parameters derived from security definitions.

3.7.7 - 2021-06-01

Fixed

Apply the Open API’s style & explode keywords to explicit examples. #1190

3.7.6 - 2021-05-31

Fixed

Disable filtering optimization for headers when there are keywords other than type. #1189

3.7.5 - 2021-05-31

Fixed

Too much filtering in headers that have schemas with the pattern keyword. #1189

3.7.4 - 2021-05-28

Changed

Internal: SerializedCase.path_template returns path templates as they are in the schema, without base path.

3.7.3 - 2021-05-28

Fixed

Invalid multipart payload generated for unusual schemas for the multipart/form-data media type.

Performance

Reduce the amount of filtering needed to generate valid headers and cookies.

3.7.2 - 2021-05-27

Added

SerializedCase.media_type that stores the information about what media type was used for a particular case.

Fixed

Internal error on unusual schemas for the multipart/form-data media type. #1152
Ignored explicit Content-Type override in Case.as_requests_kwargs.

3.7.1 - 2021-05-23

Added

Internal: FailureContext.title attribute that gives a short failure description.
Internal: FailureContext.message attribute that gives a longer failure description.

Changed

Rename JSONDecodeErrorContext.message to JSONDecodeErrorContext.validation_message for consistency.
Store the more precise schema & instance in ValidationErrorContext.
Rename ResponseTimeout to RequestTimeout.

3.7.0 - 2021-05-23

Added

Additional context for each failure coming from the runner. It allows the end-user to customize failure formatting.

Changed

Use different exception classes for not_a_server_error and status_code_conformance checks. It improves the variance of found errors.
All network requests (not WSGI) now have the default timeout of 10 seconds. If the response is time-outing, Schemathesis will report it as a failure. It also solves the case when the tested app hangs. #1164
The default test duration deadline is extended to 15 seconds.

3.6.11 - 2021-05-20

Added

Internal: BeforeExecution.verbose_name & SerializedCase.verbose_name that reflect specification-specific API operation name.

3.6.10 - 2021-05-17

Changed

Explicitly add colorama to project’s dependencies.
Bump hypothesis-jsonschema version.

3.6.9 - 2021-05-14

Fixed

Ignored $ref keyword in schemas with deeply nested references. #1167
Ignored Open API specific keywords & types in schemas with deeply nested references. #1162

3.6.8 - 2021-05-13

Changed

Relax dependency on starlette to >=0.13,<1. #1160

3.6.7 - 2021-05-12

Fixed

Missing support for the date string format (only full-date was supported).

3.6.6 - 2021-05-07

Changed

Improve error message for failing Hypothesis deadline healthcheck in CLI. #880

3.6.5 - 2021-05-07

Added

Support for disabling ANSI color escape codes via the NO_COLOR <https://no-color.org/> environment variable or the --no-color CLI option. #1153

Changed

Generate valid header values for Bearer auth by construction rather than by filtering.

3.6.4 - 2021-04-30

Changed

Bump minimum hypothesis-graphql version to 0.5.0. It brings support for interfaces and unions and fixes a couple of bugs in query generation.

3.6.3 - 2021-04-20

Fixed

Bump minimum hypothesis-graphql version to 0.4.1. It fixes a problem with generating queries with surrogate characters.
UnicodeEncodeError when sending application/octet-stream payloads that have no format: binary in their schemas. #1134

3.6.2 - 2021-04-15

Fixed

Windows: UnicodeDecodeError during schema loading via the from_path loader if it contains certain Unicode symbols. from_path loader defaults to UTF-8 from now on.

3.6.1 - 2021-04-09

Fixed

Using parametrized pytest fixtures with the from_pytest_fixture loader. #1121

3.6.0 - 2021-04-04

Added

Custom keyword arguments to schemathesis.graphql.from_url that are proxied to requests.post.
from_wsgi, from_asgi, from_path and from_file loaders for GraphQL apps. #1097, #1100
Support for data_generation_methods and code_sample_style in all GraphQL loaders.
Support for app & base_url arguments for the from_pytest_fixture runner.
Initial support for GraphQL schemas in the Schemathesis runner.

import schemathesis

# Load schema
schema = schemathesis.graphql.from_url("http://localhost:8000/graphql")
# Initialize runner
runner = schemathesis.runner.from_schema(schema)
# Emit events
for event in runner.execute():
    ...

Breaking

Loaders’ signatures are unified. Most of the arguments became keyword-only. All except the first two for ASGI/WSGI, all except the first one for the others. It forces loader calls to be more consistent.

# BEFORE
schema = schemathesis.from_uri(
    "http://example.com/openapi.json", "http://127.0.0.1:8000/", "GET"
)
# NOW
schema = schemathesis.from_uri(
    "http://example.com/openapi.json", base_url="http://127.0.0.1:8000/", method="GET"
)

Changed

Schemathesis generates separate tests for each field defined in the GraphQL Query type. It makes the testing process unified for both Open API and GraphQL schemas.
IDs for GraphQL tests use the corresponding Query field instead of HTTP method & path.
Do not show overly verbose raw schemas in Hypothesis output for failed GraphQL tests.
The schemathesis.graphql.from_url loader now uses the usual Schemathesis User-Agent.
The Hypothesis database now uses separate entries for each API operation when executed via CLI. It increases its effectiveness when tests are re-run.
Module schemathesis.loaders is moved to schemathesis.specs.openapi.loaders.
Show a more specific exception on incorrect usage of the from_path loader in the Schemathesis runner.

Deprecated

schemathesis.runner.prepare will be removed in Schemathesis 4.0. Use schemathesis.runner.from_schema instead. With this change, the schema loading part goes to your code, similar to using the regular Schemathesis Python API. It leads to a unified user experience where the starting point is API schema loading, which is much clearer than passing a callback & keyword arguments to the prepare function.

Fixed

Add the missing @schema.given implementation for schemas created via the from_pytest_fixture loader. #1093
Silently ignoring some incorrect usages of @schema.given.
Fixups examples were using the incorrect fixup name.
Return type of make_case for GraphQL schemas.
Missed operation_id argument in from_asgi loader.

Removed

Undocumented way to install fixups via the fixups argument for schemathesis.runner.prepare is removed.

3.5.3 - 2021-03-27

Fixed

Do not use importlib-metadata==3.8 in dependencies as it causes RuntimeError. Ref: https://github.com/python/importlib_metadata/issues/293

3.5.2 - 2021-03-24

Changed

Prefix worker thread names with schemathesis_.

3.5.1 - 2021-03-23

Fixed

Encoding for response payloads displayed in the CLI output. #1073
Use actual charset (from flask.Response.mimetype_params) when storing WSGI responses rather than defaulting to flask.Response.charset.

3.5.0 - 2021-03-22

Added

before_generate_case hook, that allows the user to modify or filter generated Case instances. #1067

Fixed

Missing body parameters during Open API links processing in CLI. #1069
Output types for evaluation results of $response.body and $request.body runtime expressions. #1068

3.4.1 - 2021-03-21

Added

event_type field to the debug output.

3.4.0 - 2021-03-20

Added

--debug-output-file CLI option to enable storing the underlying runner events in the JSON Lines format in a separate file for debugging purposes. #1059

Changed

Make Request.body, Response.body and Response.encoding internal attributes optional. For Request, it means that absent body will lead to Request.body to be None. For Response, body will be None if the app response did not have any payload. Previously these values were empty strings, which was not distinguishable from the cases described above. For the end-user, it means that in VCR cassettes, fields request.body and response.body may be absent.
models.Status enum now has string values for more readable representation.

3.3.1 - 2021-03-18

Fixed

Displaying wrong headers in the FAILURES block of the CLI output. #792

3.3.0 - 2021-03-17

Added

Display failing response payload in the CLI output, similarly to the pytest plugin output. #1050
A way to control which code sample style to use - Python or cURL. #908

Fixed

UnicodeDecodeError when generating cURL commands for failed test case reproduction if the request’s body contains non-UTF8 characters.

Internal

Extra information to events, emitted by the Schemathesis runner.

3.2.2 - 2021-03-11

Added

Support for Hypothesis 6. #1013

3.2.1 - 2021-03-10

Fixed

Wrong test results in some cases when the tested schema contains a media type that Schemathesis doesn’t know how to work with. #1046

3.2.0 - 2021-03-09

Performance

Add an internal caching layer for data generation strategies. It relies on the fact that the internal BaseSchema structure is not mutated over time. It is not directly possible through the public API and is discouraged from doing through hook functions.

Changed

APIOperation and subclasses of Parameter are now compared by their identity rather than by value.

3.1.3 - 2021-03-08

Added

count_operations boolean flag to runner.prepare. In case of False value, Schemathesis won’t count the total number of operations upfront. It improves performance for the direct runner usage, especially on large schemas. Schemathesis CLI will still use these calculations to display the progress during execution, but this behavior may become configurable in the future.

3.1.2 - 2021-03-08

Fixed

Percent-encode the generated . and .. strings in path parameters to avoid resolving relative paths and changing the tested path structure. #1036

3.1.1 - 2021-03-05

Fixed

Loosen importlib-metadata version constraint and update pyproject.toml #1039

3.1.0 - 2021-02-11

Added

Support for external examples via the externalValue keyword. #884

Fixed

Prevent a small terminal width causing a crash (due to negative length used in an f-string) when printing percentage
Support the latest cryptography version in Docker images. #1033

3.0.9 - 2021-02-10

Fixed

Return a default terminal size to prevent crashes on systems with zero-width terminals (some CI/CD servers).

3.0.8 - 2021-02-04

This release updates the documentation to be in-line with the current state.

3.0.7 - 2021-01-31

Fixed

Docker tags for Buster-based images.

3.0.6 - 2021-01-31

Packaging-only release for Docker images based on Debian Buster. #1028

3.0.5 - 2021-01-30

Fixed

Allow to use any iterable type for checks and additional_checks arguments to Case.validate_response.

3.0.4 - 2021-01-19

Fixed

Generating stateful tests, with common parameters behind a reference. #1020
Programmatic addition of Open API links via add_link when schema validation is disabled and response status codes are noted as integers. #1022

Changed

When operations are resolved by operationId then the same reference resolving logic is applied as in other cases. This change leads to less reference inlining and lower memory consumption for deeply nested schemas. #945

3.0.3 - 2021-01-18

Fixed

Flaky Hypothesis error during explicit examples generation. #1018

3.0.2 - 2021-01-15

Fixed

Processing parameters common for multiple API operations if they are behind a reference. #1015

3.0.1 - 2021-01-15

Added

YAML serialization for text/yaml, text/x-yaml, application/x-yaml and text/vnd.yaml media types. #1010.

3.0.0 - 2021-01-14

Added

Support for sending text/plain payload as test data. Including variants with non-default charset. #850, #939
Generating data for all media types defined for an operation. #690
Support for user-defined media types serialization. You can define how Schemathesis should handle media types defined in your schema or customize existing (like application/json).
The response_schema_conformance check now runs on media types that are encoded with JSON. For example, application/problem+json. #920
Base URL for GraphQL schemas. It allows you to load the schema from one place but send test requests to another one. #934
A helpful error message when an operation is not found during the direct schema access. #812
--dry-run CLI option. When applied, Schemathesis won’t send any data to the server and won’t perform any response checks. #963
A better error message when the API schema contains an invalid regular expression syntax. #1003

Changed

Open API parameters parsing to unblock supporting multiple media types per operation. Their definitions aren’t converted to JSON Schema equivalents right away but deferred instead and stored as-is.
Missing required: true in path parameters definition is now automatically enforced if schema validation is disabled. According to the Open API spec, the required keyword value should be true for path parameters. This change allows Schemathesis to generate test cases even for endpoints containing optional path parameters (which is not compliant with the spec). #941
Using --auth together with --header that sets the Authorization header causes a validation error. Before, the --header value was ignored in such cases, and the basic auth passed in --auth was used. #911
When hypothesis-jsonschema fails to resolve recursive references, the test is skipped with an error message that indicates why it happens.
Shorter error messages when API operations have logical errors in their schema. For example, when the maximum is less than the minimum - {"type": "integer", "minimum": 5, "maximum": 4}.
If multiple non-check related failures happens during a test of a single API operation, they are displayed as is, instead of Hypothesis-level error messages about multiple found failures or flaky tests. #975
Catch schema parsing errors, that are caused by YAML parsing.
The built-in test server now accepts --operations instead of --endpoints.
Display Collected API operations instead of collected endpoints in the CLI. #869
--skip-deprecated-endpoints is renamed to --skip-deprecated-operations. #869
Rename various internal API methods that contained endpoint in their names. #869
Bump hypothesis-jsonschema version to 0.19.0. This version improves the handling of unsupported regular expression syntax and can generate data for a subset of schemas containing such regular expressions.
Schemathesis doesn’t stop testing on errors during schema parsing. These errors are handled the same way as other errors during the testing process. It allows Schemathesis to test API operations with valid definitions and report problematic operations instead of failing the whole run. #999

Fixed

Allow generating requests without payload if the schema does not require it. #916
Allow sending null as request payload if the schema expects it. #919
CLI failure if the tested operation is GET and has payload examples. #925
Excessive reference inlining that leads to out-of-memory for large schemas with deep references. #945, #671
--exitfirst CLI option trims the progress bar output when a failure occurs. #951
Internal error if filling missing explicit examples led to Unsatisfiable errors. #904
Do not suggest to disable schema validation if it is already disabled. #914
Skip explicit examples generation if this phase is disabled via config. #905
Unsatisfiable error in stateful testing caused by all API operations having inbound links. #965, #822
A possibility to override APIStateMachine.step. #970
TypeError on nullable parameters during Open API specific serialization. #980
Invalid types in x-examples. #982
CLI crash on schemas with operation names longer than the current terminal width. #990
Handling of API operations that contain reserved characters in their paths. #992
CLI execution stops on errors during example generation. #994
Fill missing properties in incomplete explicit examples for non-body parameters. #1007

Deprecated

HookContext.endpoint. Use HookContext.operation instead.
Case.endpoint. Use Case.operation instead.

Performance

Use compiled versions of Open API spec validators.
Decrease CLI memory usage. #987
Various improvements relevant to processing of API operation definitions. It gives ~20% improvement on large schemas with many references.

Removed

Case.form_data. Use Case.body instead.
Endpoint.form_data. Use Endpoint.body instead.
before_generate_form_data hook. Use before_generate_body instead.
Deprecated stateful testing integration from our pytest plugin.

Note

This release features multiple backward-incompatible changes. The first one is removing form_data and hooks related to it - all payload related actions can be done via body and its hooks. The second one involves renaming the so-called “endpoint” to “operation”. The main reason for this is to generalize terminology and make it applicable to GraphQL schemas, as all Schemathesis internals are more suited to work with semantically different API operations rather than with endpoints that are often connected with URLs and HTTP methods. It brings the possibility to reuse the same concepts for Open API and GraphQL - in the future, unit tests will cover individual API operations in GraphQL, rather than everything available under the same “endpoint”.

2.8.6 - 2022-03-29

Added

Support for Werkzeug>=2.1.0. #1410

2.8.5 - 2020-12-15

Added

auto variant for the --workers CLI option that automatically detects the number of available CPU cores to run tests on. #917

2.8.4 - 2020-11-27

Fixed

Use --request-tls-verify during schema loading as well. #897

2.8.3 - 2020-11-27

Added

Display failed response payload in the error output for the pytest plugin. #895

Changed

In pytest plugin output, Schemathesis error classes use the CheckFailed name. Before, they had not readable “internal” names.
Hypothesis falsifying examples. The code does not include Case attributes with default values to improve readability. #886

2.8.2 - 2020-11-25

Fixed

Internal error in CLI, when the base_url is an invalid IPv6. #890
Internal error in CLI, when a malformed regex is passed to -E / -M / -T / -O CLI options. #889

2.8.1 - 2020-11-24

Added

--force-schema-version CLI option to force Schemathesis to use the specific Open API spec version when parsing the schema. #876

Changed

The content_type_conformance check now raises a well-formed error message when encounters a malformed media type value. #877

Fixed

Internal error during verifying explicit examples if an example has no value key. #882

2.8.0 - 2020-11-24

Added

--request-tls-verify CLI option, that controls whether Schemathesis verifies the server’s TLS certificate. You can also pass the path to a CA_BUNDLE file for private certs. #830

Changed

In CLI, if an endpoint contains an invalid schema, show a message about the --validate-schema CLI option. #855

Fixed

Handling of 204 responses in the response_schema_conformance check. Before, all responses were required to have the Content-Type header. #844
Catch OverflowError when an invalid regex is passed to -E / -M / -T / -O CLI options. #870
Internal error in CLI, when the schema location is an invalid IPv6. #872
Collecting Open API links behind references via CLI. #874

Deprecated

Using of Case.form_data and Endpoint.form_data. In the 3.0 release, you’ll need to use relevant body attributes instead. This change includes deprecation of the before_generate_form_data hook, use before_generate_body instead. The reason for this is the upcoming unification of parameter handling and their serialization.
--stateful-recursion-limit. It will be removed in 3.0 as a part of removing the old stateful testing approach. This parameter is no-op.

2.7.7 - 2020-11-13

Fixed

Missed headers in Endpoint.partial_deepcopy.

2.7.6 - 2020-11-12

Added

An option to set data generation methods. At the moment, it includes only “positive”, which means that Schemathesis will generate data that matches the schema.

Fixed

Pinned dependency on attrs that caused an error on fresh installations. #858

2.7.5 - 2020-11-09

Fixed

Invalid keyword in code samples that Schemathesis suggests to run to reproduce errors. #851

2.7.4 - 2020-11-07

Added

New relative_path property for BeforeExecution and AfterExecution events. It represents an operation path as it is in the schema definition.

2.7.3 - 2020-11-05

Fixed

Internal error on malformed JSON when the response_conformance check is used. #832

2.7.2 - 2020-11-05

Added

Shortcut for response validation when Schemathesis’s data generation is not used. #485

Changed

Improve the error message when the application can not be loaded from the value passed to the --app command-line option. #836
Security definitions are now serialized as other parameters. At the moment, it means that the generated values will be coerced to strings, which is a no-op. However, types of security definitions might be affected by the “Negative testing” feature in the future. Therefore this change is mostly for future-compatibility. #841

Fixed

Internal error when a “header” / “cookie” parameter were not coerced to a string before filtration. #839

2.7.1 - 2020-10-22

Fixed

Adding new Open API links via the add_link method, when the related PathItem contains a reference. #824

2.7.0 - 2020-10-21

Added

New approach to stateful testing, based on the Hypothesis’s RuleBasedStateMachine. #737
Case.validate_response accepts the new additional_checks argument. It provides a way to execute additional checks in addition to existing ones.

Changed

The response_schema_conformance and content_type_conformance checks fail unconditionally if the input response has no Content-Type header. #816

Fixed

Failure reproduction code missing values that were explicitly passed to call_* methods during testing. #814

Deprecated

Using stateful=Stateful.links in schema loaders and parametrize. Use schema.as_state_machine().TestCase instead. The old approach to stateful testing will be removed in 3.0. See the Stateful testing section of our documentation for more information.

2.6.1 - 2020-10-19

Added

New method as_curl_command added to the Case class. #689

2.6.0 - 2020-10-06

Added

Support for passing Hypothesis strategies to tests created with schema.parametrize by using schema.given decorator. #768
Support for PEP561. #748
Shortcut for calling & validation. #738
New hook to pre-commit, rstcheck, as well as updates to documentation based on rstcheck. #734
New check for maximum response time and corresponding CLI option --max-response-time. #716
New response_headers_conformance check that verifies the presence of all headers defined for a response. #742
New field with information about executed checks in cassettes. #702
New port parameter added to from_uri() method. #706
A code snippet to reproduce a failed check when running Python tests. #793
Python 3.9 support. #731
Ability to skip deprecated endpoints with --skip-deprecated-endpoints CLI option and skip_deprecated_operations=True argument to schema loaders. #715

Fixed

User-Agent header overriding the passed one. #757
Default User-Agent header in Case.call. #717
Status of individual interactions in VCR cassettes. Before this change, all statuses were taken from the overall test outcome, rather than from the check results for a particular response. #695
Escaping header values in VCR cassettes. #783
Escaping HTTP response message in VCR cassettes. #788

Changed

Case.as_requests_kwargs and Case.as_werkzeug_kwargs now return the User-Agent header. This change also affects code snippets for failure reproduction - all snippets will include the User-Agent header.

Performance

Speed up generation of headers, cookies, and formData parameters when their schemas do not define the type keyword. #795

2.5.1 - 2020-09-30

This release contains only documentation updates which are necessary to upload to PyPI.

2.5.0 - 2020-09-27

Added

Stateful testing via Open API links for the pytest runner. #616
Support for GraphQL tests for the pytest runner. #649

Fixed

Progress percentage in the terminal output for “lazy” schemas. #636

Changed

Check name is no longer displayed in the CLI output, since its verbose message is already displayed. This change also simplifies the internal structure of the runner events.
The stateful argument type in the runner.prepare is Optional[Stateful] instead of Optional[str]. Use schemathesis.Stateful enum.

2.4.1 - 2020-09-17

Changed

Hide Case.endpoint from representation. Its representation decreases the usability of the pytest’s output. #719
Return registered functions from register_target and register_check decorators. #721

Fixed

Possible IndexError when a user-defined check raises an exception without a message. #718

2.4.0 - 2020-09-15

Added

Ability to register custom targets for targeted testing. #686

Changed

The AfterExecution event now has path and method fields, similar to the BeforeExecution one. The goal is to make these events self-contained, which improves their usability.

2.3.4 - 2020-09-11

Changed

The default Hypothesis’s deadline setting for tests with schema.parametrize is set to 500 ms for consistency with the CLI behavior. #705

Fixed

Encoding error when writing a cassette on Windows. #708

2.3.3 - 2020-08-04

Fixed

KeyError during the content_type_conformance check if the response has no Content-Type header. #692

2.3.2 - 2020-08-04

Added

Run checks conditionally.

2.3.1 - 2020-07-28

Fixed

IndexError when examples list is empty.

2.3.0 - 2020-07-26

Added

Possibility to generate values for in: formData parameters that are non-bytes or contain non-bytes (e.g., inside an array). #665

Changed

Error message for cases when a path parameter is in the template but is not defined in the parameters list or missing required: true in its definition. #667
Bump minimum required hypothesis-jsonschema version to 0.17.0. This allows Schemathesis to use the custom_formats argument in from_schema calls and avoid using its private API. #684

Fixed

ValueError during sending a request with test payload if the endpoint defines a parameter with type: array and in: formData. #661
KeyError while processing a schema with nullable parameters and in: body. #660
StopIteration during requestBody processing if it has empty “content” value. #673
AttributeError during generation of “multipart/form-data” parameters that have no “type” defined. #675
Support for properties named “$ref” in object schemas. Previously, it was causing TypeError. #672
Generating illegal Unicode surrogates in the path. #668
Invalid development dependency on graphql-server-core package. #658

2.2.1 - 2020-07-22

Fixed

Possible UnicodeEncodeError during generation of Authorization header values for endpoints with basic security scheme. #656

2.2.0 - 2020-07-14

Added

schemathesis.graphql.from_dict loader allows you to use GraphQL schemas represented as a dictionary for testing.
before_load_schema hook for GraphQL schemas.

Fixed

Serialization of non-string parameters. #651

2.1.0 - 2020-07-06

Added

Support for property-level examples. #467

Fixed

Content-type conformance check for cases when Open API 3.0 schemas contain “default” response definitions. #641
Handling of multipart requests for Open API 3.0 schemas. #640
Sending non-file form fields in multipart requests. #647

Removed

Deprecated skip_validation argument to HookDispatcher.apply.
Deprecated _accepts_context internal function.

2.0.0 - 2020-07-01

Changed

BREAKING. Base URL handling. base_url now is treated as one with a base path included. You should pass a full base URL now instead:

schemathesis run --base-url=http://127.0.0.1:8080/api/v2 ...

This value will override basePath / servers[0].url defined in your schema if you use Open API 2.0 / 3.0 respectively. Previously if you pass a base URL like the one above, it was concatenated with the base path defined in the schema, which leads to a lack of ability to redefine the base path. #511

Fixed

Show the correct URL in CLI progress when the base URL is overridden, including the path part. #511
Construct valid URL when overriding base URL with base path. #511

Example:

Base URL in the schema         : http://0.0.0.0:8081/api/v1
`--base-url` value in CLI      : http://0.0.0.0:8081/api/v2
Full URLs before this change   : http://0.0.0.0:8081/api/v2/api/v1/users/  # INVALID!
Full URLs after this change    : http://0.0.0.0:8081/api/v2/users/         # VALID!

Removed

Support for hooks without context argument in the first position.
Hooks registration by name and function. Use register decorators instead. For more details, see the “Customization” section in our documentation.
BaseSchema.with_hook and BaseSchema.register_hook. Use BaseSchema.hooks.apply and BaseSchema.hooks.register instead.

1.10.0 - 2020-06-28

Added

loaders.from_asgi supports making calls to ASGI-compliant application (For example: FastAPI). #521
Support for GraphQL strategies.

Fixed

Passing custom headers to schema loader for WSGI / ASGI apps. #631

1.9.1 - 2020-06-21

Fixed

Schema validation error on schemas containing numeric values in scientific notation without a dot. #629

1.9.0 - 2020-06-20

Added

Pass the original case’s response to the add_case hook.
Support for multiple examples with OpenAPI examples. #589
--verbosity CLI option to minimize the error output. #598
Allow registering function-level hooks without passing their name as the first argument to apply. #618
Support for hook usage via LazySchema / from_pytest_fixture. #617

Changed

Tests with invalid schemas marked as errors, instead of failures. #622

Fixed

Crash during the generation of loosely-defined headers. #621
Show exception information for test runs on invalid schemas with --validate-schema=false command-line option. Before, the output sections for invalid endpoints were empty. #622

1.8.0 - 2020-06-15

Fixed

Tests with invalid schemas are marked as failed instead of passed when hypothesis-jsonschema>=0.16 is installed. #614
KeyError during creating an endpoint strategy if it contains a reference. #612

Changed

Require hypothesis-jsonschema>=0.16. #614
Pass original InvalidSchema text to pytest.fail call.

1.7.0 - 2020-05-30

Added

Support for YAML files in references via HTTPS & HTTP schemas. #600
Stateful testing support via Open API links syntax. #548
New add_case hook. #458
Support for parameter serialization formats in Open API 2 / 3. For example pipeDelimited or deepObject. #599
Support serializing parameters with application/json content-type. #594

Changed

The minimum required versions for Hypothesis and hypothesis-jsonschema are 5.15.0 and 0.11.1 respectively. The main reason is this fix that is required for stability of Open API links feature when it is executed in multiple threads.

1.6.3 - 2020-05-26

Fixed

Support for a colon symbol (:) inside of a header value passed via CLI. #596

1.6.2 - 2020-05-15

Fixed

Partially generated explicit examples are always valid and can be used in requests. #582

1.6.1 - 2020-05-13

Changed

Look at the current working directory when loading hooks for CLI. #586

1.6.0 - 2020-05-10

Added

New before_add_examples hook. #571
New after_init_cli_run_handlers hook. #575

Fixed

Passing workers_num to ThreadPoolRunner leads to always using 2 workers in this worker kind. #579

1.5.1 - 2020-05-08

Fixed

Display proper headers in reproduction code when headers are overridden. #566

1.5.0 - 2020-05-06

Added

Display a suggestion to disable schema validation on schema loading errors in CLI. #531
Filtration of endpoints by operationId via operation_id parameter to schema.parametrize or -O command-line option. #546
Generation of security-related parameters. They are taken from securityDefinitions / securitySchemes and injected to the generated data. It supports generating API keys in headers or query parameters and generating data for HTTP authentication schemes. #540

Fixed

Overriding header values in CLI and runner when headers provided explicitly clash with ones defined in the schema. #559
Nested references resolving in response_schema_conformance check. #562
Nullable parameters handling when they are behind a reference. #542

1.4.0 - 2020-05-03

Added

context argument for hook functions to provide an additional context for hooks. A deprecation warning is emitted for hook functions that do not accept this argument.
A new hook system that allows generic hook dispatching. It comes with new hook locations. For more details, see the “Customization” section in our documentation.
New before_process_path hook.
Third-party compatibility fixups mechanism. Currently, there is one fixup for FastAPI. #503

Deprecated

Hook functions that do not accept context as their first argument. They will become not be supported in Schemathesis 2.0.
Registering hooks by name and function. Use register decorators instead. For more details, see the “Customization” section in our documentation.
BaseSchema.with_hook and BaseSchema.register_hook. Use BaseSchema.hooks.apply and BaseSchema.hooks.register instead.

Fixed

Add missing validate_schema argument to loaders.from_pytest_fixture.
Reference resolving during response schema conformance check. #539

1.3.4 - 2020-04-30

Fixed

Validation of nullable properties in response_schema_conformance check introduced in 1.3.0. #542

1.3.3 - 2020-04-29

Changed

Update pytest-subtests pin to >=0.2.1,<1.0. #537

1.3.2 - 2020-04-27

Added

Show exceptions if they happened during loading a WSGI application. Option --show-errors-tracebacks will display a full traceback.

1.3.1 - 2020-04-27

Fixed

Packaging issue

1.3.0 - 2020-04-27

Added

Storing network logs with --store-network-log=<filename.yaml>. The stored cassettes are based on the VCR format and contain extra information from the Schemathesis internals. #379
Replaying of cassettes stored in VCR format. #519
Targeted property-based testing in CLI and runner. It only supports the response_time target at the moment. #104
Export CLI test results to JUnit.xml with --junit-xml=<filename.xml>. #427

Fixed

Code samples for schemas where body is defined as {"type": "string"}. #521
Showing error causes on internal jsonschema errors during input schema validation. #513
Recursion error in response_schema_conformance check. Because of this change, Endpoint.definition contains a definition where references are not resolved. In this way, it makes it possible to avoid recursion errors in jsonschema validation. #468

Changed

Added indentation & section name to the SUMMARY CLI block.
Use C-extension for YAML loading when it is possible. It can cause more than 10x speedup on schema parsing. Do not show Click’s “Aborted!” message when an error occurs during CLI schema loading.
Add a help message to the CLI output when an internal exception happens. #529

1.2.0 - 2020-04-15

Added

Per-test hooks for modification of data generation strategies. #492
Support for x-example vendor extension in Open API 2.0. #504
Sanity validation for the input schema & loader in runner.prepare. #499

1.1.2 - 2020-04-14

Fixed

Support for custom loaders in runner. Now all built-in loaders are supported as an argument to runner.prepare. #496
from_wsgi loader accepts custom keyword arguments that will be passed to client.get when accessing the schema. #497

1.1.1 - 2020-04-12

Fixed

Mistakenly applied Open API -> JSON Schema Draft 7 conversion. It should be Draft 4. #489
Using wrong validator in response_schema_conformance check. It should be Draft 4 validator. #468

1.1.0 - 2020-04-08

Fixed

Response schema check for recursive schemas. #468

Changed

App loading in runner. Now it accepts application as an importable string, rather than an instance. It is done to make it possible to execute a runner in a subprocess. Otherwise, apps can’t be easily serialized and transferred into another process.
Runner events structure. All data in events is static from now. There are no references to BaseSchema, Endpoint or similar objects that may calculate data dynamically. This is done to make events serializable and not tied to Python object, which decouples any runner consumer from implementation details. It will help make runner usable in more cases (e.g., web application) since events can be serialized to JSON and used in any environment. Another related change is that Python exceptions are not propagated anymore - they are replaced with the InternalError event that should be handled accordingly.

1.0.5 - 2020-04-03

Fixed

Open API 3. Handling of endpoints that contain multipart/form-data media types. Previously only file upload endpoints were working correctly. #473

1.0.4 - 2020-04-03

Fixed

OpenApi30.get_content_types behavior, introduced in 8aeee1a. #469

1.0.3 - 2020-04-03

Fixed

Precedence of produces keywords for Swagger 2.0 schemas. Now, operation-level produces overrides schema-level produces as specified in the specification. #463
Content-type conformance check for Open API 3.0 schemas. #461
Pytest 5.4 warning for test functions without parametrization. #451

1.0.2 - 2020-04-02

Fixed

Handling of fields in paths that are not operations, but allowed by the Open API spec. #457
Pytest 5.4 warning about deprecated Node initialization usage. #451

1.0.1 - 2020-04-01

Fixed

Processing of explicit examples in Open API 3.0 when there are multiple parameters in the same location (e.g. path) contain example value. They are properly combined now. #450

1.0.0 - 2020-03-31

Changed

Move processing of runner parameters to runner.prepare. This change will provide better code reuse since all users of runner (e.g., if you extended it in your project) need some kind of input parameters handling, which was implemented only in Schemathesis CLI. It is not backward-compatible. If you didn’t use runner directly, then this change should not have a visible effect on your use-case.

0.28.0 - 2020-03-31

Fixed

Handling of schemas that use x-* custom properties. #448

Removed

Deprecated runner.execute. Use runner.prepare instead.

0.27.0 - 2020-03-31

Deprecated

runner.execute should not be used, since runner.prepare provides a more flexible interface to test execution.

Removed

Deprecated Parametrizer class. Use schemathesis.from_path as a replacement for Parametrizer.from_path.

0.26.1 - 2020-03-24

Fixed

Limit recursion depth while resolving JSON schema to handle recursion without breaking. #435

0.26.0 - 2020-03-19

Fixed

Filter problematic path template variables containing "/", or "%2F" url encoded. #440
Filter invalid empty "" path template variables. #439
Typo in a help message in the CLI output. #436

0.25.1 - 2020-03-09

Changed

Allow werkzeug >= 1.0.0. #433

0.25.0 - 2020-02-27

Changed

Handling of explicit examples from schemas. Now, if there are examples for multiple locations (e.g., for body and query) then they will be combined into a single example. #424

0.24.5 - 2020-02-26

Fixed

Error during pytest collection on objects with custom __getattr__ method and therefore pass is_schemathesis check. #429

0.24.4 - 2020-02-22

Fixed

Resolving references when the schema is loaded from a file on Windows. #418

0.24.3 - 2020-02-10

Fixed

Not copied validate_schema parameter in BaseSchema.parametrize. Regression after implementing #383
Missing app, location and hooks parameters in schema when used with BaseSchema.parametrize. #416

0.24.2 - 2020-02-09

Fixed

Crash on invalid regular expressions in method, endpoint and tag CLI options. #403
Crash on a non-latin-1 encodable value in the auth CLI option. #404
Crash on an invalid value in the header CLI option. #405
Crash on some invalid URLs in the schema CLI option. #406
Validation of --request-timeout parameter. #407
Crash with --hypothesis-deadline=0 CLI option. #410
Crash with --hypothesis-max-examples=0 CLI option. #412

0.24.1 - 2020-02-08

Fixed

CLI crash on Windows and Python < 3.8 when the schema path contains characters unrepresentable at the OS level. #400

0.24.0 - 2020-02-07

Added

Support for testing of examples in Parameter & Media Type objects in Open API 3.0. #394
--show-error-tracebacks CLI option to display errors’ tracebacks in the output. #391
Support for schema behind auth. #115

Changed

Schemas with GET endpoints accepting body are allowed now if schema validation is disabled (via --validate-schema=false for example). The use-case is for tools like ElasticSearch that use GET requests with non-empty bodies. #383

Fixed

CLI crash when an explicit example is specified in the endpoint definition. #386

0.23.7 - 2020-01-30

Added

-x/--exitfirst CLI option to exit after the first failed test. #378

Fixed

Handling examples of parameters in Open API 3.0. #381

0.23.6 - 2020-01-28

Added

all variant for --checks CLI option to use all available checks. #374

Changed

Use built-in importlib.metadata on Python 3.8. #376

0.23.5 - 2020-01-24

Fixed

Generation of invalid values in Case.cookies. #371

0.23.4 - 2020-01-22

Fixed

Converting exclusiveMinimum & exclusiveMaximum fields to JSON Schema. #367

0.23.3 - 2020-01-21

Fixed

Filter out surrogate pairs from the query string.

0.23.2 - 2020-01-16

Fixed

Prevent KeyError when the response does not have the “Content-Type” header. #365

0.23.1 - 2020-01-15

Fixed

Dockerfile entrypoint was not working as per docs. #361

0.23.0 - 2020-01-15

Added

Hooks for strategy modification. #313
Input schema validation. Use --validate-schema=false to disable it in CLI and validate_schema=False argument in loaders. #110

0.22.0 - 2020-01-11

Added

Show multiple found failures in the CLI output. #266 & #207
Raise a proper exception when the given schema is invalid. #308
Support for None as a value for --hypothesis-deadline. #349

Fixed

Handling binary request payloads in Case.call. #350
Type of the second argument to all built-in checks set to proper Case instead of TestResult. The error was didn’t affect built-in checks since both Case and TestResult had endpoint attribute, and only it was used. However, this fix is not backward-compatible with 3rd party checks.

0.21.0 - 2019-12-20

Added

Support for AioHTTP applications in CLI. #329

0.20.5 - 2019-12-18

Fixed

Compatibility with the latest release of hypothesis-jsonschema and setting its minimal required version to 0.9.13. #338

0.20.4 - 2019-12-17

Fixed

Handling nullable attribute in Open API schemas. #335

0.20.3 - 2019-12-17

Fixed

Usage of the response status code conformance check with old requests version. #330

0.20.2 - 2019-12-14

Fixed

Response schema conformance check for Open API 3.0. #332

0.20.1 - 2019-12-13

Added

Support for response code ranges. #330

0.20.0 - 2019-12-12

Added

WSGI apps support. #31
Case.validate_response for running built-in checks against app’s response. #319

Changed

Checks receive Case instance as a second argument instead of TestResult. This was done for making checks usable in Python tests via Case.validate_response. Endpoint and schema are accessible via case.endpoint and case.endpoint.schema.

0.19.1 - 2019-12-11

Fixed

Compatibility with Hypothesis >= 4.53.2. #322

0.19.0 - 2019-12-02

Added

Concurrent test execution in CLI / runner. #91
update importlib_metadata pin to ^1.1. #315

0.18.1 - 2019-11-28

Fixed

Validation of the base-url CLI parameter. #311

0.18.0 - 2019-11-27

Added

Resolving references in PathItem objects. #301

Fixed

Resolving of relative paths in schemas. #303
Loading string dates as datetime.date objects in YAML loader. #305

0.17.0 - 2019-11-21

Added

Resolving references that point to different files. #294

Changed

Keyboard interrupt is now handled during the CLI run, and the summary is displayed in the output. #295

0.16.0 - 2019-11-19

Added

Display RNG seed in the CLI output to allow test reproducing. #267
Allow specifying seed in CLI.
Ability to pass custom kwargs to the requests.get call in loaders.from_uri.

Changed

Refactor case generation strategies: strategy is not used to generate empty value. #253
Improved error message for invalid path parameter declaration. #255

Fixed

Pytest fixture parametrization via pytest_generate_tests. #280
Support for tests defined as methods. #282
Unclosed requests.Session on calling Case.call without passing a session explicitly. #286

0.15.0 - 2019-11-15

Added

Support for OpenAPI 3.0 server variables (base_path). #40
Support for format: byte. #254
Response schema conformance check in CLI / Runner. #256
Docker image for CLI. #268
Pre-run hooks for CLI. #147
A way to register custom checks for CLI via schemathesis.register_check. #270

Fixed

Not encoded path parameters. #272

Changed

Verbose messages are displayed in the CLI on failed checks. #261

0.14.0 - 2019-11-09

Added

CLI: Support file paths in the schema argument. #119
Checks to verify response status & content type in CLI / Runner. #101

Fixed

Custom base URL handling in CLI / Runner. #248

Changed

Raise an error if the schema has a body for GET requests. #218
Method names are case insensitive during direct schema access. #246

0.13.2 - 2019-11-05

Fixed

IndexError when Hypothesis found inconsistent test results during the test execution in the runner. #236

0.13.1 - 2019-11-05

Added

Support for binary format #197

Fixed

Error that happens when there are no success checks in the statistic in CLI. #237

0.13.0 - 2019-11-03

Added

An option to configure request timeout for CLI / Runner. #204
A help snippet to reproduce errors caught by Schemathesis. #206
Total running time to the CLI output. #181
Summary line in the CLI output with the number of passed / failed / errored endpoint tests. #209
Extra information to the CLI output: schema address, spec version, and base URL. #188

Fixed

Compatibility with Hypothesis 4.42.4+ . #212
Display flaky errors only in the “ERRORS” section and improve CLI output. #215
Handling formData parameters in Case.call. #196
Handling cookies in Case.call. #211

Changed

More readable falsifying examples output. #127
Show exceptions in a separate section of the CLI output. #203
Error message for cases when it is not possible to satisfy schema parameters. It should be more clear now. #216
Do not stop on schema errors related to a single endpoint. #139
Display a proper error message when the schema is not available in CLI / Runner. #214

0.12.2 - 2019-10-30

Fixed

Wrong handling of the base_url parameter in runner and Case.call if it has a trailing slash. #194 and #199
Do not send any payload with GET requests. #200

0.12.1 - 2019-10-28

Fixed

Handling for errors other than AssertionError and HypothesisException in the runner. #189
CLI failing on the case when there are tests, but no checks were performed. #191

Changed

Display the “SUMMARY” section in the CLI output for empty test suites.

0.12.0 - 2019-10-28

Added

Display progress during the CLI run. #125

Fixed

Test server-generated wrong schema when the endpoints option is passed via CLI. #173
Error message if the schema is not found in CLI. #172

Changed

Continue running tests on hypothesis error. #137

0.11.0 - 2019-10-22

Added

LazySchema accepts filters. #149
Ability to register strategies for custom string formats. #94
Generator-based events in the runner module to improve control over the execution flow.
Filtration by tags. #134

Changed

Base URL in schema instances could be reused when it is defined during creation. Now on, the base_url argument in Case.call is optional in such cases. #153
Hypothesis deadline is set to 500ms by default. #138
Hypothesis output is captured separately, without capturing the whole stdout during CLI run.
Disallow empty username in CLI --auth option.

Fixed

User-agent during schema loading. #144
Generation of invalid values in Case.headers. #167

Removed

Undocumented support for file:// URI schema

0.10.0 - 2019-10-14

Added

HTTP Digest Auth support. #106
Support for Hypothesis settings in CLI & Runner. #107
Case.call and Case.as_requests_kwargs convenience methods. #109
Local development server. #126

Removed

Autogenerated runner.StatsCollector.__repr__ to make Hypothesis output more readable.

0.9.0 - 2019-10-09

Added

Test executor collects results of execution. #29
CLI option --base-url for specifying base URL of API. #118
Support for coroutine-based tests. #121
User Agent to network requests in CLI & runner. #130

Changed

CLI command schemathesis run prints result in a more readable way with a summary of passing checks.
Empty header names are forbidden for CLI.
Suppressed hypothesis exception about using example non-interactively. #92

0.8.1 - 2019-10-04

Fixed

Wrap each test in suppress so the runner doesn’t stop after the first test failure.

0.8.0 - 2019-10-04

Added

CLI tool invoked by the schemathesis command. #30
New arguments api_options, loader_options and loader for test executor. #90
A mapping interface for schemas & convenience methods for direct strategy access. #98

Fixed

Runner stopping on the first falsifying example. #99

0.7.3 - 2019-09-30

Fixed

Filtration in lazy loaders.

0.7.2 - 2019-09-30

Added

Support for type “file” for Swagger 2.0. #78
Support for filtering in loaders. #75

Fixed

Conflict for lazy schema filtering. #64

0.7.1 - 2019-09-27

Added

Support for x-nullable extension. #45

0.7.0 - 2019-09-26

Added

Support for the cookie parameter in OpenAPI 3.0 schemas. #21
Support for the formData parameter in Swagger 2.0 schemas. #6
Test executor. #28

Fixed

Using hypothesis.settings decorator with test functions created from from_pytest_fixture loader. #69

0.6.0 - 2019-09-24

Added

Parametrizing tests from a pytest fixture via pytest-subtests. #58

Changed

Rename module readers to loaders.
Rename parametrize parameters. filter_endpoint to endpoint and filter_method to method.

Removed

Substring match for method/endpoint filters. To avoid clashing with escaped chars in endpoints keys in schemas.

0.5.0 - 2019-09-16

Added

Generating explicit examples from the schema. #17

Changed

Schemas are loaded eagerly from now on. Using schemathesis.from_uri implies network calls.

Deprecated

Using Parametrizer.from_{path,uri} is deprecated, use schemathesis.from_{path,uri} instead.

Fixed

Body resolving during test collection. #55

0.4.1 - 2019-09-11

Fixed

Possibly unhandled exception during hasattr check in is_schemathesis_test.

0.4.0 - 2019-09-10

Fixed

Resolving all inner references in objects. #34

Changed

jsonschema.RefResolver is now used for reference resolving. #35

0.3.0 - 2019-09-06

Added

Parametrizer.from_uri method to construct parametrizer instances from URIs. #24

Removed

Possibility to use Parametrizer.parametrize and custom Parametrizer kwargs for passing config options to hypothesis.settings. Use hypothesis.settings decorators on tests instead.

0.2.0 - 2019-09-05

Added

Open API 3.0 support. #10
“header” parameters. #7

Changed

Handle errors during collection / executions as failures.
Use re.search for pattern matching in filter_method/filter_endpoint instead of fnmatch. #18
Case.body contains properties from the target schema, without the extra level of nesting.

Fixed

KeyError on collection when “basePath” is absent. #16

0.1.0 - 2019-06-28

Initial public release