Changelog

Unreleased - TBD

3.27.0 - 2024-04-14

Added

• Case.as_transport_kwargs method to simplify the creation of transport-specific keyword arguments for sending requests.

Changed

• Make Case.call work with ASGI & WSGI applications.

• Extend the JUnit XML report format to match CLI output including skipped tests, code samples, and more.

Deprecated

• Case.call_wsgi & Case.call_asgi in favor of Case.call.

• Case.as_requests_kwargs & Case.as_werkzeug_kwargs in favor of Case.as_transport_kwargs.

3.26.2 - 2024-04-06

Added

• Support for pyrate-limiter>=3.0.

Fixed

• Excluding \x00 bytes as a result of probes.

3.26.1 - 2024-04-04

Added

• Store time needed to generate each test case.

Fixed

• InvalidArgument when using from_pytest_fixture with parametrized pytest fixtures and Hypothesis settings. #2115

3.26.0 - 2024-03-21

Added

• Support for per-media type data generators. #962

• Support for application/yaml & text/yml media types in YAMLSerializer.

• EXPERIMENTAL: Run automatic schema optimization & format inference if CLI is authenticated in Schemathesis.io.

Fixed

• Not resolving references in nested security schemes. #2073

Changed

• Improve error message when the minimum possible example is too large.

3.25.6 - 2024-03-02

Fixed

• Not respecting allow_x00 and codec configs options during filling gaps in explicit examples.

• Internal error when sending multipart/form-data requests when the schema defines the */* content type.

• Internal error when YAML payload definition contains nested binary format.

• Internal error when an Open API 2.0 schema contains no swagger key and the schema version is forced.

Changed

• Indicate API probing results in CLI.

3.25.5 - 2024-02-29

Fixed

• Incorrect error message when the code inside the hook module raises ImportError. #2074

• Compatibility with Hypothesis >6.98.14

• Not respecting allow_x00 and codec configs options for data generation in some cases. #2072

3.25.4 - 2024-02-25

Changed

• Improve error message when the minimum possible example is too large.

3.25.3 - 2024-02-22

Added

• Added __contains__ method to ParameterSet for easier parameter checks in hooks.

Changed

• Suppress TLS-related warnings during API probing.

3.25.2 - 2024-02-21

Added

• Run automatic probes to detect the application capabilities before testing. They allow for more accurate data generation, reducing false positive test failures. #1840

• Support running async Python tests with trio. #1872

Fixed

• Invalid spec detection if the experimental support for Open API 3.1 is not explicit explicitly enabled.

• Invalid spec detection if the input YAML contains not allowed characters.

• AttributeError when using the experimental support for Open API 3.1 with multiple workers.

• Do not skip API operation if it is still possible to generate positive tests when -D all is passed.

3.25.1 - 2024-02-10

Changed

• CLI: Enhanced Open API 3.1.0 support messaging, now suggesting --experimental=openapi-3.1 option for partial compatibility.

Fixed

• Not reporting errors during testing of explicit examples when data generation is flaky.

3.25.0 - 2024-02-07

Added

• --hypothesis-no-phases CLI option to disable Hypothesis testing phases. #1324

• Support for loading GraphQL schemas from JSON files that contain the __schema key.

• Response validation for GraphQL APIs.

• Support tag in filters for custom auth.

• Support for testing examples inside anyOf / oneOf / allOf keywords.

• Support for the text/xml media type in XMLSerializer.

• Support for the text/json media type in JSONSerializer.

• Support for pytest 8.

Changed

• CLI: Eagerly check for permissions when writing output to a file, including JUnit XML and other reports.

• Python: Explicitly note that combining @schema.given with explicit examples from the spec is not supported. #1217

• Clarify error message when a state machine has no transitions. #1992

• Do not consider missing the paths key an error for Open API 3.1.

• Improved formatting of multiple errors within the same API operation.

• Allow arbitrary objects in array for application/x-www-form-urlencoded payloads.

Deprecated

• The --contrib-unique-data CLI option and the corresponding schemathesis.contrib.unique_data hook. The concept of this feature does not fit the core principles of Hypothesis where strategies are configurable on a per-example basis but this feature implies uniqueness across examples. This leads to cryptic error messages about external state and flaky test runs, therefore it will be removed in Schemathesis 4.0

Fixed

• CLI: Do not duplicate the error message in the output when the error has no traceback and the --show-trace option is provided.

• Open API: Internal error on path templates that contain . inside path parameters.

• Open API: YAML serialization of data generated for schemas with format: binary.

• Create parent directories when saving JUnit XML reports and other file-based output. #1995

• Internal error when an API operation contains multiple parameters with the same name and some of them contain the examples keyword.

• Internal error during query parameter generation on schemas that do not contain the type keyword.

• Example generation for request body parameters using $ref.

• Generating examples for properties that have deeply nested $ref.

• Generating examples for properties with boolean sub-schemas.

• Validating responses with boolean sub-schemas on Open API 3.1.

• TypeError on non-string pattern values. This could happen on values in YAML, such that when not quoted, they are parsed as non-strings.

• Testing examples requiring unsupported payload media types resulted in an internal error. These are now correctly reported as errors

• Internal error on unsupported regular expressions in inside properties during example generation.

• Missing XML definitions when the media type contains options like application/xml; charset=utf-8.

• Unhandled timeout while reading the response payload.

• Internal error when the header example in the schema is not a valid header.

• Handle KeyError during state machine creation.

• Deduplicate network errors that contain unique URLs in their messages.

• Not reporting multiple errors of different kinds at the same API operation.

• Group similar errors within the same API operation.

3.24.3 - 2024-01-23

Fixed

• Incorrect base URL handling for GraphQL schemas. #1987

3.24.2 - 2024-01-23

Added

• Python: Shortcut to create strategies for all operations or a subset of them via schema.as_strategy() and schema["/path/"].as_strategy(). #1982

Changed

• Python: Cleaner repr for GraphQL & Open API schemas.

• GraphQL: Show suggestion when a field is not found in schema["Query"][field_name].

Fixed

• Filter out test cases that can not be serialized when the API operation requires application/x-www-form-urlencoded. #1306

3.24.1 - 2024-01-22

Changed

• Cleanup SSL error messages.

Fixed

• Internal error when an unresolvable pointer occurs during data generation.

• Internal errors when references lead to non-objects.

• Missing schema.override on schemas created via the from_pytest_fixture loader.

• Not calling hooks for query / cookies / headers in GraphQL schemas. #1978

• Inability to access individual operations in GraphQL schema objects. #1976

3.24.0 - 2024-01-21

Added

• CLI options for overriding Open API parameters in test cases. #1676

• A way to override Open API parameters the pytest integration with the override decorator. #8

• Open API: Support for the examples keyword inside individual property schemas. #1730, #1320

• Open API: Extract explicit examples from all defined media types. #921

Changed

• Raise an error if it is not possible to generate explicit examples. #1771

• Avoid using the deprecated cgi module. #1962

Fixed

• Open API: Properly combine multiple explicit examples extracted from examples and example fields. #1360

• Open API: Ignoring examples referenced via the $ref keyword. #1692

3.23.1 - 2024-01-14

Changed

• Do not auto-detect spec if the --force-schema-version CLI option is present.

• Do not assume GraphQL when trying to auto-detect spec in an empty input file.

Fixed

• Internal error when the schema file is empty.

3.23.0 - 2023-12-29

Added

• New CLI option --contrib-openapi-fill-missing-examples to automatically generate random examples for API operations that lack explicit examples. #1728, #1376

• New CLI option --request-proxy to set HTTP(s) proxies for network calls. #1723

Changed

• Validate --generation-codec values in CLI.

• Do not deepcopy responses before passing to checks. They are not supposed to be mutated inside checks.

• Pin anyio to <4 due to incompatibility with starlette-testclient.

Fixed

• Internal error when the configured proxy is not available.

• Not using examples from shared parameters. #1729, #1513

3.22.1 - 2023-12-04

Fixed

• Internal error during network error handling. #1933

3.22.0 - 2023-12-03

Added

• Support for hypothesis-jsonschema==0.23.

• A way to control what characters are used for string generation. #1142, #1286, #1562, #1668.

• Display the total number of collected links in the CLI output. #1383.

• arm64 Docker builds. #1740.

• Use Python 3.12 in Docker images.

• Store Docker image name in Metadata.

• GraphQL scalar strategies for Date, Time, DateTime, IP, IPv4, IPv6, Long, BigInt and UUID. #1690

Changed

• Bump the minimum supported Hypothesis version to 6.84.3.

• Bump the minimum supported jsonschema version to 4.18.0.

• Bump the minimum supported hypothesis_graphql version to 0.11.0.

• Use the same random seed for all tests in CLI. #1384.

• Improve serialization error messages in CLI.

• Store skip reason in the runner events.

• Build bookworm-based Debian Docker images instead of buster-based.

• Improve error message on unknown scalar types in GraphQL.

• Better auto-detection of GraphQL schemas.

• Display parsing errors for schemas that are expected to be JSON or YAML.

Deprecated

• Using the --show-errors-tracebacks CLI option. Use --show-trace instead.

Fixed

• Internal error when a non-existing schema file is passed together with --base-url. #1912.

• Internal error during schema loading from invalid URLs.

• Ignore incompatible GraphQL checks in CLI rather than fail the whole test run. #1918.

Removed

• Support for Python 3.7.

• Unnecessary dependencies on typing-extensions and importlib-metadata.

3.21.2 - 2023-11-27

Added

• Support for hypothesis>=6.90.1.

3.21.1 - 2023-11-16

Added

• Basic support for httpx in Case.validate_response.

Changed

• Restore the ability to import NOT_SET from schemathesis.utils. #1890

3.21.0 - 2023-11-09

Added

• Add Python 3.12 compatibility. #1809

• Separate command for report upload.

Changed

• Generated binary data inside Case.body is wrapped with a custom wrapper - Binary in order to simplify compatibility with hypothesis-jsonschema.

• Do not modify Case.body inside Case.as_requests_kwargs when serializing multipart data.

• INTERNAL: Moved heavy imports inside functions to improve CLI startup time by 4.3x, not affecting overall execution speed. #1509

• Improved messaging for loading hooks and WSGI application issues.

• Refined documentation strings for CLI options.

• Added an error message if an internal error happened inside CLI event handler.

• Unified CLI messages for errors arising from network, configuration, and Hypothesis-related issues. #1600, #1607, #1782, #1835

• Try to validate JSON data even if there is no proper Content-Type header. #1787

• Refined failure reporting for clarity. #1784, #1785, #1790, #1799, #1800

3.20.2 - 2023-10-27

Fixed

• Incorrect documentation & implementation for enabling experimental features in pytest.

3.20.1 - 2023-10-20

Changed

• Improved CLI error messages for missing or invalid arguments.

3.20.0 - 2023-10-18

Added

• Support for application/xml serialization based on Open API schema definitions. #733

• Hook shortcuts (filter_query, map_header, etc.) to minimize boilerplate in extensions. #1673

• Support for colored output from docker container. #1170

• A way to disable suggestion for visualizing test results via the SCHEMATHESIS_REPORT_SUGGESTION=0 environment variable. #1802

• Automatic FastAPI fixup injecting for ASGI loaders, eliminating the need for manual setup. #1797

• Support for body hooks in GraphQL schemas, enabling custom filtering or modification of queries and mutations. #1464

• New filter_operations hook to conditionally include or exclude specific API operations from being tested.

• Added contains method to ParameterSet for easier parameter checks in hooks. #1789

• Automatic sanitization of sensitive data in the output is now enabled by default. This feature can be disabled using the --sanitize-output=false CLI option. For more advanced customization, use schemathesis.sanitizing.configure(). #1794

• --experimental=openapi-3.1 CLI option for experimental support of OpenAPI 3.1. This enables compatible JSON Schema validation for responses, while data generation remains OpenAPI 3.0-compatible. #1820

Note: Experimental features can change or be removed in any minor version release.

Changed

• Support Werkzeug>=3.0. #1819

• Refined generated reproduction code and shortened X-Schemathesis-TestCaseId for easier debugging. #1801

• Add case as the first argument to AuthContext.set. Previous calling convention is still supported. #1788

• Disable the ‘explain’ phase in Hypothesis to improve performance. #1808

• Simplify Python code samples for failure reproduction.

• Do not display InsecureRequestWarning in CLI output if the user explicitly provided --request-tls-verify=false. #1780

• Enhance CLI output for schema loading and internal errors, providing clearer diagnostics and guidance. #1781, #1517, #1472

Before:

Failed to load schema from https://127.0.0.1:6423/openapi.json
You can use `--wait-for-schema=NUM` to wait for a maximum of NUM seconds on the API schema availability.

Error: requests.exceptions.SSLError: HTTPSConnectionPool(host='localhost', port=6423): Max retries exceeded with url: /openapi.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:992)')))

Add this option to your command line parameters to see full tracebacks: --show-errors-tracebacks

After:

Schema Loading Error

SSL verification problem

    [SSL: WRONG_VERSION_NUMBER] wrong version number

Tip: Bypass SSL verification with `--request-tls-verify=false`.

Deprecated

• Defining AuthProvider.get with a single context argument. The support will be removed in Schemathesis 4.0.

Fixed

• Fixed type hint for AuthProvider. #1776

• Do not skip negative tests if the generated value is None.

• Lack of execution for ASGI events during testing. #1305, #1727

• Confusing error message when trying to load schema from a non-existing file. #1602

• Reflect disabled TLS verification in generated code samples. #1054

• Generated cURL commands now include the Content-Type header, which was previously omitted. #1783

• Improperly serialized headers in SerializedHistoryEntry.case.extra_headers.

Performance

• Optimize event data emission by postponing code sample generation, resulting in a ~4% reduction in the emitted events data size.

Removed

• Unused SerializedError.example attribute. It used to be populated for flaky errors before they became regular failures.

• Unused TestResult.overridden_headers attribute.

3.19.7 - 2023-09-03

Fixed

• Unsatisfiable error for multiple security schemes applied to the same API operation and an explicit Authorization header. #1763

3.19.6 - 2023-08-14

Fixed

• Broken --report CLI argument under click>=8.1.4. #1753

3.19.5 - 2023-06-03

Fixed

• Do not raise Unsatisfiable when explicit headers are provided for negative tests.

• Do not raise Unsatisfiable when no headers can be negated.

3.19.4 - 2023-06-03

Fixed

• Improved handling of negative test scenarios by not raising Unsatisfiable when path parameters cannot be negated but other parameters can be negated.

3.19.3 - 2023-05-25

Changed

• Support requests<3. #1742

• Bump the minimum supported Hypothesis version to 6.31.6 to reflect requirement from hypothesis-jsonschema.

Fixed

• HypothesisDeprecationWarning regarding deprecated HealthCheck.all(). #1739

3.19.2 - 2023-05-20

Added

• You can now provide a tuple of checks to exclude when validating a response.

3.19.1 - 2023-04-26

Changed

• Support requests<2.29.

Fixed

• Passing params / cookies to case.call causing TypeError. #1734

Removed

• Direct dependency on attrs.

3.19.0 - 2023-03-22

Added

• Schemathesis now supports custom authentication mechanisms from the requests library. You can use schemathesis.auth.set_from_requests to set up Schemathesis CLI with any third-party authentication implementation that works with requests. #1700

import schemathesis
from requests_ntlm import HttpNtlmAuth

schemathesis.auth.set_from_requests(HttpNtlmAuth("domain\\username", "password"))

• Ability to apply authentication conditionally to specific API operations using a combination of @schemathesis.auth.apply_to() and @schemathesis.auth.skip_for() decorators.

import schemathesis


# Apply auth only for operations that path starts with `/users/` but not the `POST` method
@schemathesis.auth().apply_to(path_regex="^/users/").skip_for(method="POST")
class MyAuth:
    ...

• Add a convenience mapping-like interface to OperationDefinition including indexing access, the get method, and “in” support.

• Request throttling via the --rate-limit CLI option. #910

Changed

• Unified Schemathesis custom authentication usage via the schema.auth decorator, replacing the previous schema.auth.register and schema.auth.apply methods:

import schemathesis

schema = schemathesis.from_uri("https://example.schemathesis.io/openapi.json")


# Schema-level auth
# Before: @schema.auth.register()
@schema.auth()
class MyAuth:
    ...


# Test-level auth
# Before: @schema.auth.apply(MyAuth)
@schema.auth(MyAuth)
@schema.parametrize()
def test_api(case):
    ...

Fixed

• Handling of query parameters and cookies passed to case.call and query parameters passed to case.call_wsgi. The user-provided values are now merged with the data generated by Schemathesis, instead of overriding it completely. #1705

• Parameter definition takes precedence over security schemes with the same name.

• Unsatisfiable error when explicit header name passed via CLI clashes with the header parameter name. #1699

• Not using the port keyword argument in schema loaders during API schema loading. #1721

3.18.5 - 2023-02-18

Added

• Support for specifying the path to load hooks from via the SCHEMATHESIS_HOOKS environment variable. #1702.

Deprecated

• Use of the --pre-run CLI option for loading hooks. Use the SCHEMATHESIS_HOOKS environment variable instead.

3.18.4 - 2023-02-16

Changed

• Support any Werkzeug 2.x in order to allow mitigation of CVE-2023-25577. #1695

3.18.3 - 2023-02-12

Added

• APIStateMachine.run method to simplify running stateful tests.

Changed

• Improved quality of generated test sequences by updating state machines in Schemathesis to always run a minimum of two steps during testing. #1627 If you use hypothesis.stateful.run_state_machine_as_test to run your stateful tests, please use the run method on your state machine class instead. This change requires upgrading Hypothesis to at least version 6.68.1.

3.18.2 - 2023-02-08

Performance

• Modify values in-place inside built-in map functions as there is no need to copy them.

• Update hypothesis-jsonschema to 0.22.1 for up to 30% faster data generation in some workflows.

3.18.1 - 2023-02-06

Changed

• Stateful testing: Only make stateful requests when stateful data is available from another operation. This change significantly reduces the number of API calls that likely will fail because of absence of stateful data. #1669

Performance

• Do not merge component schemas into the currently tested schema if they are not referenced by it. Originally all schemas were merged to make them visible to hypothesis-jsonschema, but they imply significant overhead. #1180

• Use a faster, specialized version of deepcopy.

3.18.0 - 2023-02-01

Added

• Extra information to VCR cassettes.

• The --contrib-unique-data CLI option that forces Schemathesis to generate unique test cases only. This feature is also available as a hook in schemathesis.contrib.unique_data.

• A few decorators & functions that provide a simpler API to extend Schemathesis:

  • schemathesis.auth() for authentication providers;

  • schemathesis.check for checks;

  • schemathesis.hook & BaseSchema.hook for hooks;

  • schemathesis.serializer for serializers;

  • schemathesis.target for targets;

  • schemathesis.openapi.format for custom OpenAPI formats.

  • schemathesis.graphql.scalar for GraphQL scalars.

• Open API: UUID format generation via the schemathesis.contrib.openapi.formats.uuid extension You could enable it via the --contrib-openapi-formats-uuid CLI option.

Changed

• Build: Switch the build backend to Hatch.

• Relax requirements for attrs. #1643

• Avoid occasional empty lines in cassettes.

Deprecated

• schemathesis.register_check in favor of schemathesis.check.

• schemathesis.register_target in favor of schemathesis.target.

• schemathesis.register_string_format in favor of schemathesis.openapi.format.

• schemathesis.graphql.register_scalar in favor of schemathesis.graphql.scalar.

• schemathesis.auth.register in favor of schemathesis.auth.

Fixed

• Remove recursive references from the last reference resolution level. It works on the best effort basis and does not cover all possible cases. #947

• Invalid cassettes when headers contain characters with a special meaning in YAML.

• Properly display flaky deadline errors.

• Internal error when the utf8_bom fixup is used for WSGI apps.

• Printing header that are set explicitly via get_call_kwargs in stateful testing. #828

• Display all explicitly defined headers in the generated cURL command.

• Replace starlette.testclient.TestClient with starlette_testclient.TestClient to keep compatibility with newer starlette versions. #1637

Performance

• Running negative tests filters out less data.

• Schema loading: Try a faster loader first if an HTTP response or a file is expected to be JSON.

3.17.5 - 2022-11-08

Added

• Python 3.11 support. #1632

Fixed

• Allow Werkzeug<=2.2.2. #1631

3.17.4 - 2022-10-19

Fixed

• Appending an extra slash to the / path. #1625

3.17.3 - 2022-10-10

Fixed

• Missing httpx dependency. #1614

3.17.2 - 2022-08-27

Fixed

• Insufficient timeout for report uploads.

3.17.1 - 2022-08-19

Changed

• Support requests==2.28.1.

3.17.0 - 2022-08-17

Added

• Support for exception groups in newer Hypothesis versions. #1592

• A way to generate negative and positive test cases within the same CLI run via -D all.

Fixed

• Allow creating APIs in Schemathesis.io by name when the schema is passed as a file.

• Properly trim tracebacks on Hypothesis>=6.54.0.

• Skipping negative tests when they should not be skipped.

Changed

• pytest: Generate positive & negative within the same test node.

• CLI: Warning if there are too many HTTP 403 API responses.

• Runner: BeforeExecution.data_generation_method and AfterExecution.data_generation_method changed to lists of DataGenerationMethod as the same test may contain data coming from different data generation methods.

3.16.5 - 2022-08-11

Fixed

• CLI: Hanging on CTRL-C when --report is enabled.

• Internal error when GraphQL schema has its root types renamed. #1591

3.16.4 - 2022-08-09

Changed

• Suggest using --wait-for-schema if API schema is not available.

3.16.3 - 2022-08-08

Added

• CLI: --max-failures=N option to exit after first N failures or errors. #1580

• CLI: --wait-for-schema=N option to automatically retry schema loading for N seconds. #1582

• CLI: Display old and new payloads in st replay when the -v option is passed. #1584

Fixed

• Internal error on generating negative tests for query parameters with explode: true.

3.16.2 - 2022-08-05

Added

• CLI: Warning if ALL API responses are HTTP 404.

• The after_load_schema hook, which is designed for modifying the loaded API schema before running tests. For example, you can use it to add Open API links to your schema via schema.add_link.

• New utf8_bom fixup. It helps to mitigate JSON decoding errors inside the response_schema_conformance check when payload contains BOM. #1563

Fixed

• Description of -v or --verbosity option for CLI.

Changed

• Execute before_call / after_call hooks inside the call_* methods. It makes them available for the pytest integration.

3.16.1 - 2022-07-29

Added

• CLI: Warning if the API returns too many HTTP 401.

• Add SCHEMATHESIS_BASE_URL environment variable for specifying --base-url in CLI.

• Collect anonymyzed CLI usage telemetry when reports are uploaded. We do not collect any free-form values you use in your CLI, except for header names. Instead, we measure how many times you use each free-form option in this command. Additionally we count all non-default hook types only by hook name.

Important

You can disable usage this with the --schemathesis-io-telemetry=false CLI option or the SCHEMATHESIS_TELEMETRY=false environment variable.

3.16.0 - 2022-07-22

Added

• Report uploading to Schemathesis.io via the --report CLI option.

Changed

• Do not validate schemas by default in the pytest integration.

• CLI: Display test run environment metadata only if -v is provided.

• CLI: Do not display headers automatically added by requests in code samples.

Fixed

• Do not report optional headers as missing.

• Compatibility with hypothesis>=6.49. #1538

• Handling of unittest.case.SkipTest emitted by newer Hypothesis versions.

• Generating invalid headers when their schema has array or object types.

Removed

• Previously, data was uploaded to Schemathesis.io when the proper credentials were specified. This release removes this behavior. From now on, every upload requires the explicit --report CLI option.

• Textual representation of HTTP requests in CLI output in order to decrease verbosity and avoid showing the same data in multiple places.

3.15.6 - 2022-06-23

Fixed

• Do not discard dots (.) in OpenAPI expressions during parsing.

3.15.5 - 2022-06-21

Fixed

• TypeError when using --auth-type=digest in CLI.

3.15.4 - 2022-06-06

Added

• Support generating data for Open API request payloads with wildcard media types. #1526

Changed

• Mark tests as skipped if there are no explicit examples and --hypothesis-phases=explicit is used. #1323

• Parse all YAML mapping keys as strings, ignoring the YAML grammar rules. For example, on: true will be parsed as {"on": True} instead of {True: True}. Even though YAML does not restrict keys to strings, in the Open API and JSON Schema context, this restriction is implied because the underlying data model comes from JSON.

• INTERNAL: Improve flexibility of event serialization.

• INTERNAL: Store request / response history in SerializedCheck.

3.15.3 - 2022-05-28

Fixed

• Deduplication of failures caused by malformed JSON payload. #1518

• Do not re-raise InvalidArgument exception as InvalidSchema in non-Schemathesis tests. #1514

3.15.2 - 2022-05-09

Fixed

• Avoid generating negative query samples that requests will treat as an empty query.

• Editable installation via pip.

3.15.1 - 2022-05-03

Added

• OpenAPI: Expose APIOperation.get_security_requirements that returns a list of security requirements applied to the API operation

• Attach originally failed checks to “grouped” exceptions.

Fixed

• Internal error when Schemathesis doesn’t have permission to create its hosts.toml file.

• Do not show internal Hypothesis warning multiple times when the Hypothesis database directory is not usable.

• Do not print not relevant Hypothesis reports when run in CI.

• Invalid verbose_name value in SerializedCase for GraphQL tests.

3.15.0 - 2022-05-01

Added

• GraphQL: Mutations supports. Schemathesis will generate random mutations by default from now on.

• GraphQL: Support for registering strategies to generate custom scalars.

• Custom auth support for schemas created via from_pytest_fixture.

Changed

• Do not encode payloads in cassettes as base64 by default. This change makes Schemathesis match the default Ruby’s VCR behavior and leads to more human-readable cassettes. Use --cassette-preserve-exact-body-bytes to restore the old behavior. #1413

• Bump hypothesis-graphql to 0.9.0.

• Avoid simultaneous authentication requests inside auth providers when caching is enabled.

• Reduce the verbosity of pytest output. A few internal frames and the “Falsifying example” block are removed from the output.

• Skip negative tests on API operations that are not possible to negate. #1463

• Make it possible to generate negative tests if at least one parameter can be negated.

• Treat flaky errors as failures and display full report about the failure. #1081

• Do not duplicate failing explicit example in the HYPOTHESIS OUTPUT CLI output section. #881

Fixed

• GraphQL: Semantically invalid queries without aliases.

• GraphQL: Rare crashes on invalid schemas.

• Internal error inside BaseOpenAPISchema.validate_response on requests>=2.27 when response body contains malformed JSON. #1485

• schemathesis.from_pytest_fixture: Display each failure if Hypothesis found multiple of them.

Performance

• GraphQL: Over 2x improvement from internal optimizations.

3.14.2 - 2022-04-21

Added

• Support for auth customization & automatic refreshing. #966

3.14.1 - 2022-04-18

Fixed

• Using @schema.parametrize with test methods on pytest>=7.0.

3.14.0 - 2022-04-17

Added

• Open API link name customization via the name argument to schema.add_link.

• st as an alias to the schemathesis command line entrypoint.

• st auth login / st auth logout to authenticate with Schemathesis.io.

• X-Schemathesis-TestCaseId header to help to distinguish test cases on the application side. #1303

• Support for comma separated lists in the --checks CLI option. #1373

• Hypothesis Database configuration for CLI via the --hypothesis-database option. #1326

• Make the SCHEMA CLI argument accept API names from Schemathesis.io.

Changed

• Enable Open API links traversal by default. To disable it, use --stateful=none.

• Do not validate API schema by default. To enable it back, use --validate-schema=true.

• Add the api_name CLI argument to upload data to Schemathesis.io.

• Show response status code on failing checks output in CLI.

• Improve error message on malformed Open API path templates (like /foo}/). #1372

• Improve error message on malformed media types that appear in the schema or in response headers. #1382

• Relax dependencies on pyyaml and click.

• Add --cassette-path that is going to replace --store-network-log. The old option is deprecated and will be removed in Schemathesis 4.0

Fixed

• Show the proper Hypothesis configuration in the CLI output. #1445

• Missing source attribute in the Case.partial_deepcopy implementation. #1429

• Duplicated failure message from content_type_conformance and response_schema_conformance checks when the checked response has no Content-Type header. #1394

• Not copied case & response inside Case.validate_response.

• Ignored pytest.mark decorators when they are applied before schema.parametrize if the schema is created via from_pytest_fixture. #1378

3.13.9 - 2022-04-14

Fixed

• Compatibility with pytest-asyncio>=0.17.1. #1452

3.13.8 - 2022-04-05

Fixed

• Missing media_type in the Case.partial_deepcopy implementation. It led to missing payload in failure reproduction code samples.

3.13.7 - 2022-04-02

Added

• Support for Hypothesis>=6.41.0. #1425

3.13.6 - 2022-03-31

Changed

• Deep-clone Response instances before passing to check functions.

3.13.5 - 2022-03-31

Changed

• Deep-clone Case instances before passing to check functions.

3.13.4 - 2022-03-29

Added

• Support for Werkzeug>=2.1.0. #1410

Changed

• Validate requests kwargs to catch cases when the ASGI integration is used, but the proper ASGI client is not supplied. #1335

3.13.3 - 2022-02-20

Added

• --request-tls-verify CLI option for the replay command. It controls whether Schemathesis verifies the server’s TLS certificate. You can also pass the path to a CA_BUNDLE file for private certs. #1395

• Support for client certificate authentication with --request-cert and --request-cert-key arguments for the replay command.

3.13.2 - 2022-02-16

Changed

• Use Schemathesis default User-Agent when communicating with SaaS.

Fixed

• Use the same correlation_id in BeforeExecution and AfterExecution events if the API schema contains an error that causes an InvalidSchema exception during test execution.

• Use full_path in error messages in recoverable schema-level errors. It makes events generated in such cases consistent with usual events.

3.13.1 - 2022-02-10

Added

• APIOperation.iter_parameters helper to iterate over all parameters.

Fixed

• Properly handle error if Open API parameter doesn’t have content or schema keywords.

3.13.0 - 2022-02-09

Changed

• Update integration with Schemathesis.io.

• Always show traceback for errors in Schemathesis.io integration.

3.12.3 - 2022-01-13

Fixed

• Generating illegal unicode surrogates in queries. #1370

3.12.2 - 2022-01-12

Fixed

• Not-escaped single quotes in generated Python code samples. #1359

3.12.1 - 2021-12-31

Fixed

• Improper handling of base_url in call_asgi, when the base URL has a non-empty base path. #1366

3.12.0 - 2021-12-29

Changed

• Upgrade typing-extensions to >=3.7,<5.

• Upgrade jsonschema to ^4.3.2.

• Upgrade hypothesis-jsonschema to >=0.22.0.

Fixed

• Generating values not compliant with the ECMAScript regex syntax. #1350, #1241.

Removed

• Support for Python 3.6.

3.11.7 - 2021-12-23

Added

• Support for Python 3.10. #1292

3.11.6 - 2021-12-20

Added

• Support for client certificate authentication with --request-cert and --request-cert-key arguments. #1173

• Support for readOnly and writeOnly Open API keywords. #741

3.11.5 - 2021-12-04

Changed

• Generate tests for API operations with the HTTP TRACE method on Open API 2.0.

3.11.4 - 2021-12-03

Changed

• Add AfterExecution.data_generation_method.

• Minor changes to the Schemathesis.io integration.

3.11.3 - 2021-12-02

Fixed

• Silently failing to detect numeric status codes when the schema contains a shared parameters key. #1343

• Not raising an error when tests generated by schemas loaded with from_pytest_fixture match no API operations. #1342

3.11.2 - 2021-11-30

Changed

• Use name & data_generation_method parameters to subtest context instead of path & method. It allows the end-user to disambiguate among subtest reports.

• Raise an error if a test function wrapped with schema.parametrize matches no API operations. #1336

Fixed

• Handle KeyboardInterrupt that happens outside of the main test loop inside the runner. It makes interrupt handling consistent, independent at what point it happens. #1325

• Respect the data_generation_methods config option defined on a schema instance when it is loaded via from_pytest_fixture. #1331

• Ignored hooks defined on a schema instance when it is loaded via from_pytest_fixture. #1340

3.11.1 - 2021-11-20

Changed

• Update click and PyYaml dependency versions. #1328

3.11.0 - 2021-11-03

Changed

• Show cURL code samples by default instead of Python. #1269

• Improve reporting of jsonschema errors which are caused by non-string object keys.

• Store data_generation_method in BeforeExecution.

• Use case-insensitive dictionary for Case.headers. #1280

Fixed

• Pass data_generation_method to Case for GraphQL schemas.

• Generation of invalid headers in some cases. #1142

• Unescaped quotes in generated Python code samples on some schemas. #1030

Performance

• Dramatically improve CLI startup performance for large API schemas.

• Open API 3: Inline only components/schemas before passing schemas to hypothesis-jsonschema.

• Generate tests on demand when multiple workers are used during CLI runs. #1287

3.10.1 - 2021-10-04

Added

• DataGenerationMethod.all shortcut to get all possible enum variants.

Fixed

• Unresolvable dependency due to incompatible changes in the new hypothesis-jsonschema release. #1290

3.10.0 - 2021-09-13

Added

• Optional integration with Schemathesis.io.

• New before_init_operation hook.

• INTERNAL. description attribute for all parsed parameters inside APIOperation.

• Timeouts when loading external schema components or external examples.

Changed

• Pin werkzeug to >=0.16.0.

• INTERNAL. OpenAPI20CompositeBody.definition type to List[OpenAPI20Parameter].

• Open API schema loaders now also accept single DataGenerationMethod instances for the data_generation_methods argument. #1260

• Improve error messages when the loaded API schema is not in JSON or YAML. #1262

Fixed

• Internal error in make_case calls for GraphQL schemas.

• TypeError on case.call with bytes data on GraphQL schemas.

• Worker threads may not be immediately stopped on SIGINT. #1066

• Re-used referenced objects during inlining. Now they are independent.

• Rewrite not resolved remote references to local ones. #986

• Stop worker threads on failures with exit_first enabled. #1204

• Properly report all failures when custom checks are passed to case.validate_response.

Performance

• Avoid using filters for header values when is not necessary.

3.9.7 - 2021-07-26

Added

• New process_call_kwargs CLI hook. #1233

Changed

• Check non-string response status codes when Open API links are collected. #1226

3.9.6 - 2021-07-15

Added

• New before_call and after_call CLI hooks. #1224, #700

3.9.5 - 2021-07-14

Fixed

• Preserve non-body parameter types in requests during Open API runtime expression evaluation.

3.9.4 - 2021-07-09

Fixed

• KeyError when the response_schema_conformance check is executed against responses without schema definition. #1220

• TypeError during negative testing on Open API schemas with parameters that have non-default style value. #1208

3.9.3 - 2021-06-22

Added

• ExecutionEvent.is_terminal attribute that indicates whether an event is the last one in the stream.

Fixed

• When EventStream.stop is called, the next event always is the last one.

3.9.2 - 2021-06-16

Changed

• Return response from Case.call_and_validate.

Fixed

• Incorrect deduplication applied to response schema conformance failures that happen to have the same failing validator but different input values. #907

3.9.1 - 2021-06-13

Changed

• ExecutionEvent.asdict adds the event_type field which is the event class name.

• Add API schema to the Initialized event.

• Internal: Add SerializedCase.cookies

• Convert all FailureContext class attributes to instance attributes. For simpler serialization via attrs.

3.9.0 - 2021-06-07

Added

• GraphQL support in CLI. #746

• A way to stop the Schemathesis runner’s event stream manually via events.stop() / events.finish() methods. #1202

Changed

• Avoid pytest warnings when internal Schemathesis classes are in the test module scope.

3.8.0 - 2021-06-03

Added

• Negative testing. #65

• Case.data_generation_method attribute that provides the information of the underlying data generation method (e.g. positive or negative)

Changed

• Raise UsageError if schema.parametrize or schema.given are applied to the same function more than once. #1194

• Python values of True, False and None are converted to their JSON equivalents when generated for path parameters or query. #1166

• Bump hypothesis-jsonschema version. It allows the end-user to override known string formats.

• Bump hypothesis version.

• APIOperation.make_case behavior. If no media_type is passed along with body, then it tries to infer the proper media type and raises an error if it is not possible. #1094

Fixed

• Compatibility with hypothesis>=6.13.3.

3.7.8 - 2021-06-02

Fixed

• Open API style & explode for parameters derived from security definitions.

3.7.7 - 2021-06-01

Fixed

• Apply the Open API’s style & explode keywords to explicit examples. #1190

3.7.6 - 2021-05-31

Fixed

• Disable filtering optimization for headers when there are keywords other than type. #1189

3.7.5 - 2021-05-31

Fixed

• Too much filtering in headers that have schemas with the pattern keyword. #1189

3.7.4 - 2021-05-28

Changed

• Internal: SerializedCase.path_template returns path templates as they are in the schema, without base path.

3.7.3 - 2021-05-28

Fixed

• Invalid multipart payload generated for unusual schemas for the multipart/form-data media type.

Performance

• Reduce the amount of filtering needed to generate valid headers and cookies.

3.7.2 - 2021-05-27

Added

• SerializedCase.media_type that stores the information about what media type was used for a particular case.

Fixed

• Internal error on unusual schemas for the multipart/form-data media type. #1152

• Ignored explicit Content-Type override in Case.as_requests_kwargs.

3.7.1 - 2021-05-23

Added

• Internal: FailureContext.title attribute that gives a short failure description.

• Internal: FailureContext.message attribute that gives a longer failure description.

Changed

• Rename JSONDecodeErrorContext.message to JSONDecodeErrorContext.validation_message for consistency.

• Store the more precise schema & instance in ValidationErrorContext.

• Rename ResponseTimeout to RequestTimeout.

3.7.0 - 2021-05-23

Added

• Additional context for each failure coming from the runner. It allows the end-user to customize failure formatting.

Changed

• Use different exception classes for not_a_server_error and status_code_conformance checks. It improves the variance of found errors.

• All network requests (not WSGI) now have the default timeout of 10 seconds. If the response is time-outing, Schemathesis will report it as a failure. It also solves the case when the tested app hangs. #1164

• The default test duration deadline is extended to 15 seconds.

3.6.11 - 2021-05-20

Added

• Internal: BeforeExecution.verbose_name & SerializedCase.verbose_name that reflect specification-specific API operation name.

3.6.10 - 2021-05-17

Changed

• Explicitly add colorama to project’s dependencies.

• Bump hypothesis-jsonschema version.

3.6.9 - 2021-05-14

Fixed

• Ignored $ref keyword in schemas with deeply nested references. #1167

• Ignored Open API specific keywords & types in schemas with deeply nested references. #1162

3.6.8 - 2021-05-13

Changed

• Relax dependency on starlette to >=0.13,<1. #1160

3.6.7 - 2021-05-12

Fixed

• Missing support for the date string format (only full-date was supported).

3.6.6 - 2021-05-07

Changed

• Improve error message for failing Hypothesis deadline healthcheck in CLI. #880

3.6.5 - 2021-05-07

Added

• Support for disabling ANSI color escape codes via the NO_COLOR <https://no-color.org/> environment variable or the --no-color CLI option. #1153

Changed

• Generate valid header values for Bearer auth by construction rather than by filtering.

3.6.4 - 2021-04-30

Changed

• Bump minimum hypothesis-graphql version to 0.5.0. It brings support for interfaces and unions and fixes a couple of bugs in query generation.

3.6.3 - 2021-04-20

Fixed

• Bump minimum hypothesis-graphql version to 0.4.1. It fixes a problem with generating queries with surrogate characters.

• UnicodeEncodeError when sending application/octet-stream payloads that have no format: binary in their schemas. #1134

3.6.2 - 2021-04-15

Fixed

• Windows: UnicodeDecodeError during schema loading via the from_path loader if it contains certain Unicode symbols. from_path loader defaults to UTF-8 from now on.

3.6.1 - 2021-04-09

Fixed

• Using parametrized pytest fixtures with the from_pytest_fixture loader. #1121

3.6.0 - 2021-04-04

Added

• Custom keyword arguments to schemathesis.graphql.from_url that are proxied to requests.post.

• from_wsgi, from_asgi, from_path and from_file loaders for GraphQL apps. #1097, #1100

• Support for data_generation_methods and code_sample_style in all GraphQL loaders.

• Support for app & base_url arguments for the from_pytest_fixture runner.

• Initial support for GraphQL schemas in the Schemathesis runner.

import schemathesis

# Load schema
schema = schemathesis.graphql.from_url("http://127.0.0.1:8000/graphql")
# Initialize runner
runner = schemathesis.runner.from_schema(schema)
# Emit events
for event in runner.execute():
    ...

Breaking

• Loaders’ signatures are unified. Most of the arguments became keyword-only. All except the first two for ASGI/WSGI, all except the first one for the others. It forces loader calls to be more consistent.

# BEFORE
schema = schemathesis.from_uri(
    "http://example.com/openapi.json", "http://127.0.0.1:8000/", "GET"
)
# NOW
schema = schemathesis.from_uri(
    "http://example.com/openapi.json", base_url="http://127.0.0.1:8000/", method="GET"
)

Changed

• Schemathesis generates separate tests for each field defined in the GraphQL Query type. It makes the testing process unified for both Open API and GraphQL schemas.

• IDs for GraphQL tests use the corresponding Query field instead of HTTP method & path.

• Do not show overly verbose raw schemas in Hypothesis output for failed GraphQL tests.

• The schemathesis.graphql.from_url loader now uses the usual Schemathesis User-Agent.

• The Hypothesis database now uses separate entries for each API operation when executed via CLI. It increases its effectiveness when tests are re-run.

• Module schemathesis.loaders is moved to schemathesis.specs.openapi.loaders.

• Show a more specific exception on incorrect usage of the from_path loader in the Schemathesis runner.

Deprecated

• schemathesis.runner.prepare will be removed in Schemathesis 4.0. Use schemathesis.runner.from_schema instead. With this change, the schema loading part goes to your code, similar to using the regular Schemathesis Python API. It leads to a unified user experience where the starting point is API schema loading, which is much clearer than passing a callback & keyword arguments to the prepare function.

Fixed

• Add the missing @schema.given implementation for schemas created via the from_pytest_fixture loader. #1093

• Silently ignoring some incorrect usages of @schema.given.

• Fixups examples were using the incorrect fixup name.

• Return type of make_case for GraphQL schemas.

• Missed operation_id argument in from_asgi loader.

Removed

• Undocumented way to install fixups via the fixups argument for schemathesis.runner.prepare is removed.

3.5.3 - 2021-03-27

Fixed

• Do not use importlib-metadata==3.8 in dependencies as it causes RuntimeError. Ref: https://github.com/python/importlib_metadata/issues/293

3.5.2 - 2021-03-24

Changed

• Prefix worker thread names with schemathesis_.

3.5.1 - 2021-03-23

Fixed

• Encoding for response payloads displayed in the CLI output. #1073

• Use actual charset (from flask.Response.mimetype_params) when storing WSGI responses rather than defaulting to flask.Response.charset.

3.5.0 - 2021-03-22

Added

• before_generate_case hook, that allows the user to modify or filter generated Case instances. #1067

Fixed

• Missing body parameters during Open API links processing in CLI. #1069

• Output types for evaluation results of $response.body and $request.body runtime expressions. #1068

3.4.1 - 2021-03-21

Added

• event_type field to the debug output.

3.4.0 - 2021-03-20

Added

• --debug-output-file CLI option to enable storing the underlying runner events in the JSON Lines format in a separate file for debugging purposes. #1059

Changed

• Make Request.body, Response.body and Response.encoding internal attributes optional. For Request, it means that absent body will lead to Request.body to be None. For Response, body will be None if the app response did not have any payload. Previously these values were empty strings, which was not distinguishable from the cases described above. For the end-user, it means that in VCR cassettes, fields request.body and response.body may be absent.

• models.Status enum now has string values for more readable representation.

3.3.1 - 2021-03-18

Fixed

• Displaying wrong headers in the FAILURES block of the CLI output. #792

3.3.0 - 2021-03-17

Added

• Display failing response payload in the CLI output, similarly to the pytest plugin output. #1050

• A way to control which code sample style to use - Python or cURL. #908

Fixed

• UnicodeDecodeError when generating cURL commands for failed test case reproduction if the request’s body contains non-UTF8 characters.

Internal

• Extra information to events, emitted by the Schemathesis runner.

3.2.2 - 2021-03-11

Added

• Support for Hypothesis 6. #1013

3.2.1 - 2021-03-10

Fixed

• Wrong test results in some cases when the tested schema contains a media type that Schemathesis doesn’t know how to work with. #1046

3.2.0 - 2021-03-09

Performance

• Add an internal caching layer for data generation strategies. It relies on the fact that the internal BaseSchema structure is not mutated over time. It is not directly possible through the public API and is discouraged from doing through hook functions.

Changed

• APIOperation and subclasses of Parameter are now compared by their identity rather than by value.

3.1.3 - 2021-03-08

Added

• count_operations boolean flag to runner.prepare. In case of False value, Schemathesis won’t count the total number of operations upfront. It improves performance for the direct runner usage, especially on large schemas. Schemathesis CLI will still use these calculations to display the progress during execution, but this behavior may become configurable in the future.

3.1.2 - 2021-03-08

Fixed

• Percent-encode the generated . and .. strings in path parameters to avoid resolving relative paths and changing the tested path structure. #1036

3.1.1 - 2021-03-05

Fixed

• Loosen importlib-metadata version constraint and update pyproject.toml #1039

3.1.0 - 2021-02-11

Added

• Support for external examples via the externalValue keyword. #884

Fixed

• Prevent a small terminal width causing a crash (due to negative length used in an f-string) when printing percentage

• Support the latest cryptography version in Docker images. #1033

3.0.9 - 2021-02-10

Fixed

• Return a default terminal size to prevent crashes on systems with zero-width terminals (some CI/CD servers).

3.0.8 - 2021-02-04

• This release updates the documentation to be in-line with the current state.

3.0.7 - 2021-01-31

Fixed

• Docker tags for Buster-based images.

3.0.6 - 2021-01-31

• Packaging-only release for Docker images based on Debian Buster. #1028

3.0.5 - 2021-01-30

Fixed

• Allow to use any iterable type for checks and additional_checks arguments to Case.validate_response.

3.0.4 - 2021-01-19

Fixed

• Generating stateful tests, with common parameters behind a reference. #1020

• Programmatic addition of Open API links via add_link when schema validation is disabled and response status codes are noted as integers. #1022

Changed

• When operations are resolved by operationId then the same reference resolving logic is applied as in other cases. This change leads to less reference inlining and lower memory consumption for deeply nested schemas. #945

3.0.3 - 2021-01-18

Fixed

• Flaky Hypothesis error during explicit examples generation. #1018

3.0.2 - 2021-01-15

Fixed

• Processing parameters common for multiple API operations if they are behind a reference. #1015

3.0.1 - 2021-01-15

Added

• YAML serialization for text/yaml, text/x-yaml, application/x-yaml and text/vnd.yaml media types. #1010.

3.0.0 - 2021-01-14

Added

• Support for sending text/plain payload as test data. Including variants with non-default charset. #850, #939

• Generating data for all media types defined for an operation. #690

• Support for user-defined media types serialization. You can define how Schemathesis should handle media types defined in your schema or customize existing (like application/json).

• The response_schema_conformance check now runs on media types that are encoded with JSON. For example, application/problem+json. #920

• Base URL for GraphQL schemas. It allows you to load the schema from one place but send test requests to another one. #934

• A helpful error message when an operation is not found during the direct schema access. #812

• --dry-run CLI option. When applied, Schemathesis won’t send any data to the server and won’t perform any response checks. #963

• A better error message when the API schema contains an invalid regular expression syntax. #1003

Changed

• Open API parameters parsing to unblock supporting multiple media types per operation. Their definitions aren’t converted to JSON Schema equivalents right away but deferred instead and stored as-is.

• Missing required: true in path parameters definition is now automatically enforced if schema validation is disabled. According to the Open API spec, the required keyword value should be true for path parameters. This change allows Schemathesis to generate test cases even for endpoints containing optional path parameters (which is not compliant with the spec). #941

• Using --auth together with --header that sets the Authorization header causes a validation error. Before, the --header value was ignored in such cases, and the basic auth passed in --auth was used. #911

• When hypothesis-jsonschema fails to resolve recursive references, the test is skipped with an error message that indicates why it happens.

• Shorter error messages when API operations have logical errors in their schema. For example, when the maximum is less than the minimum - {"type": "integer", "minimum": 5, "maximum": 4}.

• If multiple non-check related failures happens during a test of a single API operation, they are displayed as is, instead of Hypothesis-level error messages about multiple found failures or flaky tests. #975

• Catch schema parsing errors, that are caused by YAML parsing.

• The built-in test server now accepts --operations instead of --endpoints.

• Display Collected API operations instead of collected endpoints in the CLI. #869

• --skip-deprecated-endpoints is renamed to --skip-deprecated-operations. #869

• Rename various internal API methods that contained endpoint in their names. #869

• Bump hypothesis-jsonschema version to 0.19.0. This version improves the handling of unsupported regular expression syntax and can generate data for a subset of schemas containing such regular expressions.

• Schemathesis doesn’t stop testing on errors during schema parsing. These errors are handled the same way as other errors during the testing process. It allows Schemathesis to test API operations with valid definitions and report problematic operations instead of failing the whole run. #999

Fixed

• Allow generating requests without payload if the schema does not require it. #916

• Allow sending null as request payload if the schema expects it. #919

• CLI failure if the tested operation is GET and has payload examples. #925

• Excessive reference inlining that leads to out-of-memory for large schemas with deep references. #945, #671

• --exitfirst CLI option trims the progress bar output when a failure occurs. #951

• Internal error if filling missing explicit examples led to Unsatisfiable errors. #904

• Do not suggest to disable schema validation if it is already disabled. #914

• Skip explicit examples generation if this phase is disabled via config. #905

• Unsatisfiable error in stateful testing caused by all API operations having inbound links. #965, #822

• A possibility to override APIStateMachine.step. #970

• TypeError on nullable parameters during Open API specific serialization. #980

• Invalid types in x-examples. #982

• CLI crash on schemas with operation names longer than the current terminal width. #990

• Handling of API operations that contain reserved characters in their paths. #992

• CLI execution stops on errors during example generation. #994

• Fill missing properties in incomplete explicit examples for non-body parameters. #1007

Deprecated

• HookContext.endpoint. Use HookContext.operation instead.

• Case.endpoint. Use Case.operation instead.

Performance

• Use compiled versions of Open API spec validators.

• Decrease CLI memory usage. #987

• Various improvements relevant to processing of API operation definitions. It gives ~20% improvement on large schemas with many references.

Removed

• Case.form_data. Use Case.body instead.

• Endpoint.form_data. Use Endpoint.body instead.

• before_generate_form_data hook. Use before_generate_body instead.

• Deprecated stateful testing integration from our pytest plugin.

Note

This release features multiple backward-incompatible changes. The first one is removing form_data and hooks related to it - all payload related actions can be done via body and its hooks. The second one involves renaming the so-called “endpoint” to “operation”. The main reason for this is to generalize terminology and make it applicable to GraphQL schemas, as all Schemathesis internals are more suited to work with semantically different API operations rather than with endpoints that are often connected with URLs and HTTP methods. It brings the possibility to reuse the same concepts for Open API and GraphQL - in the future, unit tests will cover individual API operations in GraphQL, rather than everything available under the same “endpoint”.

2.8.6 - 2022-03-29

Added

• Support for Werkzeug>=2.1.0. #1410

2.8.5 - 2020-12-15

Added

• auto variant for the --workers CLI option that automatically detects the number of available CPU cores to run tests on. #917

2.8.4 - 2020-11-27

Fixed

• Use --request-tls-verify during schema loading as well. #897

2.8.3 - 2020-11-27

Added

• Display failed response payload in the error output for the pytest plugin. #895

Changed

• In pytest plugin output, Schemathesis error classes use the CheckFailed name. Before, they had not readable “internal” names.

• Hypothesis falsifying examples. The code does not include Case attributes with default values to improve readability. #886

2.8.2 - 2020-11-25

Fixed

• Internal error in CLI, when the base_url is an invalid IPv6. #890

• Internal error in CLI, when a malformed regex is passed to -E / -M / -T / -O CLI options. #889

2.8.1 - 2020-11-24

Added

• --force-schema-version CLI option to force Schemathesis to use the specific Open API spec version when parsing the schema. #876

Changed

• The content_type_conformance check now raises a well-formed error message when encounters a malformed media type value. #877

Fixed

• Internal error during verifying explicit examples if an example has no value key. #882

2.8.0 - 2020-11-24

Added

• --request-tls-verify CLI option, that controls whether Schemathesis verifies the server’s TLS certificate. You can also pass the path to a CA_BUNDLE file for private certs. #830

Changed

• In CLI, if an endpoint contains an invalid schema, show a message about the --validate-schema CLI option. #855

Fixed

• Handling of 204 responses in the response_schema_conformance check. Before, all responses were required to have the Content-Type header. #844

• Catch OverflowError when an invalid regex is passed to -E / -M / -T / -O CLI options. #870

• Internal error in CLI, when the schema location is an invalid IPv6. #872

• Collecting Open API links behind references via CLI. #874

Deprecated

• Using of Case.form_data and Endpoint.form_data. In the 3.0 release, you’ll need to use relevant body attributes instead. This change includes deprecation of the before_generate_form_data hook, use before_generate_body instead. The reason for this is the upcoming unification of parameter handling and their serialization.

• --stateful-recursion-limit. It will be removed in 3.0 as a part of removing the old stateful testing approach. This parameter is no-op.

2.7.7 - 2020-11-13

Fixed

• Missed headers in Endpoint.partial_deepcopy.

2.7.6 - 2020-11-12

Added

• An option to set data generation methods. At the moment, it includes only “positive”, which means that Schemathesis will generate data that matches the schema.

Fixed

• Pinned dependency on attrs that caused an error on fresh installations. #858

2.7.5 - 2020-11-09

Fixed

• Invalid keyword in code samples that Schemathesis suggests to run to reproduce errors. #851

2.7.4 - 2020-11-07

Added

• New relative_path property for BeforeExecution and AfterExecution events. It represents an operation path as it is in the schema definition.

2.7.3 - 2020-11-05

Fixed

• Internal error on malformed JSON when the response_conformance check is used. #832

2.7.2 - 2020-11-05

Added

• Shortcut for response validation when Schemathesis’s data generation is not used. #485

Changed

• Improve the error message when the application can not be loaded from the value passed to the --app command-line option. #836

• Security definitions are now serialized as other parameters. At the moment, it means that the generated values will be coerced to strings, which is a no-op. However, types of security definitions might be affected by the “Negative testing” feature in the future. Therefore this change is mostly for future-compatibility. #841

Fixed

• Internal error when a “header” / “cookie” parameter were not coerced to a string before filtration. #839

2.7.1 - 2020-10-22

Fixed

• Adding new Open API links via the add_link method, when the related PathItem contains a reference. #824

2.7.0 - 2020-10-21

Added

• New approach to stateful testing, based on the Hypothesis’s RuleBasedStateMachine. #737

• Case.validate_response accepts the new additional_checks argument. It provides a way to execute additional checks in addition to existing ones.

Changed

• The response_schema_conformance and content_type_conformance checks fail unconditionally if the input response has no Content-Type header. #816

Fixed

• Failure reproduction code missing values that were explicitly passed to call_* methods during testing. #814

Deprecated

• Using stateful=Stateful.links in schema loaders and parametrize. Use schema.as_state_machine().TestCase instead. The old approach to stateful testing will be removed in 3.0. See the Stateful testing section of our documentation for more information.

2.6.1 - 2020-10-19

Added

• New method as_curl_command added to the Case class. #689

2.6.0 - 2020-10-06

Added

• Support for passing Hypothesis strategies to tests created with schema.parametrize by using schema.given decorator. #768

• Support for PEP561. #748

• Shortcut for calling & validation. #738

• New hook to pre-commit, rstcheck, as well as updates to documentation based on rstcheck. #734

• New check for maximum response time and corresponding CLI option --max-response-time. #716

• New response_headers_conformance check that verifies the presence of all headers defined for a response. #742

• New field with information about executed checks in cassettes. #702

• New port parameter added to from_uri() method. #706

• A code snippet to reproduce a failed check when running Python tests. #793

• Python 3.9 support. #731

• Ability to skip deprecated endpoints with --skip-deprecated-endpoints CLI option and skip_deprecated_operations=True argument to schema loaders. #715

Fixed

• User-Agent header overriding the passed one. #757

• Default User-Agent header in Case.call. #717

• Status of individual interactions in VCR cassettes. Before this change, all statuses were taken from the overall test outcome, rather than from the check results for a particular response. #695

• Escaping header values in VCR cassettes. #783

• Escaping HTTP response message in VCR cassettes. #788

Changed

• Case.as_requests_kwargs and Case.as_werkzeug_kwargs now return the User-Agent header. This change also affects code snippets for failure reproduction - all snippets will include the User-Agent header.

Performance

• Speed up generation of headers, cookies, and formData parameters when their schemas do not define the type keyword. #795

2.5.1 - 2020-09-30

This release contains only documentation updates which are necessary to upload to PyPI.

2.5.0 - 2020-09-27

Added

• Stateful testing via Open API links for the pytest runner. #616

• Support for GraphQL tests for the pytest runner. #649

Fixed

• Progress percentage in the terminal output for “lazy” schemas. #636

Changed

• Check name is no longer displayed in the CLI output, since its verbose message is already displayed. This change also simplifies the internal structure of the runner events.

• The stateful argument type in the runner.prepare is Optional[Stateful] instead of Optional[str]. Use schemathesis.Stateful enum.

2.4.1 - 2020-09-17

Changed

• Hide Case.endpoint from representation. Its representation decreases the usability of the pytest’s output. #719

• Return registered functions from register_target and register_check decorators. #721

Fixed

• Possible IndexError when a user-defined check raises an exception without a message. #718

2.4.0 - 2020-09-15

Added

• Ability to register custom targets for targeted testing. #686

Changed

• The AfterExecution event now has path and method fields, similar to the BeforeExecution one. The goal is to make these events self-contained, which improves their usability.

2.3.4 - 2020-09-11

Changed

• The default Hypothesis’s deadline setting for tests with schema.parametrize is set to 500 ms for consistency with the CLI behavior. #705

Fixed

• Encoding error when writing a cassette on Windows. #708

2.3.3 - 2020-08-04

Fixed

• KeyError during the content_type_conformance check if the response has no Content-Type header. #692

2.3.2 - 2020-08-04

Added

• Run checks conditionally.

2.3.1 - 2020-07-28

Fixed

• IndexError when examples list is empty.

2.3.0 - 2020-07-26

Added

• Possibility to generate values for in: formData parameters that are non-bytes or contain non-bytes (e.g., inside an array). #665

Changed

• Error message for cases when a path parameter is in the template but is not defined in the parameters list or missing required: true in its definition. #667

• Bump minimum required hypothesis-jsonschema version to 0.17.0. This allows Schemathesis to use the custom_formats argument in from_schema calls and avoid using its private API. #684

Fixed

• ValueError during sending a request with test payload if the endpoint defines a parameter with type: array and in: formData. #661

• KeyError while processing a schema with nullable parameters and in: body. #660

• StopIteration during requestBody processing if it has empty “content” value. #673

• AttributeError during generation of “multipart/form-data” parameters that have no “type” defined. #675

• Support for properties named “$ref” in object schemas. Previously, it was causing TypeError. #672

• Generating illegal Unicode surrogates in the path. #668

• Invalid development dependency on graphql-server-core package. #658

2.2.1 - 2020-07-22

Fixed

• Possible UnicodeEncodeError during generation of Authorization header values for endpoints with basic security scheme. #656

2.2.0 - 2020-07-14

Added

• schemathesis.graphql.from_dict loader allows you to use GraphQL schemas represented as a dictionary for testing.

• before_load_schema hook for GraphQL schemas.

Fixed

• Serialization of non-string parameters. #651

2.1.0 - 2020-07-06

Added

• Support for property-level examples. #467

Fixed

• Content-type conformance check for cases when Open API 3.0 schemas contain “default” response definitions. #641

• Handling of multipart requests for Open API 3.0 schemas. #640

• Sending non-file form fields in multipart requests. #647

Removed

• Deprecated skip_validation argument to HookDispatcher.apply.

• Deprecated _accepts_context internal function.

2.0.0 - 2020-07-01

Changed

• BREAKING. Base URL handling. base_url now is treated as one with a base path included. You should pass a full base URL now instead:

schemathesis run --base-url=http://127.0.0.1:8080/api/v2 ...

This value will override basePath / servers[0].url defined in your schema if you use Open API 2.0 / 3.0 respectively. Previously if you pass a base URL like the one above, it was concatenated with the base path defined in the schema, which leads to a lack of ability to redefine the base path. #511

Fixed

• Show the correct URL in CLI progress when the base URL is overridden, including the path part. #511

• Construct valid URL when overriding base URL with base path. #511

Example:

Base URL in the schema         : http://0.0.0.0:8081/api/v1
`--base-url` value in CLI      : http://0.0.0.0:8081/api/v2
Full URLs before this change   : http://0.0.0.0:8081/api/v2/api/v1/users/  # INVALID!
Full URLs after this change    : http://0.0.0.0:8081/api/v2/users/         # VALID!

Removed

• Support for hooks without context argument in the first position.

• Hooks registration by name and function. Use register decorators instead. For more details, see the “Customization” section in our documentation.

• BaseSchema.with_hook and BaseSchema.register_hook. Use BaseSchema.hooks.apply and BaseSchema.hooks.register instead.

1.10.0 - 2020-06-28

Added

• loaders.from_asgi supports making calls to ASGI-compliant application (For example: FastAPI). #521

• Support for GraphQL strategies.

Fixed

• Passing custom headers to schema loader for WSGI / ASGI apps. #631

1.9.1 - 2020-06-21

Fixed

• Schema validation error on schemas containing numeric values in scientific notation without a dot. #629

1.9.0 - 2020-06-20

Added

• Pass the original case’s response to the add_case hook.

• Support for multiple examples with OpenAPI examples. #589

• --verbosity CLI option to minimize the error output. #598

• Allow registering function-level hooks without passing their name as the first argument to apply. #618

• Support for hook usage via LazySchema / from_pytest_fixture. #617

Changed

• Tests with invalid schemas marked as errors, instead of failures. #622

Fixed

• Crash during the generation of loosely-defined headers. #621

• Show exception information for test runs on invalid schemas with --validate-schema=false command-line option. Before, the output sections for invalid endpoints were empty. #622

1.8.0 - 2020-06-15

Fixed

• Tests with invalid schemas are marked as failed instead of passed when hypothesis-jsonschema>=0.16 is installed. #614

• KeyError during creating an endpoint strategy if it contains a reference. #612

Changed

• Require hypothesis-jsonschema>=0.16. #614

• Pass original InvalidSchema text to pytest.fail call.

1.7.0 - 2020-05-30

Added

• Support for YAML files in references via HTTPS & HTTP schemas. #600

• Stateful testing support via Open API links syntax. #548

• New add_case hook. #458

• Support for parameter serialization formats in Open API 2 / 3. For example pipeDelimited or deepObject. #599

• Support serializing parameters with application/json content-type. #594

Changed

• The minimum required versions for Hypothesis and hypothesis-jsonschema are 5.15.0 and 0.11.1 respectively. The main reason is this fix that is required for stability of Open API links feature when it is executed in multiple threads.

1.6.3 - 2020-05-26

Fixed

• Support for a colon symbol (:) inside of a header value passed via CLI. #596

1.6.2 - 2020-05-15

Fixed

• Partially generated explicit examples are always valid and can be used in requests. #582

1.6.1 - 2020-05-13

Changed

• Look at the current working directory when loading hooks for CLI. #586

1.6.0 - 2020-05-10

Added

• New before_add_examples hook. #571

• New after_init_cli_run_handlers hook. #575

Fixed

• Passing workers_num to ThreadPoolRunner leads to always using 2 workers in this worker kind. #579

1.5.1 - 2020-05-08

Fixed

• Display proper headers in reproduction code when headers are overridden. #566

1.5.0 - 2020-05-06

Added

• Display a suggestion to disable schema validation on schema loading errors in CLI. #531

• Filtration of endpoints by operationId via operation_id parameter to schema.parametrize or -O command-line option. #546

• Generation of security-related parameters. They are taken from securityDefinitions / securitySchemes and injected to the generated data. It supports generating API keys in headers or query parameters and generating data for HTTP authentication schemes. #540

Fixed

• Overriding header values in CLI and runner when headers provided explicitly clash with ones defined in the schema. #559

• Nested references resolving in response_schema_conformance check. #562

• Nullable parameters handling when they are behind a reference. #542

1.4.0 - 2020-05-03

Added

• context argument for hook functions to provide an additional context for hooks. A deprecation warning is emitted for hook functions that do not accept this argument.

• A new hook system that allows generic hook dispatching. It comes with new hook locations. For more details, see the “Customization” section in our documentation.

• New before_process_path hook.

• Third-party compatibility fixups mechanism. Currently, there is one fixup for FastAPI. #503

Deprecated

• Hook functions that do not accept context as their first argument. They will become not be supported in Schemathesis 2.0.

• Registering hooks by name and function. Use register decorators instead. For more details, see the “Customization” section in our documentation.

• BaseSchema.with_hook and BaseSchema.register_hook. Use BaseSchema.hooks.apply and BaseSchema.hooks.register instead.

Fixed

• Add missing validate_schema argument to loaders.from_pytest_fixture.

• Reference resolving during response schema conformance check. #539

1.3.4 - 2020-04-30

Fixed

• Validation of nullable properties in response_schema_conformance check introduced in 1.3.0. #542

1.3.3 - 2020-04-29

Changed

• Update pytest-subtests pin to >=0.2.1,<1.0. #537

1.3.2 - 2020-04-27

Added

• Show exceptions if they happened during loading a WSGI application. Option --show-errors-tracebacks will display a full traceback.

1.3.1 - 2020-04-27

Fixed

• Packaging issue

1.3.0 - 2020-04-27

Added

• Storing network logs with --store-network-log=<filename.yaml>. The stored cassettes are based on the VCR format and contain extra information from the Schemathesis internals. #379

• Replaying of cassettes stored in VCR format. #519

• Targeted property-based testing in CLI and runner. It only supports the response_time target at the moment. #104

• Export CLI test results to JUnit.xml with --junit-xml=<filename.xml>. #427

Fixed

• Code samples for schemas where body is defined as {"type": "string"}. #521

• Showing error causes on internal jsonschema errors during input schema validation. #513

• Recursion error in response_schema_conformance check. Because of this change, Endpoint.definition contains a definition where references are not resolved. In this way, it makes it possible to avoid recursion errors in jsonschema validation. #468

Changed

• Added indentation & section name to the SUMMARY CLI block.

• Use C-extension for YAML loading when it is possible. It can cause more than 10x speedup on schema parsing. Do not show Click’s “Aborted!” message when an error occurs during CLI schema loading.

• Add a help message to the CLI output when an internal exception happens. #529

1.2.0 - 2020-04-15

Added

• Per-test hooks for modification of data generation strategies. #492

• Support for x-example vendor extension in Open API 2.0. #504

• Sanity validation for the input schema & loader in runner.prepare. #499

1.1.2 - 2020-04-14

Fixed

• Support for custom loaders in runner. Now all built-in loaders are supported as an argument to runner.prepare. #496

• from_wsgi loader accepts custom keyword arguments that will be passed to client.get when accessing the schema. #497

1.1.1 - 2020-04-12

Fixed

• Mistakenly applied Open API -> JSON Schema Draft 7 conversion. It should be Draft 4. #489

• Using wrong validator in response_schema_conformance check. It should be Draft 4 validator. #468

1.1.0 - 2020-04-08

Fixed

• Response schema check for recursive schemas. #468

Changed

• App loading in runner. Now it accepts application as an importable string, rather than an instance. It is done to make it possible to execute a runner in a subprocess. Otherwise, apps can’t be easily serialized and transferred into another process.

• Runner events structure. All data in events is static from now. There are no references to BaseSchema, Endpoint or similar objects that may calculate data dynamically. This is done to make events serializable and not tied to Python object, which decouples any runner consumer from implementation details. It will help make runner usable in more cases (e.g., web application) since events can be serialized to JSON and used in any environment. Another related change is that Python exceptions are not propagated anymore - they are replaced with the InternalError event that should be handled accordingly.

1.0.5 - 2020-04-03

Fixed

• Open API 3. Handling of endpoints that contain multipart/form-data media types. Previously only file upload endpoints were working correctly. #473

1.0.4 - 2020-04-03

Fixed

• OpenApi30.get_content_types behavior, introduced in 8aeee1a. #469

1.0.3 - 2020-04-03

Fixed

• Precedence of produces keywords for Swagger 2.0 schemas. Now, operation-level produces overrides schema-level produces as specified in the specification. #463

• Content-type conformance check for Open API 3.0 schemas. #461

• Pytest 5.4 warning for test functions without parametrization. #451

1.0.2 - 2020-04-02

Fixed

• Handling of fields in paths that are not operations, but allowed by the Open API spec. #457

• Pytest 5.4 warning about deprecated Node initialization usage. #451

1.0.1 - 2020-04-01

Fixed

• Processing of explicit examples in Open API 3.0 when there are multiple parameters in the same location (e.g. path) contain example value. They are properly combined now. #450

1.0.0 - 2020-03-31

Changed

• Move processing of runner parameters to runner.prepare. This change will provide better code reuse since all users of runner (e.g., if you extended it in your project) need some kind of input parameters handling, which was implemented only in Schemathesis CLI. It is not backward-compatible. If you didn’t use runner directly, then this change should not have a visible effect on your use-case.

0.28.0 - 2020-03-31

Fixed

• Handling of schemas that use x-* custom properties. #448

Removed

• Deprecated runner.execute. Use runner.prepare instead.

0.27.0 - 2020-03-31

Deprecated

• runner.execute should not be used, since runner.prepare provides a more flexible interface to test execution.

Removed

• Deprecated Parametrizer class. Use schemathesis.from_path as a replacement for Parametrizer.from_path.

0.26.1 - 2020-03-24

Fixed

• Limit recursion depth while resolving JSON schema to handle recursion without breaking. #435

0.26.0 - 2020-03-19

Fixed

• Filter problematic path template variables containing "/", or "%2F" url encoded. #440

• Filter invalid empty "" path template variables. #439

• Typo in a help message in the CLI output. #436

0.25.1 - 2020-03-09

Changed

• Allow werkzeug >= 1.0.0. #433

0.25.0 - 2020-02-27

Changed

• Handling of explicit examples from schemas. Now, if there are examples for multiple locations (e.g., for body and query) then they will be combined into a single example. #424

0.24.5 - 2020-02-26

Fixed

• Error during pytest collection on objects with custom __getattr__ method and therefore pass is_schemathesis check. #429

0.24.4 - 2020-02-22

Fixed

• Resolving references when the schema is loaded from a file on Windows. #418

0.24.3 - 2020-02-10

Fixed

• Not copied validate_schema parameter in BaseSchema.parametrize. Regression after implementing #383

• Missing app, location and hooks parameters in schema when used with BaseSchema.parametrize. #416

0.24.2 - 2020-02-09

Fixed

• Crash on invalid regular expressions in method, endpoint and tag CLI options. #403

• Crash on a non-latin-1 encodable value in the auth CLI option. #404

• Crash on an invalid value in the header CLI option. #405

• Crash on some invalid URLs in the schema CLI option. #406

• Validation of --request-timeout parameter. #407

• Crash with --hypothesis-deadline=0 CLI option. #410

• Crash with --hypothesis-max-examples=0 CLI option. #412

0.24.1 - 2020-02-08

Fixed

• CLI crash on Windows and Python < 3.8 when the schema path contains characters unrepresentable at the OS level. #400

0.24.0 - 2020-02-07

Added

• Support for testing of examples in Parameter & Media Type objects in Open API 3.0. #394

• --show-error-tracebacks CLI option to display errors’ tracebacks in the output. #391

• Support for schema behind auth. #115

Changed

• Schemas with GET endpoints accepting body are allowed now if schema validation is disabled (via --validate-schema=false for example). The use-case is for tools like ElasticSearch that use GET requests with non-empty bodies. #383

Fixed

• CLI crash when an explicit example is specified in the endpoint definition. #386

0.23.7 - 2020-01-30

Added

• -x/--exitfirst CLI option to exit after the first failed test. #378

Fixed

• Handling examples of parameters in Open API 3.0. #381

0.23.6 - 2020-01-28

Added

• all variant for --checks CLI option to use all available checks. #374

Changed

• Use built-in importlib.metadata on Python 3.8. #376

0.23.5 - 2020-01-24

Fixed

• Generation of invalid values in Case.cookies. #371

0.23.4 - 2020-01-22

Fixed

• Converting exclusiveMinimum & exclusiveMaximum fields to JSON Schema. #367

0.23.3 - 2020-01-21

Fixed

• Filter out surrogate pairs from the query string.

0.23.2 - 2020-01-16

Fixed

• Prevent KeyError when the response does not have the “Content-Type” header. #365

0.23.1 - 2020-01-15

Fixed

• Dockerfile entrypoint was not working as per docs. #361

0.23.0 - 2020-01-15

Added

• Hooks for strategy modification. #313

• Input schema validation. Use --validate-schema=false to disable it in CLI and validate_schema=False argument in loaders. #110

0.22.0 - 2020-01-11

Added

• Show multiple found failures in the CLI output. #266 & #207

• Raise a proper exception when the given schema is invalid. #308

• Support for None as a value for --hypothesis-deadline. #349

Fixed

• Handling binary request payloads in Case.call. #350

• Type of the second argument to all built-in checks set to proper Case instead of TestResult. The error was didn’t affect built-in checks since both Case and TestResult had endpoint attribute, and only it was used. However, this fix is not backward-compatible with 3rd party checks.

0.21.0 - 2019-12-20

Added

• Support for AioHTTP applications in CLI. #329

0.20.5 - 2019-12-18

Fixed

• Compatibility with the latest release of hypothesis-jsonschema and setting its minimal required version to 0.9.13. #338

0.20.4 - 2019-12-17

Fixed

• Handling nullable attribute in Open API schemas. #335

0.20.3 - 2019-12-17

Fixed

• Usage of the response status code conformance check with old requests version. #330

0.20.2 - 2019-12-14

Fixed

• Response schema conformance check for Open API 3.0. #332

0.20.1 - 2019-12-13

Added

• Support for response code ranges. #330

0.20.0 - 2019-12-12

Added

• WSGI apps support. #31

• Case.validate_response for running built-in checks against app’s response. #319

Changed

• Checks receive Case instance as a second argument instead of TestResult. This was done for making checks usable in Python tests via Case.validate_response. Endpoint and schema are accessible via case.endpoint and case.endpoint.schema.

0.19.1 - 2019-12-11

Fixed

• Compatibility with Hypothesis >= 4.53.2. #322

0.19.0 - 2019-12-02

Added

• Concurrent test execution in CLI / runner. #91

• update importlib_metadata pin to ^1.1. #315

0.18.1 - 2019-11-28

Fixed

• Validation of the base-url CLI parameter. #311

0.18.0 - 2019-11-27

Added

• Resolving references in PathItem objects. #301

Fixed

• Resolving of relative paths in schemas. #303

• Loading string dates as datetime.date objects in YAML loader. #305

0.17.0 - 2019-11-21

Added

• Resolving references that point to different files. #294

Changed

• Keyboard interrupt is now handled during the CLI run, and the summary is displayed in the output. #295

0.16.0 - 2019-11-19

Added

• Display RNG seed in the CLI output to allow test reproducing. #267

• Allow specifying seed in CLI.

• Ability to pass custom kwargs to the requests.get call in loaders.from_uri.

Changed

• Refactor case generation strategies: strategy is not used to generate empty value. #253

• Improved error message for invalid path parameter declaration. #255

Fixed

• Pytest fixture parametrization via pytest_generate_tests. #280

• Support for tests defined as methods. #282

• Unclosed requests.Session on calling Case.call without passing a session explicitly. #286

0.15.0 - 2019-11-15

Added

• Support for OpenAPI 3.0 server variables (base_path). #40

• Support for format: byte. #254

• Response schema conformance check in CLI / Runner. #256

• Docker image for CLI. #268

• Pre-run hooks for CLI. #147

• A way to register custom checks for CLI via schemathesis.register_check. #270

Fixed

• Not encoded path parameters. #272

Changed

• Verbose messages are displayed in the CLI on failed checks. #261

0.14.0 - 2019-11-09

Added

• CLI: Support file paths in the schema argument. #119

• Checks to verify response status & content type in CLI / Runner. #101

Fixed

• Custom base URL handling in CLI / Runner. #248

Changed

• Raise an error if the schema has a body for GET requests. #218

• Method names are case insensitive during direct schema access. #246

0.13.2 - 2019-11-05

Fixed

• IndexError when Hypothesis found inconsistent test results during the test execution in the runner. #236

0.13.1 - 2019-11-05

Added

• Support for binary format #197

Fixed

• Error that happens when there are no success checks in the statistic in CLI. #237

0.13.0 - 2019-11-03

Added

• An option to configure request timeout for CLI / Runner. #204

• A help snippet to reproduce errors caught by Schemathesis. #206

• Total running time to the CLI output. #181

• Summary line in the CLI output with the number of passed / failed / errored endpoint tests. #209

• Extra information to the CLI output: schema address, spec version, and base URL. #188

Fixed

• Compatibility with Hypothesis 4.42.4+ . #212

• Display flaky errors only in the “ERRORS” section and improve CLI output. #215

• Handling formData parameters in Case.call. #196

• Handling cookies in Case.call. #211

Changed

• More readable falsifying examples output. #127

• Show exceptions in a separate section of the CLI output. #203

• Error message for cases when it is not possible to satisfy schema parameters. It should be more clear now. #216

• Do not stop on schema errors related to a single endpoint. #139

• Display a proper error message when the schema is not available in CLI / Runner. #214

0.12.2 - 2019-10-30

Fixed

• Wrong handling of the base_url parameter in runner and Case.call if it has a trailing slash. #194 and #199

• Do not send any payload with GET requests. #200

0.12.1 - 2019-10-28

Fixed

• Handling for errors other than AssertionError and HypothesisException in the runner. #189

• CLI failing on the case when there are tests, but no checks were performed. #191

Changed

• Display the “SUMMARY” section in the CLI output for empty test suites.

0.12.0 - 2019-10-28

Added

• Display progress during the CLI run. #125

Fixed

• Test server-generated wrong schema when the endpoints option is passed via CLI. #173

• Error message if the schema is not found in CLI. #172

Changed

• Continue running tests on hypothesis error. #137

0.11.0 - 2019-10-22

Added

• LazySchema accepts filters. #149

• Ability to register strategies for custom string formats. #94

• Generator-based events in the runner module to improve control over the execution flow.

• Filtration by tags. #134

Changed

• Base URL in schema instances could be reused when it is defined during creation. Now on, the base_url argument in Case.call is optional in such cases. #153

• Hypothesis deadline is set to 500ms by default. #138

• Hypothesis output is captured separately, without capturing the whole stdout during CLI run.

• Disallow empty username in CLI --auth option.

Fixed

• User-agent during schema loading. #144

• Generation of invalid values in Case.headers. #167

Removed

• Undocumented support for file:// URI schema

0.10.0 - 2019-10-14

Added

• HTTP Digest Auth support. #106

• Support for Hypothesis settings in CLI & Runner. #107

• Case.call and Case.as_requests_kwargs convenience methods. #109

• Local development server. #126

Removed

• Autogenerated runner.StatsCollector.__repr__ to make Hypothesis output more readable.

0.9.0 - 2019-10-09

Added

• Test executor collects results of execution. #29

• CLI option --base-url for specifying base URL of API. #118

• Support for coroutine-based tests. #121

• User Agent to network requests in CLI & runner. #130

Changed

• CLI command schemathesis run prints result in a more readable way with a summary of passing checks.

• Empty header names are forbidden for CLI.

• Suppressed hypothesis exception about using example non-interactively. #92

0.8.1 - 2019-10-04

Fixed

• Wrap each test in suppress so the runner doesn’t stop after the first test failure.

0.8.0 - 2019-10-04

Added

• CLI tool invoked by the schemathesis command. #30

• New arguments api_options, loader_options and loader for test executor. #90

• A mapping interface for schemas & convenience methods for direct strategy access. #98

Fixed

• Runner stopping on the first falsifying example. #99

0.7.3 - 2019-09-30

Fixed

• Filtration in lazy loaders.

0.7.2 - 2019-09-30

Added

• Support for type “file” for Swagger 2.0. #78

• Support for filtering in loaders. #75

Fixed

• Conflict for lazy schema filtering. #64

0.7.1 - 2019-09-27

Added

• Support for x-nullable extension. #45

0.7.0 - 2019-09-26

Added

• Support for the cookie parameter in OpenAPI 3.0 schemas. #21

• Support for the formData parameter in Swagger 2.0 schemas. #6

• Test executor. #28

Fixed

• Using hypothesis.settings decorator with test functions created from from_pytest_fixture loader. #69

0.6.0 - 2019-09-24

Added

• Parametrizing tests from a pytest fixture via pytest-subtests. #58

Changed

• Rename module readers to loaders.

• Rename parametrize parameters. filter_endpoint to endpoint and filter_method to method.

Removed

• Substring match for method/endpoint filters. To avoid clashing with escaped chars in endpoints keys in schemas.

0.5.0 - 2019-09-16

Added

• Generating explicit examples from the schema. #17

Changed

• Schemas are loaded eagerly from now on. Using schemathesis.from_uri implies network calls.

Deprecated

• Using Parametrizer.from_{path,uri} is deprecated, use schemathesis.from_{path,uri} instead.

Fixed

• Body resolving during test collection. #55

0.4.1 - 2019-09-11

Fixed

• Possibly unhandled exception during hasattr check in is_schemathesis_test.

0.4.0 - 2019-09-10

Fixed

• Resolving all inner references in objects. #34

Changed

• jsonschema.RefResolver is now used for reference resolving. #35

0.3.0 - 2019-09-06

Added

• Parametrizer.from_uri method to construct parametrizer instances from URIs. #24

Removed

• Possibility to use Parametrizer.parametrize and custom Parametrizer kwargs for passing config options to hypothesis.settings. Use hypothesis.settings decorators on tests instead.

0.2.0 - 2019-09-05

Added

• Open API 3.0 support. #10

• “header” parameters. #7

Changed

• Handle errors during collection / executions as failures.

• Use re.search for pattern matching in filter_method/filter_endpoint instead of fnmatch. #18

• Case.body contains properties from the target schema, without the extra level of nesting.

Fixed

• KeyError on collection when “basePath” is absent. #16

0.1.0 - 2019-06-28

• Initial public release