Changelog

2.0.0 - 2020-07-01

Changed

  • BREAKING. Base URL handling. base_url now is treated as one with a base path included. You should pass a full base URL now instead:
schemathesis run --base-url=http://127.0.0.1:8080/api/v2 ...

This value will override basePath / servers[0].url defined in your schema if you use Open API 2.0 / 3.0 respectively. Previously if you pass a base URL like the one above it was concatenated with the base path defined in the schema which lead to lack of ability to redefine the base path. #511

Fixed

  • Show correct URL in CLI progress when base URL is overridden including the path part. #511
  • Construct valid URL when overriding base URL with base path. #511

Example:

Base URL in the schema         : http://0.0.0.0:8081/api/v1
`--base-url` value in CLI      : http://0.0.0.0:8081/api/v2
Full URLs before this change   : http://0.0.0.0:8081/api/v2/api/v1/users/  # INVALID!
Full URLs after  this change   : http://0.0.0.0:8081/api/v2/users/         # VALID!

Removed

  • Support for hooks without context argument at the first position.
  • Hooks registration by name and function. Use register decorators instead, for more details see “Customization” section in our documentation.
  • BaseSchema.with_hook and BaseSchema.register_hook. Use BaseSchema.hooks.apply and BaseSchema.hooks.register instead.

1.10.0 - 2020-06-28

Added

  • loaders.from_asgi supports making calls to ASGI-compliant application (For example: FastAPI). #521
  • Support for GraphQL strategies.

Fixed

  • Passing custom headers to schema loader for WSGI / ASGI apps. #631

1.9.1 - 2020-06-21

Fixed

  • Schema validation error on schemas containing numeric values in scientific notation without a dot. #629

1.9.0 - 2020-06-20

Added

  • Pass original case’s response to add_case hook.
  • Support for multiple examples with OpenAPI examples. #589
  • --verbosity CLI option to minimize the error output. #598
  • Allow register function-level hooks without passing their name as the first argument to apply. #618
  • Support for hook usage via LazySchema / from_pytest_fixture. #617

Changed

  • Tests with invalid schemas marked as errors, instead of failures. `#622`_

Fixed

  • Crash during generation of loosely-defined headers. #621
  • Show exception information for test runs on invalid schemas with --validate-schema=false command-line option. Before the output sections for invalid endpoints were empty. `#622`_

1.8.0 - 2020-06-15

Fixed

  • Tests with invalid schemas are marked as failed instead of passed when hypothesis-jsonschema>=0.16 is installed. #614
  • KeyError during creating endpoint strategy if it contains a reference. #612

Changed

  • Require hypothesis-jsonschema>=0.16. #614
  • Pass original InvalidSchema text to pytest.fail call.

1.7.0 - 2020-05-30

Added

  • Support for YAML files in references via HTTPS & HTTP schemas. #600
  • Stateful testing support via Open API links syntax. #548
  • New add_case hook. #458
  • Support for parameters serialization formats in Open API 2 / 3. For example pipeDelimited or deepObject. #599
  • Support serializing parameters with application/json content type. #594

Changed

  • The minimum required versions for Hypothesis and hypothesis-jsonschema are 5.15.0 and 0.11.1 respectively. The main reason is this fix that is required for stability of Open API links feature when is executed in multiple threads.

1.6.3 - 2020-05-26

Fixed

  • Support for a colon symbol (:) inside of a header value passed via CLI. #596

1.6.2 - 2020-05-15

Fixed

  • Partially generated explicit examples are always valid and can be used in requests. #582

1.6.1 - 2020-05-13

Changed

  • Look at the current working directory when loading hooks for CLI. #586

1.6.0 - 2020-05-10

Added

  • New before_add_examples hook. #571
  • New after_init_cli_run_handlers hook. #575

Fixed

  • Passing workers_num to ThreadPoolRunner which lead to always using 2 workers in this worker kind. #579

1.5.1 - 2020-05-08

Fixed

  • Display proper headers in reproduction code when headers are overridden. #566

1.5.0 - 2020-05-06

Added

  • Display a suggestion to disable schema validation on schema loading errors in CLI. #531
  • Filtration of endpoints by operationId via operation_id parameter to schema.parametrize or -O command-line option. #546
  • Generation of security-related parameters. They are taken from securityDefinitions / securitySchemes and injected to the generated data. It supports generating API keys in headers or query parameters and generating data for HTTP authentication schemes as well. #540

Fixed

  • Overriding header values in CLI and runner when headers provided explicitly clash with ones defined in the schema. #559
  • Nested references resolving in response_schema_conformance check. #562
  • Nullable parameters handling when they are behind a reference. #542

1.4.0 - 2020-05-03

Added

  • context argument for hook functions to provide an additional context for hooks. A deprecation warning is emitted for hook functions that do not accept this argument.
  • A new hook system that allows generic hook dispatching. It comes with new hook locations. For more details see “Customization” section in our documentation.
  • New before_process_path hook.
  • Third-party compatibility fixups mechanism. Currently there is one fixup for FastAPI. #503

Deprecated

  • Hook functions that do not accept context as their first argument. They will become not supported in Schemathesis 2.0.
  • Registering hooks by name and function. Use register decorators instead, for more details see “Customization” section in our documentation.
  • BaseSchema.with_hook and BaseSchema.register_hook. Use BaseSchema.hooks.apply and BaseSchema.hooks.register instead.

Fixed

  • Add missing validate_schema argument to loaders.from_pytest_fixture.
  • Reference resolving during response schema conformance check. #539

1.3.4 - 2020-04-30

Fixed

  • Validation of nullable properties in response_schema_conformance check introduced in 1.3.0. #542

1.3.3 - 2020-04-29

Changed

  • Update pytest-subtests pin to >=0.2.1,<1.0. #537

1.3.2 - 2020-04-27

Added

  • Show exceptions if they happened during loading a WSGI application. Option --show-errors-tracebacks will display a full traceback.

1.3.1 - 2020-04-27

Fixed

  • Packaging issue

1.3.0 - 2020-04-27

Added

  • Storing network logs with --store-network-log=<filename.yaml>. The stored cassettes are based on the VCR format and contain extra information from the Schemathesis internals. #379
  • Replaying of cassettes stored in VCR format. #519
  • Targeted property-based testing in CLI and runner. It only supports response_time target at the moment. #104
  • Export CLI test results to JUnit.xml with --junit-xml=<filename.xml>. #427

Fixed

  • Code samples for schemas where body is defined as {"type": "string"}. #521
  • Showing error causes on internal jsonschema errors during input schema validation. #513
  • Recursion error in response_schema_conformance check. Because of this change Endpoint.definition contains a definition where references are not resolved. In this way it makes possible to avoid recursion errors in jsonschema validation. #468

Changed

  • Added indentation & section name to the SUMMARY CLI block.
  • Use C-extension for YAML loading when it is possible. It can cause more than 10x speedup on schema parsing.
  • Do not show Click’s “Aborted!” message when an error occurs during schema loading in CLI.
  • Add a help message to the CLI output when an internal exception happens. #529

1.2.0 - 2020-04-15

Added

  • Per-test hooks for modification of data generation strategies. #492
  • Support for x-example vendor extension in Open API 2.0. #504
  • Sanity validation for the input schema & loader in runner.prepare. #499

1.1.2 - 2020-04-14

Fixed

  • Support for custom loaders in runner. Now all built-in loaders are supported as an argument to runner.prepare. #496
  • from_wsgi loader accept custom keyword arguments that will be passed to client.get when accessing the schema. #497

1.1.1 - 2020-04-12

Fixed

  • Mistakenly applied Open API -> JSON Schema Draft 7 conversion. It should be Draft 4. #489
  • Using wrong validator in response_schema_conformance check. It should be Draft 4 validator. #468

1.1.0 - 2020-04-08

Fixed

  • Response schema check for recursive schemas. #468

Changed

  • App loading in runner. Now it accepts application as an importable string, rather than an instance. It is done to make it possible to execute runner in a subprocess. Otherwise apps can’t be easily serialized and transferred into another process.
  • Runner events structure. All data in events is static from now, there are no references to BaseSchema, Endpoint or similar objects that may calculate data dynamically. This is done to make events serializable and not tied to Python object which decouples any runner consumer from implementation details and will help make runner usable in more cases (e.g. web application), since events can be serialized to JSON and used in any environment. Another related change is that Python exceptions are not propagated anymore - they are replaced with InternalError event that should be handled accordingly.

1.0.5 - 2020-04-03

Fixed

  • Open API 3. Handling of endpoints that contain multipart/form-data media types. Previously only file upload endpoints were working correctly. #473

1.0.4 - 2020-04-03

Fixed

  • OpenApi30.get_content_types behavior, introduced in 8aeee1a. #469

1.0.3 - 2020-04-03

Fixed

  • Precedence of produces keywords for Swagger 2.0 schemas. Now, operation-level produces overrides schema-level produces as specified in the specification. #463
  • Content Type conformance check for Open API 3.0 schemas. #461
  • Pytest 5.4 warning for test functions without parametrization. #451

1.0.2 - 2020-04-02

Fixed

  • Handling of fields in paths that are not operations, but allowed by the Open API spec. #457
  • Pytest 5.4 warning about deprecated Node initialization usage. #451

1.0.1 - 2020-04-01

Fixed

  • Processing of explicit examples in Open API 3.0 when there are multiple parameters in the same location (e.g. path) contain example value. They are properly combined now. #450

1.0.0 - 2020-03-31

Changed

  • Move processing of runner parameters to runner.prepare. This change will provide better code reusage, since all users of runner (e.g. if you extended it in your project`) need some kind of input parameters handling, which was implemented only in Schemathesis CLI. It is not backward-compatible. If you didn’t use runner directly, then this change should not have a visible effect for your use-case.

0.28.0 - 2020-03-31

Fixed

  • Handling of schemas, that use x-* custom properties. #448

Removed

  • Deprecated runner.execute. Use runner.prepare instead.

0.27.0 - 2020-03-31

Deprecated

  • runner.execute should not be used, since runner.prepare provides a more flexible interface to test execution.

Removed

  • Deprecated Parametrizer class. Use schemathesis.from_path as a replacement for Parametrizer.from_path.

0.26.1 - 2020-03-24

Fixed

  • Limit recursion depth while resolving JSON schema to handle recursion without breaking. #435

0.26.0 - 2020-03-19

Fixed

  • Filter problematic path template variables containing "/", or "%2F" url encoded. #440
  • Filter invalid empty "" path template variables. #439
  • Typo in a help message in the CLI output. #436

0.25.1 - 2020-03-09

Changed

  • Allow werkzeug >= 1.0.0. #433

0.25.0 - 2020-02-27

Changed

  • Handling of explicit examples from schemas. Now if there are examples for multiple locations (e.g. for body and for query) then they will be combined into a single example. #424

0.24.5 - 2020-02-26

Fixed

  • Error during pytest collection on objects that have custom __getattr__ method and therefore pass is_schemathesis check. #429

0.24.4 - 2020-02-22

Fixed

  • Resolving references when schema is loaded from a file on Windows. #418

0.24.3 - 2020-02-10

Fixed

  • Not copied validate_schema parameter in BaseSchema.parametrize. Regression after implementing #383
  • Missing app, location and hooks parameters in schema when used with BaseSchema.parametrize. #416

0.24.2 - 2020-02-09

Fixed

  • Crash on invalid regular expressions in method, endpoint and tag CLI options. #403
  • Crash on non latin-1 encodable value in auth CLI option. #404
  • Crash on invalid value in header CLI options. #405
  • Crash on some invalid URLs in schema CLI option. #406
  • Validation of --request-timeout parameter. #407
  • Crash with --hypothesis-deadline=0 CLI option. #410
  • Crash with --hypothesis-max-examples=0 CLI option. #412

0.24.1 - 2020-02-08

Fixed

  • CLI crash on Windows and Python < 3.8 when the schema path contains characters unrepresentable at the OS level. #400

0.24.0 - 2020-02-07

Added

  • Support for testing of examples in Parameter & Media Type objects in Open API 3.0. #394
  • --show-error-tracebacks CLI option to display errors’ tracebacks in the output. #391
  • Support for schema behind auth. #115

Changed

  • Schemas with GET endpoints accepting body are allowed now if schema validation is disabled (via --validate-schema=false for example). The usecase is for tools like ElasticSearch that use GET requests with non empty bodies. #383

Fixed

  • CLI crash when an explicit example is specified in endpoint definition. #386

0.23.7 - 2020-01-30

Added

  • -x/--exitfirst CLI option to exit after first failed test. #378

Fixed

  • Handling examples of parameters in Open API 3.0. #381

0.23.6 - 2020-01-28

Added

  • all variant for --checks CLI option to use all available checks. #374

Changed

  • Use built-in importlib.metadata on Python 3.8. #376

0.23.5 - 2020-01-24

Fixed

  • Generation of invalid values in Case.cookies. #371

0.23.4 - 2020-01-22

Fixed

  • Converting exclusiveMinimum & exclusiveMaximum fields to JSON Schema. #367

0.23.3 - 2020-01-21

Fixed

  • Filter out surrogate pairs from query string.

0.23.2 - 2020-01-16

Fixed

  • Prevent KeyError when response does not have Content-Type. #365

0.23.1 - 2020-01-15

Fixed

  • Dockerfile entrypoint was not working as per docs. #361

0.23.0 - 2020-01-15

Added

  • Hooks for strategy modification. #313
  • Input schema validation. Use --validate-schema=false to disable it in CLI and validate_schema=False argument in loaders. #110

0.22.0 - 2020-01-11

Added

  • Show multiple found failures in the CLI output. #266 & #207
  • Raise proper exception when the given schema is invalid. #308
  • Support for None as a value for --hypothesis-deadline. #349

Fixed

  • Handling binary request payloads in Case.call. #350
  • Type of the second argument to all built-in checks set to proper Case instead of TestResult. The error was didn’t affect built-in checks since both Case and TestResult had endpoint attribute and only it was used. However this fix is not backward-compatible with 3rd party checks.

0.21.0 - 2019-12-20

Added

  • Support for AioHTTP applications in CLI. #329

0.20.5 - 2019-12-18

Fixed

  • Compatibility with the latest release of hypothesis-jsonschema and setting its minimal required version to 0.9.13. #338

0.20.4 - 2019-12-17

Fixed

  • Handling nullable attribute in Open API schemas. #335

0.20.3 - 2019-12-17

Fixed

  • Response status code conformance check applicability for old requests version. #330

0.20.2 - 2019-12-14

Fixed

  • Response schema conformance check for Open API 3.0. #332

0.20.1 - 2019-12-13

Added

  • Support for response code ranges. #330

0.20.0 - 2019-12-12

Added

  • WSGI apps support. #31
  • Case.validate_response for running built-in checks against app’s response. #319

Changed

  • Checks receive Case instance as a second argument instead of TestResult. This was done for making checks usable in Python tests via Case.validate_response. Endpoint and schema are accessible via case.endpoint and case.endpoint.schema.

0.19.1 - 2019-12-11

Fixed

  • Compatibility with Hypothesis >= 4.53.2. #322

0.19.0 - 2019-12-02

Added

  • Concurrent test execution in CLI / runner. #91
  • update importlib_metadata pin to ^1.1. #315

0.18.1 - 2019-11-28

Fixed

  • Validation of base-url CLI parameter. #311

0.18.0 - 2019-11-27

Added

  • Resolving references in PathItem objects. #301

Fixed

  • Resolving of relative paths in schemas. #303
  • Loading string dates as datetime.date objects in YAML loader. #305

0.17.0 - 2019-11-21

Added

  • Resolving references that point to different files. #294

Changed

  • Keyboard interrupt is now handled during the CLI run and the summary is displayed in the output. #295

0.16.0 - 2019-11-19

Added

  • Display RNG seed in the CLI output to allow test reproducing. #267
  • Allow to specify seed in CLI.
  • Ability to pass custom kwargs to the requests.get call in loaders.from_uri.

Changed

  • Refactor case generation strategies: strategy is not used to generate empty value. #253
  • Improved error message for invalid path parameter declaration. #255

Fixed

  • Pytest fixture parametrization via pytest_generate_tests. #280
  • Support for tests defined as methods. #282
  • Unclosed requests.Session on calling Case.call without passing a session explicitly. #286

0.15.0 - 2019-11-15

Added

  • Support for OpenAPI 3.0 server variables (base_path). #40
  • Support for format: byte. #254
  • Response schema conformance check in CLI / Runner. #256
  • Docker image for CLI. #268
  • Pre-run hooks for CLI. #147
  • A way to register custom checks for CLI via schemathesis.register_check. #270

Fixed

  • Not encoded path parameters. #272

Changed

  • Verbose messages are displayed in the CLI on failed checks. #261

0.14.0 - 2019-11-09

Added

  • CLI: Support file paths in schema argument. #119
  • Checks to verify response status & content type in CLI / Runner. #101

Fixed

  • Custom base URL handling in CLI / Runner. #248

Changed

  • Raise an error if schema has body for GET requests. #218
  • Method names are case insensitive during direct schema access. #246

0.13.2 - 2019-11-05

Fixed

  • IndexError when Hypothesis found inconsistent test results during the test execution in runner. #236

0.13.1 - 2019-11-05

Added

  • Support for binary format #197

Fixed

  • Error that happens when there are no success checks in the statistic in CLI. #237

0.13.0 - 2019-11-03

Added

  • An option to configure request timeout for CLI / Runner. #204
  • A help snippet to reproduce errors caught by Schemathesis. #206
  • Total running time to the CLI output. #181
  • Summary line in the CLI output with the number of passed / failed / errored endpoint tests. #209
  • Extra information to the CLI output: schema address, spec version and base url. #188

Fixed

  • Compatibility with Hypothesis 4.42.4+ . #212
  • Display flaky errors only in the “ERRORS” section and improve CLI output. #215
  • Handling formData parameters in Case.call. #196
  • Handling cookies in Case.call. #211

Changed

  • More readable falsifying examples output. #127
  • Show exceptions in a separate section of the CLI output. #203
  • Error message for cases when it is not possible to satisfy schema parameters. It should be more clear now. #216
  • Do not stop on schema errors related to single endpoint. #139
  • Display a proper error message when schema is not available in CLI / Runner. #214

0.12.2 - 2019-10-30

Fixed

  • Wrong handling of base_url parameter in runner and Case.call if it has a trailing slash. #194 and #199
  • Do not send any payload with GET requests. #200

0.12.1 - 2019-10-28

Fixed

  • Handling for errors other than AssertionError and HypothesisException in the runner. #189
  • CLI failing on the case when there are tests, but no checks were performed. #191

Changed

  • Display “SUMMARY” section in the CLI output for empty test suites.

0.12.0 - 2019-10-28

Added

  • Display progress during the CLI run. #125

Fixed

  • Test server generated wrong schema when endpoints option is passed via CLI. #173
  • Error message if schema is not found in CLI. #172

Changed

  • Continue running tests on hypothesis error. #137

0.11.0 - 2019-10-22

Added

  • LazySchema accepts filters. #149
  • Ability to register strategies for custom string formats. #94
  • Generator-based events in runner module to improve control over the execution flow.
  • Filtration by tags. #134

Changed

  • Base URL in schema instances could be reused when it is defined during creation. Now on, base_url argument in Case.call is optional in such cases. #153
  • Hypothesis deadline is set to 500ms by default. #138
  • Hypothesis output is captured separately, without capturing the whole stdout during CLI run.
  • Disallow empty username in CLI --auth option.

Fixed

  • User agent during schema loading. #144
  • Generation of invalid values in Case.headers. #167

Removed

  • Undocumented support for file:// uri schema

0.10.0 - 2019-10-14

Added

  • HTTP Digest Auth support. #106
  • Support for Hypothesis settings in CLI & Runner. #107
  • Case.call and Case.as_requests_kwargs convenience methods. #109
  • Local development server. #126

Removed

  • Autogenerated runner.StatsCollector.__repr__ to make Hypothesis output more readable.

0.9.0 - 2019-10-09

Added

  • Test executor collects results of execution. #29
  • CLI option --base-url for specifying base URL of API. #118
  • Support for coroutine-based tests. #121
  • User Agent to network requests in CLI & runner. #130

Changed

  • CLI command schemathesis run prints results in a more readable way with a summary of passing checks.
  • Empty header names are forbidden for CLI.
  • Suppressed hypothesis exception about using example non-interactively. #92

0.8.1 - 2019-10-04

Fixed

  • Wrap each individual test in suppress so the runner doesn’t stop after the first test failure.

0.8.0 - 2019-10-04

Added

  • CLI tool invoked by the schemathesis command. #30
  • New arguments api_options, loader_options and loader for test executor. #90
  • A mapping interface for schemas & convenience methods for direct strategies access. #98

Fixed

  • Runner stopping on the first falsifying example. #99

0.7.3 - 2019-09-30

Fixed

  • Filtration in lazy loaders.

0.7.2 - 2019-09-30

Added

  • Support for type “file” for Swagger 2.0. #78
  • Support for filtering in loaders. #75

Fixed

  • Conflict for lazy schema filtering. #64

0.7.1 - 2019-09-27

Added

  • Support for x-nullable extension. #45

0.7.0 - 2019-09-26

Added

  • Support for cookie parameter in OpenAPI 3.0 schemas. #21
  • Support for formData parameter in Swagger 2.0 schemas. #6
  • Test executor. #28

Fixed

  • Using hypothesis.settings decorator with test functions created from from_pytest_fixture loader. #69

0.6.0 - 2019-09-24

Added

  • Parametrizing tests from a pytest fixture via pytest-subtests. #58

Changed

  • Rename module readers to loaders.
  • Rename parametrize parameters. filter_endpoint to endpoint and filter_method to method.

Removed

  • Substring match for method / endpoint filters. To avoid clashing with escaped chars in endpoints keys in schemas.

0.5.0 - 2019-09-16

Added

  • Generating explicit examples from schema. #17

Changed

  • Schemas are loaded eagerly from now on. Using schemathesis.from_uri implies network calls.

Deprecated

  • Using Parametrizer.from_{path,uri} is deprecated, use schemathesis.from_{path,uri} instead.

Fixed

  • Body resolving during test collection. #55

0.4.1 - 2019-09-11

Fixed

  • Possibly unhandled exception during hasattr check in is_schemathesis_test.

0.4.0 - 2019-09-10

Fixed

  • Resolving all inner references in objects. #34

Changed

  • jsonschema.RefResolver is now used for reference resolving. #35

0.3.0 - 2019-09-06

Added

  • Parametrizer.from_uri method to construct parametrizer instances from URIs. #24

Removed

  • Possibility to use Parametrizer.parametrize and custom Parametrizer kwargs for passing config options to hypothesis.settings. Use hypothesis.settings decorators on tests instead.

0.2.0 - 2019-09-05

Added

  • Open API 3.0 support. #10
  • “header” parameters. #7

Changed

  • Handle errors during collection / executions as failures.
  • Use re.search for pattern matching in filter_method/filter_endpoint instead of fnmatch. #18
  • Case.body contains properties from the target schema, without extra level of nesting.

Fixed

  • KeyError on collection when “basePath” is absent. #16

0.1.0 - 2019-06-28

  • Initial public release