Changelog
Unreleased - TBD
3.19.7 - 2023-09-03
Fixed
Unsatisfiable
error for multiple security schemes applied to the same API operation and an explicitAuthorization
header. #1763_
3.19.6 - 2023-08-14
Fixed
Broken
--report
CLI argument underclick>=8.1.4
. #1753
3.19.5 - 2023-06-03
Fixed
Do not raise
Unsatisfiable
when explicit headers are provided for negative tests.Do not raise
Unsatisfiable
when no headers can be negated.
3.19.4 - 2023-06-03
Fixed
Improved handling of negative test scenarios by not raising
Unsatisfiable
when path parameters cannot be negated but other parameters can be negated.
3.19.3 - 2023-05-25
Changed
Support
requests<3
. #1742Bump the minimum supported
Hypothesis
version to6.31.6
to reflect requirement fromhypothesis-jsonschema
.
Fixed
HypothesisDeprecationWarning
regarding deprecatedHealthCheck.all()
. #1739
3.19.2 - 2023-05-20
Added
You can now provide a tuple of checks to exclude when validating a response.
3.19.1 - 2023-04-26
Changed
Support
requests<2.29
.
Fixed
Passing
params
/cookies
tocase.call
causingTypeError
. #1734
Removed
Direct dependency on
attrs
.
3.19.0 - 2023-03-22
Added
Schemathesis now supports custom authentication mechanisms from the
requests
library. You can useschemathesis.auth.set_from_requests
to set up Schemathesis CLI with any third-party authentication implementation that works withrequests
. #1700
import schemathesis
from requests_ntlm import HttpNtlmAuth
schemathesis.auth.set_from_requests(HttpNtlmAuth("domain\\username", "password"))
Ability to apply authentication conditionally to specific API operations using a combination of
@schemathesis.auth.apply_to()
and@schemathesis.auth.skip_for()
decorators.
import schemathesis
# Apply auth only for operations that path starts with `/users/` but not the `POST` method
@schemathesis.auth().apply_to(path_regex="^/users/").skip_for(method="POST")
class MyAuth:
...
Add a convenience mapping-like interface to
OperationDefinition
including indexing access, theget
method, and “in” support.Request throttling via the
--rate-limit
CLI option. #910
Changed
Unified Schemathesis custom authentication usage via the
schema.auth
decorator, replacing the previousschema.auth.register
andschema.auth.apply
methods:
import schemathesis
schema = schemathesis.from_uri("https://example.schemathesis.io/openapi.json")
# Schema-level auth
# Before: @schema.auth.register()
@schema.auth()
class MyAuth:
...
# Test-level auth
# Before: @schema.auth.apply(MyAuth)
@schema.auth(MyAuth)
@schema.parametrize()
def test_api(case):
...
Fixed
Handling of query parameters and cookies passed to
case.call
and query parameters passed tocase.call_wsgi
. The user-provided values are now merged with the data generated by Schemathesis, instead of overriding it completely. #1705Parameter definition takes precedence over security schemes with the same name.
Unsatisfiable
error when explicit header name passed via CLI clashes with the header parameter name. #1699Not using the
port
keyword argument in schema loaders during API schema loading. #1721
3.18.5 - 2023-02-18
Added
Support for specifying the path to load hooks from via the
SCHEMATHESIS_HOOKS
environment variable. #1702.
Deprecated
Use of the
--pre-run
CLI option for loading hooks. Use theSCHEMATHESIS_HOOKS
environment variable instead.
3.18.4 - 2023-02-16
Changed
Support any Werkzeug 2.x in order to allow mitigation of CVE-2023-25577. #1695
3.18.3 - 2023-02-12
Added
APIStateMachine.run
method to simplify running stateful tests.
Changed
Improved quality of generated test sequences by updating state machines in Schemathesis to always run a minimum of two steps during testing. #1627 If you use
hypothesis.stateful.run_state_machine_as_test
to run your stateful tests, please use therun
method on your state machine class instead. This change requires upgradingHypothesis
to at least version6.68.1
.
3.18.2 - 2023-02-08
Performance
Modify values in-place inside built-in
map
functions as there is no need to copy them.Update
hypothesis-jsonschema
to0.22.1
for up to 30% faster data generation in some workflows.
3.18.1 - 2023-02-06
Changed
Stateful testing: Only make stateful requests when stateful data is available from another operation. This change significantly reduces the number of API calls that likely will fail because of absense of stateful data. #1669
Performance
Do not merge component schemas into the currently tested schema if they are not referenced by it. Originally all schemas were merged to make them visible to
hypothesis-jsonschema
, but they imply significant overhead. #1180Use a faster, specialized version of
deepcopy
.
3.18.0 - 2023-02-01
Added
Extra information to VCR cassettes.
The
--contrib-unique-data
CLI option that forces Schemathesis to generate unique test cases only. This feature is also available as a hook inschemathesis.contrib.unique_data
.- A few decorators & functions that provide a simpler API to extend Schemathesis:
schemathesis.auth()
for authentication providers;schemathesis.check
for checks;schemathesis.hook
&BaseSchema.hook
for hooks;schemathesis.serializer
for serializers;schemathesis.target
for targets;schemathesis.openapi.format
for custom OpenAPI formats.schemathesis.graphql.scalar
for GraphQL scalars.
Open API: UUID format generation via the
schemathesis.contrib.openapi.formats.uuid
extension You could enable it via the--contrib-openapi-formats-uuid
CLI option.
Changed
Build: Switch the build backend to Hatch.
Relax requirements for
attrs
. #1643Avoid occasional empty lines in cassettes.
Deprecated
schemathesis.register_check
in favor ofschemathesis.check
.schemathesis.register_target
in favor ofschemathesis.target
.schemathesis.register_string_format
in favor ofschemathesis.openapi.format
.schemathesis.graphql.register_scalar
in favor ofschemathesis.graphql.scalar
.schemathesis.auth.register
in favor ofschemathesis.auth
.
Fixed
Remove recursive references from the last reference resolution level. It works on the best effort basis and does not cover all possible cases. #947
Invalid cassettes when headers contain characters with a special meaning in YAML.
Properly display flaky deadline errors.
Internal error when the
utf8_bom
fixup is used for WSGI apps.Printing header that are set explicitly via
get_call_kwargs
in stateful testing. #828Display all explicitly defined headers in the generated cURL command.
Replace
starlette.testclient.TestClient
withstarlette_testclient.TestClient
to keep compatibility with newerstarlette
versions. #1637
Performance
Running negative tests filters out less data.
Schema loading: Try a faster loader first if an HTTP response or a file is expected to be JSON.
3.17.5 - 2022-11-08
Added
Python 3.11 support. #1632
Fixed
Allow
Werkzeug<=2.2.2
. #1631
3.17.4 - 2022-10-19
Fixed
Appending an extra slash to the
/
path. #1625
3.17.3 - 2022-10-10
Fixed
Missing
httpx
dependency. #1614
3.17.2 - 2022-08-27
Fixed
Insufficient timeout for report uploads.
3.17.1 - 2022-08-19
Changed
Support
requests==2.28.1
.
3.17.0 - 2022-08-17
Added
Support for exception groups in newer
Hypothesis
versions. #1592A way to generate negative and positive test cases within the same CLI run via
-D all
.
Fixed
Allow creating APIs in Schemathesis.io by name when the schema is passed as a file.
Properly trim tracebacks on
Hypothesis>=6.54.0
.Skipping negative tests when they should not be skipped.
Changed
pytest: Generate positive & negative within the same test node.
CLI: Warning if there are too many HTTP 403 API responses.
Runner:
BeforeExecution.data_generation_method
andAfterExecution.data_generation_method
changed to lists ofDataGenerationMethod
as the same test may contain data coming from different data generation methods.
3.16.5 - 2022-08-11
Fixed
CLI: Hanging on
CTRL-C
when--report
is enabled.Internal error when GraphQL schema has its root types renamed. #1591
3.16.4 - 2022-08-09
Changed
Suggest using
--wait-for-schema
if API schema is not available.
3.16.3 - 2022-08-08
Added
CLI:
--max-failures=N
option to exit after firstN
failures or errors. #1580CLI:
--wait-for-schema=N
option to automatically retry schema loading forN
seconds. #1582CLI: Display old and new payloads in
st replay
when the-v
option is passed. #1584
Fixed
Internal error on generating negative tests for query parameters with
explode: true
.
3.16.2 - 2022-08-05
Added
CLI: Warning if ALL API responses are HTTP 404.
The
after_load_schema
hook, which is designed for modifying the loaded API schema before running tests. For example, you can use it to add Open API links to your schema viaschema.add_link
.New
utf8_bom
fixup. It helps to mitigate JSON decoding errors inside theresponse_schema_conformance
check when payload contains BOM. #1563
Fixed
Description of
-v
or--verbosity
option for CLI.
Changed
Execute
before_call
/after_call
hooks inside thecall_*
methods. It makes them available for thepytest
integration.
3.16.1 - 2022-07-29
Added
CLI: Warning if the API returns too many HTTP 401.
Add
SCHEMATHESIS_BASE_URL
environment variable for specifying--base-url
in CLI.Collect anonymyzed CLI usage telemetry when reports are uploaded. We do not collect any free-form values you use in your CLI, except for header names. Instead, we measure how many times you use each free-form option in this command. Additionally we count all non-default hook types only by hook name.
Important
You can disable usage this with the --schemathesis-io-telemetry=false
CLI option or the SCHEMATHESIS_TELEMETRY=false
environment variable.
3.16.0 - 2022-07-22
Added
Report uploading to Schemathesis.io via the
--report
CLI option.
Changed
Do not validate schemas by default in the
pytest
integration.CLI: Display test run environment metadata only if
-v
is provided.CLI: Do not display headers automatically added by
requests
in code samples.
Fixed
Do not report optional headers as missing.
Compatibility with
hypothesis>=6.49
. #1538Handling of
unittest.case.SkipTest
emitted by newer Hypothesis versions.Generating invalid headers when their schema has
array
orobject
types.
Removed
Previously, data was uploaded to Schemathesis.io when the proper credentials were specified. This release removes this behavior. From now on, every upload requires the explicit
--report
CLI option.Textual representation of HTTP requests in CLI output in order to decrease verbosity and avoid showing the same data in multiple places.
3.15.6 - 2022-06-23
Fixed
Do not discard dots (
.
) in OpenAPI expressions during parsing.
3.15.5 - 2022-06-21
Fixed
TypeError
when using--auth-type=digest
in CLI.
3.15.4 - 2022-06-06
Added
Support generating data for Open API request payloads with wildcard media types. #1526
Changed
Mark tests as skipped if there are no explicit examples and
--hypothesis-phases=explicit
is used. #1323Parse all YAML mapping keys as strings, ignoring the YAML grammar rules. For example,
on: true
will be parsed as{"on": True}
instead of{True: True}
. Even though YAML does not restrict keys to strings, in the Open API and JSON Schema context, this restriction is implied because the underlying data model comes from JSON.INTERNAL: Improve flexibility of event serialization.
INTERNAL: Store request / response history in
SerializedCheck
.
3.15.3 - 2022-05-28
Fixed
3.15.2 - 2022-05-09
Fixed
Avoid generating negative query samples that
requests
will treat as an empty query.Editable installation via
pip
.
3.15.1 - 2022-05-03
Added
OpenAPI: Expose
APIOperation.get_security_requirements
that returns a list of security requirements applied to the API operationAttach originally failed checks to “grouped” exceptions.
Fixed
Internal error when Schemathesis doesn’t have permission to create its
hosts.toml
file.Do not show internal Hypothesis warning multiple times when the Hypothesis database directory is not usable.
Do not print not relevant Hypothesis reports when run in CI.
Invalid
verbose_name
value inSerializedCase
for GraphQL tests.
3.15.0 - 2022-05-01
Added
GraphQL: Mutations supports. Schemathesis will generate random mutations by default from now on.
GraphQL: Support for registering strategies to generate custom scalars.
Custom auth support for schemas created via
from_pytest_fixture
.
Changed
Do not encode payloads in cassettes as base64 by default. This change makes Schemathesis match the default Ruby’s VCR behavior and leads to more human-readable cassettes. Use
--cassette-preserve-exact-body-bytes
to restore the old behavior. #1413Bump
hypothesis-graphql
to0.9.0
.Avoid simultaneous authentication requests inside auth providers when caching is enabled.
Reduce the verbosity of
pytest
output. A few internal frames and the “Falsifying example” block are removed from the output.Skip negative tests on API operations that are not possible to negate. #1463
Make it possible to generate negative tests if at least one parameter can be negated.
Treat flaky errors as failures and display full report about the failure. #1081
Do not duplicate failing explicit example in the HYPOTHESIS OUTPUT CLI output section. #881
Fixed
GraphQL: Semantically invalid queries without aliases.
GraphQL: Rare crashes on invalid schemas.
Internal error inside
BaseOpenAPISchema.validate_response
onrequests>=2.27
when response body contains malformed JSON. #1485schemathesis.from_pytest_fixture
: Display each failure if Hypothesis found multiple of them.
Performance
GraphQL: Over 2x improvement from internal optimizations.
3.14.2 - 2022-04-21
Added
Support for auth customization & automatic refreshing. #966
3.14.1 - 2022-04-18
Fixed
Using
@schema.parametrize
with test methods onpytest>=7.0
.
3.14.0 - 2022-04-17
Added
Open API link name customization via the
name
argument toschema.add_link
.st
as an alias to theschemathesis
command line entrypoint.st auth login
/st auth logout
to authenticate with Schemathesis.io.X-Schemathesis-TestCaseId
header to help to distinguish test cases on the application side. #1303Support for comma separated lists in the
--checks
CLI option. #1373Hypothesis Database configuration for CLI via the
--hypothesis-database
option. #1326Make the
SCHEMA
CLI argument accept API names from Schemathesis.io.
Changed
Enable Open API links traversal by default. To disable it, use
--stateful=none
.Do not validate API schema by default. To enable it back, use
--validate-schema=true
.Add the
api_name
CLI argument to upload data to Schemathesis.io.Show response status code on failing checks output in CLI.
Improve error message on malformed Open API path templates (like
/foo}/
). #1372Improve error message on malformed media types that appear in the schema or in response headers. #1382
Relax dependencies on
pyyaml
andclick
.Add
--cassette-path
that is going to replace--store-network-log
. The old option is deprecated and will be removed in Schemathesis4.0
Fixed
Show the proper Hypothesis configuration in the CLI output. #1445
Missing
source
attribute in theCase.partial_deepcopy
implementation. #1429Duplicated failure message from
content_type_conformance
andresponse_schema_conformance
checks when the checked response has noContent-Type
header. #1394Not copied
case
&response
insideCase.validate_response
.Ignored
pytest.mark
decorators when they are applied beforeschema.parametrize
if the schema is created viafrom_pytest_fixture
. #1378
3.13.9 - 2022-04-14
Fixed
Compatibility with
pytest-asyncio>=0.17.1
. #1452
3.13.8 - 2022-04-05
Fixed
Missing
media_type
in theCase.partial_deepcopy
implementation. It led to missing payload in failure reproduction code samples.
3.13.7 - 2022-04-02
Added
Support for
Hypothesis>=6.41.0
. #1425
3.13.6 - 2022-03-31
Changed
Deep-clone
Response
instances before passing to check functions.
3.13.5 - 2022-03-31
Changed
Deep-clone
Case
instances before passing to check functions.
3.13.4 - 2022-03-29
Added
Support for
Werkzeug>=2.1.0
. #1410
Changed
Validate
requests
kwargs to catch cases when the ASGI integration is used, but the proper ASGI client is not supplied. #1335
3.13.3 - 2022-02-20
Added
--request-tls-verify
CLI option for thereplay
command. It controls whether Schemathesis verifies the server’s TLS certificate. You can also pass the path to a CA_BUNDLE file for private certs. #1395Support for client certificate authentication with
--request-cert
and--request-cert-key
arguments for thereplay
command.
3.13.2 - 2022-02-16
Changed
Use Schemathesis default User-Agent when communicating with SaaS.
Fixed
Use the same
correlation_id
inBeforeExecution
andAfterExecution
events if the API schema contains an error that causes anInvalidSchema
exception during test execution.Use
full_path
in error messages in recoverable schema-level errors. It makes events generated in such cases consistent with usual events.
3.13.1 - 2022-02-10
Added
APIOperation.iter_parameters
helper to iterate over all parameters.
Fixed
Properly handle error if Open API parameter doesn’t have
content
orschema
keywords.
3.13.0 - 2022-02-09
Changed
Update integration with Schemathesis.io.
Always show traceback for errors in Schemathesis.io integration.
3.12.3 - 2022-01-13
Fixed
Generating illegal unicode surrogates in queries. #1370
3.12.2 - 2022-01-12
Fixed
Not-escaped single quotes in generated Python code samples. #1359
3.12.1 - 2021-12-31
Fixed
Improper handling of
base_url
incall_asgi
, when the base URL has a non-empty base path. #1366
3.12.0 - 2021-12-29
Changed
Upgrade
typing-extensions
to>=3.7,<5
.Upgrade
jsonschema
to^4.3.2
.Upgrade
hypothesis-jsonschema
to>=0.22.0
.
Fixed
Removed
Support for Python 3.6.
3.11.7 - 2021-12-23
Added
Support for Python 3.10. #1292
3.11.6 - 2021-12-20
Added
3.11.5 - 2021-12-04
Changed
Generate tests for API operations with the HTTP
TRACE
method on Open API 2.0.
3.11.4 - 2021-12-03
Changed
Add AfterExecution.data_generation_method.
Minor changes to the Schemathesis.io integration.
3.11.3 - 2021-12-02
Fixed
3.11.2 - 2021-11-30
Changed
Use
name
&data_generation_method
parameters to subtest context instead ofpath
&method
. It allows the end-user to disambiguate among subtest reports.Raise an error if a test function wrapped with
schema.parametrize
matches no API operations. #1336
Fixed
Handle
KeyboardInterrupt
that happens outside of the main test loop inside the runner. It makes interrupt handling consistent, independent at what point it happens. #1325Respect the
data_generation_methods
config option defined on a schema instance when it is loaded viafrom_pytest_fixture
. #1331Ignored hooks defined on a schema instance when it is loaded via
from_pytest_fixture
. #1340
3.11.1 - 2021-11-20
Changed
Update
click
andPyYaml
dependency versions. #1328
3.11.0 - 2021-11-03
Changed
Show
cURL
code samples by default instead of Python. #1269Improve reporting of
jsonschema
errors which are caused by non-string object keys.Store
data_generation_method
inBeforeExecution
.Use case-insensitive dictionary for
Case.headers
. #1280
Fixed
Pass
data_generation_method
toCase
for GraphQL schemas.Generation of invalid headers in some cases. #1142
Unescaped quotes in generated Python code samples on some schemas. #1030
Performance
Dramatically improve CLI startup performance for large API schemas.
Open API 3: Inline only
components/schemas
before passing schemas tohypothesis-jsonschema
.Generate tests on demand when multiple workers are used during CLI runs. #1287
3.10.1 - 2021-10-04
Added
DataGenerationMethod.all
shortcut to get all possible enum variants.
Fixed
Unresolvable dependency due to incompatible changes in the new
hypothesis-jsonschema
release. #1290
3.10.0 - 2021-09-13
Added
Optional integration with Schemathesis.io.
New
before_init_operation
hook.INTERNAL.
description
attribute for all parsed parameters insideAPIOperation
.Timeouts when loading external schema components or external examples.
Changed
Pin
werkzeug
to>=0.16.0
.INTERNAL.
OpenAPI20CompositeBody.definition
type toList[OpenAPI20Parameter]
.Open API schema loaders now also accept single
DataGenerationMethod
instances for thedata_generation_methods
argument. #1260Improve error messages when the loaded API schema is not in JSON or YAML. #1262
Fixed
Internal error in
make_case
calls for GraphQL schemas.TypeError
oncase.call
with bytes data on GraphQL schemas.Worker threads may not be immediately stopped on SIGINT. #1066
Re-used referenced objects during inlining. Now they are independent.
Rewrite not resolved remote references to local ones. #986
Stop worker threads on failures with
exit_first
enabled. #1204Properly report all failures when custom checks are passed to
case.validate_response
.
Performance
Avoid using filters for header values when is not necessary.
3.9.7 - 2021-07-26
Added
New
process_call_kwargs
CLI hook. #1233
Changed
Check non-string response status codes when Open API links are collected. #1226
3.9.6 - 2021-07-15
Added
3.9.5 - 2021-07-14
Fixed
Preserve non-body parameter types in requests during Open API runtime expression evaluation.
3.9.4 - 2021-07-09
Fixed
3.9.3 - 2021-06-22
Added
ExecutionEvent.is_terminal
attribute that indicates whether an event is the last one in the stream.
Fixed
When
EventStream.stop
is called, the next event always is the last one.
3.9.2 - 2021-06-16
Changed
Return
response
fromCase.call_and_validate
.
Fixed
Incorrect deduplication applied to response schema conformance failures that happen to have the same failing validator but different input values. #907
3.9.1 - 2021-06-13
Changed
ExecutionEvent.asdict
adds theevent_type
field which is the event class name.Add API schema to the
Initialized
event.Internal: Add
SerializedCase.cookies
Convert all
FailureContext
class attributes to instance attributes. For simpler serialization viaattrs
.
3.9.0 - 2021-06-07
Added
GraphQL support in CLI. #746
A way to stop the Schemathesis runner’s event stream manually via
events.stop()
/events.finish()
methods. #1202
Changed
Avoid
pytest
warnings when internal Schemathesis classes are in the test module scope.
3.8.0 - 2021-06-03
Added
Negative testing. #65
Case.data_generation_method
attribute that provides the information of the underlying data generation method (e.g. positive or negative)
Changed
Raise
UsageError
ifschema.parametrize
orschema.given
are applied to the same function more than once. #1194Python values of
True
,False
andNone
are converted to their JSON equivalents when generated for path parameters or query. #1166Bump
hypothesis-jsonschema
version. It allows the end-user to override known string formats.Bump
hypothesis
version.APIOperation.make_case
behavior. If nomedia_type
is passed along withbody
, then it tries to infer the proper media type and raises an error if it is not possible. #1094
Fixed
Compatibility with
hypothesis>=6.13.3
.
3.7.8 - 2021-06-02
Fixed
Open API
style
&explode
for parameters derived from security definitions.
3.7.7 - 2021-06-01
Fixed
Apply the Open API’s
style
&explode
keywords to explicit examples. #1190
3.7.6 - 2021-05-31
Fixed
Disable filtering optimization for headers when there are keywords other than
type
. #1189
3.7.5 - 2021-05-31
Fixed
Too much filtering in headers that have schemas with the
pattern
keyword. #1189
3.7.4 - 2021-05-28
Changed
Internal:
SerializedCase.path_template
returns path templates as they are in the schema, without base path.
3.7.3 - 2021-05-28
Fixed
Invalid multipart payload generated for unusual schemas for the
multipart/form-data
media type.
Performance
Reduce the amount of filtering needed to generate valid headers and cookies.
3.7.2 - 2021-05-27
Added
SerializedCase.media_type
that stores the information about what media type was used for a particular case.
Fixed
Internal error on unusual schemas for the
multipart/form-data
media type. #1152Ignored explicit
Content-Type
override inCase.as_requests_kwargs
.
3.7.1 - 2021-05-23
Added
Internal:
FailureContext.title
attribute that gives a short failure description.Internal:
FailureContext.message
attribute that gives a longer failure description.
Changed
Rename
JSONDecodeErrorContext.message
toJSONDecodeErrorContext.validation_message
for consistency.Store the more precise
schema
&instance
inValidationErrorContext
.Rename
ResponseTimeout
toRequestTimeout
.
3.7.0 - 2021-05-23
Added
Additional context for each failure coming from the runner. It allows the end-user to customize failure formatting.
Changed
Use different exception classes for
not_a_server_error
andstatus_code_conformance
checks. It improves the variance of found errors.All network requests (not WSGI) now have the default timeout of 10 seconds. If the response is time-outing, Schemathesis will report it as a failure. It also solves the case when the tested app hangs. #1164
The default test duration deadline is extended to 15 seconds.
3.6.11 - 2021-05-20
Added
Internal:
BeforeExecution.verbose_name
&SerializedCase.verbose_name
that reflect specification-specific API operation name.
3.6.10 - 2021-05-17
Changed
Explicitly add
colorama
to project’s dependencies.Bump
hypothesis-jsonschema
version.
3.6.9 - 2021-05-14
Fixed
3.6.8 - 2021-05-13
Changed
Relax dependency on
starlette
to>=0.13,<1
. #1160
3.6.7 - 2021-05-12
Fixed
Missing support for the
date
string format (onlyfull-date
was supported).
3.6.6 - 2021-05-07
Changed
Improve error message for failing Hypothesis deadline healthcheck in CLI. #880
3.6.5 - 2021-05-07
Added
Support for disabling ANSI color escape codes via the NO_COLOR <https://no-color.org/> environment variable or the
--no-color
CLI option. #1153
Changed
Generate valid header values for Bearer auth by construction rather than by filtering.
3.6.4 - 2021-04-30
Changed
Bump minimum
hypothesis-graphql
version to0.5.0
. It brings support for interfaces and unions and fixes a couple of bugs in query generation.
3.6.3 - 2021-04-20
Fixed
3.6.2 - 2021-04-15
Fixed
Windows:
UnicodeDecodeError
during schema loading via thefrom_path
loader if it contains certain Unicode symbols.from_path
loader defaults to UTF-8 from now on.
3.6.1 - 2021-04-09
Fixed
Using parametrized
pytest
fixtures with thefrom_pytest_fixture
loader. #1121
3.6.0 - 2021-04-04
Added
Custom keyword arguments to
schemathesis.graphql.from_url
that are proxied torequests.post
.from_wsgi
,from_asgi
,from_path
andfrom_file
loaders for GraphQL apps. #1097, #1100Support for
data_generation_methods
andcode_sample_style
in all GraphQL loaders.Support for
app
&base_url
arguments for thefrom_pytest_fixture
runner.Initial support for GraphQL schemas in the Schemathesis runner.
import schemathesis
# Load schema
schema = schemathesis.graphql.from_url("http://localhost:8000/graphql")
# Initialize runner
runner = schemathesis.runner.from_schema(schema)
# Emit events
for event in runner.execute():
...
Breaking
Loaders’ signatures are unified. Most of the arguments became keyword-only. All except the first two for ASGI/WSGI, all except the first one for the others. It forces loader calls to be more consistent.
# BEFORE
schema = schemathesis.from_uri(
"http://example.com/openapi.json", "http://127.0.0.1:8000/", "GET"
)
# NOW
schema = schemathesis.from_uri(
"http://example.com/openapi.json", base_url="http://127.0.0.1:8000/", method="GET"
)
Changed
Schemathesis generates separate tests for each field defined in the GraphQL
Query
type. It makes the testing process unified for both Open API and GraphQL schemas.IDs for GraphQL tests use the corresponding
Query
field instead of HTTP method & path.Do not show overly verbose raw schemas in Hypothesis output for failed GraphQL tests.
The
schemathesis.graphql.from_url
loader now uses the usual Schemathesis User-Agent.The Hypothesis database now uses separate entries for each API operation when executed via CLI. It increases its effectiveness when tests are re-run.
Module
schemathesis.loaders
is moved toschemathesis.specs.openapi.loaders
.Show a more specific exception on incorrect usage of the
from_path
loader in the Schemathesis runner.
Deprecated
schemathesis.runner.prepare
will be removed in Schemathesis 4.0. Useschemathesis.runner.from_schema
instead. With this change, the schema loading part goes to your code, similar to using the regular Schemathesis Python API. It leads to a unified user experience where the starting point is API schema loading, which is much clearer than passing a callback & keyword arguments to theprepare
function.
Fixed
Add the missing
@schema.given
implementation for schemas created via thefrom_pytest_fixture
loader. #1093Silently ignoring some incorrect usages of
@schema.given
.Fixups examples were using the incorrect fixup name.
Return type of
make_case
for GraphQL schemas.Missed
operation_id
argument infrom_asgi
loader.
Removed
Undocumented way to install fixups via the
fixups
argument forschemathesis.runner.prepare
is removed.
3.5.3 - 2021-03-27
Fixed
Do not use importlib-metadata==3.8 in dependencies as it causes
RuntimeError
. Ref: https://github.com/python/importlib_metadata/issues/293
3.5.2 - 2021-03-24
Changed
Prefix worker thread names with
schemathesis_
.
3.5.1 - 2021-03-23
Fixed
Encoding for response payloads displayed in the CLI output. #1073
Use actual charset (from
flask.Response.mimetype_params
) when storing WSGI responses rather than defaulting toflask.Response.charset
.
3.5.0 - 2021-03-22
Added
before_generate_case
hook, that allows the user to modify or filter generatedCase
instances. #1067
Fixed
3.4.1 - 2021-03-21
Added
event_type
field to the debug output.
3.4.0 - 2021-03-20
Added
--debug-output-file
CLI option to enable storing the underlying runner events in the JSON Lines format in a separate file for debugging purposes. #1059
Changed
Make
Request.body
,Response.body
andResponse.encoding
internal attributes optional. ForRequest
, it means that absent body will lead toRequest.body
to beNone
. ForResponse
,body
will beNone
if the app response did not have any payload. Previously these values were empty strings, which was not distinguishable from the cases described above. For the end-user, it means that in VCR cassettes, fieldsrequest.body
andresponse.body
may be absent.models.Status
enum now has string values for more readable representation.
3.3.1 - 2021-03-18
Fixed
Displaying wrong headers in the
FAILURES
block of the CLI output. #792
3.3.0 - 2021-03-17
Added
Display failing response payload in the CLI output, similarly to the pytest plugin output. #1050
A way to control which code sample style to use - Python or cURL. #908
Fixed
UnicodeDecodeError
when generating cURL commands for failed test case reproduction if the request’s body contains non-UTF8 characters.
Internal
Extra information to events, emitted by the Schemathesis runner.
3.2.2 - 2021-03-11
Added
Support for Hypothesis 6. #1013
3.2.1 - 2021-03-10
Fixed
Wrong test results in some cases when the tested schema contains a media type that Schemathesis doesn’t know how to work with. #1046
3.2.0 - 2021-03-09
Performance
Add an internal caching layer for data generation strategies. It relies on the fact that the internal
BaseSchema
structure is not mutated over time. It is not directly possible through the public API and is discouraged from doing through hook functions.
Changed
APIOperation
and subclasses ofParameter
are now compared by their identity rather than by value.
3.1.3 - 2021-03-08
Added
count_operations
boolean flag torunner.prepare
. In case ofFalse
value, Schemathesis won’t count the total number of operations upfront. It improves performance for the directrunner
usage, especially on large schemas. Schemathesis CLI will still use these calculations to display the progress during execution, but this behavior may become configurable in the future.
3.1.2 - 2021-03-08
Fixed
Percent-encode the generated
.
and..
strings in path parameters to avoid resolving relative paths and changing the tested path structure. #1036
3.1.1 - 2021-03-05
Fixed
Loosen
importlib-metadata
version constraint and update pyproject.toml #1039
3.1.0 - 2021-02-11
Added
Support for external examples via the
externalValue
keyword. #884
Fixed
Prevent a small terminal width causing a crash (due to negative length used in an f-string) when printing percentage
Support the latest
cryptography
version in Docker images. #1033
3.0.9 - 2021-02-10
Fixed
Return a default terminal size to prevent crashes on systems with zero-width terminals (some CI/CD servers).
3.0.8 - 2021-02-04
This release updates the documentation to be in-line with the current state.
3.0.7 - 2021-01-31
Fixed
Docker tags for Buster-based images.
3.0.6 - 2021-01-31
Packaging-only release for Docker images based on Debian Buster. #1028
3.0.5 - 2021-01-30
Fixed
Allow to use any iterable type for
checks
andadditional_checks
arguments toCase.validate_response
.
3.0.4 - 2021-01-19
Fixed
Generating stateful tests, with common parameters behind a reference. #1020
Programmatic addition of Open API links via
add_link
when schema validation is disabled and response status codes are noted as integers. #1022
Changed
When operations are resolved by
operationId
then the same reference resolving logic is applied as in other cases. This change leads to less reference inlining and lower memory consumption for deeply nested schemas. #945
3.0.3 - 2021-01-18
Fixed
Flaky
Hypothesis error during explicit examples generation. #1018
3.0.2 - 2021-01-15
Fixed
Processing parameters common for multiple API operations if they are behind a reference. #1015
3.0.1 - 2021-01-15
Added
YAML serialization for
text/yaml
,text/x-yaml
,application/x-yaml
andtext/vnd.yaml
media types. #1010.
3.0.0 - 2021-01-14
Added
Support for sending
text/plain
payload as test data. Including variants with non-defaultcharset
. #850, #939Generating data for all media types defined for an operation. #690
Support for user-defined media types serialization. You can define how Schemathesis should handle media types defined in your schema or customize existing (like
application/json
).The response_schema_conformance check now runs on media types that are encoded with JSON. For example,
application/problem+json
. #920Base URL for GraphQL schemas. It allows you to load the schema from one place but send test requests to another one. #934
A helpful error message when an operation is not found during the direct schema access. #812
--dry-run
CLI option. When applied, Schemathesis won’t send any data to the server and won’t perform any response checks. #963A better error message when the API schema contains an invalid regular expression syntax. #1003
Changed
Open API parameters parsing to unblock supporting multiple media types per operation. Their definitions aren’t converted to JSON Schema equivalents right away but deferred instead and stored as-is.
Missing
required: true
in path parameters definition is now automatically enforced if schema validation is disabled. According to the Open API spec, therequired
keyword value should betrue
for path parameters. This change allows Schemathesis to generate test cases even for endpoints containing optional path parameters (which is not compliant with the spec). #941Using
--auth
together with--header
that sets theAuthorization
header causes a validation error. Before, the--header
value was ignored in such cases, and the basic auth passed in--auth
was used. #911When
hypothesis-jsonschema
fails to resolve recursive references, the test is skipped with an error message that indicates why it happens.Shorter error messages when API operations have logical errors in their schema. For example, when the maximum is less than the minimum -
{"type": "integer", "minimum": 5, "maximum": 4}
.If multiple non-check related failures happens during a test of a single API operation, they are displayed as is, instead of Hypothesis-level error messages about multiple found failures or flaky tests. #975
Catch schema parsing errors, that are caused by YAML parsing.
The built-in test server now accepts
--operations
instead of--endpoints
.Display
Collected API operations
instead ofcollected endpoints
in the CLI. #869--skip-deprecated-endpoints
is renamed to--skip-deprecated-operations
. #869Rename various internal API methods that contained
endpoint
in their names. #869Bump
hypothesis-jsonschema
version to0.19.0
. This version improves the handling of unsupported regular expression syntax and can generate data for a subset of schemas containing such regular expressions.Schemathesis doesn’t stop testing on errors during schema parsing. These errors are handled the same way as other errors during the testing process. It allows Schemathesis to test API operations with valid definitions and report problematic operations instead of failing the whole run. #999
Fixed
Allow generating requests without payload if the schema does not require it. #916
Allow sending
null
as request payload if the schema expects it. #919CLI failure if the tested operation is GET and has payload examples. #925
Excessive reference inlining that leads to out-of-memory for large schemas with deep references. #945, #671
--exitfirst
CLI option trims the progress bar output when a failure occurs. #951Internal error if filling missing explicit examples led to
Unsatisfiable
errors. #904Do not suggest to disable schema validation if it is already disabled. #914
Skip explicit examples generation if this phase is disabled via config. #905
Unsatisfiable
error in stateful testing caused by all API operations having inbound links. #965, #822A possibility to override
APIStateMachine.step
. #970TypeError
on nullable parameters during Open API specific serialization. #980Invalid types in
x-examples
. #982CLI crash on schemas with operation names longer than the current terminal width. #990
Handling of API operations that contain reserved characters in their paths. #992
CLI execution stops on errors during example generation. #994
Fill missing properties in incomplete explicit examples for non-body parameters. #1007
Deprecated
HookContext.endpoint
. UseHookContext.operation
instead.Case.endpoint
. UseCase.operation
instead.
Performance
Use compiled versions of Open API spec validators.
Decrease CLI memory usage. #987
Various improvements relevant to processing of API operation definitions. It gives ~20% improvement on large schemas with many references.
Removed
Case.form_data
. UseCase.body
instead.Endpoint.form_data
. UseEndpoint.body
instead.before_generate_form_data
hook. Usebefore_generate_body
instead.Deprecated stateful testing integration from our
pytest
plugin.
Note
This release features multiple backward-incompatible changes. The first one is removing form_data
and hooks related to it -
all payload related actions can be done via body
and its hooks. The second one involves renaming the so-called “endpoint” to “operation”.
The main reason for this is to generalize terminology and make it applicable to GraphQL schemas, as all Schemathesis internals
are more suited to work with semantically different API operations rather than with endpoints that are often connected with URLs and HTTP methods.
It brings the possibility to reuse the same concepts for Open API and GraphQL - in the future, unit tests will cover individual API operations
in GraphQL, rather than everything available under the same “endpoint”.
2.8.6 - 2022-03-29
Added
Support for Werkzeug>=2.1.0. #1410
2.8.5 - 2020-12-15
Added
auto
variant for the--workers
CLI option that automatically detects the number of available CPU cores to run tests on. #917
2.8.4 - 2020-11-27
Fixed
Use
--request-tls-verify
during schema loading as well. #897
2.8.3 - 2020-11-27
Added
Display failed response payload in the error output for the
pytest
plugin. #895
Changed
In pytest plugin output, Schemathesis error classes use the CheckFailed name. Before, they had not readable “internal” names.
Hypothesis falsifying examples. The code does not include
Case
attributes with default values to improve readability. #886
2.8.2 - 2020-11-25
Fixed
2.8.1 - 2020-11-24
Added
--force-schema-version
CLI option to force Schemathesis to use the specific Open API spec version when parsing the schema. #876
Changed
The
content_type_conformance
check now raises a well-formed error message when encounters a malformed media type value. #877
Fixed
Internal error during verifying explicit examples if an example has no
value
key. #882
2.8.0 - 2020-11-24
Added
--request-tls-verify
CLI option, that controls whether Schemathesis verifies the server’s TLS certificate. You can also pass the path to a CA_BUNDLE file for private certs. #830
Changed
In CLI, if an endpoint contains an invalid schema, show a message about the
--validate-schema
CLI option. #855
Fixed
Handling of 204 responses in the
response_schema_conformance
check. Before, all responses were required to have theContent-Type
header. #844Catch
OverflowError
when an invalid regex is passed to-E
/-M
/-T
/-O
CLI options. #870Internal error in CLI, when the schema location is an invalid IPv6. #872
Collecting Open API links behind references via CLI. #874
Deprecated
Using of
Case.form_data
andEndpoint.form_data
. In the3.0
release, you’ll need to use relevantbody
attributes instead. This change includes deprecation of thebefore_generate_form_data
hook, usebefore_generate_body
instead. The reason for this is the upcoming unification of parameter handling and their serialization.--stateful-recursion-limit
. It will be removed in3.0
as a part of removing the old stateful testing approach. This parameter is no-op.
2.7.7 - 2020-11-13
Fixed
Missed
headers
inEndpoint.partial_deepcopy
.
2.7.6 - 2020-11-12
Added
An option to set data generation methods. At the moment, it includes only “positive”, which means that Schemathesis will generate data that matches the schema.
Fixed
Pinned dependency on
attrs
that caused an error on fresh installations. #858
2.7.5 - 2020-11-09
Fixed
Invalid keyword in code samples that Schemathesis suggests to run to reproduce errors. #851
2.7.4 - 2020-11-07
Added
New
relative_path
property forBeforeExecution
andAfterExecution
events. It represents an operation path as it is in the schema definition.
2.7.3 - 2020-11-05
Fixed
Internal error on malformed JSON when the
response_conformance
check is used. #832
2.7.2 - 2020-11-05
Added
Shortcut for response validation when Schemathesis’s data generation is not used. #485
Changed
Improve the error message when the application can not be loaded from the value passed to the
--app
command-line option. #836Security definitions are now serialized as other parameters. At the moment, it means that the generated values will be coerced to strings, which is a no-op. However, types of security definitions might be affected by the “Negative testing” feature in the future. Therefore this change is mostly for future-compatibility. #841
Fixed
Internal error when a “header” / “cookie” parameter were not coerced to a string before filtration. #839
2.7.1 - 2020-10-22
Fixed
Adding new Open API links via the
add_link
method, when the related PathItem contains a reference. #824
2.7.0 - 2020-10-21
Added
New approach to stateful testing, based on the Hypothesis’s
RuleBasedStateMachine
. #737Case.validate_response
accepts the newadditional_checks
argument. It provides a way to execute additional checks in addition to existing ones.
Changed
The
response_schema_conformance
andcontent_type_conformance
checks fail unconditionally if the input response has noContent-Type
header. #816
Fixed
Failure reproduction code missing values that were explicitly passed to
call_*
methods during testing. #814
Deprecated
Using
stateful=Stateful.links
in schema loaders andparametrize
. Useschema.as_state_machine().TestCase
instead. The old approach to stateful testing will be removed in3.0
. See theStateful testing
section of our documentation for more information.
2.6.1 - 2020-10-19
Added
New method
as_curl_command
added to theCase
class. #689
2.6.0 - 2020-10-06
Added
Support for passing Hypothesis strategies to tests created with
schema.parametrize
by usingschema.given
decorator. #768Support for PEP561. #748
Shortcut for calling & validation. #738
New hook to pre-commit,
rstcheck
, as well as updates to documentation based on rstcheck. #734New check for maximum response time and corresponding CLI option
--max-response-time
. #716New
response_headers_conformance
check that verifies the presence of all headers defined for a response. #742New field with information about executed checks in cassettes. #702
New
port
parameter added tofrom_uri()
method. #706A code snippet to reproduce a failed check when running Python tests. #793
Python 3.9 support. #731
Ability to skip deprecated endpoints with
--skip-deprecated-endpoints
CLI option andskip_deprecated_operations=True
argument to schema loaders. #715
Fixed
User-Agent
header overriding the passed one. #757Default
User-Agent
header inCase.call
. #717Status of individual interactions in VCR cassettes. Before this change, all statuses were taken from the overall test outcome, rather than from the check results for a particular response. #695
Escaping header values in VCR cassettes. #783
Escaping HTTP response message in VCR cassettes. #788
Changed
Case.as_requests_kwargs
andCase.as_werkzeug_kwargs
now return theUser-Agent
header. This change also affects code snippets for failure reproduction - all snippets will include theUser-Agent
header.
Performance
Speed up generation of
headers
,cookies
, andformData
parameters when their schemas do not define thetype
keyword. #795
2.5.1 - 2020-09-30
This release contains only documentation updates which are necessary to upload to PyPI.
2.5.0 - 2020-09-27
Added
Stateful testing via Open API links for the
pytest
runner. #616Support for GraphQL tests for the
pytest
runner. #649
Fixed
Progress percentage in the terminal output for “lazy” schemas. #636
Changed
Check name is no longer displayed in the CLI output, since its verbose message is already displayed. This change also simplifies the internal structure of the runner events.
The
stateful
argument type in therunner.prepare
isOptional[Stateful]
instead ofOptional[str]
. Useschemathesis.Stateful
enum.
2.4.1 - 2020-09-17
Changed
Hide
Case.endpoint
from representation. Its representation decreases the usability of the pytest’s output. #719Return registered functions from
register_target
andregister_check
decorators. #721
Fixed
Possible
IndexError
when a user-defined check raises an exception without a message. #718
2.4.0 - 2020-09-15
Added
Ability to register custom targets for targeted testing. #686
Changed
The
AfterExecution
event now haspath
andmethod
fields, similar to theBeforeExecution
one. The goal is to make these events self-contained, which improves their usability.
2.3.4 - 2020-09-11
Changed
The default Hypothesis’s
deadline
setting for tests withschema.parametrize
is set to 500 ms for consistency with the CLI behavior. #705
Fixed
Encoding error when writing a cassette on Windows. #708
2.3.3 - 2020-08-04
Fixed
KeyError
during thecontent_type_conformance
check if the response has noContent-Type
header. #692
2.3.2 - 2020-08-04
Added
Run checks conditionally.
2.3.1 - 2020-07-28
Fixed
IndexError
whenexamples
list is empty.
2.3.0 - 2020-07-26
Added
Possibility to generate values for
in: formData
parameters that are non-bytes or contain non-bytes (e.g., inside an array). #665
Changed
Error message for cases when a path parameter is in the template but is not defined in the parameters list or missing
required: true
in its definition. #667Bump minimum required
hypothesis-jsonschema
version to 0.17.0. This allows Schemathesis to use thecustom_formats
argument infrom_schema
calls and avoid using its private API. #684
Fixed
ValueError
during sending a request with test payload if the endpoint defines a parameter withtype: array
andin: formData
. #661KeyError
while processing a schema with nullable parameters andin: body
. #660StopIteration
duringrequestBody
processing if it has empty “content” value. #673AttributeError
during generation of “multipart/form-data” parameters that have no “type” defined. #675Support for properties named “$ref” in object schemas. Previously, it was causing
TypeError
. #672Generating illegal Unicode surrogates in the path. #668
Invalid development dependency on
graphql-server-core
package. #658
2.2.1 - 2020-07-22
Fixed
Possible
UnicodeEncodeError
during generation ofAuthorization
header values for endpoints withbasic
security scheme. #656
2.2.0 - 2020-07-14
Added
schemathesis.graphql.from_dict
loader allows you to use GraphQL schemas represented as a dictionary for testing.before_load_schema
hook for GraphQL schemas.
Fixed
Serialization of non-string parameters. #651
2.1.0 - 2020-07-06
Added
Support for property-level examples. #467
Fixed
Content-type conformance check for cases when Open API 3.0 schemas contain “default” response definitions. #641
Handling of multipart requests for Open API 3.0 schemas. #640
Sending non-file form fields in multipart requests. #647
Removed
Deprecated
skip_validation
argument toHookDispatcher.apply
.Deprecated
_accepts_context
internal function.
2.0.0 - 2020-07-01
Changed
BREAKING. Base URL handling.
base_url
now is treated as one with a base path included. You should pass a full base URL now instead:
schemathesis run --base-url=http://127.0.0.1:8080/api/v2 ...
This value will override basePath
/ servers[0].url
defined in your schema if you use
Open API 2.0 / 3.0 respectively. Previously if you pass a base URL like the one above, it
was concatenated with the base path defined in the schema, which leads to a lack of ability
to redefine the base path. #511
Fixed
Show the correct URL in CLI progress when the base URL is overridden, including the path part. #511
Construct valid URL when overriding base URL with base path. #511
Example:
Base URL in the schema : http://0.0.0.0:8081/api/v1
`--base-url` value in CLI : http://0.0.0.0:8081/api/v2
Full URLs before this change : http://0.0.0.0:8081/api/v2/api/v1/users/ # INVALID!
Full URLs after this change : http://0.0.0.0:8081/api/v2/users/ # VALID!
Removed
Support for hooks without context argument in the first position.
Hooks registration by name and function. Use
register
decorators instead. For more details, see the “Customization” section in our documentation.BaseSchema.with_hook
andBaseSchema.register_hook
. UseBaseSchema.hooks.apply
andBaseSchema.hooks.register
instead.
1.10.0 - 2020-06-28
Added
loaders.from_asgi
supports making calls to ASGI-compliant application (For example: FastAPI). #521Support for GraphQL strategies.
Fixed
Passing custom headers to schema loader for WSGI / ASGI apps. #631
1.9.1 - 2020-06-21
Fixed
Schema validation error on schemas containing numeric values in scientific notation without a dot. #629
1.9.0 - 2020-06-20
Added
Pass the original case’s response to the
add_case
hook.Support for multiple examples with OpenAPI
examples
. #589--verbosity
CLI option to minimize the error output. #598Allow registering function-level hooks without passing their name as the first argument to
apply
. #618Support for hook usage via
LazySchema
/from_pytest_fixture
. #617
Changed
Tests with invalid schemas marked as errors, instead of failures. #622
Fixed
1.8.0 - 2020-06-15
Fixed
Tests with invalid schemas are marked as failed instead of passed when
hypothesis-jsonschema>=0.16
is installed. #614KeyError
during creating an endpoint strategy if it contains a reference. #612
Changed
Require
hypothesis-jsonschema>=0.16
. #614Pass original
InvalidSchema
text topytest.fail
call.
1.7.0 - 2020-05-30
Added
Support for YAML files in references via HTTPS & HTTP schemas. #600
Stateful testing support via
Open API links
syntax. #548New
add_case
hook. #458Support for parameter serialization formats in Open API 2 / 3. For example
pipeDelimited
ordeepObject
. #599Support serializing parameters with
application/json
content-type. #594
Changed
The minimum required versions for
Hypothesis
andhypothesis-jsonschema
are5.15.0
and0.11.1
respectively. The main reason is this fix that is required for stability of Open API links feature when it is executed in multiple threads.
1.6.3 - 2020-05-26
Fixed
Support for a colon symbol (
:
) inside of a header value passed via CLI. #596
1.6.2 - 2020-05-15
Fixed
Partially generated explicit examples are always valid and can be used in requests. #582
1.6.1 - 2020-05-13
Changed
Look at the current working directory when loading hooks for CLI. #586
1.6.0 - 2020-05-10
Added
Fixed
Passing
workers_num
toThreadPoolRunner
leads to always using 2 workers in this worker kind. #579
1.5.1 - 2020-05-08
Fixed
Display proper headers in reproduction code when headers are overridden. #566
1.5.0 - 2020-05-06
Added
Display a suggestion to disable schema validation on schema loading errors in CLI. #531
Filtration of endpoints by
operationId
viaoperation_id
parameter toschema.parametrize
or-O
command-line option. #546Generation of security-related parameters. They are taken from
securityDefinitions
/securitySchemes
and injected to the generated data. It supports generating API keys in headers or query parameters and generating data for HTTP authentication schemes. #540
Fixed
1.4.0 - 2020-05-03
Added
context
argument for hook functions to provide an additional context for hooks. A deprecation warning is emitted for hook functions that do not accept this argument.A new hook system that allows generic hook dispatching. It comes with new hook locations. For more details, see the “Customization” section in our documentation.
New
before_process_path
hook.Third-party compatibility fixups mechanism. Currently, there is one fixup for FastAPI. #503
Deprecated
Hook functions that do not accept
context
as their first argument. They will become not be supported in Schemathesis 2.0.Registering hooks by name and function. Use
register
decorators instead. For more details, see the “Customization” section in our documentation.BaseSchema.with_hook
andBaseSchema.register_hook
. UseBaseSchema.hooks.apply
andBaseSchema.hooks.register
instead.
Fixed
Add missing
validate_schema
argument toloaders.from_pytest_fixture
.Reference resolving during response schema conformance check. #539
1.3.4 - 2020-04-30
Fixed
Validation of nullable properties in
response_schema_conformance
check introduced in1.3.0
. #542
1.3.3 - 2020-04-29
Changed
Update
pytest-subtests
pin to>=0.2.1,<1.0
. #537
1.3.2 - 2020-04-27
Added
Show exceptions if they happened during loading a WSGI application. Option
--show-errors-tracebacks
will display a full traceback.
1.3.1 - 2020-04-27
Fixed
Packaging issue
1.3.0 - 2020-04-27
Added
Storing network logs with
--store-network-log=<filename.yaml>
. The stored cassettes are based on the VCR format and contain extra information from the Schemathesis internals. #379Replaying of cassettes stored in VCR format. #519
Targeted property-based testing in CLI and runner. It only supports the
response_time
target at the moment. #104Export CLI test results to JUnit.xml with
--junit-xml=<filename.xml>
. #427
Fixed
Code samples for schemas where
body
is defined as{"type": "string"}
. #521Showing error causes on internal
jsonschema
errors during input schema validation. #513Recursion error in
response_schema_conformance
check. Because of this change,Endpoint.definition
contains a definition where references are not resolved. In this way, it makes it possible to avoid recursion errors injsonschema
validation. #468
Changed
Added indentation & section name to the
SUMMARY
CLI block.Use C-extension for YAML loading when it is possible. It can cause more than 10x speedup on schema parsing. Do not show Click’s “Aborted!” message when an error occurs during CLI schema loading.
Add a help message to the CLI output when an internal exception happens. #529
1.2.0 - 2020-04-15
Added
1.1.2 - 2020-04-14
Fixed
1.1.1 - 2020-04-12
Fixed
1.1.0 - 2020-04-08
Fixed
Response schema check for recursive schemas. #468
Changed
App loading in
runner
. Now it accepts application as an importable string, rather than an instance. It is done to make it possible to execute a runner in a subprocess. Otherwise, apps can’t be easily serialized and transferred into another process.Runner events structure. All data in events is static from now. There are no references to
BaseSchema
,Endpoint
or similar objects that may calculate data dynamically. This is done to make events serializable and not tied to Python object, which decouples anyrunner
consumer from implementation details. It will help makerunner
usable in more cases (e.g., web application) since events can be serialized to JSON and used in any environment. Another related change is that Python exceptions are not propagated anymore - they are replaced with theInternalError
event that should be handled accordingly.
1.0.5 - 2020-04-03
Fixed
Open API 3. Handling of endpoints that contain
multipart/form-data
media types. Previously only file upload endpoints were working correctly. #473
1.0.4 - 2020-04-03
Fixed
1.0.3 - 2020-04-03
Fixed
1.0.2 - 2020-04-02
Fixed
1.0.1 - 2020-04-01
Fixed
Processing of explicit examples in Open API 3.0 when there are multiple parameters in the same location (e.g.
path
) containexample
value. They are properly combined now. #450
1.0.0 - 2020-03-31
Changed
Move processing of
runner
parameters torunner.prepare
. This change will provide better code reuse since all users ofrunner
(e.g., if you extended it in your project) need some kind of input parameters handling, which was implemented only in Schemathesis CLI. It is not backward-compatible. If you didn’t userunner
directly, then this change should not have a visible effect on your use-case.
0.28.0 - 2020-03-31
Fixed
Handling of schemas that use
x-*
custom properties. #448
Removed
Deprecated
runner.execute
. Userunner.prepare
instead.
0.27.0 - 2020-03-31
Deprecated
runner.execute
should not be used, sincerunner.prepare
provides a more flexible interface to test execution.
Removed
Deprecated
Parametrizer
class. Useschemathesis.from_path
as a replacement forParametrizer.from_path
.
0.26.1 - 2020-03-24
Fixed
Limit recursion depth while resolving JSON schema to handle recursion without breaking. #435
0.26.0 - 2020-03-19
Fixed
0.25.1 - 2020-03-09
Changed
Allow
werkzeug
>= 1.0.0. #433
0.25.0 - 2020-02-27
Changed
Handling of explicit examples from schemas. Now, if there are examples for multiple locations (e.g., for body and query) then they will be combined into a single example. #424
0.24.5 - 2020-02-26
Fixed
Error during
pytest
collection on objects with custom__getattr__
method and therefore passis_schemathesis
check. #429
0.24.4 - 2020-02-22
Fixed
Resolving references when the schema is loaded from a file on Windows. #418
0.24.3 - 2020-02-10
Fixed
0.24.2 - 2020-02-09
Fixed
Crash on invalid regular expressions in
method
,endpoint
andtag
CLI options. #403Crash on a non-latin-1 encodable value in the
auth
CLI option. #404Crash on an invalid value in the
header
CLI option. #405Crash on some invalid URLs in the
schema
CLI option. #406Validation of
--request-timeout
parameter. #407Crash with
--hypothesis-deadline=0
CLI option. #410Crash with
--hypothesis-max-examples=0
CLI option. #412
0.24.1 - 2020-02-08
Fixed
CLI crash on Windows and Python < 3.8 when the schema path contains characters unrepresentable at the OS level. #400
0.24.0 - 2020-02-07
Added
Support for testing of examples in Parameter & Media Type objects in Open API 3.0. #394
--show-error-tracebacks
CLI option to display errors’ tracebacks in the output. #391Support for schema behind auth. #115
Changed
Schemas with GET endpoints accepting body are allowed now if schema validation is disabled (via
--validate-schema=false
for example). The use-case is for tools like ElasticSearch that use GET requests with non-empty bodies. #383
Fixed
CLI crash when an explicit example is specified in the endpoint definition. #386
0.23.7 - 2020-01-30
Added
-x
/--exitfirst
CLI option to exit after the first failed test. #378
Fixed
Handling examples of parameters in Open API 3.0. #381
0.23.6 - 2020-01-28
Added
all
variant for--checks
CLI option to use all available checks. #374
Changed
Use built-in
importlib.metadata
on Python 3.8. #376
0.23.5 - 2020-01-24
Fixed
Generation of invalid values in
Case.cookies
. #371
0.23.4 - 2020-01-22
Fixed
Converting
exclusiveMinimum
&exclusiveMaximum
fields to JSON Schema. #367
0.23.3 - 2020-01-21
Fixed
Filter out surrogate pairs from the query string.
0.23.2 - 2020-01-16
Fixed
Prevent
KeyError
when the response does not have the “Content-Type” header. #365
0.23.1 - 2020-01-15
Fixed
Dockerfile entrypoint was not working as per docs. #361
0.23.0 - 2020-01-15
Added
0.22.0 - 2020-01-11
Added
Raise a proper exception when the given schema is invalid. #308
Support for
None
as a value for--hypothesis-deadline
. #349
Fixed
Handling binary request payloads in
Case.call
. #350Type of the second argument to all built-in checks set to proper
Case
instead ofTestResult
. The error was didn’t affect built-in checks since bothCase
andTestResult
hadendpoint
attribute, and only it was used. However, this fix is not backward-compatible with 3rd party checks.
0.21.0 - 2019-12-20
Added
Support for AioHTTP applications in CLI. #329
0.20.5 - 2019-12-18
Fixed
Compatibility with the latest release of
hypothesis-jsonschema
and setting its minimal required version to0.9.13
. #338
0.20.4 - 2019-12-17
Fixed
Handling
nullable
attribute in Open API schemas. #335
0.20.3 - 2019-12-17
Fixed
Usage of the response status code conformance check with old
requests
version. #330
0.20.2 - 2019-12-14
Fixed
Response schema conformance check for Open API 3.0. #332
0.20.1 - 2019-12-13
Added
Support for response code ranges. #330
0.20.0 - 2019-12-12
Added
WSGI apps support. #31
Case.validate_response
for running built-in checks against app’s response. #319
Changed
Checks receive
Case
instance as a second argument instead ofTestResult
. This was done for making checks usable in Python tests viaCase.validate_response
. Endpoint and schema are accessible viacase.endpoint
andcase.endpoint.schema
.
0.19.1 - 2019-12-11
Fixed
Compatibility with Hypothesis >= 4.53.2. #322
0.19.0 - 2019-12-02
Added
0.18.1 - 2019-11-28
Fixed
Validation of the
base-url
CLI parameter. #311
0.18.0 - 2019-11-27
Added
Resolving references in
PathItem
objects. #301
Fixed
0.17.0 - 2019-11-21
Added
Resolving references that point to different files. #294
Changed
Keyboard interrupt is now handled during the CLI run, and the summary is displayed in the output. #295
0.16.0 - 2019-11-19
Added
Display RNG seed in the CLI output to allow test reproducing. #267
Allow specifying seed in CLI.
Ability to pass custom kwargs to the
requests.get
call inloaders.from_uri
.
Changed
Refactor case generation strategies: strategy is not used to generate empty value. #253
Improved error message for invalid path parameter declaration. #255
Fixed
0.15.0 - 2019-11-15
Added
Support for OpenAPI 3.0 server variables (base_path). #40
Support for
format: byte
. #254Response schema conformance check in CLI / Runner. #256
Docker image for CLI. #268
Pre-run hooks for CLI. #147
A way to register custom checks for CLI via
schemathesis.register_check
. #270
Fixed
Not encoded path parameters. #272
Changed
Verbose messages are displayed in the CLI on failed checks. #261
0.14.0 - 2019-11-09
Added
CLI: Support file paths in the
schema
argument. #119Checks to verify response status & content type in CLI / Runner. #101
Fixed
Custom base URL handling in CLI / Runner. #248
Changed
0.13.2 - 2019-11-05
Fixed
IndexError
when Hypothesis found inconsistent test results during the test execution in the runner. #236
0.13.1 - 2019-11-05
Added
Support for binary format #197
Fixed
Error that happens when there are no success checks in the statistic in CLI. #237
0.13.0 - 2019-11-03
Added
An option to configure request timeout for CLI / Runner. #204
A help snippet to reproduce errors caught by Schemathesis. #206
Total running time to the CLI output. #181
Summary line in the CLI output with the number of passed / failed / errored endpoint tests. #209
Extra information to the CLI output: schema address, spec version, and base URL. #188
Fixed
Compatibility with Hypothesis 4.42.4+ . #212
Display flaky errors only in the “ERRORS” section and improve CLI output. #215
Handling
formData
parameters inCase.call
. #196Handling cookies in
Case.call
. #211
Changed
More readable falsifying examples output. #127
Show exceptions in a separate section of the CLI output. #203
Error message for cases when it is not possible to satisfy schema parameters. It should be more clear now. #216
Do not stop on schema errors related to a single endpoint. #139
Display a proper error message when the schema is not available in CLI / Runner. #214
0.12.2 - 2019-10-30
Fixed
0.12.1 - 2019-10-28
Fixed
Handling for errors other than
AssertionError
andHypothesisException
in the runner. #189CLI failing on the case when there are tests, but no checks were performed. #191
Changed
Display the “SUMMARY” section in the CLI output for empty test suites.
0.12.0 - 2019-10-28
Added
Display progress during the CLI run. #125
Fixed
Test server-generated wrong schema when the
endpoints
option is passed via CLI. #173Error message if the schema is not found in CLI. #172
Changed
Continue running tests on hypothesis error. #137
0.11.0 - 2019-10-22
Added
LazySchema accepts filters. #149
Ability to register strategies for custom string formats. #94
Generator-based events in the
runner
module to improve control over the execution flow.Filtration by tags. #134
Changed
Base URL in schema instances could be reused when it is defined during creation. Now on, the
base_url
argument inCase.call
is optional in such cases. #153Hypothesis deadline is set to 500ms by default. #138
Hypothesis output is captured separately, without capturing the whole stdout during CLI run.
Disallow empty username in CLI
--auth
option.
Fixed
Removed
Undocumented support for
file://
URI schema
0.10.0 - 2019-10-14
Added
HTTP Digest Auth support. #106
Support for Hypothesis settings in CLI & Runner. #107
Case.call
andCase.as_requests_kwargs
convenience methods. #109Local development server. #126
Removed
Autogenerated
runner.StatsCollector.__repr__
to make Hypothesis output more readable.
0.9.0 - 2019-10-09
Added
Test executor collects results of execution. #29
CLI option
--base-url
for specifying base URL of API. #118Support for coroutine-based tests. #121
User Agent to network requests in CLI & runner. #130
Changed
CLI command
schemathesis run
prints result in a more readable way with a summary of passing checks.Empty header names are forbidden for CLI.
Suppressed hypothesis exception about using
example
non-interactively. #92
0.8.1 - 2019-10-04
Fixed
Wrap each test in
suppress
so the runner doesn’t stop after the first test failure.
0.8.0 - 2019-10-04
Added
CLI tool invoked by the
schemathesis
command. #30New arguments
api_options
,loader_options
andloader
for test executor. #90A mapping interface for schemas & convenience methods for direct strategy access. #98
Fixed
Runner stopping on the first falsifying example. #99
0.7.3 - 2019-09-30
Fixed
Filtration in lazy loaders.
0.7.2 - 2019-09-30
Added
Fixed
Conflict for lazy schema filtering. #64
0.7.1 - 2019-09-27
Added
Support for
x-nullable
extension. #45
0.7.0 - 2019-09-26
Added
Support for the
cookie
parameter in OpenAPI 3.0 schemas. #21Support for the
formData
parameter in Swagger 2.0 schemas. #6Test executor. #28
Fixed
Using
hypothesis.settings
decorator with test functions created fromfrom_pytest_fixture
loader. #69
0.6.0 - 2019-09-24
Added
Parametrizing tests from a pytest fixture via
pytest-subtests
. #58
Changed
Rename module
readers
toloaders
.Rename
parametrize
parameters.filter_endpoint
toendpoint
andfilter_method
tomethod
.
Removed
Substring match for method/endpoint filters. To avoid clashing with escaped chars in endpoints keys in schemas.
0.5.0 - 2019-09-16
Added
Generating explicit examples from the schema. #17
Changed
Schemas are loaded eagerly from now on. Using
schemathesis.from_uri
implies network calls.
Deprecated
Using
Parametrizer.from_{path,uri}
is deprecated, useschemathesis.from_{path,uri}
instead.
Fixed
Body resolving during test collection. #55
0.4.1 - 2019-09-11
Fixed
Possibly unhandled exception during
hasattr
check inis_schemathesis_test
.
0.4.0 - 2019-09-10
Fixed
Resolving all inner references in objects. #34
Changed
jsonschema.RefResolver
is now used for reference resolving. #35
0.3.0 - 2019-09-06
Added
Parametrizer.from_uri
method to construct parametrizer instances from URIs. #24
Removed
Possibility to use
Parametrizer.parametrize
and customParametrizer
kwargs for passing config options tohypothesis.settings
. Usehypothesis.settings
decorators on tests instead.
0.2.0 - 2019-09-05
Added
Changed
Handle errors during collection / executions as failures.
Use
re.search
for pattern matching infilter_method
/filter_endpoint
instead offnmatch
. #18Case.body
contains properties from the target schema, without the extra level of nesting.
Fixed
KeyError
on collection when “basePath” is absent. #16
0.1.0 - 2019-06-28
Initial public release