Experimental features

Introduction

This section provides an overview of experimental features in Schemathesis - features that are under development and available for testing, but not yet considered stable.

This section provides an overview of experimental features in Schemathesis. These are features that are under development, available for user testing but not yet stable. Experimental features offer a glimpse into upcoming functionalities and enhancements, providing users an opportunity to try out and contribute feedback to shape their final form.

Note

Experimental features can change or be removed in any minor version release.

Enabling Experimental Features

Schemathesis provides a few ways to enable experimental features: via the CLI, Python tests, and environment variables.

In CLI

To enable an experimental feature via the CLI, use the --experimental option.

For example, to enable experimental support for OpenAPI 3.1:

st run https://example.schemathesis.io/openapi.json --experimental=openapi-3.1

In Python Tests

To enable experimental features for your Schemathesis tests in Python, use the enable method on the desired experimental feature. It’s a good idea to put the code to enable the feature in a place that runs before your tests start. For those using pytest, putting this code in a conftest.py file at the root of your test directory would work well.

import schemathesis

# Globally enable OpenAPI 3.1 experimental feature
schemathesis.experimental.OPEN_API_3_1.enable()

Executing the above code will ensure that Schemathesis will utilize the enabled experimental features in all tests run afterwards.

Note

This action will apply globally, affecting all your tests. Use with caution.

Using Environment Variables

You can also enable experimental features through environment variables. This is particularly useful for CI/CD pipelines or when you don’t have direct control over CLI arguments or test code.

For example, to enable experimental support for OpenAPI 3.1:

export SCHEMATHESIS_EXPERIMENTAL_OPENAPI_3_1=true

This will enable the OpenAPI 3.1 experimental feature for any Schemathesis runs in the same environment.

Current Experimental Features

Open API 3.1

Provides partial support for OpenAPI 3.1. This includes compatible JSON Schema validation for API responses. Note that data generation is still compatible only with OpenAPI 3.0.

Enabling this feature also automatically enables UUID format support.

In CLI

st run https://example.schemathesis.io/openapi.json --experimental=openapi-3.1

In Python Tests

import schemathesis

# Globally enable OpenAPI 3.1 experimental feature
schemathesis.experimental.OPEN_API_3_1.enable()

Using Environment Variables

To enable OpenAPI 3.1 support via environment variables, you can set:

export SCHEMATHESIS_EXPERIMENTAL_OPENAPI_3_1=true

For more details, join the GitHub Discussion #1822.

Coverage phase

Schemathesis primarily generates randomized tests; therefore, the outcomes can vary between test runs.

During the coverage phase, Schemathesis deterministically generates test cases to cover common “positive” and “negative” testing scenarios depending on the configuration. It covers common testing patterns such as:

  • Boundary values: Minimum, maximum, and near-boundary values for numeric types and string lengths.

  • Data types: Generating values for different data types specified in the schema.

  • Constraints: Testing against schema constraints like required fields, pattern, format, and uniqueItems.

Additionally, it uses examples in the API schema as the source of “positive” values.

In CLI

st run https://example.schemathesis.io/openapi.json --experimental=coverage-phase

Using Environment Variables

export SCHEMATHESIS_EXPERIMENTAL_COVERAGE_PHASE=true

For more details, join the GitHub Discussion #2418.

Positive Data Acceptance

Verifies that schema-conforming data receives 2xx status responses, highlighting mismatches between schema and API behavior across all endpoints.

Allowed status codes: 2xx, 401, 403, 404

Note: May produce false positives with complex validation that is not reflected in the schema.

In CLI

st run https://example.schemathesis.io/openapi.json --experimental=positive_data_acceptance

Configuration options:

--experimental-positive-data-acceptance-allowed-statuses=202

Using Environment Variables

To enable Positive Data Acceptance via environment variables:

export SCHEMATHESIS_EXPERIMENTAL_POSITIVE_DATA_ACCEPTANCE=true

Configure status codes:

export SCHEMATHESIS_EXPERIMENTAL_POSITIVE_DATA_ACCEPTANCE_ALLOWED_STATUSES=201,202,204

For more details, join the GitHub Discussion #2499.

Negative Data Rejection

This feature covers configuring the negative_data_rejection check and allows for defining what status codes won’t trigger this check.

Allowed status codes: 400, 401, 403, 404, 422, 5XX

In CLI

Configuration options:

--experimental-negative-data-rejection-allowed-statuses=4XX

Using Environment Variables

Configure status codes:

export SCHEMATHESIS_EXPERIMENTAL_NEGATIVE_DATA_REJECTION_ALLOWED_STATUSES=4XX

New Stateful Test Runner

This experimental feature enables a more effective way to run stateful tests in the CLI, along with a new display style. It leverages state machines, which were previously available in the pytest integration, to generate more diverse test scenarios. By default, state machines are executed until no new failures are found in a single iteration.

Important

The new test runner does not yet support all configuration options available in the old one.

In CLI

st run https://example.schemathesis.io/openapi.json --experimental=stateful-test-runner

Using Environment Variables

export SCHEMATHESIS_EXPERIMENTAL_STATEFUL_TEST_RUNNER=true

For more details, join the GitHub Discussion #2262.

Schema Analysis

Enables integration with Schemathesis.io to optimize test case generation for better bug detection. This feature infers appropriate data generation strategies, supports uncommon media types, adjusts schemas for faster data generation, and detects the web server to generate more targeted test data.

In CLI

st run https://example.schemathesis.io/openapi.json --experimental=schema-analysis

Using Environment Variables

To enable schema analysis support via environment variables, you can set:

export SCHEMATHESIS_EXPERIMENTAL_SCHEMA_ANALYSIS=true

For more details, join the GitHub Discussion #2056.

Stabilization of Experimental Features

Criteria for moving a feature from experimental to stable status include:

  • Full coverage of planned functionality

  • API design stability, assessed through user feedback and internal review

Providing Feedback

Feedback is crucial for the development and stabilization of experimental features. We encourage you to share your thoughts via GitHub Discussions

Note

When you use an experimental feature, a notice will appear in your test output, providing a link to the corresponding GitHub discussion where you can leave feedback.