Adding Schema Conformance Validation to Existing Tests
Check requests and responses in your existing tests against your OpenAPI schema without changing how those tests generate data or call the API.
Prerequisites
- Schemathesis, pytest and
requestsinstalled - A running API that serves its OpenAPI schema
The examples use an API at http://127.0.0.1:8000 with POST /users, GET /users (with a limit query parameter) and GET /users/{user_id}.
1. Load the schema once
# conftest.py
import pytest
import schemathesis
@pytest.fixture(scope="session")
def base_url():
return "http://127.0.0.1:8000"
@pytest.fixture(scope="session")
def api_schema(base_url):
return schemathesis.openapi.from_url(f"{base_url}/openapi.json")
2. Validate responses
validate_response() raises when the response body does not match the schema documented for its status code:
# test_users.py
import requests
def test_get_user(api_schema, base_url):
response = requests.get(f"{base_url}/users/1")
api_schema["/users/{user_id}"]["GET"].validate_response(response)
is_valid_response() returns a boolean instead:
def test_create_user(api_schema, base_url):
response = requests.post(f"{base_url}/users", json={"name": "Alice"})
assert api_schema["/users"]["POST"].is_valid_response(response)
A single violation raises a Failure subclass, which is an AssertionError, so pytest reports it as a normal test failure. Several violations in one response raise a FailureGroup that contains all of them. For a response whose id is a string instead of an integer:
E schemathesis.openapi.checks.JsonSchemaError: Response violates schema
E
E "999" is not of type "integer"
E
E Validated against the response schema for status code 200.
These methods validate the response body against the schema for the returned status code. A status code the schema does not document is not an error here; use case.validate_response() in Schemathesis-generated tests for status code and header checks.
3. Validate request data
Catch malformed test data before it reaches the API. Every parameter of an operation - request body, query, path, header, cookie - carries validate() and is_valid():
def test_request_data(api_schema):
operation = api_schema["/users"]["POST"]
body = next(operation.get_bodies_for_media_type("application/json"))
# Raises `jsonschema_rs.ValidationError`
body.validate({"name": "Alice"})
assert body.is_valid({"name": "Alice"})
limit = api_schema["/users"]["GET"].get_parameter("limit", "query")
limit.validate(50)
Pass values in their parsed form, not serialized. get_parameter() returns None when the operation declares no such parameter, and get_bodies_for_media_type() yields nothing when no body matches the media type.
Complete example
import requests
def test_user_workflow(api_schema, base_url):
create_response = requests.post(f"{base_url}/users", json={"name": "Test"})
api_schema["/users"]["POST"].validate_response(create_response)
user_id = create_response.json()["id"]
get_response = requests.get(f"{base_url}/users/{user_id}")
api_schema["/users/{user_id}"]["GET"].validate_response(get_response)
Run pytest; tests pass when every response matches the schema:
test_users.py .... [100%]
============================== 4 passed in 0.09s ===============================
Troubleshooting
OperationNotFound. The path in api_schema[...] must match the schema exactly, including path parameter names such as {user_id}, not the concrete URL.
AttributeError: 'NoneType' object has no attribute 'validate'. get_parameter() returned None: check the parameter name and location ("query", "path", "header", "cookie").
StopIteration from next(...). The operation has no request body for that media type.