Targeted Testing
Steer data generation toward inputs that maximize a metric, such as response time, so Schemathesis reaches slow or failing inputs in fewer test cases.
Prerequisites
- Schemathesis installed or available via
uvx - An API with an OpenAPI schema
Which phases use targeting
Targeting guides Hypothesis-based generation, so it applies in the fuzzing and stateful phases. The examples and coverage phases use fixed inputs; metrics are computed there but do not change what gets sent. See Testing Phases for what each phase does.
Maximize response time from the CLI
Pass the metric name to --generation-maximize:
$ uvx schemathesis run http://127.0.0.1:8000/openapi.json --generation-maximize response_time
The built-in metric is response_time. In schemathesis.toml:
[generation]
maximize = "response_time"
Consider an endpoint that slows down by 10 ms for every 0 in the request body and fails with more than ten zeros:
import asyncio
import json
from starlette.applications import Starlette
from starlette.requests import Request
from starlette.responses import JSONResponse
from starlette.routing import Route
SCHEMA = {
"openapi": "3.0.0",
"info": {"title": "Demo", "version": "1.0"},
"paths": {
"/performance": {
"post": {
"requestBody": {
"required": True,
"content": {"application/json": {"schema": {"type": "array", "items": {"type": "integer"}}}},
},
"responses": {
"200": {"description": "OK"},
"422": {"description": "Invalid input"},
},
}
}
},
}
async def performance(request: Request) -> JSONResponse:
try:
numbers = json.loads(await request.body())
except ValueError:
numbers = None
if not isinstance(numbers, list) or not all(type(item) is int for item in numbers):
return JSONResponse({"detail": "Expected an array of integers"}, status_code=422)
zeros = str(numbers).count("0")
# Each zero adds 10 ms to the response time
await asyncio.sleep(0.01 * zeros)
if zeros > 10:
return JSONResponse({"detail": "Internal error"}, status_code=500)
return JSONResponse({"result": "success"})
async def openapi(request: Request) -> JSONResponse:
return JSONResponse(SCHEMA)
app = Starlette(
routes=[
Route("/performance", performance, methods=["POST"]),
Route("/openapi.json", openapi),
]
)
Save it as app.py, start it with uvicorn app:app --port 8000, and run the command above. Inputs that take longer are favored, so the fuzzing phase moves toward bodies with more zeros:
Test Phases:
⏭ Examples
✅ Coverage
❌ Fuzzing
⏭ Stateful (not applicable)
Failures:
❌ ST101 Server error: 1
❌ ST103 Undocumented status code: 1
Results vary between runs because generation is random. Compare runs with and without --generation-maximize on your own API, using the same --max-examples, to see whether targeting helps there.
Register a custom metric
A metric is a function that takes a MetricContext (with the case and its response) and returns a float:
# metrics.py
import schemathesis
@schemathesis.metric
def response_size(ctx: schemathesis.MetricContext) -> float:
return float(len(ctx.response.content))
Load the module through SCHEMATHESIS_HOOKS and pass the function name to --generation-maximize:
$ export SCHEMATHESIS_HOOKS=metrics
$ uvx schemathesis run http://127.0.0.1:8000/openapi.json --generation-maximize response_size
Target a metric in pytest
The pytest integration does not apply --generation-maximize or metrics registered with @schemathesis.metric. Compute the value from the response yourself and pass it to hypothesis.target:
# test_api.py
import hypothesis
import schemathesis
from app import app
schema = schemathesis.openapi.from_asgi("/openapi.json", app)
@schema.parametrize()
def test_api(case):
response = case.call()
hypothesis.target(response.elapsed, label="response_time")
case.validate_response(response)
response.elapsed is the response time in seconds. Run pytest test_api.py; the test fails with the server error once a body with more than ten zeros is generated:
| schemathesis.core.failures.FailureGroup: Schemathesis found 2 distinct failures
| - ST101 Server error
...
FAILED test_api.py::test_api[POST /performance] - ...
hypothesis.target only affects generated test cases; in explicit examples (the examples and coverage phases under pytest) it has no effect.
Troubleshooting
--generation-maximize rejects the metric name. The name must match a registered metric. For custom metrics, check that SCHEMATHESIS_HOOKS points to the module that defines it and that the module imports without errors.
No difference compared to an untargeted run. Targeting needs several generated test cases per operation to have an effect. Raise --max-examples, and check that the operation reaches the fuzzing phase.