Changelog¶

All notable changes to toolproof are listed here. The format follows Keep a Changelog, and the project uses Semantic Versioning.

Unreleased ¶

The GitHub Action is named "MCP toolproof", so it can be listed on the GitHub Marketplace. Workflows still use uses: NilkanthSuthar/toolproof@....

refresh, override_provider and override_model were accepted as keys in the evals: section of toolproof.yaml. They are command-line options only (--no-cache, --provider, --model) and are now rejected like any unknown key.

toolproof eval: tool-selection evals. Each case gives a model the server's tool list and a prompt, and checks the tool and arguments it picks. Providers: anthropic, openai (and compatible APIs through OPENAI_BASE_URL) and command for your own program. Accuracy is gated by min_accuracy, API answers are cached in .toolproof-cache/, and --no-cache refreshes them. The provider SDKs are optional extras: mcp-toolproof[anthropic], mcp-toolproof[openai].
evals: section in toolproof.yaml; toolproof run includes it when present (--skip-evals).
A GitHub Action: uses: NilkanthSuthar/toolproof@v0.3.0 installs toolproof, runs it, uploads the reports and writes a job summary.
--html PATH writes a single-file HTML report that works offline, with a latency chart. Everything from the server under test is escaped.
--summary PATH appends a Markdown summary, for example to $GITHUB_STEP_SUMMARY.
headers: for HTTP servers, sent with every request (for example Authorization).
${NAME} in the server: section of toolproof.yaml is replaced with the environment variable NAME, so secrets can come from CI instead of the file. An unset variable is a clear error.
run_config, run_config_async, RunReport and ConfigError are exported from toolproof as public API.
The package ships a py.typed marker, so type checkers use toolproof's type hints.
Documentation site with guides, recipes and a full reference.

import toolproof and the pytest plugin no longer load Hypothesis up front. It's loaded the first time fuzzing runs.

toolproof fuzz: generates valid and invalid inputs from each tool's input schema with Hypothesis and reports crashes, hangs, internal errors, leaked tracebacks, outputSchema violations and invalid input that was accepted, each shrunk to the smallest failing input.
Fuzzing restarts a crashed server and keeps going, skips tools marked destructive by default, and prints its seed so a run can be repeated.
toolproof bench: p50/p95/p99 latency, throughput and error rate at a chosen concurrency, with thresholds that fail the run.
bench: and fuzz: sections in toolproof.yaml; toolproof run includes them when present (--skip-bench, --skip-fuzz).
Fuzz and bench results in the console, JUnit and JSON reports.
fuzz, bench and inspect fall back to ./toolproof.yaml when no server is given.

First release.

toolproof inspect: list a server's tools, resources and prompts (table or --json).
Static checks for tool definitions: names, descriptions, input/output JSON Schemas, required fields.
YAML test cases with is_error, equals, contains, regex, jsonpath, max_latency_ms and automatic outputSchema validation.
toolproof run with per-test timeouts, retries, crash detection and automatic server restart.
Console, JUnit XML and JSON reports; exit code 0/1 for CI.
pytest plugin with an mcp_server fixture.
stdio and streamable HTTP transports.
Example weather server and a buggy server with planted bugs.