Testing through MCP: running the suite against a real homelab

For a project that touches real infrastructure, the tests that earn their keep are the ones that actually provision, install and tear things down. DevStation’s engine exposes an MCP server, so the end-to-end suite drives the CLI through it — the same boundary an agent, or any MCP client, would use. These are ordinary integration tests; what MCP adds is that an agent can run and explore the whole flow on its own, the way a QA would work through the CLI by hand.

MCP as the test surface

The Model Context Protocol is an open standard for connecting agents to tools and data. Concretely, the suite goes through the same calls an agent would: list clusters, provision a node, install a service over SSH, uninstall, destroy. By design, MCP is an inbound adapter that translates the tools into the existing JSON-RPC calls — it knows nothing about the internal structure of the contexts — with an explicit allowlist and risk metadata for the destructive operations.

Real infrastructure instead of mocks

The suite talks to a real machine instead of a simulation: it provisions a node through OpenTofu, brings the VM up, connects over SSH, installs, uninstalls and destroys. It is slower than a unit test, and in exchange it exercises real behavior from end to end rather than an approximation.

Confidence in big refactors

The payoff is clearest in refactors. When renaming deploy → install and destroy → uninstall across the whole stack, the e2e against the real node flagged in seconds a regression no mock would catch: reading the previous state had broken an endpoint. Finding that on the spot, rather than in production, is what makes broad changes practical. And because the agent drives the suite, the “provision → install → uninstall → destroy” cycle runs in a loop without constant supervision, so re-checking a big change stays cheap.

The cross-OS Windows journey

The Windows validation illustrates the method well. Two machines — one Linux, one Windows — shared a directory, with Claude Code running on both. The Linux agent compiled the binary and published the artifacts; the Windows agent validated, guided by instructions in .md files. The two ran for hours, nearly autonomous, until the whole CLI passed on Windows — and the MCP tests were run from both sides whenever needed. The operating-system boundary stopped being a manual event and became part of the loop.

Test discipline and the role of the harness

None of this means AI made the complexity disappear. What kept things honest was the harness: every feature ships with its test, every fix with its regression, and the agent runs the real suite because that is the workflow. Testing against the real thing, in a loop, is mostly what changed how much felt safe to attempt. (There’s also useful research on where AI help tapers off — on mature codebases a team already knows well, and on very large ones — linked below.)

References

• Model Context Protocol — Anthropic
• Experienced developers’ productivity with AI on mature projects — METR (2025)
• LongCodeBench: coding models at 1M-token context windows — arXiv (2025)