Microsoft code-testing-generator: How Its Open-Source AI Agent Writes and Validates Unit Tests

How the research-plan-implement (RPI) pipeline works

The agent follows a structured pipeline instead of generating tests immediately :

1. Learn from the repository – It scans the codebase to detect the language, test framework, existing test patterns, and the correct build and test commands .
2. Choose an execution path – Based on task scope, it selects one of three modes :
   • Direct – Read code, write tests, validate result.
   • Single pass – Research and plan once, then implement.
   • Iterative – Repeat the research-plan-implement cycle for large requests or coverage goals.
3. Plan and write tests – The agent starts with simple code, moves to code with more dependencies, maps each behavior to a test file, follows local conventions, and fixes compilation or assertion errors as it goes .
4. Validate the tests – Before finishing, the agent runs a multi-step quality check (see below) .

The RPI process consists of three steps: Research, Plan, and Implement .

Validation process: lightweight mutation testing and quality assurance

The agent performs a lightweight form of mutation testing and quality assurance on every generated test :

• Considers small code changes that should make the tests fail.
• Checks for weak or missing assertions.
• Verifies every requested scenario has a matching test.
• Builds the full workspace and runs the entire test suite.
• Confirms the repository's normal test command can find the new tests.

This ensures a test that "passes" is actually meaningful and discoverable in CI . The agent writes unit tests that isolate the code under test and mock external services and other outside dependencies .

Supported languages and IDE integrations

The agent ships as the dotnet-test plugin within the MIT-licensed dotnet/skills repository on GitHub . It works with :

• Languages – .NET, Python, Go, TypeScript, Java, and Rust.
• IDEs/tools – GitHub Copilot CLI, Visual Studio Code, and VS Code Insiders (via plugin support as a preview feature).

Because it is shipped as an agent definition rather than a hosted service, it runs inside whatever coding agent a team already uses, with code staying local rather than round-tripping through an external service .

Current limitations

• Test types – Only unit tests are supported. Integration tests, end-to-end tests, browser tests, and performance tests are explicitly out of scope .
• No production code changes – The agent never modifies production code during test generation .
• Avoids flaky tests – It avoids unit tests that call external URLs, open network ports, or depend on exact timing .

Benchmark performance

On Microsoft's internal benchmark of 152 tasks, the code-testing-generator agent completed 140 tasks (92.1%) versus 120 tasks (78.9%) for stock GitHub Copilot . This performance gap highlights the advantage of a specialized agent that researches the codebase before generating tests, rather than relying on a general-purpose prompt .