Test Runner MCP

A Model Context Protocol (MCP) server for running and parsing test results from multiple testing frameworks. This server provides a unified interface for executing tests and processing their outputs, supporting:

• Bats (Bash Automated Testing System)

• Pytest (Python Testing Framework)

• Flutter Tests

• Jest (JavaScript Testing Framework)

• Go Tests

• Rust Tests (Cargo test)

• Generic (for arbitrary command execution)

Installation

Related MCP server: JMeter MCP Server

Prerequisites

The following test frameworks need to be installed for their respective test types:

• Bats: apt-get install bats or brew install bats

• Pytest: pip install pytest

• Flutter: Follow Flutter installation guide

• Jest: npm install --save-dev jest

• Go: Follow Go installation guide

• Rust: Follow Rust installation guide

Usage

Configuration

Add the test-runner to your MCP settings (e.g., in claude_desktop_config.json or cline_mcp_settings.json):

Note: For Flutter tests, ensure you replace:

• /opt/homebrew/Caskroom/flutter/3.27.2/flutter with your actual Flutter installation path

• /Users/username/.pub-cache with your actual pub cache path

• Update PATH to include your system's actual paths

You can find these values by running:

Running Tests

Use the run_tests tool with the following parameters:

Example for each framework:

Security Features

The test-runner includes built-in security features to prevent execution of potentially harmful commands, particularly for the generic framework:

1. Command Validation

   • Blocks sudo and su by default

   • Prevents dangerous commands like rm -rf /

   • Blocks file system write operations outside of safe locations

2. Environment Variable Sanitization

   • Filters out potentially dangerous environment variables

   • Prevents overriding critical system variables

   • Ensures safe path handling

3. Configurable Security

   • Override security restrictions when necessary via securityOptions

   • Fine-grained control over security features

   • Default safe settings for standard test usage

Security options you can configure:

Flutter Test Support

The test runner includes enhanced support for Flutter tests:

1. Environment Setup

   • Automatic Flutter environment configuration

   • PATH and PUB_CACHE setup

   • Flutter installation verification

2. Error Handling

   • Stack trace collection

   • Assertion error handling

   • Exception capture

   • Test failure detection

3. Output Processing

   • Complete test output capture

   • Stack trace preservation

   • Detailed error reporting

   • Raw output preservation

Rust Test Support

The test runner provides specific support for Rust's cargo test:

1. Environment Setup

   • Automatically sets RUST_BACKTRACE=1 for better error messages

2. Output Parsing

   • Parses individual test results

   • Captures detailed error messages for failed tests

   • Identifies ignored tests

   • Extracts summary information

Generic Test Support

For CI/CD pipelines, GitHub Actions via act, or any other command execution, the generic framework provides:

1. Automatic Output Analysis

   • Attempts to segment output into logical blocks

   • Identifies section headers

   • Detects pass/fail indicators

   • Provides reasonable output structure even for unknown formats

2. Flexible Integration

   • Works with arbitrary shell commands

   • No specific format requirements

   • Perfect for integration with tools like act, Docker, and custom scripts

3. Security Features

   • Command validation to prevent harmful operations

   • Can be configured to allow specific elevated permissions when necessary

Output Format

The test runner produces structured output while preserving complete test output:

Results are saved in the specified output directory:

• test_output.log: Raw test output

• test_errors.log: Error messages if any

• test_results.json: Structured test results

• summary.txt: Human-readable summary

Development

Setup

1. Clone the repository

2. Install dependencies:

   npm install

3. Build the project:

   npm run build

Running Tests

The test suite includes tests for all supported frameworks and verifies both successful and failed test scenarios.

CI/CD

The project uses GitHub Actions for continuous integration:

• Automated testing on Node.js 18.x and 20.x

• Test results uploaded as artifacts

• Dependabot configured for automated dependency updates

Contributing

1. Fork the repository

2. Create your feature branch

3. Commit your changes

4. Push to the branch

5. Create a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.