run_evaluation_tests
Execute evaluation tests on CircleCI pipelines by generating temporary configuration files and triggering new pipelines. Returns a URL to monitor pipeline progress.
Instructions
This tool allows the users to run evaluation tests on a circleci pipeline.
They can be referred to as "Prompt Tests" or "Evaluation Tests".
This tool triggers a new CircleCI pipeline and returns the URL to monitor its progress.
The tool will generate an appropriate circleci configuration file and trigger a pipeline using this temporary configuration.
The tool will return the project slug.
Input options (EXACTLY ONE of these THREE options must be used):
Option 1 - Project Slug and branch (BOTH required):
- projectSlug: The project slug obtained from listFollowedProjects tool (e.g., "gh/organization/project")
- branch: The name of the branch (required when using projectSlug)
Option 2 - Direct URL (provide ONE of these):
- projectURL: The URL of the CircleCI project in any of these formats:
* Project URL with branch: https://app.circleci.com/pipelines/gh/organization/project?branch=feature-branch
* Pipeline URL: https://app.circleci.com/pipelines/gh/organization/project/123
* Workflow URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def
* Job URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def/jobs/xyz
Option 3 - Project Detection (ALL of these must be provided together):
- workspaceRoot: The absolute path to the workspace root
- gitRemoteURL: The URL of the git remote repository
- branch: The name of the current branch
Test Files:
- promptFiles: Array of prompt template file objects from the ./prompts directory, each containing:
* fileName: The name of the prompt template file
* fileContent: The contents of the prompt template file
Pipeline Selection:
- If the project has multiple pipeline definitions, the tool will return a list of available pipelines
- You must then make another call with the chosen pipeline name using the pipelineChoiceName parameter
- The pipelineChoiceName must exactly match one of the pipeline names returned by the tool
- If the project has only one pipeline definition, pipelineChoiceName is not needed
Additional Requirements:
- Never call this tool with incomplete parameters
- If using Option 1, make sure to extract the projectSlug exactly as provided by listFollowedProjects
- If using Option 2, the URLs MUST be provided by the user - do not attempt to construct or guess URLs
- If using Option 3, ALL THREE parameters (workspaceRoot, gitRemoteURL, branch) must be provided
- If none of the options can be fully satisfied, ask the user for the missing information before making the tool call
Returns:
- A URL to the newly triggered pipeline that can be used to monitor its progress
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| params | No |
Input Schema (JSON Schema)
{
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"properties": {
"params": {
"additionalProperties": false,
"properties": {
"branch": {
"description": "The name of the branch currently checked out in local workspace. This should match local git branch. For example: \"feature/my-branch\", \"bugfix/123\", \"main\", \"master\" etc.",
"type": "string"
},
"gitRemoteURL": {
"description": "The URL of the remote git repository. This should be the URL of the repository that you cloned to your local workspace. For example: \"https://github.com/user/my-project.git\"",
"type": "string"
},
"pipelineChoiceName": {
"description": "The name of the pipeline to run. This parameter is only needed if the project has multiple pipeline definitions. If not provided and multiple pipelines exist, the tool will return a list of available pipelines for the user to choose from. If provided, it must exactly match one of the pipeline names returned by the tool.",
"type": "string"
},
"projectSlug": {
"description": "The project slug from listFollowedProjects tool (e.g., \"gh/organization/project\"). When using this option, branch must also be provided.",
"type": "string"
},
"projectURL": {
"description": "The URL of the CircleCI project. Can be any of these formats:\n- Project URL with branch: https://app.circleci.com/pipelines/gh/organization/project?branch=feature-branch\n- Pipeline URL: https://app.circleci.com/pipelines/gh/organization/project/123\n- Workflow URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def\n- Job URL: https://app.circleci.com/pipelines/gh/organization/project/123/workflows/abc-def/jobs/xyz",
"type": "string"
},
"promptFiles": {
"description": "Array of prompt template files in the ./prompts directory (e.g. <relevant-name>.prompt.yml).",
"items": {
"additionalProperties": false,
"properties": {
"fileContent": {
"description": "The contents of the prompt template file",
"type": "string"
},
"fileName": {
"description": "The name of the prompt template file",
"type": "string"
}
},
"required": [
"fileName",
"fileContent"
],
"type": "object"
},
"type": "array"
},
"workspaceRoot": {
"description": "The absolute path to the root directory of your project workspace. This should be the top-level folder containing your source code, configuration files, and dependencies. For example: \"/home/user/my-project\" or \"C:\\Users\\user\\my-project\"",
"type": "string"
}
},
"required": [
"promptFiles"
],
"type": "object"
}
},
"type": "object"
}