README.md•16.1 kB
# SpiderFoot MCP Agent
A Node.js implementation of the Model Context Protocol (MCP) server that exposes SpiderFoot's functionality as tools. This project provides both an MCP server and a web client for interacting with the SpiderFoot web interface.
## Features
- **MCP Server**: Exposes SpiderFoot functionality through the Model Context Protocol
- **Web Client**: Programmatic interface to interact with SpiderFoot's web interface
- **TypeScript Support**: Full TypeScript support for better development experience
- **Docker Support**: Easy deployment using Docker
- **Modular Design**: Easy to extend with new functionality
## Requirements
- Node.js 18+ (recommended 20+)
- A local SpiderFoot instance (Docker or direct installation)
- Default web interface URL: `http://127.0.0.1:5001`
- Docker (optional, for containerized deployment)
## Setup
### Prerequisites
1. Ensure you have a running instance of SpiderFoot
2. Clone this repository:
```bash
git clone https://github.com/yourusername/Spiderfoot-MCP-Agent.git
cd Spiderfoot-MCP-Agent
```
### Installation
1. Install dependencies:
```bash
npm install
```
2. Configure environment:
```bash
cp .env.example .env
```
Edit the `.env` file with your SpiderFoot details:
```env
# Base URL of your SpiderFoot instance
SPIDERFOOT_BASE_URL=http://127.0.0.1:5001
# Authentication (if enabled in SpiderFoot)
# SPIDERFOOT_USER=username
# SPIDERFOOT_PASS=password
# Allow starting scans through the API
ALLOW_START_SCAN=true
```
## Usage
### Running the MCP Server
#### Development Mode (stdio transport)
```bash
npm run dev
```
#### Development Mode (HTTP transport)
```bash
npm run dev:http
```
#### Production Build
```bash
# Build the project
npm run build
# Start the server
npm start
```
### Using the Web Client
The package includes a web client that can be used to interact with the SpiderFoot web interface programmatically.
```javascript
import { SpiderFootWebClient } from './spiderfoot-web-client.js';
// Create a new client instance
const client = new SpiderFootWebClient('http://127.0.0.1:5001');
// List all scans
const scans = await client.listScans();
console.log('Existing scans:', scans);
// Start a new scan
try {
const result = await client.startScan('example.com', ['type_DNS_TEXT'], 'domain', 'test-scan');
console.log('Scan started:', result);
} catch (error) {
console.error('Failed to start scan:', error);
}
```
## Development
### Building the Project
```bash
npm run typecheck
npm run build
```
Start from compiled output:
```bash
npm start # stdio transport
npm run start:http # HTTP transport (dist/index-http.js)
```
## Tools
The server registers the following tools:
- `spiderfoot_ping` – GET `/ping`
- `spiderfoot_modules` – GET `/modules`
- `spiderfoot_event_types` – GET `/eventtypes`
- `spiderfoot_scans` – GET `/scanlist`
- `spiderfoot_scan_info` – GET `/scanopts?id=<sid>`
- `spiderfoot_start_scan` – POST `/startscan` (guarded by `ALLOW_START_SCAN`)
- `spiderfoot_scan_data` – POST `/scaneventresults`
- `spiderfoot_scan_data_unique` – POST `/scaneventresultsunique`
- `spiderfoot_scan_logs` – POST `/scanlog`
- `spiderfoot_export_json` – POST `/scanexportjsonmulti`
Dangerous endpoints like `/query` are intentionally omitted.
## HTTP vs stdio transports
- `src/index.ts` uses the stdio transport (`StdioServerTransport`). This is commonly used when an IDE/agent launches your process and communicates via stdio.
- `src/index-http.ts` uses the Streamable HTTP transport, listening on `/:port/mcp` (default port `3000`). Use this for remote/HTTP-based MCP clients.
Environment variable for HTTP port:
- `MCP_HTTP_PORT` (default: `3000`)
## Docker usage
This repo includes a `Dockerfile` and `docker-compose.yml` to run the MCP server in Docker.
Build the image:
```bash
docker build -t spiderfoot-mcp:local .
```
Run with Docker directly:
```bash
docker run --rm -p 3000:3000 \
-e SPIDERFOOT_BASE_URL=http://host.docker.internal:5001 \
-e ALLOW_START_SCAN=true \
-e MCP_HTTP_PORT=3000 \
--name spiderfoot-mcp spiderfoot-mcp:local
```
Or with Compose:
```bash
docker-compose up --build
```
Compose file (`docker-compose.yml`) configures:
- Service: `spiderfoot-mcp`
- Port mapping: `3000:3000`
- Default env points to your host’s SpiderFoot at `http://host.docker.internal:5001`
Notes:
- On Linux, replace `host.docker.internal` with your host IP or use the container network to reach your SpiderFoot service.
- Ensure SpiderFoot is reachable on port `5001` from inside the MCP container.
## Environment variables
- `SPIDERFOOT_BASE_URL` — Base URL of your SpiderFoot web UI/API.
- `ALLOW_START_SCAN` — `true|false`. Enables/disables `spiderfoot_start_scan` tool. Default `true`.
- `SPIDERFOOT_USER`, `SPIDERFOOT_PASS` — Optional HTTP Digest credentials if you enable auth in SpiderFoot.
- `MCP_HTTP_PORT` — Port for HTTP transport (if using `index-http.ts`). Default `3000`.
## Project layout
- `src/index.ts` — MCP server (stdio transport) and tool registration.
- `src/index-http.ts` — MCP server (HTTP transport) with session management.
- `src/spiderfootClient.ts` — Axios-based client for SpiderFoot endpoints.
- `Dockerfile` — Multi-stage image: builds TS → runs HTTP server.
- `docker-compose.yml` — Runs container with env defaults.
## Using with IDEs and MCP-compatible clients
This section provides JSON-based configuration examples for connecting this MCP server from popular IDEs and tools. Two transport modes are supported:
1. Stdio transport: the IDE launches your local process
2. HTTP transport: the IDE connects to a running server at `http://localhost:5002/mcp` (Docker with compose) or `http://localhost:3000/mcp` when running `npm run dev:http` locally
You can use both; add two separate entries if your IDE supports it.
### Docker-based JSON (stdio inside container)
If you prefer your IDE to launch the MCP server inside Docker (without needing a long-running compose service), use this stdio-in-container configuration. It runs the stdio entrypoint (`dist/index.js`) and communicates over stdin/stdout.
```json
{
"mcpServers": {
"spiderfoot-mcp-docker-stdio": {
"command": "docker",
"args": [
"run",
"-i",
"--rm",
"--add-host=host.docker.internal:host-gateway",
"-e",
"SPIDERFOOT_BASE_URL=http://host.docker.internal:5001",
"spiderfoot-mcp:local",
"node",
"dist/index.js"
],
"env": {}
}
}
}
```
Copy-paste Claude Desktop block (Docker stdio + HTTP):
```json
{
"mcpServers": {
"spiderfoot-mcp-docker-stdio": {
"command": "docker",
"args": [
"run",
"-i",
"--rm",
"--add-host=host.docker.internal:host-gateway",
"-e",
"SPIDERFOOT_BASE_URL=http://host.docker.internal:5001",
"spiderfoot-mcp:local",
"node",
"dist/index.js"
]
},
"spiderfoot-mcp-http": {
"type": "http",
"url": "http://localhost:5002/mcp"
}
}
}
```
Notes:
- Make sure you have built the image (`docker build -t spiderfoot-mcp:local .` or `docker-compose build`).
- This approach does not expose a port; it uses stdio via Docker (`-i`).
- The host SpiderFoot URL is passed via `-e SPIDERFOOT_BASE_URL=http://host.docker.internal:5001`.
### Common configuration examples
__Stdio (local process)__
```json
{
"mcpServers": {
"spiderfoot-mcp-stdio": {
"type": "stdio",
"command": "node",
"args": [
"./node_modules/tsx/dist/cli.mjs",
"src/index.ts"
],
"cwd": "C:/dev-env.local/project-repos/Spiderfoot-MCP-Agent",
"env": {
"SPIDERFOOT_BASE_URL": "http://127.0.0.1:5001",
"ALLOW_START_SCAN": "true"
}
}
}
}
```
__HTTP (connect to running server)__
```json
{
"mcpServers": {
"spiderfoot-mcp-http": {
"type": "http",
"url": "http://localhost:5002/mcp"
}
}
}
```
Notes:
- If you prefer `npm start` instead of `tsx`, update `command`/`args` accordingly, e.g. `command: "npm", args: ["run", "dev"]`.
- On Windows, keep forward slashes in `cwd` or escape backslashes (e.g., `C:\\dev-env.local\\project-repos\\Spiderfoot-MCP-Agent`).
- Ensure SpiderFoot is reachable at `SPIDERFOOT_BASE_URL` from the MCP server.
### Windsurf
Steps:
1. Open `Settings` → `MCP` (or Tools/Integrations section that manages MCP servers).
2. Add a new server entry.
3. Paste one of the JSON examples above into your MCP server configuration, merging with any existing `mcpServers` entries. Recommended options:
- Docker stdio: `spiderfoot-mcp-docker-stdio` (uses `command: docker`)
- HTTP: `serverUrl` to `http://localhost:5002/mcp`
4. Save settings.
5. Start the server if using HTTP mode (Docker Compose or `npm run dev:http`). For stdio, Windsurf will launch it automatically when needed.
Windsurf – Option 2: HTTP via serverUrl
```json
{
"mcpServers": {
"spiderfoot-mcp-http": {
"serverUrl": "http://localhost:5002/mcp"
}
}
}
```
Windsurf – Option 1: Docker stdio
```json
{
"mcpServers": {
"spiderfoot-mcp-docker-stdio": {
"command": "docker",
"args": [
"run",
"-i",
"--rm",
"--add-host=host.docker.internal:host-gateway",
"-e",
"SPIDERFOOT_BASE_URL=http://host.docker.internal:5001",
"spiderfoot-mcp:local",
"node",
"dist/index.js"
]
}
}
}
```
Notes:
- Make sure you have built the image (`docker build -t spiderfoot-mcp:local .` or `docker-compose build`).
- This approach does not expose a port; it uses stdio via Docker (`-i`).
- The host SpiderFoot URL is passed via `-e SPIDERFOOT_BASE_URL=http://host.docker.internal:5001`.
### Cursor
Steps:
1. Open Cursor settings for MCP integrations.
2. Add a new MCP server.
3. Use the Docker stdio JSON to launch in a container, or the HTTP example to connect to `http://localhost:5002/mcp`.
4. Save and test by listing tools from the MCP panel.
Cursor – Option 1: Docker stdio
```json
{
"mcpServers": {
"spiderfoot-mcp-docker-stdio": {
"command": "docker",
"args": [
"run",
"-i",
"--rm",
"--add-host=host.docker.internal:host-gateway",
"-e",
"SPIDERFOOT_BASE_URL=http://host.docker.internal:5001",
"spiderfoot-mcp:local",
"node",
"dist/index.js"
]
}
}
}
```
Cursor – Option 2: HTTP
```json
{
"mcpServers": {
"spiderfoot-mcp-http": {
"type": "http",
"url": "http://localhost:5002/mcp"
}
}
}
```
### Claude Desktop
Claude Desktop reads a JSON configuration file that can include the `mcpServers` map shown above.
Typical configuration file locations:
- Windows: `%APPDATA%/Claude/claude_desktop_config.json`
- macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`
- Linux: `~/.config/Claude/claude_desktop_config.json`
Add or merge one of the following under a top-level `mcpServers` object if your extension reads from it, or under the extension-specific key (e.g., `"cline.mcpServers"`).
Claude Desktop – Option 1: Docker stdio
```json
{
"mcpServers": {
"spiderfoot-mcp-docker-stdio": {
"command": "docker",
"args": [
"run",
"-i",
"--rm",
"--add-host=host.docker.internal:host-gateway",
"-e",
"SPIDERFOOT_BASE_URL=http://host.docker.internal:5001",
"spiderfoot-mcp:local",
"node",
"dist/index.js"
]
}
}
}
```
Claude Desktop – Option 2: HTTP
```json
{
"mcpServers": {
"spiderfoot-mcp-http": {
"type": "http",
"url": "http://localhost:5002/mcp"
}
}
}
```
### VS Code (Continue)
Configuration is typically stored in VS Code `settings.json`.
Common locations:
- Windows: `%APPDATA%/Code/User/settings.json`
- macOS: `~/Library/Application Support/Code/User/settings.json`
- Linux: `~/.config/Code/User/settings.json`
Add or merge the following under a top-level `mcpServers` object if your extension reads from it, or under the extension-specific key (e.g., `"continue.mcpServers"`).
VS Code (Continue) – Option 1: Docker stdio
```json
{
"mcpServers": {
"spiderfoot-mcp-docker-stdio": {
"command": "docker",
"args": [
"run",
"-i",
"--rm",
"--add-host=host.docker.internal:host-gateway",
"-e",
"SPIDERFOOT_BASE_URL=http://host.docker.internal:5001",
"spiderfoot-mcp:local",
"node",
"dist/index.js"
]
}
}
}
```
VS Code (Continue) – Option 2: HTTP
```json
{
"mcpServers": {
"spiderfoot-mcp-http": {
"type": "http",
"url": "http://localhost:5002/mcp"
}
}
}
```
Notes:
- Some VS Code MCP extensions expect a namespaced key (e.g., `continue.mcpServers`). If so, copy the object assigned to `mcpServers` above into that namespaced setting.
- Ensure the working directory (`cwd`) points at `Spiderfoot-MCP-Agent/`.
### VS Code (Cline)
VS Code (Cline) – Option 1: Docker stdio
```json
{
"mcpServers": {
"spiderfoot-mcp-docker-stdio": {
"command": "docker",
"args": [
"run",
"-i",
"--rm",
"--add-host=host.docker.internal:host-gateway",
"-e",
"SPIDERFOOT_BASE_URL=http://host.docker.internal:5001",
"spiderfoot-mcp:local",
"node",
"dist/index.js"
]
}
}
}
```
VS Code (Cline) – Option 2: HTTP
```json
{
"mcpServers": {
"spiderfoot-mcp-http": {
"type": "http",
"url": "http://localhost:5002/mcp"
}
}
}
```
### JetBrains (Continue plugin)
Open your JetBrains IDE settings → Continue → MCP (or Tools/Integrations) and add a server using the same JSON entries shown above.
If your IDE stores a JSON configuration file, place the same `mcpServers` map in that file and restart the IDE. Use stdio or HTTP entries per your preference.
JetBrains (Continue) – Option 1: Docker stdio
```json
{
"mcpServers": {
"spiderfoot-mcp-docker-stdio": {
"command": "docker",
"args": [
"run",
"-i",
"--rm",
"--add-host=host.docker.internal:host-gateway",
"-e",
"SPIDERFOOT_BASE_URL=http://host.docker.internal:5001",
"spiderfoot-mcp:local",
"node",
"dist/index.js"
]
}
}
}
```
JetBrains (Continue) – Option 2: HTTP
```json
{
"mcpServers": {
"spiderfoot-mcp-http": {
"type": "http",
"url": "http://localhost:5002/mcp"
}
}
}
```
### Zed
Open Zed settings JSON (e.g., `~/.config/zed/settings.json`) and add an MCP servers map. For many setups, a root-level `mcpServers` object works; otherwise, consult Zed’s MCP documentation for the exact key.
Zed – Option 1: Docker stdio
```json
{
"mcpServers": {
"spiderfoot-mcp-docker-stdio": {
"command": "docker",
"args": [
"run",
"-i",
"--rm",
"--add-host=host.docker.internal:host-gateway",
"-e",
"SPIDERFOOT_BASE_URL=http://host.docker.internal:5001",
"spiderfoot-mcp:local",
"node",
"dist/index.js"
]
}
}
}
```
Zed – Option 2: HTTP
```json
{
"mcpServers": {
"spiderfoot-mcp-http": {
"type": "http",
"url": "http://localhost:5002/mcp"
}
}
}
```
### MCP Inspector (testing)
- Stdio: run `npm run dev` and point Inspector to that command.
- HTTP: run Docker Compose (or `npm run dev:http`) and connect Inspector to `http://localhost:5002/mcp`.
## Notes
- Source files are in `src/`:
- `src/index.ts` – MCP server definition and tool registration (stdio).
- `src/index-http.ts` – Streamable HTTP transport variant.
- `src/spiderfootClient.ts` – HTTP wrapper around SpiderFoot endpoints using `axios`.
- The project uses ESM (`"type": "module"`), TypeScript 5, and zod for input validation.
- Default behavior allows starting scans; disable by setting `ALLOW_START_SCAN=false`.