MCP Server

A server that generates Master Content Plans (MCPs) based on topics. The server aggregates resources from the web and organizes them into structured learning paths.

Features

• Generate learning paths for any topic (not just technology topics)

• Find relevant resources using web search and scraping

• Organize resources into a logical sequence with customizable number of nodes

• Support for multiple languages with focus on Portuguese

• Performance optimizations for Render's free tier

• Caching system for faster responses

• Return a standardized JSON structure for consumption by client applications

• NEW: TF-IDF based resource relevance filtering to ensure resources match the requested topic

• NEW: Strategic quiz distribution across learning trees for balanced learning experiences

• NEW: YouTube integration to include relevant videos in learning paths

• NEW: Category system to generate more specific content for different types of topics

• NEW: Asynchronous task system with real-time progress feedback to improve user experience and avoid timeouts

• NEW: Enhanced caching system for improved performance and faster response times

• NEW: Optimized web scraping techniques for better resource utilization

• NEW: Adaptive scraping system that automatically chooses the most efficient method for each website

• NEW: Puppeteer instance pool for efficient browser reuse and reduced memory usage

Tech Stack

• Python 3.9+

• FastAPI

• Pyppeteer for JavaScript-heavy web scraping

• Pyppeteer-stealth for avoiding detection during web scraping

• Puppeteer instance pool for efficient browser reuse

• DuckDuckGo Search API

• BeautifulSoup for HTML parsing

• scikit-learn for TF-IDF based resource relevance filtering

• yt-dlp for YouTube video search and metadata extraction

• Redis (optional) for distributed caching

• msgpack for efficient data serialization

• cachetools for advanced in-memory caching

Installation

1. Clone the repository:

   git clone https://github.com/yourusername/mcp_server.git cd mcp_server

2. Create a virtual environment:

   python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate

3. Install Python dependencies:

   pip install -r requirements.txt

4. Install Node.js dependencies (for the optimized scraping system):

   npm install

5. Install Chrome/Chromium for Pyppeteer (if not already installed)

Usage

Running Locally

1. Start the server using the provided batch file (Windows):

   run_local.bat

   Or manually with uvicorn:

   uvicorn main:app --reload --host 0.0.0.0 --port 8000

2. Access the API at http://localhost:8000

3. Generate an MCP by making a GET request to:

   GET /generate_mcp?topic=your_topic

4. Check the API documentation at http://localhost:8000/docs

Production URL

The production server is available at:

All endpoints documented in this README are available at both the local and production URLs.

Testing the Caching System

1. Make a first request to generate an MCP (this will populate the cache):

   GET /generate_mcp?topic=python&num_nodes=15&language=pt

2. Make a second request with the same parameters (this should use the cache):

   GET /generate_mcp?topic=python&num_nodes=15&language=pt

   The second request should be significantly faster as the result will be retrieved from the cache.

Documentation

Detailed documentation is available in the docs folder:

• API Reference - Detailed API documentation

• Endpoints Reference - Complete reference of all endpoints

• Flutter Integration - Guide for integrating with Flutter apps

• Async Tasks System - Documentation for the asynchronous task system

• Performance Improvements - Overview of performance optimizations

• Caching System - Documentation for the caching system

• Web Scraping Optimization - Details on web scraping optimizations

API Endpoints

• GET /health - Health check endpoint

• GET /generate_mcp?topic={topic}&max_resources={max_resources}&num_nodes={num_nodes}&min_width={min_width}&max_width={max_width}&min_height={min_height}&max_height={max_height}&language={language}&category={category} - Generate an MCP for the specified topic synchronously

  • topic (required): The topic to generate an MCP for (minimum 3 characters)

  • max_resources (optional): Maximum number of resources to include (default: 15, min: 5, max: 30)

  • num_nodes (optional): Number of nodes to include in the learning path (default: 15, min: 10, max: 30)

  • min_width (optional): Minimum width of the tree (nodes at first level) (default: 3, min: 2, max: 10)

  • max_width (optional): Maximum width at any level of the tree (default: 5, min: 3, max: 15)

  • min_height (optional): Minimum height of the tree (depth) (default: 3, min: 2, max: 8)

  • max_height (optional): Maximum height of the tree (depth) (default: 7, min: 3, max: 12)

  • language (optional): Language for resources (default: "pt")

  • category (optional): Category for the topic (e.g., "technology", "finance", "health"). If not provided, it will be detected automatically.

• POST /generate_mcp_async?topic={topic}&max_resources={max_resources}&num_nodes={num_nodes}&min_width={min_width}&max_width={max_width}&min_height={min_height}&max_height={max_height}&language={language}&category={category} - Start asynchronous generation of an MCP

• GET /status/{task_id} - Check the status of an asynchronous task

• GET /tasks - List all tasks

• POST /clear_cache?pattern={pattern}&clear_domain_cache={clear_domain_cache} - Clear the cache based on a pattern

  • pattern (optional): Pattern to match cache keys (default: "*" for all)

  • clear_domain_cache (optional): Whether to also clear the domain method cache (default: false)

• GET /cache_stats - Get statistics about the cache, including information about the domain method cache

Examples

Basic usage (Portuguese)

Custom number of nodes

English language

Specify category manually

Full customization

Control tree structure

Asynchronous generation

Check task status

Clear cache

Clear specific cache

Performance Improvements

The MCP Server includes several performance optimizations:

• Caching System: Results are cached to improve response times for repeated queries

• Asynchronous Task System: Long-running operations are handled asynchronously

• Resource Filtering: TF-IDF based filtering to select the most relevant resources

• Optimized Web Scraping: Efficient web scraping techniques to reduce resource usage

• Adaptive Scraping System: Automatically chooses the most efficient scraping method for each website

• Puppeteer Instance Pool: Reuses browser instances to reduce memory usage and startup time

• Domain Method Cache: Remembers which scraping method works best for each domain

• Resource Blocking: Blocks unnecessary resources (images, stylesheets, fonts) during scraping

Performance Gains

• 60-80% reduction in response time for topics already in cache

• 30-50% reduction in response time for new topics

• 40-60% reduction in memory usage during web scraping

• 3-5x increase in throughput for simultaneous requests

Deployment

The server can be deployed to various platforms:

Using Docker

Deploying to Render, Fly.io, or other platforms

Follow the platform-specific instructions for deploying a Docker container or a Python application.

License

Proprietary Software - All Rights Reserved

This software is proprietary and confidential. Unauthorized copying, distribution, modification, public display, or public performance of this software is strictly prohibited. This software is intended for use under a paid subscription model only.

© 2024 ReuneMacacada. All rights reserved.

Last commit: v1.1.2 - Correção de problemas com DuckDuckGo rate limit e Puppeteer