Scrapy MCP Server

The Scrapy MCP Server is a robust, enterprise-grade web scraping platform that offers comprehensive data extraction capabilities for commercial use.

Core Scraping Capabilities:

• Multiple scraping methods: HTTP requests, Scrapy framework, Selenium, or Playwright with intelligent method selection

• Concurrent processing: Scrape multiple URLs simultaneously with exponential backoff retry mechanisms

• JavaScript support: Fully render dynamic, JavaScript-heavy websites using complete browser rendering

• Advanced data extraction: Configure flexible extraction rules using simple or advanced selectors, or automatically extract structured data like contact information, social media links, product details, and addresses

• Link extraction: Specialized link extraction with domain filtering and internal/external link options

• Form interaction: Automatically fill and submit various form types including text inputs, checkboxes, and file uploads

Anti-Detection & Performance:

• Stealth techniques: Bypass anti-bot measures using undetected-chromedriver, Playwright stealth, random User-Agent rotation, and proxy support

• Performance optimization: In-memory caching, rate limiting, and intelligent request handling to prevent server overload

• Monitoring tools: Track server metrics including request counts, success rates, cache statistics, and detailed performance monitoring

Enterprise Features:

• Ethical compliance: Check robots.txt files for responsible data collection

• Error handling: Robust error classification and handling mechanisms

• Cache management: Clear scraping results cache and manage server resources

Provides web scraping capabilities using the Scrapy framework for large-scale data extraction, with support for concurrent requests, custom pipelines, and advanced crawling features.

Enables browser automation and JavaScript-heavy website scraping through Selenium WebDriver, with support for form filling, element waiting, and dynamic content extraction.