Crawler.sh

Product details from official sources

crawler.sh is a local crawler that turns any website into clean, RAG-ready Markdown for AI training, fine-tuning, and agent context. It renders JavaScript with a custom engine (no headless Chrome), respects robots.txt with adaptive pacing, and runs entirely on the user's machine with no per-page fees or API quota. It ships as a CLI (crawl, info, export, seo subcommands), a desktop app, and an MCP server for agents; crawls handle up to 10,000 pages per session with concurrent requests, 24 SEO checks, and export to NDJSON, JSON, Sitemap XML, CSV, or Markdown archives. Built in Rust, the CLI runs on macOS and Linux (Windows on the roadmap). No account is required; signing in raises the per-crawl page limit from 50 to 400.

Capabilities and conditions

• crawler.sh is a local crawler that turns websites into RAG-ready Markdown for AI training, fine-tuning, and agent context. Source ↗
• It renders JavaScript with a custom engine, respects robots.txt, runs locally on the laptop, with no headless Chrome, no per-page fees, and no API quota. Source ↗
• The CLI runs on macOS (Apple Silicon and Intel) and Linux (x86_64). Source ↗
• No account is required to use the CLI or desktop app. Source ↗
• Signing in raises the limit from 50 to 400 pages per crawl. Source ↗

Price and free allowance

See the sourced plan statements above; confirm currency, billing period and checkout eligibility on the vendor site.

Published-source profile · collected 2026-10-17. Automated collection followed by editorial review. These are vendor-published claims, not hands-on product test results.