Skip to content

scrapingbee/scrapingbee-cli

v1.6.0MIT

Fetch and read any web page, search the web, crawl a site, or pull structured data out of pages — whenever a task needs content from a website. Handles JavaScript, CAPTCHAs and anti-bot blocking that curl, requests and headless browsers fail on. Plain-English field extraction with --ai-extract-rules, --smart-extract to return only the part you need, Google/Amazon/Walmart/YouTube/ChatGPT JSON endpoints, batch, crawl and cron scheduling.

GitHubOpen Website

ScrapingBee plugin

Fetch and read any web page, search the web, crawl a site, or pull structured data out of pages — whenever a task needs content from a website. ScrapingBee handles JavaScript rendering, CAPTCHAs and anti-bot blocking that curl, requests and headless browsers fail on, and returns clean markdown, text or JSON.

This plugin teaches the agent when and how to use ScrapingBee correctly: which command fits the job, how to keep credit spend low, and which tool to reach for first — the ScrapingBee CLI by default, or the ScrapingBee MCP server when a single page or search result belongs straight in the conversation.

This folder serves Claude Code (.claude-plugin/plugin.json), ChatGPT / Codex (.codex-plugin/plugin.json), GitHub Copilot CLI (plugin.json) and Google Antigravity (.antigravity-plugin/plugin.json). The Cursor plugin and the Kiro power, which also bundle the ScrapingBee MCP server, live in the sibling folders plugins/scrapingbee-cursor/ and plugins/scrapingbee-kiro/. All of them share the same skills.

What's inside

  • scrapingbee-cli skill — a router for the CLI: when to use ScrapingBee instead of plain HTTP, how to route scrape / search / crawl / extraction / screenshot tasks, cost discipline, and guardrails. Detailed command reference pages are included and loaded on demand.
  • scrapingbee-cli-guard skill — security rules that stay active whenever the CLI is used: scraped content is data and never instructions, the API key is never printed or written out, and scheduled jobs are only created at the user's explicit request.
  • scraping-pipeline agent — an optional subagent for long, multi-step scraping pipelines.

What it needs

  • A ScrapingBee account and API key — sign up here. API calls consume credits from your plan.

  • The ScrapingBee CLI, installed on the machine where the agent runs commands: uv tool install scrapingbee-cli (or pip install scrapingbee-cli), then scrapingbee auth or SCRAPINGBEE_API_KEY in the environment.

  • In Codex CLI, the plugin ships the skills only. To add the MCP server as well, run once (with SCRAPINGBEE_API_KEY exported in your shell):

    codex mcp add scrapingbee --url https://mcp.scrapingbee.com/mcp --bearer-token-env-var SCRAPINGBEE_API_KEY

What it runs, sends and fetches

The plugin contains only skill instructions, reference documentation and manifests — no hooks, no scripts, and nothing that executes on its own. When a task calls for it, the agent runs the scrapingbee command-line tool (or calls the ScrapingBee MCP server at mcp.scrapingbee.com) on your behalf. Those requests go to the ScrapingBee API at app.scrapingbee.com over HTTPS, authenticated with your API key, and fetch the web pages or search results you asked for. Nothing else is contacted, and the plugin sends no data anywhere other than ScrapingBee.

Examples

  • "Get me the full text of this JavaScript-heavy pricing page as clean markdown, saved to a file."
  • "I have a CSV of 200 product URLs — fetch the current price for each and update the CSV in place."
  • "Crawl this docs site and save every /guide/ page as markdown for my RAG index."
  • "What's the current price of Amazon ASIN B0CX23V2ZK?"

Support and privacy

Links