site-scan
Discover and collect all URLs of a website with the web CLI: parses robots.txt and sitemaps (XML, sitemap indexes, gzip) or follows links, with include/exclude glob patterns, a URL limit and language detection from the URL path. Triggers on: site urls, list pages, discover urls, site inventory, url collection, find all pages, site-scan, website urls, enumerate pages, map site, how many pages.
- License
- Apache-2.0
- Compatibility
- Requires Node 22+; uses the web CLI (plugins/web/cli). No browser needed.
Pinned to revision ae83a5beb6a7, so it is the text this page describes rather than whatever the author pushed since.
Files
Every link opens the file at its source, pinned to the revision this page describes.