niaid-bp-robots
Check a site's robots.txt for AI crawler groups, IETF Content-Usage (train-ai), Cloudflare Content-Signal, and Crawl-delay. One homepage request records HTTP 429, Retry-After, and a Content-Usage header. Use when the user gives a repository or site URL and asks about robots.txt, or runs /niaid-bp-robots.
- Version
- 0.1
- License
- Apache-2.0
Pinned to revision 416c766ac5f1, so it is the text this page describes rather than whatever the author pushed since.
Files
- skills/niaid-bp-robots/SKILL.md
- skills/niaid-bp-robots/scripts/__pycache__/check_robots.cpython-313.pyc
- skills/niaid-bp-robots/scripts/check_robots.py
- skills/niaid-bp-robots/tests/__pycache__/test_check_robots.cpython-313-pytest-9.0.3.pyc
- skills/niaid-bp-robots/tests/test_check_robots.py
Every link opens the file at its source, pinned to the revision this page describes.