Curated SEO, GEO & AI Search resources — technical SEO, agent discoverability, analytics, and traffic monetization.
advertools - online marketing productivity and analysis tools
🤖 The largest directory for AI-ready documentation and tools implementing the proposed llms.txt standard
A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
Tame the robots crawling and indexing your Nuxt site.
🐘 Simple PHP library to help developers 🍻 do better on-page SEO optimization 🤖
The robots.txt exclusion protocol implementation for Go language
A set of reusable Java components that implement functionality common to any web crawler
Determine if a page may be crawled from robots.txt, robots meta tags and robot headers
Ultimate Website Sitemap Parser
A simple but powerful web crawler library for .NET
Open-Source Python Based SEO Web Crawler
Opt-Out tool to check Copyright reservations in a way that even machines can understand.
NodeJS robots.txt parser with support for wildcard (*) matching.
Known tags and settings suggested to opt out of having your content used for AI training.
Terminal CLI to download complete web pages (HTML + assets) for offline viewing. Markdown export, robots.txt aware.
Answer Engine Optimization for the modern web. Make your site discoverable by ChatGPT, Claude, Perplexity & AI search engines. Generates llms.txt, robots.txt, sitemap, JSON-LD & more.
Open-source GEO / AEO skill for AI visibility readiness: check robots.txt, llms.txt, schema, citation signals, competitor gaps, and buyer prompts
Parse through any sitemap in Node.js
To associate your repository with the robots-txt topic, visit your repo's landing page and select "manage topics."