crawlers – The unofficial WordPress Plugin Directory

LJPL Armored robots.txt

Add some directives to your robots.txt file to keep your site safer

Analyze how search engines and other bots crawl and understand your web page. The official PHP WordPress plugin for log-hero.com.

Serve Markdown versions of your posts and pages to AI agents and crawlers. Content negotiation, .md URLs, auto-discovery, and crawler logging.

Easily manage your WordPress site's virtual robots.txt file with custom rules. No physical file needed!

The easiest way to manage meta robots tag.

Easy way to discourage search engines from indexing only specific pages / posts with custom meta tags.

Advanced llms.txt generator with AI metadata, Schema.org types, canonical URLs, and differentiated licenses for optimal LLM indexing.

Control how AI crawlers use your content with Cloudflare Content Signals in robots.txt. Includes EU Directive 2019/790 rights reservation.

Provide the easy managment of your robots.txt from admin side. It propose you the advanced then standard robots.txt content too.

Tell AI (Artificial Intelligence) companies not to scrape your site for their AI products.