Display the results of your Page Modified crawls right within your WordPress Admin.
Plugin Tag: crawler
-
SEObot Monitor for Googlebot, Bingbot and search engine spiders
With SEObot Monitor for Googlebot you will be able to dump the server logs to Google Analytics, easily and automatically. This will allow you to dete …
-
Block Feed and Comments via robots.txt
Blocks some Feed and Comment resources in order to avoid spam bots
-
Block Common Crawl via robots.txt
Blocks the Common Crawl bot (CCBot) that gathers data that is used to train AI algorithms like ChatGPT through the WordPress virtual robots.txt file
-
Findkit – Site Search
Powerful cloud based search engine for your website. Search as you type, real-time index updates, and embeddable UI. Free for small websites.
-
Warmer – Cache Warmer and Cache Preload Crawler
Warm your WordPress cache with scheduled crawling, cache preloading, URL source control, logs, diagnostics, and WP-CLI.
-
AI Deny
Easily block over 30 of the most common AI user-agents known for crawling websites, keeping unauthorized bots at bay.
-
Block eCommerce Assets via robots.txt
Blocks some WooCommerce assets and Search pages that should not be indexed through robots.txt
-
Block Chat GPT via robots.txt
Blocks the ChatGPT-User bot that is used by plugins in ChatGPT to crawl websites through the WordPress virtual robots.txt file
-
bodi0`s Bots visits counter
Counts the visits from web spiders, crawlers and bots in your blog, with ability to get the blog rankings.