Максимально полный список юзер-агентов AI-ботов для их блокировки

Максимально полный список юзер-агентов AI-ботов для их блокировки

AI и ML боты краулят ваш сайт и парсят контент гораздо чаще, чем вы думаете.

Чтобы остановить AI-ботов от поглощения вашего контента для обучения своих систем вы можете заюзать этот Мега-блокировщик в .HTACCESS:

.htaccess

<IfModule mod_rewrite.c>

RewriteEngine On

RewriteCond %{HTTP_USER_AGENT} (Agent\ GPT|AgentGPT|AI\ Article\ Writer|AI\ Content\ Detector|AI\ Dungeon|AI\ SEO\ Crawler|AI\ Search\ Engine|AI\ Writer|AI21\ Labs|AIBot|AI2Bot|AISearchBot|AlexaTM|Alpha\ AI|AlphaAI|Amazon\ Bedrock) [NC,OR]

RewriteCond %{HTTP_USER_AGENT} (Amazon\ Lex|Amazonbot|Amelia|anthropic-ai|AnthropicAI|AnyPicker|Anyword|Applebot|Articoolo|AutoGPT|Automated\ Writer|AwarioRssBot|AwarioSmartBot|BingAI|Brave\ Leo\ AI|Bytespider|CatBoost|CC-Crawler) [NC,OR]

RewriteCond %{HTTP_USER_AGENT} (CCBot|ChatGPT|Chinchilla|Claude-Web|ClaudeBot|ClearScope|cohere-ai|cohere-training-data-crawler|Common\ Crawl|commoncrawl|Content\ Harmony|Content\ King|Content\ Optimizer|Content\ Samurai|ContentAtScale) [NC,OR]

RewriteCond %{HTTP_USER_AGENT} (ContentBot|Contentedge|Conversion\ AI|CopyAI|Copymatic|Copyscape|Crawlspace|CrawlQ\ AI|crew\ AI|crewAI|DALL-E|DataForSeoBot|DeepAI|DeepL|DeepMind|DeepSeek|DepolarizingGPT|DialoGPT|Diffbot|DuckAssistBot) [NC,OR]

RewriteCond %{HTTP_USER_AGENT} (FacebookBot|Firecrawl|Flyriver|Frase\ AI|FriendlyCrawler|Gemini|Gemma|GenAI|Google\ Bard\ AI|Google-CloudVertexBot|Google-Extended|GoogleOther|GPT-2|GPT-3|GPT-4|GPTBot|GPTZero|Grammarly|Grok|Hemingway) [NC,OR]

RewriteCond %{HTTP_USER_AGENT} (Hugging\ Face|Hypotenuse\ AI|iaskspider|ICC-Crawler|ImagesiftBot|img2dataset|INK\ Editor|INKforall|IntelliSeek\.ai|Inferkit|ISSCyberRiskCrawler|JasperAI|Kafkai|Kangaroo|Keyword\ Density\ AI|LeftWingGPT) [NC,OR]

RewriteCond %{HTTP_USER_AGENT} (LLaMA|magpie-crawler|MarketMuse|Meltwater|Meta\ AI|Meta\ Llama|Meta\.AI|Meta-AI|Meta-ExternalAgent|Meta-ExternalFetcher|MetaAI|MetaTagBot|Mistral|Narrative\ Device|Neural\ Text|NeuralSEO|OAI-SearchBot) [NC,OR]

RewriteCond %{HTTP_USER_AGENT} (OAI\ SearchBot|omgili|OmniGPT|Open\ AI|OpenAI|OpenText\ AI|Outwrite|Page\ Analyzer\ AI|PanguBot|Paraphraser\.io|peer39_crawler|PerplexityBot|PetalBot|ProWritingAid|QuillBot|RightWingGPT|RobotSpider) [NC,OR]

RewriteCond %{HTTP_USER_AGENT} (Rytr|SaplingAI|Scalenut|Scrapy|ScriptBook|SearchGPT|SemrushBot|SEO\ Content\ Machine|SEO\ Robot|Sidetrade|Simplified\ AI|SlickWrite|Spin\ Rewriter|Spinbot|Stability|Sudowrite|Surfer\ AI|Text\ Blaze) [NC,OR]

RewriteCond %{HTTP_USER_AGENT} (TextCortex|The\ Knowledge\ AI|Timpibot|VelenPublicWebCrawler|Vidnami\ AI|WebChatGPT|Webzio|Whisper|WordAI|Wordtune|Writecream|WriterZen|Writescope|Writesonic|x\.AI|xAI|YouBot|Zero\ GTP|ZimmWriter) [NC]

RewriteRule (.*) - [F,L]

</IfModule>

Полный список из 176 юзер-агентов вот тут.

@MikeBlazerX

Источник новости https://t.me/mikeblazerx/4717...