SEO Tools Suite

Robots.txt & AI Crawler Shield Generator

Create standard search engine directives with 1-click protection against generative AI scrapers like GPTBot, Claude, and Common Crawl.

Robots.txt & AI Shield Builder

General Search Crawlers (Google, Bing)

Allow Search Bots by DefaultPermit standard indexing for Googlebot and Bingbot

AI Training & Crawler Protection

5 Shielded
OpenAI / ChatGPT (GPTBot)User-agent: GPTBot
BLOCKED
Anthropic / Claude (Claude-Web)User-agent: Claude-Web
BLOCKED
Common Crawl (CCBot)User-agent: CCBot
BLOCKED
ByteDance (Bytespider)User-agent: Bytespider
BLOCKED
Google AI Training (Google-Extended)User-agent: Google-Extended
BLOCKED

Blocked Directories (Disallow Paths)

Disallow: /admin/
Disallow: /checkout/
Disallow: /api/

Generated robots.txt Preview

Root file (/robots.txt)
# robots.txt generated by BusinessTools
# https://businesstools.com/tools/seo-suite/robots-txt-generator

User-agent: *
Disallow: /admin/
Disallow: /checkout/
Disallow: /api/

# AI Scrapers & Data Harvester Blocking Directives
User-agent: GPTBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Google-Extended
Disallow: /


Sitemap: https://businesstools.com/sitemap.xml

Where to Place robots.txt

Your robots.txt file must live in the top-level root directory of your website domain (e.g. https://yourdomain.com/robots.txt). In Next.js App Router, place the exported file inside your public folder or export dynamic rules via app/robots.ts.

AI Scraper Shielding

Disallowing AI user-agents stops automated bots from ingesting your copyrighted blog posts and product data into LLM training corpuses while preserving regular Google and Bing rankings.

Why Use This Free Browser-Based Utility on ToolPort?

Modern founders, marketers, and operators need fast, reliable answers without bloated subscriptions or mandatory account signups. Every utility on ToolPort runs with instant feedback, clean formatting, and privacy-first execution so your sensitive numbers, documents, and domain configurations remain strictly under your control.

Zero Data Retention

Calculations and formatting happen directly inside your active session. We never store your financial inputs, proprietary files, or client audit logs on external databases.

Built for Production Workflows

Whether you are preparing client deliverables, validating technical markup before a site migration, or modeling unit economics, every output is structured for immediate use.