What is robots.txt and what does it do?

robots.txt is a text file that tells search engine crawlers which pages they can or cannot access on your website. It's placed in the root directory (example.com/robots.txt) and helps manage crawler traffic, protect private areas, and optimize crawl budget.

How do I create a robots.txt file online?

Select a preset (Allow All, Block All, Standard, WordPress, E-commerce, Block AI, Block SEO) or start from scratch. Add user-agent rules, configure Allow/Disallow paths, add sitemap URLs, and download or copy the generated file.

Is robots.txt mandatory for my website?

No, but it's highly recommended. Without it, crawlers will attempt to access all pages. robots.txt helps manage crawler traffic, protect private areas, prevent duplicate content indexing, and optimize your crawl budget for important pages.

Does robots.txt block pages from appearing in search results?

No! robots.txt only prevents crawling, not indexing. Pages can still appear in search results if linked from other sites. To truly block pages from search results, use meta robots tags with "noindex" or X-Robots-Tag HTTP headers.

What does User-agent mean in robots.txt?

User-agent identifies the crawler. Use * (asterisk) for all bots, or specific names like Googlebot, Bingbot, GPTBot for targeted rules. Different crawlers can have different permissions. Rules are matched from most specific to least specific.

What is the difference between Allow and Disallow?

Disallow prevents crawling of specified paths. Allow explicitly permits crawling, useful for allowing subdirectories within disallowed paths. Example: Disallow: /admin/ blocks /admin/, but Allow: /admin/public/ permits that subdirectory.

How do I block AI crawlers like GPTBot and ChatGPT?

Add rules for AI crawler user-agents: User-agent: GPTBot, User-agent: ChatGPT-User, User-agent: CCBot, User-agent: anthropic-ai, followed by Disallow: /. The tool includes a "Block AI" preset with all known AI crawler user-agents.

What is Crawl-delay in robots.txt?

Crawl-delay tells bots to wait a specified number of seconds between requests, reducing server load. Note: Google ignores Crawl-delay; use Google Search Console instead. Bing and other crawlers respect it. Example: Crawl-delay: 10

Should I include my sitemap in robots.txt?

Yes! Adding your sitemap URL (Sitemap: https://example.com/sitemap.xml) helps search engines discover all your pages efficiently. You can list multiple sitemaps. This is especially important for large sites or sites with complex navigation.

Can I block specific file types with robots.txt?

Yes, use patterns with wildcards: Disallow: /*.pdf$ blocks all PDF files, Disallow: /*.jpg$ blocks JPG images, Disallow: /*?* blocks URLs with query parameters. The $ means "end of URL".

How do I test my robots.txt file?

Use Google Search Console's robots.txt Tester to verify your rules work as expected. Enter URLs to test if they're blocked or allowed. The tool highlights syntax errors and shows which rules match each URL.

Where should robots.txt be placed?

Always in the root directory: https://example.com/robots.txt. It won't work in subdirectories. For subdomains, each needs its own robots.txt (blog.example.com/robots.txt). The file must be accessible via HTTP/HTTPS.

What are common robots.txt mistakes to avoid?

Common mistakes: blocking CSS/JS files (breaks rendering), blocking important pages accidentally, using wrong path syntax, forgetting trailing slashes, blocking sitemap access, and assuming robots.txt prevents indexing (it doesn't).

How do I block a specific page but allow the directory?

Use specific path rules: Allow: /products/ then Disallow: /products/secret-page. More specific rules take precedence. You can also use wildcards: Disallow: /products/*-draft to block all draft pages.

Is my robots.txt configuration secure when using this tool?

Yes! All robots.txt generation happens 100% in your browser using JavaScript. Your configuration is NEVER sent to any server, stored, or logged. The tool works completely offline for maximum privacy and security.

What is robots.txt and what does it do?

robots.txt is a text file that tells search engine crawlers which pages they can or cannot access on your website. It's placed in the root directory (example.com/robots.txt) and helps manage crawler traffic, protect private areas, and optimize crawl budget.

How do I create a robots.txt file online?

Select a preset (Allow All, Block All, Standard, WordPress, E-commerce, Block AI, Block SEO) or start from scratch. Add user-agent rules, configure Allow/Disallow paths, add sitemap URLs, and download or copy the generated file.

Is robots.txt mandatory for my website?

No, but it's highly recommended. Without it, crawlers will attempt to access all pages. robots.txt helps manage crawler traffic, protect private areas, prevent duplicate content indexing, and optimize your crawl budget for important pages.

Does robots.txt block pages from appearing in search results?

No! robots.txt only prevents crawling, not indexing. Pages can still appear in search results if linked from other sites. To truly block pages from search results, use meta robots tags with "noindex" or X-Robots-Tag HTTP headers.

What does User-agent mean in robots.txt?

User-agent identifies the crawler. Use * (asterisk) for all bots, or specific names like Googlebot, Bingbot, GPTBot for targeted rules. Different crawlers can have different permissions. Rules are matched from most specific to least specific.

What is the difference between Allow and Disallow?

Disallow prevents crawling of specified paths. Allow explicitly permits crawling, useful for allowing subdirectories within disallowed paths. Example: Disallow: /admin/ blocks /admin/, but Allow: /admin/public/ permits that subdirectory.

How do I block AI crawlers like GPTBot and ChatGPT?

Add rules for AI crawler user-agents: User-agent: GPTBot, User-agent: ChatGPT-User, User-agent: CCBot, User-agent: anthropic-ai, followed by Disallow: /. The tool includes a "Block AI" preset with all known AI crawler user-agents.

What is Crawl-delay in robots.txt?

Crawl-delay tells bots to wait a specified number of seconds between requests, reducing server load. Note: Google ignores Crawl-delay; use Google Search Console instead. Bing and other crawlers respect it. Example: Crawl-delay: 10

Should I include my sitemap in robots.txt?

Yes! Adding your sitemap URL (Sitemap: https://example.com/sitemap.xml) helps search engines discover all your pages efficiently. You can list multiple sitemaps. This is especially important for large sites or sites with complex navigation.

Can I block specific file types with robots.txt?

Yes, use patterns with wildcards: Disallow: /*.pdf$ blocks all PDF files, Disallow: /*.jpg$ blocks JPG images, Disallow: /*?* blocks URLs with query parameters. The $ means "end of URL".

How do I test my robots.txt file?

Use Google Search Console's robots.txt Tester to verify your rules work as expected. Enter URLs to test if they're blocked or allowed. The tool highlights syntax errors and shows which rules match each URL.

Where should robots.txt be placed?

Always in the root directory: https://example.com/robots.txt. It won't work in subdirectories. For subdomains, each needs its own robots.txt (blog.example.com/robots.txt). The file must be accessible via HTTP/HTTPS.

What are common robots.txt mistakes to avoid?

Common mistakes: blocking CSS/JS files (breaks rendering), blocking important pages accidentally, using wrong path syntax, forgetting trailing slashes, blocking sitemap access, and assuming robots.txt prevents indexing (it doesn't).

How do I block a specific page but allow the directory?

Use specific path rules: Allow: /products/ then Disallow: /products/secret-page. More specific rules take precedence. You can also use wildcards: Disallow: /products/*-draft to block all draft pages.

Is my robots.txt configuration secure when using this tool?

Yes! All robots.txt generation happens 100% in your browser using JavaScript. Your configuration is NEVER sent to any server, stored, or logged. The tool works completely offline for maximum privacy and security.

robots.txt Generator

medium

Advanced robots.txt generator with 7 presets (Allow All, Block All, Standard, WordPress, E-commerce, Block AI, Block SEO), 50+ user agents organized by category, common path templates, import/parse existing files, and validation.

Web Developmentrobotsseocrawlersearch engine

100% Private - Runs Locally

How to Use

1Select a preset or start from scratch
2Add user-agent rules for specific crawlers
3Configure Allow and Disallow paths
4Add sitemap URLs
5Set optional crawl delay
6Import existing robots.txt to edit
7Download or copy the generated file

Examples

Allow All

Input

Allow all bots to crawl everything

Output

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Block Admin

Input

Block admin and private directories

Output

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/
Disallow: /api/

Block Specific Bots

Input

Block SEO crawlers

Output

User-agent: *
Allow: /

User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

Frequently Asked Questions

Related Tools

View all

Sitemap.xml Generator

Advanced sitemap generator with 5 types (Standard, Index, Image, Video, News), 5 presets, bulk URL import, validation, and detailed statistics for SEO optimization.

Meta Tag Generator

Generate HTML meta tags for SEO, social sharing, and search engine optimization. Includes Open Graph, Twitter Cards, and all essential meta tags.

.htaccess Generator

Advanced .htaccess generator with 5 presets (Basic, WordPress, Security, Performance, SPA), 30+ configuration options for redirects, caching, compression, security headers, HSTS, CSP, bot blocking, IP blocking, password protection, CORS, and PHP settings.