ToolNest logoToolNest.

Free Online Robots.txt Generator

Quick answer: Control how search engines crawl your site. Our free robots.txt generator builds a valid robots.txt file rule by rule — allow or block specific paths, target individual user-agents like Googlebot, add a crawl-delay, and point to your sitemap. Copy the result or download it, then upload it to your domain root. No signup, everything runs in your browser.

Control how search engines crawl your site. Our free robots.txt generator builds a valid robots.txt file rule by rule — allow or block specific paths, target individual user-agents like Googlebot, add a crawl-delay, and point to your sitemap. Copy the result or download it, then upload it to your domain root. No signup, everything runs in your browser.

User-agentDirectivePath
Upload the file to your domain root as /robots.txt (e.g. example.com/robots.txt), then test it in Google Search Console's robots.txt report.

✓ Free forever  ·  ✓ No signup  ·  ✓ Runs in your browser — your data never leaves your device.

How to Use the Robots.txt Generator

  1. Enter your site URL so the Sitemap line can be built correctly, then pick a preset or build rules manually.
  2. Add one rule per row: the user-agent it applies to (* = all crawlers), Allow or Disallow, and the path.
  3. Optionally add your sitemap URL and a crawl-delay for bots that request one politely.
  4. Click Copy or Download, then upload the file to your domain root as /robots.txt.
  5. Test it in Google Search Console (Settings → robots.txt) and re-check it any time you restructure your site.

Why Use Our Robots.txt Generator?

Frequently Asked Questions

What is a robots.txt file?
A plain-text file at your domain root (example.com/robots.txt) that tells search-engine crawlers which parts of your site they may or may not crawl. It's a request, not enforcement — well-behaved bots like Googlebot honor it.
Where do I put my robots.txt file?
At the root of your domain — https://example.com/robots.txt. Bots only check that one location, so a robots.txt in a subfolder (example.com/blog/robots.txt) is ignored.
Does blocking a page in robots.txt remove it from Google?
No — and this trips people up. A Disallow stops crawling, but Google can still index a URL it discovers via links, sometimes showing it without a description. To truly keep a page out of results, use a noindex tag or password protection instead.
What is crawl-delay and should I use it?
Crawl-delay: 10 asks polite bots to wait 10 seconds between requests. Googlebot ignores it (set crawl rate in Search Console instead), but some smaller crawlers honor it. Leave it blank unless a bot is hammering your server.
What's the difference between Allow and Disallow?
Disallow: /private/ blocks that path; Allow: /private/public/ re-permits a sub-path inside a blocked one. Rules are matched in order of specificity — the most specific matching rule wins.
How do I block AI crawlers like GPTBot?
Use the 'Block AI crawlers' preset or add rules with User-agent: GPTBot, CCBot, anthropic-ai, and PerplexityBot set to Disallow: /. It only works on crawlers that identify themselves honestly, but the major ones do.
How do I test my robots.txt?
Paste the live URL into Google Search Console under Settings → robots.txt — it validates syntax and shows what Googlebot can fetch. Fix any warnings there before relying on the file.

Related Tools

  • Sitemap Generator — Free XML sitemap generator: paste your URLs and build a valid sitemap.xml with lastmod, ch...
  • Meta Tag Generator — Free meta tag generator: create SEO meta tags, Open Graph and Twitter Card tags with live ...
  • Keyword Density Checker — Free keyword density checker: analyze any text for keyword frequency, density %, top words...