How to Create a Robots.txt File for WordPress

WordPress creates a virtual robots.txt automatically, but to customize it, generate the file and either upload it to your site's root folder via FTP or paste it into your SEO plugin's file editor (Rank Math and Yoast both have one). A WordPress robots.txt blocks /wp-admin/, allows admin-ajax.php, and lists your sitemap. ToolNest's free robots.txt generator builds the file in seconds.

Where is robots.txt in WordPress?

Every WordPress site answers at yoursite.com/robots.txt — visit it and you'll see the file live. Here's the twist: WordPress usually generates it virtually, on the fly, with no physical file on the server. SEO plugins like Yoast and Rank Math take over that virtual file and let you edit its contents from the dashboard. If you upload a real robots.txt to your site's root folder (public_html), the physical file wins and the virtual one is ignored — so pick one method and stick with it, or you'll edit one file while the other keeps serving.

The two-minute method: generate, then install

Don't hand-write robots.txt from memory — generate it. ToolNest's free robots.txt generator builds a WordPress-ready file in seconds: the standard rules plus your XML sitemap line, formatted correctly. Then install it one of two ways. Option A — SEO plugin editor: in Rank Math, go to Rank Math SEO, General Settings, Edit robots.txt and paste; in Yoast, SEO, Tools, File editor. Fastest, no FTP needed. Option B — upload: save the file as robots.txt and upload it to your domain's root folder via FTP/SFTP or your host's File Manager. Either way, verify by visiting yoursite.com/robots.txt — what you see there is what Google sees.

What to put in a WordPress robots.txt

A solid WordPress robots.txt is short. The default that works for most sites:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com/sitemap_index.xml
Line by line: User-agent: * addresses all crawlers. Disallow: /wp-admin/ keeps bots out of your login and dashboard — no SEO value there, and it cuts crawl waste. Allow: /wp-admin/admin-ajax.php re-opens the one admin file the frontend needs for AJAX features. Sitemap: points crawlers straight at your XML sitemap index, which Yoast and Rank Math generate as sitemap_index.xml automatically. What not to add: don't block /wp-includes/ or your theme's asset folders — Google needs your CSS and JavaScript to render pages properly, and blocking them can hurt rankings. Keep it lean; every extra rule is another chance to accidentally block something valuable.

How to edit robots.txt in WordPress

Three routes — pick one. Rank Math: Dashboard, Rank Math SEO, General Settings, Edit robots.txt — a simple text area, Save Changes, done. Yoast SEO: SEO, Tools, File editor, then the robots.txt tab (if the tab is missing, your site root isn't writable — ask your host or use FTP instead). Manual: connect via FTP/SFTP or open File Manager in cPanel or hPanel, navigate to public_html (the domain root), and upload or edit robots.txt there. Critical warning: a physical file overrides the plugin's virtual file silently. If you edit in Rank Math but an old physical file exists, your edits do nothing — check yoursite.com/robots.txt after every change to confirm what's actually live.

Three robots.txt mistakes that hurt WordPress SEO

1. Leaving 'Disallow: /' live. Staging sites and the 'discourage search engines' checkbox (Settings, Reading) write a blanket block — and it sometimes survives the move to production, telling Google to index nothing. Check first, always. 2. Blocking assets. Disallow rules on /wp-content/uploads/ or theme folders starve Google of the CSS, JavaScript, and images it needs to render — pages get judged on broken layouts. 3. Treating robots.txt as security. The file is public by design; anyone can read your Disallow lines, and attackers use them as a treasure map to admin paths. It also can't remove URLs from Google's index — that needs a noindex tag. Robots.txt manages crawler attention, not access and not secrecy. For the on-page side of SEO, our keyword density guide and URL slug best practices cover what crawlers should find once they arrive.

Testing it and connecting your sitemap

After any change, verify. In Google Search Console, Settings, robots.txt report shows the exact file Google fetched, with warnings for syntax problems — confirm the fetched version matches what you intended. Third-party robots.txt testers validate syntax line by line. Note that Google caches robots.txt for about 24 hours, so fixes take a day to take effect. And don't forget the sitemap line: it only helps if the sitemap exists and is submitted — pair this guide with a sitemap generator so crawlers get both the rules and the map. Rules without a map, or a map the rules contradict, both waste crawl budget.