# How to edit the robots.txt file in WordPress > Canonical: https://jethost.com/kb/how-to-edit-robots-txt-in-wordpress/ · Last modified: 2026-10-31T13:34:38+00:00 Your robots.txt file tells search engines which parts of your site they may crawl and where to find your sitemap. WordPress creates a basic one for you automatically, but you can customise it. This guide explains what the file does, the safe ways to edit it, what to put inside, and the mistakes to avoid. ## What robots.txt does robots.txt is a plain text file at the root of your site, reachable at `yourdomain.com/robots.txt`. It gives crawlers instructions: which paths to skip, and where your sitemap lives. On large sites it also helps you spend crawl budget on the pages that matter instead of on endless archive or filter URLs. Two things it does *not* do. It does not make a page private — a blocked URL can still be public and even show in results, so use a password or a noindex tag for that. And it only guides well-behaved crawlers; bad bots ignore it entirely. ## The default WordPress robots.txt Out of the box, WordPress serves a **virtual** robots.txt — there is no real file on disk, WordPress generates it on request. It usually looks like this, blocking the admin area while allowing the one file the front end needs: ``` User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://yourdomain.com/sitemap_index.xml ``` For most sites this default is fine. You only need to edit it when you want to add your sitemap, block a specific path, or give instructions to a particular crawler. ## How to edit robots.txt There are two common ways: through an SEO plugin, or by creating the file by hand. Both work — pick the one that matches how you already manage your site. ### With an SEO plugin An SEO plugin adds search-optimisation features to WordPress — titles, meta descriptions, sitemaps — and most include a built-in robots.txt editor, so you can change the file from the dashboard without FTP. The two most widely used on WordPress are Yoast SEO and Rank Math; if you already run one of them, use its editor. - **Yoast SEO** is a long-established SEO plugin. It generates your XML sitemap and, on most setups, lets you edit robots.txt under **Yoast SEO → Tools → File editor**. - **Rank Math** is a newer, popular SEO plugin with a similar feature set. You edit robots.txt under **Rank Math SEO → General Settings → Edit robots.txt**. In either plugin, make your changes in the editor and save. The plugin writes the file for you. Note that if a real robots.txt file exists on the server, the plugin editor is usually disabled, because the physical file takes priority (see below). ### By hand (File Manager or FTP) You can also create the file yourself. In cPanel [File Manager](https://jethost.com/kb/how-to-use-cpanel-file-manager/) or over [FTP](https://jethost.com/kb/category/ftp/), create a file named `robots.txt` in your site’s public\_html folder and add your rules. A real file on disk overrides the WordPress virtual one, so from then on you edit it here. Keep a copy of the contents before you change anything, so you can revert. ## What to put in it A few directives cover almost every need: - **User-agent** — which crawler the rules apply to. An asterisk means all of them. - **Disallow** — a path crawlers should not request. - **Allow** — an exception inside a disallowed path. - **Sitemap** — the full address of your sitemap, so crawlers can find every page. Always include this. ## Common mistakes to avoid - Do not block your CSS and JavaScript. If crawlers cannot load them, Google may not render your pages correctly. - Do not use robots.txt to hide private content. It only asks crawlers not to visit; the URL stays public. - Do not accidentally disallow the whole site. A lone `Disallow: /` tells every crawler to skip everything. - After editing, test the file in Google Search Console so you know the rules do what you expect. ## JetHost Experts Tip Do not use robots.txt to hide sensitive pages. It only asks crawlers not to visit, and the URL stays public. For real privacy, password-protect the page or set it to noindex instead. ## Frequently asked questions **Where is the robots.txt file in WordPress?**By default WordPress serves a virtual robots.txt at yourdomain.com/robots.txt, so there is no physical file to find. If you add a real robots.txt in your site’s public\_html folder, it overrides the virtual one. **Do I need an SEO plugin to edit robots.txt?**No. A plugin like Yoast SEO or Rank Math makes it convenient from the dashboard, but you can also create the file by hand in File Manager or over FTP. Both approaches produce the same file. **Will editing robots.txt hide a page from Google?**No. robots.txt only asks crawlers not to visit a path. The URL can still be public and appear in results. To keep a page private, password-protect it or set it to noindex. ## New Hosting Plan with 80% OFF Upgrade your website’s home without the heavy bill. Get new [hosting plan](https://jethost.com/web-hosting/ "Web hosting with free SSH access") with a huge 80% discount. Get free SSL, email accounts, and a control panel to manage your site. Enjoy fast performance, secure environment, and 24/7 support - all in one affordable package. ## Need more help? Not sure what belongs in your robots.txt? Open a ticket through **Client Area → Support** and our team will review it with you.