How to Add or Edit a robots.txt File in WordPress

by

 The robots.txt file in WordPress is used to guide search engine crawlers on how to index your website. It can restrict or allow access to specific parts of your site and is useful for managing your site's SEO and crawl efficiency.


Example of a Basic robots.txt File for WordPress:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

# Optional: Prevent indexing of certain directories
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/

# Optional: Block search results pages
Disallow: /?s=

# Optional: Sitemap
Sitemap: https://yourwebsite.com/sitemap.xml

Method 1: Using SEO Plugins

  1. Install and activate an SEO plugin like Yoast SEO or Rank Math.
  2. Navigate to the plugin's settings (e.g., Yoast: SEO > Tools > File Editor).
  3. Edit or create the robots.txt file within the plugin interface.
  4. Save changes.

Method 2: Manual Upload

  1. Create a file named robots.txt using a text editor.
  2. Add your desired rules.
  3. Upload the file to the root directory of your WordPress site (e.g., https://yourwebsite.com/robots.txt) via FTP or a file manager.

Testing Your robots.txt File:

  1. Open https://yourwebsite.com/robots.txt in a browser to verify that it is accessible.
  2. Use tools like Google Search Console to check how Google interacts with your robots.txt file.


Comments

Post a Comment

Thank you