Posts in category: Tips & Tricks

Do you want to know how you can prevent indexing your WordPress website or a specific page of a website and remove it from the search?

Then, you are at the right place.

I made this article short and simple to understand this topic quickly without any jargon. 

So, let’s start

[Updated: 20.01.2023]

There may be a lot of reasons why you may want to stop indexing your website.

Here, I have mentioned some grounds on which someone prevents the website from search engine crawlers or bots. 

The reason for doing it.

1. Under Construction or Maintenance: If your website is currently undergoing maintenance or is not yet ready for public viewing, you may want to prevent search engines from crawling it. This will prevent your website from appearing in search results and potentially getting penalized for having a poor user experience.
2. Privacy Concerns: If your website contains sensitive information, such as personal data or confidential business information, you may want to prevent search engines from crawling it to protect that information from being accessed by unauthorized parties.
3. Testing and Development: If you are testing changes to your website or developing a new version, you may want to prevent search engines from crawling it to avoid confusion and errors.
4. Staging Environment: If you have a development or staging environment, you may want to prevent search engines from crawling it to prevent confusion and errors.
5. Preventing Duplicate Content: If you have multiple versions of the same content on your website, you may want to prevent search engines from crawling them to avoid being penalized for duplicate content.

Now, I will tell you guys,

The steps to hide the complete website.

1. 1. Log in to your WordPress Dashboard.
   2. Click on the “Settings” option from the left side panel and click on the “Reading” option.
   3. Scroll down on the right side and you will find the “Discourage search engines from indexing this site” option and click on the box, and it checkmarks it.
   4. And the last thing to do is apply the settings on the website. Just click the “save changes” option.

1.  Note: After this step followed. 

A code is added in the robot.txt file that notifies the crawlers not to index the website.

But there may be a chance of getting crawlers later on.

Or,

The other way this by adding the code manually to the robot.txt

Now, the steps to hide a website from crawlers on a WordPress website through the “robots.txt” file:

1. The first step is to access your website’s “robots.txt” file. This file is located at the root directory of your website.

2. Edit the “robots.txt” file.

Once you have accessed the “robots.txt” file, you can edit it by adding the following code:

User-agent: *

Disallow: /

This code tells all web crawlers that they are not allowed to access any pages or directories on your website.

You can also specify which pages or directories you want to block by replacing the “/” with the specific URL or directory you want to block.

For example, if you want to block the “About” page on your website, you can add the following code:

User-agent: *

Disallow: /about

Save the “robots.txt” file.

Once you have edited the “robots.txt” file, be sure to save your changes. You can do this by clicking the “Save” button at the bottom of the page.

Check the “robots.txt” file to ensure it is working correctly

After you have saved the “robots.txt” file, you should check to make sure it is working correctly.

To do this, you can use a tool like Google’s “Fetch as Google” or the “Robots.txt Tester” tool in the Google Search Console. These tools will allow you to test whether the “robots.txt” file is blocking the pages or directories you specified.

Consider using the “noindex” tag

In addition to using the “robots.txt” file to hide pages or directories from search engines, you can also use the “noindex” tag in the “head” section of your website’s HTML code.

This tag tells search engines not to index a specific page or post on your website.

To use the “noindex” tag, you can add the following code to the “head” section of your website’s HTML code:

<meta name=”robots” content=”noindex”>

Conclusion:

In conclusion, the “robots.txt” file is a useful tool for hiding pages or directories on your WordPress website from search engines.

By following the steps outlined above, you can easily block web crawlers from accessing certain areas of your website.

Just be sure to check the “robots.txt” file to ensure it is working correctly, and consider using the “noindex” tag as an additional measure to prevent your pages from being indexed by search engines.

The steps to hide a Specific Webpage.

There’s no other option in WordPress, particularly for a specific page.

But you can install a plugin for it and, it will solve the purpose.

I use RankMath for SEO purposes and, it has the option to disable indexing for a specific page.

For RankMath Plugin, the steps are:

1. 1. Go to the “Pages” tab and click on the “Edit” button for that page you want to hide from the search engine robots.
   2. After that, on the right panel from the Rank Math panel, go to the “Advanced” tab (suitcase icon).
   3. Under the “Robots Meta” tab in the panel, click on the “No Index” checkbox.
   4. And now, click on the “Publish/Update” button. That’s all.

Note: These are the two methods which I have mentioned. You can use to hide a page or website from indexing that is WordPress setting and, the other one is by using a plugin.

But, if you have external links on the page or the website and the links by default are set to “follow“. Then the search engine will still follow the webpage.

And if you don’t want that, you can set it to the “no follow” option available in the panel. 

If you want to know more tips and tricks and other related aspects, then click here and sign up (It’s completely free). 

After you sign up, you will get emails related to a new post uploaded on this website.

Also, share your thoughts and suggestions on which topic you want a blog in the comments section or click here. And I will try to create a blog on the topic you asked for.