How to Noindex a Page in WordPress
When you run a website, you want it to rank as high as possible in search engine results pages (a.k.a. SERPs). To achieve that goal, you’ll need to employ all the optimization techniques in your arsenal and do your best to come up with innovative ways to convince search engines your content is relevant and valuable.
The process of getting your website to the top of SERPs starts with the indexation of its individual pages. Letting bots access your content means your pages are ready to be shared with the world.
Of course, you can always lock your pages with a password if you don’t want users to be able to access them. However, that won’t hide those pages from search engines.
But truth be told, there are plenty of times when it’s better to noindex a page in WordPress. And to hide it from search engines, you need to instruct them not to index it. This is done using the noindex directive, which lets bots crawl a page, but prevents them from ranking it.
In this article, we’ll learn the nuts and bolts of how to noindex a page in WordPress and we’ll discover:
Now let’s get started.
While this may seem strange, sometimes it’s perfectly reasonable to noindex a page in WordPress. In fact, we’ll give you 6 reasons why that may be the right to do.
1. Privacy
You don’t want to share pages that contain sensitive information, confidential details, or users’ personal and contact info with the public.
2. Duplicate Content
In case you publish your content in several formats (for instance, Adobe PDF or printer-friendly versions), we advise you to show just one version to search engines in order to avoid duplicate content issues. Furthermore, if you run an eCommerce website, you should always hide the pages that have similar or identical product descriptions (in case only the product sizes or colors vary, for instance).
3. Keyword Cannibalization Issues
If you target one keyword across multiple pages or posts on your website, the page you consider to be the most relevant for a specific query may not be the one that’ll rank the highest in search engines. To avoid this issue, enable indexing only for the pages that are the most important and relevant for specific query terms.
4. Little or No User Value
Thank You pages, admin login pages, and similar types of pages are not that important for users, so it’s not that necessary to have them ranked in search engine results. They hold no real value for your website visitors, and you don’t need them receiving organic traffic.
5. Work in Progress
In case your pages and posts are under development, you should hide them from search bots until you perfect them to your liking.
6. Third-Party Content
If your website features ads and similar content that is generated by third-party sources, you should hide it from search engines. That type of content usually has nothing to do with the primary purpose of your website.
Alright, so we know in which instances we should hide a WordPress page or post from search engines. Now it’s time we learned how to do that.
Hiding your content from search engines using Yoast SEO is the easiest option to follow, regardless of the level of your WordPress proficiency.
To use this method, you need to install and configure the Yoast plugin. Then, from your WordPress dashboard access the post or a page that you’d like to hide and look for the Yoast SEO metabox (it should be right under the page editor). Now, click on Advanced.
Several options will appear, but we’re interested in the top two. The first one allows you to prevent search engines from showing your page in search results, and the second one stops search engines from following links on your page.
Adding the nofollow tag to your page is usually not necessary. You can use it in case there are paid links on your page or user-submitted links in the comments section. The former are sponsored content and are against Google’s Webmasters Guidelines. As for the latter, the problem is that you can never know how trustworthy the users’ links are, which is why it might be for the best to prevent bots from following them.
Other than that, it’s enough to set your pages to noindex, to keep them off the search index.
When you’re done making changes in Yoast, don’t forget to save them.
Best WordPress Themes for Marketing
View Collection
This method is not recommended for WordPress beginners since it requires you to alter the robots.txt file.
To access the file, connect to your server via FTP or use the File Manager in your cPanel.
Another way of accessing the robots.txt file is by using the Yoast SEO plugin. Go to SEO > Tools and select File Editor.
If you have the robots.txt file, it’ll be displayed at the top. In case you don’t, you can make one by yourself. Google has a great article on how to create a robots.txt file. To hide a page from search engines via the robots.txt file, add the following lines of code to the file:
User-agent: * Disallow: /the-URL-of-your-page/
The first line of code refers to specific bots and search engines you may wish to target. To include all search engines, use an asterisk, just like we did. Insert the URL of the page you’d like to hide in the second line of code, but make sure to exclude your website’s domain name.
For instance, if the URL to your page is “http://aweekendinthecity.com/2020/music/indie-rock” you would put:
User-agent: * Disallow: /2020/music/indie-rock/
When you’re done, you could test your robots.txt file, just to make sure you’ve set everything up correctly. And if you’ve only just created the file, make sure you upload it to the root directory of your website.
The Risks of Using the robots.txt File to Hide Content from Search Engines
Aside from not being quite beginner-friendly, this method comes with a few risks, which is why we’re not its biggest fans.
The robots.txt file is a configuration file that tells search engine crawlers if they should access your pages and posts or not. But not all search engines follow the requests of this file. It’s possible your page will still appear in search results, just without any meta description. Furthermore, if some other website points to yours with descriptive text, your page will be indexed (even if you have just audio and/or video content on it). Plus, malicious bots and content scrapers usually ignore the file.
The robots.txt file is also publicly accessible. This means anyone can see it and easily discover the pages you wish to hide.
Another way of hiding your page from search engines is by adding a noindex meta tag in the HTML file of your page or by including a noindex header in HTTP response headers. For this to work, Google insists that you don’t block your page by a robots.txt file or else search engine bots won’t even see the noindex directive.
How to Add the ‘noindex’ Directive as a Meta Tag
To instruct all search engines not to index your WordPress page, add the noindex meta tag to the HTML <head> section of a given page:
<meta name="robots" content="noindex" />
To prevent just the Googlebot from crawling your page, use this tag:
<meta name="googlebot" content="noindex" />
N.B. Even after you modify thesection and add the noindex meta tag to it, some search engines may still crawl your page.
How to Add the ‘noindex’ Directive in the HTTP Response Header
To instruct search engines not to index your page, you can also use the X-Robots-Tag and add the following HTTP response to the head of the header.php file:
header("X-Robots-Tag: noindex", true);
If you’d like to stop bots from following the links on your page as well, go with this HTTP response:
header("X-Robots-Tag: noindex, nofollow", true);
All the directives you can use in a robots meta tag can also be applied to an X-Robots-Tag. But the latter is a lot more flexible. In fact, X-Robots-Tag controls indexing of both a page as a whole and specific elements on a page, while a robot tag lets you manage indexation on a page level only. For instance, if you wish to stop search engine crawlers from indexing just an image or a video file, you would have to use the X-Robots-Tag.
Which Method Should You Opt For?
After all is said and done, we’d recommend using the Yoast SEO plugin to successfully noindex a page in WordPress. It’s simple and effective. No advanced level of expertise is required to use it. Plus, the changes you make with Yoast won’t affect the root directory of your site.
However, if you wish, experiment with other options, too. In the end, all that matters is that you succeed in preventing search engines from accessing and indexing your WordPress page or post using whatever method you feel most comfortable with.
