{"id":18145,"date":"2020-12-26T17:00:20","date_gmt":"2020-12-26T16:00:20","guid":{"rendered":"https:\/\/qodeinteractive.com\/magazine\/?p=18145"},"modified":"2021-01-05T13:31:28","modified_gmt":"2021-01-05T12:31:28","slug":"wordpress-robots-txt-file","status":"publish","type":"post","link":"https:\/\/qodeinteractive.com\/magazine\/wordpress-robots-txt-file\/","title":{"rendered":"What is the “robots.txt” File in WordPress and How to Use it"},"content":{"rendered":"

[vc_row][vc_column][vc_column_text]The art and science of getting your website to rank high in the results of various search engines are commonly referred to as SEO (Search Engine Optimization). And there are a lot of different aspects when it comes to SEO, perhaps too many to cover in a single article. This is why we will be focusing on just one right now\u2014the WordPress robots.txt file. In this article, we will dig deeper into what the robots.txt file is and how to use it. Among other things, we will discuss different ways of creating the file and cover the best practices regarding its directives.
\n[\/vc_column_text][vc_empty_space height=”68px”][\/vc_column][\/vc_row][vc_row][vc_column][vc_column_text]<\/p>\n

What is the \u201crobots.txt\u201d file<\/h2>\n

[\/vc_column_text][vc_column_text]Robots.txt is a text file located in your root WordPress directory. You can access it by opening the your-website.com\/robots.txt<\/strong> URL in your browser. It serves to let search engine bots know which pages on your website should be crawled and which shouldn\u2019t. Strictly speaking, a website doesn\u2019t need to have a robots.txt file. If you are in the process of creating your website, it is best to focus on creating quality content first. The search engine bots will crawl your website regardless of whether you have the robots.txt file or not.<\/strong>[\/vc_column_text][vc_empty_space height=”28px”][vc_column_text]However, there are quite a few benefits of having a robots.txt file with proper directives once your WordPress website is finished. Apart from preventing crawlers from crawling unnecessary content, optimized robots.txt directives ensure that your crawl quota (maximum amount of times a crawler can crawl your website during a given time) doesn\u2019t go to waste.
\n[\/vc_column_text][vc_empty_space height=”28px”][vc_column_text]Moreover, well-written WordPress robots.txt directives can reduce the negative effects of bad bots by disallowing them access. This, in turn, might improve your overall website load speed<\/strong>. But keep in mind that robots.txt directives mustn\u2019t be your only security. Bad bots often ignore these directives so using a good security plugin<\/a> is highly advised, particularly if your website is experiencing issues caused by bad bots.[\/vc_column_text][vc_empty_space height=”28px”][vc_column_text]Finally, it is a common misconception that the robots.txt file can prevent some of the pages on your website from being indexed. The robots.txt file can contain directives that prevent crawling, not indexing<\/strong>. And, even if a page isn\u2019t crawled, it can still be indexed through external links leading to it. If you want to avoid indexing a specific page<\/a>, you should use the noindex<\/em> meta tag instead of the directives in the robots.txt file.[\/vc_column_text][vc_empty_space height=”80px”][vc_widget_sidebar sidebar_id=”new-top-picks-banner”][vc_empty_space height=”81px”][\/vc_column][\/vc_row][vc_row][vc_column][vc_column_text]<\/p>\n

How to use the \u201crobots.txt\u201d file<\/h2>\n

[\/vc_column_text][vc_column_text]Having clarified what a WordPress robots.txt file is and some of what it does, we can consider how it\u2019s used. In this section, we will cover how to create and edit a robots.txt file, some of the good practices regarding its content, and how to test it for errors.
\n[\/vc_column_text][vc_empty_space height=”72px”][\/vc_column][\/vc_row][vc_row][vc_column][vc_column_text]<\/p>\n

How to create the \u201crobots.txt\u201d file<\/h3>\n

[\/vc_column_text][vc_column_text]By default, WordPress will create a virtual robots.txt file for any website. Such a file could look something like this, for example:
\n[\/vc_column_text][vc_empty_space height=”50px”]

\n
\n \"Robots\" <\/div>\n<\/div>[vc_empty_space height=”38px”][vc_column_text]However, if you want to edit it, you would need to create a real robots.txt file. In this section, we will explain three ways you can do so. Two of those involve the use of WordPress plugins, while the third relies on the use of FTP.[\/vc_column_text][vc_empty_space height=”22px”]