Set Magento 2 noindex and nofollow tags

When you're starting a new project on a dev server or make some transactional pages, you may need to hide some of your pages or the whole website from search engines not to be indexed beforehand. The best way to tell robots not to index or follow your content is to use the robots.txt file and nofollow and noindex meta tags in html head of the page(s). Magento Open Source or Adobe Commerce website allows you to configure such settings from the admin panel with no code used.

What Is robots.txt File And Why Do You Need It?

Let's say, some web robots want to visit your site https://store.emagicone.com and index or track its content. Before starting the indexation the robots will first enter your https://store.emagicone.com/robots.txt file and check it. This file is located in the root of your Magento site and contains the instructions for robots on how to process or not process your site.

Setting Noindex and Nofollow in Magento 2

There are several methods to instruct the robots not to index certain store content. Below we will describe how to close from indexing the entire store, pages, products and categories.

Method 1. Setting Noindex/Nofollow For Magento 2 Store

If you run Magento 2.1.x go to Stores > Configuration > General (On the left) > Design section

If you run Magento 2.2.0, 2.2.1 or 2.2.2 store in the admin panel choose Content > Design section — Configuration:

You will find there a table containing all your storeviews. Click “Edit” next to a store configuration:

The window with “Global” settings will open. Scroll down to “Other Settings” and click on “Search Engine Robots” to expand this section. Here you will see the “Default Robots” field. Click to see the drop-down list:

There are four “Default Robots” options available:

• INDEX, FOLLOW - allows web robots to index your site and track it for changes;
• NOINDEX, FOLLOW - instructs robots not to index your site, but allows to track it for changes;
• INDEX, NOFOLLOW - instructs robots to index your site, but not to check it for changes furtherly.
• NOINDEX, NOFOLLOW - forbids the robots neither to index nor to track you site.

You can select the option based on your purposes or use “Edit custom instruction of robots.txt File” field, in case you need to block access to some specific folders or URLs.

Input the following directives for robots into this field depending on your needs:

To block all robots from certain folders:

User-agent: *

Disallow: /folder name

To block specific robots from certain folders:

User-agent: robot name

Disallow: /folder name

To block some bots to access one folder, but allow to access another one:

User-agent: robot name

Disallow: /folder name Allow: /folder name

Method 2. Setting Noindex/Nofollow For Specific Magento 2 Pages

There is another way to block robots from processing your web pages.

From the admin panel on the left, select “Content” section. Under the “Elements” select “Pages”. You will find a table containing all pages of your store. Click “Select” on the right and choose the “Edit' option from the drop-down list:

Expand the “Design” section and find the “Layout Update XML” field. In order to prevent this page from indexing and following, insert this code into the field:

Method 3: Setting Noindex/Nofollow For Specific Magento 2 Categories and Products

The above indicated code can also be used to configure permissions to some of your category or product pages whether in 2.1.x or in 2.2.

To block the product page select “Products” from the “Catalog” section on the admin panel. Press “Edit” on the right side of the product field:

Scroll down and find the “Design” section. Expand it and enter the piece of code into the 'Layout Update XML' field. Press “Save”:

To block category page select “Categories” from the “Catalog” section and find the necessary category in the list on the right. Find 'Design' section and enter the above indicated code into 'Layout Update XML' field:

Conclusion: Do I Need to Block Access to Some of My Links?

Definitely, yes. It is essential for an online store owner to be able to manage access to some of his/her site’s folders or URLs by forbidding or allowing web robots to index or follow them. Beside limiting access for unknown crawlers, you can considerably improve your SEO by applying these restrictions.

The thing is that, when filling your site with content, you probably strive to optimize it for Google as much as possible in order to show up in the first positions of search results. Although, some of your site links might have some negative influence on your Google rate, so it would be appropriate to disallow search engine robots to index or track them.

Magento provides its users with a majority of ways to manage site access in few clicks directly from admin panel. Take advantage of any of them and keep your web store protected from any undesirable indexing or tracking.

Related Articles:

How to Optimize your Magento 2 SEO?

Detect SEO issues of the whole Magento store at once and automatically solve them clicking on a button. Read More...

Magento 2 Meta Tags Optimization

Learn how to optimize your Magento 2 meta tags and move your pages higher in search results. Read More...

Magento 2 XML Sitemap

How to make that web crawlers correctly understood your website organization and indexed its content faster. Read More...

Magento 2 SEO Features

Find out how track, analyze and optimize your SEO Magento 2 on regular basis and omprove your SEO ranking. Read More...