When you create a website, you want search engines to understand your pages and show the right content to users. But not every page needs to appear in Google Search. Some pages may be private, temporary, duplicated, or useful only for website visitors.

This is where the meta robots tag becomes important.

A meta robots tag is an HTML element that provides instructions to search engine crawlers about how a particular webpage should be indexed and displayed in search results. It is placed in the page’s HTML and can control actions such as indexing, following links, and showing snippets.

Google for Developers+1

What Is a Meta Robots Tag?

A meta robots tag is written inside the <head> section of an HTML page.

For example:

<meta name="robots" content="noindex, nofollow">

This tells search engines not to show the page in search results and not to follow links on that page. Google supports these instructions as page-level controls.

Google for Developers

The robots name applies to search engine crawlers generally. You can also use googlebot when you want to specify instructions for Google’s crawler.

Google for Developers

Why Is the Meta Robots Tag Important for SEO?

The meta robots tag helps website owners manage how search engines handle individual pages.

For example, you may have:

• A thank-you page after a form submission

• A private customer account page

• A temporary landing page

• A duplicate version of a page

• A page that should not appear in search results

Instead of allowing every page to be indexed automatically, you can use appropriate robots instructions to control how search engines treat those pages.

However, the tag should be used carefully. If you accidentally add noindex to an important page, that page may not appear in Google Search.

Google for Developers

Common Meta Robots Directives

1. index

This allows a page to be indexed and potentially shown in search results.

<meta name="robots" content="index">

Google's default behavior is generally index, follow, so you usually do not need to add these instructions explicitly.

Google for Developers

2. noindex

This tells search engines not to show the page in search results.

<meta name="robots" content="noindex">

This can be useful for pages that should remain accessible to visitors but should not appear in search results.

Google for Developers

3. follow

This allows search engines to follow links on the page.

<meta name="robots" content="follow">

4. nofollow

This instructs search engines not to follow links on the page.

<meta name="robots" content="nofollow">

Google explains that nofollow is a link-following instruction, while noindex controls whether the page is shown in search results.

Google for Developers

5. noindex, nofollow

This combines both instructions.

<meta name="robots" content="noindex, nofollow">

It is commonly used when a page should not appear in search results and its links should not be followed.

Google for Developers

Other Useful Robots Directives

Meta robots tags can also control how much content search engines show in previews.

For example:

• nosnippet can prevent a text snippet or video preview from appearing.

• max-snippet can limit the number of characters in a text snippet.

• max-image-preview can control the maximum size of image previews.

• max-video-preview can limit the length of video previews.

• noimageindex can prevent images on a page from being indexed.

  

  Google for Developers

These options can be useful when you want more control over how your content appears in search results.

Meta Robots vs Robots.txt

Many beginners confuse the meta robots tag with the robots.txt file.

Robots.txt controls whether crawlers are allowed to access certain URLs.

Meta robots provides instructions about indexing and displaying a page after the crawler accesses it.

For example, if a page is blocked by robots.txt, Google may not be able to discover its meta robots instructions. Therefore, blocking a page from crawling can prevent those indexing rules from being read.

Google for Developers

How to Use Meta Robots Correctly

Before adding a robots tag, decide what you want search engines to do with the page.

For important pages such as your homepage, service pages, and blog posts, you generally want them to remain available for indexing.

For pages that should not appear in search results, use an appropriate directive such as noindex.

If you use WordPress, Wix, or another CMS, you may be able to manage these settings through the platform’s SEO tools instead of editing HTML manually.

Google for Developers

Conclusion

The meta robots tag is a useful SEO tool that helps website owners control how search engines index and display individual pages.

By understanding directives such as index, noindex, follow, and nofollow, you can make better decisions about which pages should appear in search results.

The key is to use robots instructions carefully and check your important pages regularly. A small SEO setting can have a big impact on how your website is discovered online.