The meta robots tag is an HTML tag that controls how search engines crawl and index a web page. It can specify which pages or sections of a website should be indexed and which should be ignored.
The meta robots tag is placed within the <head> section of an HTML document and uses the following syntax:
<meta name=”robots” content=”value”>
The “value” in the content attribute can be one or more of the following:
- “index”: Tells search engines to index the page.
- “noindex”: Tells search engines not to index the page.
- “follow”: Tells search engines to follow the links on the page.
- “nofollow”: Tells search engines not to follow the links on the page.
For example, the following meta robots tag tells search engines not to index a page but to follow its links:
<meta name=”robots” content=”noindex, follow”>
It’s important to note that the meta robots tag only applies to the search engines that support it, such as Google, Bing, and Yahoo.
Additionally, the meta robots tag can be overridden by the robots.txt file, which is a file that can be used to specify which pages or sections of a website should be crawled by search engines.
Also, See: Robots.txt In SEO