The X-Robots-Tag is an HTTP header that is used to control how search engines crawl and index a webpage, similar to the meta robots tag which is used in the HTML. It can be used to specify which pages or sections of a website should be indexed, and which should be ignored by search engines.
The X-Robots-Tag uses the same values as the meta robots tag: “index” to allow a page to be indexed, “noindex” to prevent a page from being indexed, “follow” to allow search engines to follow links on the page, and “nofollow” to prevent search engines from following links on the page.
The syntax for the X-Robots-Tag HTTP header is as follows:
X-Robots-Tag: [value]
For example, the following X-Robots-Tag HTTP header tells search engines not to index a page, but to follow its links:
X-Robots-Tag: noindex, follow
It is important to note that the X-Robots-Tag is an HTTP header and it is used by search engines that support it, such as Google, Bing, and Yahoo. Additionally, the X-Robots-Tag can be overridden by the robots.txt file, which is a file that can be used to specify which pages or sections of a website should be crawled by search engines.
Also, See: XML Sitemap | HTML Sitemap