Back to Terminology


March 25, 2024

The X-Robots-Tag is an HTTP header that is used to control how search engines crawl and index a webpage, similar to the meta robots tag which is used in the HTML. It can be used to specify which pages or sections of a website should be indexed, and which should be ignored by search engines.

The X-Robots-Tag uses the same values as the meta robots tag: “index” to allow a page to be indexed, “noindex” to prevent a page from being indexed, “follow” to allow search engines to follow links on the page, and “nofollow” to prevent search engines from following links on the page.

The syntax for the X-Robots-Tag HTTP header is as follows:

X-Robots-Tag: [value]

For example, the following X-Robots-Tag HTTP header tells search engines not to index a page, but to follow its links:

X-Robots-Tag: noindex, follow

It is important to note that the X-Robots-Tag is an HTTP header and it is used by search engines that support it, such as Google, Bing, and Yahoo. Additionally, the X-Robots-Tag can be overridden by the robots.txt file, which is a file that can be used to specify which pages or sections of a website should be crawled by search engines.

Also, See: XML Sitemap | HTML Sitemap

Guaranteed Customer Satisfaction

We Promise. We Innovate.

Do you have an idea? Our experts are there to transform it into a recognized brand. We help you innovate your business with best-in-class solutions.
  • We will respond to you within 24 hours.
  • We’ll sign an NDA if requested.
  • You'll be talking to product and tech experts (no account managers).