Skip to content
ROI Synth

Noindex with a robots meta tag or the X-Robots-Tag header: which to use and how

By ROI SynthPublished 6 min read

Both tell search engines not to index a URL, and both accept the same rules. Use the robots meta tag for ordinary HTML pages, because it lives with the page. Use the X-Robots-Tag HTTP header for files that have no HTML head, such as PDFs and images, or when one server rule must cover many URLs at once, such as a whole staging domain. Either way, the URL must not be blocked in robots.txt, or crawlers never see the instruction.

On this page
  1. Two ways to send the same instruction
  2. Which one to use, case by case
  3. Why robots.txt can cancel your noindex
  4. Setting noindex on a Next.js site
  5. How to check what search engines receive
  6. Five mistakes that cause most noindex problems
  7. Questions
  8. Sources

Noindex is one of the few search controls that does exactly what it says, and one of the easiest to break by accident. Most noindex problems we see are not about choosing the wrong method. They come from a robots.txt rule that hides the directive, a staging site that went live without anyone removing it, or a header set somewhere nobody thought to look.

This guide explains the two methods, when each one fits, how to implement them on a Next.js site, and how to check what search engines actually receive.

Two ways to send the same instruction

Google's documentation treats the two as equivalent in what they can say: any rule that works in a robots meta tag can also be sent as an X-Robots-Tag header. The difference is where the instruction travels.

Robots meta tag and X-Robots-Tag header compared
Robots meta tagX-Robots-Tag header
Where it livesIn the HTML, usually in the head: <meta name="robots" content="noindex">In the HTTP response headers: X-Robots-Tag: noindex
Works forHTML pages onlyAny response: HTML, PDF, images, video, feeds
Set byThe page or its templateThe server, CDN, framework config or middleware
Best forIndividual pages and page typesNon-HTML files and whole hosts or path patterns
Easy to seeView source or rendered HTMLResponse headers (browser dev tools or curl -I)

If a URL receives both, and they disagree, Google applies the more restrictive rule. A page with index in the meta tag and noindex in the header is not indexed.

Which one to use, case by case

The right choice depends less on preference than on what kind of response you are controlling.

  • Thin or private HTML pages

    Thank-you pages, internal search results, account pages: use the meta tag, set by the page template so every page of that type is covered.

  • PDFs, images and other files

    There is no HTML head to put a tag in, so the X-Robots-Tag header is the only option.

  • A staging or preview domain

    Use a header rule that matches the host, so every response from that domain says noindex, including files and error pages, without touching the page code.

  • Pages that should not even be crawled

    Noindex is the wrong tool on its own. Use authentication for private content; robots.txt controls crawling but is not a privacy mechanism.

Why robots.txt can cancel your noindex

Robots meta tags and X-Robots-Tag headers are only discovered when a URL is crawled. Google's documentation is explicit: if a page is disallowed in robots.txt, its indexing rules are never seen and are therefore ignored. A disallowed URL can still be indexed from links pointing to it, typically shown without a description.

So the sequence matters. To remove a page from the index, allow crawling and send noindex. Only once it has dropped out should you consider blocking crawling, and for most pages there is no need to.

Setting noindex on a Next.js site

In the App Router, page-level noindex belongs in the page's metadata. Exporting robots: { index: false, follow: true } from a page or layout renders the robots meta tag in the HTML, so it is visible in the first response rather than added later by JavaScript.

For headers, the headers() function in next.config can attach X-Robots-Tag to a path pattern, and it can match on the request host. That is how this site keeps its own preview deployments out of search: every response on a host other than the canonical domain carries X-Robots-Tag: noindex, while the production domain does not. Middleware can do the same when the rule depends on something only known at request time.

Whichever you use, set the rule once in shared code rather than page by page. A rule in a template or config is reviewed like any other change and cannot be forgotten on the next new page.

How to check what search engines receive

Check the response itself, not the code you expect to produce it. CDNs, hosting platforms and plugins can all add or strip headers.

  • Request the URL with curl -I (or the Network tab in browser developer tools) and read the response headers for X-Robots-Tag.
  • View the source of the HTML response, not just the rendered page, and look for the robots meta tag.
  • Use URL Inspection in Search Console: the live test shows whether Google sees the page as indexable and why.
  • Check robots.txt for a Disallow rule that would hide either instruction.
  • After launch, confirm that no production page still carries a noindex left over from staging.

Five mistakes that cause most noindex problems

None of these are exotic. They come from building fast and from rules that live in places nobody reviews.

  • Noindex hidden by robots.txt

    The URL is disallowed, so the instruction is never read. Allow crawling until it drops out.

  • Staging noindex shipped to production

    A site-wide rule copied from staging deindexes the live site. Make the rule depend on the host, not on a setting someone must remember to change.

  • Confusing nofollow with noindex

    Nofollow is about links; it does not stop a page being indexed.

  • Noindexed pages in the sitemap

    The sitemap asks for indexing while the page refuses it. Keep the two consistent.

  • Expecting instant removal

    Noindex takes effect when the page is recrawled. For urgent removals, Search Console's removals tool hides a URL temporarily while noindex does the permanent work.

If indexing on your site is not doing what you expect, our indexation management service audits every signal together, and the free website audit shows the robots meta tag, X-Robots-Tag and robots.txt status for any single page.

Questions people also ask

Is X-Robots-Tag better than the robots meta tag?

Neither is better; they carry the same rules. Use the meta tag for HTML pages and the header for files without HTML or for rules that cover a whole host or path pattern.

Can I put noindex in robots.txt?

No. Google does not support a noindex rule in robots.txt. Use a robots meta tag or an X-Robots-Tag header, and make sure the URL is not disallowed.

Does noindex stop Google crawling the page?

Not entirely. Google still needs to crawl the page to see the noindex, and it may crawl it less often afterwards. To stop crawling, use robots.txt, but only once the page is out of the index.

How long does noindex take to work?

It applies the next time the URL is crawled, which can be days or weeks depending on the page. URL Inspection can request a recrawl of individual URLs.

Can I target noindex at one search engine?

Yes. Both forms accept a crawler name, such as googlebot: noindex in the header or name="googlebot" in the meta tag. Rules without a crawler name apply to all crawlers that support them.

Where ROI Synth can help

  • Indexation management: Noindex, canonicals, sitemaps and robots rules aligned so the index matches your intentions.
  • Technical SEO: Indexing controls implemented in your templates and config, then verified on the live site.
  • Free website audit: See the robots meta tag, X-Robots-Tag header and robots.txt status for one page.

Sources

Search engines change their documentation and features. Facts were checked against these sources on 14 November 2026.

Talk to ROI Synth about your website

Share where your site stands and what you want more of. We will tell you plainly which work is worth doing first, and what it would cost.