noindex and meta robots explained: when to use them and how to remove them

What noindex does, the difference between meta robots and X-Robots-Tag, why staging leaks kill rankings, and how to check a live page.

By Sumit JhaUpdated November 13, 20268 min readFor: Developers and SEO beginners fighting “discovered but not indexed”

Short answer

A noindex directive tells search engines not to show the page in results. It can come from a meta robots tag in HTML or an X-Robots-Tag HTTP header. Use noindex for thank-you pages, private tools, and thin duplicates—not for pages you want to rank. If a public page is stuck out of Google, check for leftover noindex from staging, CMS defaults, or headers first.

If Google will not index a page, check for noindex first with the website checker or meta tag checker, then follow the indexing troubleshooting guide.

What noindex means

noindex is a robots directive: “you may crawl this URL, but do not show it in search results.” It is appropriate for pages that should stay out of Google—not for your homepage, pricing, or blog posts you want to rank.

Related directives you will see beside it: nofollow (do not treat links as endorsement), noarchive, nosnippet. For most marketing sites, the dangerous one is leftover noindex.

Meta robots vs X-Robots-Tag

Two common places to set it:

<!-- In HTML <head> -->
<meta name="robots" content="noindex, nofollow" />

<!-- Or HTTP response header -->
X-Robots-Tag: noindex
  • Meta robots lives in the HTML. Easy to leave on from a staging theme or SEO plugin “discourage search engines” toggle.
  • X-Robots-Tag lives on the HTTP response. Useful for PDFs and non-HTML files; also easy to miss if you only view-source the body.

Google also understands googlebot-specific meta tags. If any applicable rule says noindex, treat the page as noindexed until proven otherwise.

When to use noindex

  • Thank-you / confirmation pages after forms or checkout.
  • Internal search result URLs and thin filter combinations.
  • Staging or preview deployments that are somehow publicly reachable.
  • Duplicate print views or parameter pages you cannot canonicalize cleanly.

Do not noindex a page just because it ranks poorly—fix the content and links instead.

How to find and remove it

  1. Fetch the live URL (not localhost). Confirm meta name="robots" and response headers.
  2. In WordPress, check Settings → Reading (“Discourage search engines”) and SEO plugins (Yoast / Rank Math page-level robots).
  3. In Next.js and similar frameworks, search for robots: { index: false } in metadata or layout defaults.
  4. Remove noindex, deploy, re-check with Shipcheck, then use Search Console → URL Inspection → Request indexing.

noindex vs canonical vs robots.txt

ToolUse when
noindexPage must stay out of results but can still be linked or visited.
CanonicalSeveral URLs show similar content; you want one preferred URL to rank.
robots.txt DisallowYou want to reduce crawling of whole sections (admin, APIs)—not a substitute for noindex.
301 redirectThe URL permanently moved; send users and signals to the new address.

Questions

What does noindex do?

It asks search engines not to include the page in search results. Crawlers may still fetch the URL; they just should not show it in Google or Bing results.

Is noindex the same as blocking in robots.txt?

No. robots.txt can stop crawling of a path. noindex is usually discovered after the page is fetched. If robots.txt blocks the page, engines may never see a noindex tag on it.

How do I remove noindex?

Delete meta name="robots" content that includes noindex, and check response headers for X-Robots-Tag: noindex. Publish, then use Search Console URL Inspection and request indexing if needed.

Should I noindex my whole site while building?

Prefer a password, staging host, or auth gate. Sitewide noindex on production is a common launch disaster when someone forgets to turn it off.

Tools for this guide