How to Prevent a Website Page From Showing Up in Search Results
So you can add this tag to the page:
<meta name="robots" content="noindex" />
Or send this header for the page:
One benefit of the header approach is that you can use it for non-HTML content, like a PDF or JSON file.
noindex value tells crawlers, such as Google and Bing, not to index the
page, so it won’t show up in search results.
You might think to use the robots exclusion
robots.txt) to disallow crawling, but that doesn’t
because then the crawlers can’t see your directive to not index the page. You’ve
instructed them not to look at the page at all! So if other websites link to
your page, a crawler can still pick up and index the page.
robots.txt file is for controlling crawling, not indexing.
There are many possible directive values, and you can specify more than one by separating them with commas:
all: no restrictions (the default behavior)
noindex: exclude the page from search results
nofollow: don’t follow the links in the page
none: the same as
nocache: don’t link to a cached version of the page
nosnippet: don’t show a description, snippet, thumbnail, or video preview of the page in search results
max-snippet:[length]: limit a snippet to
[length]number of characters
max-image-preview:[setting]: set an image preview’s maximum size, where
max-video-preview:[length]: limit a video preview to
[length]number of seconds
notranslate: don’t link to a translation of the page
noimageindex: don’t index images on the page
unavailable_after:[datetime]: exclude the page from search results after
[datetime], which should be in a standard format, such as ISO 8601
If you want to use different directives based on the specific crawler, you can specify the user agent in the meta tag’s name:
<meta name="googlebot" content="noindex" /> <meta name="bingbot" content="nofollow" />
Or in the header value:
X-Robots-Tag: googlebot: noindex X-Robots-Tag: bingbot: nofollow