The noindex value of an HTML robots meta tag requests that automated Internet bots avoid indexing a web page. It is also a value of the HTTP response header X-Robots-Tag. Reasons why one might want to use this meta tag include advising robots not to index a very large database, web pages that are very transitory, web pages that are under development, web pages that one wishes to keep slightly more private, or the printer and mobile-friendly versions of pages. Since the burden of honoring a website's noindex tag lies with the author of the search robot, sometimes these tags are ignored. Also th
The noindex value of an HTML robots meta tag requests that automated Internet bots avoid indexing a web page. It is also a value of the HTTP response header X-Robots-Tag. Reasons why one might want to use this meta tag include advising robots not to index a very large database, web pages that are very transitory, web pages that are under development, web pages that one wishes to keep slightly more private, or the printer and mobile-friendly versions of pages. Since the burden of honoring a website's noindex tag lies with the author of the search robot, sometimes these tags are ignored. Also the interpretation of the noindex tag is sometimes slightly different from one search engine company to the next.
== Noindexing entire pages ==
Discovered by embedding cosine similarity (sentence-transformers MiniLM, 384-dim).