Robots.txt
The robots.txt is a publicly readable text file in the root directory of a website that tells well-behaved crawlers which areas they may visit and which are excluded from crawling.
In practice
A robots.txt prevents crawling but not necessarily indexing – blocked pages that are already linked can still turn up in search results, only without a content preview. To keep individual pages out of the index reliably, a meta robots tag with “noindex” is the right tool instead.