robots.txt
A file at the root of your site that tells search engine crawlers which addresses they may crawl. It manages crawler traffic; it doesn't keep a page out of Google.
Google says a robots.txt file is used mainly to avoid overloading your site with requests from crawlers, or to keep them away from unimportant or similar pages (Google Search Central).
Keeping a page out of Google
If other pages link to a page you’ve blocked, Google can still show its address in search results, just without a description. To keep a page out of Google Search, Google points you to a noindex rule or password protection instead.
Images, videos and audio files are different: robots.txt can keep them out of Google Search results, though people can still link to them.
On a store or website platform
If you use a platform such as Wix or Blogger, you might not be able to edit the file yourself. Google suggests looking for the platform’s search settings page to hide or show a page in search engines.
Related terms
Tools for it
Sources
- https://developers.google.com/search/docs/crawling-indexing/robots/intro (Oct 7, 2026)
- https://developers.google.com/crawling/docs/robots-txt/create-robots-txt (Oct 11, 2026)
Updated Oct 7, 2026.