Robots.txt is a text file within your website’s top-level directory that instructs search engines how to crawl your pages.