What robots.txt controls - and what it doesn't
A robots.txt file tells well-behaved search engine crawlers which parts of your site they're allowed to
request. That's all it does. It is a crawling instruction, not an indexing or ranking control - blocking a URL
in robots.txt does not reliably remove it from search results, because a page can still get indexed (sometimes
with no snippet) if other sites link to it. If your actual goal is to keep a specific page out of Google
entirely, you need a noindex meta tag or HTTP header on that page instead, not a robots.txt rule.
How this generator builds the file
Choose "Allow all robots" (the sensible default for almost every site) and optionally list specific paths to block underneath - admin areas, cart pages, internal search result URLs, or anything else you don't want crawled. Choose "Disallow all robots" only if you genuinely want to block every crawler from the entire site, which is rare and usually only appropriate for a staging environment. Add your sitemap URL and it's appended automatically, which helps crawlers discover your sitemap without needing to find it independently.
Where the finished file has to live
robots.txt only works from one exact location: the root of your domain, reachable at
https://example.com/robots.txt. A copy placed in a subfolder is simply ignored by every major
crawler - there is no alternate location that works.