robots.txt
Rules0
sitemap.xml
URLs0
How the robots.txt and sitemap generator works
robots.txt tells search engines what they may read. sitemap.xml gives them the list of pages you want indexed. You fill in the site address, the rules and the URLs, then download the two files into the domain root (example.com/robots.txt and example.com/sitemap.xml).
You can write relative paths (/about) or full URLs. Lastmod, changefreq and priority apply to every row, unless you set them on the line: url | lastmod | changefreq | priority. Everything stays in the browser.
robots.txt
Allow and Disallow
Control which folders can be read, plus a link to the sitemap.
sitemap.xml
Page list
Each URL gets lastmod, frequency and priority, ready for Google Search Console.
SEO
No server publish
You generate the text here, then upload it yourself. Nothing is published automatically.
Important:
robots.txt is not a password. A Disallow does not hide the page from people; it only asks bots not to read it. Private pages still need authentication.
Useful notes
- Put both files in the domain root, not in a subfolder, so Google can find them at /robots.txt and /sitemap.xml.
- In Search Console you can submit the sitemap after it is live. Check that each <loc> returns 200.
- Crawl-delay is not honoured by Google. It is useful for other, more polite bots.
- You can add on one line: /blog | 2026-10-19 | weekly | 0.8 to override the defaults.
- Nothing is saved on the server. If you reload the page, the fields are cleared.