Robots.txt Generator
Create a valid robots.txt to control crawler access.
Use * for all crawlers, or a name like Googlebot.
User-agent: *
Disallow:A robots.txt file tells search engine crawlers which parts of your site they may or may not request. This generator builds a valid file from simple inputs — user-agent, disallow and allow rules, an optional crawl-delay, and your sitemap URL — ready to drop at your domain root.
How the Robots.txt Generator works
Choose which crawler the rules apply to (use * for all), list the paths to block, and add your sitemap URL. The tool assembles a correctly formatted robots.txt. Save it as robots.txt in your site's root so it's reachable at yourdomain.com/robots.txt. Remember: robots.txt controls crawling, not indexing — use a noindex tag to keep a page out of search results.
Example
You block /admin and /cart and add your sitemap.
A valid robots.txt with two Disallow lines and a Sitemap entry.
Frequently asked questions
Related tools
Put the metrics that matter on autopilot
Lunroo is a team of AI agents that research, write, and publish SEO content that moves numbers like these — with a human approval step, so nothing ships without your say-so.