SEO Tools

Robots.txt Generator

Create crawler instructions from explicit rules, with safe defaults and a live text preview.

robots.txt controls crawling, not guaranteed indexing. Blocking a URL can prevent a crawler from seeing its noindex directive; use authenticated access to protect private content.

Start with a preset

User-agent group 1Primary user-agentUser-agent tokens (comma separated, supports multiple agents)

Sitemap URL (optional)Crawl-delay in seconds (optional)

Live robots.txt

Valid generator input. Review your rules before publishing.

robots.txt preview

User-agent: *
Allow: /

Crawl-delay is not supported consistently by all search engines; Googlebot ignores it. Robots.txt is not access control and does not guarantee removal from search results.

Generation is local and immediate. Upload robots.txt at your website root after reviewing your intended crawler access.