Practical templates
Review the risk and platform notes before replacing your current work. Templates are starting points, not universally safe defaults.
Free Robots.txt Generator & Tester
Drop a robots.txt file here
or choose a UTF-8 plain-text file (maximum 1 MiB)
Use only the scheme and host, without a path.
Scoped parameter suggestions
Choose only parameters whose URLs you have reviewed. Blocking every parameterized URL can hide important faceted, paginated, or canonicalized pages from crawlers.
Global directives
Advanced and imported directives
Host is non-standard. Unknown imported lines are preserved, but may be ignored by crawlers.
Interactive matching examples
Each focused example includes representative outcomes and a button to load it into the builder.
Directive support and crawler semantics
Specific groups do not inherit wildcard rules. If User-agent: Bingbot matches, that crawler ignores the User-agent: * group. Repeat every generic rule that should also apply to the specific crawler.
| Feature | RFC 9309 | Bing | Practical guidance | |
|---|---|---|---|---|
User-agent | Standard | Supported | Supported | Use a valid product token: letters, underscore, or hyphen; * is the wildcard group. |
Allow | Standard | Supported | Supported | The longest matching path wins; Allow wins an equal-length tie. |
Disallow | Standard | Supported | Supported | An empty value does not block anything. / matches the entire site. |
Sitemap | Outside the core rule grammar | Supported | Supported | Use an absolute URL. A cross-host sitemap can be intentional but requires ownership verification. |
Crawl-delay | Non-standard | Ignored | Supported | Advanced only. Bing warns that large values reduce crawl freshness. |
Host | Non-standard | Ignored | Not documented as a core directive | Advanced only; prefer canonical redirects and consistent sitemap URLs. |
* in a path | Not required by the RFC | Supported extension | Supported extension | Matches zero or more characters. Test crawler-specific behavior. |
$ end anchor | Not required by the RFC | Supported extension | Supported extension | Matches only at the end of the URL path/query string. |
How to create and deploy a robots.txt file
- Set the site and crawler groups. Enter the exact site origin, then create a wildcard or crawler-specific
User-agentgroup. - Add and validate rules. Add
AllowandDisallowpaths, resolve validation findings, and test representative URLs. - Review safety warnings. Check that important pages, CSS, JavaScript, and images remain crawlable; confirm any intentional site-wide block.
- Download and upload the file. Download
robots.txtand upload it as UTF-8 plain text to/robots.txton the exact scheme and host it controls. - Verify the deployed file. Confirm the HTTP response, review the Search Console robots.txt report, and inspect representative URLs after crawler caches refresh.
Deployment details that prevent surprises
- The file applies only to its exact protocol, host, and port.
https://example.com,http://example.com, andhttps://shop.example.comare separate origins. - Serve a successful response as UTF-8 plain text. Avoid redirect chains; crawlers may impose redirect limits.
- A missing file or most
4xxresponses generally mean no restrictions.429and5xxresponses can temporarily stop or reduce crawling, so monitor availability. - Crawlers cache robots.txt, so changes may take hours to be observed. Google Search Console’s robots.txt report can show fetch and parse issues and request an emergency recrawl.
- After deployment, open the exact file URL and use URL Inspection on a representative page. Confirm “Crawl allowed?” and inspect loaded resources when CSS or JavaScript rules changed.
Robots.txt FAQ
Does every website need a robots.txt file?
No. If every crawler may fetch every public URL, a missing robots.txt normally means there are no crawl restrictions. A file is useful when you need deliberate crawl controls or want to declare sitemap locations.
Where must robots.txt be uploaded?
Upload it as /robots.txt at the root of the exact scheme, host, and port it controls, such as https://example.com/robots.txt. Subdomains and other protocols need their own files.
What is the difference between robots.txt and noindex?
robots.txt controls crawling. A noindex robots meta tag or X-Robots-Tag controls indexing, but the crawler must be allowed to fetch the URL to see that instruction.
Why can a blocked URL still appear in search results?
Search engines may discover a URL from links and index the URL without crawling its content. Use an accessible noindex directive or authentication when exclusion from results or privacy is required.
Does robots.txt protect private data?
No. The file is public and compliant crawlers may choose whether to follow it. Protect private content with authentication and appropriate 401 or 403 responses.
How do robots.txt wildcards work?
An asterisk (*) matches zero or more characters and a dollar sign ($) anchors a match to the end of the URL path. These extensions are supported by major search engines but should be tested for each crawler.
Does Google support Crawl-delay?
No. Google documents User-agent, Allow, Disallow, and Sitemap support and ignores Crawl-delay. Bing supports Crawl-delay, but it is not part of RFC 9309.
Does a crawler-specific group inherit the wildcard group?
No. When a crawler-specific group matches, the User-agent: * rules are not inherited. Repeat every generic rule that should also apply to that crawler.
How do I verify robots.txt after deployment?
Open the exact /robots.txt URL, confirm a successful plain-text UTF-8 response, review the Search Console robots.txt report, and use URL Inspection on representative URLs.
Standards and review
Reviewed by Starlight Tools / Starlight Robotics on 16 July 2026 against RFC 9309, Google’s robots.txt specification, Bing Webmaster guidance, and the Search Console robots.txt report.
User-agent, Allow, and Disallow are standardized by RFC 9309. Sitemap, wildcards, and the end anchor are crawler extensions; Crawl-delay and Host are non-standard and support varies. This browser-based validator models documented matching rules, but only the target crawler can provide a definitive interpretation. A valid file does not guarantee indexing or higher rankings.