Loading…
Create a robots.txt file for your website.
Generate a correct robots.txt for your website in seconds. Choose whether to allow all crawlers or block the whole site, list the paths you want to keep out of search, add an optional crawl-delay, and include your sitemap URL. Copy the result into a robots.txt file at your site's root.
* means all crawlers.
User-agent: *
Disallow: /admin/
Disallow: /dashboard/
Disallow: /api/
Sitemap: https://example.com/sitemap.xml
Save this as robots.txt in your site's root.
Runs entirely in your browser — nothing is uploaded.
It's a text file at your site's root (yoursite.com/robots.txt) that tells search-engine crawlers which pages or folders they may or may not request. It guides crawling — it is not a security or privacy control.
In the root of your domain, so it's reachable at https://yoursite.com/robots.txt. It won't work in a subfolder.
Not reliably — Disallow asks crawlers not to fetch a page, but a blocked URL can still be indexed if linked elsewhere. To keep a page out of results, use a noindex meta tag or header instead.
Yes — adding a Sitemap: line helps search engines discover all your pages, which is good for indexing.