GuidesTechnical SEO
Tutorial: writing a robots.txt and a sitemap
What each file does
- The
robots.txttells crawlers which parts of the site not to explore. - The sitemap gives them the list of pages you want explored.
1. Write the robots.txt
Create a text file named robots.txt, served at the root: https://example.com/robots.txt.
User-agent: *
Disallow: /admin/
Disallow: /cart/
Sitemap: https://example.com/sitemap.xml
User-agent: * addresses all crawlers. Each Disallow line closes a folder. With no Disallow line at all, everything is open.
2. Write the sitemap
Most tools generate it on their own: a plugin on WordPress, an integration on Astro or Next.js, and Shopify does it by default. By hand, it looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/</loc>
<lastmod>2026-11-11</lastmod>
</url>
</urlset>
Only put final addresses in it: no redirects, no error pages, no noindex pages.
3. Link the two
The Sitemap: line of the robots.txt is enough for crawlers to find it. Also submit it in Google Search Console, to see how many pages are read: this is covered in the Search Console tutorial.
4. Check
Open both addresses in a browser. They must display, with no redirect and no error.
The mistake that makes a site vanish
Disallow: / closes the whole site to crawlers. That line is used during development and sometimes stays at launch. Check it on launch day.
Removing a page from Google
The robots.txt removes nothing. To remove a page, leave it readable by crawlers and add this tag to its <head>:
<meta name="robots" content="noindex">
For a site in several languages, read multilingual SEO.
Frequently asked questions
- Does robots.txt stop a page from appearing in Google?
- No. It stops the crawler from reading the page, not from indexing it: a blocked page can still appear if other sites link to it. To remove it, use the noindex tag and leave the page readable.
- Is a sitemap mandatory?
- No. A small site that is well linked internally gets discovered without one. It becomes useful as soon as the site grows or changes often.
- Where do these files go?
- The robots.txt always sits at the root of the domain. The sitemap usually does too, and its address is given in the robots.txt.