GuidesTechnical SEO

Tutorial: writing a robots.txt and a sitemap

By the Banzai Labs teamUpdated on 2 min read

What each file does

  • The robots.txt tells crawlers which parts of the site not to explore.
  • The sitemap gives them the list of pages you want explored.

1. Write the robots.txt

Create a text file named robots.txt, served at the root: https://example.com/robots.txt.

User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml

User-agent: * addresses all crawlers. Each Disallow line closes a folder. With no Disallow line at all, everything is open.

2. Write the sitemap

Most tools generate it on their own: a plugin on WordPress, an integration on Astro or Next.js, and Shopify does it by default. By hand, it looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-11-11</lastmod>
  </url>
</urlset>

Only put final addresses in it: no redirects, no error pages, no noindex pages.

The Sitemap: line of the robots.txt is enough for crawlers to find it. Also submit it in Google Search Console, to see how many pages are read: this is covered in the Search Console tutorial.

4. Check

Open both addresses in a browser. They must display, with no redirect and no error.

The mistake that makes a site vanish

Disallow: / closes the whole site to crawlers. That line is used during development and sometimes stays at launch. Check it on launch day.

Removing a page from Google

The robots.txt removes nothing. To remove a page, leave it readable by crawlers and add this tag to its <head>:

<meta name="robots" content="noindex">

For a site in several languages, read multilingual SEO.

Frequently asked questions

Does robots.txt stop a page from appearing in Google?
No. It stops the crawler from reading the page, not from indexing it: a blocked page can still appear if other sites link to it. To remove it, use the noindex tag and leave the page readable.
Is a sitemap mandatory?
No. A small site that is well linked internally gets discovered without one. It becomes useful as soon as the site grows or changes often.
Where do these files go?
The robots.txt always sits at the root of the domain. The sitemap usually does too, and its address is given in the robots.txt.

Read next

All guides