robots.txt and sitemap.xml on Shopify: What the Platform Gives You and What It Gets Wrong

Two files you do not own, and one you can

Shopify robots.txt and sitemap.xml are both generated by the platform. One of them you can edit. The other you genuinely cannot, and most of the advice online pretends otherwise.

Knowing which is which saves you a lot of wasted effort.

robots.txt: editable, via a template most themes do not have

Shopify generates a default robots.txt that is fine for most stores. If you want to change it, you add a robots.txt.liquid template — and it will not be there, because it ships in no theme by default. You create it in the theme code editor, in the templates folder, named exactly robots.txt.liquid.

Two constraints worth knowing before you start:

• It cannot be a JSON template. It has to be robots.txt.liquid.
• It supports only six Liquid objects: robots, group, rule, user_agent, sitemap, and request. No product, no collection, no settings. This is a text file with a very small window into the store.

The whole default file is mirrored through the robots object, and the base template is a loop:

{% for group in robots.default_groups %}
  {{- group.user_agent -}}
  {% for rule in group.rules %}
    {{- rule -}}
  {% endfor %}
  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

Keep the loop. You can replace the entire template with plain text rules, and Shopify strongly recommends against it, for a reason that is easy to underrate: the default rules are updated regularly to keep SEO best practices applied. Hardcode them and you have frozen a snapshot of what was correct on the day you wrote it, and you will never notice the drift.

### The three edits worth making

Add a rule to an existing group. Blocking internal search results is the most common one — check for the group and inject:

{%- if group.user_agent.value == '*' -%}
  {{ 'Disallow: /*?q=*' }}
{%- endif -%}

Remove a default rule. Shopify blocks /policies/ by default. If you want your policy pages crawlable — and on a store where returns and shipping terms are a real buying objection, you might — skip that rule while emitting the others:

{%- unless rule.directive == 'Disallow' and rule.value == '/policies/' -%}
  {{ rule }}
{%- endunless -%}

Add rules for crawlers that are not in the default set. This is where you allow or block AI crawlers, and it goes outside the default-groups loop as plain text:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot Allow: / ```

There is also a host-specific pattern using the request object, which only makes sense if you run Shopify Markets with distinct domains per market:

{%- if request.host == 'example.fr' -%}
  {{ 'Disallow: /en/' }}
{%- endif -%}

That lets a French domain block English content while everything else keeps the defaults. If you do not run multiple market domains, you do not need this and adding it is a way to break something later.

### The thing people get wrong about Disallow

Disallow stops crawling. It does not stop indexing, and it actively prevents the crawler from seeing anything on the page — including your canonical tag and your noindex meta tag.

So blocking a URL you wanted consolidated is counterproductive: the crawler can no longer read the canonical that would have consolidated it. Use robots.txt to save crawl budget on URLs with no value at all. Use canonicals for duplicates. They are not interchangeable and people use the first when they need the second constantly.

sitemap.xml: generated, and you cannot edit it

Shopify serves a sitemap index at /sitemap.xml, and it links out to a set of child sitemaps. On a live production store today, that index looks like this:

/sitemap_products_1.xml?from=1878194389061&to=7369944137808
/sitemap_collections_1.xml?from=423410055&to=279444029520
/sitemap_pages_1.xml?from=248230535&to=94637621328
/sitemap_blogs_1.xml
/sitemap_metaobject_pages_1.xml
/sitemap_agentic_discovery.xml

Worth noticing, because most write-ups on this are years old:

• The child sitemaps are numbered and paginated by resource ID range, which is what the from and to parameters are. Big catalogues get _2, _3 and so on. If you have ever wondered why a product is "missing from the sitemap," check the later numbered files before concluding anything.
• Metaobject pages have their own sitemap. If you have built content on metaobjects, it is being submitted, which surprises people who assumed metaobjects were invisible to search.
• `sitemap_agentic_discovery.xml` is new, and it is part of the same agent-discovery layer that produces /agents.md, /llms.txt and /llms-full.txt. Worth reading once, because it tells you what Shopify thinks an agent should be pointed at.

You cannot add a URL, remove a URL, reorder, set priorities, or change change-frequencies. There is no setting and no template. This is the part where a lot of migration checklists ask for something impossible.

### The two levers that do remove a URL

Since you cannot edit the file, you change what qualifies for it:

• The `seo.hidden` metafield. Set the value to 1 on a page, blog post or product and Shopify removes it from sitemaps, from search engines, and from your online store's internal search. One metafield, three effects.
• Unlisted product status. For products specifically, Unlisted keeps the product reachable by direct link while removing it from sitemaps, collection pages and store search. This is the right status for a product that exists for a specific customer, a warranty replacement part, or a channel-only SKU.

Both are better than a noindex conditional in theme.liquid, because they fix the sitemap too. A noindexed URL that is still listed in your sitemap is a contradiction — you are submitting a page and asking for it not to be indexed, and you burn crawl budget on it every cycle.

If you inherited a store with noindex conditionals in the theme, audit them against the sitemap. I usually find the theme is noindexing things the sitemap is still submitting, and nobody has looked in years.

The failure that costs the most, and does not warn you

Everything above assumes the URLs in your sitemap resolve.

I broke this on my own site. The site is served on the www host and the bare host 301s to it. My sitemap, my canonicals and my agent-discovery files all pointed at the bare host. So every URL in the sitemap redirected, and every canonical named a URL that redirected.

Nothing errored. The site looked perfect. site: search returned four indexed URLs out of a few dozen, and every service page was outside the top 100 on keywords with a difficulty of zero. I spent time assuming it was a content and authority problem before checking the mechanical one.

So: after any domain change, any migration, any CDN change, take three URLs out of your sitemap, curl them, and confirm they return 200 and not 301. It takes a minute and it is the highest-value minute in technical SEO.

The same check applies to the primary domain more broadly. Shopify serves the sitemap on the primary domain, and stores that have collected extra domains over the years frequently have the sitemap submitted for the wrong one in Search Console.

The audit

• Open /robots.txt. Read it. If it has been customised, open templates/robots.txt.liquid and confirm it still loops robots.default_groups rather than hardcoding a frozen rule set.
• Open /sitemap.xml and read the index. Note the numbered files and follow at least the last one.
• Curl three URLs from a child sitemap. Confirm 200, not 301.
• Cross-check: any URL you are noindexing in the theme should be handled with seo.hidden or Unlisted instead, so it leaves the sitemap too.
• Confirm Search Console has the sitemap submitted on the host you actually serve.

Neither of these files will win you rankings. Both of them can quietly cost you the entire index, and one of them cost me exactly that.

Related services

Crawl and index mechanics are part of Shopify SEO expert work; the theme-level implementation is Shopify expert development. If the damage came from a replatform, Shopify migration expert covers URL inventories and redirects. Related reading: Shopify canonical URLs and the duplicate content trap and llms.txt, agents.md and being citable by AI.

I audit and fix Shopify crawl and index configuration on stores where the traffic is already paid for. See [Shopify SEO expert services](/shopify-seo-expert) or [hire a Shopify developer](/hire-shopify-developer).

Direct: [WhatsApp +55 11 98851-2788](https://wa.me/5511988512788) · [contato.matheusabrahao@gmail.com](mailto:contato.matheusabrahao@gmail.com)