I use a custom robots.txt.liquid in my theme but noticed that it is missing several disallow rules from the default robots.txt when rendered. I’m not doing anything other than replacing the hostname in. I’m following the documentation from Customize robots.txt . The primary differences are:
The “Robots & Agent policy” is missing
The Disallow: /sf_private_access_tokens rule is missing from all sections
The Disallow: /recommendations/products rule from the Ahrefs sections
The Disallow: /*/recommendations/products rule from the Ahrefs sections
It seems when I use the custom robots.txt.liquid it is missing a rule from the groups.rules:
{%- for rule in group.rules -%}
{{ rule }}
{%- endfor -%}
Thanks for the explanation, this aligns with what I was thinking.
That said, this doesn’t feel like a great long‑term solution, since there’s no reliable way for me to know when Shopify updates or adds new default robots.txt rules behind the scenes. As soon as I create a custom robots.txt.liquid, I lose visibility into those changes and have to manually guess what’s missing.
It seems more like a gap (or bug) in how Shopify exposes the default rules to Liquid. Ideally, the full default rule set would remain available so custom templates don’t silently fall out of sync. Curious if the Shopify team know about this bug and if there are any plans to address it.
I ran into the exact same problem and I’m glad to see I’m not the only one. On my store, robots.default_groups clearly returns an older rule set than the robots.txt Shopify actually serves by default.
The differences go both ways. The Liquid output is missing the agent policy header, the AI agent directives (UCP/MCP, agents.md), the Allow rules, and several recent Disallow rules. But it also includes bot groups (AhrefsBot, AhrefsSiteAudit, MJ12bot, Pinterest, Nutch) that no longer appear in the current default file, and it re-blocks /policies/ and /search, which the default file actually allows to be crawled.
I just published a dedicated post about it.
Hopefully someone from the Shopify team can fix this soon.
Same issue as the other thread on this - can’t confirm the specific object mismatch, but the safe practice either way is checking the live rendered /robots.txt, not the Liquid source, since that’s what AI crawlers (GPTBot, PerplexityBot, ClaudeBot) actually see.
@luke-p if the only change youre making is swapping the hostname, you probably dont need a custom robots.txt.liquid at all. shopify already serves the default robots.txt using your stores primary domain, so a template that just sets the host is redundant, and the moment that template exists you get pinned to whatever snapshot robots.default_groups is holding. thats the missing-rules drift you and the other thread are both seeing.
so the “keep it up to date” answer is really: dont customize unless you actually need to add or remove a rule. the platform default is the only version that stays current on its own.
if you DO need a custom rule, keep the template to the default loop plus only your own Disallow/Allow lines. then the diff between your file and the live default is exactly your additions, and nothing silently falls behind without you noticing.
and yeah, check the rendered /robots.txt rather than the liquid source, thats what the crawlers actually read.
Yeah, I’d avoid owning the whole robots file unless there’s a very specific rule you need to add. The annoying part is not today’s diff — it’s that Shopify can update crawler/agent rules later and your custom template quietly becomes the stale version. Before publishing any custom robots change, I’d save the current live /robots.txt, publish, then diff the rendered output and keep that check on a monthly calendar. This matters more now because robots/agents rules are part of AI visibility too: Shopify AI Visibility Guide (2026)