Hi there
Can anyone offer me some assistance? I am trying to use Google Search Console but keep getting the following message for my pages:
Page cannot be crawled: Blocked by robots.txt
I cann’t even see a robots.txt file on my website. Help!
Okay can you provide your store URL?
@lnrjordan Are you there?
Thanks for your reply. My store is rainbow-regalia.com
Aha! It’s full of Disallows!!!
There does not seem to be robots.txt.liquid. How else can I see what is blocking pages?
DO NOT MESS WITH ROBOTS.TXT
No matter what anyone says to try to convince you:
This file is automatically set up by Shopify and it is disallowed on everything that IS SUPPOSED TO BE DISALLOWED. Understand what you’re looking at. Don’t listen to anyone who says you need to allow all this stuff.
The pages that are blocked are blocked for a reason.
DON’T MESS WITH IT UNLESS YOU KNOW EXACTLY WHAT YOU’RE DOING!!!
You have been warned. And for goodness sake, please use the search. There are hundreds of posts for you to read about it.
Fair enough but it is blocking a lot of pages so any idea what I should be doing?
Don’t do anything. They’re blocked for a good reason. You’re good unless you tell is that a specific collection or your about us page is blocked or something. But pages that are meant to be blocked need to stay blocked. This is the way. Again, understand what you’re looking at. Customer accounts. Blocked. Checkout. Blocked. Search pages. Blocked. There is no reason to touch these.
Okay I have checked your store but have you got it solve or you still need help for me to guide you step by step to undo the robot.txt @lnrjordan
I always tell my clients to add the following:
User-agent: Googlebot
Disallow: /admin/
Allow: /
User-agent: Googlebot-image
Disallow: /admin/
Allow: /
User-agent: Storebot-Google
Disallow: /admin/
Allow: /
Because it is not added by default.
Shopify auto-generates robots.txt dynamically - you won’t find a file you can just open and read in your theme files unless you’re on a 2.0 theme and have specifically added a robots.txt.liquid template. What you’re seeing at yourstore.com/robots.txt is Shopify’s default output, and yes, it has a lot of Disallow rules. That’s by design.
Maximus3 is right that most of those blocks are intentional - checkout, cart, account pages, search result pages with filter parameters, etc. Google doesn’t need to crawl those, and letting it do so just creates duplicate content noise. So a long list of Disallows isn’t itself a problem.
The real question is: are any of your actual public pages being blocked? Things like collection pages, product pages, blog posts, your About or Contact page. Those should never be in the Disallow list. To check, go into Google Search Console, click on one of the flagged URLs, and use the URL Inspection tool to see exactly what’s being blocked and why. That’ll tell you immediately whether it’s a real issue or just expected behavior showing up as a warning.
On EmmanuelFlossie’s suggestion - I’d be careful with that approach. Broadly allowing everything except /admin/ would open up checkout, cart, and account pages to crawling, which Shopify intentionally blocks for good reason. It’s not something I’d recommend unless you’ve confirmed a specific legitimate page is being restricted and you understand the impact of the change.
If after inspecting your actual collection and product URLs in GSC they come back fine, then there’s nothing to fix here. The ““blocked”” count in the Coverage report looks alarming but in most cases reflects the expected Shopify defaults.