Robots.txt

Product Development & Sourcing
1

Hi,

I need help to check if there are any blocked pages by robot.txt. and solution if there is a serious problem.

2

Hi @mobyflex ,

The robots.txt is created by default by Shopify. So in case you have not changed any rule in the files, all rules set there are okay.

Every created store all have these same rules. You do not need to worry about it.

In case you still want me to check, you can share your store URL.

Hope this can help!

Best,

Daisy

3

we use Shopify as our ecommerce platform

User-agent: *
Disallow: /a/downloads/-/*
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /checkout
Disallow: /89883672894/checkouts
Disallow: /89883672894/orders
Disallow: /carts
Disallow: /account
Disallow: /collections/sort_by
Disallow: //collections/sort_by
Disallow: /collections/+*
Disallow: /collections/%2B
Disallow: /collections/%2b
Disallow: //collections/+*
Disallow: //collections/%2B*
Disallow: //collections/%2b*
Disallow: /collections/filter&filter
Disallow: /blogs/+*
Disallow: /blogs/%2B
Disallow: /blogs/%2b
Disallow: //blogs/+*
Disallow: //blogs/%2B*
Disallow: //blogs/%2b*
Disallow: /?oseid=
Disallow: /preview_theme_id
Disallow: /preview_script_id
Disallow: /policies/
Disallow: //policies/
Disallow: //?ls=&ls=*
Disallow: //?ls%3D%3Fls%3D*
Disallow: //?ls%3d%3fls%3d*
Disallow: /search
Disallow: /apple-app-site-association
Disallow: /.well-known/shopify/monorail
Disallow: /cdn/wpm/.js
Disallow: /recommendations/products
Disallow: //recommendations/products
Sitemap: https://mobyflex.shop/sitemap.xml

Google adsbot ignores robots.txt unless specifically named!

User-agent: adsbot-google
Disallow: /checkouts/
Disallow: /checkout
Disallow: /carts
Disallow: /orders
Disallow: /89883672894/checkouts
Disallow: /89883672894/orders
Disallow: /?oseid=
Disallow: /preview_theme_id
Disallow: /preview_script_id
Disallow: /cdn/wpm/.js

User-agent: Nutch
Disallow: /

User-agent: AhrefsBot
Crawl-delay: 10
Disallow: /a/downloads/-/*
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /checkout
Disallow: /89883672894/checkouts
Disallow: /89883672894/orders
Disallow: /carts
Disallow: /account
Disallow: /collections/sort_by
Disallow: //collections/sort_by
Disallow: /collections/+*
Disallow: /collections/%2B
Disallow: /collections/%2b
Disallow: //collections/+*
Disallow: //collections/%2B*
Disallow: //collections/%2b*
Disallow: /collections/filter&filter
Disallow: /blogs/+*
Disallow: /blogs/%2B
Disallow: /blogs/%2b
Disallow: //blogs/+*
Disallow: //blogs/%2B*
Disallow: //blogs/%2b*
Disallow: /?oseid=
Disallow: /preview_theme_id
Disallow: /preview_script_id
Disallow: /policies/
Disallow: //policies/
Disallow: //?ls=&ls=*
Disallow: //?ls%3D%3Fls%3D*
Disallow: //?ls%3d%3fls%3d*
Disallow: /search
Disallow: /apple-app-site-association
Disallow: /.well-known/shopify/monorail
Disallow: /cdn/wpm/.js
Disallow: /recommendations/products
Disallow: //recommendations/products
Sitemap: https://mobyflex.shop/sitemap.xml

User-agent: AhrefsSiteAudit
Crawl-delay: 10
Disallow: /a/downloads/-/*
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /checkout
Disallow: /89883672894/checkouts
Disallow: /89883672894/orders
Disallow: /carts
Disallow: /account
Disallow: /collections/sort_by
Disallow: //collections/sort_by
Disallow: /collections/+*
Disallow: /collections/%2B
Disallow: /collections/%2b
Disallow: //collections/+*
Disallow: //collections/%2B*
Disallow: //collections/%2b*
Disallow: /collections/filter&filter
Disallow: /blogs/+*
Disallow: /blogs/%2B
Disallow: /blogs/%2b
Disallow: //blogs/+*
Disallow: //blogs/%2B*
Disallow: //blogs/%2b*
Disallow: /?oseid=
Disallow: /preview_theme_id
Disallow: /preview_script_id
Disallow: /policies/
Disallow: //policies/
Disallow: //?ls=&ls=*
Disallow: //?ls%3D%3Fls%3D*
Disallow: //?ls%3d%3fls%3d*
Disallow: /search
Disallow: /apple-app-site-association
Disallow: /.well-known/shopify/monorail
Disallow: /cdn/wpm/.js
Disallow: /recommendations/products
Disallow: //recommendations/products
Sitemap: https://mobyflex.shop/sitemap.xml

User-agent: MJ12bot
Crawl-delay: 10

User-agent: Pinterest
Crawl-delay: 1

4

Hi @mobyflex ,

Then I see the rules are all okay.

It sets to disallow all the unnecessary links, like login pages, filters, and sorting options,…

You do not need to worry about that!

Best,

Daisy

5

Thank you very much.

6

@mobyflex

My pleasure.

You can mark my response as solution so that other merchants can look at. Thank you.

Have a great day!

7

How can I select this option (solution)? . I am new here