Shopify robots.txt: what to edit and what to leave alone

This is the single most impactful customization available in Shopify robots.txt. Shopify creates tag-filtered collection URLs in the pattern /collections/summer-sale+red and faceted navigation URLs with parameters like ?filter.p.m.color=red. A store with 50 collections and 20 tags can generate over 1,000 unique URLs from these combinations — none of which you want indexed.

In your robots.txt.liquid, you can add a rule like Disallow: /collections/*+* to block tag-combined URLs while keeping individual collection pages fully crawlable. For filter parameters, Disallow: /*?filter and Disallow: /*?sort_by handle the most common cases. This is a well-documented pattern in the Shopify community and is safe when applied precisely. For more on managing crawl budget across your store, see our guide on the complete Shopify SEO checklist.

Shopify’s internal search generates URLs like /search?q=red+dress — pages that are essentially dynamic search results with no stable content and no link equity. If Googlebot crawls these, it is spending your crawl budget on pages that will never rank and may trigger thin-content signals. A Disallow: /search rule prevents this entirely.

Similarly, URL parameters like ?sort_by=price-ascending and ?view=list create duplicate versions of your collection pages. Adding Disallow: /*?sort_by and Disallow: /*?view keeps Googlebot focused on your canonical collection pages. Check Google Search Console’s Coverage report to confirm these URLs are not being indexed before and after adding the rules. You can also cross-reference this with your Shopify SEO audit process.

Shopify disallows AhrefsBot and AhrefsSiteAudit by default, which is a reasonable choice for stores that do not use the tool — Ahrefs crawls aggressively and generates real server load. However, if your SEO team relies on Ahrefs for site audits, technical health reports, and rank tracking, blocking their crawler means your Ahrefs data will be incomplete or inaccurate.

To remove this default block selectively in robots.txt.liquid, use a Liquid conditional: wrap the AhrefsBot group output in an unless condition so it is omitted from the rendered file. This is explicitly documented as a supported pattern in Shopify’s developer guides. It is the one default rule where removal has a clear, legitimate business justification. For deeper Ahrefs-to-Shopify workflow integration, see our post on the best Shopify SEO tools.

Shopify’s default robots.txt does not include specific rules for Bingbot or DuckDuckBot, which means they both inherit the catch-all User-agent: * rules. For most stores this is perfectly adequate. However, if you have sections of your store that you want to allow or disallow specifically for Bing (for example, if you run Microsoft Shopping campaigns and want to ensure Bingbot can access all product pages), adding a dedicated Bingbot group gives you that precision.

The syntax in robots.txt.liquid follows the same Liquid conditional pattern as other customizations. Use the group.user_agent.value == 'Bingbot' check to inject rules for that specific crawler. This customization is low-risk when done correctly, but unnecessary for the vast majority of Shopify stores. Focus on the higher-priority edits (tag pages, faceted navigation) before worrying about per-crawler tuning.

Crawl-delay is a robots.txt directive that tells crawlers to wait a specified number of seconds between requests. It sounds like a useful tool for managing server load, but there is a critical caveat: Googlebot completely ignores the crawl-delay directive. It manages its own crawl rate based on server response times and signals from Google Search Console. A crawl-delay in your robots.txt will never slow down Google.

What crawl-delay does affect is secondary crawlers like Bingbot, Yandex, and other compliant bots. If your Shopify store is on a plan with constrained server resources and you are seeing response-time degradation during heavy crawl periods, a crawl-delay for specific non-Google agents can help. For most Shopify stores this is a non-issue — Shopify’s infrastructure handles crawl load well. If you do add a crawl-delay, set it within the specific user-agent group rather than the wildcard group, and test server response times in Search Console before and after. For a full technical SEO walkthrough see our Shopify technical SEO guide.