Webflow Sitemap and Robots.txt: The Complete Indexing Setup Guide
If Google cannot crawl and index your pages, nothing else about your SEO matters. Here is how Webflow's automatic sitemap and robots.txt work, and how to configure them so the right pages get found.
Webflow Sitemap and Robots.txt: Getting Indexed Before You Get Ranked
Ranking is step two. Step one is getting indexed — and if you skip it, every hour you spend on keywords and content is wasted on pages Google never sees. Two small files govern this: the XML sitemap, which tells search engines what exists, and robots.txt, which tells them what they are allowed to crawl.
Webflow generates both automatically, which is convenient and occasionally a trap. Here is how each one works and how to configure them so the pages you care about get discovered and the ones you do not stay out of the index.
The automatic sitemap
By default, Webflow builds and maintains an XML sitemap for you at yourdomain.com/sitemap.xml. Every published static page and CMS item is added automatically, and the file updates each time you publish. For most sites, leaving auto-generation on is exactly right.
To confirm it is enabled, go to Project Settings, open the SEO tab, and check the Sitemap section. You will see a toggle for auto-generation and a link to the live file.
- Keep auto-generation on unless you have a specific reason to override it — it is one less thing to maintain.
- Exclude pages you do not want indexed by opening each page's settings and disabling 'Sitemap indexing' for that page, or by setting it to noindex.
- Switch to a custom sitemap only when you need fine control Webflow does not offer, such as excluding a whole pattern of URLs. Turning off auto-generation means you paste in and maintain the XML yourself.
Submitting the sitemap to Google
Webflow generating a sitemap does not mean Google has seen it. Submit it once in Google Search Console under Indexing then Sitemaps, entering sitemap.xml. From then on Google rechecks it periodically and discovers new pages faster than it would by crawling links alone.
A sitemap does not force indexing — it invites it. Google still decides what is worth keeping, but you have to at least send the invitation.
How robots.txt fits in
robots.txt is a plain-text file at the root of your domain that tells crawlers which paths they may or may not request. In Webflow you edit it under Project Settings, SEO tab, in the robots.txt section. A minimal, healthy file for a normal marketing site looks like this in spirit:
- Allow the whole site to be crawled by default — do not block paths you want ranked.
- Point to your sitemap by adding a Sitemap line so crawlers that read robots.txt find it immediately.
- Block only genuine noise — internal search-results URLs, staging paths, or utility pages that should never appear in Google.
The most damaging robots.txt mistake is blocking something you meant to rank. A stray Disallow rule can remove an entire section of your site from Google. And remember: blocking a page in robots.txt is not the same as noindex. A blocked page can still appear in results with no description if other sites link to it, because Google never crawled it to see the noindex tag.
Index vs. noindex vs. disallow — the distinction that trips people up
- Indexable — Google can crawl it and is allowed to show it. This is your default for real content.
- noindex — Google can crawl it but must not show it in results. Use for thank-you pages, thin utility pages, and duplicates. Set it per page in Webflow's SEO settings.
- Disallow (robots.txt) — Google must not crawl it at all. Use sparingly, and never on a page you also want to noindex, because Google needs to crawl the page to see the noindex.
A quick indexing health check
- Visit yourdomain.com/sitemap.xml and confirm it lists your real pages and nothing sensitive.
- Visit yourdomain.com/robots.txt and confirm it is not blocking anything important.
- In Search Console, submit the sitemap and watch the Pages report for 'Indexed' vs. 'Not indexed' counts.
- Spot-check a few key URLs with the URL Inspection tool to confirm Google can fetch and index them.
Get these two files right once and indexing largely takes care of itself. Get them wrong and you can spend months wondering why great content never ranks — because it was never in the index to begin with.
Quilly is the AI content engine that writes, publishes, and ranks CMS content on autopilot — pulling real keyword demand, drafting on-brand articles, and pushing them straight to Webflow, WordPress, or a headless CMS. If you would rather rank than spend your week wrestling with SEO settings, start free.