Fluxx TechTwo small files help Google crawl your website properly: the XML sitemap and robots.txt. Most business owners never look at them, until a mistake hides important pages from Google. Here is what they do and how to check them.
What is an XML sitemap?
An XML sitemap is a list of the important pages on your website that you want search engines to find. It usually lives at yourwebsite.com/sitemap.xml (or a similar address created by your platform).
What it should include:
• Homepage, service pages, product and category pages
• Blog posts and other pages you want indexed
• Only live, indexable pages that return a normal status
What it should not include:
• Redirected or deleted pages
• Pages marked “noindex”
• Thank-you pages, admin pages and test pages
Most platforms, such as WordPress SEO plugins and Shopify, create sitemaps automatically.
How to submit a sitemap
1. Open Google Search Console
2. Go to Sitemaps
3. Enter your sitemap URL and submit
4. Check back for errors or “Couldn’t fetch” messages
Resubmitting isn’t needed every time you add a page; Google rechecks sitemaps regularly.
What is robots.txt?
Robots.txt is a text file at yourwebsite.com/robots.txt that tells search engine crawlers which parts of your site they may or may not crawl.
A simple robots.txt might allow everything and point to your sitemap:
User-agent: *
Disallow:
Sitemap: https://www.yourwebsite.com/sitemap.xml
Important: robots.txt is not for hiding pages
Blocking a page in robots.txt stops crawling, but the URL can still appear in search results if other sites link to it. To keep a page out of Google, use a “noindex” tag or password protection instead.
Common mistakes
• Blocking the whole site with “Disallow: /” left over from development
• Blocking CSS and JavaScript files Google needs to render pages
• Sitemaps full of redirects, 404s or noindexed pages
• No sitemap submitted in Search Console
• Old sitemaps from a previous website still referenced
Quick check
• ☐ Visit yourwebsite.com/robots.txt; make sure it doesn’t block important sections
• ☐ Visit your sitemap; check it lists your key pages
• ☐ In Search Console, check the Sitemaps report and the Pages report
Frequently asked questions
Does every website need a sitemap? Small sites with good internal links can be found without one, but a sitemap helps and is easy to set up.
Can robots.txt improve rankings? Not directly. It helps Google spend its crawling on the pages that matter.
What if I accidentally blocked my site? Fix robots.txt, then use Search Console to request recrawling of key pages.
