Robots.txt, sitemaps, and noindex are often talked about together because they all affect how search engines handle a website. The problem is that they do very different jobs.
One tells crawlers where they are allowed to go, one gives search engines a list of important URLs, and one tells search engines not to include a page in search results. Mixing them up can cause real problems, especially during launches, redesigns, migrations, and cleanup.
This guide will help explain what each one does, when to use it, and where website owners commonly get into trouble.
Quick Comparison
| Item | Main Job | Use It When | Do Not Use It For |
|---|---|---|---|
| robots.txt | Controls crawler access to parts of the site. | You want to manage what crawlers can request. | Keeping a page private or reliably out of search results. |
| XML sitemap | Lists important URLs for search engines to discover. | You want to help search engines find important pages. | Forcing Google to index every URL listed. |
| noindex | Tells search engines not to show a page in search results. | A page can be crawled, but should not appear in search. | Blocking crawler access in robots.txt at the same time. |
In short - robots.txt is about crawling, sitemaps are about discovery, and noindex is about search result inclusion.
Robots.txt - The Crawler Access File
A robots.txt file sits at the root of a website, usually at a URL like /robots.txt. It gives instructions to search engine crawlers about which areas of the site they are allowed to request.
For example, a site may allow crawlers to access public pages while discouraging them from crawling internal search results, filtered URLs, staging sections, duplicate paths, or areas that are not as useful for search.
Example robots.txt
User-agent: *
Disallow: /admin/
Disallow: /search/
Allow: /
Sitemap: https://www.example.com/sitemap.xml
In this example, crawlers are being told not to crawl the admin and search sections. The sitemap location is also provided so search engines can find the XML sitemap more easily.
What robots.txt is good for
- Reducing crawler access to low-value sections
- Keeping internal search results out of crawl paths
- Pointing crawlers toward the sitemap location
- Avoiding unnecessary crawling of duplicate or filtered URLs
Where robots.txt gets misunderstood
It's important to note that robots.txt doesn't make a page private. If a URL is blocked in robots.txt but linked from another website, search engines may still know the URL exists. They may not be able to crawl the page content, but blocking crawl access is not the same as removing the page from search results.
Google's robots.txt documentation explains that robots.txt is mainly for managing crawler access and should not be used as the method for keeping a page out of Google.
If a page contains private information, use proper access control. That may mean password protection, account login, server restrictions, or removing the content entirely. Robots.txt is public and easy to view.
Sitemaps - A List of Important URLs
An XML sitemap helps search engines discover the pages, posts, resources, products, or other URLs that you find important on your site. It is especially useful when a site is large, new, recently changed, or has pages that are not linked in the main navigation.
A sitemap does not force search engines to index every URL. Think of it more like a clean list of recommendations - these are the pages the site owner wants search engines to know about.
Example sitemap entry
<url>
<loc>https://www.example.com/services/website-maintenance.php</loc>
<lastmod>2026-05-13</lastmod>
</url>
The <loc> value tells search engines the page URL. The <lastmod> value can help communicate when the page was last updated, as long as it's accurate.
Good sitemap candidates
- Main service pages
- Important resources and articles
- Product or category pages that should be discoverable
- Location pages that are useful and unique
- Important pages that may not be linked in the main menu
What Should Not Be in a Sitemap
A sitemap should not become a dumping ground for every URL the website can generate. Keep it focused on pages that are useful, canonical, and intended for search.
Remove URLs that are blocked, redirected, duplicated, outdated, thin, private, or intentionally noindexed. A sitemap full of weak URLs can make cleanup harder because search engines are being handed a messy set of signals.
Avoid listing these
- Admin pages
- Login pages
- Cart and checkout URLs
- Thank-you pages
- Internal search result pages
- Test or staging URLs
- Duplicate filtered pages
- Pages with a noindex directive
Google's sitemap documentation explains how sitemaps help Google crawl a site more intelligently.
Noindex - The Search Results Control
Noindex is used when a page can be crawled, but should not appear in search results. This is a common need. A business may have thank-you pages, campaign pages, duplicate print pages, internal utility pages, thin archive pages, or temporary pages that should be accessible but not indexed.
Example noindex meta tag
<meta name="robots" content="noindex">
This tag belongs in the <head> of the page. Search engines need to crawl the page to see it, which is why noindex and robots.txt can conflict when used incorrectly.
The important detail
If a page is blocked by robots.txt, Google may not be able to crawl the page and see the noindex tag. When the goal is to remove a page from search results, the page generally needs to be crawlable long enough for the noindex directive to be found.
Google's noindex documentation covers how noindex can prevent a page from appearing in Google Search.
Meta Noindex vs X-Robots-Tag
Most business websites use a meta robots tag because it is easy to add to an HTML page. There is also an HTTP header option called X-Robots-Tag, which can be useful for files that do not have normal HTML head sections.
| Method | Common Use |
|---|---|
<meta name="robots" content="noindex"> |
HTML pages such as landing pages, archives, thank-you pages, or duplicate content pages. |
X-Robots-Tag: noindex |
Non-HTML files such as PDFs, generated files, or server-controlled responses. |
If you need to check whether a page or file is sending special HTTP directives, an HTTP Header Checker tool can help review the response headers.
Which One Should You Use?
Use the goal to choose the method.
If you want search engines to find important pages
Use a sitemap. Make sure the URLs are live, canonical, useful, and intended for search. Submit or review the sitemap in Google Search Console when appropriate.
If you want to reduce crawler access to low-value areas
Use robots.txt carefully. This is useful for areas like internal search result pages or sections that do not need crawler attention. Avoid blocking important pages by accident.
If you want a page kept out of search results
Use noindex, and make sure search engines can crawl the page to see the directive. For private or sensitive content, use authentication instead of relying on SEO directives.
Common Setup Mistakes
Blocking a page in robots.txt and adding noindex
This is a classic and common problem - the page is blocked, so the crawler may not see the noindex tag. If the goal is to remove the page from search results, this setup can work against you.
Leaving a staging block live after launch
During development, staging websites are often blocked from crawling - that is normal. The problem happens when a launch goes live and the blocking rules are copied over to production.
Before launch, check the live robots.txt file carefully. A line like Disallow: / can tell crawlers not to access the entire website.
Putting noindexed pages in the sitemap
A sitemap should list pages intended for discovery and indexing. If a URL is intentionally noindexed, including it in the sitemap sends a mixed signal and makes maintenance harder.
Using robots.txt for privacy
Robots.txt is public. Anyone can visit it. Do not use it to hide sensitive directories, private files, client information, invoices, backups, or admin areas that should be protected properly.
Letting old URLs stay in the sitemap
After a redesign or migration, old URLs sometimes remain in the sitemap even though they redirect, 404, or no longer represent current content. Sitemaps should be reviewed after major website changes.
Launch and Migration Checks
Robots.txt, sitemaps, and noindex settings should always be reviewed during launches and migrations. Small mistakes can affect search visibility quickly.
- Check robots.txt on the live domain. Make sure it is not accidentally blocking the entire website or important sections.
- Review the sitemap. Confirm that it includes the correct live URLs, not staging links or old development paths.
- Look for leftover noindex tags. Development pages, landing pages, or templates sometimes carry noindex settings into production.
- Test important pages. Service pages, location pages, resources, articles, and product pages should be crawlable if they are meant to appear in search.
- Submit or inspect in Search Console. Use Google Search Console to review sitemap status and inspect key URLs after launch.
This kind of review also fits well with a broader launch process. If you are preparing a website to go live, use the Pre-Launch Website Checklist alongside the technical SEO checks here.
Quick Decision Guide
| Situation | Use |
|---|---|
| You want Google to discover a new service page. | Add it to the XML sitemap and link to it from the site where appropriate. |
| You want a thank-you page kept out of search results. | Add noindex to the page. |
| You want to reduce crawling of internal search results. | Use robots.txt carefully. |
| You want to hide private client files. | Use proper authentication or remove the files. Do not rely on robots.txt. |
| You launched a new website and search engines are not finding pages. | Check robots.txt, review the sitemap, inspect key URLs, and look for accidental noindex tags. |
| You have a PDF that should not appear in search. | Consider an X-Robots-Tag noindex header or restrict access if the file is private. |
Basic File Locations
These are the common locations for a standard website:
- Robots.txt:
https://www.example.com/robots.txt - XML sitemap:
https://www.example.com/sitemap.xml - Sitemap index:
https://www.example.com/sitemap_index.xml
Some CMS platforms use different sitemap URLs. WordPress SEO plugins, ecommerce platforms, and hosted website builders may generate their own sitemap paths. The exact path matters less than making sure the sitemap is live, accurate, and referenced where needed.
Useful Waterloo Code Tools
These tools do not replace Search Console or a full technical SEO review, but they can still help:
- Text Diff Checker can help compare old and new versions of technical files, page templates, meta tags, or sitemap exports before and after a launch.
- Regex Tester is useful when reviewing URL patterns, redirects, filtered URLs, or rule logic before applying changes to a live website.
- Schema Starter helps generate basic structured data for important pages once crawling and indexing settings are in good shape.
