Search engines need to discover and crawl your website before they can understand its content and potentially show its pages in search results.
But does Google need to crawl every URL on your website?
Not necessarily.
That’s where a robots.txt file comes in.
A robots.txt file gives search engine crawlers instructions about which areas of your website they are allowed to crawl. When configured correctly, it can help you manage crawler activity and keep bots away from URLs that don’t need to be crawled.
When configured incorrectly, however, robots.txt can accidentally prevent Googlebot from accessing important pages, resources, or entire sections of your website.
Google describes robots.txt primarily as a way to manage crawler traffic, not as a method for keeping webpages out of Google Search.
In this beginner’s guide, we’ll explain what robots.txt is, how it works, why it matters for SEO, how to create one, and which common mistakes you should avoid.
What Is a Robots.txt File?
A robots.txt file is a plain-text file placed at the root of a website that provides crawling instructions to search engine bots and other web crawlers.
For example, BizeeBay’s robots.txt URL would be located at:
https://bizeebay.com/robots.txt
The file can contain rules telling specific crawlers which URL paths they may or may not crawl.
A simple robots.txt file could look like this:
User-agent: *
Disallow: /private/
Here’s what those instructions mean:
User-agent: *
The rule applies to all crawlers that respect the protocol.
Disallow: /private/
The crawler is being instructed not to crawl URLs under the /private/ path.
The concept is straightforward, but robots.txt becomes much more important as websites grow in size and technical complexity.
How Does Robots.txt Work?
When a compliant search engine crawler arrives at a website, it can retrieve the site’s robots.txt file before crawling other URLs.
The crawler then interprets the applicable rules to determine which URLs it may access.
For example:
User-agent: Googlebot
Disallow: /checkout/
This rule applies specifically to Googlebot and instructs it not to crawl URLs under /checkout/.
Another example:
User-agent: *
Disallow:
An empty Disallow means there is no crawling restriction from that rule.
Robots.txt therefore acts somewhat like a set of traffic instructions for crawlers.
It doesn’t control what human visitors can access, and it should never be treated as a security system.
Why Is Robots.txt Important for SEO?
Robots.txt is part of technical SEO because search engines need to access your content before they can properly process it.
Google’s minimum technical requirements for Search include allowing Googlebot to access pages you want considered for indexing.
A properly configured robots.txt file can help with several areas.
1. Manage Search Engine Crawling
Not every URL deserves crawler attention.
A website might contain:
- Internal search URLs
- Filtered URLs
- Shopping-cart pages
- Duplicate URL patterns
- Staging-related paths
- Utility pages
- Large numbers of parameterized URLs
Robots.txt can help manage crawler access to appropriate sections.
2. Help Manage Crawling on Large Websites
For a small website with 20 pages, crawl management may not be a major concern.
For an ecommerce website containing hundreds of thousands of URLs, it’s a different story.
Filters, sorting options, faceted navigation, parameters, and duplicate URL combinations can create enormous numbers of crawlable URLs.
Google specifically notes that large sites with many duplicate or less-important URLs may benefit from controlling crawling so Google can focus on more important content.
3. Prevent Accidental Crawling of Unnecessary Areas
Some website sections simply don’t need to be crawled.
Depending on the website, examples could include cart URLs or certain internal tools.
Blocking unnecessary crawling can help keep your site’s crawl structure cleaner.
But you need to be careful.
A broad rule such as:
User-agent: *
Disallow: /
instructs compliant crawlers not to crawl the entire website.
One misplaced slash can therefore create a serious SEO problem.
Robots.txt Does Not Mean Noindex
This is probably the most important concept in this entire guide.
Blocking a URL in robots.txt does not reliably prevent that URL from appearing in Google Search.
Robots.txt controls crawling.
Noindex controls indexing.
Google explicitly warns against using robots.txt as a mechanism for preventing webpages from being indexed. A URL blocked from crawling can potentially still appear in search results if Google discovers it through links or other sources.
So if you have:
User-agent: *
Disallow: /example-page/
Google may avoid crawling the page, but that doesn’t necessarily guarantee the URL will never appear in search results.
Robots.txt vs. Noindex: What's the Difference?
Understanding robots.txt vs. noindex is essential for technical SEO.
Feature | Robots.txt | Noindex |
Primary Purpose | Controls crawling | Controls indexing |
Stops Googlebot Crawling? | Yes, when applicable | No |
Designed to Remove Page From Search? | No | Yes |
Location | /robots.txt | HTML meta tag or HTTP header |
Good for Private Content? | No | No—use authentication for genuinely private content |
A typical noindex tag looks like this:
<meta name=”robots” content=”noindex”>
Google says a page must remain accessible to its crawler for Google to discover and process the noindex directive. If robots.txt blocks the page, Googlebot cannot crawl it to see the noindex rule.
In other words, don’t do this expecting it to work properly:
Disallow: /page/
while simultaneously relying on a noindex tag located on /page/.
You’ve told Googlebot not to visit the very page where it needs to discover the noindex instruction.
Common Robots.txt Directives Explained
Let’s look at the most important elements you’ll encounter in a robots.txt file.
User-agent
The User-agent identifies which crawler a set of rules applies to.
Example:
User-agent: Googlebot
Or:
User-agent: *
The asterisk represents crawlers generally.
Disallow
The Disallow directive tells the specified crawler not to crawl matching paths.
Example:
User-agent: *
Disallow: /admin/
This asks crawlers covered by the rule not to crawl URLs under /admin/.
Allow
Allow can be used to permit crawling of a more specific URL or path within an otherwise disallowed area when supported by the crawler.
For example:
User-agent: Googlebot
Disallow: /folder/
Allow: /folder/public-page/
This can be useful when you need more granular crawling instructions.
Sitemap
You can also reference an XML sitemap from robots.txt.
For example:
Sitemap: https://example.com/sitemap.xml
Google supports discovering sitemaps through robots.txt, and multiple sitemap lines can be included.
This doesn’t guarantee Google will crawl or index every URL in the sitemap, but it provides another way to help search engines discover your preferred URLs.
What About Crawl-Delay?
You may encounter robots.txt examples containing:
Crawl-delay: 10
Don’t assume every search engine supports every robots.txt directive.
For example, Google has previously clarified that crawl-delay is not a supported Google robots.txt rule.
Always check the documentation of the crawler you’re trying to manage rather than copying a generic robots.txt template from another website.
A Simple Robots.txt Example
Here’s a basic example:
User-agent: *
Disallow: /private/
Disallow: /temporary/
Sitemap: https://example.com/sitemap.xml
This tells crawlers covered by the rules not to crawl the /private/ and /temporary/ paths while also providing the location of the XML sitemap.
However, there is no universal “perfect robots.txt file.”
Your configuration should depend on your website architecture.
Where Is Robots.txt Located?
A robots.txt file belongs at the root of the website it controls.
For example:
https://example.com/robots.txt
If you want to check a website’s robots.txt file, simply add:
/robots.txt
to the domain.
Google notes that most modern content management systems can generate robots.txt automatically, so many website owners never need to create one manually.
How to Create a Robots.txt File
If your CMS or website platform doesn’t automatically manage robots.txt, you can create a plain-text file called:
robots.txt
Then add the rules appropriate for your website.
A very simple configuration might look like:
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xml
Before publishing your file, make sure:
- The filename is robots.txt.
- The syntax is valid.
- Important website sections aren’t accidentally blocked.
- Your sitemap URL is correct if included.
- Rules target the crawlers you actually intend to control.
- You understand what every Disallow rule does.
For complex websites, don’t blindly copy another site’s robots.txt file.
Their URL structure may be completely different from yours.
Robots.txt for WordPress
WordPress websites commonly have a robots.txt file generated or managed through WordPress, SEO plugins, hosting configurations, or custom server settings.
A WordPress site may have areas that don’t need significant crawler attention, but the correct configuration depends on the site’s architecture.
The biggest mistake is blocking resources Google needs to properly understand or render important pages.
Google recommends using its URL Inspection tool to verify that Google can access and render a page as expected when diagnosing blocked resources.
For a business website, your important:
- Service pages
- Blog posts
- Product pages
- Location pages
- Images and resources needed for rendering
shouldn’t be accidentally blocked.
Robots.txt for Shopify
Shopify also handles much of its robots.txt configuration automatically.
For ecommerce websites, robots.txt can become especially relevant because stores may generate URLs related to:
- Products
- Collections
- Search
- Filters
- Cart functionality
- Parameters
- Sorting
Ecommerce crawl management can become complex quickly.
That doesn’t mean you should aggressively block everything that looks unnecessary.
Changes should be made based on the site’s actual technical SEO requirements rather than a generic “best Shopify robots.txt” template.
Robots.txt and XML Sitemaps
Robots.txt and XML sitemaps serve different purposes.
Think of it this way:
Robots.txt:
“Here are areas crawlers should or shouldn’t access.”
XML sitemap:
“Here are URLs I want search engines to know about.”
Google recommends including in a sitemap the URLs you want to see in search results and allows sitemap locations to be declared within robots.txt. (Google for Developers)
For example:
Sitemap: https://example.com/sitemap.xml
A sitemap doesn’t guarantee indexing, but it can help Google discover important URLs.
Robots.txt vs. Meta Robots
These are also frequently confused.
robots.txt operates at the crawling level.
A robots meta tag operates at the individual HTML page level and can provide instructions related to indexing and how content is presented in search results.
For example:
<meta name=”robots” content=”noindex”>
Google also supports equivalent directives through the X-Robots-Tag HTTP header, which can be particularly useful for non-HTML resources.
Choosing the right method depends on what you’re trying to accomplish.
Common Robots.txt SEO Mistakes
A small robots.txt mistake can create a large SEO problem.
Here are some of the most serious mistakes to avoid.
Blocking Your Entire Website
This rule can prevent crawling across the site:
User-agent: *
Disallow: /
It may be useful in very specific environments, but it can be disastrous if accidentally left on a live website.
This is especially worth checking after a website redesign or migration. Google specifically recommends removing robots.txt blocks that were needed only during a migration.
Using Robots.txt to Hide Sensitive Information
Robots.txt is publicly accessible.
It is not a security tool.
If content is genuinely confidential, use proper access controls such as authentication rather than assuming a crawler directive makes the content private.
Google recommends password protection for confidential or private content.
Using Robots.txt Instead of Noindex
If your goal is preventing a webpage from appearing in Google, robots.txt isn’t the correct mechanism.
Use an appropriate indexing control such as noindex while keeping the page crawlable so Google can see the directive.
Blocking Important Website Resources
Blocking resources required to render a page can make it harder for Google to understand the page correctly.
Review rules carefully before blocking JavaScript, CSS, images, or other resources.
Copying Another Website's Robots.txt
A competitor’s robots.txt file might look sophisticated.
That doesn’t mean it’s right for your website.
Different websites have different:
- CMS platforms
- URL structures
- E-commerce functionality
- Filters
- Search parameters
- Plugins
- Applications
- Technical SEO requirements
Build your rules around your own website architecture.
Does Robots.txt Improve SEO Rankings?
Not directly.
Adding a robots.txt file doesn’t suddenly make your website rank higher.
Its SEO value comes from crawl management and preventing technical mistakes.
A strong robots.txt configuration can help search engines crawl your website appropriately, while a bad configuration can make valuable pages inaccessible to crawlers.
Think of robots.txt as infrastructure.
It doesn’t make your content better, but incorrect infrastructure can prevent good content from getting the opportunity it deserves.
How to Check Your Robots.txt for SEO Problems
Start by opening:
yourdomain.com/robots.txt
Then review the rules carefully.
Ask:
Are important pages blocked?
Are entire directories blocked unnecessarily?
Can Googlebot access important resources?
Does the sitemap reference point to the correct location?
Were staging or development rules accidentally pushed to production?
For individual pages, Google’s URL Inspection tool can help determine whether Google can access a URL. Google also recommends Search Console’s Page Indexing and Crawl Stats reports when diagnosing crawling problems.
Robots.txt Best Practices for SEO
Keep your robots.txt configuration as simple as your website allows.
A good approach includes:
- Allowing search engines to crawl valuable public content.
- Blocking crawling only when there is a clear reason.
- Avoiding robots.txt for confidential information.
- Using noindex rather than robots.txt when the goal is preventing indexing.
- Keeping sitemap references accurate.
- Reviewing rules after website migrations.
- Testing important pages after robots.txt changes.
- Monitoring Google Search Console for crawling and indexing problems.
- Reviewing ecommerce filters and URL parameters carefully.
- Avoiding unnecessary or unsupported directives.
Most importantly, understand why you’re adding a rule before you add it.
Build a Stronger Technical SEO Foundation
Robots.txt may be a tiny text file, but it can have a significant impact on how search engine crawlers interact with your website.
The most important thing to remember is simple:
Robots.txt controls crawling, not reliable removal from search results.
Use it strategically to manage crawler access. Use noindex when you need to control indexing. Use XML sitemaps to help search engines discover important URLs. And always test significant changes before assuming your configuration is working correctly.
Robots.txt is only one piece of technical SEO. Crawlability, indexing, site architecture, redirects, canonicalization, structured data, page performance, internal linking, and XML sitemaps all contribute to a search-friendly website.
For businesses that don’t want to manage those technical details alone, BizeeBay’s SEO and website development team can help identify crawling and indexing issues, improve technical SEO, and build websites with a stronger search foundation.
Frequently Asked Questions
What is robots.txt used for?
Robots.txt is primarily used to tell compliant web crawlers which URLs or sections of a website they can access. Google describes its main purpose as managing crawler traffic rather than preventing webpages from appearing in Search.
Is robots.txt important for SEO?
Yes, particularly from a technical SEO perspective. Incorrect rules can prevent Googlebot from accessing important pages or resources, while appropriate rules can help manage crawler activity.
Where is the robots.txt file located?
It is normally available at the root of the host, such as:
https://example.com/robots.txt
Can robots.txt prevent indexing?
Not reliably. A blocked URL can potentially still appear in Google Search if Google discovers it elsewhere. Google recommends noindex or proper access controls when the goal is preventing a page from appearing in Search.
What does Disallow mean in robots.txt?
Disallow tells the targeted crawler not to crawl URLs matching the specified path.
