Robots.txt is a simple text file that tells search engines which pages they can and cannot crawl. A misconfigured robots.txt can accidentally block important pages from appearing in search results.
What is Robots.txt?
Located at yourdomain.com/robots.txt, this file gives instructions to search engine crawlers about which parts of your site they should access.
Basic Robots.txt Syntax
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/
Sitemap: https://yourdomain.com/sitemap.xmlCommon Directives
- User-agent: Which crawler the rules apply to (* = all)
- Allow: Pages crawlers CAN access
- Disallow: Pages crawlers should NOT access
- Sitemap: Location of your XML sitemap
What to Block
- Admin/login areas
- Internal search results
- Shopping cart/checkout
- Private user areas
- Duplicate/filtered pages
What NOT to Block
- CSS, JavaScript, and image files (Google needs these)
- Any page you want in search results
- Your homepage or main content
Testing Robots.txt
Use Google Search Console's robots.txt tester to verify your rules work as intended before deploying changes.
Common Mistakes
- Blocking CSS/JS files (breaks rendering)
- Using wildcards incorrectly
- Forgetting trailing slashes
- Blocking the entire site accidentally
Related: Fix Crawlability Issues
0 comments