robots.txt for your site
This page is for website creators who tell search engine crawlers what to skip. A new installation ships a
robots.txt at the web root with sensible rules (A useful robots.txt out of the box).
Warning
.htaccess slows down every request. When .htaccess files are enabled (AllowOverride anything but None), let alone
used, Apache looks for them and reads and evaluates them on every request, in every directory of the path, and
page loading times go up drastically (measured). The only real fix is to switch the feature off for
the virtual host: AllowOverride None in the HTTPS virtual host, and the rules in the virtual host's own
configuration as mod_rewrite rules. That is far faster in production speed tests and well worth the time it takes
to optimize a production configuration. Switch over right away: copy the rules of the
example HTTPS virtual host into your virtual host (with the static cache, also
staticcache_offload.conf), set AllowOverride None, run apachectl configtest, reload Apache and
test the site. The same rules for other servers: nginx, LiteSpeed,
Caddy, IIS, Velocity (built in); where virtual host rules go in a control
panel: Plesk, cPanel, DirectAdmin, ISPConfig.
On LiteSpeed or OpenLiteSpeed, the server's own page cache can also answer anonymous visitors and crawlers without PHP: Page caching with LiteSpeed and OpenLiteSpeed.
Contents
What ships
User-agent: *
Allow: /
Disallow: /content/search/
Disallow: /content/advancedsearch/
Disallow: /content/tipafriend/
Disallow: /layout/set/print/
Disallow: /media/
Disallow: /test-area/
Crawl-delay: 1Each disallowed path is also listed in its capitalized form, because a crawler may have met either spelling. The file starts with a block of comments you may remove, and holds commented sitemap lines.
Make it yours
- Remove
/media/and/test-area/if they do not match your tree; add your own sections, using the addresses visitors see (URL aliases). - Add your sitemap:
Sitemap: https://www.example.com/sitemap.xml(XML sitemaps). - Check:
curl -s https://www.example.com/robots.txt.
Keep it in version control with your deployment, not edited on the server.
How it is served
.htaccess_root has RewriteRule ^robots\.txt - [L], so Apache hands the file over directly. Velocity's built-in
rewrite rules are the same list (8.1 The choice in one table).
On nginx make sure /robots.txt reaches the file.
Several sites
One file serves every host of the installation. When sites need different rules, let the web server serve a file per
host (for example robots-site-b.txt for www.site-b.example), or keep the rules general enough for all.
What robots.txt is not
- It is not access control: anyone can read what you disallow. Protect private content with permissions (Sections and object states).
- A disallowed page can still be listed by address if other sites link to it; use a
noindexmeta tag in its template to keep it out of results. - Staging and test sites should require a login rather than rely on
robots.txt.
Related pages
- SEO basics for Exponential sites
- XML sitemaps
- Serving the site
- Free documentation: this page is free to copy, translate and improve under the GNU FDL 1.3.
Become an Exponential Partner
Partners share their information, provide funding, contribute back and develop the products with us, and that is how Exponential grows. A partner website and dashboard are planned for mid-2027: please become a partner today to speed up our growth rate and land new features faster.