Runbook: slow site #

For on-call staff and administrators when a customer reports that their Exponential site is slow. The levers and their order come from 10.12 Performance tuning.

Contents #

1. Measure first
2. Is it everyone, or one page?
3. Checks in order
4. Fixes in order of effect
5. After the incident

Measure first #

A slow site is a number, not a feeling. Measure the front page and the pages the customer named:

curl -s -o /dev/null -w '%{http_code} %{time_total}\n' https://www.example.com/
php bin/php/console exp:benchmark --allow-root-user

exp:benchmark requests the front page, the first menu pages, the search page and the admin login page, 100 times each after 5 warm-up requests, 4 at a time, and prints median, p90 and p99. If you saved a baseline when the site was healthy, compare with it:

php bin/php/console exp:benchmark --baseline=var/benchmark/base.json --threshold=15 --allow-root-user

Is it everyone, or one page? #

Checks in order #

1. The machine: uptime, free -m, df -h, iostat if available. Swap use or a full disk explain everything else (Disk full).
2. The caches: php bin/php/cache.php status --json --allow-root-user should end in PASS. Check that [ContentSettings] ViewCaching=enabled and [TemplateSettings] TemplateCache=enabled (both shipped) were not switched off in an override.
3. The engine:
   • Velocity: php bin/php/console exp:velocity status --allow-root-user (workers as configured); exp:velocity cache stats for the response cache.
   • PHP-FPM: are all children busy? See PHP-FPM tuning.
4. OPcache: php bin/php/cache.php opcache status --allow-root-user. Never run an Exponential site with opcache.validate_timestamps=0; under Velocity the shipped settings matter (OPcache).
5. The database: on SQLite after an install or a large import, "the query planner has no statistics": run sqlite3 var/storage/sqlite3/exponential.db ANALYZE once (12.10). On MongoDB, check that the indexes from bin/mongodb/create_indexes.js exist.
6. Traffic: security.log and 404.log show bursts of probes and not-found requests; a crawler or an attack can look like slowness (Request logs).
7. The static cache: it should be off ([ContentSettings] StaticCache=disabled, the shipped value). With it on, every publish regenerates affected pages inside the editor's request (Static cache).

Fixes in order of effect #

From 10.12:

1. Serve with Velocity and its response cache; persistent workers keep the application warm.
2. Keep the content view cache and template-block cache on, and warm the site after a clear: php bin/php/warm.php --allow-root-user.
3. OPcache, configured as OPcache describes.
4. APCu for Velocity's response cache memory tier and the SQL query cache in shared mode.
5. The SQL query cache, measured on the site.
6. The database tuning of chapter 9.

Change one lever at a time and measure again with the baseline.

Slow under crawlers and bots #

When the slow times match bursts of crawlers (search engines, AI crawlers, SEO tools; the access log shows many anonymous requests from a few user agents within a minute), the cheapest fix is a cache that answers anonymous pages without PHP:

• Velocity: its response cache does this out of the box.
• LiteSpeed or OpenLiteSpeed: the server's page cache, switched on with the commented block of .htaccess_root or the same rules in the virtual host, no plugin needed: Page caching with LiteSpeed and OpenLiteSpeed.
• Apache or nginx: cacheable headers for anonymous visitors and a caching proxy in front (HTTP cache headers, Varnish and reverse proxies), or the role-aware HTTP cache.

robots.txt can slow polite crawlers down; it does nothing against the others.

After the incident #

• To do: Save a new baseline when the site is healthy again: php bin/php/console exp:benchmark --save=var/benchmark/base.json --allow-root-user.
• To do: Note the cause and the fix in the site's record.
• To do: If the site has outgrown its plan, tell the customer with the measurements (Designing hosting plans).

Related pages #

• Caching layers
• Cache clearing and warming
• Benchmarking Exponential
• Free documentation: this page is free to copy, translate and improve under the GNU FDL 1.3.