Runbook: site down
For on-call staff when a customer's Exponential site does not serve pages. Work top to bottom; stop when the cause is found. The error pages and their causes come from 12.9 Pages and errors at run time.
Warning
.htaccess slows down every request. When .htaccess files are enabled (AllowOverride anything but None), let alone
used, Apache looks for them and reads and evaluates them on every request, in every directory of the path, and
page loading times go up drastically (measured). The only real fix is to switch the feature off for
the virtual host: AllowOverride None in the HTTPS virtual host, and the rules in the virtual host's own
configuration as mod_rewrite rules. That is far faster in production speed tests and well worth the time it takes
to optimize a production configuration. Switch over right away: copy the rules of the
example HTTPS virtual host into your virtual host (with the static cache, also
staticcache_offload.conf), set AllowOverride None, run apachectl configtest, reload Apache and
test the site. The same rules for other servers: nginx, LiteSpeed,
Caddy, IIS, Velocity (built in); where virtual host rules go in a control
panel: Plesk, cPanel, DirectAdmin, ISPConfig.
On LiteSpeed or OpenLiteSpeed, the server's own page cache can also answer anonymous visitors and crawlers without PHP: Page caching with LiteSpeed and OpenLiteSpeed.
Contents
Symptoms
- The external check alerts.
- Customers report "Something went wrong", "We'll be right back", a blank page, or a browser connection error.
First five minutes
Open the site yourself and note exactly what it shows, with the HTTP status:
curl -sI https://www.example.com/ | head -5Open the administration too: does it fail the same way?
Read the newest errors:
./console exp:logs --tail=var/log/error.log --lines=50Check the machine: disk space (
df -h), memory, load. A full disk causes most of the symptoms below; see Disk full.
Which page does the visitor see?
| The visitor sees | Cause | Do |
|---|---|---|
"We'll be right back" (503, Retry-After: 30) | the database could not be reached | check that the database server runs; check the credentials in settings/override/site.ini.append.php; the log line holds the driver's message |
| "Down for maintenance" (503) | maintenance mode is on | php bin/php/maintenance.php status --allow-root-user; if the work is done: php bin/php/maintenance.php off --allow-root-user; deleting var/maintenance.json ends it whatever else is wrong |
| "The site is missing the software libraries it needs" (503) | vendor/ is missing, moved or renamed | put vendor/ back or run composer install --no-dev; then php bin/php/ezcache.php --clear-all, and restart Velocity if it serves the site |
"Something went wrong on our side" (500) with ERR-... | an uncaught error | ./console exp:logs --reference=ERR-...; the line names the message, file and line (Finding an error by its reference) |
A blank page, or 500 with nothing in error.log | PHP failed before Exponential could log: syntax error, missing extension, memory exhausted | read the web server's or PHP-FPM's error log; php -l on a recently changed file |
| 404 for every page under Apache | .htaccess not read or mod_rewrite off | allow overrides, enable mod_rewrite (Apache with PHP-FPM) |
Tip
Errors that started right after a deploy or an update: the cause is almost always in that change. Under Velocity a
changed class or setting needs exp:velocity restart or exp:velocity deploy; see
Deploying changes.
Nothing answers at all
The connection is refused or times out.
Under Velocity:
php bin/php/console exp:velocity status --allow-root-userstatus reports "running" only when the server answers a request; otherwise it prints the problems it finds and
exits with 1. If it is not running, start it (exp:velocity start) and read var/vc/qbix/run/console.log if it does
not come up; see Velocity will not start. A server that runs but does not answer:
exp:velocity restart, and if a worker is wedged, exp:velocity kill then start.
Under Apache or nginx with PHP-FPM:
- Is the web server running? Is the site's PHP-FPM pool running? (
systemctl status <service>; on Plesk the service isplesk-php<XY>-fpm.) - Are all PHP-FPM children busy? The pool's status page or its log says so; see Slow site.
In front of it: a load balancer, reverse proxy, CDN or firewall rule may be the part that fails. Test the origin directly.
After the incident
- To do: Clear the caches the fix needs, and with Velocity its response cache after any restart
(
php bin/php/console exp:velocity cache clear --apply --allow-root-user). - To do: Verify: front page, administration, a sign-in, a publish.
- To do: Write the incident note: start, detection, cause, fix, end, what prevents a repeat.
- To do: Tell the customer.
- To do: If the cause is a bug in Exponential, report it (When something goes wrong).
Related pages
- Slow site
- Velocity will not start
- Chapter 12: Troubleshooting
- Free documentation: this page is free to copy, translate and improve under the GNU FDL 1.3.
Become an Exponential Partner
Partners share their information, provide funding, contribute back and develop the products with us, and that is how Exponential grows. A partner website and dashboard are planned for mid-2027: please become a partner today to speed up our growth rate and land new features faster.