Incident handling #

This page is for engineers on call for an Exponential site. It gives a sequence for any incident and points to the runbooks for the common ones. It does not replace your organization's incident process; it fills in the Exponential-specific steps.

Contents #

1. The first ten minutes
2. Symptoms and the tools for them
3. Stopping the damage
4. Fixing: roll forward
5. Security incidents
6. After the incident

The first ten minutes #

1. Confirm and scope. Request the front page and the administration login from outside. The front page may be served from a cache while the database is down, so the login page tells you more (Health checks and uptime).
2. Check the servers. ./console exp:velocity status for Velocity; the PHP-FPM service and the database server otherwise.
3. Read the newest errors. Setup > Log files, or ./console exp:logs; a visitor's error reference leads to the lines of their request (Finding an error by its reference).
4. Ask what changed. The audit trail records deploys, settings writes and cache clears with who and when: ./console exp:audit tail --allow-root-user.
5. Decide: stop the damage first, or fix directly?

Symptoms and the tools for them #

Stopping the damage #

• Maintenance mode takes the public site offline with a 503 and Retry-After, and brings it back with one command:

  php bin/php/maintenance.php on --message="Back at 06:00" --until=2h --allow-root-user
  php bin/php/maintenance.php off --allow-root-user

  (Maintenance mode).

• Switch off the extension or setting that the change introduced, through settings/override/ and an INI cache clear (and a Velocity restart).

• Restore from backup when data is damaged (Restore from backup).

Fixing: roll forward #

Fix forward with a new commit and, for a release, a new version. A published tag or release is never moved, replaced or re-cut, even when it is the cause; the next version's notes say what was wrong with the previous one (Version numbers and immutable tags). On an installation made of Git clones, you can use a fix before it is released (Develop with Git).

Security incidents #

• A vulnerability in Exponential itself is reported privately to the address in SECURITY.md, not in the public issue tracker. Include the version, the engine, the database, what an attacker can do and the steps to reproduce, and leave out passwords, secrets and personal data.
• Verify the audit trail's integrity (./console exp:audit verify --allow-root-user, exit status 1 when a chain is broken) before you rely on it as evidence.
• Rotate the secrets an attacker may have seen, from settings/override/.

After the incident #

• Write down the timeline, the cause, what detected it and what would have detected it earlier.
• Add the test that would have caught it (Testing strategy).
• If the cause was a behavior change, check that its upgrade notes in doc/eng-us/bc/6.0/ said what to do; if not, correct them (Changelogs and upgrade notes).
• File a public issue for a bug in the software, with the diagnostics the help's Report an error page prepares (When something goes wrong).

Related pages #

• Observability: logs, debug bar and audit trail
• Runbooks for hosts
• Troubleshooting
• Free documentation: this page is free to copy, translate and improve under the GNU FDL 1.3.