Runbook: compromised site #

For the host's security lead and system administrators when there are signs that an attacker changed a customer's Exponential site: defaced pages, unknown administrator accounts, files that should not exist, spam sent from the site, or an alert from the audit trail. This is a severity 1 incident (Support tiers).

Contents #

1. Contain
2. Preserve evidence
3. Investigate
4. Eradicate and rebuild
5. Recover
6. Report and notify
7. Prevent a repeat

Contain #

1. Take the public site offline with maintenance mode, letting only your own addresses in:

   php bin/php/maintenance.php on --message="Maintenance" --allow-ip=<your address> --allow-root-user

2. If the attacker may hold an administrator account or session, restrict the administration at the network level now (Restricting the administration).

3. If the machine itself may be compromised (unknown processes, changed system files), isolate it at the network level and work from a copy.

4. Do not delete anything yet.

Preserve evidence #

• Copy the current files and a database dump to a place the site's user cannot write.
• Copy the logs: var/log/ (with error.log, security.log, auth.log, audit/), Velocity's var/vc/qbix/log/, and the web server's and PHP-FPM's logs.
• Note the time of every action you take.

Investigate #

The principle behind the file check, from 13.11: "If the PHP process can change only var/ and settings/, and the web server refuses to run anything below the root, a flaw in an extension cannot plant code."

Eradicate and rebuild #

1. Code from a known-good source: a fresh checkout of the version's tag and composer install, plus the custom extensions from their repositories. Do not trust files from the compromised tree.
2. Data from a clean backup: the newest backup from before the attacker's first change, found during the investigation (Restore from backup). If no clean backup exists, review the current database for unknown accounts, roles and content before keeping it.
3. Update Exponential and extensions to the newest 6.0.x before reopening, if the hole may be a known one (Upgrading customer sites).

Recover #

1. Credentials: change the administrator passwords; the database password (and update settings/override/site.ini.append.php); API keys and tokens the site holds; SFTP and panel passwords.
2. Sessions: a password change signs out every other session of that user ([PasswordSettings] EndOtherSessions=enabled); after a suspected compromise also remove all sessions (Setup > Sessions, with the database session handler) (Security hardening).
3. Accounts: remove accounts and role assignments the attacker created.
4. Caches: clear everything, so no cached page from the compromised state is served: php bin/php/ezcache.php --clear-all --allow-root-user, and with Velocity restart it and clear its response cache.
5. Verify, then switch maintenance mode off.

Report and notify #

• To the customer: what happened, what data may be affected, what you did.
• To the project, if a vulnerability in Exponential or one of its extensions was the cause: privately, as SECURITY.md describes, never in a public issue (Security updates and disclosure).
• To authorities and affected people, where the law requires it: the customer is usually the controller of the personal data and decides with their legal adviser; you support them with facts and logs (Compliance and GDPR).

Prevent a repeat #

• Go through the Security review checklist.
• Make the code read-only for the site's user (13.11).
• Forward the audit trail off the machine with a sink ([AuditSink_syslog], [AuditSink_webhook]), "so a compromised machine cannot quietly rewrite its own history" (Security hardening).
• Consider request-shield in front (request-shield and request rules).

Related pages #

• Hardening for hosts
• Restore from backup
• Chapter 13: Security hardening
• Free documentation: this page is free to copy, translate and improve under the GNU FDL 1.3.