Observability: logs, debug bar and audit trail #

This page is for engineers who need to see what a running installation is doing: to debug a page, to investigate an incident, or to wire the system into monitoring. Exponential has no metrics endpoint of its own; what it has is structured logs, a log viewer with search, a debug bar, a tamper-evident audit trail and status commands.

Contents #

1. The signals at a glance
2. Logs
3. The log viewer and exp:logs
4. The debug bar
5. The audit trail
6. Health and status
7. An investigation, end to end
8. Privacy in the signals

The signals at a glance #

Logs #

• The debug logs and eZLog rotate when a file outgrows site.ini [FileSettings] LogFileMaxSize (2 MB), keeping LogFileRotateCount (3) archives (Log rotation and pruning).
• Not-found answers go to 404.log with the reason the kernel found; what the request classifier takes for a probe, a traversal, an injection, an attack tool or a burst of not-found answers goes to security.log. Each request is written to one log only, and nothing is blocked by the classification (Request logs).
• With several sites, logs can go to each site's directory or another file system (Logs, INI cache and expiry timestamps per site).
• Code writes through eZDebug::writeError(), writeWarning(), writeNotice() and eZLog, never with echo or var_dump (eZDebug and logging).

The log viewer and exp:logs #

Setup > Log files (setup/logs) lists every log of the installation, follows one live, searches one file, every kernel log or every file with their rotations, parses lines into time, client, siteaccess, request and message, groups lines by request, and links them to the audit records of the same minute. It is read-only by default. The same from a shell:

./console exp:logs --help
./console exp:logs --reference=ERR-0123456789

See The log viewer and Finding an error by its reference.

The debug bar #

The Exp Debug bar replaces the classic debug report with a pinned bar: page time, SQL count and time, peak memory, templates used, warnings and errors, and tabs for messages, settings, cache, timing, SQL, templates, included files, memory and Velocity. Show it only to your own address:

[DebugSettings]
DebugOutput=enabled
DebugByIP=enabled
DebugIPList[]=203.0.113.10

Behind a proxy, the address is only right when the proxy is listed in TrustedProxies[] (The Exp Debug bar, Debug bar guide).

The audit trail #

Security-relevant events (sign-ins and refusals, content changes, settings writes, cache clears, deploys, orders, VAT changes) are recorded in a hash-chained log with signed manifests, on by default. Alert rules watch for brute-force sign-ins, an administrator role granted, settings written out of hours, mass deletes and a broken chain.

./console exp:audit status --allow-root-user
./console exp:audit tail --channel=access --name='access.session.*' --lines=3 --allow-root-user
./console exp:audit verify --allow-root-user        # exit 1 when a chain is broken

The frequent cronjob group must run for indexing, alerts and rotation (The audit trail, Audit guide).

Health and status #

Exponential has no single health URL. Combine a check from outside (the front page and the administration login, because the front page may be answered from a cache while the database is down) with status commands on the server (Health checks and uptime, Monitoring overview). Maintenance mode answers 503 with Retry-After, so a planned window is distinguishable from an outage.

An investigation, end to end #

1. A visitor reports an error page with a reference: search it in the log viewer or with exp:logs --reference.
2. Read every line of the same request, then the audit records of the same minute: what changed just before?
3. Reproduce with the debug bar on for your address.
4. Check security.log if the request looks unusual.
5. Record what you found (Incident handling).

Privacy in the signals #

Log viewer, request logs and audit trail mask e-mail and IP addresses, tokens, keys, session ids and cookies by default; passwords and secrets are never recorded. Unmasking in the log viewer needs the policy setup/unmasklogs. Keep it that way when you export logs into another system.

Related pages #

• Debugging and logs
• Log viewer and rotation, for hosts
• Security and audit
• Free documentation: this page is free to copy, translate and improve under the GNU FDL 1.3.