10. After installing: the operations handbook #

The installer leaves you with a site that answers its first request. This chapter turns it into a site you can run: securing the administrator account, getting the site addresses right, scheduling the cronjobs that publish, notify and clean up, sending real mail, knowing which cache to clear and when, keeping the class autoloads and the search index current, reading and rotating the logs, taking backups you have actually restored, watching the site, tuning it for speed, managing extensions and their order, and adding languages. Each part explains what the mechanism does, gives the commands with the settings they read, and ends with a way to check the result. Work through it once in order after the install; afterwards it serves as a reference.

Contents · Previous: 9. Databases · Next: 11. Upgrading

Contents #

• Conventions in this chapter
• 10.1 First login and the administrator password
• 10.2 Siteaccesses and site addresses
• 10.3 Cronjobs
• 10.4 Mail: the file transport and real sending
• 10.5 Notifications and newsletters
• 10.6 Caches
• 10.7 Class autoloads
• 10.8 Search indexing and image handling
• 10.9 Backups and restore
• 10.10 Logs and log rotation
• 10.11 Monitoring and health checks
• 10.12 Performance tuning
• 10.13 Extension management
• 10.14 Multi-language sites
• 10.15 A checklist for the first week
• 10.16 When something does not work
• References

Conventions in this chapter #

• Every command runs from the installation root, the directory that holds index.php.
• php bin/php/console is the command console; many installations link it as ./console. php bin/php/console list shows every command, php bin/php/console list cron one namespace.
• Scripts refuse to run as root unless given --allow-root-user. Run routine work, and above all cron, as the site's own user; then the flag is not needed and files are created with the right owner. The examples carry the flag so they can be copied as they are. bin/php/ezpgenerateautoloads.php has no root check of its own; it accepts the flag so the same habit works there too (10.7).
• Files created by root belong to root. A cache or log file that root created and the web server's user cannot overwrite is the most common cause of errors after maintenance work (chapter 12). When you must work as root, run the command as the site user instead: sudo -u <site user> php ....
• Settings are never edited in settings/*.ini. Overrides go to settings/override/<file>.ini.append.php (the whole installation) or settings/siteaccess/<name>/<file>.ini.append.php (one siteaccess). php bin/php/console exp:ini reads and writes them in every scope (exp:ini); after a hand edit run php bin/php/ezcache.php --clear-tag=ini --allow-root-user.
• site and admin stand for your public and administration siteaccess names (the defaults of exp:install).

10.1 First login and the administrator password #

1. Find the address and the password. The installer's summary printed the admin login address (with the default URL access, /admin/user/login below the site address) and the user, admin. A generated password was shown once and written to var/log/initial-admin-password, readable by its owner only.

2. Sign in and change the password at once: in the administration, Change password in the user menu, or directly /admin/user/password. The page lists the requirements beside the field: at least [UserSettings] MinPasswordLength characters (shipped 10, settings/site.ini), not a well-known or common password (WeakPasswordCheck, since 6.0.15) and the [PasswordSettings] rules you switched on. Choose a long, random password. The user gets an e-mail that the password was changed (ChangeNotificationMail), which is also the first test of your mail settings (10.4).

3. Remove the password file once you have signed in with the new password:

   rm var/log/initial-admin-password

4. Give the administrator a real e-mail address (edit the administrator's user object in the administration), so the password reset by mail reaches someone.

A lost password is reset on the command line. As the operating system's root user no other login is needed:

php bin/php/resetuserpassword.php -u admin -g --allow-root-user

-u names the user (default admin); -g generates a password (-l <length>, default 16) and prints it; -p <password> sets one; -a <login> with -ap <password> authorises the reset with another administrator's login instead, for when you are not the operating system's root. A password given with -p is visible in the process list and the shell history; prefer -g and change the password after logging in. See Reset a user password and Changing your password. Every way to change it, the default per install method, and how to retire the admin login: Change the administrator password, The default administrator password.

10.2 Siteaccesses and site addresses #

A siteaccess is one way into the installation: a name, a set of settings in settings/siteaccess/<name>/, and a rule for which requests it answers. The installer creates at least the public siteaccess and the administration. The rules live in site.ini [SiteAccessSettings]:

In site.ini [SiteSettings] of each siteaccess:

The installer chose one of three layouts (exp:install --access=url|host|port):

• URL (/admin): one host name for everything; simplest behind any web server.
• Host (admin.example.com): a host name per siteaccess; best for production and for HTTPS certificates per host.
• Port (:8081): one port per siteaccess; useful for local work.

To change from URL to host access after the install, for example:

# settings/override/site.ini.append.php
[SiteAccessSettings]
MatchOrder=host;uri
HostMatchType=map
HostMatchMapItems[]
HostMatchMapItems[]=www.example.com;site
HostMatchMapItems[]=admin.example.com;admin

and in settings/siteaccess/site/site.ini.append.php and settings/siteaccess/admin/site.ini.append.php set [SiteSettings] SiteURL=www.example.com and admin.example.com. Clear the INI cache, add the host names to the web server (chapter 8) and to DNS, and check which siteaccess an address reaches. The administration asks for a login and the public site does not, which makes the difference easy to see:

php bin/php/ezcache.php --clear-tag=ini --allow-root-user
curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' https://admin.example.com/
curl -s -o /dev/null -w '%{http_code}\n' https://www.example.com/

Expect a redirect to .../user/login (or the login page itself) from the admin host and 200 from the public host. If both answer the same, the host map does not match: the host name the browser sends must be exactly the one in HostMatchMapItems[], without a port and in lower case.

Behind a proxy or load balancer. The scheme (http or https) and the host name the visitor asked for are taken from forwarded headers (X-Forwarded-Proto, X-Forwarded-Host, ...) only when the request comes from an address in site.ini [HTTPHeaderSettings] TrustedProxies[] (shipped: 127.0.0.1 and ::1, a proxy on the same machine); the visitor's address is read from X-Forwarded-For only when, in addition, ClientIpByCustomHTTPHeader=X-Forwarded-For is set. Anyone can send these headers, which is why they are believed only from the proxies you list. A proxy on another machine must be added, or the site treats HTTPS requests as plain HTTP and records the proxy's address as every visitor's:

# settings/override/site.ini.append.php
[HTTPHeaderSettings]
TrustedProxies[]
TrustedProxies[]=127.0.0.1
TrustedProxies[]=::1
TrustedProxies[]=10.0.0.5
ClientIpByCustomHTTPHeader=X-Forwarded-For

The empty TrustedProxies[] line replaces the shipped list, so repeat the loopback entries when you still need them. See chapter 8 and Trusted proxies.

Settings per site inside an extension, for hosting several sites from one installation, are explained in Per-site settings inside extensions.

10.3 Cronjobs #

Cronjobs do the work no visitor waits for: they publish and hide content on schedule, send notifications, resume content jobs, run the audit trail's maintenance, import feeds, index content and clean up. Without them a site works, but scheduled content never appears, notifications never go out and caches and drafts pile up on disk.

How runcronjobs.php works #

runcronjobs.php runs parts: named lists of scripts in settings/cronjob.ini. Extensions add their own parts and directories through [CronjobSettings] ExtensionDirectories[]. Run without a part name, it runs the scripts of [CronjobSettings] Scripts[], the default part.

php runcronjobs.php --list --allow-root-user                       # every part and its scripts
php runcronjobs.php -s site frequent --allow-root-user              # one part
php runcronjobs.php -s site --script=notification.php --allow-root-user   # one script
php runcronjobs.php -s site --allow-root-user                       # the default part

-s (--siteaccess) chooses the siteaccess whose settings the scripts read; use the public siteaccess unless a part says otherwise (the Oracle parts want the administration). -q suppresses output except errors. In --list each script appears once per directory searched; that repetition is normal. Every part also has a console alias, php bin/php/console cron:<part> (for example cron:frequent), and the administration runs any part from Setup > Cronjobs, where the output follows while it runs and the page proposes crontab lines. Parts listed in [AdminSettings] ForbiddenParts[] (shipped: cluster_maintenance, unlock) cannot be started from there. MaxScriptExecutionTime (shipped 43200 seconds) bounds a script's run.

The parts and their scripts #

The kernel's parts (settings/cronjob.ini):

Single parts exist for contentjobs, audit, notification, mailpreferences and mailbounces, so each can run on its own schedule. cronjobs/ holds more scripts that no shipped part lists, to be run with --script= or added to a part of your own: session_gc.php (expired sessions, see below), old_drafts_cleanup.php, trashpurge.php (empties the trash, or with content.ini [TrashSettings] KeepItemsForDays only what is older), staticcache_cleanup.php, subtreeexpirycleanup.php and updateviewcount.php.

Active extensions add more parts; php bin/php/console list cron shows them with a description, for example the newsletter parts (10.5), the sitemap parts of bcgooglesitemaps and xrowmetadata, and the Oracle parts (chapter 9).

A crontab #

Install it in the crontab of the site user (crontab -e -u <site user>, or php bin/php/console crontab:edit; crontab:list shows the current one):

# m   h  dom mon dow  command
*/5   *  *   *   *    cd /path/to/installation && php runcronjobs.php -q -s site frequent >> var/log/cron-frequent.log 2>&1
*/15  *  *   *   *    cd /path/to/installation && php runcronjobs.php -q -s site >> var/log/cron-default.log 2>&1
17    *  *   *   *    cd /path/to/installation && php runcronjobs.php -q -s site infrequent >> var/log/cron-infrequent.log 2>&1
40    3  *   *   *    cd /path/to/installation && php runcronjobs.php -q -s site cache_cleanup >> var/log/cron-cleanup.log 2>&1
50    3  *   *   *    cd /path/to/installation && php runcronjobs.php -q -s site --script=session_gc.php >> var/log/cron-cleanup.log 2>&1

• Use the same PHP binary the web server uses (/opt/.../php on hosts with several PHP versions); php -v in the cron environment may differ from your shell's.
• frequent every few minutes is what notifications and content jobs expect; infrequent hourly or daily.
• session_gc.php removes sessions older than site.ini [Session] SessionTimeout (shipped 31104000 s, 360 days), where the operating system's PHP session cleanup does not run. The same by hand: php bin/php/ezsessiongc.php --allow-root-user.
• Rotate the cron-*.log files with the system's logrotate (10.10).

Check that the jobs run: the files grow, Setup > Cronjobs shows the last output, and the notification status command reports the last run (10.5).

Velocity has a scheduler of its own that runs handler functions at times or intervals while the server runs. It does not replace the system cron for these parts out of the box: it needs a handler you write, and it has no catch-up for times missed while the server was down.

10.4 Mail: the file transport and real sending #

Exponential sends mail for password changes, registrations, forms, notifications and newsletters. The kernel's transport is set in site.ini [MailSettings]:

The file transport writes every mail as a file instead of sending it: Transport=file puts them in var/log/mail/ (or the directory of [MailSettings] FileTransportDirectory), one <time>-<random>.mail file per message. Use it on a test or staging site, to see exactly what would go out without anyone receiving it:

ls -lt var/log/mail/ | head

Real sending. For production, use SMTP to your organisation's mail server or a relay service:

# settings/override/site.ini.append.php
[MailSettings]
Transport=SMTP
TransportServer=smtp.example.com
TransportPort=587
TransportConnectionType=tls
TransportUser=<smtp user>
TransportPassword=<smtp password>
SenderHost=www.example.com
AdminEmail=[email protected]
EmailSender=[email protected]

Mail that should reach real inboxes needs a sender address of a domain you control, with SPF (and preferably DKIM and DMARC) records that allow the sending server, for every address family the server sends from: if the server sends over IPv6 and the SPF record lists only IPv4 addresses, large providers reject the mail. Send a test by changing a test user's password, or by a form with an e-mail receiver, and read the mail server's log.

Some extensions have their own transport settings. The newsletter extension's are described in 10.5; the e-mail preference system (categories, consent, suppression list, the bounce mailbox) in E-mail preferences: setting them up and running them. php bin/php/console exp:mailstatus shows the preference system's state.

10.5 Notifications and newsletters #

Notifications #

Users subscribe to subtrees; publishing creates events; the notification.php script (in frequent, and as the part notification) turns events into mails and digests. The settings are in settings/notification.ini.

php runcronjobs.php -s site notification --allow-root-user          # run it once by hand
php bin/php/console exp:notificationstatus                          # pending events, last runs, problems
php bin/php/console exp:notificationrun --dry-run                   # who would get what, nothing sent

The status page in the administration and the status command flag the usual faults: events waiting with no run recorded (cron is not set up), a failed run, an overdue digest, the file transport still active, a missing sender. The complete procedure, with each fault and its fix, is Notifications: running them and fixing problems.

Newsletters (cjw_newsletter) #

When the newsletter extension cjw_newsletter is active, sending is a two-step job run by cron:

*/10  *  *   *   *    cd /path/to/installation && php runcronjobs.php -q -s site cjw_newsletter >> var/log/cron-newsletter.log 2>&1
25    *  *   *   *    cd /path/to/installation && php runcronjobs.php -q -s site cjw_newsletter_mailbox >> var/log/cron-newsletter.log 2>&1

The extension has its own transports, in cjw_newsletter.ini [NewsletterMailSettings] (override in settings/override/cjw_newsletter.ini.append.php):

So a fresh installation simulates newsletter sending: the editions and confirmations land as files in var/log/mail/, which lets you test subscription, double opt-in, sending and unsubscribing without a mail server and without real recipients. To send for real, set the methods to smtp (with the SmtpTransport* keys) or sendmail. The directory must be writable by the web server user and by the cron user. Deliverability, throttling and bounce handling are documented with the extension and in the cjw_newsletter specification.

10.6 Caches #

Exponential caches at many levels, and most "my change does not show" questions are cache questions. The rule is: clear the smallest cache that explains the problem, and clear caches in front of PHP only after PHP runs the new code.

The caches by id and tag #

bin/php/ezcache.php clears by id (one cache) or tag (every cache that carries it). The list below is what --list-ids --verbose prints on a default installation, with the tags from kernel/classes/ezcache.php and the extensions' [Cache_*] blocks:

The tags are therefore content, template, ini, codepage, i18n, image, user and rest.

php bin/php/ezcache.php --list-ids --verbose --allow-root-user
php bin/php/ezcache.php --clear-tag=ini --allow-root-user
php bin/php/ezcache.php --clear-id=template-override,design_base --allow-root-user
php bin/php/ezcache.php --clear-all --allow-root-user
# remove the files for real (saves disk space), only those older than two days:
php bin/php/ezcache.php --clear-id=content --purge --expiry="-2 days" --allow-root-user

Without --purge a clear only moves an expiry timestamp (or moves a directory aside), so it is instant; the cache_cleanup cron part removes what can never be served again. -s <siteaccess> clears the cache of another siteaccess's var directory.

Which to clear #

The caches in front of PHP #

php bin/php/cache.php (console: exp:cache) does every Setup > Cache action from the command line, ends with PASS or FAIL, accepts --dry-run and --json, and also reaches the caches ezcache.php does not:

php bin/php/cache.php list --sizes --allow-root-user     # every cache, its tag, size and how it is cleared
php bin/php/cache.php status --allow-root-user           # static, HTTP, query, Velocity, precompress, PHP caches
php bin/php/cache.php velocity clear --allow-root-user   # Velocity's response cache
php bin/php/cache.php httpcache status --allow-root-user
php bin/php/cache.php opcache status --allow-root-user

Velocity's response cache answers anonymous repeat requests in the server's parent process, without PHP. After a template or stylesheet change clear it (no restart needed):

php bin/php/console exp:velocity cache clear --apply --allow-root-user
php bin/php/console exp:velocity cache stats --allow-root-user

The order after a code change #

A page rendered by the old code between a cache clear and a PHP reload is cached again and looks as if the change had not worked. The order is: INI and template caches, then reload PHP-FPM and restart Velocity, and only then the content, HTTP, template-block and response caches. Under Velocity one command does it, with PASS or FAIL per step and a stop at the first failure:

php bin/php/console exp:velocity deploy --dry-run --allow-root-user   # what it would do
php bin/php/console exp:velocity deploy --allow-root-user             # do it
php bin/php/console exp:velocity deploy --kernel --allow-root-user    # a kernel class was added or renamed

--packer also clears the packed scripts and styles; --no-fpm, --no-velocity and --no-autoload skip steps. The PHP-FPM service it reloads is velocity.ini [DeploySettings] PhpFpmService (auto finds it). See Velocity engines: deploying a PHP change.

10.7 Class autoloads #

Exponential finds classes through generated autoload arrays in autoload/ (the kernel) and var/autoload/ (the extensions). A class that is not in an array is not found, even if its file is in place: "Class ... not found" right after adding an extension or a class is the classic symptom.

php bin/php/ezpgenerateautoloads.php -e        # extensions: after adding, renaming or removing a class
php bin/php/ezpgenerateautoloads.php -k        # the kernel: only when kernel/ or lib/ changed
php bin/php/ezpgenerateautoloads.php -o        # kernel overrides
php bin/php/ezpgenerateautoloads.php -e -n     # dry run: report, write nothing
php bin/php/ezpgenerateautoloads.php -e -p     # with progress output

The script has no root check; it accepts --allow-root-user (without effect) so that the flag can be given as with every other script. An option it does not know stops it with exit status 1 and nothing written:

$ php bin/php/ezpgenerateautoloads.php -e --bogus
The referenced parameter '--bogus' is not registered.

Older releases printed their usage and exited with 0 on an unknown option, --allow-root-user included, so a deployment script never noticed that nothing had been generated. If a deployment script of yours ran the generator and checked nothing, check it once by hand now. Run it as the site user: the arrays it writes in var/autoload/ must stay writable for the next run.

Which directories it walks is limited by .autoloadignore in the installation root, one directory per line, anchored at the root, # for comments. The shipped file excludes vendor (Composer resolves those classes itself), ai, var and .claude (working copies, which would otherwise map every kernel class to a copy). Add a directory there when it holds PHP files that are not application classes; --exclude does the same for one run.

After regenerating, clear the INI cache and, under Velocity, restart the workers: they keep the arrays they loaded when they started. exp:velocity deploy (with --kernel for the kernel array) does all three. php bin/php/console exp:checkclasses loads every declared class and reports those PHP refuses to load.

10.8 Search indexing and image handling #

Search #

The built-in engine is eZSearchEngine (site.ini [SearchSettings] SearchEngine); it keeps its index in the database. Content is indexed when it is published, unless DelayedIndexing is enabled (or classbased with DelayedIndexingClassList[]): then publishing only queues it and the default cron part's indexcontent.php indexes it later, which makes publishing faster for large objects.

The queue is the table ezpending_actions: index_object reindexes the object, index_moved_node the object and every object below its main node (for a search engine that keeps paths, such as ezfind's Solr). Each row carries the time it was queued (created); the cronjob takes the oldest first and removes a row once it is indexed. What an operation queues with delayed indexing:

Rebuild the whole index after an import, a database move or when search results are wrong:

php bin/php/updatesearchindex.php --clean --allow-root-user      # --clean removes the old index first

Search statistics (LogSearchStats) are disabled by default, because they store every phrase visitors type, which can be personal data. For a Solr server, php bin/php/console exp:solr controls it once installed and configured.

Images #

Uploaded images live in var/<site var dir>/storage/images; their variations (aliases) are generated on first use. The rules are in settings/image.ini:

• [AliasSettings] AliasList[] names the aliases (reference, small, tiny, medium, large, rss and those of your design), each with a block of filters.
• [ImageConverterSettings] ImageConverters[] lists the converters in order: GD first (PHP's own extension, no external program per image), ImageMagick as the fallback for formats GD cannot read (PSD, TIFF, PDF, WebP) and filters only it has. [ImageMagick] ExecutablePath and Executable (convert) find the program; check with convert -version.
• [OutputSettings] AllowedOutputFormat[] (JPEG, PNG, WebP, GIF) lists the formats pages may show.
• [ImageConverterSettings] LockTimeout (shipped 60): seconds a process waits while another generates the same alias; after that the waiting process ends the other one and generates the alias itself.

After changing an alias definition, remove the generated variations so they are made again:

php bin/php/cache.php imagealias clear --allow-root-user

10.9 Backups and restore #

A backup is the database and the files, taken at the same moment. Files that matter:

Caches (var/cache, var/<site>/cache) need no backup: they are regenerated.

Procedure #

1. Put the site in maintenance mode, so no write happens between the database and the file backup. Visitors get a 503 page; --allow-admin keeps the administration reachable, --allow-ip lets given addresses through:

   php bin/php/maintenance.php on --message="Back soon" --until=30m --allow-root-user
   php bin/php/maintenance.php status --allow-root-user

2. Back up the database with the tool of its engine:

   Engine	Backup	Restore
   SQLite	sqlite3 var/storage/sqlite3/exponential.db ".backup '/backup/db.sqlite'"	sqlite3 var/storage/sqlite3/exponential.db ".restore '/backup/db.sqlite'"
   MySQL, MariaDB	mysqldump --single-transaction -u USER -p DATABASE > /backup/db.sql	mysql -u USER -p DATABASE < /backup/db.sql
   PostgreSQL	pg_dump -U USER -Fc DATABASE > /backup/db.dump	pg_restore -U USER -d DATABASE --clean /backup/db.dump
   MongoDB	mongodump --uri="mongodb://USER@HOST/DATABASE" --out=/backup/mongo	mongorestore --uri="..." --drop /backup/mongo
   Oracle	php bin/php/console ext:ezoracle:datapump --export=db.dmp (or expdp)	ext:ezoracle:datapump --import=db.dmp

   -p makes the client ask for the password; never write it on the command line. For SQLite read the details in chapter 9: the online backup is safe while the site runs, a plain copy of a busy file is not.

3. Back up the files:

   tar czf /backup/files-$(date +%F).tar.gz var/storage settings/override settings/siteaccess

   (add var/<site>/storage when the site's var directory is separate, and your own extensions).

4. Switch maintenance off: php bin/php/maintenance.php off --allow-root-user.

Store backups away from the server, and keep several generations. On SQLite and MySQL with InnoDB the database backup is consistent without maintenance mode; maintenance mode makes the database and the files agree.

Restore #

1. Maintenance on. Under Velocity also stop the server (php bin/php/console exp:velocity stop --allow-root-user), so no worker holds the old database.

2. Restore the database (table above) and unpack the files over the installation; fix the owner and mode (chapter 8).

3. Clear every cache and regenerate the autoloads:

   php bin/php/ezcache.php --clear-all --allow-root-user
   php bin/php/ezpgenerateautoloads.php -e

4. Start Velocity, clear its response cache, switch maintenance off, and open the front page and the administration.

Test a restore on another machine at least once and after every major upgrade: a backup that was never restored is a hope, not a backup. See Maintenance mode and Operating a site, part 6.

Maintenance windows #

For an upgrade, an import or a restore, take the public site offline from Setup > Maintenance (message, expected duration, who still gets in, a preview and a confirmation) or with php bin/php/maintenance.php on|off|status. Visitors get a 503 maintenance page, the administration under /admin stays reachable, and deleting var/maintenance.json ends it whatever else is wrong. The full guide: Maintenance mode.

10.10 Logs and log rotation #

tail -n 50 var/log/error.log
php bin/php/audit.php status --allow-root-user
php bin/php/audit.php tail --channel=access --lines=20 --allow-root-user

Built-in rotation. The kernel rotates its own logs: when a file passes 200 KB it becomes .1, the older ones move up, and three are kept (error.log.1 to error.log.3). Change the limits in config.php, the installation's PHP configuration file (config.php-RECOMMENDED lists them):

define( 'EZPUBLISH_LOG_MAX_FILE_SIZE', 10485760 );   // 10 MB per file
define( 'EZPUBLISH_LOG_ROTATE_FILES', 10 );          // keep ten; 0 switches built-in rotation off
define( 'CUSTOM_LOG_MAX_FILE_SIZE', 10485760 );       // logs written through eZLog
define( 'CUSTOM_LOG_ROTATE_FILES', 10 );

Without an editor, the console writes the same lines (a backup of config.php first, a php -l check before it lands); ./console exp:setup:config --apply creates config.php from the template when there is none yet:

./console exp:config:set EZPUBLISH_LOG_MAX_FILE_SIZE 10485760 --apply
./console exp:config:set EZPUBLISH_LOG_ROTATE_FILES 10 --apply
./console exp:config:list --only-set

See Setting up .htaccess and config.php from the console.

System rotation. Files the kernel does not write itself (your cron-*.log files) need the system's logrotate. If you switch the built-in rotation off, logrotate takes over the kernel's logs as well:

# /etc/logrotate.d/exponential
/path/to/installation/var/log/cron-*.log {
    weekly
    rotate 8
    compress
    missingok
    notifempty
    copytruncate
    su <site user> <site group>
}

copytruncate because the writers keep the file open. Never leave [DebugSettings] DebugOutput=enabled on in production: it shows internals to visitors and writes a lot.

10.11 Monitoring and health checks #

There is no single health URL; combine a check from outside with the status commands inside. Every command below ends with a recognisable status, and most have --json for a monitoring system.

A small script in the site user's crontab that runs these and mails on a non-zero exit status is enough for a single server. In the administration, Setup > System information shows versions, PHP and the server engine.

10.12 Performance tuning #

Measure before and after every change. exp:benchmark requests the front page, the first menu pages, the search page (when visitors may search) and the admin login page, always in that order, 100 times each after 5 warm-up requests, 4 at a time, and prints median, p90, p99, minimum, maximum and standard deviation; kernel mode times the parts of a page in-process (boot, render cold and warm, INI load, content fetch, database round trip, cache write and read), and micro mode times pure-PHP hot paths without HTTP or a database:

php bin/php/console exp:benchmark --save=var/benchmark/base.json --allow-root-user
# ... make a change ...
php bin/php/console exp:benchmark --baseline=var/benchmark/base.json --threshold=15 --allow-root-user
php bin/php/console exp:benchmark --runs=200 --warmup=10 --format=json --allow-root-user > var/benchmark/run.json
php bin/php/console exp:benchmark kernel --repeat=30 --allow-root-user
php bin/php/console exp:benchmark micro --allow-root-user                 # no web server, no database
php bin/php/console exp:benchmark --compare=https://www.example.com,https://www.example.com:8443 --allow-root-user
php bin/php/console exp:benchmark --cold --requests=50 --allow-root-user   # past the response caches

Every run records its environment (PHP, OPcache and JIT, Xdebug, CPUs, load, the commit, the date), and a comparison with a baseline made under another set-up says so. For numbers that compare: the same machine, nothing else running, OPcache as in production, Xdebug off, and judge two servers only by runs made back to back. The repository's CI runs the micro mode against tests/benchmark/baseline.json as a warning-only check.

It is polite by default: against another host than this machine more than 1000 requests per URL or 16 at a time needs --force. The guide Benchmarking Exponential teaches every use with real output: reading median, p90 and p99, comparing runs fairly, the noise checklist, workflows for a code change, Apache against Velocity and a deploy, and the CI check; Benchmark: exp:benchmark is the reference.

The levers, in order of effect #

1. Serve with Velocity and its response cache (chapter 8). A cached anonymous page is answered without PHP; persistent workers (ForkPerRequest=disabled) keep the application, the database connection and the compiled files warm.
2. Keep the content view cache and template-block cache on ([ContentSettings] ViewCaching=enabled, [TemplateSettings] TemplateCache=enabled, both shipped). Warm the site after a clear: php bin/php/warm.php --allow-root-user requests every published page.
3. OPcache (next part).
4. APCu for Velocity's response cache memory tier and the SQL query cache in shared mode: velocity.ini [PHPSettings] IniOptions[]=apc.shm_size=256M is shipped; too small and the two evict each other.
5. The SQL query cache (querycache.ini [QueryCacheSettings] Mode=request or shared), measured on your site (SQL query cache).
6. The database: chapter 9 (SQLite ANALYZE, Oracle persistent connections and DRCP).

Leave the static cache off. [ContentSettings] StaticCache=disabled is the shipped value, and the installer writes it so. With the static cache on, every publish regenerated every affected page on every public siteaccess over HTTP inside the editor's request, so a publish that changed nothing took seconds instead of a fraction of one. Velocity's response cache gives the same benefit to visitors without that cost.

OPcache #

Under PHP-FPM, enable OPcache in the pool's php.ini (OPcache configuration):

opcache.enable=1
opcache.memory_consumption=256
opcache.max_accelerated_files=20000
opcache.interned_strings_buffer=32
; keep timestamp checks on: Exponential rewrites PHP cache files (INI caches,
; compiled templates, expiry.php) under the same name
opcache.validate_timestamps=1
opcache.revalidate_freq=2

Never set opcache.validate_timestamps=0 on an Exponential site: a cache file rewritten after a clear would be served in its old form until PHP restarts.

Under Velocity, the server passes its own PHP options (settings/velocity.ini [PHPSettings] IniOptions[]), and two of them differ from PHP-FPM for a reason worth understanding. OPcache compares a file's time against the request's start time, and a Velocity worker is one long request that started when the worker did:

• opcache.revalidate_freq=0: with the default of 2 a worker never checked a file again, so a rewritten cache file was served as it was when the worker first compiled it.
• opcache.file_update_protection=0: OPcache does not keep a script changed less than this many seconds before the request's start. For a worker, every file written after it started (INI caches, override caches, compiled templates, all regenerated after a deploy or clear) counted as "too new" for the worker's whole life and was compiled again on every include: the cache served a few percent of includes and a render took about four times the CPU it takes under PHP-FPM. The protection is not needed because the server reads includes whole and drops the cached copy of a file changed in the last two seconds itself.

The shipped velocity.ini sets both, with opcache.enable_cli=1 (the command-line PHP that runs the server may have OPcache off system-wide), memory_consumption=256, max_accelerated_files=20000 and interned_strings_buffer=32. Check with php bin/php/cache.php opcache status --allow-root-user; read Velocity and the opcode cache for how to profile it. Any PHP setting that compares against "the request time" behaves as frozen in a Velocity worker; suspect that first when something works under PHP-FPM and not under Velocity.

10.13 Extension management #

Extensions live in extension/<name>/ (more roots with site.ini [ExtensionSettings] AdditionalExtensionDirectories[]). Most arrive through Composer (composer.json); some are copied in.

Activating. An extension does nothing until it is active:

# settings/override/site.ini.append.php
[ExtensionSettings]
ActiveExtensions[]=myextension

ActiveAccessExtensions[] in a siteaccess's site.ini.append.php activates one for that siteaccess only (loaded after the siteaccess is chosen). The administration does the same under Setup > Extensions. After activating:

php bin/php/ezpgenerateautoloads.php -e
php bin/php/ezcache.php --clear-tag=ini --allow-root-user
php bin/php/ezcache.php --clear-id=template-override,design_base,active_extensions --allow-root-user

and restart Velocity (or run exp:velocity deploy). If the extension brings database tables, install them as its documentation says (often share/db_schema.dba with bin/php/ezsqlinsertschema.php).

Order. When two extensions provide the same setting, template or design file, the earlier one in ActiveExtensions[] wins: each extension's settings directory is prepended in turn, so a later extension ends up with lower priority. settings/override/ always wins over every extension. A siteaccess's settings win over the defaults and over the extensions in its ActiveAccessExtensions[], but not over those in ActiveExtensions[], which are read after it (Configuring, section 2). [ExtensionSettings] ExtensionOrdering=enabled (shipped) also sorts extensions by the dependencies they declare in extension.xml, which cannot fix every case (it does not reorder across ActiveExtensions and ActiveAccessExtensions). So:

• a site or theme extension that should override another goes before it;
• an extension that only adds settings to one that ships a full file (for example connection settings for ezoracle) goes before that extension;
• change the order in Setup > Extensions, card Loading order, which saves only ActiveExtensions and keeps a backup copy (Extension loading order).

Check the effective value of a setting with php bin/php/console exp:ini get <file>/<Block>/<Key> --allow-root-user after every change of order. The installed versions, licenses and websites are listed on the about page (/ezinfo/about), which counts the extensions active in the current siteaccess and those on disk (for example "63 active in this siteaccess, 85 on disk"), the same figures as the dashboard. Building and releasing extensions of your own: Extensions.

The job queue and Exponential Intelligence. expqueue and expai are on disk after composer install and inactive until you switch them on:

1. ActiveExtensions[]=expqueue in settings/override/site.ini.append.php (the queue must be active wherever jobs are queued and run), then ./console ext:expqueue:install-schema.
2. ActiveAccessExtensions[]=expai in the site.ini.append.php of each admin siteaccess (or ActiveExtensions[]=expai for all), then php extension/expai/bin/php/expaischema.php --install -s <admin siteaccess>.
3. The provider key in settings/override/expai.ini.append.php (mode 0640, never committed): [AISettings] Enabled=true and ApiKey=..., or a local model (local models).
4. Autoloads, caches and exp:velocity deploy --packer as above; the policy expai/write for editors.
5. Cron, both with the admin siteaccess when expai is active there only: * * * * * ... php runcronjobs.php -q -s <admin siteaccess> expqueue and */5 * * * * ... php runcronjobs.php -q -s <admin siteaccess> expai.

Details: Exponential Intelligence, the job queue.

10.14 Multi-language sites #

Content languages and interface languages are separate:

• Content languages are the languages objects are translated into (the guide). Add one in the administration under Setup > Languages (the content/translations view); afterwards editors can translate objects into it.
• The interface language of a siteaccess is site.ini [RegionalSettings] Locale (shipped eng-GB); an administrator can choose their own in the user preferences.

Per siteaccess, [RegionalSettings] decides what a visitor sees:

# settings/siteaccess/site_de/site.ini.append.php
[RegionalSettings]
Locale=ger-DE
ContentObjectLocale=ger-DE
# the languages shown, in order of preference; the first that exists is used
SiteLanguageList[]
SiteLanguageList[]=ger-DE
SiteLanguageList[]=eng-GB
# disabled: objects in none of these languages are not shown
ShowUntranslatedObjects=disabled
TextTranslation=enabled

A typical multi-language site has one public siteaccess per language (site_en, site_de), matched by URL element or host name (10.2), and lists them all in RelatedSiteAccessList[] because they share one database. Do not change ContentObjectLocale of an existing site lightly: objects that do not exist in the new default language disappear from listings. At install time, give every language in --languages= (or Languages[] in kickstart.ini): a site package whose content uses a locale that is not installed shows pages without images or sub-items.

Interface translations: php bin/php/ezchecktranslation.php ger-DE --allow-root-user reports how complete one is, and after a change to a .ts file clear --clear-tag=i18n. See Translations and languages.

10.15 A checklist for the first week #

1. The administrator password is changed, var/log/initial-admin-password is gone, the admin's e-mail is real.
2. AdminEmail and the mail transport are set; a password-change mail arrived in a real inbox.
3. The crontab runs frequent, the default part, infrequent and cache_cleanup; the logs grow.
4. php bin/php/cache.php status --allow-root-user ends in PASS.
5. A backup was taken and restored on another machine.
6. var/log/error.log holds nothing you cannot explain.
7. A benchmark baseline is saved in var/benchmark/.
8. Logs rotate (built-in limits in config.php, see ./console exp:config:list --only-set; logrotate for cron logs).
9. Behind a proxy on another machine: its address is in TrustedProxies[], and HTTPS pages link to https://.
10. ./console exp:expfetch --allow-root-user shows the expected versions and engine, and its Hints block has no error (The system card).

10.16 When something does not work #

The faults that show up most often in the first weeks, with the part of this chapter that explains them. The full list is in chapter 12.

On a public demo (environment variable EXP_DEMO_MODE=1) the steps above about changing the administrator password do not apply: the published password is meant to stay, and in demo mode it cannot be changed or reset. Instead check that the Demo mode banner shows in the administration and that var/log/security.log has demo/mode lines for the web server and cron (Demo mode, Demo mode problems).

References #

In this repository:

• Operating a site: caches, cronjobs, static cache, preload, backups, logs, repairs.
• Getting started, Deploying, Security and audit, Extensions, Upgrading, the learning path.
• Notifications: running them and fixing problems, Notifications specification, E-mail preferences for administrators, cjw_newsletter specification.
• Cronjobs console, Commands, cronjob parts and views as classes, Content jobs, Velocity scheduler.
• Cache console, Cache clears that move directories aside, HTTP cache, SQL query cache, Velocity response cache, Static cache defaults, Preload Sites view.
• Velocity engines and exp:velocity deploy, Velocity persistent-worker server, Velocity engine settings, Velocity and the opcode cache.
• Benchmark, Maintenance mode, Reset a user password, Changing your password, Audit trail.
• Trusted proxies, exp:ini, Setting up .htaccess and config.php from the console, Extension loading order, Additional extension directories, Per-site settings inside extensions, Translations and languages.
• Repairing an installation, Glossary.
• The settings themselves: settings/site.ini, settings/cronjob.ini, settings/image.ini, settings/velocity.ini, settings/querycache.ini, .autoloadignore, config.php-RECOMMENDED.

External:

• PHP: OPcache configuration (including opcache.file_update_protection and opcache.revalidate_freq), APCu, session garbage collection.
• crontab(5), logrotate(8).
• Mail: SPF (RFC 7208), DKIM (RFC 6376), DMARC (RFC 9989, which replaced RFC 7489).
• Databases: the references of chapter 9.

Contents · Previous: 9. Databases · Next: 11. Upgrading