Kubernetes #

This page is for platform teams at hosting companies that want to run Exponential on Kubernetes. It answers what the project's sources say about Kubernetes, and what changes when a site runs in more than one replica: the database, shared files, sessions, cron, the proxy in front and code deploys. It does not give manifests: the sources ship none, and every cluster differs. Treat it as the list of decisions your manifests must encode.

Contents #

1. What the sources say
2. One replica or several
3. The database
4. Shared files
5. Sessions
6. Cron
7. Ingress and trusted proxies
8. Deploying code
9. Checklist

What the sources say #

• Kubernetes is tier 3, expected to work: "it runs the containers above", with the note "one writer for SQLite" (Tier 3). Tier 3 is "a reasoned expectation, not a test result".
• "Several replicas need a database server (PostgreSQL first ...) and shared storage for var/storage" (Containers, cloud and hosting panels).
• Everything on Containers: Docker and Podman applies: images, volumes, the entry point, secrets.

One replica or several #

A single replica with SQLite is the simplest Kubernetes deployment and stays within the one-writer rule. Plan the move to several replicas as its own project.

The database #

In the official order: 1. SQLite 3 (the default) · 2. PostgreSQL · 3. MongoDB · 4. MariaDB · 5. Oracle · 6. MySQL. Microsoft SQL Server is known to be possible with additional work at this time.

With several replicas, SQLite is out: "SQLite's WAL needs shared memory on one host, so several replicas cannot share one database file over a network volume." The sources recommend 2. PostgreSQL first; every server database in the list has "Several web servers on one database: yes" (At a glance). Under Velocity each worker keeps its own connection, so the database must accept workers times replicas connections.

Shared files #

var/storage holds uploads and image variations; every replica must see the same files. Two approaches exist in the sources:

1. A shared file system mounted in every replica (a volume that several pods can write), with the default file handler eZFSFileHandler (settings/file.ini [ClusteringSettings] FileHandler).
2. The cluster file handler eZDFSFileHandler, described in file.ini as "distributed filesystem handler with a DB overlay. Required for NFS based architectures"; its [eZDFSClusteringSettings] name the mount point (MountPointPath) and a database backend for the file metadata.

Clustering is covered in Clustering and shared storage and Several web nodes.

Caches (var/cache, var/<site>/cache) can be shared or local per replica; a local cache means a cache clear must reach every replica. Per-site log and cache paths (LogVarDir, CacheVarDir) let you put logs and caches on other file systems than storage (Logs, INI cache and expiry timestamps per site).

Sessions #

By default site.ini [Session] Handler is empty, which means ezpSessionHandlerPHP: "PHP keeps them in its own storage (files)" (Sessions). Files on one pod are invisible to the next, so a visitor signed in on one replica is a stranger on another. Choose one:

• Handler=ezpSessionHandlerDB, which keeps sessions in the ezsession table of the shared database; or
• a PHP session save handler that every replica shares (configured in PHP, outside Exponential); or
• sticky sessions at the ingress, which hides the problem until a pod is replaced.

# settings/override/site.ini.append.php
[Session]
Handler=ezpSessionHandlerDB

Cron #

runcronjobs.php must run on one schedule for the site, not once per replica: several replicas running the frequent part at the same minute would send notifications and run workflows twice. A Kubernetes CronJob that starts a container from the same image, with the same volumes, database settings and user, matches the documented crontab (Chapter 10, Cronjobs):

*/5   * * * *   php runcronjobs.php -q -s site frequent
*/15  * * * *   php runcronjobs.php -q -s site
17    * * * *   php runcronjobs.php -q -s site infrequent

Velocity's own scheduler "does not replace the system cron for these parts out of the box".

Ingress and trusted proxies #

Behind an ingress controller or load balancer, Exponential must know which addresses may tell it the visitor's address and protocol:

• Under PHP-FPM, list the proxies in site.ini [HTTPHeaderSettings] TrustedProxies[] (shipped: 127.0.0.1 and ::1); set ClientIpByCustomHTTPHeader only together with that list (13.10 Behind a proxy).
• Under Velocity, configure the trusted proxies in Velocity's own list, Q.webserver.proxy.trusted; "Configure proxies in front of Velocity there" (Trusted proxies).
• "Make sure the application server cannot be reached around them".

Deploying code #

Velocity's workers keep the application in memory; a code change reaches them only after a restart. In Kubernetes the natural unit is a new image rolled out to new pods. Two cautions from the sources:

• Schema changes of an upgrade (the update files in update/database/, one directory per engine) run once against the shared database, not per pod (Upgrading customer sites).
• After the rollout, clear the caches that the replicas share (Cache clearing and warming).

Every pod needs the same config.php, for example the CLUSTER_* constants of a shared cluster storage. Make it in the image or in the entry point with exp:setup:config --apply and exp:config:set ... --apply, with passwords from the pod's secrets (--value="$CLUSTER_DB_PASS"), never in the image. exp:config:list --only-set --format=json, run in each pod, shows whether they agree (Setting up .htaccess and config.php from the console).

Checklist #

• To do: One replica with SQLite, or several with a database server (PostgreSQL first).
• To do: var/storage shared by every replica, or the cluster file handler configured.
• To do: Sessions shared (ezpSessionHandlerDB or a shared PHP handler).
• To do: Exactly one scheduler for runcronjobs.php.
• To do: Trusted proxies set for the ingress, in site.ini or in Velocity's list.
• To do: Rollout procedure covers schema updates and cache clearing.
• To do: A test installation on the cluster before any customer site, because this is tier 3.

Related pages #

• Containers: Docker and Podman
• Several web nodes, Clustering and shared storage
• Varnish and reverse proxies
• Setting up .htaccess and config.php from the console
• Free documentation: this page is free to copy, translate and improve under the GNU FDL 1.3.