Clustering and shared storage #

This page is for system administrators who design or run an Exponential offer with more than one web server, and for product managers who need to know what such an offer requires. It answers: how several servers share uploaded files and caches, what the DFS cluster needs (a shared mount and a metadata database), how files are served, which commands maintain the cluster, and where the limits are.

The facts come from settings/file.ini, config.php-RECOMMENDED, the cluster documents in doc/eng-us/features/ and the migration chapters of the installation book. The other half of a multi-server setup (sessions, caches in front of PHP, proxies) is on Several web nodes.

Contents #

1. Two file handlers
2. How the DFS cluster works
3. Setting it up
4. Serving files: index_cluster.php
5. Maintenance commands
6. What to watch
7. Checklist

Two file handlers #

file.ini [ClusteringSettings] FileHandler decides how Exponential reads and writes the files of var/:

The older handlers eZDBFileHandler and eZFS2FileHandler were removed; a site that still uses them is moved as described in chapter 14, the cluster.

How the DFS cluster works #

From How-to configure eZDFS cluster:

• The reference copy of every clustered file is on an NFS export, mounted at the same path on every web server.
• Its metadata (name, size, modification time, expiry) is kept in a database table, ezdfsfile.
• Cache files are copied to the local disk of the server that uses them.
• Images and binary files requested by a browser are streamed from the mount.
• The var directories must not be shared between the servers: each keeps its own, and the cluster handler keeps them in step with the central copy.

The metadata database can be the site's own database server or a separate one; the document notes that "for large scale websites, a dedicated MySQL server can really improve performances".

Database backends #

Source: 15.17 Cluster and DFS and the backends listed in config.php-RECOMMENDED (dfsmysqli, dfspostgres, dfsoracle). The storage side can be replaced as well: [eZDFSClusteringSettings] DFSBackend names a class that implements eZDFSFileHandlerDFSBackendInterface (Configurable DFS cluster backend).

Setting it up #

1. Create the table. Its definition is in the MySQLi backend class and printed in the eZDFS document (CREATE TABLE ezdfsfile ..., InnoDB).

2. Mount the export on every web server at the same path, writable by the site's user.

3. Configure the handler in the global override settings/override/file.ini.append.php. Cluster settings must be in that global file, not in an extension or a siteaccess, because of the order in which settings are read:

   # settings/override/file.ini.append.php
   [ClusteringSettings]
   FileHandler=eZDFSFileHandler
   
   [eZDFSClusteringSettings]
   MountPointPath=/path/to/nfs
   DBBackend=eZDFSFileHandlerMySQLiBackend
   DBHost=<cluster database host>
   DBPort=3306
   DBName=<cluster database>
   DBUser=<user>
   DBPassword=<password>

   The shipped file also has DBConnectRetries=3, DBExecuteRetries=20 and MaxCopyRetries=5.

4. Move the existing files into the cluster once: php bin/php/clusterize.php (see below).

5. Point the rewrite rules for files at index_cluster.php and give it its constants (next section).

On an upgrade, apply the dbupdate-cluster-... file of every step that has one (11.3 The update files).

Serving files: index_cluster.php #

Since 4.7 one file in the document root, index_cluster.php, serves cluster files over HTTP (Revamped cluster index). It reads config.php and, when present, config.cluster.php, and expects the CLUSTER_* constants that config.php-RECOMMENDED documents: CLUSTER_STORAGE_BACKEND (for example dfsmysqli), CLUSTER_STORAGE_HOST, CLUSTER_STORAGE_PORT, CLUSTER_STORAGE_USER, CLUSTER_STORAGE_PASS, CLUSTER_STORAGE_DB and CLUSTER_MOUNT_POINT_PATH, plus optional flags such as CLUSTER_ENABLE_HTTP_CACHE and CLUSTER_EXPIRY_TIMEOUT.

Set them from the console on every web server, with the same values as [eZDFSClusteringSettings]. Each command shows the change as a diff, writes only with --apply, backs config.php up to var/backup/config/<time>/ first and checks the new file with php -l:

./console exp:setup:config --apply                           # once, when there is no config.php yet
./console exp:config:set CLUSTER_STORAGE_BACKEND dfsmysqli --apply
./console exp:config:set CLUSTER_STORAGE_HOST db1.example.com --apply
./console exp:config:set CLUSTER_STORAGE_PORT 3306 --apply
./console exp:config:set CLUSTER_STORAGE_USER <user> --apply
./console exp:config:set CLUSTER_STORAGE_PASS --value="$CLUSTER_DB_PASS" --apply
./console exp:config:set CLUSTER_STORAGE_DB <cluster database> --apply
./console exp:config:set CLUSTER_MOUNT_POINT_PATH /path/to/nfs --apply
./console exp:config:list --only-set                         # the password shows as ********
chmod 0640 config.php                                        # with the web server's group

The password is masked in every output; the command warns when it lands in a config.php every user can read. ./console exp:config:list --only-set --format=json, run on each server, gives documents you can compare to see that all web nodes agree. See Setting up .htaccess and config.php from the console.

The web server sends it only the paths its rewrite rules name. With per-site caches in CacheVarDir, add their public caches as well (Cluster (DFS)):

RewriteRule ^/var_cache/([^/]+/)?cache/(texttoimage|public)/.* /index_cluster.php [L]

Maintenance commands #

Run dfscleanup.php with -S and -B first and read the result before you add -D (15.17). clusterpurge.php is also the cronjob part cluster_maintenance; the shipped cronjob.ini [AdminSettings] ForbiddenParts[] keeps it off the buttons of Setup > Cronjobs, because it is destructive and belongs to an administrator on a shell (Cron jobs).

What to watch #

• The mount. If the NFS mount goes away, every server fails to read files. Monitor the mount and the NFS server as part of the site.
• The metadata database. It is on the path of every file read; it needs the same backup and monitoring as the site's database.
• Consistency. Run dfscleanup.php -S and -B after an incident and before a migration.
• Pausing cron. Cronjob pause files are local to each server unless cronjob.ini [PauseSettings] Directory points at a shared directory (Pausing cronjob parts).
• Cluster events. file.ini [ClusterEventsSettings] ClusterEvents=disabled ships off; with a listener class they let an extension react to cluster operations, for the DFS cluster with the MySQLi backend only (Cluster events).

Checklist #

• To do: The NFS export is mounted at the same path on every web server, writable by the site's user.
• To do: ezdfsfile exists in the metadata database, which is backed up and monitored.
• To do: Cluster settings are in the global settings/override/file.ini.append.php.
• To do: Each server has its own var/ directory; nothing but the mount is shared.
• To do: The rewrite rules send images and binary files to index_cluster.php, which has its CLUSTER_* constants.
• To do: dfscleanup.php -S and -B report no differences after clusterize.php.
• To do: clusterpurge.php runs on a schedule from one server only.

Related pages #

• Several web nodes
• Storage layout
• Backups: what to back up
• Databases in the official order
• Choosing a hosting model
• Setting up .htaccess and config.php from the console: exp:config:set for the CLUSTER_* constants
• Database support, Operating system support
• Free documentation: this page is free to copy, translate and improve under the GNU FDL 1.3.