Migrations: deploying to remote installations #

This page is the guide to deploying the migrations of one installation (for example beta) to one or many other installations (dev, stage, production, a group of production nodes) and running them there, all at once and in sync: every selected remote ends in the same migration state, or none of them changes.

Guides · Migrations · Related: Packages, deploys and long runs, Rollback and captions

What a deploy does #

./console exp:migration:deploy --remotes=<ids or groups> works in phases. Every phase runs on all selected remotes in parallel, and a phase only starts when the one before it passed on every remote:

Without --apply the deploy is a dry run: it ends after the preflight and lists, per remote, the files it would send and the migrations that would run. Nothing is sent or changed.

./console exp:migration:deploy --remotes=stage,production
./console exp:migration:deploy --remotes=stage,production --apply

Only the site's migrations ([MigrationSettings] DirectoryList[]) and those of the active extensions are sent. Migrations of packages reach a remote with their package, and kernel steps with the kernel. A migration the remote already has at the same path with the same checksum is not sent again.

All or nothing #

When one remote fails in the switch, the deploy takes every remote that applied back to the caption it took right before the run (exp:migration:rollback <deploy id> --apply there). The remote that failed has already restored its own caption, as every failed run does. The deploy then reports rolled-back: every remote is where it was before.

That is why the verify phase refuses a remote whose captions are off ([CaptionSettings]): without the caption taken before the run it could not be taken back.

A remote that cannot be taken back (its rollback fails, or the connection dropped during the run so its state is unknown) makes the deploy out-of-sync. The report names the state of every remote; check them with ./console exp:migration:remote:status and resolve the one in question by hand before the next deploy.

The switch and the rollback are never retried automatically: a run that may have started is not started a second time.

Remotes #

Remotes are named in migration.ini (in settings/override/migration.ini.append.php), one section each:

[RemoteSettings]
RemoteList[]
RemoteList[]=stage
RemoteList[]=production1
RemoteList[]=production2

[Remote_stage]
Environment=stage
Transport=rsync
Host=stage.example.com
User=deploy
Path=/var/www/site
Credential=sshkey:deploy_stage

[Remote_production1]
Environment=production
Transport=ssh
Host=node1.example.com
User=deploy
Path=/var/www/site
Php=/usr/bin/php8.4
Credential=sshkey:deploy_production
Groups[]=production
HostKeyFingerprint=SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU

[Remote_production2]
Environment=production
Transport=sftp
Host=node2.example.com
User=deploy
Path=/var/www/site
Credential=sshkey:deploy_production
Groups[]=production

--remotes=all selects every remote. [RemoteSettings] also sets Parallel (how many remotes at a time), Timeout (seconds per remote and phase), Retries (how often a connection failure is tried again) and StoreReports.

./console exp:migration:remote:list

lists the remotes and every problem of their settings without connecting to any.

Credentials #

A remote section never holds a secret: a section with a Password, Key, Token or Secret setting is refused. Credential names a secret of the secret store (Setup > Secrets) or an environment variable (env:VARIABLE, for CI).

• SSH key (sshkey:): the private key is written to a file only the deploy can read, for the time of the deploy, and overwritten and removed afterwards. Put the public key into the remote user's ~/.ssh/authorized_keys.
• Password (password:): for ssh, ssh asks the helper bin/shell/expstaticaskpass.sh, which answers from the environment of the ssh process; for FTP it goes into a netrc file only the deploy can read. It is never on a command line.

Every output kept in a report or shown is redacted: the secret's value is replaced by [secret <kind>:<name>].

FTP remotes #

FTP cannot run commands. Without more, a deploy to an FTP remote only sends and verifies the files:

./console exp:migration:deploy --remotes=webspace --transfer-only --apply

Every selected migration is sent (the remote's state cannot be read over FTP) and read back to check it arrived unchanged.

FTP remotes with the HTTP trigger #

When the remote installation offers its signed HTTP trigger, an FTP remote is deployed like any other: its status is read, the files are sent over FTP, a dry run there verifies them, the switch runs them, and a failure anywhere takes it back. The source sends each command as a signed POST to the view migration/trigger of the remote, which runs the console script there (exp:migration:status, migrate, rollback, heal) and answers with its JSON report.

On the remote (the installation that is reached by FTP), in settings/override/migration.ini.append.php:

[TriggerSettings]
Trigger=enabled
Secret=migration_trigger             # a secret of Setup > Secrets (or env:VARIABLE), at least 16 characters
AllowedAddresses[]
AllowedAddresses[]=203.0.113.10      # the address the source connects from (CIDR ranges too)
Php=/usr/bin/php8.4                  # the PHP CLI that runs the console scripts there

On the source, the remote's section names the address and the same secret by name:

[Remote_webspace]
Transport=ftps
Host=ftp.example.com
User=web
Path=/htdocs
Credential=password:webspace_ftp
Trigger=https://www.example.com/migration/trigger
TriggerSecret=migration_trigger

What the trigger takes:

Each request is signed afresh, also when it is tried again after a connection failure. The secret never travels: it only signs. Each request that reaches the checks is written to the audit log of the remote (system.migration.deploy, verb trigger-<action>). The view needs no sign-in (site.ini [SiteAccessSettings] AnonymousAccessList[] and [RoleSettings] PolicyOmitList[] name migration/trigger); with Trigger=disabled, the default, it takes nothing.

Self-healing and the checks of the preflight #

Self-healing is off until [RemoteSettings] SelfHeal[] on the source names the fixes a deploy may make on a remote. Each runs there as ./console exp:migration:heal --fix=... --apply, and each fix is written to the remote's fixes in the report (the deploy's page shows them under the remote):

[RemoteSettings]
SelfHeal[]
SelfHeal[]=stale-lock
SelfHeal[]=modes
SelfHeal[]=caches

On the remote itself, ./console exp:migration:heal --fix=stale-lock,paths,modes lists what it would do; add --apply to do it.

Never fixed, only reported: a PHP or extension version that differs, a migration's own error, a checksum the guard refuses.

The checks of the preflight read the remote's status (it reports its PHP version, its extensions with their versions and the free disk space of its file system) and stop the deploy before anything is sent:

A remote with an older kernel that does not report these is not checked; the report says so in a warning.

Status and history #

./console exp:migration:remote:status
./console exp:migration:remote:status --remotes=production --format=json

reads every remote's migration status in parallel and lists, per remote, its kernel version, the run lock and how many migrations are done, pending and failed, then whether the remotes are in sync (every migration done on all of them or on none, nothing failed) and the latest deploys of this installation.

Every deploy that sends or runs something keeps its report (exp-migration-deploy/1) in var/<site>/migration/deploy/reports/<deploy id>.json, and the audit log records it (system.migration.deploy; system.migration.deploy.rollback when remotes were taken back).

Staged deploys, gates and rollback #

--staged (with --apply) puts the selected remotes in stages by their Environment, in the order of [RemoteSettings] StageOrder[] (alpha, dev, test, stage, beta, production, post, arch; an environment that is not listed comes last). The preflight still runs on every remote first. Then the first stage is transferred, verified and switched, all or nothing within the stage, and the deploy ends in the state gated: the later remotes are waiting.

./console exp:migration:deploy --remotes=all --staged --apply
./console exp:migration:deploy --continue=20261102-120000-a1b2c3 --apply     # approve the gate: the next stage
./console exp:migration:deploy --rollback=20261102-120000-a1b2c3 --apply     # take back every remote that applied it

--continue=<deploy id> approves the gate: the next stage is prepared and preflighted again (time has passed) and switched the same way; the report records who approved each gate. When a later stage fails, it is taken back and the stages before it stay deployed (the report says the remotes are not in sync). --rollback=<deploy id> takes back every remote that applied a deploy (exp:migration:rollback <deploy id> --apply on each, all at once), staged or not.

With the upgrade step 6.0.15-migration-deploy-tables, every stored report is also written to the tables exp_migration_deploy (one row per deploy) and exp_migration_deploy_host (one row per remote). A deploy that fails and a staged deploy that waits at a gate send a plain-text notice by e-mail to [RemoteSettings] NotifyEmail (empty: site.ini [MailSettings] AdminEmail); Notify[] names the kinds (failure, gate, success).

All of this is also in the administration: Setup > Migrations > Remotes and Deploys (the wizard of a remote, Test connection, a deploy with live progress per remote, the history, the gate and the rollback).

Recipes #

Deploy beta's migrations to production. On beta: ./console exp:migration:deploy --remotes=production (the dry run), read the plan, then the same with --apply.

Keep dev, stage, beta and production in sync. Put all of them in RemoteList[], deploy with --remotes=all, and check ./console exp:migration:remote:status afterwards: "In sync: yes".

Only one migration. --only=20261102090000_add_news_class sends and runs only that one on every remote.

The same in the administration. Every step above has a screen in Setup > Migrations, explained with screenshots in Add a remote step by step and Deploy step by step.

Troubleshooting #