URL map import and export #

Read this page when you upgrade to Exponential 6.0.15 and want to know what importing and exporting URL maps adds to an existing installation, and whether anything you have changes. In short: your URL aliases and wildcards work as before; there is a new table of redirects that is checked first, and it is empty until you import into it.

Importing and exporting URL maps · Migrating URLs · The URL map file formats · Changelog 6.0.15

What is added #

What does not change #

• Global URL aliases, the custom aliases of nodes and wildcards are stored and answered as before. A 301 redirect to a page of the site is imported as an ordinary global URL alias that redirects, exactly as the URL translator makes one; it shows up there and can be removed there.
• Without the tables (before the upgrade step has run) nothing is read: no request is answered differently, and an import is refused with a message that names the step.
• A wildcard of type 1 still answers with 301, an alias that redirects still answers with 301.

Where an imported row goes #

A row whose old address is a page of this site (the address of a node), a module address (/content/view/full/12, /user/login) or the front page / without a query is a conflict and is never imported. /index.php without a query (the front controller) is not importable; /index.php?id=12 is.

Redirects checked on the full request #

The redirects of exp_url_redirect are checked once per request, on the address the visitor asked for, before the URL aliases and wildcards (in ezpKernelWeb::dispatchLoop(), so for PHP-FPM, Velocity and every other front end alike):

• The path is compared normalized as in the URL map files (one slash between segments, no trailing slash, dot segments resolved, percent-encoded UTF-8 decoded) and without regard to case, as aliases and wildcards are. It is tried as the request's whole path (/index.php, /eng/old) and as the path inside the siteaccess (/old).
• The query is compared exactly, but the order of its parameters does not count: the parameters are decoded (+ is a space, a counts as a=) and sorted. A redirect without a query answers only a request without a query.
• A redirect of a siteaccess answers only there; one of every siteaccess (* in the file's siteaccess column, stored without one) everywhere, after those of the siteaccess.
• The target is node:<id> (the node's address at the time of the request, so it follows a moved node), a path of this site, or a full URL. The request's query is not carried over to the target.
• The redirect answers with its status (301, 302, 303, 307, 308) when error.ini has the [HTTPError-<status>] group, else with 301.

The table is read from a cache file next to the wildcard cache (var/<site>/cache/wildcard/urlredirect_*.php), which the URL alias cache clears too and which an import or undo replaces at once; a request with no redirects costs one small cached file.

Import, history and undo #

An import is planned first (expUrlMapPlanner): every row is new, unchanged, changed, duplicate (an earlier row of the file has the same old address; the first one counts), conflict, broken-target, unknown-host or not-importable. New rows are imported; changed ones only with the conflict policy overwrite (--on-conflict=overwrite). The import (expUrlMapImporter) runs in one transaction where the database has them, and records the batch and, for each row it wrote, the row before (none for a new one) and after. Before each write it compares the entry at the old address with the plan's and skips a row whose entry changed in between.

exp:url:undo <batch> puts back every row as it was, the last first. A row changed after the import is kept (the batch is then partly-undone) unless --force is given. exp:url:batches lists the imports.

./console exp:upgrade --apply                                  # makes the tables (step 6.0.15-url-map)
./console exp:url:import old-site.csv -s site --dry-run --report=var/tmp/plan.csv
./console exp:url:import old-site.csv -s site
./console exp:url:import old-site.htaccess --host=www.old-site.com=site --on-conflict=overwrite
./console exp:url:batches
./console exp:url:undo 3
./console exp:url:export --output=var/tmp/url-map.xlsx
./console exp:url:export --kind=wildcard,redirect --format=nginx > var/tmp/rules.nginx

A full old address picks the siteaccess by its host, through the installation's siteaccess matching; a host that is not one of the installation's is listed as unknown-host until --host=<old host>=<siteaccess> maps it (or the host points at the installation). Bare paths belong to the siteaccess of -s, else the default siteaccess; the language is the row's, else --language, else the siteaccess's content language.

exp:url:import and exp:url:export take the file's format with --format and print JSON (exp-url/1) with --output-format=json; exp:url:batches and exp:url:undo print it with --format=json. The exit codes are exp:upgrade's: 0 done, 1 failed, 2 usage, 3 the database unreachable, 4 internal error.

Export #

exp:url:export writes the global URL aliases of the URL translator (with --include-node-aliases also the custom aliases of nodes, target node:<id>), the wildcards and the redirects, in every format the import reads and as .ods. Importing the export into the same installation finds every row unchanged.

Helpers for moving a site #

Four console commands build and check a URL map; none of them changes the installation:

• The 404 source is the not-found log the kernel already writes (site.ini [RequestLogSettings] NotFoundLog=enabled, the default; var/log/404.log and its rotated files). No new log or table is added. The log writes every query value as ***, so an address's query is dropped there; access logs keep it.
• The suggestions read the URL aliases, the tree and the object names once (three queries), visible nodes below the siteaccess's content root only, and write targets as visitors use them, without the PathPrefix.
• Requests go through expUrlMapHttp: GET only, http and https only, a timeout, the user agent Exponential URL map, a size limit, the proxy of site.ini [ProxySettings]. A sitemap's XML is never read with a DOCTYPE or from the network.
• The commands follow the contract of the other exp:url commands: --format=json prints the machine output exp-url/1, --output and --output-format write the map in any format the import reads, the exit codes are the same.
• In the administration the import screen has the same helpers as tabs: From the old sitemap (an address of a public host, never one of the server's own network, or an uploaded sitemap or list), From the 404 log and Suggest targets, each downloaded as CSV or opened as an import to preview; after an import, Test requests its old addresses (the first 500) on the site of its siteaccess and lists the rows that did not land.

In the administration #

The URL translator and the wildcards pages have the buttons Import, Export, Import history and Redirects (URL maps: import, export and undo):

• The function content/urltranslator_import is new and has no limitations. A role with content/* or */* (Administrator) has it; a role that had only content/urltranslator can open the export and the redirects, not the import. No role is changed by the upgrade: give the function to the roles that should import.
• site.ini [ReauthGroup_system] gains content/urlimport ImportButton and content/urlimporthistory UndoButton.
• The uploaded file and its plan stay on the server until the import is done or thrown away, at most a day, in var/<site>/urlimport/<token>/, for the user who uploaded it only; the forms post nothing but the decisions of the rows shown.
• The wildcard pages show a temporary forward wildcard (type 3) as Temporary redirect (302), and the filter Redirecting lists both forward types.

For developers #

• The planner asks the installation through expUrlMapEnvironment; expUrlMapKernelEnvironment is the real one, a test passes its own, so the classification is tested without a database.
• expUrlMapPlan::setDecision( $index, 'import'|'skip' ) overrides a row's decision before apply(); toArray() and writeReport() give the preview (schema exp-url-map-plan/1).
• eZURLWildcard::isForwardType(), redirectStatusOfType(), typeName() and lastRedirectStatus() are new; code that compares a wildcard's type with TYPE_FORWARD should use isForwardType() to include the temporary type.

Oracle and MongoDB #

The upgrade step makes the tables from share/db_schema.dba on Oracle (through ezoracle's schema handler) and MongoDB (collections) too. The import writes with the same SQL statements the kernel's URL alias code uses; it was tested on SQLite, MySQL 8.0 and PostgreSQL 13.