Importing and exporting URL maps #

When a site moves to Exponential, or gets a new structure, its old addresses have to keep working: links on other sites, bookmarks and search engine results point at them. Exponential 6.0.15 takes those addresses in as a URL map, a file that lists each old address and where it goes now. It imports the whole file in one step, with a preview first, and can take the whole import back again. It also writes the site's own redirects out as a file, for a web server or as a backup.

Before 6.0.15 a global URL alias (the URL translator) or a URL wildcard had to be created one at a time in the administration, and there was no way to export them.

Features 6.0 · Tutorial: Migrating URLs · Example files: URL map examples

What it does #

• Imports a URL map into the URL translator, the URL wildcards and a new list of redirects, as one batch.
• Previews every row before anything is written: what is new, what would change, and what cannot be imported and why. A dry run checks every row against the site and writes nothing.
• Undoes an import as a whole: every entry it created is removed and every entry it replaced is put back.
• Exports the site's global URL aliases, wildcards and redirects in every format it reads. Importing an export into the same site changes nothing.
• Redirects old addresses with a query, such as /index.php?id=12 of an old CMS, to the new page.
• Keeps temporary redirects temporary (302, 303, 307), so search engines keep the old address where that is meant.
• Helps build the map: starts it from the old site's sitemap or from the addresses visitors do not find (the 404 log), suggests the new page for each old address, and tests the old addresses on the site after the import.

It works in the administration (the URL translator and URL wildcards pages) and on the command line.

Formats #

The format is recognized from the file's name or its content. A spreadsheet or CSV file has the columns old_url, new_url, kind, siteaccess, language, note and status; only the first two are needed:

old_url,new_url
/about-us.html,/fitness
/recipes/hummus.html,/recipes/home-made-hummus
/old-recipes/*,/recipes/{1}

Every detail of every format, including what a rewrite rule can and cannot be turned into, is in the specification The URL map file formats.

Old and new addresses #

• The old address is a path (/about-us.html) or a full address (https://www.old-site.com/about-us.html). A full address picks the siteaccess by its host. An old host that this installation does not answer for is mapped to a siteaccess in the import screen, or with --host=www.old-site.com=site on the command line.
• The new address is a path of this site (/recipes), a node (node:163, which follows the page when it moves), or a full address on another site.
• A * in the old address makes a wildcard: {1} in the new address stands for what the first * matched.

Kinds and statuses #

The kind column decides what a row becomes. Left empty, a row with * is a redirecting wildcard and every other row a permanent redirect.

Global URL aliases and wildcards are the same entries you would create by hand; they show up on the URL translator and URL wildcards pages and can be changed there. Redirects that an alias cannot hold (an old address with a query, a temporary status, a target on another site) go into a list of their own, Redirects.

Query-string redirects #

Old content management systems often had addresses such as /index.php?id=12 or /shop.php?category=3. A row with such an old address is a redirect of the full request:

old_url,new_url
/index.php?id=12,/recipes
/shop.php?category=3,/recipes/kale-salad

These redirects are checked first on every request, before the URL aliases and wildcards. The order of the query's parameters does not count (?b=2&a=1 is ?a=1&b=2); another query, or none, does not match. The old query is not carried over to the new address.

Preview, conflicts and dry run #

Each row of a file is planned before it is imported, and gets one of these states:

A dry run goes through every row against the site as the import would, counts what it would do and writes nothing.

Undo #

Every import is recorded as one batch, with every entry it wrote as it was before and after. Undo removes what the import created and puts back what it replaced, the last first. An entry that someone changed after the import is kept, unless you choose to undo it too. The import history lists who imported which file when, with the counts.

Export #

An export writes the site's global URL aliases (optionally also the custom aliases of pages, as node:<id>), the wildcards and the redirects, in any of the formats above. Use it as a backup before a big import, to edit the map in a spreadsheet and import it again, or to hand the redirects to the web server: the Apache and nginx exports are rules the server answers itself, without PHP.

Helpers for a migration #

Four helpers build and check a map, on the command line and as tabs of the import screen. None of them imports anything; they write a map you can edit and import.

Exponential's own 404 log (var/log/404.log, site.ini [RequestLogSettings] NotFoundLog, on by default) hides the values of a query as ***. exp:url:from-404 therefore gives paths only; old addresses with a query come from the sitemap or from an access log read with --log. The tab From the old sitemap fetches only addresses of public hosts, never one of the server's own network. In the administration, Test checks the first 500 old addresses on the site address of the import's siteaccess (site.ini [SiteSettings] SiteURL); exp:url:test --base=<address> tests any site, a staging site too.

In the administration #

On the URL translator and URL wildcards pages (Setup tab), the buttons Import, Formats and examples, Export, Import history and Redirects lead to the screens:

Importing and undoing ask you to confirm who you are first ("Verify it's you"). The screens are explained one by one in the guide URL maps: import, export and undo.

Policies #

The Administrator role has both. Give content/urltranslator_import only to the people who may change the site's addresses in bulk: an import can redirect hundreds of addresses at once. Every import and undo is recorded in the audit trail (content.urlalias.import, content.urlalias.import_undo).

On the command line #

./console exp:url:import old-site.csv -s site --dry-run --report=var/tmp/plan.csv
./console exp:url:import old-site.csv -s site
./console exp:url:import old-site.htaccess -s site --host=www.old-site.com=site
./console exp:url:batches
./console exp:url:undo 3
./console exp:url:export --output=var/tmp/url-map.xlsx
./console exp:url:export --kind=wildcard,redirect --format=nginx
./console exp:url:from-sitemap https://www.old-site.com/sitemap.xml --output=var/tmp/old-site.csv
./console exp:url:from-404 -s site --min-hits=3 --since=30d --output=var/tmp/not-found.csv
./console exp:url:suggest var/tmp/old-site.csv -s site --fill-above=85 --output=var/tmp/map.csv
./console exp:url:test var/tmp/map.csv --base=https://www.example.com -s site

Bare paths belong to the siteaccess given with -s (else the default siteaccess). With --on-conflict=overwrite an import replaces the entries at addresses that have one; by default it skips them. --output-format=json (import, export) and --format=json (batches, undo) print machine-readable output.

Installing it #

Upgrading to 6.0.15 adds three tables, made by the upgrade step 6.0.15-url-map (./console exp:upgrade --apply). Until the step has run, nothing changes and an import is refused with a message that names the step. What is added and what stays the same: URL map import and export.

Related pages #

• Migrating URLs: the step-by-step tutorial
• URL map examples: example files that import cleanly
• The URL map file formats
• URL map import and export
• Links, URL aliases, wildcards and search statistics
• URL wildcards and custom aliases
• Changelog 6.0.15