Specification: the URL map file formats #

This page is the normative description of the files that carry a URL map in Exponential 6.0.15: a list of old addresses and where each one goes, as used for the global URL aliases (the URL translator) and the URL wildcards. It covers the row model, the columns and kinds, how addresses are normalized, and each file format with what it reads, what it writes and what it cannot express. It is for developers and for administrators who prepare such files by hand or with other tools.

Version of this specification: 1 (exp-url-map/1).

The classes are in kernel/classes/urlmap/ and kernel/classes/spreadsheet/; they read and write files only and use no database. Importing the rows into an installation, the preview and the undo are not part of this page.

The row #

Every format carries the same seven columns (status came after the first six; a file without it is read as before):

Kinds #

A kind is compared in lower case, and _ or a space counts as - (Wildcard_Forward is wildcard-forward). temporary-redirect and redirect-temp are read as redirect-temporary, wildcard-temporary as wildcard-forward-temporary. auto is the same as empty. Any other value is kept as written and makes the row not importable. A status that contradicts the kind (redirect with 302) or is given for a kind that does not redirect is a problem too.

Where each kind is stored is the importer's business: see URL map import.

Normalization #

The old and the new address are normalized when a row is read, so that the same address gives the same row in every format:

• A scheme and a host are split off. The scheme and the host are written in lower case; a default port (:80 for http, :443 for https) and a trailing dot of the host are dropped. //host/path is a host without a scheme; www.example.com/path is a host because it begins with www.. Any other text without a scheme is a path.
• Only http and https are accepted; a user name or password in the address is a problem.
• The path starts with /; repeated slashes become one; . and .. segments are resolved.
• Trailing slash: a trailing slash is dropped (the root / stays). Exponential serves /about and /about/ as the same page, so /about/ and /about are the same old address.
• Percent-encoding: %XX sequences that encode UTF-8 text are decoded (/%C3%9Cber is /Über). %2F, %3F, %23 and %25 stay encoded, because decoding them would change the address; spaces and control characters are encoded (/Über uns is /Über%20uns); the hex digits that stay are written in upper case. Bytes that are not UTF-8 stay encoded.
• The fragment (#...) is dropped; the query (?...) is kept as written. A wildcard cannot have a query.
• The case of the path is kept.
• A target node:123 or eznode://123 is written node:123. A target with a host but no scheme gets https.

Validation #

A row is importable when it has none of these problems:

Each row keeps the place it came from (line()): the line a CSV record starts on, the row number in a spreadsheet, the entry number in JSON (from 1, without the header), the line of the directive in Apache and nginx files. A file that cannot be read at all (invalid JSON, not a workbook, an unknown format) is refused as a whole; a single bad row is a row with its problems, so a preview can list it.

CSV #

Read and written. Extensions .csv, .tsv and .txt.

• RFC 4180: fields may be quoted with "; a quoted field may hold the delimiter, line breaks and doubled quotes.
• The delimiter is ,, ; or a tab, found from the first 20 lines (the one that splits most lines into the same number of fields). Spreadsheet programs with a German locale write ;, "Unicode text" from Excel is tab-separated.
• Encoding: UTF-8, with or without a byte order mark; UTF-16 with a byte order mark is converted; text that is not valid UTF-8 is read as Windows-1252. Line ends \r\n, \n and \r.
• A header row is optional. It is recognized when it names old_url or new_url (also old, from, source, new, to, target, destination, and the German Alte URL, Quelle, Neue URL, Ziel and the like) and its first cell is not an address. Its columns may be in any order; unknown columns are ignored. Without a header the columns are in the order above.
• Blank lines and lines whose first cell starts with # are skipped. A row without a header that has more than seven filled columns is a problem.
• Written with a header, CRLF line ends, fields quoted only when they need it, and a UTF-8 byte order mark (the option bom, on by default, so that Excel reads UTF-8). A .tsv file is written with tabs.

old_url,new_url,kind,siteaccess,language,note,status
/about-us.html,/About,,,,old about page
https://old-site.com/news/2019/,/News,redirect,site,eng-GB,
/products/*,/Shop/{1},,,,"catalogue, all pages"
/team,node:42,alias,,,

JSON #

Read and written. Extension .json.

An array of objects with the canonical keys. The first entry may be the header {"schema": "exp-url-map/1"}; an object {"schema": "exp-url-map/1", "rows": [...]} is read as well. Another schema is refused. The other column names of CSV headers are accepted as keys; numbers are read as text; an entry that is not an object, or a value that is an array, is a row with a problem. Written as the array with the header first and all seven keys in every object, pretty-printed, slashes and Unicode not escaped.

[
    { "schema": "exp-url-map/1" },
    { "old_url": "/about-us.html", "new_url": "/About", "kind": "", "siteaccess": "", "language": "", "note": "", "status": "" },
    { "old_url": "/products/*", "new_url": "/Shop/{1}", "kind": "", "siteaccess": "", "language": "", "note": "", "status": "" }
]

Excel (.xlsx) #

Read and written. Extension .xlsx. Needs the PHP extension zip (ZipArchive); without it, reading and writing spreadsheets stops with a message that says so, and CSV and JSON still work.

• Reading takes the first sheet of the workbook (by the workbook's own order) as a table, as CSV: the same header detection, the same columns. Shared strings (also with formatted runs), inline strings, numbers (whole numbers without .0 or an exponent), booleans (TRUE, FALSE) and the last result of formulas are read; styles, other sheets, merged cells and comments are ignored. A cell with an error is empty.
• Writing makes one sheet "URL map" with the header row (bold and frozen) and column widths; every cell is an inline string, so nothing turns into a number, date or formula. The package has only the parts a workbook needs: [Content_Types].xml, _rels/.rels, xl/workbook.xml, xl/_rels/workbook.xml.rels, xl/styles.xml and xl/worksheets/sheet1.xml.
• A part with a DOCTYPE or larger than 100 MB uncompressed is refused.

OpenDocument spreadsheet (.ods) #

Read and written (LibreOffice Calc). Extension .ods. Needs the PHP extension zip.

• Reading: the first table of content.xml, with a header row as in CSV. A cell's paragraphs are joined with line ends; text:s (spaces), text:tab and text:line-break are read as the characters, spans and links by their text, annotations are left out; a cell without paragraphs gives its value. Repeated rows and columns are read as LibreOffice writes them: an empty one only moves the position on (a sheet ends with an empty row repeated about a million times), one with text is repeated at most 1,024 times. Header row groups count as rows. A content.xml with a DOCTYPE or over 100 MB, and an OpenDocument file that is not a spreadsheet, are refused.
• Writing: one sheet "URL map" with the header row in bold; the mimetype entry comes first and uncompressed, as OpenDocument requires; spaces, tabs and line breaks in a cell are written as OpenDocument elements so that they survive.

Temporary files of the spreadsheet classes go to the cache directory of the installation (var/<site>/cache/spreadsheet), else var/tmp, never the system's temporary directory, so they work under open_basedir.

Apache #

Read and written. Extensions .htaccess, .apache, .apacheconf, the file name .htaccess, and .conf when the content has Apache directives.

Read:

A redirect keeps its status: 301 and 308 are permanent kinds, 302, 303 and 307 temporary ones; a status other than the kind's own (301, 302) goes in the column status. A RewriteRule with R and no status is 302, as in Apache.

Not importable, with the reason in the row and the directive in its note: gone/410 and other status codes, other conditions (%{REQUEST_FILENAME}, [OR]), the flags F, G, P, C, S, E, CO, T, H, N, a negated pattern, the target -, %1 and %{...} in the target. The flags L, NC, QSA, QSD, NE, END and PT do not change the row. Lines continued with \ are joined. Other directives are ignored.

Written, for a <VirtualHost> or the .htaccess of the document root (R= is the row's status):

RewriteEngine On

# old about page
RewriteRule ^/?about\-us\.html/?$ /About [R=301,L]

RewriteCond %{HTTP_HOST} ^old\-site\.com$ [NC]
RewriteRule ^/?news/?$ /News [R=301,L]

RewriteCond %{QUERY_STRING} ^id\=5$
RewriteRule ^/?index\.php/?$ /Home? [R=301,L]

RewriteRule ^/?products/(.*)$ /Shop/$1 [R=301,L]

#exp:alias /team /Team
#exp:redirect /staff node:42

^/? and /?$ make the rule work in both places and match with and without a trailing slash. A query condition ends the target with ?, so the old query is not carried over. Aliases and wildcards (direct) are served by Exponential itself without a redirect, and a target node:<id> is only known to Exponential, so they are written as #exp: comments, which Apache ignores and the reader reads back. A note, siteaccess and language go in a comment above the rule. Rows that are not importable are left out with a # not written: comment.

nginx #

Read and written. Extension .nginx, and .conf when the content has nginx directives.

Read:

$scheme://$host (or https://$host) in front of a target is the same host and is dropped. Not importable: a return in a prefix location (location /old matches every path that starts with /old) or outside a location, a return with another status (410, 404), a rewrite inside an if, a return inside an if with another condition, other variables in the target, a map of something other than a path.

Written, for inclusion in the server block of the site (return with the row's status; a temporary wildcard is rewrite ... redirect; for 302 and location ~ ... { return 307 ...; } for the other statuses):

location = /about-us.html {
    return 301 /About;
}

location = /index.php {
    if ($args = "id=5") { return 301 /Home; }
    return 301 /;
}

rewrite ^/products/(.*)$ /Shop/$1 permanent;

#exp:alias /team /Team

server {
    server_name old-site.com;

    location = /news {
        return 301 /News;
    }
}

Exact redirects of one path are one location = block (with an if per query); a second redirect of the same address is left out with a comment, as nginx refuses two equal locations. Rows of another host go into a server block of their own, which needs the site's listen lines added. location = matches the path exactly, including a trailing slash.

Regular expressions and wildcards #

A rewrite rule becomes a wildcard when its expression is one a wildcard can say:

• anchored at the start with ^; a / or /? after it is optional (per-directory rules have no leading slash)
• literal text, with \ before punctuation; an unescaped . is read as a literal dot, as nearly every rule with one means it
• the groups (.*), (.+), (.*?), (.+?) and ([^/]+), ([^/]*), each a * (the last two with a note: a * also matches /); a bare .* is a * too
• an optional trailing slash (/?) before the end, which is dropped
• $ at the end; without it the expression matches every path that starts with it, which is a * at the end

$1, $2 in the target become the placeholders of their groups' * (a bare .* counts as a * but not as a group, so the numbers are mapped). Anything else, such as \d, [a-z], a|b, x?, x+, {2} or (?:...), makes the rule not importable, with the construct named in the reason.

When a map is written as rules, each * becomes (.*) and each {N} becomes $N; literal text is escaped.

Choosing the format #

expUrlMapFormats chooses the reader and writer: by an explicit format name (csv, json, xlsx, ods, apache, nginx), else by the file's extension, else by its content (a ZIP archive is .ods when its first entry is the OpenDocument spreadsheet mimetype, else .xlsx; [ or { first is JSON, Apache or nginx directives, else CSV).

$rows = expUrlMapFormats::readFile( 'old-site.htaccess' );
foreach ( $rows as $row )
{
    if ( !$row->isImportable() )
        echo $row->line(), ': ', implode( '; ', $row->problems() ), "\n";
}
expUrlMapFormats::writeFile( 'map.xlsx', $rows );

Related pages #

• Importing and exporting URL maps, Migrating URLs and URL map examples
• The migration file format
• Specifications 6.0
• Changelog 6.0.15