Bulk data exchange with xrowextract #

This page is for integrators whose partner system exchanges files: a nightly product file from an ERP, a weekly export for a data warehouse, a spreadsheet round trip with an agency. The xrowextract extension (required as ~2.5.18) is the Export tab of the administration. It exports and imports content as files, runs on a schedule and delivers the files to other systems, without a line of custom code. The full feature page is xrowextract, the reference the xrowextract specification.

Contents #

1. What it covers
2. Exports
3. Imports
4. Schedules and destinations
5. The command line
6. Security notes
7. xrowextract or your own code

What it covers #

Exports #

Columns are the class attributes plus special columns: object id, remote id, main node, parent node, URL alias, dates, and for users login and e-mail. Filters cover dates, section, state, visibility, name, owner, depth and any class attribute, joined with and/or. A saved preset turns a selection into a reusable, named export with {placeholders}. Export the remote id column whenever another system will match the rows back to its own records.

Imports #

1. Upload the file (chunked and resumable, any size) and choose the class, the parent node and the language.
2. Map the columns to attributes; the page lists every column and element the importer understands.
3. Press Preview: a dry run that names each change and writes nothing.
4. Press Apply. A large import runs as a background job with a resume point.

An imported file URL (for images and files referenced by address) must be of the site's own scheme, host and port. Scheduled imports always do a dry run first and are applied only when it found no errors.

Schedules and destinations #

A schedule runs a preset, a site archive, an Export as package or an import from a folder or a destination, hourly, daily, weekly, monthly or on a five-field cron expression, in full or as a delta (only changes since the last successful run). Start due schedules with the cronjob part:

php runcronjobs.php xrowextract

With xrowextract.ini [Notifications] WebhookSecret set, the webhook carries X-Xrowextract-Signature: sha256=<HMAC-SHA256 of "<timestamp>.<sha256 of the body>">, so the receiver can check the sender. Notifications go by e-mail on failure (always to the owner), optionally on success, and to the webhook. History lists every run with who, what, rows, size, checksum, delivery and warnings.

The command line #

Every function has a command, as php extension/xrowextract/bin/php/<name>.php or through the console:

# articles changed in the last 30 days that are not hidden, as JSON
php extension/xrowextract/bin/php/csv.php --class=ng_article --node=2 --format=json \
  --visibility=visible --date-field=modified --date=30 --output=var/tmp/recent.json --allow-root-user

--user=<login> exports with that login's read access (default admin): give a service account instead, so the export cannot contain more than it should.

Security notes #

• Destination credentials are encrypted with libsodium; the key file is generated with mode 0600 at settings/override/xrowextract-secrets.key. Never commit it; losing it means entering the secrets again.
• Exports can hold personal data (user columns, collected answers): deliver them only to destinations that may hold such data, and set a retention on the receiving side.
• Limit the policies of xrowextract to the people who run exchanges; read them in the feature page's Policies section.

xrowextract or your own code #

Related pages #

• xrowextract
• Packages for import and export
• Writing an export or an outbound sync
• Free documentation: this page is free to copy, translate and improve under the GNU FDL 1.3.