Queues and background jobs #

This page is for developers who dispatch messages or jobs to a queue (Symfony Messenger, Laravel queues, Celery, Sidekiq, Active Job, Spring @Async). Exponential's job queue is the expqueue extension: database-backed, with handlers, retries, priorities and workers on the command line. Two more mechanisms cover kernel work: content jobs for very large content operations and workflow events deferred to cron (Background work: queues, cronjobs and content jobs).

Contents #

1. Which mechanism
2. expqueue in five steps
3. A handler
4. Content jobs
5. Rules for jobs
6. Compared with other frameworks

Which mechanism #

expqueue in five steps #

From Queues with expqueue:

1. Activate it: [ExtensionSettings] ActiveExtensions[]=expqueue in settings/override/site.ini.append.php.
2. Create its tables (idempotent): ./console ext:expqueue:install-schema.
3. Regenerate autoloads and deploy.
4. Run a worker every minute from cron (php runcronjobs.php -q expqueue), or as a systemd service (./console ext:expqueue:service install) for shorter waits.
5. Check: ./console ext:expqueue:enqueue expqueue.noop, ./console ext:expqueue:work --once, ./console ext:expqueue:status.

Enqueue from any code:

$job = expQueue::enqueue( 'expdemo.archive', array( 'object_id' => $objectID ), array(
    'idempotency_key' => "archive:$objectID:$version",
    'priority' => 50 ) );

Options include class, priority, group, idempotency_key, resource (a concurrency key), not_before, max_attempts, timeout, depends_on and user_id. The payload is stored as JSON and shown in the administration: never put a secret in it.

A handler #

# extension/expdemo/settings/expqueue.ini.append.php
[JobType_expdemo.archive]
Handler=expDemoArchiveHandler
MaxAttempts=4
Timeout=300
Description=Sends one published object to the archive

class expDemoArchiveHandler extends expQueueHandlerBase
{
    public function run( expQueueJob $job, expQueueContext $context )
    {
        $payload = $context->payload();
        $object = eZContentObject::fetch( (int)$payload['object_id'] );
        if ( !$object )
            return expQueueResult::fail( 'The object is gone' );
        expDemoArchiveClient::send( $object );
        return array( 'sent' => true );
    }
}

run() returns nothing or an array (done), or an expQueueResult: done(), retry( $message, $retryAfter ), fail(), defer(), hold(), cancelled(); other exceptions count as retryable. Setup > Job queue shows the dashboard and lets administrators cancel, retry and pause; the policies are expqueue/view and expqueue/manage (expqueue).

Content jobs #

Removing, copying, moving, hiding or revealing large subtrees, changing sections or states and adding or removing locations can run as content jobs in batches, each batch its own transaction, started from the administration or the command line and watched with progress (Content jobs, Content jobs for applications).

Rules for jobs #

• Idempotent: a job can run again after a crash or a retry.
• No database transaction around a slow call.
• A job runs as the user who queued it; check permissions as that user.
• Workers run on the command line, never inside a web request.

Compared with other frameworks #

There is no broker to run: the queue lives in the installation's database, which keeps installations simple at the cost of throughput compared with a dedicated broker.

Related pages #

• Background work: queues, cronjobs and content jobs
• Exponential Intelligence for applications
• Events and listeners
• Free documentation: this page is free to copy, translate and improve under the GNU FDL 1.3.