exp:benchmark

Available since Exponential 6.0.15 (in development, not in a release yet).

Summary

Measure this installation: page timings over HTTP, A/B between servers, kernel probes, regression check

Usage

./console exp:benchmark [options]
php bin/php/benchmark.php [options]
./console bench [options]

Aliases

bench

Help

The output of ./console help exp:benchmark (the script run with --help), without the standard options:

Usage: bin/php/benchmark.php [OPTION]... [MODE]...
Exponential benchmark

Measures this installation and compares it with a saved run.

Modes:
  http      (the default) request pages and time them: the front page, the first
            pages of the site's menu, the search page and the admin login page,
            or --url / --urls-file
  kernel    time the parts of a page in-process: boot, full view render cold and warm,
            INI load, content fetch, node list, database round trip, cache write and
            read, image alias lookup
  micro     (or --micro) pure-PHP hot paths without HTTP or a database: template compile
            and render, INI parsing, autoload, eZURI, translations, datatype validation;
            normalised against a calibration loop, so runs on different machines compare
  compare <baseline.json> <run.json>   compare two saved runs
  show <run.json>                      print a saved run as a table

Examples:
  ./console exp:benchmark
  ./console exp:benchmark --compare=https://example.com,https://example.com:8080
  ./console exp:benchmark --cold --requests=50
  ./console exp:benchmark kernel --repeat=30
  ./console exp:benchmark --runs=200 --warmup=10 --format=json > run.json
  ./console exp:benchmark micro --baseline=tests/benchmark/baseline.json
  ./console exp:benchmark micro --save=tests/benchmark/baseline.json   (refresh the baseline)
  ./console exp:benchmark --save=var/benchmark/base.json
  ./console exp:benchmark --baseline=var/benchmark/base.json --threshold=15

Polite by default: 100 requests per URL, 4 at a time. Against a host other than
this machine's loopback, more than 1000 requests per URL or 16 at a time needs --force,
and a host the site does not serve is only measured when its URLs are named with
--url or --urls-file. Nothing is written but cache entries.

Exit code: 0 ok, 1 a metric regressed against --baseline, 2 usage error or refused.
Guide: doc/eng-us/guides/benchmarking.md


Options:
  --base=VALUE         Server to measure (default: https:// and the SiteURL of the siteaccess)
  --compare=VALUE      Two or more servers, comma separated, measured back to back (A/B)
  --url=VALUE...       A URL or path to measure; repeat it for more
  --urls-file=VALUE    A file with one URL or path per line (# starts a comment)
  --pages=VALUE        How many pages of the menu the default list takes (default 3)
  --no-admin           Leave the admin login page out of the default list
  --no-search          Leave the search page out of the default list
  --search=VALUE       The words the default search page searches for (default: the first word of the first menu page)
  --requests=VALUE     Requests per URL and server (default 100)
  --runs=VALUE         The same as --requests: measured runs per URL and server
  --concurrency=VALUE  Requests at the same time (default 4)
  --warmup=VALUE       Unmeasured runs first: requests per URL and server (default 5, at least one per connection), micro: iterations per probe (default 3)
  --rounds=VALUE       Split the requests into rounds, alternating the server order (default 2)
  --micro              The same as the micro mode
  --format=VALUE       Output: table (default), json or csv
  --cold               A unique query string on every request, so no response cache answers
  --cold-clear         Clear the content view cache once before the run (this installation only)
  --insecure           Do not verify TLS certificates (self-signed servers)
  --auth=VALUE         HTTP basic auth, user:password (or the EXP_BENCHMARK_AUTH variable)
  --header=VALUE...    An extra request header, "Name: value"; repeat it for more
  --cookie=VALUE       A Cookie header to send, e.g. to measure a signed-in page
  --resolve=VALUE...   host:port:address, as curl --resolve, to reach a host at another address
  --http=VALUE         HTTP version: 1.1 or 2 (default: what curl negotiates)
  --encoding=VALUE     The Accept-Encoding to send (default gzip; "none" for none)
  --timeout=VALUE      Seconds per request (default 30)
  --no-keepalive       A new connection for every request
  --tool=VALUE         curl (default, built in), ab, wrk or oha when installed
  --duration=VALUE     Seconds per URL for wrk (default 10)
  --repeat=VALUE       kernel: runs of every probe (default 20); micro: measured iterations of every probe (default 30)
  --boot-repeat=VALUE  kernel: new processes for boot and the cold render (default 5)
  --node=VALUE         kernel: the node to render and fetch (default: the front page)
  --probe=VALUE...     kernel, micro: only these probes; repeat it for more
  --json               Print the run as JSON
  --csv                Print the rows as CSV
  --save=VALUE         Write the run as JSON to this file
  --baseline=VALUE     Compare with this saved run; exit 1 when a metric regressed
  --threshold=VALUE    Percent a metric may get worse before it counts as a regression (default 15; micro: 40, after normalisation)
  --min-delta=VALUE    Milliseconds a latency must grow by, too, to count (default 1)
  --all-checks         Show every comparison, not only the regressions
  --force              Allow more than the polite limits against a host that is not loopback

Options

Arguments

[MODE*]

Standard options

Every script also takes the standard options of the script framework: --help, --quiet, --siteaccess, --debug, --colors, --no-colors, --logfiles, --no-logfiles, --verbose and --allow-root-user.

Source

bin/php/benchmark.php, kernel/private/classes/commands/benchmark.php

See also

Generated example: exp:benchmark

Added to the reference for Exponential 6.0.15.