Benchmarking a hosted site #

This page is for system administrators and product managers at hosting companies who must say how fast a customer site is, compare two server setups, prove a deploy did not make things slower, or set the limits of a plan. It answers: what exp:benchmark measures, how to use it safely on production machines, how to compare fairly, and how to turn the numbers into decisions. The sources are Benchmarking Exponential and Benchmark: exp:benchmark.

Contents #

1. What it measures and what it does not
2. Safe by default
3. The first run
4. Cached and rendered
5. Comparing two setups
6. Checking a deploy
7. Noise on shared machines
8. From measurements to plan limits
9. Checklist

What it measures and what it does not #

"It does not answer 'how many visitors can this machine take'; that needs a load generator against a copy of the site." --tool=ab|wrk|oha hands the requests to an installed load generator and is "the start of that, not the end".

Safe by default #

• HTTP mode sends GET only, never posts, no cookies unless --cookie.
• Against a host other than this machine's loopback, more than 1000 requests per URL, more than 16 at a time, a warm-up above 100 or more than 20,000 requests in all needs --force.
• The default page list is sent only to hosts this installation serves; another host must be named with --url.
• --cold-clear is the one option that clears something (the content view cache, once); it is refused for servers the installation does not serve. Do not use it on a server other people are working on.
• Run as root, the command prints a warning and waits ten seconds; run it as the site's user where you can.

The first run #

From the installation root:

./console exp:benchmark --allow-root-user

It requests the front page, the first three menu pages, the search page (when visitors may search) and the admin login page, 100 times each after 5 warm-up requests, 4 at a time, and prints median, p90, p99, minimum, maximum and standard deviation per page, plus the environment (PHP, OPcache, Xdebug, CPUs, load, commit). For a list that never changes, name the pages:

./console exp:benchmark --urls-file=benchmark-urls.txt --allow-root-user

Cached and rendered #

Most visitors get the warm number; editors, searches and cache misses get the cold one. Report both.

Comparing two setups #

Run both servers on the same installation (Velocity on its own port beside Apache, for example) and measure them in one run, back to back:

./console exp:benchmark --compare=https://www.example.com,https://www.example.com:8080 --allow-root-user
./console exp:benchmark --compare=https://www.example.com,https://www.example.com:8080 --cold --runs=30 --allow-root-user

The guide's own example shows why both runs matter: cached, Velocity answered in 1.3 to 2.6 ms against Apache's 8.7 to 10.7 ms; rendered, Apache was faster in that run. "A single 'which server is faster' number misleads" (Benchmarking, section 4.4). Three or more servers can be compared at once; each is compared with the first.

Checking a deploy #

./console exp:benchmark --cold --runs=50 --save=var/benchmark/pre-deploy.json --allow-root-user
./console exp:velocity deploy --allow-root-user
php bin/php/warm.php --allow-root-user
./console exp:benchmark --cold --runs=50 --baseline=var/benchmark/pre-deploy.json --min-delta=5 --allow-root-user
echo $?        # 0 no regression, 1 a metric regressed, 2 usage error or refused

A metric counts as regressed when it grew by more than --threshold percent (15) and by more than --min-delta milliseconds (1). "A deploy script can use the exit code; on a shared machine give it --threshold=25 or more and a --min-delta, or it will cry wolf" (Benchmarking, section 11.5).

Noise on shared machines #

The guide repeated a saved run nine minutes later with no change at all and got 13 "regressions": the machine was busier. Cache hits of 1 to 3 ms doubled from one minute to the next on a shared machine. What to do (Benchmarking, section 10):

1. Same machine, same PHP, OPcache, JIT and Xdebug setting (the environment line says which).
2. Same command: the same pages, runs, warm-up, concurrency, warm or cold.
3. Nothing else running that you can stop; note the load in the header line.
4. Before and after close together; for two servers, one --compare run, never two separate ones.
5. The caches in the same state: the same cache column in both runs.
6. Run it twice. A difference that does not repeat is noise.
7. Judge the median first, p90 and p99 second, max never alone.

From measurements to plan limits #

The project publishes no capacity per server, and this help gives none. What the sources do give is a method:

Figures quoted in the documentation (for example 8,941 requests a second in the README, or the instance numbers on Velocity workers and pools) were measured on particular machines and sites. The sources say so: they "describe this site's templates on this machine. They are not a property of the server and should not be quoted as one." Do not put them in a plan or an offer.

Checklist #

• To do: Benchmarks run as the site's user, against hosts this installation serves.
• To do: Every report gives warm and cold numbers, and the environment line.
• To do: Comparisons made in one --compare run, back to back.
• To do: Deploy checks use --baseline with a --min-delta suited to a shared machine.
• To do: Plan limits come from your own measurements, not from documentation figures.

Related pages #

• Performance
• Velocity workers and pools
• Designing hosting plans
• Benchmarking Exponential
• Benchmark: exp:benchmark
• Free documentation: this page is free to copy, translate and improve under the GNU FDL 1.3.