Testing a bundle #

This page is for bundlers who test a release before it goes out. A bundle combines many parts; the only proof that they work together is installing and using the bundle as a recipient would.

Contents #

1. The test matrix
2. Build checks
3. Install tests
4. Running-site checks
5. Upgrade test
6. The test log

The test matrix #

List what you claim and test each combination you claim, not more:

PHP:        8.x, 8.y
Database:   SQLite 3, PostgreSQL <version>
Server:     Velocity (qbix) / Apache + PHP-FPM / nginx + PHP-FPM
Install:    exp:install / kickstarter / setup wizard
Network:    online / offline
Upgrade:    from bundle <previous version>

The project's own CI tests PostgreSQL 15 and MySQL 8.0 and runs driver unit tests for the others (Database support); a combination your bundle claims beyond that is yours to prove.

Build checks #

Install tests #

1. Dry run: php bin/php/console exp:install --dry-run ... or exp:kickstarter run --dry-run; exit 0.
2. Install: exit 0; var/log/setup.log ends with a RESULT line and no errors. The log includes "health checks after the install (database, declared tables, content classes, objects, content root, name lists, settings, siteaccesses, file ownership)" (Install in one command).
3. Second start: the already-installed test skips the install; data is still there.
4. Offline: the same install with no outgoing network (Offline bundles).
5. Failure path: a wrong database password gives exit 1, maintenance mode on, a readable reason in var/log/setup.log.

Running-site checks #

• To do: The site and the administration answer at the documented addresses.
• To do: Sign in with the handed-over password; change it; the file var/log/initial-admin-password is gone.
• To do: Publish an article with an image; it appears on the site; image variations are created.
• To do: Search finds it.
• To do: Cron runs (runcronjobs.php) as the site user without permission errors.
• To do: /.git/config, /settings/site.ini, /composer.json are not served (13.16).
• To do: No Demo mode banner (except in demo bundles).
• To do: Info > About and Info > Copyright show the versions and licenses.
• To do: No files owned by root under var/.
• To do: After a restart of the whole bundle, everything above still holds.

Upgrade test #

1. Install the previous bundle release; add some content.
2. Apply your update the way recipients will (Updates and upgrade paths).
3. Check the version row (expected value for the 6.0.15 line: 6.0.15stable, query in 11.2), the content, the sign-in and the checks above.
4. Follow 11.7 Verify the upgrade.

The test log #

Keep, per release: the matrix with results, var/log/setup.log and var/log/kickstart.log of each install, the build record (Lock files and reproducible builds) and the names of the testers. Partners who test Exponential releases themselves can feed results back (Testing releases).

Related pages #

• Bundle release checklist
• Testing in a sandbox (site packages)
• Benchmarking a hosted site
• Free documentation: this page is free to copy, translate and improve under the GNU FDL 1.3.