Static publishing #

This page is for administrators and developers who want a public copy of an Exponential site that any static host can serve: GitHub Pages, any other git host's pages, or a plain web server with no PHP at all. Static publishing turns a front siteaccess into a folder of HTML pages and files, with every link rewritten so the folder works on its own, at the root of a domain or below a path such as /my-site/. Editors keep working in Exponential; the public site is a copy.

It is new in Exponential 6.0.15. This page describes the build engine, publishing to git, their commands and the screens under Setup > Static publishing. The GitHub setup is added to it when it arrives.

Features of 6.0 · Related: Static cache generator, Host static sites on GitHub Pages with Exponential 6.0.15+, The build format

What you get #

• Targets in staticcache.ini: each [Target_<id>] block names one front siteaccess, or several that share one host by their URI prefix (SiteAccessList[], one copy with the DefaultSiteAccess at its root and the others in their folders: Static publishing for multisite installations), how its links are written and where the copy is meant to go. An installation can have many targets.
• exp:static:build <target>: crawls the siteaccess as the anonymous user, takes out what a public copy must not carry, rewrites the links, copies the files the pages need and writes the extras a static host expects.
• exp:static:serve <target>: a local preview of the build, served below the target's base path the way GitHub Pages serves a project site.
• exp:static:diff <target>: what the last build changed, whether the folder still matches it, and a link check of every page and file.
• exp:static:targets: every target with its settings checked.
• exp:static:publish <target>: commits the build in the target's own git repository, tags it static/<target>/<n> and pushes it, never forced, signed in with a token or an SSH deploy key that never reaches a command line or a log. exp:static:rollback, exp:static:push, exp:static:keys and exp:static:status go with it.

How a build works #

1. Crawl as the anonymous user. The build uses the crawler of the static cache generator in an export mode. It sends no cookie, so every page is what a visitor who is not signed in sees. It starts at the site's front page, follows only the links the site itself writes, and leaves out module views such as the search, the login and the editing screens. A target can fetch from an internal address (CrawlURL) instead of the site's public SiteURL.
2. Take out what must not be public. Every request carries a signed X-Exp-Static-Build header, so this server renders the page without debug output even when its own address is in DebugIPList[] (see "Debug output and the build" in the troubleshooting guide). Every page loses its form tokens (ezxform_token fields and the csrf-token tags), session ids in addresses and Content Security Policy nonces.
3. Refuse what cannot be cleaned. A page whose response set a session cookie, that still comes with debug output (it is not stripped: the server did not take the signature), or that still carries a form token, is refused. The build then fails and the target's folder is not changed. Pages that only exist with a query string (/blog?page=2) and paths that cannot be a file name on every system are skipped and reported.
4. Collect the files. Every file the pages and their style sheets name is copied: the designs' style sheets, scripts, fonts and images, the packed files in var/<site>/cache/public, image aliases and downloads. Files the web server serves directly are read from the installation (var/ through the cluster file handler, so a cluster's files are found too). Everything else is requested from the site like a visitor would: a download, an image alias that has not been made yet (the request makes it), a packed file that has not been written yet. With Assets=referenced-and-storage, every image in var/<site>/storage/images that belongs to the published version of content the anonymous user may read is copied as well, linked from a page or not.
5. Rewrite the links. See Links below.
6. Write the extras and the manifest. .nojekyll, CNAME, 404.html, robots.txt and sitemap.xml as the target asks, and .static-publish/manifest.json, which lists every file with its SHA-256, its size and, for a page, its node and when its content was last modified.
7. Compare, check and replace. The build is made in a staging folder first. It is compared with the target's folder (the diff), and its links are checked. Unless it is a dry run, it then replaces the folder's content: new and changed files are copied, files the last manifest listed and this one does not are removed, and nothing else in the folder is touched. A .git folder or a README you added stays.

A build of the demo site, 560 pages and 3,547 files (485 MB, most of it storage images), took about three minutes on a copy of its installation; most of it is the crawl, which renders each page once. The link check of that build found no broken link or file among 18,534, at the root and below /<repository>/.

Links #

Relative links work wherever the folder is served, at / and below any path. Links=base-path writes every link below BasePath, as a site that is always served at one place wants. 404.html always links below BasePath: GitHub Pages serves it for every missing address at any depth, where relative links would point to the wrong place.

The siteaccess prefix (/site/about-us on a site matched by URI) and index.php are taken out of every page address. Addresses in scripts (other than structured data) and comments are left alone.

Settings #

All settings are in staticcache.ini, next to the static cache's [CacheSettings]. Put yours in settings/override/staticcache.ini.append.php.

[StaticPublishSettings]:

[Target_<id>] (the shipped [Target_example] block is a template with every setting; it is not built):

The generated reference lists every setting with its comment: staticcache.ini [StaticPublishSettings] and [Target_example].

exp:static:targets checks each target: the siteaccess exists and is a front one, BasePath has the form /name/, addresses are http or https, Remote holds no password or token, Credential names a secret instead of holding one, the lists hold known values. Mistakes are errors; doubtful combinations (a PublicURL whose path is not BasePath, a sitemap without PublicURL) are warnings.

In the administration #

Setup > Static publishing (the module staticpublish) does what the commands do, with screens: a wizard that makes and changes targets (written through the INI editor into settings/override/staticcache.ini.append.php), a build started in the background with its progress followed live, a preview of the build (what changes with sizes, the build's report, the size against the host's limits, the copy served behind a signed link) with Publish and Discard, the history with the rollback, the page of every job and the deploy key. A dashboard block lists the targets with their stale pages and last publish, and Setup > Cache links to it. The policies staticpublish/read, build, publish and setup take a Target limitation; publishing, rolling back, saving a target and the deploy key ask "Verify it's you". The help manual: Static publishing.

The administration starts its jobs as console processes (exp:static:build, exp:static:publish and exp:static:rollback with --login, --run-id and --progress), so a long build never holds a web request; the job files are in <WorkDir>/.jobs and [StaticPublishSettings] JobHistory sets how many are kept.

Commands #

Every command takes --format=json and then prints one JSON document, schema exp-static/1. The exit codes are those of exp:upgrade: 0 done, 1 failed, 2 usage, 3 the database unreachable, 4 internal error. The generated pages: exp:static:build, exp:static:diff, exp:static:serve, exp:static:targets, exp:static:publish, exp:static:rollback, exp:static:push, exp:static:keys, exp:static:status, exp:static:github, exp:static:dns.

Publishing to git #

The target's folder var/<site>/static-publish/<target>/ is a git repository of its own (never inside the site's repository). exp:static:publish commits only the build's files in it, as one commit per publish with the annotated tag static/<target>/<n>, and pushes the branch and the tags to the target's Remote. A push is never forced and a tag is never moved; a push that fails keeps the commit local with the push state pending, and exp:static:push or the cronjob part staticpublish sends it again. A rollback is a new commit holding the files of an earlier tag, so the history only grows. One build, publish, rollback or push of a target runs at a time (a lock per target).

Credentials are names: a token in Setup > Secrets or an environment variable reaches git through GIT_ASKPASS and the environment of the one git process; an SSH deploy key, made per target by exp:static:keys, through a file only the process can read for the time of the push and GIT_SSH_COMMAND. Neither is ever on a command line, in the remote address or in a log.

Every publish and rollback is a row of exp_static_run (state, counts, bytes, commit, tag, push state, who, when), and the pages of the last publish are in exp_static_page; the upgrade step 6.0.15-static-publish-tables makes both tables on every database. The audit records system.staticpublish.publish, .rollback and .credential.

Guides: Repositories and credentials, History and rollback, SSH deploy keys.

GitHub #

For a target with Kind=github-pages and a GitHub address, static publishing uses GitHub's REST API, behind a feature check (expStaticGitHub::available(): PHP's curl and openssl, and GitHubAPI=enabled). It signs in with the target's own credential, a token (token:) or a GitHub App installation (app:), sent in the Authorization header only. A deploy key or none only pushes.

• Setup (exp:static:github setup, and Set up on GitHub in step 4 of the wizard): creates the repository, public unless asked otherwise and empty, for the account or an organization; switches GitHub Pages on from the branch and folder, or from GitHub Actions; sets the custom domain and enforces HTTPS once GitHub has its certificate. Audited as system.staticpublish.target with the verb github-setup.
• Status (exp:static:github status, and the box On GitHub in History and Preview): the repository, GitHub Pages and the last Pages build (built, building, queued, failed), with what differs from the target's settings.
• GitHub Actions: with Delivery=actions (or Extras[]=workflow) the build writes .github/workflows/static-publish-pages.yml, which deploys the pushed files as they are. It needs the build at the root of the branch, and a token with the Workflows permission to push it.
• Limits: every push of a GitHub Pages target is checked first. A build over MaxSiteBytes or with a file over MaxFileBytes is not pushed (the push stays pending); above 90 % of the site limit, a file over 50 MB and a private repository are warnings.
• GitHub App: Credential=app:<installation> signs in with an installation token of the App set in GitHubAppId and GitHubAppKey, made with a JWT signed by the App's private key and kept for the process until it expires; it reaches git through GIT_ASKPASS like a token.
• OAuth: exp:static:github signin signs a user in with an OAuth App's device flow and keeps the token in the secret store.
• DNS: exp:static:dns and Check the DNS in step 6 compare a custom domain's records with GitHub Pages' addresses (expStaticDnsCheck).

Help: Connect to GitHub, Branch or GitHub Actions, Use your own domain, GitHub Pages limits and terms.

What needs a server #

A static host cannot search, take a form or sign anybody in. Each build finds these parts (expStaticDegrader) and gives them a fallback; the report of the build lists them as degraded:

• Search (Search=enabled): the site's search forms go to search.html, which searches an index per language in the browser (expStaticSearchIndex).
• Information collectors (Forms=endpoint): they post to the form endpoint staticpublish/form of the live site, which checks the origin, a token signed at build time and a rate limit, then hands the post to the normal information collection. With Forms=hide they show a short text with a link to the live page.
• Comments, the basket, the login: the short text, or left out.
• Redirect pages (Extras[]=redirects) for old addresses, custom aliases and URL wildcards without *.
• Post-processing: minify, precompress, fingerprint.

A target can publish itself: Triggers[]=publish (pages marked stale on content publish, built after PublishDebounce) and Triggers[]=schedule (Schedule), both in the cronjob part staticpublish, with the e-mail of Notify[] and a signed Webhook afterwards. Details: Dynamic features, The form endpoint. The audit records system.staticpublish.build and system.staticpublish.form too.

Privacy #

• Builds run as the anonymous user: the crawl sends no cookie, and the database checks (which storage images may be copied) are made for the anonymous user.
• The build's requests are signed so the server sends no debug output; a page that still comes with debug output stops the build. Form tokens and session ids are cleaned; a page that cannot be cleaned, or whose response set a session cookie, stops the build before anything is written to the target's folder.
• Storage images are copied only for the published version of content the anonymous user may read. Images of drafts (storage/images-versioned) and files in storage/original are never copied from the storage; a download goes into the copy only when a page links it and the site hands it to an anonymous visitor.
• ExcludePaths[] and ExcludeSections[] leave pages out.
• Read the diff of a build (exp:static:build --dry-run, exp:static:diff) before you push it anywhere public.

Platforms #

• PHP 7.4 is usable, 8.0 and newer are fully supported.
• Every database Exponential runs on; the build reads only content and image tables.
• Apache with PHP-FPM, Velocity and any other web server: the build fetches pages over HTTP like a visitor.
• Cluster storage: files below var/ are read through the cluster file handler.
• Windows: paths are written with / in the manifest whatever the system, page paths that cannot be file names on Windows (:, *, ?, CON and the like) are skipped and reported, and the preview server is started without a shell.