Skip to content

Commit a9e6d9b

Browse files
SeigerSeiger
authored
Merge pull request #2453 from elcreator/add-mysql-pgsql-CI
add(CI): MySQL and PgSQL
2 parents 3094a79 + ad05687 commit a9e6d9b

11 files changed

Lines changed: 1223 additions & 0 deletions

File tree

‎.dockerignore‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# Keeps the build context small for both images in .github/docker. Neither of
2+
# them needs the history, the editor settings or the scratch directory.
3+
.git
4+
.github/docker/**/*.md
5+
.idea
6+
tmp
7+
node_modules
8+
core/database/*.sqlite

‎.github/docker/ci/Dockerfile‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# Runs the CLI installer against a real database server, so the MySQL and
2+
# PostgreSQL paths of install/cli-install.php get the same coverage the sqlite
3+
# build (.github/workflows/build.yml) already gives the sqlite one.
4+
#
5+
# php:<version>-cli, not one of the Salo runtimes: the checks here are the
6+
# installer, the migrations and the seeded data, and the built in web server is
7+
# enough to prove the installed site answers. Which web server serves it is what
8+
# the Salo images in .github/docker/Dockerfile are for.
9+
ARG PHP_VERSION=8.3
10+
FROM php:${PHP_VERSION}-cli
11+
12+
# libpq/libzip/libicu are the build inputs of the extensions below; the -dev
13+
# packages stay out of the final image only in multi stage builds, and a CI
14+
# image that lives for one workflow run does not earn the extra stage.
15+
RUN apt-get update \
16+
&& apt-get install -y --no-install-recommends \
17+
libpq-dev libzip-dev libicu-dev libxml2-dev libonig-dev unzip \
18+
&& docker-php-ext-install -j"$(nproc)" pdo_mysql pdo_pgsql zip intl \
19+
&& apt-get clean \
20+
&& rm -rf /var/lib/apt/lists/*
21+
22+
# Every diagnostic the installer or a migration emits has to reach the log the
23+
# entrypoint greps. display_errors alone is not enough: the installer runs the
24+
# migrations inside an output buffer it later discards, which would take the
25+
# warnings printed during them with it. Logging to stderr as well puts each one
26+
# outside the buffer, where nothing can swallow it.
27+
RUN { \
28+
echo 'error_reporting = E_ALL'; \
29+
echo 'display_errors = On'; \
30+
echo 'display_startup_errors = On'; \
31+
echo 'log_errors = On'; \
32+
echo 'error_log = /dev/stderr'; \
33+
} > /usr/local/etc/php/conf.d/evo-ci.ini
34+
35+
WORKDIR /var/www/html
36+
37+
# The repository ships core/vendor, so there is no composer step here and the
38+
# installer is called with --skipComposer=y. A copy rather than a bind mount:
39+
# the installer rewrites core/config and deletes install/, which would leave a
40+
# developer running this locally with a dirty working tree.
41+
COPY . /var/www/html
42+
43+
RUN chmod +x .github/docker/ci/install-and-smoke.sh
44+
45+
ENTRYPOINT ["/var/www/html/.github/docker/ci/install-and-smoke.sh"]

‎.github/docker/ci/README.md‎

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
# Install and upgrade checks
2+
3+
Two scripts, one set of assertions:
4+
5+
- `install-and-smoke.sh` installs this tree onto a database and checks what it
6+
put there, then runs the updater over it to catch anything not idempotent.
7+
- `upgrade-and-smoke.sh` installs an **older release**, copies this tree over it
8+
the way unpacking a release archive would, and runs the updater. That is the
9+
only check that replays the migration chain over a schema this branch did not
10+
create - what every existing site does when it moves to a new release.
11+
12+
Both take a `sqlite`, `mysql` or `pgsql` database from `EVO_DB_TYPE`.
13+
14+
## Installing
15+
16+
Run either database leg from the repository root:
17+
18+
```sh
19+
docker compose -f .github/docker/ci/docker-compose.yml --profile mysql up \
20+
--build --abort-on-container-exit --exit-code-from install-mysql
21+
22+
docker compose -f .github/docker/ci/docker-compose.yml --profile pgsql up \
23+
--build --abort-on-container-exit --exit-code-from install-pgsql
24+
```
25+
26+
Clean up with `docker compose -f .github/docker/ci/docker-compose.yml --profile
27+
<mysql|pgsql> down -v`. The container works on a copy of the tree, not a bind
28+
mount, so a run leaves the working tree alone.
29+
30+
sqlite needs no container at all:
31+
32+
```sh
33+
EVO_DB_TYPE=sqlite EVO_APP_DIR="$PWD" .github/docker/ci/install-and-smoke.sh
34+
```
35+
36+
`PHP_VERSION`, `MYSQL_IMAGE_TAG`, `POSTGRES_IMAGE_TAG`, `MYSQL_PORT` and
37+
`POSTGRES_PORT` override the defaults (PHP 8.3, MySQL 9.7 LTS, PostgreSQL 18.6
38+
LTS, the servers' own ports).
39+
40+
## Upgrading
41+
42+
`upgrade-and-smoke.sh` needs two trees: the older one it installs and upgrades
43+
in place, and this one to copy over it. The older tree is left dirty, so give it
44+
a checkout that can be thrown away.
45+
46+
```sh
47+
git clone --depth 1 --branch 3.5.7 https://github.com/evolution-cms/evolution /tmp/evo-3.5.7
48+
EVO_FROM_DIR=/tmp/evo-3.5.7 EVO_NEW_DIR="$PWD" EVO_DB_TYPE=mysql EVO_DB_HOST=127.0.0.1 EVO_DB_USER=root EVO_DB_PASSWORD=secret .github/docker/ci/upgrade-and-smoke.sh
49+
```
50+
51+
Two things about the source version are worth knowing:
52+
53+
- **No released version installs on sqlite.** `cli-install.php` offered only
54+
`pgsql` and `mysql` through 3.5.7; the sqlite branch is new in this tree. So a
55+
sqlite upgrade leg has to take its source from a ref that carries it - the
56+
`nightly-3.5.x` tag, or any 3.5.x commit after it landed. A tag works here
57+
like any other ref once a release ships sqlite support.
58+
- **No `--skipComposer` before 3.5.8 either.** An older `composerUpdate()` runs
59+
whatever it finds at `core/vendor/bin/composer` and only warns when that is
60+
missing, so the script moves the shim aside: `core/vendor` is committed at
61+
every tag, and a composer update would test the network rather than the CMS.
62+
63+
The source install is checked for hard failures only. A release run on a newer
64+
PHP may emit deprecations that are not this branch's to fix, and they are not
65+
what is under test - the updater's own log is checked strictly.
66+
67+
## What CI runs
68+
69+
The `install` job in `.github/workflows/ci.yml` does not use the image above.
70+
It starts only the database service from this compose file — an official image,
71+
pulled, never built — and runs `install-and-smoke.sh` directly on the runner,
72+
whose PHP `shivammathur/setup-php` already provides with `pdo_mysql` and
73+
`pdo_pgsql` prebuilt. Building the image there would spend two or three minutes
74+
per leg compiling extensions the runner hands over ready-made, on every push.
75+
76+
The script only needs `EVO_DB_*` and a reachable server, so both paths run the
77+
same checks; the image is what makes a local run reproducible on a machine with
78+
no PHP on it, and what pins the PHP version when a version question is the one
79+
being investigated.
80+
81+
One PHP version, every database: what the job proves is the installer's database
82+
paths, which do not vary with the PHP minor. 8.3 and 8.4 are both covered by the
83+
analysis and unit test jobs. The sqlite leg starts no container.
84+
85+
The upgrade checks live in their own workflow, `.github/workflows/upgrade.yml`,
86+
off `push` and `pull_request` on purpose: an upgrade regression comes from a
87+
change to the migrations or the seeders, not from every commit, and each leg
88+
costs a full install plus a full update. It runs nightly, and on demand with the
89+
versions to go from and to as inputs (Actions -> Upgrade -> Run workflow):
90+
`from` takes one or more refs of any repository, `to` a ref of this one or blank
91+
for the checked out tree, which is the default - so the unreleased version is
92+
what an upgrade is tested against unless you say otherwise.
93+
94+
## What a run asserts
95+
96+
`install-and-smoke.sh` drives it:
97+
98+
1. The installer creates the database itself — neither server image pre-creates
99+
the one being installed into — and runs the whole migration and seed chain.
100+
2. Nothing PHP would call a diagnostic was printed. The installer runs the
101+
migrations inside an output buffer it later discards, so the image also
102+
routes `error_log` to stderr; a warning raised during a migration cannot be
103+
swallowed.
104+
3. `smoke.php --mode=install` checks what reached the database, reading the connection the
105+
installer just wrote: every table created, the columns the core migrations
106+
add, the seeded document, template, event names, settings, roles and
107+
permissions, the admin account with a hashed password and the Administrator
108+
role, and the bundled plugins and modules.
109+
4. The updater (`--typeInstall=2`) replays the same chain over the installed
110+
site, and `smoke.php` runs again — its row counts have to match the first
111+
run, which is what catches a migration or seeder that is not idempotent.
112+
5. The site answers over HTTP: the front page renders the seeded document, and
113+
the manager renders its login form.
114+
115+
An upgrade run asserts the same things through `smoke.php --mode=upgrade`, with
116+
the expectations that belong to a fresh install relaxed - the content and the
117+
settings are whichever ones the older site had, not the ones this installer
118+
would have chosen. Two checks replace them, and they are the point of the mode:
119+
120+
- the counts recorded from the older site by `--mode=baseline`, taken before the
121+
copy, must not have **dropped** anywhere. A seeded catalogue may grow, because
122+
topping those up is what the update seeder is for; losing rows is a bug.
123+
- `site_content`, `users` and `user_attributes` must be **unchanged**. Those are
124+
the site, not the release, and an upgrade may not touch them.
125+
126+
`manager_language` and `emailsender` stay exact in both modes: they are answers
127+
the operator gave the installer, and a seeder writing defaults over them would
128+
reset a live site.

‎.github/docker/ci/docker-compose.yml‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# Installs Evolution CMS onto a real database server and asserts the result.
2+
#
3+
# docker compose -f .github/docker/ci/docker-compose.yml --profile mysql up \
4+
# --build --abort-on-container-exit --exit-code-from install-mysql
5+
# docker compose -f .github/docker/ci/docker-compose.yml --profile pgsql up \
6+
# --build --abort-on-container-exit --exit-code-from install-pgsql
7+
#
8+
# One profile per database, so a failing leg names the database in its own
9+
# service. Run from the repository root: the build context is the whole tree.
10+
name: evo-ci
11+
12+
services:
13+
mysql:
14+
profiles: [mysql]
15+
# 9.7 is the LTS of the MySQL 9 series.
16+
image: mysql:${MYSQL_IMAGE_TAG:-9.7}
17+
# Published so the installer can also be driven from the host - which is
18+
# what CI does, to avoid building a PHP image for something the runner
19+
# already has. The install-* services below reach it over the network
20+
# compose creates and ignore this.
21+
ports:
22+
- "${MYSQL_PORT:-3306}:3306"
23+
environment:
24+
MYSQL_ROOT_PASSWORD: secret
25+
# No MYSQL_DATABASE on purpose: the installer is meant to create the
26+
# database itself, and that branch of checkConnectToDatabaseWithBase()
27+
# only runs when the database is missing.
28+
healthcheck:
29+
test: ["CMD", "mysqladmin", "ping", "-h", "127.0.0.1", "-uroot", "-psecret"]
30+
interval: 3s
31+
timeout: 5s
32+
retries: 30
33+
start_period: 30s
34+
35+
pgsql:
36+
profiles: [pgsql]
37+
image: postgres:${POSTGRES_IMAGE_TAG:-18.6}
38+
ports:
39+
- "${POSTGRES_PORT:-5432}:5432"
40+
environment:
41+
POSTGRES_USER: evo
42+
POSTGRES_PASSWORD: secret
43+
# The database the installer connects to first. The one it installs into
44+
# (EVO_DB_NAME below) is deliberately a different, missing database, so
45+
# the CREATE DATABASE branch of the installer is exercised.
46+
POSTGRES_DB: evo
47+
healthcheck:
48+
test: ["CMD-SHELL", "pg_isready -U evo -d evo"]
49+
interval: 3s
50+
timeout: 5s
51+
retries: 30
52+
start_period: 30s
53+
54+
install-mysql:
55+
profiles: [mysql]
56+
build:
57+
context: ../../..
58+
dockerfile: .github/docker/ci/Dockerfile
59+
args:
60+
PHP_VERSION: ${PHP_VERSION:-8.3}
61+
depends_on:
62+
mysql:
63+
condition: service_healthy
64+
environment:
65+
EVO_DB_TYPE: mysql
66+
EVO_DB_HOST: mysql
67+
EVO_DB_USER: root
68+
EVO_DB_PASSWORD: secret
69+
EVO_DB_NAME: evolution
70+
EVO_DB_PREFIX: evo_
71+
72+
install-pgsql:
73+
profiles: [pgsql]
74+
build:
75+
context: ../../..
76+
dockerfile: .github/docker/ci/Dockerfile
77+
args:
78+
PHP_VERSION: ${PHP_VERSION:-8.3}
79+
depends_on:
80+
pgsql:
81+
condition: service_healthy
82+
environment:
83+
EVO_DB_TYPE: pgsql
84+
EVO_DB_HOST: pgsql
85+
EVO_DB_USER: evo
86+
EVO_DB_PASSWORD: secret
87+
EVO_DB_NAME: evolution
88+
EVO_DB_PREFIX: evo_

‎.github/docker/ci/install-and-smoke.sh‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
#!/bin/bash
2+
# Installs Evolution CMS onto the database named by the EVO_DB_* environment,
3+
# then proves the installation is sound: no diagnostics from the migrations, the
4+
# seeded data present, the updater able to replay the chain over it without
5+
# changing anything, and the site answering over HTTP.
6+
#
7+
# Runs as the container entrypoint (see Dockerfile) and, in CI, directly on the
8+
# runner; every knob has a default so `docker run` with only EVO_DB_HOST set
9+
# does something sensible. EVO_DB_TYPE=sqlite needs no server at all.
10+
#
11+
# For the upgrade of an older release to this tree, see upgrade-and-smoke.sh.
12+
# pipefail: the installer output goes through tee, and a pipeline that hides
13+
# the exit code of its first command would report a failed install as a pass.
14+
set -euo pipefail
15+
16+
APP_DIR=${EVO_APP_DIR:-/var/www/html}
17+
18+
export EVO_DB_TYPE=${EVO_DB_TYPE:-mysql}
19+
export EVO_DB_HOST=${EVO_DB_HOST:-127.0.0.1}
20+
export EVO_DB_USER=${EVO_DB_USER:-root}
21+
export EVO_DB_PASSWORD=${EVO_DB_PASSWORD:-secret}
22+
export EVO_DB_NAME=${EVO_DB_NAME:-evolution}
23+
export EVO_DB_PREFIX=${EVO_DB_PREFIX:-evo_}
24+
25+
# Exported: smoke.php reads these to check what the installer stored.
26+
export EVO_ADMIN=${EVO_ADMIN:-admin}
27+
export EVO_ADMIN_EMAIL=${EVO_ADMIN_EMAIL:-admin@evo.local}
28+
export EVO_ADMIN_PASSWORD=${EVO_ADMIN_PASSWORD:-Passw0rd123}
29+
export EVO_LANGUAGE=${EVO_LANGUAGE:-en}
30+
31+
# shellcheck source=lib.sh
32+
. "$(dirname "$0")/lib.sh"
33+
34+
LOG=/tmp/install.log
35+
UPDATE_LOG=/tmp/update.log
36+
37+
wait_for_db
38+
39+
say "Installing Evolution CMS (${EVO_DB_TYPE}, database '${EVO_DB_NAME}', prefix '${EVO_DB_PREFIX}')"
40+
# --skipComposer=y: core/vendor is committed, so there is nothing to install and
41+
# a composer update here would test the network rather than the CMS.
42+
# --removeInstall=n: keeping install/ lets the run report what it used, and the
43+
# sqlite leg of the build workflow already covers removal.
44+
# Interactive prompts read from stdin; </dev/null makes an unanswered prompt an
45+
# immediate failure instead of a hung job.
46+
mapfile -t db_args < <(installer_db_args)
47+
cd "$APP_DIR/install"
48+
set +e
49+
php cli-install.php \
50+
--typeInstall=1 \
51+
"${db_args[@]}" \
52+
--cmsAdmin="$EVO_ADMIN" \
53+
--cmsAdminEmail="$EVO_ADMIN_EMAIL" \
54+
--cmsPassword="$EVO_ADMIN_PASSWORD" \
55+
--language="$EVO_LANGUAGE" \
56+
--removeInstall=n \
57+
--skipComposer=y \
58+
< /dev/null 2>&1 | tee "$LOG"
59+
status=$?
60+
set -e
61+
[ "$status" -eq 0 ] || fail "the installer exited with ${status}"
62+
cd "$APP_DIR"
63+
64+
say "Checking the installer output for diagnostics"
65+
check_log "$LOG" "installer"
66+
# The banner is the last thing install() prints, and the only progress message
67+
# that survives: everything the installer says while migrating and seeding goes
68+
# into the output buffer index.php opens, which checkRemoveInstall() discards.
69+
# What actually reached the database is checked below instead.
70+
grep -q 'Now you use' "$LOG" || fail "the installer did not run to completion"
71+
72+
say "Checking the seeded data"
73+
php "$APP_DIR/.github/docker/ci/smoke.php" "$APP_DIR" || fail "the seeded data is not what the installer promises"
74+
75+
say "Running the updater over the site just installed"
76+
# The update path replays the same migration chain against a populated database,
77+
# which is the only way to find a migration that is not idempotent - and it is
78+
# what every existing site runs when it moves to a new release.
79+
cd "$APP_DIR/install"
80+
set +e
81+
php cli-install.php --typeInstall=2 --removeInstall=n < /dev/null 2>&1 | tee "$UPDATE_LOG"
82+
status=$?
83+
set -e
84+
[ "$status" -eq 0 ] || fail "the updater exited with ${status}"
85+
cd "$APP_DIR"
86+
87+
# update() prints its own "Evolution CMS updated!" before checkRemoveInstall()
88+
# discards the buffer, so a successful update says nothing at all; its exit code
89+
# above and the checks below are the evidence. Warnings still arrive, because
90+
# php.ini routes them to stderr as well.
91+
check_log "$UPDATE_LOG" "updater"
92+
93+
say "Checking the data survived the update unchanged"
94+
# Same checks, plus the row counts recorded by the run above: a seeder that
95+
# inserts instead of updating doubles its table here.
96+
php "$APP_DIR/.github/docker/ci/smoke.php" "$APP_DIR" || fail "the update changed the installed data"
97+
98+
say "Checking the installed site answers over HTTP"
99+
http_check "$APP_DIR"
100+
101+
say "OK: ${EVO_DB_TYPE} installation is clean, migrated and seeded"

0 commit comments

Comments
 (0)