[andrew8er]

Short description

Run tests in a Docker Compose environment, on dev machines as well as CI. The proposed changes result in a speedup for local development (a major speedup on ARM machines anyway) and comparable runtime on CI, but with an 8-fold reduction in parallelism (no parallelism used actually), using the same resource_class: large. This translates into a substantial reduction in spent credits per run.

We could continue to use parallelism on CI, but the speedup would not scale linearly while complicating the test setup, and there are a few optimizations still on the table that we should try first. I therefore decided to not use parallelism for now.

Proposed changes

The current test setup tries to make the tests' Docker environment match the CircleCi environment. This is misguided IMHO. The dev and test environments (including the CI environment) should instead strive to match the production environment.

Pinning the Docker images' architectures on the other hand takes a huge performance toll when there is a mismatch, and in fact, running the tests on Arm MacBooks is currently practically impossible (see table below). Also, the actual benefit of pinning is dubious. The chances of encountering a significant behavioral difference between architectures is pretty small anyway, and if we do encounter one, it will be immediately obvious with the proposed setup.

Test run	time
original, local, test.sh (on ARM MacBook)	28m25s (!)
PR test.py	2m29s
PR test.py --preset coverage	6m40s

Main changes

• Move pytest settings to pytest.toml. I really think it is not a good idea to overload the pyproject.toml with all kinds of settings as it becomes unwieldy. It also has the side effect that the Docker images get rebuilt on every settings change, even when the settings do not affect the build at all. pyproject.toml should only contain project metadata, dependencies and build settings. Anything that you might want to ship to pypi and that affects the package build process basically.

• Refactor the test.sh shell script to a Python script (test.py), which is more portable and hopefully easier to understand. The script offers some predefined presets – sets of pytest arguments – that will be passed to the pytest runner. The currently available presets are: changed, ci, coverage and default.

  It also has a --local option for running tests locally (please test this). All other arguments will be passed to pytest as is.

• The webpack-stats.json file was moved to integreat_cms/static/shared. I think it is the more appropriate place, and it needs to be in a separate directory for reasons explained in the next section.

• Improve the Docker Compose setup. A few special settings worth mentioning:

  • The functionality from test_entrypoint.sh is now included in test.py, since the Dockerfile.dev image is supposed to be universally usable in the future. Also, this makes the setup less complicated.
  • Put the docker-compose.test.yaml at the project root and use official images for our 3rd-party services as well as a special custom image built from Dockerfile.dev, that inherits from debian:trixie to match our production setup.
  • On CI, the Docker build tries to use our Github package registry (ghcr.io) as a layer cache, which significantly speeds up image building. It will fall back to using no cache in case of an error (e.g. Github outage). This is to avoid CircleCI's Docker layer cache (DLC), which is ridiculously expensive.
  • The image built from Dockerfile.dev contains all Python as well as NPM dependencies compiled in and will be rebuilt when any of those changes. In the case of a change, rebuilds should still be relatively fast, since there are cache mounts in place for apt, uv and npm during the image building process.
  • There is a special web service that compiles the web assets via webpack.
  • The cms and web services share a shared directory in /app/integreat_cms/static/shared that contains the webpack stats file. This is a tmpfs mount, so the local files stay unaffected.
  • The cms service /app/integreat_cms/pdf directory also gets a tmpfs mount, so it is always clean and local files are unaffected.
  • The db service gets a tmpfs mount on /var/lib/postgresql, so Postgres effectively becomes an in-memory DB. This gives a major speedup.

Minor changes

• Ignore some pointless ruff rules around TODOs. It should not be a major chore to write a TODO, otherwise nobody will do it.
• Fix some formatting.
• Upgrade CircleCI orb qltysh/qlty-orb.

Side effects

There should be no side effects.

Faithfulness to issue description and design

There are no intended deviations from the issue and design.

How to test

Note

Usage with Podman works now, but the default machine VM size created by Podman is too small to run tests in parallel. Run the following commands to resize your machine:

podman machine stop
podman machine set --memory 8192
podman machine start

Then prepend CONTAINER_ENGINE=podman to the commands below.

• Run tools/test.py.
• Run tools/test.py --preset ci.
• Run tools/test.py --preset changed.
• Run tools/test.py --preset coverage.
• Run tools/test.py --local (if possible).
• Run tools/test.py with any arguments you usually need.
• Examine CI run.

🟢 I would really like to know, if you experience any speed-ups or downs with your setup!

[jarlhengstmengel]

I'm a bit unsure if I know how the testing should run 😅 My setup is with VSCode and a devcontainer. The testing functionality of VSCode doesn't work correctly for me, none of the tests aren passing so something does not work for that yet. I did get ./tools/test.py --local to run with installing psql and adding the DB variables in test.py. Somehow the via the iniit file set ones weren't succesfully set in test.py. When it ran, it took about 5 minutes to complete.
But I have the feeling I'm aproaching this completely wrong 😅

[andrew8er]

@jarlhengstmengel

The test script is currently only meant to run "as is", i.e. directly from a local shell, not within a devcontainer. I have not looked into this yet, but I guess, in a devcontainer setup, your IDE's shell is running in a container itself. That means running Docker is theoretically possible ("docker in docker") but a lot of features, such as volume mounts are not present.

I will see, how I can integrate this, but it should actually be not very different from the state before. The previous shell script started "docker compose" as well, just with different settings.

[andrew8er]

@jarlhengstmengel I improved the argument forwarding of test.py, maybe that fixes your problem.

[jarlhengstmengel]

It's building the container, but I'm encountering the following error when running tools/test.py

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/app/.venv/lib/python3.13/site-packages/_pytest/main.py", line 326, in wrap_session
INTERNALERROR>     config._do_configure()
INTERNALERROR>     ~~~~~~~~~~~~~~~~~~~~^^
INTERNALERROR>   File "/app/.venv/lib/python3.13/site-packages/_pytest/config/__init__.py", line 1205, in _do_configure
INTERNALERROR>     self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR>     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/app/.venv/lib/python3.13/site-packages/pluggy/_caller.py", line 222, in call_historic
INTERNALERROR>     res = self._hookexec(self.name, self._hookimpls.copy(), kwargs, False)
INTERNALERROR>   File "/app/.venv/lib/python3.13/site-packages/pluggy/_manager.py", line 183, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/app/.venv/lib/python3.13/site-packages/pluggy/_execution.py", line 195, in _multicall
INTERNALERROR>     raise exception
INTERNALERROR>   File "/app/.venv/lib/python3.13/site-packages/pluggy/_execution.py", line 147, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/app/.venv/lib/python3.13/site-packages/testmon/pytest_testmon.py", line 270, in pytest_configure
INTERNALERROR>     init_testmon_data(config)
INTERNALERROR>     ~~~~~~~~~~~~~~~~~^^^^^^^^
INTERNALERROR>   File "/app/.venv/lib/python3.13/site-packages/testmon/pytest_testmon.py", line 207, in init_testmon_data
INTERNALERROR>     testmon_data: TestmonData = TestmonData.for_local_run(
INTERNALERROR>                                 ~~~~~~~~~~~~~~~~~~~~~~~~~^
INTERNALERROR>         rootdir=config.rootdir.strpath,
INTERNALERROR>         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>     ...<2 lines>...
INTERNALERROR>         system_packages=system_packages,
INTERNALERROR>         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>     )
INTERNALERROR>     ^
INTERNALERROR>   File "/app/.venv/lib/python3.13/site-packages/testmon/testmon_core.py", line 197, in for_local_run
INTERNALERROR>     instance = cls(rootdir, database=database, readonly=False)
INTERNALERROR>   File "/app/.venv/lib/python3.13/site-packages/testmon/testmon_core.py", line 172, in __init__
INTERNALERROR>     self.db = db.DB(
INTERNALERROR>               ~~~~~^
INTERNALERROR>         os.path.join(self.rootdir, get_data_file_path()), readonly=readonly
INTERNALERROR>         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>     )  # pylint: disable=invalid-name
INTERNALERROR>     ^
INTERNALERROR>   File "/app/.venv/lib/python3.13/site-packages/testmon/db.py", line 67, in __init__
INTERNALERROR>     connection = connect(datafile, readonly)
INTERNALERROR>   File "/app/.venv/lib/python3.13/site-packages/testmon/db.py", line 25, in connect
INTERNALERROR>     return sqlite3.connect(
INTERNALERROR>            ~~~~~~~~~~~~~~~^
INTERNALERROR>         f"file:{datafile}{'?mode=ro' if readonly else ''}", uri=True, timeout=60
INTERNALERROR>         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>     )
INTERNALERROR>     ^
INTERNALERROR> sqlite3.OperationalError: unable to open database file

Running '['docker', 'compose', '-f', 'docker-compose.test.yaml', 'run', '--rm', '--remove-orphans', 'cms', 'uv', 'run', '/app/tools/test.py']' failed with code 3

[andrew8er]

@jarlhengstmengel

This is probably a UID mapping issue with Docker and bind mounts on Linux. These steps may help:

• Paste me the output of this command: ls -l tests-results.
• If there is a directory testmondata.docker, delete it: rm -r tests-results/testmondata.docker.
• Try again.
• If the same error appears, you could try this: chmod +rwx tests-results/.
• Try again.

[deliverino]

LLM PR Review (verdigado-code)

Analyzed diff, but not the whole repository. Important context might be missing.

Scope

The PR matches its description: Docker Compose test setup, test.sh → test.py, pytest.toml, webpack-stats relocation, ruff rule changes, CI orb upgrade. The mise.toml addition of circleci-cli (commit 3a75eebb) is not mentioned in the PR body.

Correctness

tools/test.py — wrong cwd for npm run build:
run(["npm", "run", "build"], check=True, cwd=_cms_package_dir, …) sets the working directory to integreat_cms/, but package.json and webpack.config.js live at the project root. This will fail in --local mode. Should be cwd=_project_home.

tools/test.py — tests-results/ not created in --local mode:
_test_results_dir.mkdir(exist_ok=True) is only in the Docker branch of _ensure_docker_compose_run. In --local mode the directory is never created, so pytest's COVERAGE_FILE / TESTMON_DATAFILE writes (and the JUnit XML path) will hit a non-existent path on a fresh checkout.

docker-compose.test.yaml — QUICK_ROLES not forwarded:
The old docker-compose.test.yml had QUICK_ROLES: ${QUICK_ROLES:-}. The new file omits it, so QUICK_ROLES=1 ./tools/test.py (still documented in docs/src/testing.rst) is silently ignored in Docker mode.

docker-compose.test.yaml — INTEGREAT_CMS_DB_NAME missing:
The old compose file set INTEGREAT_CMS_DB_NAME: integreat. The new file sets INTEGREAT_CMS_DB_HOST, INTEGREAT_CMS_DB_USER, and INTEGREAT_CMS_DB_PASSWORD but not the name. Whether this works depends on the default in integreat_cms/core/settings.py (not shown in the diff).

Dockerfile.dev — logo COPY hidden by bind mount:

COPY integreat_cms/static/src/logos/integreat/integreat-logo.png /app/integreat_cms/media/global/integreat-logo.png

The cms service bind-mounts ./integreat_cms:/app/integreat_cms, which overlays everything under /app/integreat_cms/. These files will not be visible in the container unless they also exist in the host tree. The # TODO Why do we need this? comment acknowledges the uncertainty.

tools/test.py — JUnit XML parent directory not created:
The ci preset writes to tests-results/junit/junit.xml, but neither the compose file nor the script creates the junit/ subdirectory. Whether pytest creates it automatically is version-dependent.

pytest.toml — may not be recognised:
Pytest's documented configuration files are pytest.ini, pyproject.toml ([tool.pytest.ini_options]), tox.ini, and setup.cfg. pytest.toml is not among them in any release I can verify. The DJANGO_SETTINGS_MODULE is always passed via --ds in test.py, so that setting is moot, but the markers (unit, slow) would go unregistered, producing PytestUnknownMarkWarning on -m unit / -m "not slow". Worth confirming the pinned pytest version actually reads this file.

--local mode regression:
The old test.sh ran require_installed, ensure_webpack_bundle_exists, and require_database (which could auto-start a Dockerised PostgreSQL). The new test.py --local only runs npm run build and compilemessages before pytest. A developer without a running database or with stale dependencies will get a raw connection / import error rather than a helpful message.

Conventions

• tools/test.py is a new Python file but lacks from __future__ import annotations after the module docstring.

Minor

• .circleci/config.yml — check-migrations still has attach_workspace but no longer needs the workspace contents (the Docker image is self-contained). It adds an unnecessary dependency on the webpack job completing.
• .gitignore — the new /shared entry (replacing the old webpack-stats.json line) matches a root-level shared path, not integreat_cms/static/shared/. The latter is already covered by the adjacent integreat_cms/static/shared/* line. The /shared entry appears to be a no-op or a typo.
• tools/test.py — npm run build has timeout=120; a full webpack build on a slow machine or a cold cache may exceed this.
• Commit 8f448c80 ("Update CircleCI config") is vague given the scope of changes (machine executor switch, removal of parallelism, Docker-based execution). A more specific summary would help.

[jarlhengstmengel]

Ok, I have a different error now 😅

Container integreat-cms-test-db-1 Waiting 
Container integreat-cms-test-web-1 Waiting 
Container integreat-cms-test-db-1 Healthy 
Container integreat-cms-test-web-1 Error service "web" didn't complete successfully: exit 2
service "web" didn't complete successfully: exit 2
Running '['docker', 'compose', '-f', 'docker-compose.test.yaml', 'run', '--rm', '--remove-orphans', 'cms', 'uv', 'run', '/app/tools/test.py']' failed with code 1

When I look into the logs for integreat-cms-test-web-1:

> build
> webpack build

[webpack-cli] ✖ Error: EACCES: permission denied, open '/app/integreat_cms/static/shared/webpack-stats.json'
    at Object.writeFileSync (node:fs:2380:20)
    at BundleTrackerPlugin._writeOutput (/app/node_modules/webpack-bundle-tracker/lib/index.js:118:8)
    at BundleTrackerPlugin._handleCompile (/app/node_modules/webpack-bundle-tracker/lib/index.js:142:10)
    at Hook.eval [as call] (eval at create (/app/node_modules/tapable/lib/HookCodeFactory.js:19:10), <anonymous>:7:1)
    at Hook.CALL_DELEGATE [as _call] (/app/node_modules/tapable/lib/Hook.js:16:14)
    at /app/node_modules/webpack/lib/Compiler.js:1594:23
    at Hook.eval [as callAsync] (eval at create (/app/node_modules/tapable/lib/HookCodeFactory.js:31:10), <anonymous>:6:1)
    at Hook.CALL_ASYNC_DELEGATE [as _callAsync] (/app/node_modules/tapable/lib/Hook.js:21:14)
    at Compiler.compile (/app/node_modules/webpack/lib/Compiler.js:1591:28)
    at /app/node_modules/webpack/lib/Compiler.js:700:12 {
  errno: -13,
  code: 'EACCES',
  syscall: 'open',
  path: '/app/integreat_cms/static/shared/webpack-stats.json'
}

[jonbulz]

Ok, I finally got around to testing this. I also run Apple Silicon, but the performance difference is not as dramatic for me (M3 Pro). Still an improvement, though 👍

Branch	Flags	Time in s
develop	--no-cov --numprocesses=8	131
develop	--numprocesses=8	190
improve-test-setup	--numprocesses=8	107
improve-test-setup	--cov=integreat_cms --numprocesses=8	126

I needed to add COMPOSE_DISABLE_ENV_FILE=1, otherwise my .env confuses docker. Would be nice to fix this, the old script passed --env-file /dev/null

@jarlhengstmengel

Ok, I have a different error now 😅

I also encountered this when I tried to compare the times to a linux system. It's a permission issue with the shared volume. Temporary fix is to change the permissions of /app/integreat_cms/static/shared manually inside the container:

DOCKER_USER=0:0 docker compose --env-file /dev/null -f docker-compose.test.yaml   run --rm --no-deps --user 0 web chown "$(id -u):$(id -g)" /app/integreat_cms/static/shared

The permanent fix should be in the Dockerfile.

I'll follow up with a more thorough review in the coming days

[jarlhengstmengel]

I also encountered this when I tried to compare the times to a linux system. It's a permission issue with the shared volume. Temporary fix is to change the permissions of /app/integreat_cms/static/shared manually inside the container:

Thank you @jonbulz, that helped!

I can run test.py now in 165 seconds on ubuntu. The old test.sh takes 475 seconds. So quit a speedup. I ran both without any other flags

[andrew8er]

@jonbulz
I added the COMPOSE_DISABLE_ENV_FILE env var to the docker compose command.

@jarlhengstmengel
The /app/integreat_cms/static/shared is now added to the repository (as an empty dir with a .gitkeep dummy file). Its contents are ignored however. That should fix the permission errors. When pulling, you might get an error now. In this case, you must delete this directory (possibly as root, depending on which exact permission it has).

Please give it another run and report any problems.