Upgrade certifi, pip, uv and Node.js for the eMudhra CA incident #1515

Activity

1. tutor-project-automation

   added this to Tutor project managementon Oct 22, 2026
2. github-project-automation

   moved this to Pending Triage in Tutor project managementon Oct 22, 2026
3. ahmed-arb commented on Oct 23, 2026

   

   ahmed-arb

   on Oct 23, 2026

   CollaboratorAuthor

   More actions

   Smoke results (smoke.yml on GitHub runners). Plugin images were built from the PR branches by installing them from git in requirements/plugins.txt on throwaway branches.

   Images	Verawood	Ulmo	Teak
   openedx	✅	✅	✅ build + launch
   credentials, discovery	✅	✅	✅ build + launch
   notes, xqueue	✅	✅	✅ build + launch
   mfe	✅	✅	✅ build + launch
   android	✅	✅	✅ build + launch

   Tutor 20 has no do tests smoke job, so Teak runs stop after a successful build and launch.

   Fixes found along the way:

   • discovery: NODE_OPTIONS=--openssl-legacy-provider for make static, since its webpack hashes with md4 (all lines).
   • mfe (Ulmo): keep npm 11.6.1, since npm 11.19.1 from Node.js 24.22.1 rejects the frontend-app-communications lockfile.
   • tutor: PIP_BUILD_CONSTRAINT, since pip 26 no longer applies PIP_CONSTRAINT to build dependencies (all lines), plus a backport of fix: pin build-time setuptools to stabilize source builds #1411 to Teak.
4. Abdul-Muqadim-Arbisoft commented on Oct 23, 2026

   

   Abdul-Muqadim-Arbisoft

   on Oct 23, 2026

   Collaborator

   More actions

   I built the Verawood images from the PR branches locally and checked each one against the four hosts from https://emudhra.we-have-a-problem.com/, with whatever TLS clients the image has (python system store, certifi and requests, curl, node, uv, and java for android). For comparison, the current 22.x openedx, credentials, discovery, notes and xqueue images all still trust dv-g3b, so the rebuilds are definitely needed.

   Verawood PR	Result
   #1516	openedx image rejects everything, uv 0.13.5 included. openedx-dev gets the old certifi back from development.txt (commented there)
   overhangio/tutor-mfe#366	builds, node 24.22.1 rejects the bad chains, and the build stage pulls the fixed bookworm ca-certificates
   overhangio/tutor-credentials#83	builds and passes
   overhangio/tutor-discovery#133	passes with the legacy OpenSSL provider change
   overhangio/tutor-notes#63	builds and passes
   overhangio/tutor-xqueue#51	passes with the certifi step, also when built from release/verawood.1
   overhangio/tutor-android#66	builds, and both the system store and java reject the bad chains

   One thing to keep in mind for the release order: tutor-mfe takes the Caddy image for its production image from DOCKER_IMAGE_CADDY in core, so the MFE image only gets the fixed Caddy once the core release with #1516 is out.

   I read through the Ulmo and Teak PRs as well. They match the Verawood fixes (certifi step after the requirements install everywhere, the discovery legacy provider, node 22.23.4/24.22.1), but I didn't build those locally. The one gap is the same dev image issue in #1518 and #1517: release/ulmo.4 and release/teak.3 pin certifi 2025.10.5 and 2025.1.31 in development.txt, and both still trust the bad chain. I left comments on both.

   A few official plugins aren't on the list yet:

   • tutor-jupyter: the current hub image and the jupyterhub/singleuser:4 base for the lab image trust all four hosts, including the AAA cross-signed ones, since their stores are from 2024. The lab Dockerfile's apt install -y git leaves ca-certificates at 20230311 even though 20260816 is available, and conda's own certifi and ca-certificates are 2024.7.4. Both images need an explicit ca-certificates upgrade plus pip and certifi, and the lab image needs the conda packages updated too.
   • tutor-cairn: apache/superset:4.0.0 trusts all four hosts (certifi 2023.7.22), and the plugin's apt step leaves ca-certificates at 20230311 even though 20250419deb12u2 is available. The SSO calls the LMS API over https when ENABLE_HTTPS is on, so it needs a certifi upgrade and a ca-certificates upgrade. clickhouse-server:24.2.3.70 is on ubuntu 20.04, and the public focal archive has no fixed ca-certificates (the newest is 2024020320.04.1), so that one needs a newer base image.
   • tutor-objectstorage (main only for now): the rclone (1.75.1), rustfs (1.0.0) and rustfs/rc (v0.1.36) images all predate the disclosure. rclone talks to Azure over TLS when it's pointed at a real account, so it should get bumped once upstream ships fixed images.
   • tutor-minio: it's pinned to 2022 MinIO builds from the Rancher mirror and there's nothing newer to move to. Since it's deprecated and goes away in Willow, maybe just mention this in the notice and point people to tutor-objectstorage.

   indigo, forum, livedeps and deck don't need anything, they either run inside the openedx/mfe images or on the host.
5. arbrandes

   added 2 commits that reference this issue on Oct 23, 2026

   build: upgrade Node.js

   ...

   3e480d0

   build: upgrade Node.js

   ...

   8af3b6e
6. ahmed-arb commented on Oct 26, 2026

   

   ahmed-arb

   on Oct 26, 2026

   CollaboratorAuthor

   More actions

   Added the dev-stage certifi step and fixed the changelogs on #1516, #1517 and #1518; I hadn't tested openedx-dev since the issue only matters on production (deployed) instances.
7. Abdul-Muqadim-Arbisoft

   added 3 commits that reference this issue on Oct 26, 2026

   build: upgrade Caddy (#66)

   ...

   9ce1c3a

   build: upgrade Caddy (Ulmo backport) (#68)

   ...

   18b6c85

   build: upgrade Caddy (Teak backport) (#67)

   ...

   0d6be61