Fedora RPM Packaging Issues #137

Pre-flight checklist

• I have searched existing GitHub issues and MySQL Bug database - https://bugs.mysql.com/ - and did not find a duplicate.
• I have removed or redacted sensitive information.
• I can reproduce this issue or provide enough information for the team to investigate.

MySQL component

RPM Build / Packaging

MySQL version

Explicitly tested and checked against MySQL 26.7.0 fedora rpm packaging as this is your most up to date code.

• Expecting the same problems to be present on all earlier versions

Installation or build method

mysql-rpm-builder configuration

Operating system and platform

rpm packaging issues with fedora vs OL enterprise builds causing issues

Bug summary

There appear to be some packaging inconsistencies in the fedora rpm packaging in packaging/rpm-fedora compared to RHEL compatible builds in packaging/rpm-oel. These have lead me to some issues which I thought I'd report here. I hope to provide some fixes but need to discuss some issues found with the packaging maintainers first if this is possible to see how to resolve some of them.

These issues have come up based on me testing repeatable builds using my project on https://github.com/sjmudd/mysql-rpm-builder/ something to allow me to verify I can rebuild RPMs based on the packaging information provided by Oracle in the .spec files and the src rpms. I have found a few different issues in the RHEL builds and thought I would check how the fedora builds compare to them.

• The spec's %build cmake invocation is missing BuildRequires: for zstd/editline/lz4/libevent. This gap is not seen on the RHEL builds as other package dependencies implicitly pull these in. However, this does not happen on Fedora 44 so the build fails on a minimal docker container unless you explicitly install them first. That should not be necessary: the dependencies should be part of the BuildRequires: section in a spec file.
• rpm-fedora never passes -DWITH_AUTHENTICATION_CLIENT_PLUGINS=1 to cmake the way rpm-oel does, yet %files server/%files server-debug list the six resulting client auth-plugin .so files unconditionally, so a stock build fails. It's not clear to me how Oracle build these rpms in such circumstances but in my case if fails.
• A structural inconsistency between builds on Fedora vs RHEL-compatible builds is runselftest's default: rpm-oel sets it to 0 while rpm-fedora defaults it to 1 (run the full suite) on every Fedora major back to at least fc40. Doing the self-tests may be good but it exposes other problems.
• During rpmbuild -bas %check section the spec files invokes ./mtr which is mysql-test-run.pl. That script shells out to the openssl binary for feature detection but the binary is missing. Hence a BuildRequires: entry for this is needed.
• similarly the mysql-community-test package does not have Requires: openssl and could be bitten by a similar issue.
• %checks ./mtr includes a --mem option. This triggers the code to run the tests on a /dev/shm backed memory filesystem.
  • No documentation mentions what size this should be or ensures it should be large enough for the tests. My initial testing seems to imply that a value of 16 GB may be sufficient but this can have quite an impact on a low resource computer which may have insufficient memory to provide the backing store and to run the very parallelised MTR tests based on the number of CPUs present. In my case the OL builds work fine on a computer down to 16 GB of RAM, but I have so far been unable to complete the build due to memory pressure even on a 64 GB mini computer (with 12 cores).
  • In a docker container the default /dev/shm size is too small run the tests unless configured explicitly and when configured explicitly the MTR tests runs trigger heavy memory pressure (more than on the whole GCC build process) due to running a lot of things in parallel. My thought on this is that the code should know what data set size is needed and fail if insufficient space is available fall back to using a disk instead in such cases rather than failing.
  • The MTR runner perhaps should be a bit more aware of amount of memory and number of CPUs to configure an appropriate amount of parallelism to avoid OOMing the container. Currently it does not do that.

I can provide a PR to fix all of this as I have done in the past, or at least for the missing BuildRequires: settings but the fact I've been continually unable to build the fedora rpms unless I patch the spec file to disable the MTRs has really somewhat surprised me.

Reproducible rpm builds are good as you confirm that the build process you follow is fully reproducible by others. Often this is assumed but sometimes the environment used by the upstream packagers is less obvious than expected.

Also to be clear some of the rpm packaging may be similar or the same between rpm-fedora vs rpm-oel, but the issues manifest themselves different because:

• base OS containers for Fedora provide less packages than the OEL equivalents
• the MTR invocation in rpm-fedora triggers multiple issues. This is not invoked in rpm-oel so although it's likely some of the same issues exist there they are not applicable so do not surface.

Sorry for the long issue text here. I'm not sure how to more directly interact with the packagers which would be simpler.

Steps to reproduce

I can provide reproducible build information using my mysql-rpm-builder repo if desired.

Relevant configuration

I have several small home mini PCs, NUC like, and have been able to build RHEL compatible rpms on these from containers with at least 16 GB of RAM, ol7..ol10 and 8.0 and later.

Is this a regression?

No, Fedora rpms have always been built this way. It's just the OEL equivalents are built differently. Why this is I do not know.

• The MTR run behaviour is seen to be present since it began on FC40.

Impact

• Causes usability issues
• Inability to build rpms for Fedora from a docker container following a clear reproducible process.

Workaround

• adding the missing BuildRequires: sections will solve some of the issues
• disabling MTR in Fedora also resolves the issues, this requires a patch to in the git tree so a src rpm can be built and from there a reproducible builds works fine. You may not want to do that.
• Using a larger computer will solve the issue. Providing some minimal specs would be useful.

Additional context

Check my site for the workflows and references to different but related RHEL-compatible rpm build issues seen previously.

• I can't call the MTR testing a bug as clearly if you have a large enough machine this should work fine. However, it is useful to document a minimal build environment and the Fedora rpm builds seem to require significantly more resources than an EL build, which is somewhat unexpected and at least inconsistent.
• The missing BuildRequires: is I think a bug as dependencies are expected to be labelled in the rpm spec file and are missing.