[Proposal]: MySQL Package & Extension Manager #144

Pre-flight Checklist

• I have searched existing GitHub issues and did not find a duplicate proposal.
• I have removed or redacted sensitive information.

Primary Contact Name

Alex Gaetano Padula

Primary Contact Email

agpadula@tidesdb.com

Company / Organization

TidesDB Corp.

Role

Founder, CEO & Chief Architect

Additional Authors / Contributors

No response

Component

Server

Target Release (Optional)

MySQL 26.7.0 and later

Roadmap Section

Extensibility/Ecosystem

Related Issues / Pull Requests / References (Optional)

No response

Executive Summary

MySQL has one of the broadest extension surfaces of any relational database with eleven and growing implemented
plugin interfaces, pluggable storage engines with real alternatives, loadable functions, and a
component service registry in which one component may re-implement another's service. It has no
distribution mechanism for any of it.

Installing an out-of-tree extension today requires the operator to obtain the MySQL source tree at
the exact version they are running, place the extension inside it, configure and build the whole
server, and copy the resulting shared object into plugin_dir by hand. This is not a gap in
tooling; it follows from three properties of the server:

1. Storage engine plugins are bound to one exact server version.
   MYSQL_HANDLERTON_INTERFACE_VERSION is defined as (MYSQL_VERSION_ID << 8), and
   sql_plugin.cc sets both the lower and upper bound of its check from it, so the comparison
   collapses to equality. A plugin built against 26.7.0 is refused by 26.7.1 with "API version for
   STORAGE ENGINE plugin is too different". Patch releases count.
2. No development headers ship with the server. An installed 26.7.0 prefix contains only client
   headers (mysql_com.h, errmsg.h, field_types.h, mysql/). There is no handler.h and no
   mysql/plugin.h, so no extension of any kind can be compiled against an installed server.
3. Plugins can only be loaded from one directory, by bare filename.
   INSTALL PLUGIN ... SONAME passes the library name through check_valid_path(), which rejects
   any string containing a directory separator.

This proposal adds the missing layer, a package format, a compatibility key that makes artifact
reuse safe, a resolver that prefers verified pre-built binaries and falls back to building from
source, and a SQL surface for installing, updating and removing packages. It also proposes the two
small server changes without which none of it is possible - shipping the plugin SDK headers, and
publishing the build identity an artifact must match.

Two findings shape this proposals design:

• Only three of the eleven plugin interfaces are version-locked (storage engine, daemon,
  information schema). The other eight carry their own interface-version constants that change only
  when that interface changes. A package of a portable kind can be published once and serve many
  server releases, so the expensive per-release build matrix applies to a minority of packages.
• MySQL's component loader is already scheme-dispatched and the scheme handler is itself a
  replaceable service. file:// is implemented, builtin:// is reserved with no implementation,
  and the loader resolves a scheme by acquiring the service named dynamic_loader_scheme_<scheme>.
  A pkg:// scheme therefore requires no change to the server at all. This has been prototyped and
  works.

User / Developer Stories

• As a DBA deploying an alternative storage engine, I want to install it with one statement
  against the server I already run, rather than reproducing the vendor's build environment.

• As a DBA planning a minor upgrade, I want to know before I upgrade which of my installed
  extensions have no artifact for the new server version, because today a 26.7.0 >> 26.7.1 upgrade
  silently invalidates every version-locked plugin and the server starts with tables it cannot open.

• As an extension author, I want to build against an installed server the way a PostgreSQL
  extension author builds against pg_config/PGXS, instead of distributing instructions that begin
  "clone the MySQL source".

• As a security-conscious operator, I want extension installation to be an audited, separately
  privileged action against an allow-listed registry with signature verification, rather than "copy
  this .so into plugin_dir as root".

• As a platform integrator, I want a declared dependency graph, because real engines are not
  self-contained - TideSQL links libtidesdb and pins it by major version, and a manager that ignores
  native dependencies has solved the easy half of the problem.

• As a MySQL maintainer, I want the extension ecosystem to grow without the server taking on
  responsibility for third-party code, so the trust boundary and the privilege model must be explicit.

Proposed Scope

• A package manifest format (TOML) describing a package's identity, kind, server compatibility,
  native dependencies, prebuilt artifacts and source build recipe.
• A compatibility key - server version, platform, toolchain ABI, build type - that determines
  whether a given artifact may be loaded into a given server.
• A resolver that selects a version satisfying both the request and the server, prefers a verified
  pre-built artifact, and falls back to building from a tagged source checkout.
• Installation support for the extension kinds MySQL already has registries for:
  plugins (mysql.plugin), loadable functions (mysql.func) and components (mysql.component).
• A SQL surface with INSTALL PACKAGE, UPDATE PACKAGE, REMOVE PACKAGE, SHOW PACKAGES,
  SHOW PACKAGE STATUS.
• A mysql.package system table recording, per installed package, the resolved version, the full
  key it was installed for, the constraint originally requested, and the artifact digest.
• A pkg:// dynamic loader scheme service for components.
• Pre-upgrade compatibility reporting driven by mysql.package.
• Two server-side prerequisites - ship the plugin SDK headers and a CMake package; publish the build
  identity (including the numeric version id and the ABI-affecting flag set).

Out of Scope / Future Work

• Operating a public registry (maybe on https://planet.mysql.com/)?. This specifies the client/server mechanism and the manifest format, not who hosts an index or funds a build farm.
• Packages that carry SQL objects. PostgreSQL extensions install SQL objects with versioned
  upgrade scripts. That would make loadable-function and information-schema packages considerably
  more useful and is a natural second phase, but it reopens the upgrade-script design question and
  is deliberately excluded here.
• Sandboxing the build helper beyond process isolation. A first version runs builds
  out-of-process and unprivileged; container or seccomp confinement is future work.
• Cross-version artifact compatibility. Nothing here attempts to create a stable
  storage-engine ABI. The proposal works within the exact-version rule rather than changing it.
• Automatic rebuild on server upgrade. The proposal reports and stages; it does not silently
  rebuild extensions during an upgrade.
• Windows and macOS packaging specifics. The design is platform-neutral and the key has a
  platform field, but only Linux x86-64 has been exercised in prototypes.

References

Functional Requirements

Non-functional Requirements

Impact Areas

• SQL syntax or statements
• Configuration options or system variables
• Command-line options or utilities
• User-visible behavior
• Observability
• Security or privilege model
• Protocol or replication behavior
• Upgrade / downgrade compatibility
• Performance or resource usage
• Files, persistence, or metadata formats
• APIs or internal interfaces
• Testing or QA coverage needs

Summary of the Approach

An extension artifact is addressed by a compatibility key with four fields:

tidesdb/2.0.0 @ mysql-26.7.0 / linux-x86_64 / gcc13-cxx11abi / release
               └─ server ─┘   └─ platform ─┘   └─ toolchain ─┘  └ build ┘

Each field corresponds to a demonstrated failure mode rather than defensive over-specification:

Resolution prefers a verified prebuilt artifact and falls back to a source build. The server drives
the operation through SQL; the work of fetching, verifying and building is delegated to a separate
process. Installed state lives in mysql.package, which is what makes updates and pre-upgrade
checks possible.

User Interface

-- install
INSTALL PACKAGE 'tidesdb';                      -- latest compatible with this server
INSTALL PACKAGE 'tidesdb' VERSION '2.0.0';      -- exact
INSTALL PACKAGE 'tidesdb' VERSION '^2.0';       -- range
INSTALL PACKAGE 'tidesdb' FROM 'vendor-registry';

-- update, re-resolving under the recorded constraint
UPDATE PACKAGE 'tidesdb';
UPDATE PACKAGE 'tidesdb' VERSION '2.1.0';       -- move the constraint deliberately

-- remove
REMOVE PACKAGE 'tidesdb';

-- inspect
SHOW PACKAGES;
SHOW PACKAGE STATUS 'tidesdb';
SELECT * FROM mysql.package;

Components additionally become installable through the existing loader with no new syntax, by
registering a scheme service:

INSTALL COMPONENT 'pkg://some_component@1.2';

Configuration / Knobs

New read-only variables publishing the build identity (prerequisite, see §5):

Observability

• mysql.package - one row per installed package >> name, resolved version, kind, compatibility
  key, requested constraint, artifact digest, install timestamp, source (binary or source).
• SHOW PACKAGE STATUS <name> - resolved version against available versions, whether an update
  is available under the recorded constraint, and whether a staged-but-pending artifact exists.
• Audit events - one per install, update and removal, carrying resolved key and digest, so
  extension installation is reviewable after the fact.
• Error log - resolution failures name the key sought and the keys available; build failures
  include the helper's captured output, truncated.
• Performance Schema (optional) - stages for resolve, fetch, verify, build, stage,
  load, so a slow install is attributable.

User Procedure

Installing an engine on a server that has a matching prebuilt artifact:

mysql> SET GLOBAL package_registry_allowlist = 'https://pkg.example';
mysql> GRANT INSTALL_PACKAGE ON *.* TO 'dba'@'localhost';
mysql> INSTALL PACKAGE 'tidesdb';
mysql> SELECT * FROM mysql.package;
mysql> CREATE TABLE t (id INT PRIMARY KEY) ENGINE=TidesDB;

Preparing for a server upgrade:

mysql> SHOW PACKAGE STATUS FOR SERVER '26.7.1';
+---------+---------+------------------+--------------------------------+
| package | version | target_available | note                           |
+---------+---------+------------------+--------------------------------+
| tidesdb | 2.0.0   | NO               | no artifact for mysql-26.7.1   |
| connlog | 1.0.0   | YES              | portable kind, range satisfied |
+---------+---------+------------------+--------------------------------+

The operator stages the new-key artifacts before upgrading, so the upgraded server finds its
extensions already present rather than starting with unloadable plugins.

Security Considerations

Server-driven installation is the most useful shape and the most dangerous. It gives mysqld
outbound network access and, on the source path, a compiler. Four containments, all mandatory:

1. Trust. Manifests and artifacts are signed; the server verifies against trust roots configured
   at startup, never material supplied by the registry. An unsigned or unknown-signer package is
   refused with no override flag - an override becomes the documented workaround.
2. Privilege. INSTALL_PACKAGE is its own dynamic privilege, default-granted to nobody, and is
   not implied by SUPER or INSTALL PLUGIN. Installing a package is strictly more dangerous than
   loading a file an administrator already placed on the host.
3. Network. package_registry_allowlist is empty by default. There is no implicit default
   registry, so a server that is never configured for packages cannot be induced to fetch anything.
4. Build isolation. The source path never compiles inside mysqld. mysql_package_build runs
   as a separate unprivileged process with no database connection, a scratch directory, and the
   named recipe from the manifest. On hosts with no toolchain the helper is absent and the source
   path reports that plainly.

Residual risks that should be stated rather than designed away, a signed package is still arbitrary
code in the server's address space; the registry operator is a trusted party; and the build helper
executes a recipe from a manifest, which is why package_allow_source_build defaults off and why a
binary-only first version is a reasonable scope reduction.

Compatibility and Behavior Changes

• No change to existing extension loading. INSTALL PLUGIN, CREATE FUNCTION … SONAME and
  INSTALL COMPONENT 'file://…' behave exactly as before.
• New system table. mysql.package requires a dictionary upgrade step. A downgrade leaves the
  table present and unused.
• The plugin SDK becomes a published interface. This is the significant new commitment, once
  headers ship, their contents become something third parties build against. The SDK should
  therefore carry its own version and an explicit stability statement - in particular that it offers
  no cross-version guarantee for the version-locked kinds, matching the loader's existing rule.
• Replication. Unaffected; package state is local to a server.
• Upgrade hazard addressed, not introduced. A server version change already invalidates every
  version-locked extension. This proposal makes that visible and recoverable instead of leaving it
  to be discovered at startup.

Block Diagram

flowchart TD
    A["INSTALL PACKAGE 'tidesdb'"] --> B["Privilege check: INSTALL_PACKAGE"]
    B --> C["Read compatibility key from this server"]
    C --> D["Resolve version: constraint AND server compat"]
    D --> E["Verify manifest signature against trust roots"]
    E --> F["Resolve native dependencies from [requires]"]
    F --> G{"Prebuilt artifact<br/>for this key?"}
    G -- yes --> H["Verify sha256"]
    G -- no --> I{"package_allow_source_build?"}
    I -- no --> J["Fail: report key sought + keys available"]
    I -- yes --> K["mysql_package_build:<br/>unprivileged, no DB access"]
    K --> L["Run named recipe against SDK"]
    L --> H
    H --> M["Stage into plugin_dir or package_store_dir"]
    M --> N{"kind"}
    N -- "plugin / storage_engine" --> O["INSTALL PLUGIN"]
    N -- loadable_function --> P["CREATE FUNCTION ... SONAME"]
    N -- component --> Q["INSTALL COMPONENT 'pkg://name'"]
    O --> R["Record in mysql.package + audit event"]
    P --> R
    Q --> R

Interface Specification

Manifest format. TOML, one file per package version.

[package]
name        = "tidesdb"
version     = "2.0.0"
kind        = "storage_engine"      # selects the version-locking rule
plugin_name = "TidesDB"          # what INSTALL PLUGIN registers
soname      = "ha_tidesdb.so"
license     = "GPL-2.0"
maturity    = "beta"

[compat]
server = "=26.7.0"               # exact storage engines are version-locked
                                            # an audit plugin would write server = ">=8.0.0"

[requires]
libtidesdb = "^10.1.1"           # native dependency, major pinned

[[binary]]
key    = "mysql-26.7.0/linux-x86_64/gcc13-cxx11abi/release"
url    = "https://pkg.example/tidesdb/2.0.0/…"
sha256 = "…"

# for multiplatform / architecture 
[[binary]]
key    = "mysql-26.7.0/linux-aarch64/gcc13-cxx11abi/release"
url    = "https://pkg.example/tidesdb/2.0.0/linux-aarch64.so"
sha256 = "…"

[[binary]]
key    = "mysql-26.7.0/macos-arm64/clang17-libc++/release"
url    = "…"
key    = "mysql-26.7.0/macos-arm64/clang17-libc++/release"
url    = "…"
sha256 = "…"

[[binary]]
key    = "mysql-26.7.0/windows-x64/msvc193/release"
url    = "…"
sha256 = "…"

[source]
git      = "https://github.com/tidesdb/tidesql-mysql"
tag      = "v2.0.0"
build    = "./install.sh --rebuild-plugin --mysql-src ${MYSQL_SDK_SRC}"
produces = "ha_tidesdb.so"

kind is what lets the resolver apply the right version rule without the manifest restating it.
[source] is a named recipe, not an arbitrary shell string, so the build helper has something it
can constrain.

mysql.package system table.

The pkg:// scheme service. No core change is required. The loader resolves a scheme by name:

/* components/libminchassis/dynamic_loader.cc, get_scheme_service_from_urn */
size_t scheme_end = urn.find("://");
my_string scheme(urn.begin(), urn.begin() + scheme_end);
my_service<SERVICE_TYPE(dynamic_loader_scheme)> service(
    (my_string("dynamic_loader_scheme_") + scheme).c_str(),
    &imp_mysql_minimal_chassis_registry);

A component providing dynamic_loader_scheme_pkg therefore extends a running server. The service
name is declared as an alias of the service type, exactly as the built-in file scheme does:

typedef SERVICE_TYPE_NO_CONST(dynamic_loader_scheme)
    SERVICE_TYPE_NO_CONST(dynamic_loader_scheme_pkg);

BEGIN_SERVICE_IMPLEMENTATION(pkg_scheme, dynamic_loader_scheme_pkg)
pkg_load, pkg_unload END_SERVICE_IMPLEMENTATION();

The scheme contract carries an invariant the service definition does not express. The
definition is only load and unload, but a scheme implementation must refuse to load the
same library twice. The built-in file scheme guards on the address of the library's
list_components entry point (dynamic_loader_scheme_file.cc:160-168); without that guard the
same component is registered twice and the second unload frees state the first still references.
A first prototype that omitted it crashed the server with SIGSEGV. If the scheme mechanism is to
be opened to third parties, this contract must be specified, or a helper supplied that enforces
it.

The plugin SDK. Measured from the compiler's own dependency output, not estimated:

storage engine header closure          366 headers,  5.1 MB
  include/ 188   sql/ 97   libs/ 69   sql-common/ 4   extra/ 4   vector-common/ 1
  generated 3 - my_config.h, mysql_version.h, mysqld_error.h   (configuration-specific)

+ component service headers           +161 headers
union across extension kinds              527 headers

Three properties of the SDK follow:

1. libmysqlservices.a already ships. It supplies mysql_malloc_service and
   plugin_registry_service, and is already installed into the prefix. Only the headers are
   missing, which makes this ask smaller than it appears.
2. The flag set is part of the ABI. A complete engine compiled and linked cleanly but would not
   load until built with the server's own -ftls-model=initial-exec, -D_FILE_OFFSET_BITS=64 and
   -Wl,-Bsymbolic-functions. Publishing a toolchain tag is insufficient; the flags themselves
   must be published.
3. The SDK must be a union, not a sample. The storage-engine closure appeared to be a superset -
   it already contained plugin_audit.h, udf_registration_types.h and
   component_implementation.h - yet a component failed to compile, because a storage engine never
   includes any components/services/*.h.

Proposed Implementation Plan

The ordering is chosen so that each phase is independently useful and the riskiest work comes last.

QA Notes

Correctness and refusal paths

Scheme-specific

Cross-version and upgrade

Automation opportunities

• The refusal matrix above is mostly table-driven and suits mysql-test-run with a fixture registry
  served from the test's own directory, no network needed.
• Cross-version refusal can be tested without a second server build by publishing a manifest whose
  artifact key names a fabricated version; the loader's check is what is under test.
• The duplicate-library guard deserves a dedicated crash-regression test, since it is an invariant
  a third-party scheme must honour and a prototype has already violated it.
• Digest and signature failures should be tested with deliberately corrupted artifacts rather than
  mocks, since the whole value of the key is that it is checked against reality.

Known gaps in current validation

• Only Linux x86-64 has been exercised. The platform and toolchain fields of the key are designed but
  untested on macOS and Windows.
• Signature verification is specified but not implemented in the prototype; only digests are checked.
• The server-driven surface (INSTALL PACKAGE) is designed and not implemented; the prototype drives
  installation from a client, which is the same resolution logic in a different process.
• libtidesdb-style native dependency resolution is declared in the manifest format and checked for
  presence only; version-range satisfaction across a dependency graph is not implemented.