Skip to content

fix(install): serialize module activation with an advisory file lock - #241

Open
kavix wants to merge 3 commits into
wso2:mainfrom
kavix:fix/serialize-module-activation
Open

kavix wants to merge 3 commits into
wso2:mainfrom
kavix:fix/serialize-module-activation

Conversation

@kavix

@kavix kavix commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Two shell processes can stage, move aside the same version, restore the shared .replaced directory, and write policy and active state independently. The final state can mix decisions from different installs (#240).

Solution

Serialize module activation per product namespace using the shell's advisory file lock (internal/lockfile) in the state root:

  • Added Store.LockPath(namespace) in internal/modules/store.go to return the namespace lock path (<store>/<namespace>.lock). Per ADR 0007, the lock sits beside the namespace directory so rollback cleanup of a failed first install never unlinks the lock file.
  • Wrapped Installer.activate() in internal/install/install.go inside lockfile.With(), bounding wait time under 15 * time.Second (above declarationTimeout of 10s plus margin to accommodate running the installed executable).
  • Contention and filesystem lock errors are mapped to storeFailure.
  • Added <namespace>.lock to the storage layout in docs/architecture.md.

Acceptance

  • Concurrent install test: TestConcurrentInstallsActivateCleanlyMatchingExactlyOneInstall in internal/install/install_test.go and TestConcurrentInstallsMatchExactlyOneInstall in test/acceptance/moduleinstall_test.go.
  • Failure-injection test: TestInterruptedInstallLeavesPreviousVersionActive in internal/install/install_test.go exercising same-version reinstall rollback from .replaced.

Closes #240

@coderabbitai

coderabbitai Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 2b6def9a-78a0-483f-a79c-98ed26aed009

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@CLAassistant

CLAassistant commented Sep 23, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@kavix
kavix force-pushed the fix/serialize-module-activation branch from f65e8e4 to 5e5f78a Compare September 23, 2026 16:14
@kanushka

Copy link
Copy Markdown
Contributor

The concurrent-install test passes and the namespace lock addresses the main race. Before merging, could you please:

  • run gofmt on the three new test files;
  • add the persistent <namespace>.lock file to the storage layout in docs/architecture.md;
  • strengthen the failure-injection test to exercise a same-version reinstall after the existing version has moved to .replaced. The current different-version validation failure does not test that rollback path.

@kavix

kavix commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review @kanushka! I have updated the branch with the requested changes

@kanushka kanushka left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lock deadline can expire while a healthy activation still holds the lock.

Comment thread internal/install/install.go Outdated
Comment thread internal/install/install.go
kavix added 3 commits October 10, 2026 14:18
Serialize activation per product namespace with an advisory file lock in
the state root, preventing concurrent shell processes from interleaving
staging, moving aside versions, restoring .replaced, or writing policy
and active state.

- Add Store.LockPath(namespace) placing the lock file beside the namespace directory
- Wrap Installer.activate() in lockfile.With()
- Add concurrent install test and failure-injection rollback test
- Add end-to-end acceptance test for concurrent installs

Fixes wso2#240

Signed-off-by: kavix <kavix@yahoo.com>
… layout

- Run gofmt on new test files
- Add <namespace>.lock to storage layout in docs/architecture.md
- Strengthen failure-injection test to verify rollback from .replaced on same-version reinstall failure

Signed-off-by: kavix <kavix@yahoo.com>
@kavix
kavix force-pushed the fix/serialize-module-activation branch from 4e37f37 to a8737a5 Compare October 10, 2026 08:50
@kavix

kavix commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks @kanushka! Updated lockDeadline to 15s (above declarationTimeout), updated the comment to mention the executable run, and rebased onto main.

@kavix
kavix requested a review from kanushka October 10, 2026 08:58

@kanushka kanushka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the lock is in the right place and all earlier comments are addressed. Two things before merging:

  • The concurrency tests don't catch the race. With the install.go from main and the tests from this PR, TestConcurrentInstallsActivateCleanlyMatchingExactlyOneInstall passes 300/300 and TestConcurrentInstallsMatchExactlyOneInstall passes 20/20. Both install different versions (1.0.0 and 1.1.0), so they never share .replaced. Could you make them run concurrent installs of the same version, or add a test hook that pauses activation between staging and the policy write, and confirm they fail without the lock?
  • The busy-lock error is wrong (internal/install/install.go:353-358). When the lock is held, lockfile.ErrBusy goes through storeFailure. The user is told "locking the module namespace failed: lockfile: …" and is asked to check the state-directory permissions. Please map busy the same way internal/contexts/save.go:169 does: another invocation is updating the product, retry the command. Please keep the lockfile: package name out of the message as well, and add a test for the busy path.

Non-blocking, fine as follow-ups:

  • Store.Remove (wso2 product remove) doesn't take the namespace lock, so a remove that runs during an install can leave a half-removed namespace.
  • updateOne reads the channel and pin before taking the lock, so a pin that lands in between can be overwritten.
  • Per ADR 0007, declarationTimeout should point back to lockDeadline, since the two values are coupled.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Concurrent installs of the same module namespace can leave mixed state

3 participants