Add POSIX shared-memory support - #1809
Open
dg1sbg wants to merge 5 commits into
Open
Conversation
New src/core/shmem.cc binds shm_open/shm_unlink/ftruncate/mmap/munmap/ mprotect/msync/mlock/munlock/getpagesize plus a flag-constants alist and strerror, each as a CORE sys-* CL_DEFUN returning (values result errno). Registered in src/core/cscript.lisp.
src/lisp/kernel/lsp/shmem.lisp wraps the CORE sys-* primitives with keyword-flag translation, a syscall-error condition, a GC-finalized mapping object (idempotent munmap, definalize on close), the mmap family, and open/close-shared-memory plus with-shared-memory. Public symbols are re-exported through CLASP-POSIX. Loaded after fli.lisp in the base image so clasp-ffi pointer accessors are available; mapping-pointer returns a clasp-ffi:foreign-data usable directly by cffi:mem-ref. FTRUNCATE is defined on the CLASP-POSIX symbol to avoid clobbering CL:FTRUNCATE.
src/lisp/regression-tests/shmem.lisp covers round-trip persistence, two simultaneous MAP_SHARED mappings sharing backing, syscall-error on a missing segment, idempotent munmap with mapping-pointer-after-unmap error, and getpagesize. Registered in run-all.lisp. Byte access uses clasp-ffi (base image) so the suite runs under test-boehm.
The parent maps a named segment, forks, and the child writes a sentinel through the inherited MAP_SHARED mapping and hard-exits; the parent waits and reads the sentinel back, proving real shared memory between processes. The child stays allocation-minimal (Boehm HANDLE_FORK + single-threaded suite) and exits via core:cexit, matching the fork-server precedent.
CORE::MAPPING is already an exported Clasp class (hashTable.h Mapping_O, the abstract base of StrongMapping_O and WeakKeyMapping_O). Defining a struct of that name inside (in-package #:core) redefined it as a STRUCTURE-CLASS, so loading the image died with "When redefining a class, the metaclass can not change" -- every image built from this branch was unbootable, including any built before these fixes. Renamed to SHM-MAPPING with (:conc-name mapping-), so every accessor and the predicate keep their names; only the exported type name changes. MAPPING-POINTER returns a ForeignData holding no reference back to the MAPPING, while MMAP registered a GC finalizer that munmaps. Since the finalizer table is weak-keyed, dropping the mapping while retaining the pointer let the region be unmapped underneath a live pointer. Removed the finalizer; lifetime is now explicit via MUNMAP or WITH-SHARED-MEMORY, the same resolution taken for foreign-alloc in clasp-developers#1792. A POSIX shm object may be sized only by its creator: ftruncate on an existing one fails EINVAL regardless of access mode, and even to the same size. OPEN-SHARED-MEMORY defaulted :create to T and then ftruncated unconditionally, but plain O_CREAT means "create or open", so every call against an existing region failed in every direction. %SHM-OPEN-OR-CREATE now attempts O_CREAT|O_EXCL|O_RDWR first and only sizes the object when it actually created it, falling back to a plain open on EEXIST. Exposes EEXIST from shmem.cc for that test. Adds four regression tests covering the reopen and :input paths, the absence of a finalizer, and a mapping surviving GC while only its raw pointer is held.
dg1sbg
force-pushed
the
feat/posix-shmem
branch
from
August 13, 2026 16:38
015128a to
c88fb47
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds POSIX named shared memory to Clasp as a thin syscall layer plus a small Lisp ergonomics layer, exposed through
CLASP-POSIX.What this adds
src/core/shmem.cc— thinCL_DEFUNwrappers overshm_open,shm_unlink,ftruncate,mmap,munmap,mprotect,msync,mlock,munlock,getpagesize,strerror. Every wrapper returns(values result errno)witherrno0 on success, so no C++ exceptions cross the FFI boundary and there is no errno race.core__sys_shm_constantsreturns an alist of theO_*/PROT_*/MAP_*/MS_*values so the Lisp side never hardcodes platform numbers.src/lisp/kernel/lsp/shmem.lisp— flag handling that accepts a keyword, a list of keywords, or a raw integer; asyscall-errorcondition carryingerrnoand the call name with astrerror-based report; a single%checkedmacro that turns(values result errno)into either the result or a signalled condition; ashm-mappingstruct; and the high-levelopen-shared-memory,close-shared-memory, andwith-shared-memory.src/lisp/regression-tests/shmem.lisp— 10 tests: round-trip through a mapping, two mappings of one fd aliasing the same page, error on a missing object,munmapidempotence, page-size sanity, cross-process IPC viafork, plus the four regression tests described below.Notes for review
SHM-MAPPING, notMAPPING.CORE::MAPPINGis already an exported Clasp class —Mapping_OinhashTable.h, the abstract base ofStrongMapping_OandWeakKeyMapping_O. Adefstruct mappinginside(in-package #:core)redefines it as aSTRUCTURE-CLASS, and the image then fails to load with "When redefining a class, the metaclass can not change". The struct is therefore namedshm-mapping, with(:conc-name mapping-)so the accessors read naturally (mapping-address,mapping-size,mappingp). Happy to rename the accessors too if you'd prefer they carry theshm-prefix.Mappings are not GC-reclaimed, by design.
mapping-pointerhands out aForeignDatathat holds no reference back to theshm-mapping, so a GC finalizer that unmaps would be free to run while that pointer is still live — the finalizer table is weak-keyed, so nothing keeps the mapping alive. Rather than hand out a pointer that can be invalidated underneath the caller, there is no finalizer: lifetime is explicit viamunmap/close-shared-memory, withwith-shared-memoryfor the scoped case. This mirrors the resolution taken forforeign-allocin #1792.Only a creator may size a shm object.
ftruncateon an already-existing POSIX shm object failsEINVALregardless of the fd's access mode, and even when the requested size is unchanged.open-shared-memorytherefore attemptsO_CREAT|O_EXCL|O_RDWRfirst and sizes the object only when that succeeds, falling back to a plain open onEEXIST. Without this, any call against an already-existing region fails, in every:direction— which is the normal case for the second and every later participant.Testing
macOS arm64, native build, LLVM 22.1.8, rebased onto current
main(da70d6ff1):boehmboehmprecise(clean build from scratch)The four expected failures (
SBCL-CROSS-COMPILE-4,INCLUDE-LEVEL-2B,INCLUDE-LEVEL-3,TYPES-CLASSES-10) are pre-existing and unrelated. The count differs between variants only becauseboehmprecisealso runs the variant-specific snapshot tests.boehmpreciseis the interesting one here:shmem.cccompiles and links under precise stack scanning with no GC layout descriptor, confirming it introduces no managed C++ class. The cross-processforkIPC test passes, as does a probe that forces GC while holding only the raw mapping pointer — the failure mode the removed finalizer would have caused.For disclosure, the
boehmpreciserun came from a tree that also carried #1810 on top of this branch; #1810 touches onlyclos/slot-value.lispand no shmem code.