Skip to content

Add POSIX shared-memory support - #1809

Open
dg1sbg wants to merge 5 commits into
clasp-developers:mainfrom
dg1sbg:feat/posix-shmem
Open

Add POSIX shared-memory support#1809
dg1sbg wants to merge 5 commits into
clasp-developers:mainfrom
dg1sbg:feat/posix-shmem

Conversation

@dg1sbg

@dg1sbg dg1sbg commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Adds POSIX named shared memory to Clasp as a thin syscall layer plus a small Lisp ergonomics layer, exposed through CLASP-POSIX.

What this adds

src/core/shmem.cc — thin CL_DEFUN wrappers over shm_open, shm_unlink, ftruncate, mmap, munmap, mprotect, msync, mlock, munlock, getpagesize, strerror. Every wrapper returns (values result errno) with errno 0 on success, so no C++ exceptions cross the FFI boundary and there is no errno race. core__sys_shm_constants returns an alist of the O_* / PROT_* / MAP_* / MS_* values so the Lisp side never hardcodes platform numbers.

src/lisp/kernel/lsp/shmem.lisp — flag handling that accepts a keyword, a list of keywords, or a raw integer; a syscall-error condition carrying errno and the call name with a strerror-based report; a single %checked macro that turns (values result errno) into either the result or a signalled condition; a shm-mapping struct; and the high-level open-shared-memory, close-shared-memory, and with-shared-memory.

src/lisp/regression-tests/shmem.lisp — 10 tests: round-trip through a mapping, two mappings of one fd aliasing the same page, error on a missing object, munmap idempotence, page-size sanity, cross-process IPC via fork, plus the four regression tests described below.

Notes for review

SHM-MAPPING, not MAPPING. CORE::MAPPING is already an exported Clasp class — Mapping_O in hashTable.h, the abstract base of StrongMapping_O and WeakKeyMapping_O. A defstruct mapping inside (in-package #:core) redefines it as a STRUCTURE-CLASS, and the image then fails to load with "When redefining a class, the metaclass can not change". The struct is therefore named shm-mapping, with (:conc-name mapping-) so the accessors read naturally (mapping-address, mapping-size, mappingp). Happy to rename the accessors too if you'd prefer they carry the shm- prefix.

Mappings are not GC-reclaimed, by design. mapping-pointer hands out a ForeignData that holds no reference back to the shm-mapping, so a GC finalizer that unmaps would be free to run while that pointer is still live — the finalizer table is weak-keyed, so nothing keeps the mapping alive. Rather than hand out a pointer that can be invalidated underneath the caller, there is no finalizer: lifetime is explicit via munmap / close-shared-memory, with with-shared-memory for the scoped case. This mirrors the resolution taken for foreign-alloc in #1792.

Only a creator may size a shm object. ftruncate on an already-existing POSIX shm object fails EINVAL regardless of the fd's access mode, and even when the requested size is unchanged. open-shared-memory therefore attempts O_CREAT|O_EXCL|O_RDWR first and sizes the object only when that succeeds, falling back to a plain open on EEXIST. Without this, any call against an already-existing region fails, in every :direction — which is the normal case for the second and every later participant.

Testing

macOS arm64, native build, LLVM 22.1.8, rebased onto current main (da70d6ff1):

GC variant tree tested successes unexpected failures shmem tests
boehm this branch alone 1971 none 10/10
boehmprecise (clean build from scratch) this branch + #1810 1973 none 10/10

The four expected failures (SBCL-CROSS-COMPILE-4, INCLUDE-LEVEL-2B, INCLUDE-LEVEL-3, TYPES-CLASSES-10) are pre-existing and unrelated. The count differs between variants only because boehmprecise also runs the variant-specific snapshot tests.

boehmprecise is the interesting one here: shmem.cc compiles and links under precise stack scanning with no GC layout descriptor, confirming it introduces no managed C++ class. The cross-process fork IPC test passes, as does a probe that forces GC while holding only the raw mapping pointer — the failure mode the removed finalizer would have caused.

For disclosure, the boehmprecise run came from a tree that also carried #1810 on top of this branch; #1810 touches only clos/slot-value.lisp and no shmem code.

dg1sbg added 5 commits August 13, 2026 18:37
New src/core/shmem.cc binds shm_open/shm_unlink/ftruncate/mmap/munmap/
mprotect/msync/mlock/munlock/getpagesize plus a flag-constants alist and
strerror, each as a CORE sys-* CL_DEFUN returning (values result errno).
Registered in src/core/cscript.lisp.
src/lisp/kernel/lsp/shmem.lisp wraps the CORE sys-* primitives with
keyword-flag translation, a syscall-error condition, a GC-finalized
mapping object (idempotent munmap, definalize on close), the mmap family,
and open/close-shared-memory plus with-shared-memory. Public symbols are
re-exported through CLASP-POSIX. Loaded after fli.lisp in the base image so
clasp-ffi pointer accessors are available; mapping-pointer returns a
clasp-ffi:foreign-data usable directly by cffi:mem-ref. FTRUNCATE is defined
on the CLASP-POSIX symbol to avoid clobbering CL:FTRUNCATE.
src/lisp/regression-tests/shmem.lisp covers round-trip persistence, two
simultaneous MAP_SHARED mappings sharing backing, syscall-error on a
missing segment, idempotent munmap with mapping-pointer-after-unmap error,
and getpagesize. Registered in run-all.lisp. Byte access uses clasp-ffi
(base image) so the suite runs under test-boehm.
The parent maps a named segment, forks, and the child writes a sentinel
through the inherited MAP_SHARED mapping and hard-exits; the parent waits
and reads the sentinel back, proving real shared memory between processes.
The child stays allocation-minimal (Boehm HANDLE_FORK + single-threaded
suite) and exits via core:cexit, matching the fork-server precedent.
CORE::MAPPING is already an exported Clasp class (hashTable.h Mapping_O,
the abstract base of StrongMapping_O and WeakKeyMapping_O). Defining a
struct of that name inside (in-package #:core) redefined it as a
STRUCTURE-CLASS, so loading the image died with "When redefining a class,
the metaclass can not change" -- every image built from this branch was
unbootable, including any built before these fixes. Renamed to SHM-MAPPING
with (:conc-name mapping-), so every accessor and the predicate keep their
names; only the exported type name changes.

MAPPING-POINTER returns a ForeignData holding no reference back to the
MAPPING, while MMAP registered a GC finalizer that munmaps. Since the
finalizer table is weak-keyed, dropping the mapping while retaining the
pointer let the region be unmapped underneath a live pointer. Removed the
finalizer; lifetime is now explicit via MUNMAP or WITH-SHARED-MEMORY, the
same resolution taken for foreign-alloc in clasp-developers#1792.

A POSIX shm object may be sized only by its creator: ftruncate on an
existing one fails EINVAL regardless of access mode, and even to the same
size. OPEN-SHARED-MEMORY defaulted :create to T and then ftruncated
unconditionally, but plain O_CREAT means "create or open", so every call
against an existing region failed in every direction. %SHM-OPEN-OR-CREATE
now attempts O_CREAT|O_EXCL|O_RDWR first and only sizes the object when it
actually created it, falling back to a plain open on EEXIST. Exposes
EEXIST from shmem.cc for that test.

Adds four regression tests covering the reopen and :input paths, the
absence of a finalizer, and a mapping surviving GC while only its raw
pointer is held.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant