EQL-specialized generic functions never memoize: every call is a full dispatch miss

[dg1sbg]

Summary

Any generic function with an eql specializer takes a full dispatch miss on every call. Its call history stays permanently empty, so compute-applicable-methods, compute-effective-method and a fresh effective-method-function run on every single call.

Class-specialized generic functions are unaffected.

Version: clasp-boehm-3.0.1-73-g2cf5bb5e4, macOS arm64. Reproduces identically on boehm and boehmprecise.

Reproducer

(defun bytes-per (thunk n)
  (funcall thunk)
  (let ((start (gctools:bytes-allocated)))
    (dotimes (i n) (funcall thunk))
    (/ (float (- (gctools:bytes-allocated) start)) n)))

(defgeneric egf (k))
(defmethod egf ((k (eql :a))) 1)
(defmethod egf ((k (eql :b))) 2)

(bytes-per (lambda () (egf :a)) 500)                      ; => 6784.0
(length (clos::generic-function-call-history #'egf))      ; => 0   <-- never memoized

A class-specialized gf for comparison gives 0.0 B/call and a call history of length 1.

Impact

clasp-ffi:%mem-ref / %mem-set define one eql method per foreign type (fli.lisp, generate-methods), so every CFFI foreign memory access hits this:

clasp-ffi:%mem-ref   19448 B/call, ~321 us/call, call history 0
clasp-ffi:%mem-set   20240 B/call

Cost scales with method count (~6300 B + ~288 B/method) and is identical for the first or last key, since it is not a search.

Root cause

src/lisp/kernel/clos/miss.lisp, in miss-info:

((eq (class-of generic-function)
     #.(find-class 'standard-generic-function))
 (memoize-eql-specialized generic-function method-combination
                          call-history argument-classes))

In the built image this comparison is always false, so miss-info falls through to the (t nil) clause and never calls memoize-eql-specialized. Nothing is added to the call history, so the next call misses again. Note updatedp is therefore always NIL and force-discriminator never runs — the discriminating function is not being rebuilt, the raw miss path is simply taken every time.

Replacing the read-time literal with a runtime (find-class 'standard-generic-function) fixes it completely: 0 B/call, and %mem-ref drops to 137 ns with a 27-entry call history.

Why this is filed rather than just patched: the literal is not stale

Everything checkable says the comparison should succeed:

• The module literal, the live class, and (class-of gf) are all eq to one another:
  (eq (class-of #'egf) (aref (core:bytecode-module/literals (core:bytecode-simple-fun/code #'clos::miss-info)) 9)) => T, and both are eq to (find-class 'standard-generic-function).
• The bytecode decodes correctly. Raw bytes around the test: 60 8 (called-fdefinition EQ), 60 1 (CLASS-OF), 2 0 (closure 0 = the gf), 4 1, 1 9 (const literal 9 = the class), 4 2, 23 12 (jump-if-8 +12). Offset 12 from the opcode lands exactly on the called-fdefinition of MEMOIZE-EQL-SPECIALIZED.
• Every input to the cond is correct. Wrapping clos::outcome shows, inside the compiled function's own extent: closure 0 is #'egf, (eq (class-of gf) (find-class 'standard-generic-function)) is T, ok is NIL, final-methods has length 1, call-history is NIL.
• Yet witness counters on memoize-eql-specialized, specializers-combinate and call-history-find-key all stay at zero, while a direct call to memoize-eql-specialized with those exact arguments returns 3 entries and trips all three counters.
• Redefining miss-info at runtime from identical source (with either the #. literal or a runtime find-class) works correctly.

So the source is right, the constant is right, the bytecode is right, and the inputs are right, but the branch is not taken in cross-clasp-compiled code. That looks like a compiler or VM issue rather than a CLOS one, which is why it is worth a look beyond the one-line workaround.

Second occurrence of the same pattern

src/lisp/kernel/clos/static-gfs/svuc.lisp, uncustomizable-slot-p, has the same read-time-literal guard and is dead in the same way — it returns NIL for a plain standard-class with a standard-effective-slot-definition:

(defclass plainc () ((a :initform 1)))
(let ((c (find-class 'plainc)))
  (eq (class-of c) (find-class 'standard-class)))                  ; => T
(static-gfs::uncustomizable-slot-p c slotd)                        ; => NIL, expected T

This silently disables the static slot-value / slot-boundp optimization for every standard class.

Not all such guards fail: #.(find-class 't) in update-specializer-profile (generic.lisp:284) and #.(find-class 'standard-class) in std-create-slots-table (class.lisp:307) both behave correctly, so it is not simply "all read-time class literals are broken".

[dg1sbg]

Audited the remaining #.(find-class ...) sites in the kernel. A third one is dead, and this one is a correctness bug rather than a lost optimization.

validate-superclass rejects AMOP-legal hierarchies

class.lisp:171-181 uses read-time literals to permit a standard-class and a funcallable-standard-class to appear in each other's superclass chain. Both clauses evaluate false in the built image, so the mixed case is rejected:

(defclass plain-sc () ((a :initform 1)))

(defclass fsc-from-sc (plain-sc) () (:metaclass clos:funcallable-standard-class))
;; => Class #<STANDARD-CLASS PLAIN-SC> is not a valid superclass for
;;    #<FUNCALLABLE-STANDARD-CLASS FSC-FROM-SC>

Called directly, both directions return NIL where T is expected:

(clos:validate-superclass (find-class 'standard-generic-function)  ; funcallable-standard-class
                          (find-class 'plain-sc))                  ; standard-class     => NIL
(clos:validate-superclass (find-class 'plain-sc)
                          (find-class 'standard-generic-function)) ; => NIL

Full audit

Site	Guard	Result
miss.lisp:266	(eq (class-of gf) #.(find-class 'standard-generic-function))	dead
svuc.lisp:21-23	(eq (class-of class) #.(find-class 'standard-class)) …	dead
class.lisp:174-177	(eq c1 #.(find-class 'standard-class)) …	dead
class.lisp:184	(eq superclass #.(find-class 't))	works
class.lisp:307-309	(eq metaclass #.(find-class 'standard-class)) …	works
generic.lisp:284	(eql spec #.(find-class 't))	works
method.lisp:117-121	(class-prototype #.(find-class 'standard-generic-function))	works

What the split is not

Several plausible explanations are ruled out by the table:

• Not per-class. #.(find-class 'standard-class) is dead at svuc.lisp:21 and at class.lisp:176, but works at class.lisp:307.
• Not per-file. Within class.lisp, line 174-177 is dead while 184 and 307 work.
• Not defun vs defmethod. Both groups contain both.
• Not test-vs-value. The working method.lisp case uses the literal as a value, but the working class.lisp:307 and generic.lisp:284 cases are eq/eql tests.
• Not class-of vs si::instance-class. The two dead class-of sites and the working instance-class site suggested this, but (eq (class-of c) (si::instance-class c)) is T for the classes involved.

The only consistent observation remains the one in the issue body: recompiling the identical source at runtime produces working code, so the divergence is introduced at build time. Note the built-in class T works in both places it appears, while every dead site names a CLOS metaobject class — but class.lisp:307 is a counterexample to that being sufficient on its own.

PR #1812 fixes the first two by switching to a runtime find-class. The validate-superclass case is left alone there pending a decision on whether to paper over each site individually or fix the underlying cause.

[dg1sbg]

Root cause found

It is not specific to #. or to CLOS at all. (eq X <constant>) is folded to NIL whenever X has a declared CLOS class type and the constant is an instance of that type.

The #.(find-class ...) guards were just the place it happened to bite: class-of is proclaimed (ftype (function (t) class) class-of) (cleavir/proclamations.lisp:269), so (eq (class-of x) #.<a class object>) has a class-typed operand and a constant-instance operand — exactly the broken shape.

Minimal reproducer

Compiled into the kernel (cross-clasp / native), then called:

(defun probe-t   (x) (declare (type t x))              (eq x #.(find-class 'clos:funcallable-standard-class)))
(defun probe-cls (x) (declare (type class x))          (eq x #.(find-class 'clos:funcallable-standard-class)))
(defun probe-sc  (x) (declare (type standard-class x)) (eq x #.(find-class 'standard-class)))
(defun probe-cof ()  (eq (class-of (find-class 'standard-generic-function))
                         #.(find-class 'clos:funcallable-standard-class)))

form	result	expected
(probe-t (find-class 'clos:funcallable-standard-class))	T	T
(probe-cls (find-class 'clos:funcallable-standard-class))	NIL	T
(probe-sc (find-class 'standard-class))	NIL	T
(probe-cof)	NIL	T

Declaring the argument t works. Declaring it class or standard-class — a strictly more precise, and correct, declaration — silently makes the comparison constant-NIL.

Where

src/lisp/kernel/cleavir/transform.lisp:406:

;;; If the types of the arguments are disjoint, EQ (and EQL) are false.
(deftransform eq (((x t) (y t)) :argstype args)
  (with-transformer-types (x y) args
    (if (ctype:disjointp x y *clasp-system*)
        'nil
        (decline-transform "types not disjoint"))))

and Ctype/other-functions.lisp:63:

(defun disjointp (ct1 ct2 system)
  (bottom-p (conjoin/2 ct1 ct2 system) system))

So conjoining a CLOS class ctype with a member/eql ctype whose element is a CLOS instance yields bottom, and disjointp reports disjoint. That is unsound: #<FUNCALLABLE-STANDARD-CLASS ...> is of type class, so the conjunction is the member type, not bottom. disjointp's docstring allows false negatives ("may return false even if the ctypes are disjoint"), but a false positive silently miscompiles code.

Why it looked mysterious

• Only the native/cross-clasp path is affected; the bytecode compiler does not run these transforms, so redefining the identical source at runtime always worked.
• The literal really is eq to the live class — the constant is fine. Nothing is stale. The comparison is discarded before it ever runs.
• Every dead guard routed through class-of; every working one did not (class.lisp:307 uses si::instance-class, which has no such proclamation, and works with the same literal in the same file).

Scope

Wider than the three guards in #1812. Any code of the form (eq x <literal instance>) where x is declared to a CLOS class type compiles to NIL — including user code. eql has the same transform immediately below and presumably the same exposure.

I have not attempted a fix: the defect is in ctype conjunction/bottom-p for member types over CLOS instances, and guessing at Cleavir's type-inference invariants seemed more likely to cause harm than the documented bug. #1812 remains a correct workaround for the three kernel sites regardless of how this is resolved.