monotouch-test: random CoreCLR GC crash (maybe in MonoTouchFixtures.CoreAnimation.LayerTest?)

[rolfbjarne]

Thread stack for the crashing thread:

Thread 43 Crashed:
0   ???                           	               0x0 ???
1   libxamarin-dotnet-coreclr-debug.dylib	       0x100ac2c24 xamarin_coreclr_reference_tracking_is_referenced_callback + 116
2   libcoreclr.dylib              	       0x101777d18 ObjCMarshalNative::IsTrackedReference(Object*, bool*) + 96
3   libcoreclr.dylib              	       0x1016dd7ec GCToEEInterface::RefCountedHandleCallbacks(Object*) + 76
4   libcoreclr.dylib              	       0x1017904d4 PromoteRefCounted(Object**, unsigned long*, unsigned long, unsigned long) + 96
5   libcoreclr.dylib              	       0x10178e5fc BlockScanBlocksWithoutUserData(TableSegment*, unsigned int, unsigned int, ScanCallbackInfo*) + 116
6   libcoreclr.dylib              	       0x10178ff1c TableScanHandles(HandleTable*, unsigned int const*, unsigned int, TableSegment* (*)(HandleTable*, TableSegment*, Holder<CrstBase*, &CrstBase::AcquireLock(CrstBase*), &CrstBase::ReleaseLock(CrstBase*), 0ul, &int CompareDefault<CrstBase*>(CrstBase*, CrstBase*), true>*), void (*)(TableSegment*, unsigned int, unsigned int, ScanCallbackInfo*), ScanCallbackInfo*, Holder<CrstBase*, &CrstBase::AcquireLock(CrstBase*), &CrstBase::ReleaseLock(CrstBase*), 0ul, &int CompareDefault<CrstBase*>(CrstBase*, CrstBase*), true>*) + 392
7   libcoreclr.dylib              	       0x10178b014 HndScanHandlesForGC(HandleTable*, void (*)(Object**, unsigned long*, unsigned long, unsigned long), unsigned long, unsigned long, unsigned int const*, unsigned int, unsigned int, unsigned int, unsigned int) + 372
8   libcoreclr.dylib              	       0x101791b74 Ref_TraceNormalRoots(unsigned int, unsigned int, ScanContext*, void (*)(Object**, ScanContext*, unsigned int)) + 600
9   libcoreclr.dylib              	       0x1017f82a0 WKS::gc_heap::mark_phase(int) + 1124
10  libcoreclr.dylib              	       0x101810c28 WKS::gc_heap::gc1() + 840
11  libcoreclr.dylib              	       0x1017eb92c WKS::gc_heap::garbage_collect(int) + 1804
12  libcoreclr.dylib              	       0x1017eaee0 WKS::GCHeap::GarbageCollectGeneration(unsigned int, gc_reason) + 980
13  libcoreclr.dylib              	       0x1017ea980 WKS::GCHeap::GarbageCollect(int, bool, int) + 444
14  libcoreclr.dylib              	       0x1016bcbc0 GCInterface_Collect + 184
15  monotouchtest                 	       0x1060f0494 S_P_CoreLib_System_GC__Collect_0 + 116

Full crash report: https://gist.github.com/rolfbjarne/0909e427b8eb228da56bd88dcf6342af

This is what the test run was doing:

[...]
MonoTouchFixtures.CoreAnimation.LayerTest
	[PASS] AddAnimation
	[PASS] CAActionTest
	[PASS] ConvertPoint
	[PASS] ConvertRect
	[PASS] ConvertTime
	[PASS] Mask

Full output log: https://gist.github.com/rolfbjarne/741fda3807e021b9a5ff7b709a4c6537

[rolfbjarne]

From #25308 (comment):

stderr-20260505_054656.log

ExcUserFault_monotouchtest-2026-05-05-054711-20260505_054757.ips.zip

[rolfbjarne]

Created PR #25330 with debug logging in LayerTest to help diagnose this crash.

The logging covers:

• Entry/exit of every test method with thread IDs
• All explicit GC.Collect() calls (before and after)
• Background thread operations in TestBug26532 and TestCALayerDelegateDispose

The crash is in xamarin_coreclr_reference_tracking_is_referenced_callback where info->data appears to be NULL during GC. The most suspicious tests are TestBug26532 (creates 50 CALayer sublayers on a background thread + multiple GC.Collect calls) and TestCALayerDelegateDispose (creates/disposes layers across threads + GC.Collect).

Will monitor the PR runs for crash reproduction.

[rolfbjarne]

Run 2 on PR #25330 (build 14017041): monotouch-test/iOS - simulator/Release (trimmable static registrar, NativeAOT, ARM64) crashed. Retriggering for more data.

[rolfbjarne]

Root Cause Analysis

The crash has been definitively identified as a race condition in xamarin_release_managed_ref in runtime/runtime.m.

The Race

In xamarin_release_managed_ref, the framework_peer_waypoint (which prevents objc_release from racing with the GC toggle-ref callbacks) was only called for non-user types. For user types (like TextCALayer in the failing test), objc_release was called without any synchronization against the GC.

The race manifests as follows:

1. Thread A (main thread, in NSObject_Disposer.Drain): calls ReleaseManagedRef → clears HasManagedRef flag → calls xamarin_release_managed_ref → objc_release without waiting for the GC to finish
2. Thread B (background thread): calls GC.Collect() → GC locks framework_peer_release_lock → calls is_referenced_callback → reads handle → calls [handle retainCount]

If Thread A's objc_release deallocates the native object while Thread B is between reading the handle and calling retainCount, Thread B crashes with SIGSEGV.

This is particularly likely on ARM64 where memory ordering is weak — even though HasManagedRef = false is written before the release, the GC thread may still read the stale HasManagedRef = true value and proceed to access the handle.

The Fix

Move the xamarin_framework_peer_waypoint_safe() call outside the if (user_type) / else block so it applies to both user types and non-user types. This ensures objc_release never races with the GC's reference tracking callbacks.

The fix is in PR #25330 (commit c53e459).

Crash Confirmation

The crash was reproduced in CI run 2 of PR #25330. The crash report (monotouchtest-2026-05-05-101829.ips) confirmed:

• Crash in objc_msgSend called from xamarin_gc_toggleref_callback (runtime.m:733)
• Called from xamarin_coreclr_reference_tracking_is_referenced_callback (coreclr-bridge.m:306)
• During GC.Collect() on TestBug26532's background thread
• Main thread was in performSelectorOnMainThread path (consistent with NSObject_Disposer.Drain)
• NativeAOT ARM64 build (weak memory ordering)

Why This Test Triggers It

TestBug26532 creates TextCALayer (user type) objects on a background thread and calls GC.Collect(). If objects from a previous test run are still pending finalization, NSObject_Disposer.Drain runs on the main thread concurrently with the GC, creating the exact race condition described above.