Feature: SPSC Lock-Free Log Engine (Phases 1, 2, 4)

[doomedraven]

Overhauls the BSON logging engine loop entirely for wait-free throughput. Replaces g_mutex across all loq calls. Routes encoding bytes to a 512KB thread-local SPSC ring array inside thread_log_context_t. _send_log opportunistically drains all rings using contiguous chunk writes. Relies on the TEB cache hooks established in PR #203.

[doomedraven]

Depends on the #164 and #203

Architectural Execution:

• Phase 1 (Atomic Identifiers): Replaced g_mutex explanation blocking with InterlockedCompareExchange on an array of LONG.
• Phase 2 (SPSC Rings & Thread-Locals):
• Purged all g_mutex lock loops inside loq.
• Eradicated static lastlog_t lastlog completely. Duplicate tracking is now memory-mapped strictly inside the TEB pointers ctx->last_buf and ctx->last_api_logged.
• Re-routed log_raw_direct to bypass g_buffer appending entirely. Serialization dumps straight into your thread's pre-allocated 512KB Ring using monotonic write_idx tracking.
• Phase 4 (Opportunistic Drain): Hard-refactored _send_log(). Only the flushing pipe-writer uses TryEnterCriticalSection(&g_writing_log_buffer_mutex). We iterate through pItem = (entry_t *)g_log_contexts.root;, chunk out the length deltas via read_idx, and commit WriteFile flushes concurrently.

[doomedraven]

Pushed a round of fixes addressing correctness problems in the ring implementation. Summary of what changed and why.

Compile / link

• loq() referenced a ctx local that was never declared.
• The g_mutex definition had been removed from log.c while capemon.c still externed and initialised it. Both sites removed; no references remain.

Ring memory safety

• The wrap marker could be 1-3 bytes, so the drainer's 4-byte length read could run past the end of the buffer. Records are now padded to a 4-byte boundary, which keeps write_idx aligned and guarantees a full marker always fits.
• Wrap padding was not charged against free space, so a write could overrun read_idx and corrupt records the drainer had not consumed yet. It is now accounted for before a record is admitted.
• Partial WriteFile results were ignored. The old global-buffer path handled short writes with a memmove; the ring now loops in write_all(). Without this a short write truncates a BSON document and desynchronises the stream permanently.
• Records too large to ever fit the ring drain the thread's ring and then go straight to the pipe, preserving that thread's ordering, instead of being dropped.

Record ordering

This was the significant one. Per-thread rings broke two ordering guarantees the result server depends on, and neither was fixable by tightening the ring itself.

1. Netlog header. announce_netlog() wrote the BSON <pid> header through a per-thread ring, making it just another record. The drain walks threads in lookup order, so another thread's ring could reach the pipe first and corrupt the stream from byte zero. The header now bypasses the rings and goes straight to the pipe.

2. Explain frames. The per-index explain frame was gated by a global atomic. The thread that won the CAS parked explain(id) in its own ring while every other thread proceeded immediately and emitted record(id) into a different ring — which could drain first, leaving the parser with an unknown index.

   The gate is now per-thread, so explain(id) and the record that needs it always land in the same ring and drain in FIFO order. Cost is a duplicate explain frame per thread per id; the parser treats that as an idempotent map update. This also removes the InterlockedCompareExchange, which was the thing creating the split in the first place.

Record loss

• log_flush() could silently skip the drain via TryEnterCriticalSection. It now blocks; only the periodic drain from the logging thread stays best-effort.
• Rings were unlinked on thread exit without being drained. lookup_del() only unlinks, so anything still queued became unreachable for the rest of the process. Teardown now flushes and drains before unlinking.
• Dropped records were counted into a field nothing read. Overflow is now reported on the command pipe with the owning thread ID, rearmed only when the count grows so a saturated ring cannot flood.

Cleanup

Removed the unreferenced exit: label, an empty block, and a stale comment describing the old cross-thread race.

---

Still open

Two things I have deliberately not addressed here, both worth resolving before this comes out of draft:

• Per-thread ring memory. Reduced 512KB to 256KB, but it is still allocated per thread and never freed, since lookup_del() leaks by design. A thread-spamming sample grows memory without bound. Bounding it properly needs either a free list of retired contexts or a cap on tracked threads.

• TEB slot choice (inherited from the context-caching change, not this one). NtTib.ArbitraryUserPointer is not a free slot — ntdll writes it during DLL and path-name resolution, e.g. LdrLoadDll stores the module name there. Clobbering it from a hook running underneath loader activity is a risk that should be either verified empirically under loader-heavy samples or avoided by moving to a TLS slot index reserved at load time.

Both x86 and x64 configurations were syntax-checked after the change.