[Bug]: FsEventWatcher deadlocks when initializing

[pkolaczk]

Prerequisites

• I searched existing issues to avoid duplicates

Component

notify (core library)

Operating System

macOS

Notify Version

8.2.0

Notify Backend

fsevents (macOS)

Bug Report

What happened?

I'm using notify to notify me about configuration file changes. I have a bunch of unit tests, where each of the tests updates the config file in a different way (e.g. by overwriting, by moving, by relinking to a different file etc). When I run each of those tests separately one at a time, everything works fine. When I run them all at once, they block forever. It seems that creating the second watcher blocks in the watch function:

        let mut watcher = notify::recommended_watcher(handler)?;
        for p in paths_to_watch.as_ref() {
            if p.exists() {
                watcher.watch(p.as_path(), RecursiveMode::NonRecursive)?;  // <--- blocked forever
            }
        }

This does not happen when the watcher is run for the first time.

I ran the code under lldb and found that one of the fsevent runloops is stuck in fsevent.rs:474:

fs::FSEventStreamStart(stream);

so it never reaches this

 // the calling to CFRunLoopRun will be terminated by CFRunLoopStop call in drop()
      rl_tx
            .send(CFSendWrapper(cur_runloop))
            .expect("Unable to send runloop to watcher");

and never unblocks the thread waiting in line 495.

See attached stacktraces for more details.

Steps to reproduce

Unfortunately this is a closed source program, cannot share full source.

Environment details

macOS Sequoia 15.7.7
Mac M2 Pro
Rustc

Error messages or logs

I grabbed the following stacktraces (I omitted the irrelevant stuff) that should explain pretty well what's happening:

thread #2, name = 'config_updater::test::reload_watcher'
    frame #0: 0x0000000190cf9bb0 libsystem_kernel.dylib`semaphore_wait_trap + 8
    frame #1: 0x0000000190b85960 libdispatch.dylib`_dispatch_sema4_wait + 28
    frame #2: 0x0000000190b85f10 libdispatch.dylib`_dispatch_semaphore_wait_slow + 132
    frame #3: 0x0000000100844c3c shadow_proxy-1d32326ccc00f96b`<std::sys::sync::thread_parking::darwin::Parker>::park at darwin.rs:74:15 [opt] [inlined]
    frame #4: 0x0000000100844c1c shadow_proxy-1d32326ccc00f96b`<std::thread::thread::Thread>::park at thread.rs:121:47 [opt]
    frame #5: 0x0000000100719e08 shadow_proxy-1d32326ccc00f96b`std::sync::mpmc::context::Context::wait_until::h1cb9bfd4016a4cce(self=0x000000017103fb10, deadline=Option<std::time::Instant> @ 0x000000017103f970) at context.rs:143:44
    frame #6: 0x00000001007183fc shadow_proxy-1d32326ccc00f96b`std::sync::mpmc::list::Channel$LT$T$GT$::recv::_$u7b$$u7b$closure$u7d$$u7d$::hf417f51c79dfadfc(cx=0x000000017103fb10) at list.rs:453:39
    frame #7: 0x000000010071a6b4 shadow_proxy-1d32326ccc00f96b`std::sync::mpmc::context::Context::with::_$u7b$$u7b$closure$u7d$$u7d$::h05758b28567229b8(cx=0x000000017103fb10) at context.rs:49:13
    frame #8: 0x000000010071a654 shadow_proxy-1d32326ccc00f96b`std::sync::mpmc::context::Context::with::_$u7b$$u7b$closure$u7d$$u7d$::h12337541634910d3(cell=0x0000000153004af0) at context.rs:57:31
    frame #9: 0x000000010072a1bc shadow_proxy-1d32326ccc00f96b`std::thread::local::LocalKey$LT$T$GT$::try_with::hcc235304a2099934(self=0x0000000100b6a428, f={closure_env#1}<std::sync::mpmc::list::{impl#3}::recv::{closure_env#1}<notify::fsevent::{impl#5}::run::CFSendWrapper>, ()> @ 0x000000017103fcd0) at local.rs:462:12
    frame #10: 0x0000000100719f50 shadow_proxy-1d32326ccc00f96b`std::sync::mpmc::context::Context::with::h0ef30676b9699a58(f=<unavailable>) at context.rs:53:14
    frame #11: 0x0000000100718284 shadow_proxy-1d32326ccc00f96b`std::sync::mpmc::list::Channel$LT$T$GT$::recv::ha12533528e7eba28(self=0x000000010190c000, deadline=Option<std::time::Instant> @ 0x000000017103fdb0) at list.rs:442:13
    frame #12: 0x000000010071fca4 shadow_proxy-1d32326ccc00f96b`std::sync::mpmc::Receiver$LT$T$GT$::recv::h2b4b94c01e04a348(self=0x000000017103fff0) at mod.rs:1012:48
    frame #13: 0x000000010072f570 shadow_proxy-1d32326ccc00f96b`std::sync::mpsc::Receiver$LT$T$GT$::recv::h4c39461d0af09b0b(self=0x000000017103fff0) at mpsc.rs:872:20
    frame #14: 0x000000010072c94c shadow_proxy-1d32326ccc00f96b`notify::fsevent::FsEventWatcher::run::h182ccb53a52e6b79(self=0x00000001710405a0) at fsevent.rs:495:36
    frame #15: 0x000000010072c4c0 shadow_proxy-1d32326ccc00f96b`notify::fsevent::FsEventWatcher::watch_inner::hff9e5881f78c70d3(self=0x00000001710405a0, path=&std::path::Path @ 0x0000000171040308, recursive_mode=NonRecursive) at fsevent.rs:312:22
    frame #16: 0x000000010072b6ec shadow_proxy-1d32326ccc00f96b`_$LT$notify..fsevent..FsEventWatcher$u20$as$u20$notify..Watcher$GT$::watch::haf0d3a9989377638(self=0x00000001710405a0, path=&std::path::Path @ 0x0000000171040348, recursive_mode=NonRecursive) at fsevent.rs:586:14
    frame #17: 0x00000001003b933c shadow_proxy-1d32326ccc00f96b`shadow_proxy::config_updater::FileWatcher::try_new_watcher::h9db2bccc11830d3a(path=&std::path::Path @ 0x0000000171040b80, sender=Sender<notify_types::event::Event> @ 0x00000001710404b8) at config_updater.rs:211:25

thread #6, name = 'notify-rs fsevents loop'
    frame #0: 0x0000000190cf9c34 libsystem_kernel.dylib`mach_msg2_trap + 8
    frame #1: 0x0000000190d0c338 libsystem_kernel.dylib`mach_msg2_internal + 76
    frame #2: 0x0000000190d02764 libsystem_kernel.dylib`mach_msg_overwrite + 484
    frame #3: 0x0000000190cf9fa8 libsystem_kernel.dylib`mach_msg + 24
    frame #4: 0x0000000190e26c0c CoreFoundation`__CFRunLoopServiceMachPort + 160
    frame #5: 0x0000000190e25528 CoreFoundation`__CFRunLoopRun + 1208
    frame #6: 0x0000000190e249e8 CoreFoundation`CFRunLoopRunSpecific + 572
    frame #7: 0x0000000190e9e4a4 CoreFoundation`CFRunLoopRun + 64
    frame #8: 0x000000010072cc08 shadow_proxy-1d32326ccc00f96b`notify::fsevent::FsEventWatcher::run::_$u7b$$u7b$closure$u7d$$u7d$::h46b2c4c53bb6820f at fsevent.rs:481:21
    frame #9: 0x00000001007218b4 shadow_proxy-1d32326ccc00f96b`std::sys::backtrace::__rust_begin_short_backtrace::h63d1c519c0938ddf(f=<unavailable>) at backtrace.rs:166:18
    frame #10: 0x0000000100730b98 shadow_proxy-1d32326ccc00f96b`std::thread::lifecycle::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::ha14ce9d14e99efcb at lifecycle.rs:91:13
    frame #11: 0x000000010072af78 shadow_proxy-1d32326ccc00f96b`_$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h2758ce0839fe56d2(self=<unavailable>, (null)=<unavailable>) at unwind_safe.rs:275:9
    frame #12: 0x0000000100730ed0 shadow_proxy-1d32326ccc00f96b`std::panicking::catch_unwind::do_call::he5e1d41897388b64(data="\U00000001") at panicking.rs:581:40
    frame #13: 0x0000000100731a2c shadow_proxy-1d32326ccc00f96b`__rust_try + 32
    frame #14: 0x0000000100730a10 shadow_proxy-1d32326ccc00f96b`std::panicking::catch_unwind::h35cd268af39f95c4(f=AssertUnwindSafe<std::thread::lifecycle::spawn_unchecked::{closure#1}::{closure_env#0}<notify::fsevent::{impl#5}::run::{closure_env#0}, ()>> @ 0x000000017006ed18) at panicking.rs:544:19
    frame #15: 0x00000001007309c4 shadow_proxy-1d32326ccc00f96b`std::panic::catch_unwind::hd8eab32641d9251b(f=AssertUnwindSafe<std::thread::lifecycle::spawn_unchecked::{closure#1}::{closure_env#0}<notify::fsevent::{impl#5}::run::{closure_env#0}, ()>> @ 0x000000017006ed18) at panic.rs:359:14 [inlined]
    frame #16: 0x00000001007309c4 shadow_proxy-1d32326ccc00f96b`std::thread::lifecycle::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::heaa9d7ab3c917db0 at lifecycle.rs:89:26
    frame #17: 0x00000001007239c0 shadow_proxy-1d32326ccc00f96b`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h742d5b436a3b7129((null)=0x00000001023f1880, (null)=<unavailable>) at function.rs:250:5
    frame #18: 0x0000000100855688 shadow_proxy-1d32326ccc00f96b`<alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output = ()> + core::marker::Send> as core::ops::function::FnOnce<()>>::call_once at boxed.rs:2249:9 [opt] [inlined]
    frame #19: 0x000000010085567c shadow_proxy-1d32326ccc00f96b`<std::sys::thread::unix::Thread>::new::thread_start at unix.rs:118:17 [opt]
    frame #20: 0x0000000190d3bbc8 libsystem_pthread.dylib`_pthread_start + 136

thread #7, name = 'notify-rs fsevents loop'
    frame #0: 0x0000000190cf9c34 libsystem_kernel.dylib`mach_msg2_trap + 8
    frame #1: 0x0000000190d0c338 libsystem_kernel.dylib`mach_msg2_internal + 76
    frame #2: 0x0000000190d02764 libsystem_kernel.dylib`mach_msg_overwrite + 484
    frame #3: 0x0000000190cf9fa8 libsystem_kernel.dylib`mach_msg + 24
    frame #4: 0x000000019a56d8d0 FSEvents`f2d_register_rpc + 284
    frame #5: 0x000000019a56d6b8 FSEvents`register_with_server + 840
    frame #6: 0x000000019a56d284 FSEvents`FSEventStreamStart + 244
    frame #7: 0x000000010072cb9c shadow_proxy-1d32326ccc00f96b`notify::fsevent::FsEventWatcher::run::_$u7b$$u7b$closure$u7d$$u7d$::h46b2c4c53bb6820f at fsevent.rs:474:21
    frame #8: 0x00000001007218b4 shadow_proxy-1d32326ccc00f96b`std::sys::backtrace::__rust_begin_short_backtrace::h63d1c519c0938ddf(f=<unavailable>) at backtrace.rs:166:18
    frame #9: 0x0000000100730b98 shadow_proxy-1d32326ccc00f96b`std::thread::lifecycle::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::ha14ce9d14e99efcb at lifecycle.rs:91:13
    frame #10: 0x000000010072af78 shadow_proxy-1d32326ccc00f96b`_$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h2758ce0839fe56d2(self=<unavailable>, (null)=<unavailable>) at unwind_safe.rs:275:9
    frame #11: 0x0000000100730ed0 shadow_proxy-1d32326ccc00f96b`std::panicking::catch_unwind::do_call::he5e1d41897388b64(data="\U00000001") at panicking.rs:581:40
    frame #12: 0x0000000100731a2c shadow_proxy-1d32326ccc00f96b`__rust_try + 32
    frame #13: 0x0000000100730a10 shadow_proxy-1d32326ccc00f96b`std::panicking::catch_unwind::h35cd268af39f95c4(f=AssertUnwindSafe<std::thread::lifecycle::spawn_unchecked::{closure#1}::{closure_env#0}<notify::fsevent::{impl#5}::run::{closure_env#0}, ()>> @ 0x000000017027ad18) at panicking.rs:544:19
    frame #14: 0x00000001007309c4 shadow_proxy-1d32326ccc00f96b`std::panic::catch_unwind::hd8eab32641d9251b(f=AssertUnwindSafe<std::thread::lifecycle::spawn_unchecked::{closure#1}::{closure_env#0}<notify::fsevent::{impl#5}::run::{closure_env#0}, ()>> @ 0x000000017027ad18) at panic.rs:359:14 [inlined]
    frame #15: 0x00000001007309c4 shadow_proxy-1d32326ccc00f96b`std::thread::lifecycle::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::heaa9d7ab3c917db0 at lifecycle.rs:89:26
    frame #16: 0x00000001007239c0 shadow_proxy-1d32326ccc00f96b`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h742d5b436a3b7129((null)=0x00000001023f1b80, (null)=<unavailable>) at function.rs:250:5
    frame #17: 0x0000000100855688 shadow_proxy-1d32326ccc00f96b`<alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output = ()> + core::marker::Send> as core::ops::function::FnOnce<()>>::call_once at boxed.rs:2249:9 [opt] [inlined]
    frame #18: 0x000000010085567c shadow_proxy-1d32326ccc00f96b`<std::sys::thread::unix::Thread>::new::thread_start at unix.rs:118:17 [opt]
    frame #19: 0x0000000190d3bbc8 libsystem_pthread.dylib`_pthread_start + 136