Reworks GIL related types to prevent some forms of program termination.

This change is a rework of some GIL related types in the library to prevent and avoid some unexpected crashes and program terminations, which would occur most often at process exit:

- A new exception type named `InterpreterFinalizingException` is introduced. It is thrown by some functions in the library when an operation fails because the interpreter is currently finalizing.
- `GILAcquire` constructor now throws a `InterpreterFinalizingException` when the interpreter is finalizing and therefore the GIL could not be acquired.
- `GILRelease` destructor now does not reacquire the GIL when the interpreter is finalizing.
- The `Guarded` and `GILGuardedObject` types are removed, and replaced with a `SharedObject` type. This type now wraps a Python object value as a shared reference-counted value that does not require the GIL to copy or move. The last copy of a `SharedObject` value will acquire the GIL to release the object, if possible, or do nothing (and leak it) if the GIL cannot be acquired.
- The `DeleteInOtherThread` deleter type is removed. Its implementation was incorrect and it has become unneeded.
- Destructors of `Application` and `ApplicationSession` do not use the `DeleterInOtherThread` deleter type anymore. The hypothesis is that the `DeleteInOtherThread` deleter was previously needed because `Application` values were wrapped in shared pointers and could be destroyed in callbacks executed by native threads. This is not the case anymore and instead, they destroy the application in the same thread that the Python `Application` object is garbage collected, which most likely is the interpreter main thread (due to `Application` being a singleton in the Python code, and its destruction is scheduled in an `atexit` callback).
- A new `invokeCatchPythonError` is introduced. It acquires the GIL, then invokes a function with the given arguments, but catches any `pybind11::error_already_set` exception thrown during this invocation, and throws a `std::runtime_error` instead.
- The `currentThreadHoldsGil` function now returns an optional denoting if the interpreter is initialized and not finalizing. Another function `gilExistsAndCurrentThreadHoldsIt` is introduced that returns just a bool.
- The `ObjectDecRef` internal type is removed, as it is unused.

This change mitigates the problem observed in issue #SW-4658.

Author: nyibbang

CMakeLists.txt

@@ -115,6 +115,7 @@ target_sources(
 
   PUBLIC
     qipython/common.hpp
+    qipython/common.hxx
     qipython/pyapplication.hpp
     qipython/pyasync.hpp
     qipython/pyclock.hpp

qipython/common.hpp

@@ -125,6 +125,28 @@ inline boost::optional<bool> interpreterIsFinalizing()
 #endif
 }
 
+/// Acquires the GIL, then invokes a function with the given arguments, but
+/// catches any `pybind11::error_already_set` exception thrown during this
+/// invocation, and throws a `std::runtime_error` instead. This is useful to
+/// avoid the undefined behavior caused by the `what` member function of
+/// `pybind11::error_already_set` when the GIL is not acquired.
+template<typename F, typename... Args>
+auto invokeCatchPythonError(F&& f, Args&&... args);
+
+/// Exception type thrown when an operation fails because the interpreter is
+/// finalizing.
+///
+/// Some operations are forbidden during interpreter finalization as they may
+/// terminate the thread they are called from. This exception allows stopping
+/// the flow of execution in this case before such functions are called.
+struct InterpreterFinalizingException : std::exception
+{
+  inline const char* what() const noexcept override
+  {
+    return "the interpreter is finalizing";
+  }
+};
+
 } // namespace py
 } // namespace qi
 
@@ -203,4 +225,6 @@ namespace detail
 
 PYBIND11_DECLARE_HOLDER_TYPE(T, boost::shared_ptr<T>);
 
+#include <qipython/common.hxx>
+
 #endif // QIPYTHON_COMMON_HPP

qipython/common.hxx

@@ -0,0 +1,37 @@
+/*
+**  Copyright (C) 2023 Aldebaran Robotics
+**  See COPYING for the license
+*/
+
+#pragma once
+
+#ifndef QIPYTHON_COMMON_HXX
+#define QIPYTHON_COMMON_HXX
+
+#include <pybind11/pybind11.h>
+#include <qipython/common.hpp>
+#include <qipython/pyguard.hpp>
+
+namespace qi
+{
+namespace py
+{
+
+template<typename F, typename... Args>
+auto invokeCatchPythonError(F&& f, Args&&... args)
+{
+  GILAcquire acquire;
+  try
+  {
+    return std::invoke(std::forward<F>(f), std::forward<Args>(args)...);
+  }
+  catch (const pybind11::error_already_set& err)
+  {
+    throw std::runtime_error(err.what());
+  }
+}
+
+} // namespace py
+} // namespace qi
+
+#endif // QIPYTHON_COMMON_HXX