Pickling of function pointer

[NanoNabla]

When python has to pickle functions it will raise an exception that pickling failed.

Using Python 3.10, Score-P 8.0, Bindings 4.2.0

Minimal example

import multiprocessing as mp

def hello(x):
    print("hello_world {}".format(x))

if __name__ == "__main__":
    nums = [i for i in range(4)]
    pool = mp.Pool()
    pool.map(hello, nums)

results in

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/tmp/venv/lib/python3.10/site-packages/scorep/__main__.py", line 142, in <module>
    scorep_main()
  File "/tmp/venv/lib/python3.10/site-packages/scorep/__main__.py", line 119, in scorep_main
    tracer.run(code, globs, globs)
  File "/tmp/venv/lib/python3.10/site-packages/scorep/_instrumenters/scorep_instrumenter.py", line 55, in run
    exec(cmd, globals, locals)
  File "test.py", line 9, in <module>
    pool.map(hello, nums)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 540, in _handle_tasks
    put(task)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function hello at 0x7feaa75b3490>: attribute lookup hello on __main__ failed

although running the code without -m scorep works.

Another code reported by @maximilian-tech can be found in #106

[NanoNabla]

A more minimal example

import pickle                                                                                                                                                                                                                                 

def a(x):
    print(x)
def b(x, y): 
      x(y) 
def main():
    b(a,3)
    pickle.dumps(a) # without this pickle call everything works fine

if __name__ == '__main__':
    main()

[NanoNabla]

The problem seems to be caused by some design decision made by python and a well known issue e.g. with using multiprocessing in interactive sessions.

Functionality within this package requires that the main module be importable by the children. [...] This means that some examples, such as the multiprocessing.pool.Pool examples will not work in the interactive interpreter. For example: [from the multiprocessing docu]

As a workaround until I find a proper solution I recommend as a workaround to put the code into a separate file than the one to call the python interpreter on

e.g.

# foo.py

import pickle                                                                                                                                                                                                                                 

def a(x):
    print(x)
def b(x, y): 
      x(y) 
def main():
    b(a,3)
    pickle.dumps(a) # without this pickle call everything works fine

# bar.py

import foo

if __name__ == '__main__':
    foo.main()

python -m scorep bar.py