RuntimeError: dictionary changed size during iteration

[mastak]

Hi there!

I'm using aioprometheus with aiohttp.

Traceback (most recent call last):
  File "/app/utils_web.py", line 131, in _call_handler
    return await handler(request)
  File "/app/service.py", line 250, in handle_metrics
    content, http_headers = render(REGISTRY, request.headers.getall(ACCEPT, []))
  File "/usr/local/lib/python3.8/dist-packages/aioprometheus/renderer.py", line 37, in render
    content = formatter.marshall(registry)
  File "/usr/local/lib/python3.8/dist-packages/aioprometheus/formats/text.py", line 246, in marshall
    blocks.append(self.marshall_collector(i))
  File "/usr/local/lib/python3.8/dist-packages/aioprometheus/formats/text.py", line 237, in marshall_collector
    result = self.marshall_lines(collector)
  File "/usr/local/lib/python3.8/dist-packages/aioprometheus/formats/text.py", line 226, in marshall_lines
    for i in collector.get_all():
  File "/usr/local/lib/python3.8/dist-packages/aioprometheus/collectors.py", line 165, in get_all
    for k in self.values:
RuntimeError: dictionary changed size during iteration

[claws]

Are you able to provide any additional information about how this might be replicated to help investigate it?

[huwcbjones]

The underlying issue is an inherent race condition (iterating over a Collector's value whilst adding/removing elements to the collector).
The script below eventually explodes on my machine after a while with this stacktrace

Exception in thread Thread-1 (get_all):
Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "test.py", line 10, in get_all
    for i in g.get_all():
             ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aioprometheus/collectors.py", line 165, in get_all
    for k in self.values:
RuntimeError: dictionary changed size during iteration

from threading import Thread

from aioprometheus import Gauge

g = Gauge(name="g", doc="")


def get_all():
    while True:
        for i in g.get_all():
            print(i)


t = Thread(daemon=True, target=get_all)
t.start()

for i in range(100000):
    g.set({"l" * i: str(i)}, i)

[huwcbjones]

Fix with test available in #100