docs: add retry loop to version check in K8s tutorial charm

[dwilding]

This part of the K8s tutorial charm has a potential race:

container.replan()
version = fastapi_demo.get_version(port=8000)

In practice we're OK because our workload starts serving with Pebble's 1 second early-exit check. But for a production workload, it would be better to wrap get_version() in a retry loop.

I observed the race in an integration test run, so we ought to make the charm code more robust.

This PR:

• Updates the charm code to include a retry loop (in all five charms)
• In "Create a minimal Kubernetes charm":
  • Adds an explanation of why the loop is needed.
  • Adds an instruction to import time and urllib.error.
  • Moves the "Add logger functionality" section earlier so that we've introduced logger.

Preview doc

[dwilding]

I'm changing this part so that talking about replan() later makes more sense.

[dwilding]

Moving this back into draft status. An integration test run hit the race condition, so we ought to update the charm code. I'll propose an update.

[dwilding]

I'm removing the last part because we actually do need additional imports now.

[dwilding]

@tromai I updated the charm code and tutorial to include a retry loop - see the updated PR description for details. Would you mind reviewing again? Thanks!

I also thought how we might test what the charm does when the race occurs. I think we'd have to change mock_version in the unit tests so that it fails a certain number of times, or until sufficient time has passed. It seems fiddly for the tutorial, so I didn't include it. What do you think?

[tromai]

Thanks for the PR. It looks good to me.

I think we'd have to change mock_version in the unit tests so that it fails a certain number of times, or until sufficient time has passed.

Ah, that makes sense.

I originally thought about an "un-happy" path test, where I just mock get_version to always trigger urllib.error.URLError. The test would run ctx.run(ctx.on.pebble_ready(container), state_in) and expects RuntimeError. An example:

    mock_sleep = Mock()
    monkeypatch.setattr("time.sleep", mock_sleep)
    mock_get = Mock(side_effect=urllib.error.URLError("not ready"))
    monkeypatch.setattr("fastapi_demo.get_version", mock_get)

    ctx = testing.Context(FastAPIDemoCharm)
    container = testing.Container(
        name="demo-server", can_connect=True, layers={"rock": ROCK_LAYER}
    )
    state_in = testing.State(
        containers={container},
        leader=True,
    )

    with pytest.raises(RuntimeError, match="workload is not available"):
        ctx.run(ctx.on.pebble_ready(container), state_in)

I agree that the setup can be fiddly. I am happy to exclude this test.

[dwilding]

@hpidcock this is ready for review now after rework - thanks!

[hpidcock]

LGTM

[hpidcock]

Showing a more durable retry strategy in tutorials is a good idea.

Suggested change

	# The workload may not be ready immediately after replan(), so try get_version() in a loop.
	for attempt in range(3):
	if attempt:
	time.sleep(1) # If not the first attempt, wait before retrying.
	try:
	version = fastapi_demo.get_version(port=8000)
	break
	except urllib.error.URLError:
	continue
	else:
	logger.error("The workload was not available within the expected time")
	raise RuntimeError("workload is not available")
	# The workload may not be ready immediately after replan(), so try get_version() in a loop.
	for attempt in range(6):
	if attempt:
	time.sleep(2 ** attempt) # If not the first attempt, an exponential back-off before retrying.
	try:
	version = fastapi_demo.get_version(port=8000)
	break
	except urllib.error.URLError:
	continue
	else:
	logger.error("The workload was not available within the expected time")
	raise RuntimeError("workload is not available")

[dwilding]

It's overkill for this particular charm, because the workload really does just need a couple more seconds to get ready. If it's going to fail for someone working through the tutorial, I would want it to fail in less than ~1 minute.

But I get your point that we want to teach a robust strategy. And now I remember that we use exponential back-off in the httpbin demo charm too (here).

How about this:

for attempt in range(3):  # In general, allow more attempts for a complex workload.