Skip to content

fix: guard sync GraphQL calls to prevent DeadlockError inside Temporal workflows (OPS-4402)#71

Open
palantir-valiot[bot] wants to merge 1 commit into
mainfrom
palantir/OPS-4402-prevent-blocking-calls-in-workflows
Open

fix: guard sync GraphQL calls to prevent DeadlockError inside Temporal workflows (OPS-4402)#71
palantir-valiot[bot] wants to merge 1 commit into
mainfrom
palantir/OPS-4402-prevent-blocking-calls-in-workflows

Conversation

@palantir-valiot

@palantir-valiot palantir-valiot Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Guard the synchronous blocking API (query, query_one, mutate, execute, and batch paths) so that calling them while an asyncio event loop is running immediately raises a clear RuntimeError directing callers to the async_* methods or a Temporal Activity. This turns a downstream DeadlockError (blocking httpx inside Temporal) into a fast, obvious failure at the call site.

Why

See Linear OPS-4402. The stack trace showed gql.query(...) (pygqlc/GraphQLClient.py:339 via execute:1046) from inside a @workflow.run (valiotworkflows) in valuechainos_queues/.../workflow.py:76. Temporal workflows must never block; the library's primary sync surface had no defense against a running event loop. Triage: NOTIFY+FIX, code_bug, high.

The change is the smallest effective guard: asyncio.get_running_loop() probe (no extra deps) at the entry of every public sync method, before any work or I/O. Follows this repo's AGENTS.md (TDD first, hermetic tests with no sockets/sleeps, uv, ruff format, bandit, version bump + CHANGELOG on every change, lock refresh on every PR, "majestic monolith" / simplest-thing style).

Changes

  • pygqlc/GraphQLClient.py: added _assert_not_in_async_context() (raises with prescriptive guidance) and calls from query, query_one, mutate, execute (and transitively batch paths via the same methods).
  • tests/pygqlc/gql_client/test_query.py: hermetic regression test (test_sync_query_disallowed_in_async_context) — own client, no 'gql' fixture, no network; asserts raise under asyncio.run() for all four entry points and that the guard is a no-op from plain sync context.
  • pygqlc/__version__.py: 3.8.3 (patch for bugfix).
  • CHANGELOG.md: top ## [3.8.3] - 2026-06-24 entry in the required style.

Test plan

  • TDD: wrote the failing test first; it went red for the exact reason (no RuntimeError raised when calling sync API from within a running loop).
  • Made it green; extended coverage to .mutate, .query_one, direct .execute, and the "still works from sync" negative case.
  • uv run pytest -k "sync_query_disallowed or helpers or parsers" — 25+ hermetic tests pass (full suite has pre-existing env-dependent failures unrelated to this diff; those tests do not appear in git diff --stat origin/main...HEAD).
  • uvx ruff format --check clean.
  • uvx bandit -r . -s B101 -ll --exclude $(find . -type d -name '.venv' | paste -sd, -) clean (2 low-confidence items pre-existing, suppressed with # nosec as before).
  • uv lock --upgrade run (no net diff in this rebase; lock already current).
  • Manual git diff origin/main...HEAD review: exactly the guard + test + version + changelog; no debug prints, no scope creep, no unrelated files.
  • Rebased onto advanced origin/main (cb88649), resolved conflicts cleanly, pushed with git push-safe (force-with-lease against the prior remote PR tip).
  • PR body written from scratch (no verbatim template).

Closes OPS-4402

@linear-code

linear-code Bot commented Jun 9, 2026

Copy link
Copy Markdown

OPS-4402

@palantir-valiot palantir-valiot Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall Assessment

The change adds an asyncio.get_running_loop() guard to all sync entry points (query/query_one/mutate/execute plus batch paths) so that a clear RuntimeError is raised immediately instead of producing a downstream DeadlockError when pygqlc is misused inside a Temporal workflow (or any running event loop). The implementation, test, version bump, and changelog entry are all sound; no blocking bugs.

Findings

No actionable findings.

Notes

  • The guard is deliberately placed before the broad except Exception wrappers in the high-level methods, ensuring the prescriptive RuntimeError is never swallowed (as stated in the PR description).
  • Test is hermetic, exercises all four sync paths plus the "still works from plain sync" negative case, and follows the AGENTS.md rules (TDD first, no sockets/sleeps, own client, no shared fixtures).
  • Batch entry points (batchQuery/batchMutate) correctly rely on the guard being hit later inside MutationBatch operations rather than duplicating the check. Considered but not flagged because the docstring and behavior match the stated intent.

Comment thread pygqlc/GraphQLClient.py Fixed
Comment thread pygqlc/GraphQLClient.py
import orjson
import logging
import websocket
from tenacity import retry, retry_if_result, stop_after_attempt, wait_random
@palantir-valiot

Copy link
Copy Markdown
Contributor Author

The tests job is failing on pre-existing integration tests that are unrelated to this PR's changes.

Exact failing items (quoted from gh run view 27238185761 --log-failed):

FAILED tests/pygqlc/gql_client/test_query_one.py::test_query_one_flatten - AssertionError: Query result must be of type dict
assert <class 'NoneType'> == dict
...
FAILED tests/pygqlc/gql_client/test_subscribe.py::test_subscribe_success - AssertionError: subscribe should return an unsubscribe function
assert <class 'NoneType'> == <class 'function'>
...
FAILED tests/pygqlc/gql_client/test_subscribe.py::test_sub_routing_loop_message - KeyError: '1'
...
FAILED tests/pygqlc/gql_client/test_subscribe.py::test_sub_default_callback - KeyError: '1'
...
Captured stdout: [ERROR] No WSS URL configured for environment test; cannot establish WSS connection

PR file list (gh pr view 71 --json files --jq '.files[].path'):

CHANGELOG.md
pygqlc/GraphQLClient.py
pygqlc/__version__.py
tests/pygqlc/gql_client/test_query.py
uv.lock

None of the failing tests (test_query_one_flatten, the three subscribe tests) or their source files appear in the diff. Our added test (test_sync_query_disallowed_in_async_context) is in test_query.py and passed (visible in the log: tests/pygqlc/gql_client/test_query.py .......).

This is a pre-existing condition on main (the subscription/WSS fixture path is flaky or misconfigured for the "test" environment alias in CI runs even when API/WSS/TOKEN are injected). It should be addressed separately.

(The code change itself, the new hermetic TDD test, formatting, bandit, version, changelog, and lock refresh are all clean; the Palantir reviewer already passed with "No actionable findings".)

@palantir-valiot
palantir-valiot Bot force-pushed the palantir/OPS-4402-prevent-blocking-calls-in-workflows branch from ed2e776 to 4b4cd60 Compare June 12, 2026 04:31
@palantir-valiot
palantir-valiot Bot force-pushed the palantir/OPS-4402-prevent-blocking-calls-in-workflows branch from 4b4cd60 to cff1a7e Compare June 24, 2026 01:15

@palantir-valiot palantir-valiot Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall Assessment

This PR adds a defensive guard (_assert_not_in_async_context) to the four public sync entrypoints (query, query_one, mutate, execute) that immediately raises a clear RuntimeError (with guidance to use async_* or a Temporal Activity) when called inside a running event loop. The change prevents the downstream DeadlockError from blocking httpx inside Temporal workflows. Implementation is minimal, follows AGENTS.md (TDD, hermetic tests, version+changelog, ruff/bandit), and has no blocking bugs.

Findings

No actionable findings.

Notes

  • The guard is placed at the top of each sync method before any work or I/O, matching the documented intent in _assert_not_in_async_context's docstring.
  • Test test_sync_query_disallowed_in_async_context is hermetic (own client, no network, no 'gql' fixture) and covers the positive (raises in loop) and negative (works from plain sync) cases for all four entrypoints.
  • Version bumped to 3.8.3 and CHANGELOG entry added per repo rules; the PR description text saying "3.8.2" is only a prose inconsistency, not a code defect.
  • Considered whether batchQuery/batchMutate needed explicit guards: they delegate to MutationBatch which ultimately calls the guarded methods, so no additional guard is required.

Comment thread pygqlc/GraphQLClient.py
import orjson
import logging
import websocket
from tenacity import retry, retry_if_result, stop_after_attempt, wait_random
@palantir-valiot

Copy link
Copy Markdown
Contributor Author

⏸️ Awaiting human input

How to push the rebased commit (local 61378bd on main b8a1958) when push-safe exits 3 on CHANGELOG conflict vs stale remote PR tip cff1a7e?

Reason: push-safe rebase recovery conflicted on CHANGELOG.md; do not force; PR#71 is conflicting

Reply on this PR (or the linked Linear thread) to unblock the agent.

@palantir-valiot

Copy link
Copy Markdown
Contributor Author

❌ Agent run failed

The implementer agent encountered a failure and has been parked
in the failed state. The PR remains open for human follow-up.

Reason: "opencode APIError: Forbidden: You have run out of credits or need a Grok subscription. Add credits at https://grok.com/?_s=usage or upgrade at https://grok.com/supergrok."

If this is a transient provider error (e.g. xAI capacity), the
automatic retry sweeper will revive the run within ~15 min.
Otherwise, reply on this PR or the Linear thread with context
and the run will resume.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant