Skip to content

Link-header pagination: responseToken-only config deadlocks the read loop, and the extracted next URL is never used as the request URL #123

Description

@jeffreyaven

Environment

  • stackql v0.10.582 (c926a33), Windows and Linux, local file registry
  • Found building the rebuilt snowflake provider; reproduced against both a live API and a local mock server

Summary

Link-header pagination (RFC 5988 Link: <url>; rel="next") is broken under both possible config shapes:

  1. The natural config for the scheme - a responseToken only - deadlocks the binary: every list SELECT hangs indefinitely after the first response, whether or not a Link header is present.
  2. Adding a requestToken (semantically meaningless for this scheme - there is no token, only a URL) unblocks the loop, but the URL extracted by defaultLinkHeaderTransformer is then injected verbatim as the request token query parameter value instead of being used as the next request URL.

Repro 1: responseToken-only config hangs

Service-level config (root x-stackQL-config in the service doc):

x-stackQL-config:
  pagination:
    responseToken:
      key: link
      location: header

Any list SELECT against the service hangs forever (no timeout, no error). Reproduced:

  • against a live API that returns 200 + JSON with NO Link header
  • against a mock that returns a correctly formatted header (<url>; rel="next", single space after the semicolon per the linksNextRegex in internal/anysdk/pagination.go)
  • with and without a response transform on the method

Expected: either traverse the chain (header present) / return the single page (header absent), or fail fast at config validation if a responseToken-only pagination block is unsupported. A hang should never be the outcome of a config that passes the published stackql-config.schema.json.

Repro 2: with a requestToken, the next URL is sent as a query param value

x-stackQL-config:
  pagination:
    requestToken:
      key: fromName
      location: query
    responseToken:
      key: link
      location: header

defaultLinkHeaderTransformer correctly extracts the next-page URL from rel="next", but the follow-up request goes out as:

GET /api/v2/databases?fromName=http%3A%2F%2Flocalhost%3A63218%2Fapi%2Fv2%2Fdatabases%3FfromName%3DDB3

i.e. the entire URL is injected as the fromName value. Expected: the extracted URL replaces the request URL for the next page (the whole point of link-header pagination). Wire capture is from a mock-server integration suite that logs every request; happy to share the harness.

Notes

  • Token-style pagination is unaffected: the k8s provider's continue token config works correctly on the same binary, so this is specific to the link-header path's consumer.
  • docs/provider_spec.md documents algorithm: link_header_next for this pattern, but that identifier is not recognized anywhere in code (it merely diverts from the defaultLinkHeaderTransformer fast path into the generic header transformer, which behaves identically) - worth reconciling docs and code in the same fix.
  • Practical impact for snowflake today is nil (the vendor specs declare Link headers but the live control plane does not emit them), so that provider ships with no pagination config. Any provider whose API genuinely paginates via Link headers hits this immediately.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions