feat: Respect `RateLimit` headers in default REST backoff implementation

[ReubenFrankel]

Feature scope

Taps (catalog, state, stream maps, tests, etc.)

Description

This IETF draft defines a series of RateLimit headers:

• X-RateLimit-Limit
• X-RateLimit-Remaining
• X-RateLimit-Reset

Taps built with the SDK around REST APIs that respond with these headers would benefit from a default RESTStream backoff implementation that understands and respects them - namely X-RateLimit-Reset.

[stale]

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

[edgarrmondragon]

I'm tempted to get rid of (in a backwards-compatible way) the backoff integration in favor of native retries. They support exponential backoff, factor and jitter, and automatically respect the Retry-After header. That wouldn't work with RetriableAPIError, though...

[ReubenFrankel]

I'd be interested in that too. Sounds like a breaking change for RetriableAPIError, unless it's possible to leverage a retry callback somehow e.g. response hook (not sure if this get fired for retries) or extending the Retry class.

[Nazib65]

I'd be interested in working on this.
From the thread, I understand the goal is a default RESTStream backoff that reads the standard X-RateLimit-* headers — particularly respecting X-RateLimit-Reset so taps wait the right amount before retrying.
I also saw the later discussion about possibly moving to requests' native retries (which respect Retry-After and support backoff/jitter) instead of the current backoff integration, along with the open question around RetriableAPIError compatibility.
Before I start, could you clarify which direction you'd prefer?

-Implement the X-RateLimit-aware backoff within the existing mechanism, or
-Tackle it as part of the larger native-retries change you mentioned

Happy to set up the dev environment and scope a PR around whichever approach fits your plans. Thanks!

[ReubenFrankel]

Hey @Nazib65

I'd be interested in working on this.

That would be awesome! 🔥

Implement the X-RateLimit-aware backoff within the existing mechanism

I think this to start with. Sounds like the proposed native retries refactor needs some more consideration around compatibility with RetriableAPIError before starting.

[ReubenFrankel]

This old Slack thread is probably useful context here (I created this issue when working on tap-auth0).

[Nazib65]

Thanks @ReubenFrankel! That makes sense — I'll start with the X-RateLimit-aware backoff in the existing mechanism and leave the native-retries refactor aside for now given the open RetriableAPIError questions.
Appreciate the Slack thread context from the tap-auth0 work — I'll read through it before I start. I'll get the dev environment set up and dig into where the current RESTStream backoff is handled. Will follow up here soon!