feature(connectors): add open-meteo-marine-weather connector#37
Open
kellykohlleffel wants to merge 1 commit into
Open
feature(connectors): add open-meteo-marine-weather connector#37kellykohlleffel wants to merge 1 commit into
kellykohlleffel wants to merge 1 commit into
Conversation
3 tasks
Contributor
|
Hi @kellykohlleffel , Could you please rebase your PR so that the latest checks can run and pass? Thank you! |
Adds a new community connector that syncs hourly and daily marine weather data from the Open-Meteo Marine Weather API (free, no authentication required) for a configured coastal location. Tables synced: - marine_hourly: wave height, direction, period, swell, wind waves, ocean current (15 columns, composite PK: location_id + timestamp) - marine_daily: daily aggregated marine metrics (12 columns, composite PK: location_id + date) Key implementation details: - Incremental sync with date-based cursor (last_synced_date) and 1-day overlap for deduplication at destination - Exponential backoff retry for transient errors (408, 429, 500-504) - Intermediate checkpointing every 100 records to enable mid-batch resume - Configurable forecast window (1-16 days) and historical window (0-92 days) - Type-safe configuration validation with placeholder detection Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
kellykohlleffel
force-pushed
the
feature/open-meteo-marine-weather
branch
from
62fdc08 to
961ebe5
Compare
Author
|
Thanks @fivetran-JenasVimal — rebased onto the latest |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MARINE_HOURLY(15 columns, composite PK:location_id+timestamp) andMARINE_DAILY(12 columns, composite PK:location_id+date)last_synced_date) and 1-day overlap for deduplicationfivetran debugresultsSDK Version: 2.25.1230.001 | Sync: SUCCEEDED | Records: 375 upserts (2 tables)
Run 2 validates cursor resume: previous state
{"last_synced_date": "2026-06-23"}was picked up correctly, date range advanced to 2026-06-22 (with 1-day overlap), upserts correctly deduplicated via composite PK.Run 1 — raw
fivetran debugoutput (initial sync)Run 2 — raw
fivetran debugoutput (resume sync)Test plan
black --line-length 99— no changes neededflake8— no issuesfivetran debugrun 1 (initial sync): SUCCEEDED, 375 upserts, 4 checkpointsfivetran debugrun 2 (resume sync): SUCCEEDED, 225 upserts, cursor resume verifiedopen_meteo_marine_weatherconfiguration.jsoncontains placeholder values only