tap-dagster

tap-dagster is a Singer tap for Dagster.

Built with the Meltano Tap SDK for Singer Taps.

Installation

Install from GitHub:

uv tool install git+https://github.com/MeltanoLabs/tap-dagster.git@main

Configuration

Accepted Config Options

Setting	Required	Default	Description
api_token	True	None	The API token to authenticate against the Dagster Cloud API
base_url	True	None	The base URL for the Dagster Cloud API
start_date	False	None	The earliest record date to sync
stream_maps	False	None	Config object for stream maps capability. For more information check out Stream Maps.
stream_maps.else	False	None	Currently, only setting this to __NULL__ is supported. This will remove all other streams.
stream_map_config	False	None	User-defined config values to be used within map expressions.
faker_config	False	None	Config for the Faker instance variable fake used within map expressions. Only applicable if the plugin specifies faker as an additional dependency (through the singer-sdk faker extra or directly).
faker_config.seed	False	None	Value to seed the Faker generator for deterministic output: https://faker.readthedocs.io/en/master/#seeding-the-generator
faker_config.locale	False	None	One or more LCID locale strings to produce localized output for: https://faker.readthedocs.io/en/master/#localization
flattening_enabled	False	None	'True' to enable schema flattening and automatically expand nested properties.
flattening_max_depth	False	None	The max depth to flatten schemas.
batch_config	False	None	Configuration for BATCH message capabilities.
batch_config.encoding	False	None	Specifies the format and compression of the batch files.
batch_config.encoding.format	False	None	Format to use for batch files.
batch_config.encoding.compression	False	None	Compression format to use for batch files.
batch_config.storage	False	None	Defines the storage layer to use when writing batch files
batch_config.storage.root	False	None	Root path to use when writing batch files.
batch_config.storage.prefix	False	None	Prefix to use when writing batch files.

A full list of supported settings and capabilities is available by running: tap-dagster --about.

Configure using environment variables

This Singer tap will automatically import any environment variables within the working directory's .env if the --config=ENV is provided, such that config values will be considered if a matching environment variable is set either in the terminal context or in the .env file.

Source Authentication and Authorization

Generate a new API token in the Dagster Cloud UI by navigating to the "Organization Settings" page and clicking the "Tokens" tab.

Usage

You can easily run tap-dagster by itself or in a pipeline using Meltano.

Executing the Tap Directly

tap-dagster --version
tap-dagster --help
tap-dagster --config CONFIG --discover > ./catalog.json

Developer Resources

Follow these instructions to contribute to this project.

Initialize your Development Environment

Prerequisites:

• Python 3.10+
• uv

uv sync

Create and Run Tests

Create tests within the tests subfolder and then run:

uv run pytest

You can also test the tap-dagster CLI interface directly using uv run:

uv run tap-dagster --help

Testing with Meltano

Note: This tap will work in any Singer environment and does not require Meltano. Examples here are for convenience and to streamline end-to-end orchestration scenarios.

Next, install Meltano (if you haven't already) and any needed plugins:

# Install meltano
uv tool install meltano
# Initialize meltano within this directory
cd tap-dagster
meltano install

Now you can test and orchestrate using Meltano:

# Test invocation:
meltano invoke tap-dagster --version

# OR run a test ELT pipeline:
meltano run tap-dagster target-jsonl

SDK Dev Guide

See the dev guide for more instructions on how to use the SDK to develop your own taps and targets.