DAGs for Slurm.

✅ DAG management: Organize your workflows in clean DAGs	✅ Caching: Completed jobs won't run again
✅ Code first API: Familiar Python API equipped with a Rust scheduler	✅ Failure recovery: Handle errors and retries
✅ Visualization: View workflows and details directly in the terminal	✅ Control flow: Adapt your pipelines dynamically
✅ CLI: Manage all pipelines from a unique entrypoint	✅ Local or Slurm: Run on your laptop or scale across a Slurm cluster

Why sdag

Historically, managing interdependent jobs on Slurm clusters has been tough. While in the Cloud tools like AirFlow and Kubeflow are available, their deployment in air-gapped HPC environments is usually very challenging as internet connection and tools like Docker or Kubernetes are not available. Therefore, teams often resort to relying on custom Bash scripts and makefiles, which quickly become large and hard to maintain as the project complexity grows.

sdag tries to fill this gap by providing a typed, code-first API. Its design loosely follows the one of Kubeflow, so it will feel natural if you already worked with it. However, it's way lighter! So it can easily run on HPC infra. Tasks are compiled down to a graph and executed by a Rust-based scheduler.

Installation

sdag is not available on pypi yet (it's coming soon!). If you manage dependencies with uv, you can directly install sdag from GitHub:

uv add "sdag @ git+https://github.com/igeniusai/sdag"

You can also install sdag as a tool to gain global access to it:

uv tool install "sdag @ git+https://github.com/igeniusai/sdag"

To install sdag with pip, first clone the GitHub repository:

git clone https://github.com/igeniusai/sdag.git

and then install sdag:

pip install path/to/sdag

If you face issues in the process, you likely have to install Rust or update it to a more recent version via:

rustup update

Getting started

Paste the following code into a module in the main project directory:

from sdag import Script, pipeline


@pipeline
def hello_world() -> None:
    say_hello(name="sdag")


@hello_world.task(Script("sdag-execute"))
def say_hello(name: str) -> None:
    print(f"hello from {name}")

To run the hello_world pipeline locally, execute:

sdag run hello_world --local

Check out the official documentation for a detailed description of the features offered by sdag.

Building from source and running tests

After cloning the repository, run:

uv sync --all-groups

to build the environment and:

source .venv/bin/activate

to activate it. Execute:

pytest

to run the the quick Python test suite and:

pytest --runslow

to run every Python test. Rust tests can be executed through:

cargo test

Building the docs

Execute:

mkdocs serve

to build and serve the documentation. The server will be running at http://127.0.0.1:8000/.