Skip to content

Migrate Python environment management from conda to uv #1082

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rly merged 12 commits into from
Jul 16, 2026
Merged

Migrate Python environment management from conda to uv #1082

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
b677ab4
Migrate Python environment management from conda to uv
rly Jul 13, 2026
cd8e78a
Merge remote-tracking branch 'origin/main' into migrate/conda-to-uv
rly Jul 14, 2026
9fef659
Make the pyproject version dynamic to avoid duplicating package.json
rly Jul 14, 2026
796f037
Harden matplotlib removal and apply it in the deploy builds
rly Jul 14, 2026
20d3100
Ignore *.egg-info produced by the PyInstaller build
rly Jul 14, 2026
073096f
Use the non-login bash shell so the venv wins on macOS PATH
rly Jul 14, 2026
e5a5734
Log chokidar change events to diagnose intel-mac E2E restart
rly Jul 15, 2026
1f61d5c
Revert "Log chokidar change events to diagnose intel-mac E2E restart"
rly Jul 15, 2026
1d5b568
Cache npm downloads in CI and retry the flaky x64 E2E runner
rly Jul 15, 2026
e46ae41
Raise intel E2E retry attempts to 5
rly Jul 15, 2026
9745ecd
Pin setup-uv to the immutable v8.3.2 release
rly Jul 16, 2026
40612be
Explain why the workflows uninstall matplotlib
rly Jul 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 24 additions & 9 deletions .github/workflows/build_and_deploy_mac.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,33 +8,48 @@ on:
jobs:
deploy-on-mac:
runs-on: macos-15-intel
# NOTE: if changing this to macos-latest (arm64), make sure to use the apple-silicon conda environment.

defaults:
run:
shell: bash -l {0}
shell: bash

steps:
- uses: actions/checkout@v6

- name: Set up conda
uses: conda-incubator/setup-miniconda@v3
- name: Install uv
uses: astral-sh/setup-uv@v8.3.2
with:
miniforge-version: latest
activate-environment: nwb-guide
conda-remove-defaults: true
enable-cache: true

- name: Create and activate environment
run: conda env update --name nwb-guide --file environments/environment-MAC-intel.yml
- name: Install Python dependencies
run: uv sync --locked

- name: Add the virtual environment to PATH
run: |
if [ "$RUNNER_OS" == "Windows" ]; then
echo "$GITHUB_WORKSPACE\\.venv\\Scripts" >> "$GITHUB_PATH"
else
echo "$GITHUB_WORKSPACE/.venv/bin" >> "$GITHUB_PATH"
fi
echo "VIRTUAL_ENV=$GITHUB_WORKSPACE/.venv" >> "$GITHUB_ENV"

- name: Use Node.js 20
uses: actions/setup-node@v6
with:
node-version: 20
cache: 'npm'

- name: Install package.json modules and their dependencies (not via `npm ci`)
run: npm install --verbose

# Keep the released bundle consistent with the one exercised by the build tests.
Comment thread
pauladkisson marked this conversation as resolved.
# matplotlib arrives transitively through neuroconv's extras and GUIDE never imports it.
# PyInstaller bundles whatever is installed, and matplotlib's hidden libraries break the
# packaged build, so every job drops it to keep tests and releases on the same environment.
# `uv tree --invert --package matplotlib` lists what currently pulls it in.
- name: Manually remove matplotlib
run: uv pip uninstall matplotlib || true

- uses: apple-actions/import-codesign-certs@v6
with:
# Currently this is set to Ryan's Developer ID certificate
Expand Down
32 changes: 24 additions & 8 deletions .github/workflows/build_and_deploy_win.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,28 +10,44 @@ jobs:

defaults:
run:
shell: bash -l {0}
shell: bash

steps:
- uses: actions/checkout@v6

- name: Set up conda
uses: conda-incubator/setup-miniconda@v3
- name: Install uv
uses: astral-sh/setup-uv@v8.3.2
with:
miniforge-version: latest
activate-environment: nwb-guide
conda-remove-defaults: true
enable-cache: true

- name: Create and activate environment
run: conda env update --name nwb-guide --file environments/environment-Windows.yml
- name: Install Python dependencies
run: uv sync --locked

- name: Add the virtual environment to PATH
run: |
if [ "$RUNNER_OS" == "Windows" ]; then
echo "$GITHUB_WORKSPACE\\.venv\\Scripts" >> "$GITHUB_PATH"
else
echo "$GITHUB_WORKSPACE/.venv/bin" >> "$GITHUB_PATH"
fi
echo "VIRTUAL_ENV=$GITHUB_WORKSPACE/.venv" >> "$GITHUB_ENV"

- uses: actions/setup-node@v6
with:
node-version: "20"
cache: 'npm'

- name: Install package.json modules and their dependencies
run: npm install --verbose

# Keep the released bundle consistent with the one exercised by the build tests.
# matplotlib arrives transitively through neuroconv's extras and GUIDE never imports it.
# PyInstaller bundles whatever is installed, and matplotlib's hidden libraries break the
# packaged build, so every job drops it to keep tests and releases on the same environment.
# `uv tree --invert --package matplotlib` lists what currently pulls it in.
- name: Manually remove matplotlib
run: uv pip uninstall matplotlib || true

- name: Build and deploy on Windows
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Expand Down
49 changes: 20 additions & 29 deletions .github/workflows/testing_dev.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,63 +15,54 @@ jobs:
runs-on: ${{ matrix.os }}
defaults:
run:
shell: bash -l {0}
shell: bash
strategy:
fail-fast: false
matrix:
include:
- os: ubuntu-latest
label: environments/environment-Linux.yml

- os: macos-latest # Mac arm64 runner
label: environments/environment-MAC-apple-silicon.yml

- os: macos-15-intel # Mac x64 runner
label: environments/environment-MAC-intel.yml

- os: windows-latest
label: environments/environment-Windows.yml

steps:
- uses: actions/checkout@v6

# see https://github.com/conda-incubator/setup-miniconda#caching-environments
- name: Set up conda
uses: conda-incubator/setup-miniconda@v3
- name: Install uv
uses: astral-sh/setup-uv@v8.3.2
with:
miniforge-version: latest
activate-environment: nwb-guide
conda-remove-defaults: true
enable-cache: true

- name: Set cache date
id: get-date
run: echo "today=$(/bin/date -u '+%Y%m%d')" >> $GITHUB_OUTPUT
shell: bash
- name: Install Python dependencies
run: uv sync --locked

- name: Cache Conda env
uses: actions/cache@v5
with:
path: ${{ env.CONDA }}/envs
key: conda-${{ runner.os }}-${{ runner.arch }}-${{steps.get-date.outputs.today }}-${{ hashFiles(matrix.label) }}-${{ env.CACHE_NUMBER }}
id: cache

- if: steps.cache.outputs.cache-hit != 'true'
name: Create and activate environment
run: conda env update --name nwb-guide --file ${{ matrix.label }}
- name: Add the virtual environment to PATH
run: |
if [ "$RUNNER_OS" == "Windows" ]; then
echo "$GITHUB_WORKSPACE\\.venv\\Scripts" >> "$GITHUB_PATH"
else
echo "$GITHUB_WORKSPACE/.venv/bin" >> "$GITHUB_PATH"
fi
echo "VIRTUAL_ENV=$GITHUB_WORKSPACE/.venv" >> "$GITHUB_ENV"

- name: Use Node.js 20
uses: actions/setup-node@v6
with:
node-version: 20
cache: 'npm'

- name: Install GUIDE
run: npm ci --verbose

- name: Install testing dependencies
run: pip install pytest pytest-cov

# matplotlib arrives transitively through neuroconv's extras and GUIDE never imports it.
# PyInstaller bundles whatever is installed, and matplotlib's hidden libraries break the
# packaged build, so every job drops it to keep tests and releases on the same environment.
# `uv tree --invert --package matplotlib` lists what currently pulls it in.
- name: Manually remove matplotlib
run: pip uninstall matplotlib --yes
run: uv pip uninstall matplotlib || true

- if: matrix.os != 'ubuntu-latest'
name: Run tests
Expand Down
61 changes: 31 additions & 30 deletions .github/workflows/testing_dev_e2e_with_live_services.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,70 +17,71 @@ jobs:
runs-on: ${{ matrix.os }}
defaults:
run:
shell: bash -l {0}
shell: bash
strategy:
fail-fast: false
matrix:
include:
- os: ubuntu-latest
label: environments/environment-Linux.yml

- os: macos-latest # Mac arm64 runner
label: environments/environment-MAC-apple-silicon.yml

- os: macos-15-intel # Mac x64 runner
label: environments/environment-MAC-intel.yml

- os: windows-latest
label: environments/environment-Windows.yml

steps:
- uses: actions/checkout@v6

# see https://github.com/conda-incubator/setup-miniconda#caching-environments
- name: Set up conda
uses: conda-incubator/setup-miniconda@v3
- name: Install uv
uses: astral-sh/setup-uv@v8.3.2
with:
miniforge-version: latest
activate-environment: nwb-guide
conda-remove-defaults: true

- name: Set cache date
id: get-date
run: echo "today=$(/bin/date -u '+%Y%m%d')" >> $GITHUB_OUTPUT
shell: bash
enable-cache: true

- name: Cache Conda env
uses: actions/cache@v5
with:
path: ${{ env.CONDA }}/envs
key: conda-${{ runner.os }}-${{ runner.arch }}-${{steps.get-date.outputs.today }}-${{ hashFiles(matrix.label) }}-${{ env.CACHE_NUMBER }}
id: cache
- name: Install Python dependencies
run: uv sync --locked

- if: steps.cache.outputs.cache-hit != 'true'
name: Create and activate environment
run: conda env update --name nwb-guide --file ${{ matrix.label }}
- name: Add the virtual environment to PATH
run: |
if [ "$RUNNER_OS" == "Windows" ]; then
echo "$GITHUB_WORKSPACE\\.venv\\Scripts" >> "$GITHUB_PATH"
else
echo "$GITHUB_WORKSPACE/.venv/bin" >> "$GITHUB_PATH"
fi
echo "VIRTUAL_ENV=$GITHUB_WORKSPACE/.venv" >> "$GITHUB_ENV"

- name: Use Node.js 20
uses: actions/setup-node@v6
with:
node-version: 20
cache: 'npm'

- name: Install GUIDE
run: npm ci --verbose

- name: Install testing dependencies
run: pip install pytest pytest-cov

# matplotlib arrives transitively through neuroconv's extras and GUIDE never imports it.
# PyInstaller bundles whatever is installed, and matplotlib's hidden libraries break the
# packaged build, so every job drops it to keep tests and releases on the same environment.
# `uv tree --invert --package matplotlib` lists what currently pulls it in.
- name: Manually remove matplotlib
run: pip uninstall matplotlib --yes
run: uv pip uninstall matplotlib || true

- name: Create env file
run: |
touch .env
echo DANDI_SANDBOX_API_KEY=${{ secrets.DANDI_SANDBOX_API_KEY }} >> .env

- if: matrix.os != 'ubuntu-latest'
# The macOS x64 runner intermittently crashes Chromium/Electron subprocesses (GPU and
# network-service), which fails the E2E run. Retry only on that runner.
- if: matrix.os == 'macos-15-intel'
name: Run tests (retried on the flaky x64 runner)
uses: nick-fields/retry@ad984534de44a9489a53aefd81eb77f87c70dc60 # v4.0.0

@pauladkisson pauladkisson Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this job do?

@pauladkisson pauladkisson Jul 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And also, why can't you just @v4?

@rly rly Jul 16, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The job retries the below command (npm run coverage:tutorial) if it fails, with 5 max attempts and a timeout of 25 minutes, in case it gets stuck.

Recent best practices for github actions, especially non-official actions, are to pin a specific commit instead of the mutable version tag.

Last year, a popular GitHub Action tj-actions/changed-files was compromised and most release tags were updated to point to a new commit with code that leaked secrets. https://www.stepsecurity.io/blog/harden-runner-detection-tj-actions-changed-files-action-is-compromised.

To prevent these attacks, GitHub now supports "immutable releases", but adoption has been slow across the actions community. Surprisingly, not even the github-maintained core "actions/checkout" action is immutable: actions/checkout#2316 . The setup-uv action has started to make immutable releases, and so pinning to a version tag there is fine (note that they stopped publishing "floating tags" (v4, v4.0) for similar reasons). Until actions are immutable, pinning to a specific release commit gives the highest security possible, at the costs of the workflow being less readable and the action not auto-updated. I think in a year, actions will be immutable, and we can return to tag-based pointers.

I did not update the action pins in all the workflows in this PR, just the changed ones. The others ones will be updated in a separate PR.

@pauladkisson pauladkisson Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh, good to know. Thanks for the info!

with:
timeout_minutes: 25
max_attempts: 5
command: npm run coverage:tutorial

- if: matrix.os == 'macos-latest' || matrix.os == 'windows-latest'
name: Run tests
run: npm run coverage:tutorial

Expand Down
49 changes: 20 additions & 29 deletions .github/workflows/testing_dev_with_live_services.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,64 +15,55 @@ jobs:
runs-on: ${{ matrix.os }}
defaults:
run:
shell: bash -l {0}
shell: bash
strategy:
fail-fast: false
matrix:
include:
- os: ubuntu-latest
label: environments/environment-Linux.yml

- os: macos-latest # Mac arm64 runner
label: environments/environment-MAC-apple-silicon.yml

- os: macos-15-intel # Mac x64 runner
label: environments/environment-MAC-intel.yml


- os: windows-latest
label: environments/environment-Windows.yml

steps:
- uses: actions/checkout@v6

# see https://github.com/conda-incubator/setup-miniconda#caching-environments
- name: Set up conda
uses: conda-incubator/setup-miniconda@v3
- name: Install uv
uses: astral-sh/setup-uv@v8.3.2
with:
miniforge-version: latest
activate-environment: nwb-guide
conda-remove-defaults: true
enable-cache: true

- name: Set cache date
id: get-date
run: echo "today=$(/bin/date -u '+%Y%m%d')" >> $GITHUB_OUTPUT
shell: bash
- name: Install Python dependencies
run: uv sync --locked

- name: Cache Conda env
uses: actions/cache@v5
with:
path: ${{ env.CONDA }}/envs
key: conda-${{ runner.os }}-${{ runner.arch }}-${{steps.get-date.outputs.today }}-${{ hashFiles(matrix.label) }}-${{ env.CACHE_NUMBER }}
id: cache

- if: steps.cache.outputs.cache-hit != 'true'
name: Create and activate environment
run: conda env update --name nwb-guide --file ${{ matrix.label }}
- name: Add the virtual environment to PATH
run: |
if [ "$RUNNER_OS" == "Windows" ]; then
echo "$GITHUB_WORKSPACE\\.venv\\Scripts" >> "$GITHUB_PATH"
else
echo "$GITHUB_WORKSPACE/.venv/bin" >> "$GITHUB_PATH"
fi
echo "VIRTUAL_ENV=$GITHUB_WORKSPACE/.venv" >> "$GITHUB_ENV"

- name: Use Node.js 20
uses: actions/setup-node@v6
with:
node-version: 20
cache: 'npm'

- name: Install GUIDE
run: npm ci --verbose

- name: Install testing dependencies
run: pip install pytest pytest-cov

# matplotlib arrives transitively through neuroconv's extras and GUIDE never imports it.
# PyInstaller bundles whatever is installed, and matplotlib's hidden libraries break the
# packaged build, so every job drops it to keep tests and releases on the same environment.
# `uv tree --invert --package matplotlib` lists what currently pulls it in.
- name: Manually remove matplotlib
run: pip uninstall matplotlib --yes
run: uv pip uninstall matplotlib || true

- if: matrix.os != 'ubuntu-latest'
name: Run tests
Expand Down
Loading
Loading