Skip to content

fix: ignore "directory" blobs in the Azure backend when walking. - #36

Merged
peterhuene merged 4 commits into
stjude-rust-labs:mainfrom
peterhuene:fix-directory-blobs
Jun 24, 2026
Merged

peterhuene merged 4 commits into
stjude-rust-labs:mainfrom
peterhuene:fix-directory-blobs

Conversation

@peterhuene

@peterhuene peterhuene commented Jun 17, 2026 •

Copy link
Copy Markdown
Collaborator

This PR fixes including "directory" blobs as output from the walk method, which should only be (conceptually) "files".

Azure Blob Storage has a feature for creating empty blobs that act as "directories" in a hierarchical namespace; the directory objects hold metadata such as Access Control Lists (ACLs).

By including the directory as output from the walk function, callers might attempt to download the directory as a file and then create it as a directory when also downloading the directory's contents, which would result in a conflicting file system error.

Additionally included a check for such conflicts so that walking returns an error.

Replaced localstack with floci for testing.

Before submitting this PR, please make sure:

  • You have added a few sentences describing the PR here.
  • You have added yourself or the appropriate individual as the assignee.
  • You have added at least one relevant code reviewer to the PR.
  • Your code builds clean without any errors or warnings.
  • You have added tests (when appropriate).
  • You have updated the README or other documentation to account for these
    changes (when appropriate).
  • You have added an entry to the relevant CHANGELOG.md (see
    "keep a changelog" for more information).
  • Your commit messages follow the conventional commit style.

This commit fixes including "directory" blobs as output from the `walk` method,
which should only be (conceptually) "files".

Azure Blob Storage has a feature for creating empty-sized blobs that act as
"directories" in a hierarchical namespace; the directory objects hold metadata
such as Access Control Lists (ACLs).

By including the directory as output from the `walk` function, callers might
attempt to download the directory as a file and then create it as a directory
when also downloading the directory's contents, which would result in a
conflicting file system error.

Additionally included a check for such conflicts so that walking returns an
error.

Replaced `localstack` with `floci` for testing.
@peterhuene
peterhuene requested a review from claymcleod June 17, 2026 16:09
@peterhuene peterhuene self-assigned this Jun 17, 2026
@peterhuene peterhuene mentioned this pull request Jun 24, 2026
8 tasks done

@claymcleod claymcleod left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One path-safety issue could not be attached inline: in src/transfer.rs:851, walked object names are joined into local destination paths without rejecting .. or other non-normal components, so a crafted object name can escape the requested download directory. That looks pre-existing, but the new walk-conflict validation is a natural place to cover it.

Comment thread src/backend/azure.rs Outdated
Comment thread src/lib.rs
Comment thread src/backend/azure.rs Outdated
Comment thread README.md Outdated
Comment thread src/backend/azure.rs Outdated
Remove new deserialization code in favor of filtering out directories in
the request to "list" a container's content.
@peterhuene
peterhuene requested a review from claymcleod June 24, 2026 20:23
@peterhuene
peterhuene merged commit 387f756 into stjude-rust-labs:main Jun 24, 2026
9 checks passed
@peterhuene
peterhuene deleted the fix-directory-blobs branch June 24, 2026 22:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants