Skip to content

[FEA] Add fixed-width list support to JIT transforms - #24767

Draft
lamarrr wants to merge 21 commits into
NVIDIA:mainfrom
lamarrr:transform-list-support
Draft

lamarrr wants to merge 21 commits into
NVIDIA:mainfrom
lamarrr:transform-list-support

Conversation

@lamarrr

@lamarrr lamarrr commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

Allow CUDA transforms to consume lists with fixed-width children, including dictionary-encoded list keys, and produce fixed-width lists using caller-supplied INT32 offsets. Add list_element and mutable_list_element, nested output specifications, list allocation/finalization, and output-offset validation.

Cover dictionary keys, scalar/sliced inputs, Decimal128 children, parent and child validity, empty/all-null inputs, a size-scan-write workflow, mixed outputs, and unsupported-child rejection in eleven new integration tests. Update the pylibcudf declaration of transform_output.

This PR is based on the string/representation-access migration. Nested lists and string children remain rejected by the transform API.

Validation

  • Full libcudf build and all 122 CTest entries pass in the CUDA 13.3 devcontainer using sccache (the default RAPIDS CUDA architecture set).
  • Focused accessor GPU probes and Compute Sanitizer memcheck pass with zero errors.
  • Doxygen passes with warnings as errors.
  • Python package builds and pytest suites were not run.

PR stack

Depends on #24425 and #24766. This draft targets main, so its PR diff includes those prerequisites until they merge. The changes specific to this part are in commit 4984f7207db7c52054aef595629085ed72ac19ed and this comparison.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@copy-pr-bot

copy-pr-bot Bot commented Oct 22, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added libcudf Affects libcudf (C++/CUDA) code. Python Affects Python cuDF API. CMake CMake build issue pylibcudf Issues specific to the pylibcudf package labels Oct 22, 2026
Reconcile source-type reflection and table mappings with core device-view
accessors. Preserve decoded dictionary and string UDF value types, and
normalize default column indices to the accessor's int32_t type.

Add regression coverage for equivalent mappings, invalid mappings,
CUDA/PTX/LTO type selection, UDF signatures, and storage-derived specs.
Keep four-parameter kernel accessors and mutable string outputs while
aligning precompiled fragment names with signed runtime indices.

Verify all 61 precompiled fixed-width and string configurations against
runtime reflection to prevent silent fallback to JIT compilation.
… includes

Remove JIT reflection test and associated configurations
rapids-bot Bot pushed a commit that referenced this pull request Oct 24, 2026
Transform reflection and output-storage wrappers currently live inside `transform.cu.`

They have been moved into another header to allow them to be reused for other JIT implementations like `filter_join_indices` and `reduce` (#24445 )

Follow-up PRs: 

- #24766 
- #24767

Both are stacked on this PR

Authors:
  - Basit Ayantunde (https://github.com/lamarrr)

Approvers:
  - Muhammad Haseeb (https://github.com/mhaseeb123)
  - Yunsong Wang (https://github.com/PointKernel)

URL: #24425
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CMake CMake build issue libcudf Affects libcudf (C++/CUDA) code. pylibcudf Issues specific to the pylibcudf package Python Affects Python cuDF API.

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

1 participant