Skip to content

feat(dynamictable): add ragged and doubly-ragged array column helpers #855

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
ehennestad wants to merge 8 commits into
base: main
Choose a base branch
from
Draft

feat(dynamictable): add ragged and doubly-ragged array column helpers #855

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
21326dd
feat(dynamictable): add ragged and doubly-ragged array column helpers
ehennestad Jul 7, 2026
9ea2656
Update DynamicTableBase.m
ehennestad Jul 7, 2026
1644c89
docs: add tested ragged-array how-to guide
ehennestad Jul 8, 2026
7628b10
Update create_doubly_indexed_column.m
ehennestad Jul 8, 2026
bd6dbd7
Update unit times roundtrip tests (pynwb<->matnwb)
ehennestad Jul 8, 2026
1e1c12a
test(system): round-trip multi-electrode doubly-ragged Units waveforms
ehennestad Jul 9, 2026
bab50cc
test(system): rename UnitTimesIOTest to UnitsTableIOTest
ehennestad Jul 9, 2026
b369001
feat: support n-dimensional doubly ragged arrays
ehennestad Jul 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
140 changes: 140 additions & 0 deletions +matnwb/+neurodata/DynamicTableBase.m
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
properties (Abstract)
id
colnames
vectordata
end

methods
Expand Down Expand Up @@ -112,6 +113,145 @@ function addColumn(obj, columnName, columnVector)
types.util.dynamictable.addVarargColumn(obj, columnVectorPairs{:});
end

function addRaggedArray(obj, columnName, data, options)
% addRaggedArray - Add a ragged-array column to the DynamicTable.
%
% A ragged array stores a variable number of elements per row. The
% values are held in a single VectorData column, and a companion
% VectorIndex ('<columnName>_index') marks each row's boundary. See
% the "Tables and ragged arrays" section of the NWB format
% specification.
%
% Syntax:
% dynamicTable.addRaggedArray(columnName, data) build and add a
% ragged column named columnName, plus its VectorIndex.
%
% dynamicTable.addRaggedArray(__, Name, Value) provide optional
% arguments (see below).
%
% Input Arguments:
% - columnName (string) -
% Name of the new column.
%
% - data (cell) -
% A cell array with one cell per row; each cell holds that row's
% elements (e.g. {[1 2 3], [4 5]} for a 2-row table).
%
% Name-Value Arguments:
% - description (string) -
% Description stored on the VectorData column.
%
% - table (DynamicTable) -
% If provided, the column is created as a DynamicTableRegion that
% references this table (row indices) instead of a VectorData.
%
% See also util.create_indexed_column, addColumn, addDoublyRaggedArray

arguments
obj (1,1) {matnwb.common.validation.mustBeDynamicTable}
columnName (1,1) string
data cell
options.description (1,1) string = "no description"
options.table = []
end

if isempty(options.table)
[vector, index] = util.create_indexed_column( ...
data, char(options.description));
else
[vector, index] = util.create_indexed_column( ...
data, char(options.description), options.table);
end
obj.addColumn(columnName, vector, columnName + "_index", index);
end

function addDoublyRaggedArray(obj, columnName, data, options)
% addDoublyRaggedArray - Add a doubly-ragged-array column to the DynamicTable.
%
% A doubly-ragged array stores, for each row, a variable number of
% sub-groups, each holding a variable number of fixed-length elements
% (e.g. the Units table 'waveforms' column: per unit, a variable number
% of spike events, each with a waveform per electrode). It is backed by
% a VectorData column and two VectorIndex levels
% ('<columnName>_index' over sub-groups and '<columnName>_index_index'
% over rows). See the "Doubly ragged arrays" section of the NWB format
% specification.
%
% Syntax:
% dynamicTable.addDoublyRaggedArray(columnName, data) build and add a
% doubly-ragged column named columnName, plus its two VectorIndex
% levels.
%
% Input Arguments:
% - columnName (string) -
% Name of the new column.
%
% - data (cell) -
% A cell array with one cell per row. Each cell is either a numeric
% matrix [nSubGroups x nSamples] (one element per sub-group), or a
% cell array whose j-th entry is a [nElements x nSamples] matrix for
% sub-group j. See util.create_doubly_indexed_column.
%
% Name-Value Arguments:
% - description (string) -
% Description stored on the VectorData column.
%
% See also util.create_doubly_indexed_column, addColumn, addRaggedArray

arguments
obj (1,1) {matnwb.common.validation.mustBeDynamicTable}
columnName (1,1) string
data cell
options.description (1,1) string = "no description"
end

[vector, index, indexIndex] = ...
util.create_doubly_indexed_column(data, options.description);

% The number of rows equals the length of the outermost index.
rowCount = numel(indexIndex.data);

% Initialize id for a new table before checking editability (which
% inspects the id column).
if isempty(obj.id) || isempty(obj.id.data)
types.util.dynamictable.internal.initDynamicTableId(obj, rowCount);
end

obj.assertIsEditable('NWB:DynamicTable:AddDoublyRaggedArray:Uneditable')

tableHeight = types.util.dynamictable.internal.getColumnHeight(obj.id);
assert(rowCount == tableHeight, ...
'NWB:DynamicTable:AddDoublyRaggedArray:MissingRows', ...
'Column `%s` has %d rows, but the table height is %d.', ...
columnName, rowCount, tableHeight)

% Assign the data column and both index levels to the appropriate
% storage (a typed property for schema-defined columns such as
% Units.waveforms, otherwise the generic vectordata set). addColumn
% is not used here because its height check only follows a single
% index level.
names = [columnName, columnName + "_index", columnName + "_index_index"];
values = {vector, index, indexIndex};
for iName = 1:numel(names)
thisName = char(names(iName));
storageTarget = types.util.dynamictable.resolveColumnStorage( ...
obj, thisName, values{iName});
switch storageTarget
case 'property'
obj.(thisName) = values{iName};
case 'vectordata'
obj.vectordata.set(thisName, values{iName});
end
end

% Only the data column is listed in colnames; the index levels are
% implicit. Schema-defined columns are added by a property post-set
% hook, so guard against duplicates.
if ~any(strcmp(obj.colnames, char(columnName)))
obj.colnames{end+1} = char(columnName);
end
end

function row = getRow(obj, rowIndices, options)
% getRow - Return one or more DynamicTable rows.
%
Expand Down
28 changes: 24 additions & 4 deletions +tests/+system/PyNWBIOTest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -214,10 +214,30 @@ def getContainer(self, file):
return file


class UnitTimesIOTest(PyNWBIOTest):
class UnitsTableIOTest(PyNWBIOTest):
def addContainer(self, file):
self.file.units = Units('units', waveform_rate=1., resolution=3.)
self.file.units.add_unit(waveform_mean=[5], waveform_sd=[7], waveforms=np.full((1, 1), 9),
spike_times=[11])
# Build a Units table with multi-electrode, doubly-ragged waveforms
# using the idiomatic add_unit method. This mirrors the MatNWB
# addRaggedArray / addDoublyRaggedArray construction in UnitsTableIOTest.m.
#
# Each unit's waveforms are passed as a 3-D array with shape
# (num_spikes, num_electrodes, num_samples). Note the LAST dimension is
# samples, not electrodes: the `waveforms` column is doubly indexed, so
# add_unit stores a 2-D [num_waveforms, num_samples] dataset with the
# electrode dimension carried by waveforms_index. (The add_unit docstring
# line "the third dimension shows ... different electrodes" describes a
# non-indexed 3-D dataset and does not match this input order.)
#
# Unit 1 has 2 spikes, unit 2 has 3 spikes; each spike has 2 electrodes
# and 3 samples.
self.file.units = Units(name='units', waveform_rate=1., resolution=3.,
description='data on spiking units')
self.file.units.add_unit(spike_times=[1., 2.],
waveform_mean=[1., 2., 3.], waveform_sd=[7., 8., 9.],
waveforms=np.arange(1, 13, dtype=np.int32).reshape(2, 2, 3))
self.file.units.add_unit(spike_times=[3., 4., 5.],
waveform_mean=[4., 5., 6.], waveform_sd=[10., 11., 12.],
waveforms=np.arange(13, 31, dtype=np.int32).reshape(3, 2, 3))

def getContainer(self, file):
return file.units
57 changes: 0 additions & 57 deletions +tests/+system/UnitTimesIOTest.m
View file
Open in desktop

This file was deleted.

63 changes: 63 additions & 0 deletions +tests/+system/UnitsTableIOTest.m
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
classdef UnitsTableIOTest < tests.system.PyNWBIOTest
methods
function addContainer(~, file)
% Build a Units table with multi-electrode, doubly-ragged waveforms
% using the idiomatic addRaggedArray / addDoublyRaggedArray methods.
% This mirrors PyNWB's Units.add_unit in PyNWBIOTest.py so the two
% round-tripped containers compare equal.
file.units = types.core.Units('description', 'data on spiking units');

% spike_times: a ragged column, one list of times per unit.
file.units.addRaggedArray('spike_times', {[1 2], [3 4 5]});

% waveforms: a doubly-ragged column. data{unit}{spike} is a
% [numElectrodes x numSamples] matrix (one row per electrode's
% waveform). This matches PyNWB add_unit's per-unit 3-D input
% (num_spikes, num_electrodes, num_samples): both store a 2-D
% [num_waveforms, num_samples] dataset with the electrode dimension
% carried by waveforms_index.
% Unit 1 has 2 spikes, unit 2 has 3 spikes; each spike has 2
% electrodes and 3 samples.
waveforms = { ...
{int32([1 2 3; 4 5 6]), int32([7 8 9; 10 11 12])}, ...
{int32([13 14 15; 16 17 18]), int32([19 20 21; 22 23 24]), int32([25 26 27; 28 29 30])} ...
};
file.units.addDoublyRaggedArray('waveforms', waveforms);

% waveform_mean / waveform_sd: fixed 2-D columns [numSamples x numUnits].
file.units.waveform_mean = types.hdmf_common.VectorData( ...
'description', 'the spike waveform mean for each spike unit', ...
'data', [1 4; 2 5; 3 6]);
file.units.waveform_sd = types.hdmf_common.VectorData( ...
'description', 'the spike waveform standard deviation for each spike unit', ...
'data', [7 10; 8 11; 9 12]);

% Match PyNWB's column order and auto-generated descriptions so the
% round-tripped containers compare equal.
file.units.colnames = {'spike_times'; 'waveform_mean'; 'waveform_sd'; 'waveforms'};
file.units.spike_times.description = 'the spike times for each unit in seconds';
file.units.spike_times_index.description = 'Index for VectorData ''spike_times''';
file.units.waveforms.description = ['Individual waveforms for each spike. ' ...
'If the dataset is three-dimensional, the third dimension shows the response ' ...
'from different electrodes that all observe this unit simultaneously. ' ...
'In this case, the `electrodes` column of this Units table should be used to ' ...
'indicate which electrodes are associated with this unit, ' ...
'and the electrodes dimension here should be in the same order as the ' ...
'electrodes referenced in the `electrodes` column of this table.'];
file.units.waveforms_index.description = 'Index for VectorData ''waveforms''';
file.units.waveforms_index_index.description = 'Index for VectorData ''waveforms_index''';

% Optional Units dataset attributes (match PyNWB waveform_rate / resolution).
file.units.spike_times_resolution = 3;
file.units.waveform_mean_sampling_rate = 1;
file.units.waveform_sd_sampling_rate = 1;

% Skip waveforms_sampling_rate because PyNWB does not export it.
file.units.waveforms_sampling_rate = 1;
end

function c = getContainer(~, file)
c = file.units;
end
end
end
Loading
Loading