Add EventTable type

[oruebel]

Fix #285

Add support for EventsTable

• Added new EventTable type
• Added unit tests for the new EventTable type
• Updated NWBFile to create events/ group and added NWBFile::createEventsTable convenience function

Enhance DynamicTable to improve support for use during acquisition

• Refactored DynamicTable so it creates and initializes its ElementIdentifiers id column internally during initialize(). This is to ensure the ElementIdentifiers are always create before we start the recording. Calling DynamicTable::setRowIDs() previously required creation of ElementIdentifiers but may be called later during recording. As part of this change, setRowIDs() now takes only the row ID values and writes them directly to the table's built-in id column instead of accepting a separate ElementIdentifiers object. In practice this aligns the API with the intended usage, since callers should have always used the table's own id dataset. (@oruebel, #302)
• Updated DynamicTable column-name management to be immediate and table-driven. addColumnName() now flushes new column names to the file as columns are added, setColNames() is now restricted to reordering the existing set of columns (it no longer acts as a general overwrite mechanism), and finalize() no longer writes the colnames attribute because column-name updates are persisted when they occur. This ensures that the colnames attribute is always consistent with the actual columns in the table and helps avoid creation of invalid files. (@oruebel, #304)

Other

• Fix bugs in TimestampVectorData and DurationVectorData; e.g, bad namespace and inconsistent ordering of parameters in initialize
• Update Changelog

From #305 : Row-based Recordings

Fix #303

Problem: The current implementation of DynamicTable is mainly designed for the purpose of metadata tables, e.g., ElectrodeTable, where we can usually fill in the whole table before the actual acquisition. With the new EventTable, however, we need enhanced support for acquiring the data in the table as part of the acquisition workflow.

Goal: Update DynamicTable and EventTable to simplify row-based recording and configuration of the table

Changes:

• Support configuration of the columns of a DynamicTable directly via initialize, rather than just incremental viaaddColumn
  • Added Data::DataSpecBase and Data::DataSpec<T> to support runtime configuration of DynamicTable column layouts and to decouple table schema definition from object initialization. Each Data subclass (e.g., VectorData, ElementIdentifiers, TimestampVectorData, DurationVectorData) now exposes a nested DataSpec struct that bundles the dataset configuration, description, and any type-specific parameters needed to create and initialize the column.
  • Added DynamicTable::createDefaultDataSpecs static factory to return the default ordered list of DataSpec objects for a table (currently just the id column). Subclasses (ElectrodesTable, EventsTable, MeaningsTable) override this to append their own required columns, allowing callers to retrieve, inspect, and customize the default column configuration before passing it to initialize().
  • Updated DynamicTable::initialize, ElectrodesTable::initialize, EventsTable::initialize, and MeaningsTable::initialize to accept a std::vector<DataSpecPtr> column-spec list instead of individual rowChunkSize / resolution / flag parameters.
  • Added DynamicTable::addColumn(DataSpecPtr) overload consistent with the new DataSpec-based API used by initialize() and createDefaultDataSpecs().
  • Added DyanmicTable::validateDataSpecs (and DynamicTable::checkRequiredColumnNames helper function) to validate column specifications as part of DyanmicTable::initialize before initialization. Also updated subclassed of DynamicTable (e.g., ElectrodesTable, EventsTable, MeaningsTable) to override validateDataSpecs to provide their specific validation logic.
• Support row-based write of DynamicTable across all columns of the table
  • Added DynamicTable::addRow and DynamicTable::addRows methods to support row-based data insertion into a DynamicTable. Rows are specified as DynamicTable::RowData (an unordered_map from column name to CellValue variant), enabling type-safe, column-keyed writes without requiring callers to manage per-column buffers directly. Row IDs are auto-generated if not provided.
  • Added BaseDataType::BaseDataVariant scalar variant type and BaseDataType::createEmptyVectorVariant helper to support runtime-typed single-cell and buffer operations needed by addRow/addRows. (@cline, @oruebel, #305)
• Update Documentation
  • Added tutorial to demonstrate acquisition workflow for DynamicTable. The tutorial uses EventsTable as an example and shows: 1) row-based appending of data and 2)column-based write and addition of full columns to a table, and 3) adding of a MeaningsTable to a columns
  • Updated the CHANGELOG
• Created Follow-up issues
  • Appending rows for reference columns is not yet working. See Support appending rows and DataSpec configuration for reference columns in DynamicTable #313
  • Update schema_to_aqnwb.py utility to follow the new structure for DynamicTable. See Update schema_to_aqnwb.py utility to follow the new structure for DynamicTable #314

[cboulay]

Hi @oruebel and @rly , we are very much looking forward to this and the associated Events table changes. Let me know if there's anything we can do to help move this forward.

[oruebel]

Hi @oruebel and @rly , we are very much looking forward to this and the associated Events table changes. Let me know if there's anything we can do to help move this forward.

@cboulay great to hear. My current goal is to have this ready for a 0.4 release early/mid August. The main parts for EventTable are functional, but I want to spent a bit more time testing and refining things.

All this is implemented in a sequence of PRs that build on each other. I.e., if you want to give it a try you will only need to checkout the rowbased_table_acquisition branch from this PR #305 , which has a first, full working implementation of the new EventsTable, MeaningsTable, and various other required new types. I also added a tutorial on how to record event-based data as part of the PR in events.dox and the source-code for the tutorial is in testEventsWorkflowExamples.cpp. For an overview of the changes to AqNWB please see the CHANGLOG.md on the PR.

In terms of moving this forward, it would be really helpful if you could try it out and let me know if there are any issues. As usual, feel free to create separate issues, add comments on the PRs directly, or by creating follow-up PR's.

Side Note: Another reason I am holding of with the release is because I would like hdmf-dev/hdmf-common-schema#97 to be released first, since that is changing the schema for MeaningsTable, and I would like to avoid releasing this in AqNWB and then having to immediately update it again.

[oruebel]

@cboulay I've merge all those PRs into #300 so please use the schema_2_10 for your tests