Add MultiCameraPoseEstimation and CalibratedCamera neurodata types

[alessandratrapani]

Add CalibratedCamera and MultiCameraPoseEstimation neurodata types

Motivation

PoseEstimation is designed for single-camera, pixel-space pose data (e.g. DeepLabCut).
It does not model multi-camera 3D setups well: there is no place to store calibration
parameters, the relationship between cameras and their videos is implicit (order-based
string paths), and 3D world-space estimates have no structural separation from 2D pixel
estimates.

This PR adds two new neurodata types to handle multi-camera 3D pose estimation
(DANNCE, Anipose, etc.), and extends PoseEstimation to serve as the per-camera
building block for multi-camera setups.

Update: an earlier version of this PR introduced two additional types, CameraView
and CameraCalibration. Following review feedback, both have been removed:

• CameraView duplicated PoseEstimation almost entirely (device link, source video,
  2D PoseEstimationSeries, scorer, source software). Rather than ship two overlapping
  per-camera containers that would drift apart over time, PoseEstimation itself is now
  scoped to a single camera view and reused directly inside MultiCameraPoseEstimation.
• CameraCalibration stored calibration as parallel, row-indexed arrays alongside a
  separate list of Device links, which required duplicating the whole rig (and keeping
  the row order in sync) for every subject in a multi-subject session (e.g. sDANNCE).
  Calibration now lives on the camera itself via the new CalibratedCamera type, so the
  camera + calibration are added once to the NWBFile and shared by reference.

---

New types

CalibratedCamera (Device)

A Device extended with intrinsic and extrinsic calibration parameters for that single
camera. Because it is a Device, it is added once to the NWBFile (e.g. in
general/devices) and can be linked to by reference from multiple PoseEstimation/
MultiCameraPoseEstimation objects — e.g. one per subject in a multi-subject recording
session — so the camera rig and its calibration are never duplicated, and there is no
row-order matching to maintain.

Field	Required	Shape	Description
intrinsic_matrix	yes	(3, 3)	Camera matrix K
rotation_matrix	no	(3, 3)	Rotation from world to this camera's frame
translation_vector	no	(3,)	Translation from world to this camera's frame
distortion_coefficients	no	(N,)	Lens distortion coefficients

PoseEstimation (NWBDataInterface) — updated

Now explicitly represents pose estimates from a single camera view, computed with
one tool/algorithm. This is the same type used for standalone single-camera files; when
used as a child of MultiCameraPoseEstimation, one PoseEstimation is added per camera.

Field	Required	Description
device	no	Link to a Device (ideally a CalibratedCamera) already in the NWBFile
source_video / labeled_video	no	Links to ImageSeries objects stored in acquisition
pose_estimation_series	no	PoseEstimationSeries children (2D pixel-space, or 3D if used standalone)
scorer, source_software, source_software_version	no	Metadata about the estimation procedure
skeleton	no	Link to a Skeleton in a Skeletons object
devices (list)	no	Deprecated. Use device instead; passing more than one device now raises an error

MultiCameraPoseEstimation (NWBDataInterface)

Top-level container for 3D multi-camera pose estimation. Holds 3D world-space keypoints
and one PoseEstimation per camera view.

Field	Required	Description
pose_estimation_series	no	3D PoseEstimationSeries children in world-space coordinates
pose_estimations	no	PoseEstimation children, one per camera view
skeleton	no	Link to a Skeleton in a Skeletons object
description	no	Description of the pose estimation procedure
scorer	no	Name of the scorer / algorithm
source_software	no	Name of the software tool
source_software_version	no	Version string of the software tool

---

Design decisions

• PoseEstimation reused instead of a new CameraView type: avoids two structurally
  identical containers drifting apart over time. It also means anything that already
  reads/writes PoseEstimation (DeepLabCut/Anipose-style per-camera 2D outputs) composes
  directly into a multi-camera file without a new type to learn.
• Calibration lives on the camera (CalibratedCamera), not in a separate rig object:
  cameras are Devices, so they're added once to the NWBFile and referenced by link.
  This removes the row-order coupling between a calibration object and its device list,
  and avoids duplicating the rig per subject in multi-subject sessions.
• PoseEstimation.device is now singular: PoseEstimation represents one camera
  view. The previous devices (list) argument is deprecated (warns once, still readable)
  and raises if more than one device is passed — multi-camera setups should add one
  PoseEstimation per camera to a MultiCameraPoseEstimation instead.
• labeled_video omitted from the per-camera 2D estimates: labeled videos (pose
  overlays) are derived visualization artifacts, not raw data, consistent with the
  original design decision.
• Links to ImageSeries in acquisition: unchanged — PoseEstimation.source_video
  follows the pattern introduced in PR Add source_video link field to PoseEstimation #56, with the ImageSeries living in acquisition
  and linked from the pose container.

---

Files changed

File	Change
spec/ndx-pose.extensions.yaml	Added CalibratedCamera; updated PoseEstimation (singular device link, deprecated multi-video fields); updated MultiCameraPoseEstimation to contain PoseEstimation children instead of CameraView/CameraCalibration
src/pynwb/ndx_pose/pose.py	Added CalibratedCamera; updated PoseEstimation device deprecation logic; rewrote MultiCameraPoseEstimation
src/pynwb/ndx_pose/io/pose.py	Removed unused imports; kept MultiCameraPoseEstimationMap for source_software_version IO mapping
src/pynwb/ndx_pose/__init__.py	Exported CalibratedCamera instead of CameraCalibration/CameraView
src/pynwb/ndx_pose/testing/mock/pose.py	Added mock_CalibratedCamera; updated mock_PoseEstimation (singular device, add_to_nwbfile flag); rewrote mock_MultiCameraPoseEstimation to build per-camera PoseEstimation children
examples/write_multicamera_pose_estimates.py	Rewritten for the new type structure
src/pynwb/tests/unit/test_pose.py	Added TestCalibratedCameraConstructor; added PoseEstimation deprecation tests; rewrote TestMultiCameraPoseEstimationConstructor
src/pynwb/tests/integration/hdf5/test_pose.py	Added TestCalibratedCameraRoundtrip; rewrote TestMultiCameraPoseEstimationRoundtrip
README.md, CHANGELOG.md	Updated diagrams, type descriptions, and changelog entry

Previous proposal

### New types

CameraCalibration (NWBDataInterface)

Stores intrinsic and extrinsic calibration parameters for a set of cameras. Each row of
every dataset corresponds to one camera, in the same order as the linked Device objects.

Field	Required	Shape	Description
intrinsic_matrix	yes	(n_cameras, 3, 3)	Camera matrix K
rotation_matrix	no	(n_cameras, 3, 3)	Rotation from world to camera frame
translation_vector	no	(n_cameras, 3)	Translation from world to camera frame
distortion_coefficients	no	(n_cameras, N)	Lens distortion coefficients
devices	no	—	Links to Device objects, one per camera (must already be in the NWBFile)

CameraView (NWBDataInterface)

Represents a single camera's contribution to the pose estimation pipeline. Groups a
camera device, its raw video, and optionally its per-camera 2D keypoint estimates in
pixel space.

Field	Required	Description
device	yes	Link to a Device already in the NWBFile
source_video	no	Link to an ImageSeries stored in acquisition
pose_estimation_series	no	2D PoseEstimationSeries children (pixel-space estimates for this camera)

MultiCameraPoseEstimation (NWBDataInterface)

Top-level container for 3D multi-camera pose estimation. Holds 3D world-space keypoints,
one CameraView per camera, and optionally a calibration object and skeleton link.

Field	Required	Description
pose_estimation_series	no	3D PoseEstimationSeries children in world-space coordinates
camera_views	no	CameraView children, one per camera
calibration	no	CameraCalibration child
skeleton	no	Link to a Skeleton in a Skeletons object
description	no	Description of the pose estimation procedure
scorer	no	Name of the scorer / algorithm
source_software	no	Name of the software tool
source_software_version	no	Version string of the software tool

---

Design decisions

• labeled_video omitted: labeled videos (pose overlays) are derived visualization
  artifacts, not raw data. They do not belong in the primary data storage path and are
  therefore excluded from CameraView.
• CameraView as a named container per camera: each camera has its own named group,
  making the device ↔ video ↔ 2D-estimates relationship explicit and structural rather
  than relying on implicit ordering of parallel lists.
• Links to ImageSeries in acquisition: CameraView.source_video follows the same
  pattern introduced in PR Add source_video link field to PoseEstimation #56 for PoseEstimation.source_video — the ImageSeries
  lives in acquisition and is linked from the pose container.
• Calibration row order matches device link order: CameraCalibration datasets are
  row-indexed to match the order of the linked Device list, making the
  camera-to-parameters mapping unambiguous.

---

Files changed

File	Change
spec/ndx-pose.extensions.yaml	Added CameraCalibration, CameraView, MultiCameraPoseEstimation type definitions
spec/ndx-pose.namespace.yaml	Version bumped to 0.2.2
src/pynwb/ndx_pose/pose.py	Implemented all three classes
src/pynwb/ndx_pose/io/pose.py	Added MultiCameraPoseEstimationMap for source_software_version IO mapping
src/pynwb/ndx_pose/__init__.py	Exported CameraCalibration, CameraView
src/pynwb/ndx_pose/testing/mock/pose.py	Added mock_CameraCalibration, mock_CameraView; rewrote mock_MultiCameraPoseEstimation
src/pynwb/tests/unit/test_pose.py	Added TestCameraCalibrationConstructor, TestCameraViewConstructor; rewrote TestMultiCameraPoseEstimationConstructor
src/pynwb/tests/integration/hdf5/test_pose.py	Added TestCameraCalibrationRoundtrip, TestCameraViewRoundtrip; rewrote TestMultiCameraPoseEstimationRoundtrip and TestMultiCameraPoseEstimationRoundtripPyNWB

[rly]

Thanks @alessandratrapani . #56 is merged and there are some conflicts

[h-mayorquin]

Hey, I finally took a look at this. We should fix the tests and conflicts but right now this is what I think. There are basically two ideas.

I think that CameraView is the PoseEstimation container specialized for one camera and 2D. In that light I suggest: let's modify PoseEstimation so we really make it single camera and we can reuse it here. The risk otherwise is that we ship a second per-camera container that overlaps PoseEstimation by about ninety percent and the two drift apart over time. Let me expand, for example for this case, the current CameraView should also have things like scorer and source_software because Anipose reuses that data from DeepLabCut by default and that deserves a place. Instead of adding a new type that almost overlaps by 90% with the current PoseEstimation I say let's fix PoseEstimation and use only one. What do you think @rly?

In the current proposal the MultiCameraPoseEstimation holds the full rig. This is problematic for two-subject scenarios like sDANNCE which would make us duplicate the rig. I propose a normalized version here where we extend Device to CalibratedCamera that contains the extrinsic (localization in lab space) and intrinsic (calibration) attributes and is only linked to the PoseEstimation as a device. When the coordinates are available we link a CalibratedCamera, otherwise a simple Device. This solves the duplication: the cameras live once and are shared by reference, so a second subject (sDANNCE) or another run costs nothing extra, and the calibration travels with the camera it describes, so there is no fragile row-order matching between views and calibration.

[alessandratrapani]

@rly I implemented the suggestion from @h-mayorquin, and it is now ready for review.

[h-mayorquin]

The tests are still failing, can you take a look at that before I take a final look?

[rly]

I'm reviewing it and fixing up the tests (pynwb 4.0 breaking change) now

[rly]

Overall looks good. I made some minor updates: bumping the version, adding your name as a contributor @alessandratrapani , fixing the changelog, and fixing some pynwb 4.0 compatibility issues. And I made some comments above. The failing spec validation relates to the dims key I commented on. @h-mayorquin could you please review this PR? Thanks!

@alessandratrapani It looks like you modified and reformatted the YAML directly, which is totally fine. Once it's final, I'll port those changes into create_extension_spec.py and generate the YAML from that in this PR, because I like to use the python API to check for issues and deterministically generate and format the YAML spec.

[alessandratrapani]

@rly Thanks for the first round of review!

[h-mayorquin]

What is the policy for reading files with multiple cameras from previous versions?

[h-mayorquin]

Thanks guys, looks good to me. Three things that I spotted:

1. source_software_version bug, fixed in #63. Both types store the version as an attribute on the source_software dataset, so setting it without source_software silently drops it on roundtrip. Not caused by this PR (PoseEstimation on main does the same), and #63 adds the guard there. Once #63 merges and this branch picks it up, MultiCameraPoseEstimation should get the same check, since it inherits the layout.

2. Backwards compatibility. A 0.2.1 file with a PoseEstimation linked to two cameras fails to read on this branch (incorrect type for 'device' (got 'list', expected 'Device')); it reads fine on main. hdmf resolves links by target type, so it collects both Device links and passes them as a list, which the new singular device docval rejects before the deprecation path can run. Single-camera files still read, but devices was dropped from __nwbfields__, so pe.devices now raises AttributeError.

Worth noting why CI missed this: src/pynwb/tests/back_compat/ has no fixture with any device, so the device-link path this PR rewrites is untested. A 0.2.x two-camera fixture would have caught it and would pin down the policy. I am fine if we don't want to add support for that though, should we?

3. Example doesn't run. examples/write_multicamera_pose_estimates.py fails on the pynwb 4.0 num_samples requirement that #62 already handled everywhere else; this example was written separately and missed it. Adding num_samples=n_frames to the ImageSeries in step 3 fixes it. CI stays green because the examples aren't run in CI.

[alessandratrapani]

Thanks for the thorough review!

1. source_software_version bug, fixed in #63. Both types store the version as an attribute on the source_software dataset, so setting it without source_software silently drops it on roundtrip. Not caused by this PR (PoseEstimation on main does the same), and #63 adds the guard there. Once #63 merges and this branch picks it up, MultiCameraPoseEstimation should get the same check, since it inherits the layout.

Added the same check PoseEstimation got in #63: MultiCameraPoseEstimation.__init__ now raises if
source_software_version is set without source_software, for the same reason (it's stored as an
attribute on the source_software dataset).

2. Backwards compatibility. A 0.2.1 file with a PoseEstimation linked to two cameras fails to read on this branch (incorrect type for 'device' (got 'list', expected 'Device')); it reads fine on main. hdmf resolves links by target type, so it collects both Device links and passes them as a list, which the new singular device docval rejects before the deprecation path can run. Single-camera files still read, but devices was dropped from __nwbfields__, so pe.devices now raises AttributeError.

I've made device accept (Device, list, tuple) and, when it arrives as a list, route it through the
same logic as the deprecated devices argument:

• One device : resolves to .device transparently, no error.
• Two or more devices: raises a clear, actionable ValueError explaining that multi-camera
  PoseEstimation isn't supported anymore and pointing at MultiCameraPoseEstimation, instead of HDMF's
  generic "incorrect type" message.

I also added a deprecated PoseEstimation.devices property (returning [device] or [], with a
DeprecationWarning, mirroring the existing .nodes/.edges pattern) so pe.devices on an old
single-camera file warns instead of raising AttributeError.

3. Example doesn't run. examples/write_multicamera_pose_estimates.py fails on the pynwb 4.0 num_samples requirement that #62 already handled everywhere else; this example was written separately and missed it. Adding num_samples=n_frames to the ImageSeries in step 3 fixes it. CI stays green because the examples aren't run in CI.

I added num_samples=n_frames to the ImageSeries in step 3 of write_multicamera_pose_estimates.py, matching what #62 did elsewhere. Ran all three example scripts locally against pynwb 4.0.0 / hdmf 6.1.0 and they pass.

[h-mayorquin]

LGTM.

If the decision is not read multi-device files created with previous versions I think we should note that in the changelog but all my feedback has been addressed. I think what remains rests on @rly to decide. Thanks.