Skip to content

Add AMD APU (MI300A) support with monitoring energy, thermals, and power - #591

Open
slabasan wants to merge 18 commits into
devfrom
AMD_APU
Open

Add AMD APU (MI300A) support with monitoring energy, thermals, and power#591
slabasan wants to merge 18 commits into
devfrom
AMD_APU

Conversation

@slabasan

@slabasan slabasan commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Latest TODO

  • check this in cpx, tpx and spx to ensure it is correct and matches amd-smi CLI
  • Make sure power-node-watts is correct
  • Check thermal and energy APIs too similarly. Energy also has a ‘node-joules’ I think and we need to ensure that is correct across different modes too

Update 7/17 end of hackathon:

  • Get JSON energy and thermals not working with regular and sudo user (HSMP related or claude issue?) ANything going through RSMI may not be privileged, double check this. Dev energy count does not need sudo per documentation.
    --> On tioga, you were able to collect energy with sudo using RSMI API (only CPU or both CPU and GPU)
  • Get JSON power works with regular user
$ ./variorum-get-thermals-json-example 
tuolumne1048:/g/g19/brink2/git_root_pub/variorum/src/variorum/AMD_APU/amd_apu_power_features.c:get_thermals_json():613: _ERROR_VARIORUM_PLATFORM_ENV: RSMI temp API was not successful
tuolumne1048:/g/g19/brink2/git_root_pub/variorum/src/variorum/AMD_APU/amd_apu_power_features.c:get_thermals_json():613: _ERROR_VARIORUM_PLATFORM_ENV: RSMI temp API was not successful
tuolumne1048:/g/g19/brink2/git_root_pub/variorum/src/variorum/AMD_APU/amd_apu_power_features.c:get_thermals_json():613: _ERROR_VARIORUM_PLATFORM_ENV: RSMI temp API was not successful
tuolumne1048:/g/g19/brink2/git_root_pub/variorum/src/variorum/AMD_APU/amd_apu_power_features.c:get_thermals_json():613: _ERROR_VARIORUM_PLATFORM_ENV: RSMI temp API was not su

This commit adds comprehensive support for AMD APU architectures, specifically targeting the MI300A with energy monitoring capabilities via the ROCm SMI rsmi_dev_energy_count_get() API.

Major changes:

  • Created new AMD_APU directory structure under src/variorum/AMD_APU/

  • config_amd_apu.{c,h}: APU detection and function pointer registration

  • amd_apu_power_features.{c,h}: Layer 1 ROCm-SMI API wrappers including energy monitoring (get_energy_data, get_energy_json)

  • mi300aAPU.{c,h}: Layer 2 MI300A-specific wrappers

  • CMakeLists.txt: Build configuration for AMD_APU

  • Updated config_architecture.{c,h}:

  • Added amd_apu_arch_e enum with AMD_MI300A = 1

  • Added P_AMD_APU_IDX to supported_platforms_e

  • Integrated APU detection and function pointer registration

  • Increased MAX_PLATFORMS from 2 to 3

  • Updated build system:

  • src/CMakeLists.txt: Added VARIORUM_WITH_AMD_APU option

  • src/variorum/CMakeLists.txt: Added AMD_APU subdirectory and build rules

  • src/CMake/Setup3rdParty.cmake: Include FindROCM.cmake for APU support

Energy monitoring features:

  • variorum_get_energy_json(): Returns cumulative energy counters in microjoules
  • variorum_print_energy(): Prints energy data in standard Variorum format
  • Uses ROCm SMI rsmi_dev_energy_count_get() API
  • Gracefully handles unsupported devices (RSMI_STATUS_NOT_SUPPORTED)

The implementation follows Variorum's three-layer architecture pattern and mirrors the existing AMD_GPU structure for consistency.

Description

Please provide a short summary of the change and tag any issues being fixed. Please also include relevant motivation
and context. List any dependencies that may be required for this change.

Fixes #

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature/architecture support (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Build/CI update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Please provide hardware architecture specs and
instructions so we can reproduce.

  • Test A: Hardware architecture, machine name, example/test run
  • Test B: Hardware architecture, machine name, example/test run
  • ...

Checklist:

  • I have run ./scripts/check-code-format.sh and confirm my code code follows the style guidelines of variorum
  • I have added comments in my code
  • My changes generate no new warnings (build with -DENABLE_WARNINGS=ON)
  • New and existing unit tests pass with my changes

Thank you for taking the time to contribute to Variorum!

slabasan and others added 6 commits July 17, 2026 13:45
This commit adds comprehensive support for AMD APU architectures,
specifically targeting the MI300A with energy monitoring capabilities
via the ROCm SMI rsmi_dev_energy_count_get() API.

Major changes:
- Created new AMD_APU directory structure under src/variorum/AMD_APU/
  - config_amd_apu.{c,h}: APU detection and function pointer registration
  - amd_apu_power_features.{c,h}: Layer 1 ROCm-SMI API wrappers including
    energy monitoring (get_energy_data, get_energy_json)
  - mi300aAPU.{c,h}: Layer 2 MI300A-specific wrappers
  - CMakeLists.txt: Build configuration for AMD_APU

- Updated config_architecture.{c,h}:
  - Added amd_apu_arch_e enum with AMD_MI300A = 1
  - Added P_AMD_APU_IDX to supported_platforms_e
  - Integrated APU detection and function pointer registration
  - Increased MAX_PLATFORMS from 2 to 3

- Updated build system:
  - src/CMakeLists.txt: Added VARIORUM_WITH_AMD_APU option
  - src/variorum/CMakeLists.txt: Added AMD_APU subdirectory and build rules
  - src/CMake/Setup3rdParty.cmake: Include FindROCM.cmake for APU support

Energy monitoring features:
- variorum_get_energy_json(): Returns cumulative energy counters in microjoules
- variorum_print_energy(): Prints energy data in standard Variorum format
- Uses ROCm SMI rsmi_dev_energy_count_get() API
- Gracefully handles unsupported devices (RSMI_STATUS_NOT_SUPPORTED)

The implementation follows Variorum's three-layer architecture pattern
and mirrors the existing AMD_GPU structure for consistency.

Co-Authored-By: Claude <noreply@anthropic.com>
1) Update AMD APU to use ROCm 7 APIs exclusively

Simplified power monitoring implementation to use only ROCm 7+ APIs,
removing legacy ROCm 5 compatibility code.

Changes:
- Removed ROCM_VERSION_MAJOR preprocessor checks
- Removed rocm-core/rocm_version.h include (not needed)
- Use rsmi_dev_power_get() exclusively (recommended for ROCm 7+)
  - This API automatically tries current power first, falls back to average
  - Returns RSMI_POWER_TYPE indicating which type was retrieved
  - Provides best compatibility across APU generations
- Removed deprecated rsmi_dev_power_ave_get() calls

Benefits:
- Cleaner, simpler code without version conditionals
- Uses recommended ROCm 7 APIs
- Better forward compatibility
- Automatically adapts to device capabilities

The rsmi_dev_power_get() API automatically handles the differences
between devices that support current vs average power, making it
the preferred choice for ROCm 7+.

2) Implement complete power and thermal monitoring APIs

    Added full implementations for power and thermal monitoring functions
    in AMD APU support, mirroring the AMD_GPU patterns.

    Power Monitoring:
    - get_power_data(): Print power consumption per device
      - Uses rsmi_dev_power_get() for ROCm 6+ or rsmi_dev_power_ave_get() for ROCm 5
      - Reports power in Watts (converted from microwatts)
      - Handles RSMI_STATUS_NOT_SUPPORTED gracefully

    - get_json_power_data(): JSON format power data
  - Per-socket and per-device power in watts
  - Aggregates total node power (power_node_watts)
  - Compatible with multi-socket systems

Thermal Monitoring:
- get_thermals_data(): Print temperature per device
  - Uses rsmi_dev_temp_metric_get() with RSMI_TEMP_TYPE_EDGE
  - Reports edge temperature (die edge) in Celsius
  - Converts from millidegrees to Celsius

- get_thermals_json(): JSON format thermal data
  - Per-device temperatures in Celsius
  - Organized by socket

Output Format Examples:
_AMD_APU_POWER_USAGE Host Socket DeviceID Power Timestamp_sec
_AMD_APU_TEMPERATURE Host Socket DeviceID Temperature Timestamp_sec

JSON: {"socket_0": {"power_apu_watts": {"APU_0": 150.5}, "APU": {"temp_celsius_apu_0": 65.2}}}

All functions include proper error handling, ROCm version compatibility,
and follow Variorum's established formatting patterns.

3) Fix: Correct rsmi_dev_energy_count_get() API signature

Updated to use the correct 4-parameter API signature for rsmi_dev_energy_count_get():
- Parameter 1: uint32_t dv_ind (device index)
- Parameter 2: uint64_t *power (energy counter in microjoules)
- Parameter 3: float *counter_resolution (resolution in microjoules)
- Parameter 4: uint64_t *timestamp (timestamp in nanoseconds)

Changes:
- Fixed get_energy_data() to use all 4 parameters
- Fixed get_energy_json() to use all 4 parameters
- Updated JSON output to include counter_resolution
- Changed timestamp from float to uint64_t (nanoseconds)
- Updated JSON field names to reflect units (_ns, _uJ)
- Added comments documenting each parameter

Reference: ROCm SMI Lib documentation v6.0.0
https://rocm.docs.amd.com/projects/rocm_smi_lib/en/docs-6.0.0/.doxygen/docBin/html/group__PowerQuer.html#ga1f61b24edaca83a0e395a34c466fcf86

4) Fix: Add VARIORUM_WITH_AMD_APU to variorum_config.h.in

The P_AMD_APU_IDX constant was undefined during compilation because
the VARIORUM_WITH_AMD_APU preprocessor macro wasn't being defined.
This adds the CMake configuration variable to variorum_config.h.in
so it gets properly defined when AMD_APU support is enabled.

Co-Authored-By: Claude <noreply@anthropic.com>
more than 2 not possible
@slabasan slabasan changed the title Add AMD APU (MI300A) support with energy monitoring Add AMD APU (MI300A) support with monitoring energy, thermals, and power Jul 17, 2026
@tpatki

tpatki commented Jul 18, 2026

Copy link
Copy Markdown
Member

Spent a few hours debugging the segfault with amd_smi APIs. Summary from claude below.

(Caveat: Not sure I trust claude here, but I think it is heading in the right direction based on my independent debugging of this as well. Worth looking into a local build of hwloc and testing).

================

Problem:
variorum-get-power-json-example and variorum-get-energy-json-example segfault on Tuolumne (MI300A, ROCm 7.2.0) with SIGSEGV inside amd::smi::GpuMetricsBase_v17_t::~GpuMetricsBase_v17_t(), invoked via _dl_init/call_init from /usr/lib64/hwloc/librocm_smi64.so.7. That file has no dependency on libamd_smi.so, it's a full separate copy. Because Variorum links /opt/rocm-7.2.0/lib/libamd_smi.so.26 directly and links hwloc (for unrelated CPU/socket topology detection) which transitively loads that bundled copy, both libraries define the same mangled C++ symbols (e.g. GpuMetricsBase_v17_t's destructor) with different internal layouts in one process. This is a One Definition Rule (ODR) violation — C++ requires that any given symbol have exactly one definition throughout a program, but the dynamic linker doesn't enforce or even check this across shared libraries, so it silently resolves calls to whichever definition it happens to load first, meaning code built against one layout can end up running against the other's, corrupting memory on destruction. A standalone AMD SMI test program that doesn't link hwloc at all doesn't hit this, since it never has two conflicting copies of that code loaded simultaneously.

Fix:
Two options, both removing the conflicting copy from the process:

  1. Build hwloc with --disable-rsmi and point Variorum's HWLOC_DIR at it. Confirmed safe: Variorum's own APU discovery (amdsmi_get_socket_handles/amdsmi_get_processor_handles in config_amd_apu.c/amd_apu_power_features.c) never goes through hwloc — hwloc is only used elsewhere in the codebase for CPU socket/core/thread topology (HWLOC_OBJ_SOCKET/CORE/PU/NUMANODE), with zero calls to hwloc_rsmi_* or any GPU-identification API.
  2. Build hwloc with --with-rocm=/opt/rocm-7.2.0 instead, so its rsmi backend dynamically links the same libamd_smi.so.26 Variorum uses, rather than disabling GPU discovery in hwloc entirely. This keeps hwloc's GPU-topology feature working, but ties the hwloc build to that specific ROCm version going forward — switching ROCm modules later without rebuilding hwloc in lockstep reintroduces the same class of bug with a different version pair.

Option 1 is simpler and more robust long-term since Variorum doesn't use hwloc's GPU features at all; option 2 is worth it only if something does need hwloc's GPU-locality API later.
Also worth flagging to LC/TOSS as a hwloc packaging issue, since any application linking both a ROCm module's libamd_smi.so and the system hwloc will hit this, not just Variorum.

@tpatki

tpatki commented Jul 21, 2026

Copy link
Copy Markdown
Member

Update from 7/21.

  • Segfault fixed with a --disable-rsmi build of hwloc-2.13.0 under /usr/workspace/variorum
  • Power and energy is getting reported
  • Thermals not working, looks like a parsing bug. Uses the amdsmi_get_temp_metric API Fixed.

@slabasan

Copy link
Copy Markdown
Collaborator Author
  • Update JSON fields to stdout
$ ./variorum-get-energy-json-example 
{
    "tuolumne1041": {
        "timestamp": 1784747836023678,
        "num_gpus_per_socket": 1,
        "num_apus_per_node": 4,
        "socket_0": {
            "energy_apu_joules": {
                "APU_0": 399758137.22862118
            }
        },
        "energy_node_joules": 1459563955.1508536,
        "socket_1": {
            "energy_apu_joules": {
                "APU_1": 347587168.35872364
            }
        },
        "socket_2": {
            "energy_apu_joules": {
                "APU_2": 365700637.52314699
            }
        },
        "socket_3": {
            "energy_apu_joules": {
                "APU_3": 346518012.04036188
            }
        }
    }
}
$ ./variorum-get-thermals-json-example 
{
    "tuolumne1041": {
        "timestamp": 1784747878530999,
        "num_gpus_per_socket": 1,
        "num_apus_per_node": 4,
        "socket_0": {
            "APU": {
                "temp_celsius_apu_0_junction": 47.0,
                "temp_celsius_apu_0_hbm": 42.0
            }
        },
        "socket_1": {
            "APU": {
                "temp_celsius_apu_1_junction": 44.0,
                "temp_celsius_apu_1_hbm": 43.0
            }
        },
        "socket_2": {
            "APU": {
                "temp_celsius_apu_2_junction": 44.0,
                "temp_celsius_apu_2_hbm": 39.0
            }
        },
        "socket_3": {
            "APU": {
                "temp_celsius_apu_3_junction": 44.0,
                "temp_celsius_apu_3_hbm": 40.0
            }
        }
    }
}
$ ./variorum-get-power-json-example 
{
    "tuolumne1041": {
        "timestamp": 1784747893078522,
        "num_gpus_per_socket": 1,
        "num_apus_per_node": 4,
        "socket_0": {
            "power_apu_watts": {
                "APU_0": 132.0
            }
        },
        "socket_1": {
            "power_apu_watts": {
                "APU_1": 77.0
            }
        },
        "socket_2": {
            "power_apu_watts": {
                "APU_2": 123.0
            }
        },
        "socket_3": {
            "power_apu_watts": {
                "APU_3": 116.0
            }
        },
        "power_node_watts": 448.0
    }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants