feat: Add HarmActionsEval for evaluation of action-level safety in AI agents

[prane-eth]

Background

AI agents have a growing adoption across the industry, including critical applications. AI agents that have access to tools can currently call tools directly with no centralized validation layer that inspects these calls before execution, allowing harmful or disallowed tool calls to be executed without oversight. Existing agent benchmarks evaluate the safety of final responses instead of actions.

HarmActionsEval benchmark evaluates actions. It found that 80% of the LLMs tested executed actions at the first attempt for over 95% of the harmful prompts.

Related work: https://github.com/Pro-GenAI/Agent-Action-Guard. It received 962 downloads on PyPI, and 247 clones on GitHub in the first week.

Changes

I integrated HarmActionsEval benchmark.

All the test cases passed using: pytest tests/probes/test_probes_harmactionseval.py tests/detectors/test_detectors_harmactionseval.py
and pytest tests/detectors/test_detectors.py::test_detector_detect[detectors.harmactionseval.ToolCallMatch].

Example command used:

garak --target_type openai \
	--target_name "$OPENAI_MODEL" \
	--probes harmactionseval.HarmActionsEval \
	--generations 1

Verification

List the steps needed to make sure this thing works

• Supporting configuration such as generator configuration file. -- Not applicable. I added the dataset.
• garak -t <target_type> -n <model_name>
• Run the tests and ensure they pass python -m pytest tests/
• Verify the thing does what it should
• Verify the thing does not do what it should not
• Document the thing and how it works (Example)

[github-actions]

DCO Assistant Lite bot All contributors have signed the DCO ✍️ ✅

[prane-eth]

I have read the DCO Document and I hereby sign the DCO

[prane-eth]

recheck