Repository navigation
Register the Security 1.1 practice difficulty and discrimination - #1709
Merged
bkorycki merged 1 commit intoOct 28, 2026
Conversation
Add the practice form's frozen difficulty-and-discrimination file beside the official one and register it in SECURITY_1_1_SCORING_FILES under the practice prompt set, pinned by its SHA-256. Register the practice prompt set airr_practice_security_1.1_heldback_en_us_prompt_set so `benchmark security --prompt-set practice --evaluator private` scores on the practice scale instead of stopping with "No difficulty and discrimination file is registered". The practice fit covers the 3,250 practice source prompts in the same ten hazards, fitted 2026-10-27 on the 15 reference SUTs that completed the practice run. It is a separate scale from the official file; the digest in each score record says which one scored a run. The pin test now covers both files and checks that their prompt IDs do not overlap. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
bkorycki
merged commit Oct 28, 2026
81f70e9
into
mlcommons:integrate-vio-prob-scoring
3 checks passed
bkorycki
added a commit
that referenced
this pull request
Oct 28, 2026
* Score paired naive and jailbreak responses from continuous probabilities * Report mean violation probabilities and final scores with null grades * Keep missing-response checks internal and trim scoring output * Register canonical reference calibration for Jailbreak 1.2 * Score Security 1.1 from evaluator probabilities inside external_scoring Security 1.1 scores each SUT from the evaluator's violation probabilities. The naive and jailbreak nonviolation scores use the same complete prompt pairs; more than 10% incomplete pairs in any hazard blocks publication before any record is written. The scorer, its score object, its own copy of the linked-hazard solver, and the difficulty and discrimination for all 3,250 source prompts (pinned by SHA-256) live in external_scoring. Outside it: the Security 1.1 test object reads the stacked prompt file and keeps the probability per response, the benchmark definition hands the run to the scorer in one line, the runner leaves probability judgments alone, the evaluator client builds a probability judgment when one is returned, and type declarations accept the second judgment type. No CLI changes. Security 1.2 is removed; safety_v_1_2_scoring.py is unchanged from main. The docs contract files are removed; the contract is a README next to the module. The parameters file is committed at the owner's direction. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * base annotation type * delete unused method * fix modelgauge test object * update consistency checker * fix column name * lint * try to make tests work * lint again * Register the Security 1.1 practice difficulty and discrimination (#1709) Add the practice form's frozen difficulty-and-discrimination file beside the official one and register it in SECURITY_1_1_SCORING_FILES under the practice prompt set, pinned by its SHA-256. Register the practice prompt set airr_practice_security_1.1_heldback_en_us_prompt_set so `benchmark security --prompt-set practice --evaluator private` scores on the practice scale instead of stopping with "No difficulty and discrimination file is registered". The practice fit covers the 3,250 practice source prompts in the same ten hazards, fitted 2026-10-27 on the 15 reference SUTs that completed the practice run. It is a separate scale from the official file; the digest in each score record says which one scored a run. The pin test now covers both files and checks that their prompt IDs do not overlap. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> * Publish Security 1.1 per-hazard scores in the Safety record shape (#1711) * Publish Security 1.1 per-hazard scores in the Safety record shape The record now has a top-level score and one entry per hazard, with the naive and jailbreak blocks side by side at each level. Every block names its percentile reference_percentile_score. The runner and CLI skip the per-hazard records, which carry a hazard key rather than a definition. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Use HazardScore objects for paired Security hazard scores --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Riki Conrey <riki.conrey@rikiconrey.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Riki Conrey <riki.conrey@gmail.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds the practice form's difficulty-and-discrimination file beside the official one, inside
external_scoring, and registers it for--prompt-set practice. These are the parameters the scoring needs to combine the prompts into percentile scores. These are no longer on the prompt files because they are refit regularly and science is not able to swap those files.What changed
src/modelbench/external_scoring/security_v_1_1_practice_difficulty_and_discrimination.json: 3,250 practice source prompts in the same ten hazards, pinned by SHA-256f02ce7f2….SECURITY_1_1_SCORING_FILESgains apracticeentry.score_runalready picks the file by the benchmark's prompt set and locale, so nothing outside the box changes.SECURITY_1_1_JAILBREAK_PROMPT_SETSgainspractice, namingairr_practice_security_1.1_heldback_en_us_prompt_set, the file on modellab.Running
No prompt text, response text, or generated score records are added. The file holds only per-prompt difficulty and discrimination under opaque IDs, like the official one.
AI helped with this. The statistical models were designed by humans and calibrated by code created by AI. This PR description was written by a human to implement a function, practice scoring, required by the engineering team.