Question: sanctioned validation path for custom ContextContract agent

[ombhojane]

Request: sanctioned validation path for ContextContract agent

I am testing a local ContextContract controller with STATE-Bench.

Current status:

• A custom BaseAgent loads in the public checkout.
• A direct no-score run_task() smoke works on two shopping tasks.
• I am not claiming an official score or external validation.
• The official CLI path is blocked locally by STATE_BENCH_EVAL_ENDPOINT.

Question:

What is the clean sanctioned path from adapter compatibility to valid STATE-Bench evidence?

Useful answers would be:

1. how to access/configure the locked eval endpoint,
2. whether a deterministic no-score smoke can count as nonofficial adapter evidence,
3. whether this belongs in Main Track or Agent Learning Track.

I want to avoid private simulator hacks or protocol-unsafe claims.

[nisyad-ms]

Hi Om,

I don't fully understand what the ContextContract agent is doing? If you are generating any kind of learning / memory from the train set and testing on the test set, then it belongs to the Agent Learning Track. If you simply want to test your agent on the whole set, please use the Main track.

Regarding the eval endpoint, the user must provide the endpoint - the benchmark only recommends which model to use (GPT 5.4 high) for a fair comparison.

whether a deterministic no-score smoke can count as nonofficial adapter evidence?
Please elaborate on this question.