State scorer collapses sibling `order_items` (and orders) — `_record_identity` keys child records by their parent id, causing false `task_completion = 0`

[SimonHegelich]

Summary

evaluate_state_requirements reconstructs the final DB snapshot from the task-env
file, which stores entities as lists. scoring.py::_record_identity derives a
record's key from the first matching id field, but it checks owner/foreign keys
(order_id, customer_id, cart_id) before the record's own id (item_id,
…). An order_item carries both item_id and its parent order_id, so all line
items in the same order collapse under one key — only the last survives and its
siblings disappear from the reconstructed snapshot. Their state_requirements
then report as missing_assertions, failing the task even when the agent behaved
perfectly.

Environment

• STATE-Bench v0.7.1 (commit a0ffc65)
• state_bench/scoring.py, function _record_identity (around line 206)

Reproduction

Run from a STATE-Bench checkout (uses only shipped files — task
customer_support/13-cancel_partial):

import json
from pathlib import Path
from state_bench.scoring import _record_identity, evaluate_state_requirements
from state_bench.schemas import TaskDefinition, StateDiff

# Root cause: three sibling order_items collapse to one key.
items = [{"item_id": f"ITEM-{n}", "order_id": "ORD-1", "item_status": "confirmed"} for n in (1, 2, 3)]
print([_record_identity(i) for i in items])   # -> ['ORD-1', 'ORD-1', 'ORD-1']

# End-to-end: a correct partial cancellation of only the shirt (ITEM-9105).
task = TaskDefinition.from_dict(json.loads(
    Path("state_bench/domains/customer_support/tasks/13-cancel_partial.json").read_text()))
state_diff = StateDiff(modified={
    "orders": {"ORD-6016": {"status": {"old": "processing", "new": "partially_cancelled"}}},
    "order_items": {"ITEM-9105": {
        "item_status": {"old": "confirmed", "new": "cancelled"},
        "refund_amount": {"old": None, "new": 59},
        "refund_method": {"old": None, "new": "credit_card"},
    }},
})
res = evaluate_state_requirements(task, state_diff)
print(res.score, res.reasoning, res.details)

Actual output (v0.7.1)

['ORD-1', 'ORD-1', 'ORD-1']
0 State requirements failed: missing_assertions=2 {'missing_assertions': [
  {'entity_type': 'order_items', 'record_key': 'ITEM-9106', 'field': 'item_status', 'value': 'confirmed'},
  {'entity_type': 'order_items', 'record_key': 'ITEM-9107', 'field': 'item_status', 'value': 'confirmed'}]}

ITEM-9106 and ITEM-9107 are never modified by the agent and should remain
confirmed, but they vanish from the reconstructed snapshot because all three
items collapsed under ORD-6016.

Expected output

['ITEM-1', 'ITEM-2', 'ITEM-3']
1 All required state assertions matched the saved state_diff. None

Impact

Any task asserting state on a sibling line item that is not the last in its order
gets phantom missing_assertions and a false task_completion = 0. Observed on
at least customer_support/13-cancel_partial and 15-cancel_backordered. The
same class affects orders keyed by customer_id for customers with multiple
orders (currently masked when the task modifies the order, so the diff re-adds the
correct key).

Suggested fix

Order _record_identity so a record's own id precedes owner/foreign keys.
Records without their own id (orders, customers, products) keep their previous
keying; only records that have both their own id and a parent id change:

 def _record_identity(record: dict[str, Any], fallback_index: int | None = None) -> str | None:
     for key in (
         "cart_item_id",
-        "cart_id",
-        "customer_id",
-        "order_id",
         "item_id",
         "booking_id",
         "reservation_id",
         "rental_id",
+        "cart_id",
+        "order_id",
+        "customer_id",
         "id",
         "product_id",
     ):

[nisyad-ms]

Hi Simon,

Thank you for the detailed report. This is helpful. We are aware of a related bug (_record_idenity key ordering) and working on it to fix in the next version.

[SimonHegelich]

Thanks for confirming — glad it's already on your radar.

While preparing a local fix I found that the same _record_identity key-ordering class affects more than order_items. Sharing the details in case they're useful for the next-version fix.

Two distinct failure modes

The priority list checks owner/foreign keys before a record's own id and omits several own ids entirely. That produces:

1. Sibling collapse (the originally reported case): order_items sharing one order_id collapse to a single key.
2. Unresolvable record_key: entities whose own id is missing from the list key by a parent id, so state_requirements that reference them by record_key can't be found in the reconstructed snapshot.

Mode 2 affects, in the shipped data:

entity	domain	own id	currently keyed by
warranties	customer_support	warranty_id	item_id
users	travel	user_id	fallback index
flights	travel	flight_id	fallback index
hotel_inventory	travel	hotel_id	fallback index
car_inventory	travel	car_id	fallback index

warranties is the concrete one: 30 state_requirements across customer_support reference warranties by record_key = WRT-xxxx (e.g. tasks 31-warranty_recurring_defect, 43-compound_warranty_then_return, 127, 130), but the reconstructed snapshot keys them by item_id.

Reproduction (shipped files only)

import json
from pathlib import Path
from state_bench.scoring import _as_record_mapping

env = json.loads(Path(
    "state_bench/domains/customer_support/task_envs/31-warranty_recurring_defect.json"
).read_text())
mapping = _as_record_mapping(env["warranties"])
print("snapshot keys:", list(mapping.keys()))   # -> ['ITEM-9030']

task = json.loads(Path(
    "state_bench/domains/customer_support/tasks/31-warranty_recurring_defect.json"
).read_text())
keys = [r["record_key"] for r in task["state_requirements"]
        if r.get("entity_type") == "warranties"]
print("requirement record_keys:", keys)         # -> ['WRT-4001', 'WRT-4001', 'WRT-4001']
print("WRT-4001 in snapshot:", "WRT-4001" in mapping)  # -> False

The warranty record carries warranty_id=WRT-4001, item_id=ITEM-9030, order_id=ORD-6004, product_id=PROD-2030; it is keyed by ITEM-9030, so the WRT-4001 assertions miss.

This is masked whenever the agent mutates the asserted field (the state_diff re-adds the WRT-xxxx key), but surfaces on any unchanged-state assertion against these entities.

Suggested fix

List every own id before any foreign key, ordering the two-own-id entities so the more specific one wins (warranty_id before item_id; reservation_id before hotel_id; rental_id before car_id):

def _record_identity(record, fallback_index=None):
    for key in (
        "cart_item_id",
        "warranty_id",
        "item_id",
        "reservation_id",
        "rental_id",
        "booking_id",
        "flight_id",
        "hotel_id",
        "car_id",
        "user_id",
        "cart_id",
        "order_id",
        "customer_id",
        "id",
        "product_id",
    ):
        value = record.get(key)
        if value is not None:
            return str(value)
    if fallback_index is not None:
        return str(fallback_index)
    return None

Verified offline against the shipped env data for all three domains: every entity now keys by its own id, there are zero sibling collapses, warranties[WRT-4001] resolves, and the 13-cancel_partial end-to-end repro from the original report scores 1. Reference tables with no _id field (promotions, keyed by promo_code) keep index-keying and aren't referenced by record_key.