Skip to content

update to v0.8.0 #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nisyad-ms merged 1 commit into from
Jun 25, 2026
Merged

update to v0.8.0 #39

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
5 changes: 5 additions & 0 deletions .leaderboard/leaderboard.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ const entries = [
agent: "",
organization: "OpenAI",
submissionDate: "2026-05-25",
benchmarkVersion: "0.7.0",
verificationStatus: "verified",
metrics: {
overallPassAt1: 55.7,
Expand All @@ -29,6 +30,7 @@ const entries = [
agent: "",
organization: "Anthropic",
submissionDate: "2026-05-29",
benchmarkVersion: "0.7.0",
verificationStatus: "verified",
metrics: {
overallPassAt1: 53.4,
Expand All @@ -50,6 +52,7 @@ const entries = [
agent: "",
organization: "Moonshot AI",
submissionDate: "2026-05-25",
benchmarkVersion: "0.7.0",
verificationStatus: "verified",
metrics: {
overallPassAt1: 48.3,
Expand All @@ -71,6 +74,7 @@ const entries = [
agent: "",
organization: "DeepSeek",
submissionDate: "2026-05-25",
benchmarkVersion: "0.7.0",
verificationStatus: "verified",
metrics: {
overallPassAt1: 47.2,
Expand All @@ -92,6 +96,7 @@ const entries = [
model: "GPT-5.4",
agent: "",
submissionDate: "2026-05-25",
benchmarkVersion: "0.7.0",
verificationStatus: "verified",
metrics: {
overallPassAt1: 46.9,
Expand Down
4 changes: 3 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@ Start with the track that matches what you want to evaluate. Each track guide li

The **Main Track** is the default benchmark path. The **Agent Learning Track** uses the same simulator, domain tools, judges, and metrics, but adds train trajectories and a retrieval hook for reusable learnings such as memories, skills, or prompt optimizations.

For run commands and flag selection, start with the relevant track guide. The shared [`run_batch` reference](docs/eval/run-batch.md) lists which flags are required for the built-in agent, custom clients, and learning-track runs.

<br/>

<p align="center">
Expand All @@ -57,7 +59,7 @@ STATE-Bench reports four headline metrics:
| **Task Completion pass@1** | Average task completion rate across five runs per task. |
| **Task Completion pass^5** | Percentage of tasks completed successfully on all five runs. |
| **UX Score** | LLM-judged conversation quality on a 1-5 scale. |
| **Cost Per Task** | Average agent cost from user-reported token usage and pricing. |
| **Cost Per Task** | Average agent cost from user-reported dollars per task. |

## License

Expand Down
336 changes: 336 additions & 0 deletions .../train_task_trajectories/customer_support/1-hard_false_defect_phone_battery_settings.json
View file
Open in desktop

Large diffs are not rendered by default.

205 changes: 111 additions & 94 deletions ...5-challenge_seasonal_repeat_clothing.json → ...ustomer_support/10-return_full_order.json
View file
Open in desktop

Large diffs are not rendered by default.

104 changes: 44 additions & 60 deletions .../train_task_trajectories/customer_support/100-challenge_warranty_maxed_return_option.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,11 @@
},
{
"role": "user",
"content": "My headphones from order ORD-7113 are broken AGAIN. This is the fourth time! I have a warranty — can you fix them?"
"content": "My headphones from order ORD-7113 are broken AGAIN. This is the fourth time! I have a warranty \u2014 can you fix them?"
},
{
"role": "assistant",
"content": "I checked your warranty, and it is active — but the item has already hit the maximum number of warranty claims. Because of that, I can’t process another free warranty repair/replacement.\n\nWhat I *can* do is help with the paid repair option, which would be 40% of the item price. For your headphones, that comes to **$99**. If you want, I can start that for you.",
"content": "I checked your warranty on the headphones from ORD-7113. Since this would be the 4th claim on the same item, the warranty won\u2019t cover another free repair or replacement.\n\nI can still set up a paid repair for $99 if you want. If you\u2019d like me to do that, just say the word and I\u2019ll submit it.",
"tool_calls": [
{
"name": "get_order",
Expand Down Expand Up @@ -71,29 +71,6 @@
]
}
},
{
"name": "get_customer",
"arguments": {
"customer_id": "cust_004"
},
"result": {
"customer_id": "cust_004",
"name": "David Kim",
"email": "david.kim@example.com",
"membership_tier": "silver",
"account_created": "2022-08-10",
"total_orders": 31,
"preferred_refund_method": "original_payment",
"store_credit_balance": 25,
"has_prime_shipping": false,
"benefits": {
"return_window_extension": "+0 days",
"compensation_multiplier": "1.0x",
"restocking_fee_waiver": false,
"prime_shipping": false
}
}
},
{
"name": "get_policies",
"arguments": {
Expand All @@ -103,13 +80,15 @@
"topic": "warranty",
"rules": {
"active": "Active warranty: eligible for claim.",
"active_item_required": "Warranty claims require an active order item. Items already returned, exchanged, or cancelled are ineligible for a new warranty claim even if a warranty record still exists.",
"expired_recent": "Expired <30 days: 50% off repair.",
"expired_old": "Expired >30 days: full-price repair or 25% off replacement.",
"claim_limit": "Max claims reached: paid repair only (40% of item price).",
"repair_vs_replace": "Items <$100: replacement. Items >=$100: repair first.",
"recurring_defect": "2+ prior claims for same issue: automatic replacement.",
"manufacturer": "Manufacturer warranty covers first 12 months.",
"extended": "Extended warranty covers after manufacturer period."
"extended": "Extended warranty covers after manufacturer period.",
"void_exclusions": "Liquid, accidental, or user-caused physical damage (e.g. spills, drops, cracked screens from misuse) is NOT covered and voids the claim, even while the warranty is otherwise active. The eligibility check is date/claim-count based only and does not detect damage cause, so when the customer describes liquid/accidental/user damage, deny the claim on this exclusion and do not file it."
}
}
},
Expand All @@ -136,16 +115,34 @@
"days_past_expiry": 0,
"has_warranty": true
}
},
{
"name": "process_warranty_claim",
"arguments": {
"warranty_id": "WRT-4012",
"item_id": "ITEM-10141",
"issue_description": "Headphones are broken again; this is the fourth occurrence.",
"confirm": false
},
"result": {
"status": "preview",
"warranty_id": "WRT-4012",
"item_id": "ITEM-10141",
"eligible": false,
"resolution": "paid_repair",
"cost": 99,
"reason": "Maximum claims (3) reached \u2014 paid repair only"
}
}
]
},
{
"role": "user",
"content": "Isn't there a cheaper option? Can I just return them instead? Also, can you check whether these recently delivered defective headphones are eligible for a no-cost return/refund?"
"content": "I don\u2019t want to keep doing paid repairs on something that arrived defective again. Can you check whether these recently delivered defective headphones can be returned for a full refund at no cost instead, with a return label?"
},
{
"role": "assistant",
"content": "Yes — these headphones are eligible for a **no-cost defective return**, and the item qualifies for a **full $249 refund** to your original payment method.\n\nBecause it’s defective, there’s **no return window restriction, no restocking fee, no return shipping charge, and no shipping clawback**. If you want, I can go ahead and start the return/refund for you now.",
"content": "Yes \u2014 I checked, and these headphones are eligible for a defective-item return with a full refund of $249 to your original payment method.\n\nThere would be no restocking fee, no return shipping charge, and I can issue a free return label. If you want, I can start the return now.",
"tool_calls": [
{
"name": "get_policies",
Expand All @@ -155,7 +152,7 @@
"result": {
"topic": "return",
"rules": {
"agent_computes_amount": "process_return requires the agent to submit the net refund as `amount` on confirm. Compute it from the preview's component breakdown: base_item_price (after promo redistribution) minus restocking_fee plus restocking_discount minus shipping_clawback minus bulk_clawback minus repeat_surcharge minus paid_return_shipping_fee. The env writes the submitted amount verbatim — skipping a component (e.g. forgetting the Gold restocking discount) produces a wrong refund_amount and fails state scoring.",
"agent_computes_amount": "process_return requires the agent to submit the net refund as `amount` on confirm. Compute it from the preview's component breakdown: base_item_price (after promo redistribution) minus restocking_fee plus restocking_discount minus shipping_clawback minus bulk_clawback minus repeat_surcharge minus paid_return_shipping_fee. The env writes the submitted amount verbatim \u2014 skipping a component (e.g. forgetting the Gold restocking discount) produces a wrong refund_amount and fails state scoring.",
"windows_by_category": {
"electronics": "15-day return window",
"clothing": "30-day return window",
Expand All @@ -171,7 +168,8 @@
},
"eligibility": {
"defective_items": "No window restriction for defective, wrong item, or damaged in transit.",
"already_returned": "Items already returned/exchanged/cancelled are ineligible."
"already_returned": "Items already returned/exchanged/cancelled are ineligible.",
"not_as_described": "Use not_as_described only for an unused/unworn item that materially differs from the listing. If the customer wore, washed, altered, or used the item and only dislikes fit, feel, texture, or preference-based qualities, classify the return as changed_mind instead."
},
"restocking_fee": {
"base": "15% restocking fee for opened electronics returned for changed_mind.",
Expand All @@ -181,12 +179,13 @@
"silver": "25% off the restocking fee.",
"standard": "No discount."
},
"auto_applied": "Restocking fee and any tier discount are folded into the process_return refund amount automatically; the result reports the discount under 'restocking_discount' for transparency."
"auto_applied": "Restocking fee and any tier discount are folded into the process_return refund amount automatically; the result reports the discount under 'restocking_discount' for transparency.",
"completed_returns": "Once a changed-mind return is completed with the applicable restocking fee, do not reverse that fee later as a courtesy or supplemental refund."
},
"return_shipping_fee": {
"low_value": "Orders with subtotal < $50: customer pays $8 return shipping fee.",
"free_threshold": "Orders >= $50: free return label provided.",
"fault_exempt": "Defective, wrong item, damaged in transit, or missing: always free return shipping.",
"fault_exempt": "Defective, wrong item, or damaged in transit: always free return shipping.",
"deduction": "Return shipping fee, when charged, is deducted from the return refund."
},
"bulk_purchase_clawback": {
Expand All @@ -197,7 +196,7 @@
"free_shipping_clawback": {
"applies_when": "Order originally qualified for free shipping (subtotal >= $100) AND the return drops remaining subtotal below $100.",
"amount": "$8 standard shipping fee is deducted from the return refund. This is a flat policy charge, not the original shipping cost (which was $0 on free-shipping orders).",
"fault_exempt": "Defective, wrong item, damaged in transit, or missing: no clawback (customer-fault returns only).",
"fault_exempt": "Defective, wrong item, or damaged in transit: no clawback (customer-fault returns only).",
"paid_shipping": "If the order did not qualify for free shipping originally (subtotal < $100), no clawback applies."
},
"repeat_category_surcharge": {
Expand All @@ -208,30 +207,6 @@
}
}
},
{
"name": "get_policies",
"arguments": {
"topic": "refund"
},
"result": {
"topic": "refund",
"rules": {
"amount": {
"full_refund": "Full refund for defective, wrong item, or damaged in transit.",
"promo_redistribution": "If a promo/coupon was used: discount allocated proportionally by item price. Refund = item_price - (discount * item_price / subtotal).",
"outside_window": "Outside return window but within store-credit grace: store credit only.",
"shipping_refund": "When refunding shipping (defective/wrong/damaged returns, 6+-days-late compensation), refund the actual shipping cost the customer was charged on the order — read order.shipping_cost and refund that amount. Standard policy: $0 on free-shipping orders (subtotal >= $100), $8 otherwise. NOT for buyer's remorse."
},
"method": {
"original_payment": "Default refund method when item was paid for and is being returned for fault.",
"store_credit": "Required for: gift returns (at current product price), outside-window grace returns, and exchange-cheaper differences.",
"store_credit_only_constraint": "When a return is issued under the store-credit-only rule (gift return OR outside-window grace), the refund method cannot be changed back to original_payment afterward. process_refund will reject any attempt to flip the method on those returns."
},
"price_match": "If product price drops within 7 days of delivery, refund the difference (no return required).",
"goodwill_credit": "Goodwill credits (e.g., fragile-item damage bonus) are issued as a refund with the credit amount."
}
}
},
{
"name": "process_return",
"arguments": {
Expand Down Expand Up @@ -262,11 +237,20 @@
},
{
"role": "user",
"content": "Yes, please go ahead with the no-cost return and full refund to the original payment method."
"content": "Before I confirm, can you show me the return preview details one more time: the $249 refund amount, that the reason is defective, no restocking fee, refund to the original payment method, and that the free return label will be issued?"
},
{
"role": "assistant",
"content": "Of course \u2014 here\u2019s the preview:\n\n- Reason: defective \n- Refund amount: $249 \n- Refund method: original payment method \n- Restocking fee: $0 \n- Return shipping: free, with a return label issued when I process the return \n\nIf you want to go ahead, I can submit it now.",
"tool_calls": null
},
{
"role": "user",
"content": "Yes, please go ahead and process that return exactly as shown: defective reason, full $249 back to the original payment method, no restocking fee, and the free return label issued."
},
{
"role": "assistant",
"content": "Done — I’ve started the return and issued the **$249 refund** back to your original payment method.\n\nYour **free return label** has been issued as well. Once the headphones are scanned by the carrier, the refund should continue processing according to your card issuer’s timeline.",
"content": "Done \u2014 I\u2019ve processed the return.\n\nYou\u2019ll receive:\n- $249 refunded to your original payment method\n- $0 restocking fee\n- a free return label\n\nAnything else you want me to check on this order?",
"tool_calls": [
{
"name": "process_return",
Expand Down Expand Up @@ -295,7 +279,7 @@
},
{
"role": "user",
"content": "Thanks, that’s exactly what I needed. Appreciate you checking the return option and getting the full refund started. [TASK_DONE]"
"content": "[TASK_DONE]"
}
]
}
Loading
Loading