[anhappdev]

Summary

The "Core ML quick wins" from #1188. None of them depend on Core AI. One commit per item:

• Cache the compiled model. MLModel.compileModel used to run on every mlperf_backend_create. The compiled .mlmodelc now lives in Caches/mlperf_coreml_models (one entry per source path) and is reused while a fingerprint of the source files (path, size, mtime) still matches. An unreadable entry gets recompiled. If caching fails, the backend uses the freshly compiled model. The model URL is built with URL(fileURLWithPath:). URL(string:) returns nil on iOS 15/16 for paths that need percent-encoding (iOS 17+ encodes them silently).
• Wire up ft_callback. It is called after the prediction when the benchmark sets custom_setting { id: "report-first-token" value: "true" }, and is off by default. Calling it for every benchmark is unsafe: non-token runs don't expect token latencies, and in Offline mode MlperfDriver puts sample.back().index where loadgen's FirstTokenComplete dereferences a SampleMetadata*. No Core ML benchmark sets the flag yet; it's plumbing for a future LLM path. A failed CoreMLExecutor creation now also logs the error.
• Raise the xcframework minimum to iOS 15.0, to match the app's deployment target.
• Remove the dead image_classification / image_classification_offline entries (v3_0 MobilenetEdgeTPU). image_classification is only the task_set of the _v2 tasks, not a task id.
• Refresh models/requirements.txt. It now pins coremltools 9.0, torch 2.7.0 / torchvision 0.22.0 / timm 1.0.15, and tensorflow-macos 2.12.0 (the newest torch and TF that coremltools 9.0 is tested with), on Python 3.11. coremltools 7 changed the default ct.convert target to mlprogram, which breaks the TF scripts' .mlmodel, quantization_utils and NeuralNetworkBuilder calls. So mobilebert.py, mobiledet.py, mobilenet_edgetpu.py and mosaic.py now pass convert_to='neuralnetwork', which keeps their previous output.

Also fixed: when model loading failed, mlperf_backend_create left backendExists = true and leaked backend_data, so every later create was refused. The app aborts on a failed create anyway (LOG(FATAL) in external.cc), so this only affected callers that keep running, such as dev-utils.

Test plan

• bazel build --config=ios //mobile_back_apple/cpp/backend_coreml:libcoremlbackend (Xcode 26.6): the device slice has minos 15.0 and links CryptoKit.
• swiftc -typecheck -swift-version 6 for arm64-apple-ios15.0 and arm64-apple-macos13.1, swift-format lint --strict, clang-format 16 --dry-run --Werror, and markdownlint.
• macOS harness around CoreMLExecutor, with .mlpackage and .mlmodel files under a path containing a space. Checked: cold compile, warm reuse (entry untouched), recompile after the source changes, recovery from a corrupted entry, direct .mlmodelc load (nothing cached), and a clear error for a missing file. Predictions were correct in every case.
• Real v4_1/apple/mobilenetv4_fp32_NCHW.mlpackage (CPU & GPU, macOS VM): backend create took 266–428 ms before this change and 21–25 ms once cached. The cache entry is 62 MB.
• Fresh Python 3.11 venv from requirements.txt: TF SavedModel to NeuralNetwork plus fp16 weights, and timm MobileNetV4 to ML Program plus W8 and the experimental W8A8 API, then predict.
• Real devices on BrowserStack, all with this PR's test package (ios_tests-apple-1080):
  • iPhone 16 Pro, iOS 18 (CI apple-iPhone 16 Pro-18): All tests passed!. The device log shows each ML Program model loaded three times from one Library/Caches/mlperf_coreml_models/<sha256>/model.mlmodelc entry, so compiled models are reused across backend creates.
  • iPhone 13, iOS 15.5 (manual run): +14: All tests passed!, models served from the same cache, and no cache or CoreMLExecutor errors.
  • MobileBERT logs about 900 Espresso::overflow_error lines on iOS 15. The base commit 846c58d logs 928 on the same device model, so they come from the model, not from this PR.

Note: CI's test package doesn't start on iOS 15/16 for any PR. dyld can't find lib_TestingInterop.dylib or _Testing_Foundation.framework: Xcode 26 doesn't embed them, and on iOS 18 they load from elsewhere on the device. For the iOS 15 runs above I copied both from Xcode into Runner.app/Frameworks of the downloaded package. Fixing that in make flutter/ios/test-package is a separate change.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[freedomtan]

@anhappdev check Core ML's built-in caching mechanism works or not. Or, in other words, does app-level cache pay off or not. If model compilation time is a concern, we can just load compiled model (.modelc/).

[anhappdev]

@anhappdev check Core ML's built-in caching mechanism works or not. Or, in other words, does app-level cache pay off or not. If model compilation time is a concern, we can just load compiled model (.modelc/).

@freedomtan The problem is that the app explicitly compiles the model everytime it runs the benchmark, so we need to fix the logic there.
let compiledModelURL = try? MLModel.compileModel(at: url)

[freedomtan]

To avoid compiling every time, we can simply compile a model offline.
.mlpackage .mlmodel -> .mlmodelc/ on either Linux or macOS machine. Provide .mlmodelc. from our server/cloud side.

@freedomtan to check it.