Add conformal prediction: calibrated coverage/risk guarantees for zero-shot NER

[ALI-AL-MARJANI]

Motivation

GLiNER scores every candidate (span, type) pair with an independent sigmoid and filters with
threshold=0.5 by default — a convenient default, not a statistical guarantee. It doesn't say
what fraction of true entities a deployment should expect to miss, and it isn't calibrated to
any particular dataset or entity-type mix. This is a symptom several open issues already point
at: #69 (feature request just to expose confidence values), #192 (label ordering changing
confidence scores — a miscalibration symptom), #324 (transformers v5 causing uniformly
low/meaningless scores).

This PR adds gliner.conformal, a small additive module that replaces the arbitrary
threshold=0.5 cutoff with a threshold calibrated on a held-out labeled set, backed by
finite-sample, distribution-free guarantees (Vovk et al.; Angelopoulos & Bates,
arXiv:2107.07511; Conformal Risk Control, arXiv:2208.02814). Two very recent papers
(arXiv:2601.16999, arXiv:2605.18812) establish conformal-prediction theory for NER
specifically, but neither ships code and neither addresses open-vocabulary/zero-shot label
sets — as far as I can find, no released implementation of conformal prediction for NER exists
anywhere. This is, to my knowledge, the first one, and the first that works with GLiNER's
inference-time arbitrary label sets.

What this is not claiming

Not a rigorous zero-shot coverage guarantee for entity types never seen in calibration.
Split-conformal validity requires calibration/test exchangeability; a type with zero
calibration occurrences has no well-defined quantile — undefined, not merely wide — and no
theorem in the literature I surveyed licenses a coverage claim for it. ConformalGLiNER
handles this honestly: types below the calibration floor get a loud warning and GLiNER's
original uncalibrated behavior, flagged "calibrated": False on every affected entity — never
silently blended into a guaranteed-looking number. Full discussion in docs/conformal.md's
Limitations section.

API

from gliner import GLiNER
from gliner.conformal import ConformalGLiNER

model = GLiNER.from_pretrained("gliner-community/gliner_small-v2.5")
cg = ConformalGLiNER(model)                                  # wraps, never mutates, the model
cg.calibrate(calib_data, alpha=0.1, mode="risk_control")      # span_filter | risk_control | mondrian
entities = cg.predict_entities(text, labels)                  # entities + "conformal" metadata
report = cg.coverage_report(test_data)                        # empirical validation
cg.save_calibration(path) / ConformalGLiNER.load_calibration(path, model)

Three modes: span_filter (marginal per-entity coverage), risk_control (Conformal
Risk Control bounding expected missed-entity rate — the flagship mode for compliance/PII use
cases; GLiNER's independent-sigmoid decode rule satisfies CRC's required monotonicity condition
by construction, proven in the code's own docstrings), and mondrian (per-type
calibration so rare types aren't systematically under-covered by the marginal guarantee).

Design

No changes to any existing model file. Raw pre-sigmoid, pre-decode span scores are already
reachable via the public model.run_batch(); gliner/conformal/ is a pure additive package:

gliner/conformal/
├── scores.py        # raw score extraction, span-mode only (see Scope below)
├── calibrators.py   # pure-Python split-conformal quantile, CRC λ-search, Mondrian partitioning
└── wrapper.py        # ConformalGLiNER

calibrators.py has no GLiNER dependency and is independently unit-tested against synthetic
scores with analytically known coverage.

Scope: span-mode models only (UniEncoderSpanGLiNER, BiEncoderSpanGLiNER — the default
span_mode="markerV0" architecture). Token-mode, generative-decoder, and relation-extraction
variants apply their confidence threshold inside the forward pass to prune candidates before
returning scores, so run_batch()'s output isn't the full candidate universe for those
architectures — calibrating against it would silently understate true coverage, so it's
explicitly unsupported (NotImplementedError, not a silent wrong answer).

Tests

- tests/test_conformal_calibrators.py — synthetic, no network. Includes a 20,000-trial
empirical coverage check (0.9016 ± 0.0021 against a 0.9 target) and a 200-trial risk check.
- tests/test_conformal_gliner.py — integration tests against gliner-community/gliner_small-v2.5
(mirrors the existing test_models.py::test_span_model pattern). Covers all three modes,
calibration-floor enforcement, the out-of-calibration warn+fallback path, empty predictions,
save/load round-trip, and a deterministic regression test for a real per-sentence-vs-per-entity
metric bug found during validation (see below).
- 364 tests pass total (1 skipped, matches pre-existing), zero regressions — confirms this
is fully additive.

Empirical validation

Ran the module against real CoNLL-2003 and WNUT-17 data (gliner-community/gliner_small-v2.5):
in-domain coverage on both, plus a zero-shot experiment calibrating on CoNLL-2003 and measuring
coverage on WNUT-17. Every in-domain row tracks its target within ~0.3–2 standard deviations
across α ∈ {0.05, 0.1, 0.2}. The zero-shot experiment gives the concrete number behind the
"not a zero-shot guarantee" claim above: types shared with calibration (location, person)
reach 0.87–0.98 coverage; types never seen in calibration (corporation, creative-work,
group, product) sit flat at 0.551 regardless of α or mode.

Two real bugs surfaced and got fixed during this validation, not glossed over: (1) an early run
calibrated on CoNLL-2003's official validation split and tested on its official test split as
static separate pools, which showed a systematic ~5-standard-deviation coverage undershoot —
traced to those splits not being fully exchangeable for this model, a real property of the
benchmark, not a defect, fixed by pooling and re-splitting per trial; (2) coverage_report
reported per-entity-pooled coverage for risk_control mode when that mode actually calibrates
and guarantees a per-sentence average miss rate — a real bug in the shipped library, fixed
with a deterministic regression test.

Reproducible via scripts/conformal_validation.py (not part of the package; uses datasets
and matplotlib, which aren't currently declared dependencies of this repo — matches the
existing convention that scripts/ tooling is installed ad hoc, e.g. convert_to_onnx.py).

Documentation

docs/conformal.md — practitioner guide: motivation, the three modes explained, a runnable
example, and a Limitations section covering the exchangeability caveat, domain shift, and the
span-mode-only scope.

Additive, reviewed for scope creep

- Zero changes to gliner/modeling/, gliner/decoding/, or any other existing file. gliner/model.py now has one small addition — BaseEncoderGLiNER.calibrate() / .conformal — added in response to review feedback to support calibrating directly on the model (see comment below).
- No new required dependencies for the shipped package — gliner/conformal/ uses only
NumPy/PyTorch (already required).
- gliner.conformal is not imported by gliner/__init__.py by default — opt-in only.

[Ingvarstep]

Hi @ALI-AL-MARJANI, thank you for implementing the good idea. I think it would be better to update the main class API to support per-label thresholds and an additional calibration method. Also, please, fix ruff errors.

Best regards,
Ihor

[ALI-AL-MARJANI]

Ruff fixed.

• Per-label thresholds: ConformalGLiNER now exposes calibrated_types and thresholds() — the latter returns the nonconformity threshold actually applied per label, genuinely different per type under mode="mondrian".

• Additional calibration method: added model.calibrate(calib_data, alpha=0.1, mode=...) directly on GLiNER, plus a model.conformal property exposing the resulting ConformalGLiNER instance, so calibration and inference can live on the same object:
  model.calibrate(calib_data, alpha=0.1, mode="risk_control")
  model.conformal.predict_entities(text, labels)

Also merged latest main in along the way (streaming feature), no conflicts now