feat: activate Focal Loss, add Dice Loss + span-width weighting, OpenVINO INT8 pipeline - #361
Conversation
Ingvarstep
left a comment
Ingvarstep
left a comment
There was a problem hiding this comment.
Thank you very much for your great contribution. I have a few suggestions:
- First, use importing utils so the project is consistent from this perspective;
- Move ablation scripts into scripts/dice_loss_study, or something like that;
| import numpy as np | ||
| import torch | ||
| import onnxruntime as ort | ||
| try: |
There was a problem hiding this comment.
Please, use
Line 61 in dfc0061
Addresses review feedback from Ingvarstep on urchade#361: - gliner/onnx/model.py: replaced the local `try/except ImportError` around `import onnxruntime` with the same is_module_available() pattern used elsewhere in the codebase (gliner/modeling/encoder.py's IS_PEFT/IS_LLM2VEC etc.), for consistency. - Moved scripts/{baseline_eval,convert_to_openvino,train_ablation, visualize_results}.py into scripts/dice_loss_study/ -- they're four numbered steps (Step 1-4) of one self-contained ablation study pipeline, not independent general-purpose scripts. Updated each script's own docstring usage example to the new path. No cross-script imports and no __file__-relative path logic exist between them, so the move doesn't change how they locate the shared results/ directory (all paths are CWD-relative, assuming repo-root invocation, unchanged). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…x ruff
Found while addressing the consistency review comment, beyond the two
explicitly requested files:
- gliner/model.py had onnxruntime imported twice: an unguarded try/except
at the top of the file (dead code, immediately shadowed by the existing
is_module_available()-gated import further down) and the real one.
Removed the redundant one.
- The existing is_module_available()-gated block had been changed to set
ONNX_AVAILABLE = True unconditionally, even without onnxruntime
installed, while _check_onnx_export_preconditions still raised
"onnxruntime is not available" behind a now-dead `if not ONNX_AVAILABLE`
check. Traced the actual export call chain
(export_to_onnx -> _run_torch_onnx_export): it only uses
torch.onnx.export, never `ort` -- onnxruntime is only needed to later
*load*/*run* an exported model, not to export one. So the underlying
behavior change (allow export without onnxruntime) was correct; the
dead/misleading check was the bug. Removed the check, restored
ONNX_AVAILABLE to truthfully reflect is_module_available("onnxruntime").
- The lazy Trainer/TrainingArguments import (avoids a torch.distributed
import deadlock on macOS ARM) had dropped create_training_args'
entire docstring (all Args: entries) and both methods' type hints in the
process. Restored the docstring verbatim and the return-type hints via
`if TYPE_CHECKING: from .training import Trainer, TrainingArguments` +
string forward-refs, so IDEs/static analysis get full types back without
reintroducing the eager import.
- Moved the lazy-import helper out of the middle of the top-level import
block (was causing every import after it to trip E402).
Ruff (gliner/config.py, gliner/model.py, gliner/modeling/base.py,
gliner/onnx/model.py, gliner/training/trainer.py): same PLR0917
version-drift issue as the other two branches, plus a few genuinely
introduced by this PR's own diff -- missing docstring args for the new
loss_type/dice_gamma params, an unused unpacked variable, two lazy imports
needing noqa, and numpy/torch movable into a TYPE_CHECKING block in
gliner/onnx/model.py (that file already has `from __future__ import
annotations`; every use is in a type hint, none at runtime).
317 passed, 1 skipped, ruff check gliner clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
# Conflicts: # gliner/model.py
Also merged latest main (52 commits — the branch was quite stale) to pick up the streaming feature and clear the merge conflict. Along the way restored a docstring that had been accidentally dropped from create_training_args during the lazy-import fix for the macOS deadlock, and fixed the ruff errors introduced by that version drift (same story as the other two PRs — pip install ruff with no version pin, ended up on 0.16.0 mid-review). |
2 participants
Summary
1. Loss functions (
gliner/modeling/loss_functions.py,base.py,trainer.py)focal_loss_with_logitsalready exists in the codebase but is disabled by default(
alpha=-1, gamma=0). This PR:loss_type,focal_loss_alpha,focal_loss_gammainTrainingArgumentssousers can activate it
span_dice_loss()— span-level Dice Loss adapted from Li et al. (ACL 2020),applied element-wise over the
(B, L×K, T)logit tensor withignore_indexmaskinguse_span_width_weightflag: positive spans of width k receivew(k) = 1 + log(k+1)— zero inference overheadMotivation: WNUT-17 has 187× more negative spans than positive entities (0.53%
positive ratio). BCE's gradient is dominated by trivial negatives. Focal α=0.25 delivers
+0.99 pp WNUT-17 F1; Dice delivers +0.70 pp.
2. Bug fixes (
gliner/utils.py,gliner/modeling/encoder.py,gliner/model.py,gliner/onnx/model.py)is_module_available()changed from__import__()toimportlib.util.find_spec()—prevents optional packages (peft, tensorflow) from being eagerly imported, which caused
OpenMP deadlocks on macOS ARM
encoder.py:kwargs.pop("token_lengths", None)prevents crash on bi-encoder modelsthat pass this GLiNER-internal kwarg to HuggingFace forward methods
Trainer/TrainingArgumentsinmodel.py— avoids importingtorch.distributed at module load time
3. OpenVINO INT8 pipeline (
scripts/convert_to_openvino.py)New script: ONNX → OpenVINO IR → INT8 weight compression via
nncf.compress_weights(INT8_ASYM).contains
Ifnodes with dynamic rank that the CPU plugin rejects during calibration-basedquantization
Benchmark
Model:
knowledgator/gliner-bi-small-v1.0, 200 fine-tuning steps on CoNLL-2003, eval onWNUT-17.