Repository navigation
IndependentScorer batch training, partitioned (Hive) datasets, and fi… - #7
Merged
Merged
Conversation
…xes found along the way IndependentScorer crashed in train() when given a dict of per-item estimators (the coordinator asked the dict for support_batch_training), and could never batch-train at all. Training dispatch now asks the scorer (BaseScorer.support_batch_training), and IndependentScorer batch-trains one estimator per item from only that item's rows: - Interactions partitioned into ITEM_ID=<value>/ directories are trained from directly, with no copy. - Otherwise one pass over the partitions writes each item's rows to buffered temporary parquet chunks under batch_chunk_dir (constructor argument or scorer_config; default: system temp directory). - The binary-label check runs over the whole dataset, items without interactions are dropped, and validation columns are checked, matching the in-memory path. Mixing batch and in-memory estimators raises. Datasets: - Partitioned datasets are listed recursively (local now matches S3); _/. directories such as Spark's _temporary are skipped on both. - Hive key=value/ directories: ITEM_ID is restored as a column; other keys (e.g. month=) are ignored unless the client schema declares them. - Fixed reading a single partition file of a directory dataset (S3 crashed; local returned no columns with a client schema). - Dataset.peek_columns() reads column names from parquet metadata. Batch training generally: - train() no longer loads the full interactions/users up front in batch mode; retrievers and GCSL load interactions on demand, and GCSL keeps them with the model. - Batch readers apply the recommender's outcome-column handling (dropped by default, kept as features by GCSL); users are fetched once per iterator. - recommend() no longer fails when inference input lacks training-time OUTCOME_* columns. - In-memory retraining of IndependentScorer keeps the binary-label check. Docs: capability matrix, scorers, datasets, training and orchestration.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
IndependentScorer batch training + partitioned (Hive) datasets
IndependentScorercrashed intrain()when given a dict of per-item estimators, and couldn't batch-train at all. It now can: each item trains from only its own rows.ITEM_ID=<value>/, e.g. SparkpartitionBy): each item reads its own files, with no copy.batch_chunk_dir(new constructor /scorer_configoption).Datasets
_temporary-style folders are skipped.key=value/folders:ITEM_IDis restored as a column; other keys (e.g.month=) are ignored unless the client schema declares them.Batch training generally
train()no longer loads all interactions/users up front in batch mode.OUTCOME_*as features).recommend()no longer fails when inference input lacks training-timeOUTCOME_*columns.Behaviour changes: undeclared Hive keys are no longer returned by whole-directory reads; local partitioned datasets now include files in subfolders.
Docs updated (capability matrix, scorers, datasets, training, orchestration). Tests:
test_independent_batch_training.py,test_hive_partitioning.py(incl. S3 via moto); full suite passes (1253).