Add ColumnArithmetic, ColumnConcat and TypeCast converters

[Felipedino]

Add ColumnArithmetic, ColumnConcat and TypeCast converters

Summary

• Redesigns ColumnArithmetic so operands are derived from the columns selected in scope instead of typed column names, and fixes a silent data-corruption bug in its integer path.
• Adds ColumnConcat, a new converter for concatenating string columns.
• Adds TypeCast, a new converter for changing a column's DashAI type from within a pipeline, reusing the same validation used by the dataset upload preview.

Motivation

ColumnArithmetic required the user to (1) select a scope of columns in the pipeline builder UI, and then (2) separately type the exact column names into column_a/column_b text fields — completely redundant, since the converter's fit() already receives only the scoped columns. This PR removes that redundancy across the whole "binary operand" converter family (arithmetic and the new concat converter), and adds a way to change column types mid-pipeline that previously only existed in the dataset-upload preview screen.

Changes

ColumnArithmetic (redesigned)

• column_a, column_b, and operand_b_mode are no longer schema parameters. Operands are now derived from the columns selected in scope during fit():
  • 2 columns selected → operate between them, in ascending dataset-column-index order (the platform's fixed scope ordering — see ConverterJob).
  • 1 column selected → operate against a new constant parameter.
• New swap_operands boolean lets the user reverse the two-column order (relevant for subtract/divide, which aren't commutative).
• metadata.input_cardinality = {"min": 1, "max": 2} reuses the existing explorer mechanism to constrain the frontend's column selector to 1–2 columns, so no frontend changes are required.
• Bug fix: when both operand columns are Integer and either has a missing value, transform() previously cast the underlying NaN to int64 via .to_numpy(dtype="int64"), which numpy silently converts to the sentinel -9223372036854775808 instead of raising or propagating a null. fit() now checks null_count on the operand columns and falls back to the Float output path (where missing values propagate correctly as NaN) whenever either operand has missing values.
• Bug fix: a constant of NaN or Infinity (reachable via direct API calls, not the browser UI) previously passed validation and then crashed with a confusing low-level ValueError/OverflowError from int(nan). Now explicitly rejected with a descriptive message.
• Tightened the per-column type check so a scoped column with no recorded type also fails validation (previously silently skipped).

ColumnConcat (new)

• Concatenates the columns selected in scope, mirroring ColumnArithmetic's scope-driven design: 2 columns → join them; 1 column → join with a constant string.
• Optional separator inserted between the two operands (none by default).
• swap_operands to reverse concatenation order, same as ColumnArithmetic.
• Null-propagating: if either operand is missing for a row, the result for that row is None.
• metadata.allowed_types = [Text, Categorical].

TypeCast (new)

• Casts every column in scope to a target new_type (Integer, Float, Text, or Categorical), reusing DashAI.back.types.type_validation.validate_type_change — the exact same validation and conversion logic used by the /datasets/validate_type_changes endpoint behind the upload preview screen, so behavior is consistent between upload-time and pipeline-time type changes.
• Columns already of the target type are left untouched.
• on_error controls behavior when a column can't be safely converted (e.g. non-numeric text targeting Integer): "raise" (default) stops with a descriptive, column-specific error; "skip" leaves that column unchanged, prints a warning, and continues with the rest of the columns in scope.
• Caches each column's conversion result computed during fit() and reuses it in transform() when called with the same dataset (the common case), instead of recomputing the full validation/conversion pass twice.

NumericExpansion (fixed, pre-existing converter)

• Bug fix: same class of bug as the ColumnArithmetic fix above — square on an Integer column with missing values previously kept the declared output type as Integer while transform() cast the underlying NaN to the int64 sentinel via .to_numpy(dtype="int64") and squared it, silently producing garbage. fit() now checks null_count on the source column and falls back to Float output whenever it has missing values.

FeatureEngineeringConverter (docstring only)

• Updated the base class docstring, which described this category as producing only "numeric features" — no longer accurate now that ColumnConcat (Text output) belongs to the same category.

Registration

• ColumnConcat and TypeCast registered in DashAI/back/initial_components.py alongside the existing simple converters.

Design notes / trade-offs

• Operand order is not user click-order. The platform (ConverterJob) resolves a scope's columns in ascending dataset-column-index order, not the order the user selected them in the UI — this is a pre-existing, unchangeable platform behavior (confirmed not just for these converters but for the whole scope-selection mechanism used by explorers too). swap_operands is the chosen mitigation: a simple, low-risk escape hatch rather than a deeper fix to the shared column-selection component, which would carry more risk and affect every converter/explorer using scope selection.
• No backwards-compatibility shim was added for ColumnArithmetic's old column_a/column_b/operand_b_mode constructor signature. This converter was introduced and rewritten within the same development cycle and was never released, so there is no persisted data to migrate.
• Dividing/subtracting a column from itself (previously possible by passing the same column name as both column_a and column_b) is no longer directly expressible, since a scope can't contain the same dataset column twice. This was a capability of an unreleased draft, not a shipped feature.

Testing

No automated tests were added in this PR. tests/back/converters/ only covers converter metadata (test_base_converter_metadata.py); there is no test file for simple_converters/ — this is a pre-existing gap in the codebase, not something this PR fixes.

What was actually done to validate the changes: manual, ad hoc verification scripts run locally against real DashAIDataset instances during development (not committed as test files). These covered:

• ColumnArithmetic: 2-column add/subtract (with and without swap_operands), 1-column + constant, integer-preserving output when no missing values, Float fallback with correct NaN propagation when a missing value is present, rejection of out-of-range column counts, rejection of NaN/Infinity constants.
• ColumnConcat: 2-column concatenation with a separator and null propagation, 1-column + constant (including an empty-string constant), swap_operands.
• TypeCast: Text→Integer/Float/Categorical, Float→Integer rejection on decimal loss (both raise and skip modes), no-op on already-matching type, and confirmed (via call-count instrumentation) that transform() reuses fit()'s cached conversion when given the same dataset, falling back to a fresh conversion when given a different one.
• NumericExpansion: square on an Integer column with a missing value now falls back to Float with correct NaN propagation, and still returns Integer when there are no missing values.
• ruff check / ruff format pass clean on all changed/added files.

Follow-up work should add real pytest coverage under tests/back/converters/simple_converters/ for this converter family.

Follow-ups (not in this PR)

• Add automated test coverage for simple_converters/ (see Testing section above).
• No preview images (column_arithmetic.png, column_concat.png, type_cast.png) exist yet in DashAI/back/static/images/ for these converters.
• The scope-derivation boilerplate (cardinality check + 2-vs-1-column branching) is duplicated between ColumnArithmetic.fit() and ColumnConcat.fit(). Left un-extracted for now (only two instances, with different per-type validation), but worth revisiting if a third "binary operand from scope" converter is added.

[Copilot]

Pull request overview

This PR expands DashAI’s tabular pipeline capabilities by introducing new feature-engineering converters (arithmetic, concatenation, numeric expansion) and a pipeline-time type casting converter, while also adding new classification models/metrics and improving sklearn-wrapper behavior around missing values.

Changes:

• Adds new simple converters: ColumnArithmetic, ColumnConcat, NumericExpansion, and TypeCast (feature engineering + in-pipeline type conversion).
• Adds new ML components: XGBClassifier, LGBMClassifier, and BalancedAccuracy, including registration and test coverage for the new model/metric.
• Improves conversion/pipeline infrastructure: normalizes missing values passed to sklearn transformers, adds a small fit→transform pandas cache, and optimizes/clarifies converter-job dataset rebuilding and scoping.

Reviewed changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/back/models/test_tabular_class_models.py	Extends classifier smoke tests to cover new XGBoost/LightGBM model wrappers.
tests/back/metrics/test_classification_metrics.py	Adds tests for the new BalancedAccuracy metric (including sklearn reference comparison).
pyproject.toml	Adds lightgbm and xgboost runtime dependencies.
DashAI/back/tasks/classification_task.py	Registers BalancedAccuracy as compatible with classification tasks.
DashAI/back/models/scikit_learn/xgboost_classifier.py	Introduces XGBClassifier component + schema and DashAI mixin workaround for XGBoost MRO assumptions.
DashAI/back/models/scikit_learn/lightgbm_classifier.py	Introduces LGBMClassifier component + schema and DashAI mixin workaround for LightGBM MRO assumptions.
DashAI/back/models/scikit_learn/svc.py	Adds class_weight support to SVC schema.
DashAI/back/models/scikit_learn/sgd_classifier.py	Adds class_weight support to schema and forwards it into the underlying raw estimator.
DashAI/back/models/scikit_learn/random_forest_classifier.py	Adds class_weight schema field for RandomForestClassifier.
DashAI/back/models/scikit_learn/logistic_regression.py	Adds class_weight schema field for LogisticRegression.
DashAI/back/models/scikit_learn/linear_svc_classifier.py	Adds class_weight schema field and forwards it into the underlying raw estimator.
DashAI/back/models/scikit_learn/hist_gradient_boosting_classifier.py	Adds class_weight schema field for HistGradientBoostingClassifier.
DashAI/back/models/scikit_learn/extra_trees_classifier.py	Adds class_weight schema field for ExtraTreesClassifier.
DashAI/back/models/scikit_learn/decision_tree_classifier.py	Adds class_weight schema field for DecisionTreeClassifier.
DashAI/back/metrics/classification/balanced_accuracy.py	Adds the BalancedAccuracy classification metric implementation.
DashAI/back/job/predict_job.py	Adjusts how prediction outputs are merged/saved (drop any existing output column before adding predictions).
DashAI/back/job/converter_job.py	Optimizes dataset rebuild logic and avoids redundant column selection when no row-level scope is set.
DashAI/back/initial_components.py	Registers the new converters, models, and metric in the initial component set.
DashAI/back/converters/sklearn_wrapper.py	Normalizes missing values for sklearn, and caches the fit input DataFrame for reuse in transform when possible.
DashAI/back/converters/scikit_learn/simple_imputer.py	Improves output type preservation based on strategy/statistics; adds fit-time bookkeeping for typing.
DashAI/back/converters/category/feature_engineering.py	Adds a converter category base class for “Feature Engineering” converters.
DashAI/back/converters/simple_converters/column_arithmetic.py	Adds ColumnArithmetic converter (scope-derived operands + constant mode).
DashAI/back/converters/simple_converters/column_concat.py	Adds ColumnConcat converter (scope-derived operands + constant mode).
DashAI/back/converters/simple_converters/numeric_expansion.py	Adds NumericExpansion converter (log1p/square/sqrt derived columns).
DashAI/back/converters/simple_converters/type_cast.py	Adds TypeCast converter reusing dataset-upload type validation logic.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.