Skip to content

feat: try_cast helper for stream map expressions #3693

Description

@ReubenFrankel

Feature scope

Inline mapping (stream maps, flattening, etc.)

Description

Problem

Stream map expressions are evaluated with simpleeval, which has no try/except and no lambda. This makes safe type coercion of dirty upstream data awkward: int(value), float(value), uuid.UUID(value), etc. raise on malformed input, and there is no way to catch that inside an expression — so a single bad record can error the whole stream instead of being coerced to a sensible default.

There's no clean workaround today. Registering a module doesn't help (simpleeval blocks module objects), and there's no way to wrap the call in try/except.

Proposal

Register a try_cast helper in the stream map function namespace that coerces a value and falls back to a default on failure, instead of raising:

def try_cast(value, caster, default=None, exceptions=(ValueError, TypeError)):
    """Coerce value via caster(value); on failure return default instead of raising."""
    try:
        return caster(value)
    except exceptions:
        return default

The key point that makes this work under simpleeval: pass the callable plus its argument (callables are values), and let the helper — which is ordinary Python, not sandboxed — perform the try/except. This sidesteps the missing-lambda limitation.

Example usage

stream_maps:
  my_stream:
    amount: try_cast(amount, float, default=0.0)   # garbage -> 0.0, not a crash
    count:  try_cast(count, int)                    # unparseable -> None

It also composes into __filter__ as an existence check:

    __filter__: try_cast(str(value), int) is not None

Design considerations

  • Narrow exception scope by default. Defaulting to (ValueError, TypeError) rather than bare Exception keeps genuine config bugs (e.g. a misspelled field name raising KeyError/NameError) loud instead of silently returning default on every row.
  • default=None, not False. A False default breaks the common ... is not None idiom (False is not None is True).
  • Caster name resolution. For try_cast(value, int) to work with a bare caster name, the caster must be resolvable in the expression context. To stay robust across simpleeval configurations, either register intended casters in both the names and functions maps, or accept the caster as a string label (e.g. try_cast(value, "int")) resolved internally against a whitelist — the latter also lets the SDK constrain which casts are permitted.

Follows this Slack discussion - would go hand-in-hand with #3694.

🤖 Generated with Claude

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions