Feature scope
Inline mapping (stream maps, flattening, etc.)
Description
Problem
Stream map expressions are evaluated with simpleeval, which has no try/except and no lambda. This makes safe type coercion of dirty upstream data awkward: int(value), float(value), uuid.UUID(value), etc. raise on malformed input, and there is no way to catch that inside an expression — so a single bad record can error the whole stream instead of being coerced to a sensible default.
There's no clean workaround today. Registering a module doesn't help (simpleeval blocks module objects), and there's no way to wrap the call in try/except.
Proposal
Register a try_cast helper in the stream map function namespace that coerces a value and falls back to a default on failure, instead of raising:
def try_cast(value, caster, default=None, exceptions=(ValueError, TypeError)):
"""Coerce value via caster(value); on failure return default instead of raising."""
try:
return caster(value)
except exceptions:
return default
The key point that makes this work under simpleeval: pass the callable plus its argument (callables are values), and let the helper — which is ordinary Python, not sandboxed — perform the try/except. This sidesteps the missing-lambda limitation.
Example usage
stream_maps:
my_stream:
amount: try_cast(amount, float, default=0.0) # garbage -> 0.0, not a crash
count: try_cast(count, int) # unparseable -> None
It also composes into __filter__ as an existence check:
__filter__: try_cast(str(value), int) is not None
Design considerations
- Narrow exception scope by default. Defaulting to
(ValueError, TypeError) rather than bare Exception keeps genuine config bugs (e.g. a misspelled field name raising KeyError/NameError) loud instead of silently returning default on every row.
default=None, not False. A False default breaks the common ... is not None idiom (False is not None is True).
- Caster name resolution. For
try_cast(value, int) to work with a bare caster name, the caster must be resolvable in the expression context. To stay robust across simpleeval configurations, either register intended casters in both the names and functions maps, or accept the caster as a string label (e.g. try_cast(value, "int")) resolved internally against a whitelist — the latter also lets the SDK constrain which casts are permitted.
Follows this Slack discussion - would go hand-in-hand with #3694.
🤖 Generated with Claude
Feature scope
Inline mapping (stream maps, flattening, etc.)
Description
Problem
Stream map expressions are evaluated with
simpleeval, which has notry/exceptand nolambda. This makes safe type coercion of dirty upstream data awkward:int(value),float(value),uuid.UUID(value), etc. raise on malformed input, and there is no way to catch that inside an expression — so a single bad record can error the whole stream instead of being coerced to a sensible default.There's no clean workaround today. Registering a module doesn't help (
simpleevalblocks module objects), and there's no way to wrap the call intry/except.Proposal
Register a
try_casthelper in the stream map function namespace that coerces a value and falls back to a default on failure, instead of raising:The key point that makes this work under
simpleeval: pass the callable plus its argument (callables are values), and let the helper — which is ordinary Python, not sandboxed — perform thetry/except. This sidesteps the missing-lambdalimitation.Example usage
It also composes into
__filter__as an existence check:Design considerations
(ValueError, TypeError)rather than bareExceptionkeeps genuine config bugs (e.g. a misspelled field name raisingKeyError/NameError) loud instead of silently returningdefaulton every row.default=None, notFalse. AFalsedefault breaks the common... is not Noneidiom (False is not NoneisTrue).try_cast(value, int)to work with a bare caster name, the caster must be resolvable in the expression context. To stay robust acrosssimpleevalconfigurations, either register intended casters in both the names and functions maps, or accept the caster as a string label (e.g.try_cast(value, "int")) resolved internally against a whitelist — the latter also lets the SDK constrain which casts are permitted.Follows this Slack discussion - would go hand-in-hand with #3694.
🤖 Generated with Claude