Polars integration using ta-lib

[meet-brad-ch]

Great job guys on the project.
I created a wrapper for the ta-lib to use it with polars: https://github.com/meet-brad-ch/polars-talib

[mario4tier]

@meet-brad-ch Thanks, particularly with doing it right (with ta-abstract). A good test for it will be the coming v0.8.1 which adds >40 new TA functions 😄

Help me understand what would be the best long term strategy for polar. I do not know the answer yet. Inviting @mrjbq7 and @Yvictor to the discussion.

===

I assume the scope of this adaptor is for python devs only. A solution for Rust devs is a whole other subject. Correct me if wrong.

ta-lib-python uses always an intermediate numpy layer, so anything that has a "to_numpy" adaptor already works (including polars/pandas etc...).

Now would a polar specific adaptor (direct to C-FFI instead of through numpy layer) be significantly more performant to justify a separate python facade to ta-lib?

Should the essence of this wrapper be folded into ta-lib-python? (reduce ecosystem confusion).

Feedback very welcome, and I think some benchmark would be helpful to identify tradeoffs.

[mrjbq7]

Thats kind of rad integration, I have some comments -- I guess as a usability tool that's amazing.

but I would also be interested in a Python or C library change that took polars dataframes and had a zero copy pass reference to the backend for the computation and pre-allocated the result array back into the data frame in a more efficient way.

So I view this as

1. Improved developer experience (e.g. plta.sma(timeperiod=20).alias("sma20"))

2. Improved performance due to understanding what types (polars, pandas, numpy) we are being called with and where their output is expected to go.

Happy to take PRs if you'd like to contribute any of that.

[meet-brad-ch]

@mario4tier thanks. And yes, 0.8.1 with more than 40 new functions is exactly the test I want for this design.

On scope: you're right. This is for Python people using Polars. Rust users will use the Rust crate directly once it ships, and a Polars-rs integration would be a separate, much smaller project.

Is the direct C call faster? Honestly, no.

I ran the three routes side by side on the same machine: Polars 1.44, TA-Lib 0.7.1, 24 threads, median of 10 runs. The symbol rows were each measured in a fresh process. All three give identical values - the plugin's test suite checks all 161 functions against ta-lib-python bit for bit.

scenario	polars-talib plugin	ta-lib-python on to_numpy	ta-lib-python via map_batches
1 indicator, 1M rows	2.6 ms	2.6 ms	2.7 ms
8 indicators in one select, 1M rows	13.3 ms	33.2 ms	34.3 ms
8 indicators, 10M rows	129 ms	344 ms	341 ms
1 indicator over 2 000 symbols	11.3 ms	8.9 ms, hand-written loop	35 ms
1 indicator over 20 000 symbols	72 ms	54 ms, hand-written loop	289 ms

Your intuition about the numpy layer holds: for one indicator on one series, going straight to C buys nothing. to_numpy is zero-copy for a dense float column, and the plugin borrows the column the same way, so the two routes cost the same. I should admit the first release of the plugin took about 7 ms here, not 2.6. It copied the input, filled the whole output with NaN, and looked the function up twice per call. I fixed all three - the table shows the numbers after the fix.

Where the plugin does win is everything Polars does around the call, and that's the part I care about:

• Independent expressions run in parallel. Eight indicators in one select run about 2.5 times faster.
• Under .over("symbol") it's 3 to 4 times faster than a map_batches wrapper. The wrapper calls back into Python once per group, and holds the GIL while it does.
• It's an expression, so it lives inside lazy and streaming plans with no Python step in the middle.

@mrjbq7, on zero copy and the PR offer

Your framing is the right one - know what type you were called with, and where the result goes. On the input side, though, I think ta-lib-python is already there. It accepts polars.Series and polars.DataFrame today, and to_numpy() on a dense float column is a view, not a copy. The first row of the table is the proof. Pre-allocating the output into the frame would be a nicer API. But it saves one 8 MB buffer at most - a fraction of a millisecond.

The second and third wins above need a Polars expression, so they need a plugin. The first one is reachable from ta-lib-python with a small change. That's the PR I'd like to offer: release the GIL around the C call. If I read generate_func.py right, it emits retCode = lib.TA_XXX(...) with the GIL held. The arguments are all C pointers and ints, and the output arrays are allocated before the call. So wrapping that one line in with nogil: should be safe. With that, Polars' map_batches route and anyone calling from several threads would overlap the C loops. Polars does run map_batches calls in parallel once the GIL is free - I checked with a UDF that sleeps. My estimate is that the 8-indicator map_batches case drops from 33 ms toward the plugin's 13 ms. I'd rather measure than guess, so the PR would come with the benchmark.

The per-group cost would stay: a Python call per symbol is what it is, and only an expression avoids it. So I'd frame it this way. ta-lib-python with the GIL released becomes a good no-Rust option for Polars and pandas users. The plugin stays the option for people who want TA-Lib inside the query engine. If that sounds right, I'll open the GIL PR first. I'll also push the benchmark script to the polars-talib repo, so the comparison is reproducible on your side.

Should the plugin be folded into ta-lib-python?

I share the worry about a split, but I'd keep the code separate. ta-lib-python is Cython over numpy. This is a Rust plugin bound to the Polars plugin ABI. Different build tools, different release rhythms, and I don't see what either project would gain from carrying the other's.

What actually kept the ecosystem split small for me wasn't code, it was the abstract interface. All 161 functions run through one Rust expression that reads ta_abstract, and no function is written by hand. That's why 0.8.1 should cost me a submodule bump and a stub regeneration. It's also why I was so happy to find ta_abstract in the first place.

The Rust port

This one makes my life easier, not harder. Everything in polars-talib that knows about C is three files: the build script, the FFI declarations, and the safe layer over ta_abstract. If abstract_api carries the same metadata, those three files go and the crate takes their place. The expression, the Python API generation, the stubs, and the tests all stay. And with no C compiler in the build, Linux and macOS wheels become easier.

So my one real ask

Please treat the abstract interface as a stable, first-class contract in both the C library and the Rust crate. Keep the same metadata in both: inputs, parameter defaults, output names, lookback, group, hint. That contract is what makes a binding like this one cheap to maintain. Where it lives, under the TA-Lib organization or on its own, matters much less to me. If you'd want it under the organization once it sits on the Rust crate, I'm open to that.

[mario4tier]

@mrjbq7 I concur with @meet-brad-ch for nogil... coincidently last week the AI recommended that this is the low hanging fruit for ta-lib-python performance for its users.

ta-lib-python could have a multi-thread model matching C: TA_Initalize/TA_Shutdown and settings global API should remain under GIL, but calling the TA functions concurrently should be allowed.

Tweaking global settings and calling TA functions concurrently should be documented as "undefined behavior", and we drop protecting the user from it.

All IMHO.

---

@meet-brad-ch should be benchmark also closer to 1000 inputs (more typical), and will help make the "call cost" more visible. I would like to see the numbers with nogil, so multi-thread is no longer a factor.

Overall, I am curious to see perf focusing on generated static layer (ta-lib-python) versus a dynamic call (polar-talib), that might influence how we move forward.

To answer your ask: I am being careful to not change the abstract ABI for C (Rust will be frozen in the coming months, still tweaking).

[meet-brad-ch]

@mario4tier thanks, that answers my ask. Not changing the C abstract ABI is all I need, and a Rust freeze in the coming months is fine by me.

On the threading model: I agree with every point. Initialize, shutdown and the global settings stay under the GIL. TA function calls run concurrently. We document a settings change during a running call as undefined behavior, and we stop protecting people from it. That matches the C library as it is: the indicator functions read the globals with no lock, and the library guards only init and shutdown. So it's the honest contract.

Per-call cost at 1 000 rows, single thread

You're right that small inputs are the typical case, and they make the call cost visible. Same machine, one Polars thread, median of 7 timed loops, microseconds per call:

rows	talib.EMA (generated, direct call)	abstract.EMA	polars-talib select(ema)	polars-talib select(supersmoother)	select(close + 0.5)	select(map_batches(EMA))
100	0.8	4.1	16.1	13.3	9.7	15.2
1 000	2.4	5.8	17.5	18.5	9.9	16.6
10 000	17.6	21.1	33.2	68.8	10.8	32.2

What I read from it:

• ta-lib-python's generated layer is very cheap. A whole talib.EMA call on 100 rows, Python call included, is under a microsecond. The Python abstract layer adds about 3 µs.
• A Polars select costs about 10 µs before any plugin runs. That's the query engine, and it's the same for a plain arithmetic expression.
• The plugin's own dispatch adds about 3.5 µs to that. supersmoother is the control here: same plugin machinery, no TA-Lib call, and its loop costs about what EMA's does at 100 rows.
• The dynamic call through the abstract interface costs about 3 µs. Your generated direct call, Python overhead included, costs under 1 µs. The difference is the parameter holder, the calls that fill it, and the TA_CallFunc dispatch. So the answer to "generated versus dynamic" is: the dynamic path loses a few microseconds per call, and nothing per row.

So at 1 000 rows and one thread, the plugin and a map_batches wrapper cost the same inside Polars. Both are about 7 times the cost of calling talib.EMA directly. That's the price of being inside the query engine. It is worth paying only when the engine does something for you: parallel expressions, .over() groups, lazy plans. For a single small call, a plain talib.EMA is the right tool, and I'd say so in the docs.

Numbers with nogil

The nogil numbers will come with the GIL pull request to ta-lib-python, in its description. The single-thread numbers above shouldn't change with nogil, which is why I measured them that way first. What changes is the map_batches column of the multi-thread table from my earlier post. I'll rerun that with the patched build so the comparison is fair.

@mrjbq7 I'll open the PR against generate_func.py as discussed.

[mrjbq7]

I keep thinking beyond the Python objects and down to the Rust/C level. If we can elide the python nonsense I bet the polars expression overhead would get lower.

[meet-brad-ch]

Let me look :-)

[meet-brad-ch]

@mrjbq7 I had the same hunch, so I measured where the microseconds go. Single thread, 100 rows, microseconds per call:

what runs	µs
talib.EMA on a numpy array	0.8
Series + 0.5, the Python binding with no query engine	2.2
pl.select(pl.lit(1.0)), the query engine with no frame at all	10.6
one more trivial expression inside an existing select	5.5
one more supersmoother expression, plugin but no TA-Lib call	8.8
one more ema expression, plugin plus the abstract-interface call	12.2

There is no Python left to elide. Once select starts, the plugin path is Rust and C down to the loop. The kwargs arrive as bytes and a Rust deserializer reads them. The series cross the plugin boundary through the Arrow C data interface. Python is the 1 µs or so between the second row and the first.

The cost sits in Polars itself. About 10 µs is the engine planning and running a query, and it costs the same with no data at all. Every extra expression costs about 5.5 µs of engine scheduling before any plugin code runs. After that, the plugin's own boundary costs about 3 µs, and the dynamic call through the abstract interface another 3.

So a lower-level plugin wouldn't move the big numbers. It could shave the 3 µs boundary a little, and a generated direct call instead of TA_CallFunc would shave the other 3. The 10 µs and the 5.5 µs per expression belong to Polars, and only Polars can lower them. That's fine by me: they are the price of a query plan, and a query plan is the reason to be there. For a small array and one indicator, talib.EMA at 0.8 µs is the right tool, and nothing built on expressions will get near it.

[mrjbq7]

Thats not the thing to compare to -- the thing would be a Rust script using Rust polars calling Rust ta-lib and measuring that time without Python, making sure its zero-copy and zero-conversion.

[meet-brad-ch]

@mrjbq7 fair, so I wrote that program. Rust only: polars-rs calling the TA-Lib C library, input borrowed from the series buffer, one allocation for the output, no conversion, no Python in the process. One thread, microseconds per call:

rows	direct C call	abstract call	select(lit), no frame	select(close + 0.5)	select(ema), direct	select(ema), abstract	one more ema expression
100	0.1	0.2	5.4	7.5	7.2	7.3	4.5
1 000	1.7	1.7	5.4	7.7	8.8	9.0	5.9
10 000	16.8	16.9	5.3	9.0	24.3	24.7	25.4

You were right about Python. The engine alone plans and runs an empty query in 5.4 µs. Through the Python binding the same query costs about 10. So half of the fixed cost is the binding, and a Rust user does not pay it.

The other half stays. Each extra expression costs about 4.5 µs in pure Rust. That is Polars deciding what to run and running it, and no lower level removes it.

One correction to my last post. The abstract interface does not cost 3 µs. In Rust it costs 0.1 µs more than the direct call. The 3 µs I saw in Python is the plugin deserializing its kwargs twice, once at plan time and once at run time. That one is mine to fix.

So one EMA at 100 rows costs 0.1 µs as a bare C call. Inside Polars it costs 7.2 µs from Rust and 16.1 µs from Python.

[mario4tier]

Thanks that answer the direct versus abstract call... no significant difference (thanks for checking this). I did independently came to the same conclusion (and the abstract call cost is insignificant compare to the polar/query processing).

=====
For ta-lib-python, it seems that .to_numpy() already works faster for pandas/polar users when they organize their data in contiguous array. Which I recommend anyway so that the output remains bit-exact compared to TA-Lib C/Rust/Java/Python etc....

polars-talib introduce processing in "chunk/segment". That means "unstable period" injected here and there... so we loose the "bit exact" quality. I expect many users will just say "it's broken" and you might have to defend this often... no matter how well you document it 🤔

In short, I think that polars-talib have different trade-offs than ta-lib-python. polars-talib for where its extra query/expressiveness is favored (like @meet-brad-ch was saying I think), while simple arrays performance and "bit-exact" is more the ta-lib-python objectives.

Correct me if wrong, I'm learning as we go here.

The nogil should be done regardless 😄

[meet-brad-ch]

@mario4tier agreed on the positioning, and yes, nogil regardless.

On chunks and segments, let me separate what the plugin does from what a user can do with it. The first part is stronger than you think. The second part is the real risk.

The plugin itself never segments. Polars is not a partitioned engine like Dask or Spark. The plugin registers its expression as a whole-column function, not a row-by-row one, so Polars hands it the whole column. If the column sits in several memory chunks, the plugin joins them into one contiguous buffer before TA-Lib sees it. And the query optimizer does not move a filter or a slice below the expression. Computing an EMA and then filtering rows still computes on the full history. I checked all of that today with EMA, RSI, ATR, MACD and SAR on a million rows. I ran them through the streaming engine, on a frame stored in 15 chunks, and with a filter and a slice after the expression. Every result was bit-identical to one whole-array talib call. The test suite already asserts the same for all 161 functions. It runs them on float64 and float32, on a frame with leading NaN rows and a null, and on TA-Lib's 252-bar reference data. Bit-exact is the plugin's oracle, not a goal.

What does restart the unstable period is the user's own partitioning: .over("symbol"), a group_by, or a filter before the expression. Each segment then warms up on its own. A numpy user who slices first gets the same result. In a dataframe that partition is one method call away, so it will happen more often. That is your "it's broken" report, and I agree it will come. The defence is the test suite plus one clear line in the docs: the function sees exactly the rows of the segment you give it, and nothing before them.

On .to_numpy(): yes. For a contiguous float column it is a view. The first row of my earlier table shows ta-lib-python on that view costs the same as the plugin. Your recommendation to keep the data contiguous is the right one for both.

[mario4tier]

After digging a bit more found that .over("symbol") would NOT segment. In most common scenario .over() will work properly because polars-talib specify to polars that the data must be fed in whole columns.

It would need something like .over(["symbol","years"]), in which case the slicing is explicit in the user code.

What I fear were "silent/implicit" segmentation and now it seems this was just not a true problem.

Frankly, waking up this morning and not sure where to go next with all this. Should it be on its own or a convenience integrated with ta-lib-python? Not sure anymore.

====

I think also @mrjbq7 is hinting that more work is needed to evaluate for a Rust adaptor (For Rust users?) or PyO3+Rust (for python users?)

Not sure, so taking a break to hear more from you two.

[mrjbq7]

Not trying to make work for either of you guys, I'm trying to understand why this feels slow despite the code looking elegant. And then trying to go down a layer for two theoretical users: 1) of Polars and TA-Lib that is only using Rust and not Python and 2) of Python who wants to have the same speed that lower layer would yield.

I don't have time to dig into it this week myself, though.