LowMemoryCondition when using Event House with CostSummary.kql.pbit

[lundejd]

I'm trying to update to using Fabric Event House and switch away from storage, but am getting a LowMemoryCondition when I try go back and re-process past months. I can consume about 2 months total using Number of Months, anything higher gives the error. With previous versions of the storage version of the Power BI file I could set a date range instead of number of months, e.g. Jan 1, 2025 to Jan 31, 2025 and process just that month. I'm wondering if that would help get around the error. (which seems to be a limitation of using Event Hub). Our cost data is a good 20M records a month. I need to process everything back to 2014 eventually so number of months as the only filter is not helpful.

[flanakin]

Are you still hitting this? This is very strange. I've never seen this with the latest implementation. The last time I remember seeing this was when we tried to split the Costs table into a star model. I think this usually happens when you summarize with a lot of group by columns. I'm assuming you didn't change anything, so I'm not sure what could be causing it.

[lundejd]

I'm trying to bring the Event house data into a dataflow so I can further manipulate it with some splitting of costs we have between business groups. I used to pull right from Azure blob but it seems you are doing a lot of the manipulations in fabric event house now so I'm trying to feed from that - but now I can't pull more than the latest month. With dataflow and an older version of the toolkit I could pull February 2014 if I wanted and manipulate it, pushing it back into our lakehouse for consumption by Power BI. I could show 10 years of Azure spend month by month without much effort that way but the more advanced the toolkit gets the less I seem to be able to use it.

My current flow: MS Cost Management exports->Finops toolkit blob->Dataflow step one (basically your Power BI code)->Dataflow step 2 (our customized splitting and manipulating) -> Lakehouse then consume with Power BI. I have 2014 through to today so if I upgrade I need to be able to re-run all those months again. I can't do that without setting a date range which seems to have disappeared.

I can't use "number of months" with the 20 million rows of data per month we have, it limits me to the last month, maybe 2 no matter what. I'm stuck with version 6 of Finops toolkit right now. I'd love to get to 12 but I don't see how it's going to work unless I dig in and put back the date range part.

[lundejd]

I'm a bit confused when it comes to the Event House vs. the old way of using storage. What enhancements do we get over the other method? The reason I ask is that the event house seems to run just fine, but we can't manipulate the data further because we can't pull from it without getting memory errors.

What we normally do once we get a clean ingestion is run another job that processes cost splits based on another table we have. for example, we might want to take a series of AI resource groups and split them by percentage to our business units. The way we've done this so far is to have one dataflow pull from a shortcut and place it into a Lakehouse database (this based on your storage .pbit re-created as a dataflow). A second dataflow then runs on that data and does our internal manipulations.

With the event house I don't see how we can do it since we can't even pull a full month of cost data without a memory error, so I'm just curious why we would even continue to try. What would we be missing?

Our next thought is to create a notebook that copies the data out of the event house back into a Lakehouse so we can pull it without the memory restrictions.

We have tens of millions of lines a month and people want to see at least a year back if not longer for forecasting and budgeting. We also report on projects that have been running for years, so we have built out all our cost data dating back to 2014 month by month. This wasn't possible via Power BI alone, but using a live connection into fabric makes this possible. Now we can see years of data and publish reports others can view. One of the most valuable things is to present the year of cost data to an app team so they can see their costs changing over time and can drill in to find out why. (Microsoft's built-in split capabilities ignores the RI costs and is worthless to us. They don't realize that some companies want to allocate the amortized data, so we have to build that out ourselves.)

I mentioned earlier we're still on a much older version of the FinOps toolkit because it still works but we want to upgrade to move to the newer focus data and enhanced columns.

here is the initial dataflow and then some of the manipulations we run in the second pass.