Multilog Analysis (Logx) by Jeffrey-Moon · Pull Request #52 · Penn-Electric-Racing/PER-Data-Analyzer

Jeffrey-Moon · 2026-07-17T22:36:10Z

Added tests to test the features but they were AI generated as a result of me being lazy... If you can think of edge cases, let me know.

Zero-Copy Parser Optimization

Replaced Polars group_by loops with an $O(N)$ contiguous boundary detector.
Slices telemetry values into zero-copy NumPy array views of master columns, dropping parsing time to 60ms and saving 280MB of RAM per log for a test log with size $736.9\text{MB}$.
Parses log start timestamps from the header.

Run Catalog & Scanner

Indexes directory log headers in under 1ms per file.
Implemented date filtering and lazy evaluation in check_any with automatic garbage collection to prevent OOM leaks.
Added unified path resolution for directories, wildcards, and list inputs.

Overlay Graphing & Downsampling

Added plot_comparison and compare_summary to overlay relative time coordinates
Implemented max_points parameter using zero-copy step-slicing.
Fixed step calculation using ceiling division (n_points + max_points - 1) // max_points to enforce a strict upper bound.

Hybrid Semantic Search

Added optional SentenceTransformer vector search
Caches embeddings in-memory with automatic invalidation.
Lazy-loads imports to execute on keyword search if PyTorch/models are missing offline

alex-yang-upenn

First pass

alex-yang-upenn · 2026-08-12T04:46:37Z

+   "source": [
+    "#### Parameterized Downsampling\n",
+    "\n",
+    "To speed up rendering or save network bandwidth, you can optionally restrict the number of points plotted using the `max_points` parameter. This applies a memory-efficient zero-copy step downsampling."


What is this about network bandwidth? Not sure that this is accurate.

That was more from the MCP perspective, I'll take that out

alex-yang-upenn · 2026-08-12T04:47:38Z

            parsing_errors_limit=parsing_errors_limit,
            verbose=verbose,
        )
+        self.file_path = filepath


Is this used anywhere?

If it's only for the sake of the run collection, I suspect your metadata Pydantic model can cover this functionality.

alex-yang-upenn · 2026-08-12T04:51:03Z

This file needs to be split up.

Make (or reuse!) a generic plotting function in the plotting module, then whatever lives in the Analyzer is a wrapper around the generic plotting function.

Try not to let plotting logic leak into the Analyzer, and vice versa, try not to write plotting functions that are application specific. It really blows up the size of our codebase for no good reason

alex-yang-upenn · 2026-08-12T04:52:02Z

+
+class RunMetadata(BaseModel):
+    """Internal model for telemetry run index metadata."""
+    file_path: str = Field(description="Absolute path to the log file")


Why is this a string object. Python should have Path objects right?

alex-yang-upenn · 2026-08-12T05:28:09Z

+    def __getitem__(self, index: int) -> Analyzer:
+        return self.get_run(index)
+
+    def filter(self, predicate: Callable[[RunMetadata], bool]) -> "RunCollection":


future annotations should fix this quotation marks

alex-yang-upenn · 2026-08-12T05:31:21Z

+        def predicate(item: RunMetadata) -> bool:
+            d = item.date
+            if d is None:
+                return False
+            if start_date and d < start_date:
+                return False
+            if end_date and d > end_date:
+                return False
+            return True


Lambda function please since you're only using it once / immediately passing it in.

Also, can the if statements be a bit more condensed and easier to reason with? Something like

return (
d is not None
and (start_date is None or d >= start_date)
and (end_date is None or d <= end_date)
)

Usually easier to think about the positive criteria rather than the negative criteria

alex-yang-upenn · 2026-08-12T05:34:59Z

+    def check_any(self, cpp_name: str, condition_fn: Callable[[np.ndarray], bool]) -> List[str]:
+        """Find log filenames in the collection where a condition holds for a given variable.
+
+        Evaluates runs lazily to keep memory usage bounded.
+
+        Parameters
+        ----------
+        cpp_name : str
+            C++ name of the variable to check.
+        condition_fn : Callable[[np.ndarray], bool]
+            Boolean condition function operating on the variable's numpy array values.
+
+        Returns
+        -------
+        List[str]
+            List of filenames where the condition evaluated to True.
+        """
+        matching_filenames = []
+        for i in range(len(self._items)):
+            item = self._items[i]
+            if item.analyzer is not None:
+                aly = item.analyzer
+            else:
+                try:
+                    aly = Analyzer(item.file_path, verbose=0)
+                except Exception:
+                    continue  # Skip unreadable file
+
+            if cpp_name in aly.data:
+                di = aly.data[cpp_name]
+                if condition_fn(di.value_np):
+                    matching_filenames.append(item.filename)
+        return matching_filenames


Maybe Im tweaking, but is this not just a filter function with a more powerful predicate? Surely you can accomplish all of this with filter, if you just write a predicate that actually accesses data from the Analyzer?

alex-yang-upenn · 2026-08-12T05:35:44Z

+        go.Figure
+            Plotly Figure with overlaid traces from all runs.
+        """
+        return plot_comparison(self.runs, cpp_name, max_points=max_points)


Reiterating my point above. I think you should make the sacrifice of greater complexity here in exchange for being able to have a generic plotting function (and perhaps even reuse existing generics)

alex-yang-upenn · 2026-08-12T05:37:08Z

+
+    def compare_summary(self, cpp_name: str) -> Dict[str, Dict[str, float]]:
+        """Compare variable statistics across all runs in the collection."""
+        return compare_summary(self.runs, cpp_name)


Reiterating what I said above about consolidation around Analyzer or SingleRunData having full ownership over how to generate a summary for a single cpp_name, and then everyone else constructs their summaries using that canonical summary as a building block

alex-yang-upenn · 2026-08-12T05:41:07Z

+        comparison[run_label] = {
+            "min": float(np.min(vals)),
+            "max": float(np.max(vals)),
+            "mean": float(np.mean(vals)),
+            "std": float(np.std(vals)),
+            "len": int(len(vals)),


This is a clear spot where you would want a strongly typed pydantic model

Jeffrey-Moon · 2026-08-14T00:23:13Z

+    Parameters
+    ----------
+    header_line : str
+        First line of the log file, e.g. ``"PER Log: Thu Jun 11 17:06:37 2026 v2.0"``.


Removed the csv_utils.py and moved the helpers to here. Also renamed the function names to be more specific.

Jeffrey-Moon · 2026-08-14T00:23:41Z

+COMPARISON_TIMESTAMP_UNIT = Timescale.US
+
+
+class RunMetadata(BaseModel):


Stored the filepaths in the RunMetaData

Jeffrey-Moon · 2026-08-14T00:24:03Z

+                )
+            )
+
+        return plot_single_axis(


reused plot_single_axis :)

Jeffrey-Moon · 2026-08-14T00:26:06Z

+                continue
+            yield run, analyzer
+
+    def filter(self, predicate: Callable[[RunMetadata], bool]) -> RunCollection:


renamed to properly represent what the functions are doing: filtering

Jeffrey-Moon · 2026-08-14T00:29:25Z

Tried to follow the format of analyzer.py and analyzer_factory.py

Jeffrey-Moon · 2026-08-14T00:32:06Z

 from .integrate import average_over_time_range


+class DataInstanceSummary(BaseModel):


Added new pydantic model for DI level summary as discussed

Jeffrey-Moon · 2026-08-14T00:36:04Z

Just edited minor points corresponding to the change of data_instance_summary

Jeffrey-Moon · 2026-08-14T00:36:26Z

The global path insert to fix pytest issue

Jeffrey-Moon · 2026-08-14T00:40:02Z

   - All numpy typing must be precise (e.g. `np.ndarray` is not allowed; use `NDArray[Float64]` instead). Use `numpy.typing` for this purpose.

- DO NOT use scoped imports, only use imports at the top of the file. When you have many imports, use asterisk to avoid long import lists
+- DO NOT use scoped imports, only use imports at the top of the file.


I changed this because it contrasts the precommit formatting.

Jeffrey-Moon · 2026-08-14T00:40:57Z

    rev: v2.3.1
    hooks:
      - id: autoflake
+        exclude: conftest\.py$


This was bc it kept removing the "unused imports" in conftest.py files

Jeffrey-Moon requested review from alex-yang-upenn and chinmaygovind July 17, 2026 22:38

Jeffrey-Moon force-pushed the multilog_analysis branch 2 times, most recently from 710b482 to 5623020 Compare July 17, 2026 23:37

chinmaygovind reviewed Jul 18, 2026

View reviewed changes

Comment thread perda/analyzer/__init__.py

Jeffrey-Moon force-pushed the multilog_analysis branch from 5623020 to ecedc94 Compare July 18, 2026 21:18

Jeffrey-Moon added 6 commits July 30, 2026 15:24

Plotting Max Number of Points Downsampling

204fa2c

Optimized Hybrid Search Using all-MiniLM-L6-V2

6efff65

Multilog Analysis and CSV Loading Optimization

7be2aaf

Downsampling Fix

ce9f0ec

Added Tests for New Features

86923ab

Formatting Fix

01f280b

Jeffrey-Moon force-pushed the multilog_analysis branch from ecedc94 to 01f280b Compare July 30, 2026 19:29

alex-yang-upenn reviewed Aug 12, 2026

View reviewed changes

Jeffrey-Moon added 6 commits August 12, 2026 13:24

Fix autoflake stripping conftest fixture wiring

7e266e9

Apply pre-commit hooks to existing files

4b60676

Deduplicate CSV parsing helpers

641844a

Return DataInstanceSummary model instead of printing

7dc37a6

Rework RunCollection and remove comparison module

bbd9c66

Add rootdir on sys.path for pytest import

a2f30c9

Jeffrey-Moon commented Aug 14, 2026

View reviewed changes

Comment thread pyproject.toml

Jeffrey-Moon Aug 14, 2026

Copy link
Copy Markdown

Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The global path insert to fix pytest issue

Jeffrey-Moon commented Aug 14, 2026

View reviewed changes

		COMPARISON_TIMESTAMP_UNIT = Timescale.US


		class RunMetadata(BaseModel):

		from .integrate import average_over_time_range


		class DataInstanceSummary(BaseModel):

Uh oh!

Conversation

Jeffrey-Moon commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Zero-Copy Parser Optimization

Run Catalog & Scanner

Overlay Graphing & Downsampling

Hybrid Semantic Search

Uh oh!

Uh oh!

alex-yang-upenn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alex-yang-upenn Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Jeffrey-Moon commented Jul 17, 2026 •

edited

Loading

alex-yang-upenn Aug 12, 2026 •

edited

Loading