feat: eval navsim

.gitignore

@@ -223,12 +223,11 @@ fusion_result.json
 *output*
 *kernel_meta/
 data/*.json
-maps
 data/metric_cache
-data/maps
 tmp
 navsim_v1
 nuscenes
 extra-info/
 pretrained_model/
-.DS_Store
+.DS_Store
+exp/

README.md

@@ -3,7 +3,7 @@
     <a href='https://github.com/YoucanBaby' target='_blank'>Yifang Xu</a><sup>1*</sup>&emsp;
     <a href='https://cuijh26.github.io/' target='_blank'>Jiahao Cui</a><sup>1*</sup>&emsp;
     <a href='https://github.com/fudan-generative-vision/WAM-Flow' target='_blank'>Feipeng Cai</a><sup>2*</sup>&emsp;
-    <a href='https://github.com/SSSSSSuger' target='_blank'>Zhihao Zhu</a><sup>1</sup>&emsp;
+    <a href='https://github.com/SSSSSSuger' target='_blank'>Zhihao Zhu</a><sup>1*</sup>&emsp;
     <a href='https://github.com/NinoNeumann' target='_blank'>Hanlin Shang</a><sup>1</sup>&emsp;
     <a href='https://github.com/isan089' target='_blank'>Shan Luan</a><sup>1</sup>&emsp;
 </div>
@@ -30,6 +30,7 @@
 
 
 ## 📰 News
+- **`2026/02/21`**: 🎉🎉🎉 WAM-Flow is accepted by [CVPR 2026](https://cvpr.thecvf.com/).
 - **`2026/02/01`**: 🎉🎉🎉 Release the pretrained models on [Huggingface](https://huggingface.co/fudan-generative-ai/WAM-Flow).
 - **`2025/12/06`**: 🎉🎉🎉 Paper submitted on [Arxiv](https://arxiv.org/pdf/2512.06112).
 
@@ -41,7 +42,7 @@
 | :----: | :----------------------------------------------------------------------------------------------------: | :--------: |
 |   ✅   | **[Release the SFT and inference code](https://github.com/fudan-generative-vision/WAM-Flow)**   | 2025.12.19 |
 |   ✅   | **[Pretrained models on Huggingface](https://huggingface.co/fudan-generative-ai/WAM-Flow)**    | 2026.02.01        |
-|   🚀   | **[Release the evaluation code](https://huggingface.co/fudan-generative-ai/WAM-Flow)**    | TBD |
+|   ✅   | **[Release the evaluation code](https://huggingface.co/fudan-generative-ai/WAM-Flow)**    | 2026.03.03 |
 |   🚀   | **[Release the RL code](https://github.com/fudan-generative-vision/WAM-Flow)**   | TBD |
 |   🚀   | **[Release the pre-processed training data](#training)**                                       | TBD        |
 
@@ -82,13 +83,14 @@ cd WAM-Flow
 Install dependencies:
 
 ```sh
-conda create --name wam-flow python=3.10
+conda create --name wam-flow python=3.9
 conda activate wam-flow
-pip install -r requirements.txt
+pip install -e ./nuplan-devkit
+pip install -e .
 ```
 
 
-### Model Download
+### Model
 
 Download models using huggingface-cli:
 
@@ -99,6 +101,24 @@ huggingface-cli download LucasJinWang/FUDOKI --local-dir ./pretrained_model/fudo
 ```
 
 
+### Dataset
+
+**NAVSIM**
+
+Please [download NAVSIM dataset](https://github.com/autonomousvision/navsim/blob/main/docs/install.md#2-download-the-dataset) and [run metric caching](https://github.com/autonomousvision/navsim/blob/main/docs/cache.md).
+
+
+
+
+### Evaluation
+
+**NAVSIM**
+
+```sh
+# Please change NAVSIM and METRIC_CACHE path
+sh scripts/evaluation/run_wam_flow_agent_pdm_score_evaluation.sh
+```
+
 
 ### Inference
 
@@ -137,4 +157,4 @@ The integration of Vision-Language-Action models into autonomous driving introdu
 
 
 ## 🤗 Acknowledgements
-We gratefully acknowledge the contributors to the [Recogdrive](https://github.com/xiaomi-research/recogdrive), [Janus](https://github.com/deepseek-ai/Janus), [FUDOKI](https://github.com/fudoki-hku/FUDOKI) and [flow_matching](https://github.com/facebookresearch/flow_matching) repositories, whose commitment to open source has provided us with their excellent codebases and pretrained models.
+We gratefully acknowledge the contributors to the [WAM-Diff](https://github.com/fudan-generative-vision/WAM-Diff), [RecogDrive](https://github.com/xiaomi-research/recogdrive), [Janus](https://github.com/deepseek-ai/Janus), [FUDOKI](https://github.com/fudoki-hku/FUDOKI) and [flow_matching](https://github.com/facebookresearch/flow_matching) repositories, whose commitment to open source has provided us with their excellent codebases and pretrained models.

config/pretrain.yaml

@@ -11,7 +11,7 @@ ckpt_path: ""
 train_llm_emb: false
 
 data_list: [
-  "path/to/llava_v1_5_mix665k_2.jsonl"
+  "path/to/llava_v1_5_mix665k_2.jsonl",
   "path/to/dataset_coda_lm.jsonl",
   "path/to/dataset_drivegpt4.jsonl",
   "path/to/dataset_lingoqa.jsonl",

download/download_maps.sh

@@ -0,0 +1,4 @@
+wget https://motional-nuplan.s3-ap-northeast-1.amazonaws.com/public/nuplan-v1.1/nuplan-maps-v1.1.zip
+unzip nuplan-maps-v1.1.zip
+rm nuplan-maps-v1.1.zip
+mv nuplan-maps-v1.0 maps

download/download_mini.sh

@@ -0,0 +1,22 @@
+wget https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_metadata_mini.tgz
+tar -xzf openscene_metadata_mini.tgz
+rm openscene_metadata_mini.tgz
+mv openscene-v1.1/meta_datas mini_navsim_logs
+rm -r openscene-v1.1
+
+for split in {0..31}; do
+    wget https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_sensor_mini_camera/openscene_sensor_mini_camera_${split}.tgz
+    echo "Extracting file openscene_sensor_mini_camera_${split}.tgz"
+    tar -xzf openscene_sensor_mini_camera_${split}.tgz
+    rm openscene_sensor_mini_camera_${split}.tgz
+done
+
+for split in {0..31}; do
+    wget https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_sensor_mini_lidar/openscene_sensor_mini_lidar_${split}.tgz
+    echo "Extracting file openscene_sensor_mini_lidar_${split}.tgz"
+    tar -xzf openscene_sensor_mini_lidar_${split}.tgz
+    rm openscene_sensor_mini_lidar_${split}.tgz
+done
+
+mv openscene-v1.1/sensor_blobs mini_sensor_blobs
+rm -r openscene-v1.1

download/download_navtrain.sh

@@ -0,0 +1,26 @@
+wget https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_metadata_trainval.tgz
+tar -xzf openscene_metadata_trainval.tgz
+rm openscene_metadata_trainval.tgz
+mv openscene-v1.1/meta_datas trainval_navsim_logs
+rm -r openscene-v1.1
+
+mkdir -p trainval_sensor_blobs/trainval
+for split in {1..4}; do
+    wget https://s3.eu-central-1.amazonaws.com/avg-projects-2/navsim/navtrain_current_${split}.tgz
+    echo "Extracting file navtrain_current_${split}.tgz"
+    tar -xzf navtrain_current_${split}.tgz
+    rm navtrain_current_${split}.tgz
+
+    rsync -rv current_split_${split}/* trainval_sensor_blobs/trainval
+    rm -r current_split_${split}
+done
+
+for split in {1..4}; do
+    wget https://s3.eu-central-1.amazonaws.com/avg-projects-2/navsim/navtrain_history_${split}.tgz
+    echo "Extracting file navtrain_history_${split}.tgz"
+    tar -xzf navtrain_history_${split}.tgz
+    rm navtrain_history_${split}.tgz
+
+    rsync -rv history_split_${split}/* trainval_sensor_blobs/trainval
+    rm -r history_split_${split}
+done

download/download_private_test_e2e.sh

@@ -0,0 +1,10 @@
+wget https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_metadata_private_test_e2e.tgz
+tar -xzf openscene_metadata_private_test_e2e.tgz
+rm openscene_metadata_private_test_e2e.tgz
+mv openscene-v1.1/meta_datas/ private_test_e2e_navsim_logs
+rm -r openscene-v1.1
+
+wget https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_sensor_private_test_e2e.tgz
+tar -xzf openscene_sensor_private_test_e2e.tgz
+rm openscene_sensor_private_test_e2e.tgz
+mv competition_test private_test_e2e_sensor_blobs

download/download_test.sh

@@ -0,0 +1,21 @@
+wget https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_metadata_test.tgz
+tar -xzf openscene_metadata_test.tgz
+rm openscene_metadata_test.tgz
+
+for split in {0..31}; do
+    wget https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_sensor_test_camera/openscene_sensor_test_camera_${split}.tgz
+    echo "Extracting file openscene_sensor_test_camera_${split}.tgz"
+    tar -xzf openscene_sensor_test_camera_${split}.tgz
+    rm openscene_sensor_test_camera_${split}.tgz
+done
+
+for split in {0..31}; do
+    wget https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_sensor_test_lidar/openscene_sensor_test_lidar_${split}.tgz
+    echo "Extracting file openscene_sensor_test_lidar_${split}.tgz"
+    tar -xzf openscene_sensor_test_lidar_${split}.tgz
+    rm openscene_sensor_test_lidar_${split}.tgz
+done
+
+mv openscene-v1.1/meta_datas test_navsim_logs
+mv openscene-v1.1/sensor_blobs test_sensor_blobs
+rm -r openscene-v1.1

download/download_trainval.sh

@@ -0,0 +1,21 @@
+wget https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_metadata_trainval.tgz
+tar -xzf openscene_metadata_trainval.tgz
+rm openscene_metadata_trainval.tgz
+
+for split in {0..199}; do
+    wget https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_sensor_trainval_camera/openscene_sensor_trainval_camera_${split}.tgz
+    echo "Extracting file openscene_sensor_trainval_camera_${split}.tgz"
+    tar -xzf openscene_sensor_trainval_camera_${split}.tgz
+    rm openscene_sensor_trainval_camera_${split}.tgz
+done
+
+for split in {0..199}; do
+    wget https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_sensor_trainval_lidar/openscene_sensor_trainval_lidar_${split}.tgz
+    echo "Extracting file openscene_sensor_trainval_lidar_${split}.tgz"
+    tar -xzf openscene_sensor_trainval_lidar_${split}.tgz
+    rm openscene_sensor_trainval_lidar_${split}.tgz
+done
+
+mv openscene-v1.1/meta_datas trainval_navsim_logs
+mv openscene-v1.1/sensor_blobs trainval_sensor_blobs
+rm -r openscene-v1.1

download/download_warmup_two_stage.sh

@@ -0,0 +1,3 @@
+wget https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/navsim-v2/navsim_v2.1.1_warmup_two_stage.tar.gz
+tar -xzvf navsim_v2.1.1_warmup_two_stage.tar.gz
+rm navsim_v2.1.1_warmup_two_stage.tar.gz

flow_matching/path/mixture.py

@@ -123,15 +123,13 @@ def __init__(self, mode, embedding_path):
         self.c = 3
         assert mode in ['image', 'text'], f"Unsupported mode probability path: {mode}"
         self.mode = mode
-        self.embedding_path = embedding_path
         self.embedding = self.get_embedding(embedding_path)
         self.embedding.weight.requires_grad = False
-        self.embedding = self.embedding
         torch.cuda.empty_cache()
 
     def get_embedding(self, embedding_path):
         # with torch.serialization.safe_globals([torch.nn.modules.sparse.Embedding]):
-        embedding = torch.load(embedding_path, map_location="cpu")
+        embedding = torch.load(embedding_path, map_location="cpu", weights_only=False)
         embedding.requires_grad_(False)
         torch.cuda.empty_cache()
         return embedding.cuda()

fudoki/janus/models/heading_mlp.py

@@ -0,0 +1,29 @@
+import torch
+import torch.nn as nn
+
+
+class TrajectoryHeadingMLP(nn.Module):
+    """
+    input: [B, 8, 2]
+    output: [B, 8, 1]
+    """
+    def __init__(self, hidden_dims=[512, 512, 256, 128], dropout=0.2):
+        super().__init__()
+        layers = []
+        input_dim = 8 * 2
+
+        for h_dim in hidden_dims:
+            layers.append(nn.Linear(input_dim, h_dim))
+            layers.append(nn.LayerNorm(h_dim))   
+            layers.append(nn.GELU())             
+            layers.append(nn.Dropout(dropout))   
+            input_dim = h_dim
+
+        layers.append(nn.Linear(input_dim, 8))
+        self.net = nn.Sequential(*layers)
+
+    def forward(self, traj):
+        x = traj.reshape(traj.size(0), -1)      # [B, 8, 2] → [B, 16]
+        out = self.net(x)
+        out = torch.tanh(out) * 3.14159
+        return out.unsqueeze(-1)                # [B, 8, 1]

navsim/agents/abstract_agent.py

@@ -0,0 +1,110 @@
+from abc import abstractmethod, ABC
+from typing import Dict, Union, List
+import torch
+import pytorch_lightning as pl
+
+from navsim.common.dataclasses import AgentInput, Trajectory, SensorConfig
+from navsim.planning.training.abstract_feature_target_builder import AbstractFeatureBuilder, AbstractTargetBuilder
+
+
+class AbstractAgent(torch.nn.Module, ABC):
+    """Interface for an agent in NAVSIM."""
+
+    def __init__(
+        self,
+        requires_scene: bool = False,
+    ):
+        super().__init__()
+        self.requires_scene = requires_scene
+
+    @abstractmethod
+    def name(self) -> str:
+        """
+        :return: string describing name of this agent.
+        """
+        pass
+
+    @abstractmethod
+    def get_sensor_config(self) -> SensorConfig:
+        """
+        :return: Dataclass defining the sensor configuration for lidar and cameras.
+        """
+        pass
+
+    @abstractmethod
+    def initialize(self) -> None:
+        """
+        Initialize agent
+        :param initialization: Initialization class.
+        """
+        pass
+
+    def forward(self, features: Dict[str, torch.Tensor]) -> Dict[str, torch.Tensor]:
+        """
+        Forward pass of the agent.
+        :param features: Dictionary of features.
+        :return: Dictionary of predictions.
+        """
+        raise NotImplementedError
+
+    def get_feature_builders(self) -> List[AbstractFeatureBuilder]:
+        """
+        :return: List of target builders.
+        """
+        raise NotImplementedError("No feature builders. Agent does not support training.")
+
+    def get_target_builders(self) -> List[AbstractTargetBuilder]:
+        """
+        :return: List of feature builders.
+        """
+        raise NotImplementedError("No target builders. Agent does not support training.")
+
+    def compute_trajectory(self, agent_input: AgentInput) -> Trajectory:
+        """
+        Computes the ego vehicle trajectory.
+        :param current_input: Dataclass with agent inputs.
+        :return: Trajectory representing the predicted ego's position in future
+        """
+        self.eval()
+        features: Dict[str, torch.Tensor] = {}
+        # build features
+        for builder in self.get_feature_builders():
+            features.update(builder.compute_features(agent_input))
+
+        # add batch dimension
+        features = {k: v.unsqueeze(0) for k, v in features.items()}
+
+        # forward pass
+        with torch.no_grad():
+            predictions = self.forward(features)
+            poses = predictions["trajectory"].squeeze(0).numpy()
+
+        # extract trajectory
+        return Trajectory(poses)
+
+    def compute_loss(
+        self,
+        features: Dict[str, torch.Tensor],
+        targets: Dict[str, torch.Tensor],
+        predictions: Dict[str, torch.Tensor],
+    ) -> torch.Tensor:
+        """
+        Computes the loss used for backpropagation based on the features, targets and model predictions.
+        """
+        raise NotImplementedError("No loss. Agent does not support training.")
+
+    def get_optimizers(
+        self,
+    ) -> Union[torch.optim.Optimizer, Dict[str, Union[torch.optim.Optimizer, torch.optim.lr_scheduler.LRScheduler]]]:
+        """
+        Returns the optimizers that are used by thy pytorch-lightning trainer.
+        Has to be either a single optimizer or a dict of optimizer and lr scheduler.
+        """
+        raise NotImplementedError("No optimizers. Agent does not support training.")
+
+    def get_training_callbacks(self) -> List[pl.Callback]:
+        """
+        Returns a list of pytorch-lightning callbacks that are used during training.
+        See navsim.planning.training.callbacks for examples.
+        """
+        return []