Help with Sim-to-Sim transfer to MuJoCo

[Lrogic]

Intro

Hi!

I am a graduate student at UBC using MuJoCo for humanoid manipulation and sim-to-sim transfer.

My setup

MuJoCo 3.10.1, Python, Linux
ManiSkill 3.0.1

My question

Problem

I am trying to transfer a policy trained in ManiSkill to MuJoCo for grasping an apple. When I run the same trained policy directly in MuJoCo, it fails. My understanding is that small differences in how the two simulators implement position actuation produce slightly different joint states. These errors accumulate, pushing the MuJoCo observations outside the policy’s training distribution and causing the behavior to diverge.

I matched the timestep, stiffness, damping, force limits, gravity compensation, and controller update frequency as closely as possible, but the trajectories still differed. See the Exact control and observation configuration section below for more details.

As a diagnostic, I used ManiSkill as a leader and MuJoCo as a follower. The policy runs in ManiSkill, and after each control step, MuJoCo receives the joint positions achieved by ManiSkill as its next position-actuator targets. For example, if ManiSkill reaches [1.2, 0.3], MuJoCo uses [1.2, 0.3] as its next target.

This improved the transfer, but contact-related joint-tracking errors remain large enough that MuJoCo often knocks the apple away instead of grasping it.

I also copied MuJoCo's object pose into the corresponding ManiSkill environment so that the policy could observe the follower's object state. This still did not produce reliable grasping.

I am trying to determine which differences between the two simulators, particularly position actuators, contact dynamics, gravity compensation, or robot parameters, are causing the remaining mismatch.

Exact control configuration

• Policy output: A 13-dimensional normalized action controlling six torso/right-arm joints and seven right-finger joints. Each action is converted into a per-control-step delta from the current measured joint position, with a configured range of ±0.03 rad.
• Control timing: The policy runs at 20 Hz. Both simulators run at 500 Hz, giving 25 physics substeps per policy action.
• Robot embodiment: A simplified Unitree G1 upper body without legs. The pelvis/root is fixed to the world. The policy controls the torso, right arm, and right hand; the left-arm, left-hand and leg joints are passive.
• ManiSkill actuation: PhysX articulation drives with $k_p = 1000$, $k_d = 100$, and a force limit of ±100 for all controlled arm and finger joints.
• MuJoCo actuation: Position actuators with the same nominal $k_p = 1000$, $k_d = 100$, and forcerange="-100 100".
• Gravity compensation: In ManiSkill 3.0.1, balance_passive_force=True is implemented by disabling gravity on all robot links. In MuJoCo, I set gravcomp="1" on the corresponding robot bodies. Gravity remains enabled for the apple.
• Leader–follower mode: Instead of applying the policy's delta action directly in MuJoCo, MuJoCo receives the joint positions actually achieved by ManiSkill at the end of the preceding control step and uses those as its next absolute position targets.

Potential issues and questions

Below are the main issues and possible sources of mismatch that I am currently considering. I would appreciate any feedback, corrections, or suggestions on these points or on the overall approach.

1. Stiffness, damping, and force limits

I use $k_p = 1000$, $k_d = 100$, and ±100 force limits for both the arm and finger joints, matching the ManiSkill G1 model. These values seem aggressive for low-inertia fingers. Are these values unusually high for grasping?

2. Other model mismatches

Potential differences include:

• Gravity compensation: ManiSkill disables gravity on the robot links, while MuJoCo uses gravcomp="1" on the corresponding robot bodies. These may produce similar generalized forces, but they are implemented differently.
• Collision geometry: The engines may generate different convex hulls or convex decompositions.
• Contact and friction: PhysX and MuJoCo use different contact, friction, restitution, and solver models.

3. Policy robustness

The original policy was trained without observation or action noise. I am now training with both. Are these sufficient, or should I also randomize actuator gains, control delay, friction, contact parameters, and object dynamics?

4. Force saturation and implicit damping

In a low-inertia single-joint test without contact, removing forcerange eliminated large velocity jumps and made MuJoCo’s joint trajectory more closely match PhysX.

The MuJoCo 3.5.0 changelog mentions a fix to actuator velocity derivatives when force is clamped by forcerange. Since I use MuJoCo 3.10.1, I assume this fix is included.

Is the intended behavior that a saturated actuator has zero local velocity derivative in the implicit update, thereby reducing its implicit damping contribution? What is the recommended way to approximate PhysX’s force-limited implicit PD drive in MuJoCo? Should I use a MuJoCo position actuator with kp, kv, and forcerange, or compute and clip the PD torque myself at every physics step and apply it through a motor actuator?

5. Overall transfer strategy

Beyond the specific sources of mismatch discussed above, I am unsure which overall transfer approach would be most effective and how best to diagnose the remaining contact failures. I would appreciate feedback on the following:

• The ManiSkill-leader/MuJoCo-follower approach performs substantially better than running the policy directly in MuJoCo, although small tracking errors during contact still cause failures. Does this seem like a reasonable transfer strategy to continue developing, or would it be better to focus on making the policy operate directly from MuJoCo observations?
• To reduce actuator mismatch, would it make sense to implement the same explicit PD controller outside both simulators and apply only its resulting torque commands? Alternatively, should I continue tuning each simulator’s native position actuators to try and produce similar closed-loop trajectories?
• Another option would be to retain the native controllers and train the policy to tolerate their differences through noise and domain randomization.
• Are there any MuJoCo solver settings or diagnostic quantities that would be particularly useful for identifying actuator mismatches that arise primarily during contact?

6. Other sim-to-sim advice

I would also appreciate any other tips for PhysX-to-MuJoCo transfer, including approaches or common issues that I may not have considered.

Minimal model and/or code that explain my question

Minimal model and/or code that explain my question

The complete setup spans a fairly large ManiSkill and MuJoCo codebase, so I have not attached all of it initially. For now, I have included a side-by-side video illustrating the leader–follower behavior:

mujoco_vs_maniskill.mp4

The top row shows the trained policy running in ManiSkill. The bottom row shows the MuJoCo follower receiving ManiSkill's achieved joint positions as its position-actuator targets.

If additional details would help, I would be happy to provide the relevant MJCF and assets, leader–follower script, controller configuration, or anything else that is helpful.

Confirmations

• I searched the latest documentation thoroughly before posting.
• I searched previous Issues and Discussions, I am certain this has not been raised before.

[Omc12]

I think your diagnosis is largely correct, but I would separate free-space actuator mismatch from contact mismatch.

Matching kp=1000, kd=100, timestep, and force limits does not make a PhysX articulation drive dynamically equivalent to a MuJoCo position actuator. So I agree that this can explain the accumulated tracking error before contact.

However, looking at the video, the two trajectories seem reasonably close until the hand interacts with the apple. The larger divergence happens during contact, which makes me suspect collision geometry, friction, contact constraints, and object dynamics are at least as important as the actuator implementation.

The experiment I would try next is to remove the actuator difference entirely: record one ManiSkill rollout and replay the same torque sequence in both simulators, first without contact and then with contact. For example:

tau = clip(kp * (q_target - q) - kd * qvel, -tau_max, tau_max)

If the trajectories match in free space but diverge at first contact, that gives you strong evidence that the remaining problem is contact/model mismatch rather than the position actuator itself.

I would also verify the actual physical parameters match: masses, COMs, inertias, joint axes, damping, armature, collision meshes, and self-collision filtering. In particular, for grasping, even small differences in fingertip/apple collision geometry can change the contact points and frictional wrench enough to produce exactly this kind of failure.

I would also test gravity separately, since ManiSkill's balance_passive_force=True and MuJoCo's gravcomp=1 are not implemented in exactly the same way.

So I would keep the ManiSkill-leader/MuJoCo-follower setup as a useful diagnostic, but use the same torque sequence in both simulators as the next controlled experiment. Once you know whether the divergence starts before or at contact, it becomes much easier to decide whether to tune actuators or focus on contact parameters.

[Lrogic]

Quick update on this:

I retrained the policy with domain randomization over actuator stiffness and damping, hand/object friction, observation noise, action jitter, object spawn pose, and small random perturbation forces.

This helped substantially. The policy can now run directly from MuJoCo observations/actions and reaches toward the object reasonably well, without using ManiSkill as a leader.

The remaining problem now seems very contact-specific. The MuJoCo and ManiSkill trajectories stay fairly close before contact, but as soon as the hand makes even light contact with the object in MuJoCo, the object often gets launched and the interaction becomes unstable. The same policy can make stable contact and grasp the object in ManiSkill.

I am currently logging qpos, qvel, and estimated joint torques in both simulators at physics-step resolution around first contact. So far, these quantities remain relatively close until contact begins, after which the MuJoCo trajectory diverges very quickly.

At this point this seems more like a contact/physics mismatch than a policy or free-space actuator mismatch, but I am not sure which MuJoCo parameters would be most useful to investigate. In particular, I am wondering whether I should focus on contact softness/stiffness, solver parameters, friction, collision geometry, timestep/integration settings, or something else.

Are there any MuJoCo contact or solver diagnostics/parameters you would recommend looking at first for this kind of "object gets launched on first contact" behavior?

I'm also attaching a video of the new setup, where the policy runs independently in Mujoco and Maniskill below.

policy_eval_in_mj_and_ms.mp4

@yuvaltassa , do you have any suggestions or pointers on this? I thought you might have some insight into the issue, and any help would be really appreciated. Sorry for calling you out directly!