[crepuscular-drizzle]

What and why

Expand native policy evaluation across LIBERO, RoboCasa365, RoboTwin 2.0, MolmoSpaces, SimplerEnv and EmbodiedBench EB-Navigation. Orbit registers the model and benchmark independently, freezes the protocol and samples for each run, and verifies the saved episode evidence on CPU.

• Fast-WAM and LeRobot add both first_frame and idm to all 40 LIBERO tasks across Spatial, Object, Goal and Long. The LeRobot 0.7.0 runtime supports SmolVLA, MolmoAct2 and LingBot-VLA 2.0.

• RoboTwin 2.0 registers the September evaluator alongside the original version, with 50 tasks each in Clean and Randomized. The September suites freeze the native instruction split and use separate sample identities.

• LingBot-VA runs RoboTwin Clean with native Cartesian controls, camera keyframes, feedback from the preceding action prediction, and per-episode cache resets. Protocol regressions cover early success, the horizon and inconsistent evidence.

• LingBot-VLA 2.0 serves the released 14-joint RoboTwin checkpoint. The authors' recipe uses RoboTwin 13c3c47. Compatible revisions can be evaluated as separate benchmark configurations.

• GR00T N1.7 refreshes the pinned inference source and checkpoint revisions for LIBERO and SimplerEnv and checks the policy image build manifest at registration.

• MolmoSpaces adds all seven MolmoBot v2 conditions, with 1,000 samples per condition from five catalogs. RandCam has its own DROID Policy registration with explicit camera semantics and shared checkpoint weights.

• LIBERO-Mem registers all 10 tasks and 200 published initial states. The CPU verifier checks ordered progress against the frozen BDDL branches using original object IDs. Reports include subgoal completion and full-sequence success.

• Xiaomi-Robotics-1 and GigaWorld-Policy-0.1 each register the 50 RoboCasa365 Target50 tasks with 50 seeded samples per task. The CPU verifier checks fixed replanning intervals of 16 and 20 controls, respectively.

• EB-Navigation registers all 300 released episodes across five subsets and runs the pinned AI2-THOR 5.0.0 simulator locally. SeeNav-Agent and Qwen2.5-VL-3B-Instruct share the annotated views, prompt, action feedback and stopping rule. The CPU verifier checks native action IDs, goal distance and stopping from saved evidence. The Trial timeline synchronizes the raw first-person and overhead videos with the annotated view sent to the policy. The SeeNav image build tests the pinned planner's image assembly, action parsing and fallback, feedback, and episode reset.

Validation

• pnpm verify and all 34 embodied adapter tests passed after the code changes in 5c3bcf3. The Python 3.11 SDK run had 167 passes and two skips. The documented Python 3.8 subset passed 64 tests.

• GPU samples for Fast-WAM Spatial in both modes, Fast-WAM Object in first_frame mode, LingBot-VA Clean, LingBot-VLA 2.0 on original Clean and Randomized, GR00T N1.7 on LIBERO Spatial, and DROID on MolmoBot v2 RandCam completed with matching independent CPU verification and rescore. RandCam reached its 303-control limit.

• Native reset, camera and first-control checks covered both RoboTwin September suites and all seven MolmoBot v2 conditions. Orbit registered all 7,000 MolmoBot v2 condition and sample combinations against the same offline asset bundle. The LeRobot image ran offline GPU predictions with SmolVLA and MolmoAct2.

• On 5c3bcf3, the refreshed GR00T N1.7 image ran one previously verified frozen SimplerEnv sample per endpoint on GPU 1: WidowX with Bridge weights (Spoon on Towel) and Google Robot with Fractal weights (Pick Coke Can). Both completed with native success, and independent CPU verification and rescore agreed on both samples.

• π₀.₅ × LIBERO Spatial. The upstream and Orbit runners agreed exactly on reset, policy inputs, full action chunks, physics traces and stopping for two frozen initial states of the first task. Each path shared the same compiled policy and used its own fresh key-0 stream per initial state. Independent CPU verification confirmed both Orbit episodes. The parity helper now uses the production policy loader.

• DROID × MolmoBot v2. Two initial states were paired across Filament and RandCam. Reset physics, instructions and wrist-camera frames matched within each pair. The exterior camera differed, and so did the first arm action chunk. All four rollouts reached the native step limit and passed independent CPU evidence verification.

• LingBot-VLA 2.0 on RoboTwin September passed configuration checks. Full-suite model statistics remain unmeasured.

• At bea5e6d, pnpm verify passed. Python 3.11 SDK discovery passed 174 tests with one skip, and the documented Python 3.8 subset passed 70 tests.

• A native LIBERO-Mem replay of 78 saved controls completed the goal and passed the revised CPU verifier. Regression tests cover both legal empty-basket branches and reject a branch with the wrong basket ID. Older LIBERO-Mem evidence requires a regenerated registration and sample plan and a new solve to capture the original predicates.

• One frozen Target50 sample per new model ran with released weights on GPU 0. Xiaomi-Robotics-1 and GigaWorld reached their 900- and 1,350-control horizons with valid native failure scores. Independent CPU verification and rescore agreed. Both saved traces passed the revised replanning checks, and synthetic early replanning was rejected for both protocols.

• EB-Navigation validation. All 300 published episodes passed native reset and render checks. One fixed-action replay per subset matched the pinned upstream environment on reward, distance, action validity and stopping. Each model ran two matching frozen samples, with agreement between CPU verification and rescore. A final SeeNav rerun confirmed the three synchronized recordings and Web timeline parsing. The CPU planner tests, pnpm verify, Python 3.11 SDK suite and documented Python 3.8 subset passed. Full-suite model statistics remain unmeasured.

Compatibility matrix · Task and validation records

• pnpm verify passes
• Task record / design / status updated when needed

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.