DSI-Bench — Gemma4-31B — sample reverse__row_000109

Task Analysis The objective is to determine the 3D movement of a character relative to their own starting orientation and location. This is an object-perspective motion problem. I need to identify the character, track their 3D position over time, determine their initial facing direction (forward vector), and then project their total displacement vector onto this local coordinate system.

• Target Variable: Direction of 3D displacement (Forward, Backward, Left, Right).
• Coordinate System: Object-relative (Local) 3D space. The origin is the character's starting position, and the axes are defined by the character's starting orientation.
• Answer Format: Multiple choice (A, B, C, or D).

Information Needs

1. Character Identification: Which object in the scene is the "character"?
2. 3D Trajectory: A sequence of 3D world coordinates for the character's center of mass across the clip.
3. Starting Orientation: The world-space vector representing the direction the character is facing at the first frame.
4. Displacement Vector: The vector from the character's starting 3D position to their ending 3D position.

Computation Plan

1. Character Identification and Segmentation:

   • Use show(InputImages[0]) and vlm.ask_with_thinking(InputImages[0], "Identify the main character in this image. Describe their appearance.") to confirm the target.
   • Use tools.SAM3.segment_video_by_text(prompts=["character"], labels=["character"]) to track the character across all frames.
   • Verify the segmentation quality by calling show(seg.visualize(fi)) for the first, middle, and last frames of the sequence.

2. 3D Scene Reconstruction:

   • Call recon = tools.Reconstruct.Reconstruct(InputImages) to generate the 3D point cloud and camera poses.

3. Trajectory Extraction:

   • For every frame index fi in seg.frame_indices, compute the 3D centroid: pos = seg.get_centroid_3d(recon, frame=fi, object='character').
   • Store these as a list of coordinates trajectory_3d.

4. Orientation Grounding:

   • Use vlm.ask_with_thinking(InputImages[0], "In the first frame, which direction is the character facing? Describe it relative to the camera (e.g., facing away from camera, facing left, etc.).").
   • Convert this qualitative direction into a world-space vector char_fwd.
     • Method: Get the camera pose at frame 0 (pose = recon.extrinsics[InputImages.frame_indices[0]]). Use cam_fwd = pose[:3, 2] and cam_right = pose[:3, 0]. If the VLM says "facing away", char_fwd is roughly cam_fwd. If "facing right", char_fwd is roughly cam_right.
   • Define the local right vector char_right as the cross product of the world-up vector [0, 1, 0] and char_fwd (or simply orthogonal to char_fwd in the ground plane).

5. Motion Analysis:

   • Calculate the total displacement vector: motion_vec = trajectory_3d[-1] - trajectory_3d[0].
   • Compute the projection of motion_vec onto the local axes:
     • dot_fwd = np.dot(motion_vec, char_fwd)
     • dot_right = np.dot(motion_vec, char_right)
   • Classification:
     • If dot_fwd is the dominant component and positive $\rightarrow$ Moving forward (B).
     • If dot_fwd is the dominant component and negative $\rightarrow$ Moving backward (A).
     • If dot_right is the dominant component and positive $\rightarrow$ Moving right (C).
     • If dot_right is the dominant component and negative $\rightarrow$ Moving left (D).

6. Visual Cross-Validation:

   • Render a BEV plot: recon.render_bev(masks=seg).
   • Inspect the trajectory line (red $\rightarrow$ blue) and compare its direction with the character's initial facing direction observed in the first frame.

Verification Checklist

CHECKLIST

[
  {"priority": "HIGH", "description": "Verify that the SAM3 masks accurately track the character throughout the video without drifting to other objects."},
  {"priority": "HIGH", "description": "Confirm the starting orientation vector (char_fwd) correctly reflects the character's facing direction in the first frame."},
  {"priority": "MEDIUM", "description": "Check for 3D centroid stability; ensure no sudden jumps in trajectory_3d that would skew the displacement vector."},
  {"priority": "MEDIUM", "description": "Cross-reference the quantitative dot-product result with the qualitative trajectory seen in the BEV render."}
]

Fallbacks

• Segmentation Failure: If segment_video_by_text fails to produce masks, use vlm.locate on the first frame to get a bounding box and use tools.SAM3.segment_video_by_box.
• Reconstruction Noise: If the 3D centroids are too noisy, use vlm.ask_with_thinking on a sequence of frames (e.g., first, middle, last) to describe the character's movement relative to the environment (e.g., "The character starts at the door and walk