ERQA — Gemma4-31B — sample ERQA

Keyframes

Question

<image> <image> How did the camera move to go from the first image to the second image? Choices: A. translate right. B. translate left. C. pan left. D. pan right. Please answer directly with only the letter of the correct option and nothing else. This question is from the ERQA (Embodied Reasoning Question Answer) benchmark, which evaluates multimodal spatial reasoning and world knowledge in robotics scenarios. The question may contain <image> tags indicating where images should be viewed in context. Questions are multiple-choice with answer options embedded in the question text. Answer with ONLY the correct option letter (e.g., A, B, C, or D). Do not include any explanation or additional text in your final answer.

Ground truth D

SpatialClaw D ✓

Agent wins

No-tool Baseline

✗ Incorrect

To determine the camera movement between the first and second images, we analyze the positions of static objects in the scene:

Door Frame: In the first image, the right edge of the brown door frame is positioned roughly in the center of the image. In the second image, this edge has shifted to the left, occupying a smaller portion of the left side of the frame.
Smoke Detector/Sensor: The small white circular object on the ceiling also shifts to the left when moving from the first image to the second.
Right-side Elements: In the first image, only a small portion of the light source (top right) and the patterned tiles (bottom right) are visible. In the second image, more of these elements have entered the frame from the right, meaning they have moved leftward across the field of view.

When all objects in a scene shift to the left in the image, it indicates that the camera has moved to the right (translation) or rotated to the right (panning). Comparing the two images, the change in perspective is a consistent shift, which is characteristic of a translation to the right.

Therefore, the camera moved to the right.

Single-pass code

✗ Incorrect

Purpose

Determine the camera movement from the first image to the second image.

Reasoning

1. **Visual Analysis**: - In the first image, the door frame is on the left, a small white sensor is in the center-right, and a ceiling light is at the top right. - In the second image, all these elements have shifted to the left relative to the frame. The door frame has moved further left (partially out of frame), the sensor is more central, and the light is more visible and shifted left. 2. **Interpreting Motion**: When objects in a scene shift to the left in the image plane, it indicates that the camera has either moved (translated) to the right or rotated (panned) to the right. 3. **Translation vs. Pan**: Translation typically introduces parallax, where closer objects (like the door frame) move more across the image than distant objects (like the wall/sensor). In these images, the door frame moves significantly to the left, while the sensor and light also move left but the relative shift is consistent with a rightward translation of the camera. 4. **Conclusion**: The movement is a translation to the right.

Next Goal

Return the final answer.

Code

ReturnAnswer("A")

Output

[ReturnAnswer] Answer submitted: A

Final answer

Structured tool-call

✗ Incorrect

Plan

Task Analysis

The objective is to determine the specific type of camera motion between two frames: translation (right/left) or panning (right/left).

Coordinate System Resolution:

Translation Right/Left: Refers to the camera moving along its local X-axis (the axis pointing to the right of the image).
Pan Right/Left: Refers to the camera rotating around its local Y-axis (the axis pointing "up" relative to the camera sensor, though the world Y is gravity-aligned). A "pan right" means the camera's optical axis rotates toward the right.

This is a 3D spatial reasoning task requiring the distinction between linear motion (translation) and angular motion (rotation).

Information Needs

To distinguish between translation and panning, I need:

Camera Poses: The 3D position and orientation (extrinsics) of the camera for both frames.
Trajectory Data: The displacement vector between the two camera centers.
Rotation Data: The change in the camera's forward-facing vector.
Visual Evidence of Parallax: Translation causes parallax (objects at different depths move at different speeds in the image), whereas panning causes a uniform rotation of the scene.

Computation Plan

3D Reconstruction:
- Call recon = tools.Reconstruct.Reconstruct(InputImages) using both frames. This provides the camera-to-world (c2w) matrices.
Quantitative Pose Analysis:
- Extract the camera positions: pos0 = recon.extrinsics[0][:3, 3] and pos1 = recon.extrinsics[1][:3, 3].
- Extract the camera "right" vector for the first frame: cam_right0 = recon.extrinsics[0][:3, 0].
- Calculate the translation vector: delta_p = pos1 - pos0.
- Compute the projection of the translation onto the right axis: dist_right = np.dot(delta_p, cam_right0).
  - If dist_right is significantly positive, the camera translated right.
  - If dist_right is significantly negative, the camera translated left.
- Extract the camera "forward" vectors: fwd0 = recon.extrinsics[0][:3, 2] and fwd1 = recon.extrinsics[1][:3, 2].
- Calculate the angle between fwd0 and fwd1 using tools.Geometry.angle_between_vectors.
- Determine the rotation direction: Use the cross product of fwd0 and fwd1 relative to the camera's "up" vector to see if it's a pan right or left.
Visual Trajectory Verification:
- Render a BEV plot: recon.render_bev(ego_trajectory=True).
- Inspect the green/yellow dashed line. A clear line indicates translation; a point with a change in orientation indicates a pan.
Visual Reasoner Cross-Check:
- Call vlm.ask_with_thinking([InputImages[0], InputImages[1]], "Analyze the motion from the first image to the second. Does the scene exhibit parallax (objects shifting relative to each other based on depth), which indicates translation, or does it look like a rotation around a fixed point, which indicates panning? Based on this, did the camera move right, left, pan right, or pan left?").
Synthesis:
- Compare the magnitude of translation ($|delta_p|$) against the magnitude of rotation.
- If translation dominates and dist_right > 0 $\rightarrow$ Translate Right.
- If translation dominates and dist_right < 0 $\rightarrow$ Translate Left.
- If rotation dominates and it's a clockwise rotation around the Y-axis (from top view) $\rightarrow$ Pan Right.
- If rotation dominates and it's a counter-clockwise rotation around the Y-axis (from top view) $\rightarrow$ Pan Left.

CHECKLIST

[
  {"priority": "HIGH", "description": "Verify if the reconstruction succeeded and the camera poses are physically plausible"},
  {"priority": "HIGH", "description": "Distinguish between translation and rotation by comparing the magnitude of position change vs orientation change"},
  {"priority": "MEDIUM", "description": "Use BEV trajectory to visually confirm the direction of movement"},
  {"priority": "MEDIUM", "description": "Cross-validate the geometric result with the VLM's analysis of parallax"}
]

Fallbacks

Reconstruction Failure: If the images have too little overlap for tools.Reconstruct, I will rely entirely on vlm.ask_with_thinking to identify parallax. Parallax is the definitive visual cue for translation over panning.
Ambiguous Motion: If both translation and rotation occur, I will determine which component is dominant in terms of the resulting image shift.

Step 0

LLM · Tool Call

Purpose

Reconstruct the 3D scene to obtain camera poses (extrinsics) and positions for both frames.

Reasoning

To distinguish between translation and panning, I need the precise camera-to-world (c2w) matrices. `tools.Reconstruct.Reconstruct` will provide the camera positions and orientations, allowing me to calculate the displacement vector and the change in the forward-facing vector.

Next Goal

Analyze the camera poses to determine if the motion was primarily translation or rotation and in which direction.

Tool Call

{
  "tool": "tools.Reconstruct.Reconstruct",
  "args": {
    "frames": "InputImages"
  }
}

Feedback

=== Step 0 Execution Feedback ===

[SUCCESS] Code executed in 27.63s.

[Variables] New/changed:
  result_0: Reconstruction, frames=[0, 1]

[Tool Calls] 1 tool/VLM call(s) this step.

Step 1

LLM · Tool Call

Purpose

Calculate the 3D distance between the camera centers of the two frames to quantify translation.

Reasoning

By computing the Euclidean distance between the camera positions (the translation component of the c2w matrices), I can determine if the camera physically moved in space. A significant distance suggests translation, while a near-zero distance suggests a pure pan.

Next Goal

Calculate the angle between the camera's forward vectors to quantify rotation (panning).

Tool Call

{
  "tool": "tools.Geometry.euclidean_distance",
  "args": {
    "p1": "result_0.extrinsics[0][:3, 3]",
    "p2": "result_0.extrinsics[1][:3, 3]"
  }
}

Feedback

=== Step 1 Execution Feedback ===

[SUCCESS] Code executed in 0.00s.

[Variables] New/changed:
  result_1: float

[Tool Calls] 1 tool/VLM call(s) this step.

Step 2

LLM · Tool Call

**Purpose**: Calculate the angle between the camera's forward-facing vectors to quantify rotation. **Reasoning**: A pure pan involves rotation around the camera's center without translation. By calculating the angle between the forward vectors of the camera in the first and second frames, I can determine the magnitude of the rotation. Comparing this to the translation distance (from `result_1`) will help me decide if the motion was primarily a translation or a pan. **Next Goal**: Determine the direction of the motion (right/left) by projecting the translation onto the camera's right axis or analyzing the rotation. **Tool Call (translated)**: ```python result_2 = tools.Geometry.angle_between_vectors(v1=result_0.extrinsics[0][:3, 2], v2=result_0.extrinsics[1][:3, 2]) ```

Feedback

=== Step 2 Execution Feedback ===

[ERROR] Skipped: LLM response was not in the required format.

Step 3

LLM · Tool Call

Purpose

Visualize the camera trajectory in BEV to distinguish between translation and panning.

Reasoning

`render_bev` with `ego_trajectory=True` draws the camera's path. A clear line segment indicates translation, while a stationary point (or very short segment) with a change in orientation indicates a pan. This will provide a clear visual answer to whether the motion is translation or rotation and in which direction.

Next Goal

Based on the BEV, determine if the motion was primarily translation or rotation and identify the direction.

Tool Call

{
  "tool": "show",
  "args": {
    "image": "result_0.render_bev(ego_trajectory=True)"
  }
}

Execute

Output

 [image: show_0001]

show_0001

Feedback

=== Step 3 Execution Feedback ===

[SUCCESS] Code executed in 0.17s.

[Variables] New/changed:
  result_3: NoneType

[Inline Images] 1 image(s) attached below.
  Labels: result_0.render_bev(ego_trajectory=True)
[show() budget] 249/250 images remaining.

Step 4

LLM · Tool Call

**Purpose**: Use the VLM to analyze the images for parallax to distinguish between translation and panning. **Reasoning**: Translation causes parallax, where closer objects (like the door frame) shift more than distant objects (like the smoke detector). Panning causes a uniform rotation where all objects shift in a way consistent with a rotation around the camera center. By analyzing the relative motion of the door frame and the smoke detector, I can definitively tell if the motion was translation or panning. **Next Goal**: Synthesize the VLM's visual analysis with the BEV trajectory to select the final answer. **Tool Call (translated)**: ```json {"tool": "vlm.ask_with_thinking", "args": {"visual_input": ["InputImages[0]", "InputImages[1]"], "question": "Analyze the motion from the first image to the second. Does the scene exhibit parallax (objects shifting relative to each other based on depth), which indicates translation, or does it look like a rotation around a fixed point, which indicates panning? Specifically, compare the movement of the door frame on the left and the smoke detector on the ceiling. Based on this, did the camera move right, move left, pan right, or pan left?"}} ```

Feedback

=== Step 4 Execution Feedback ===

[ERROR] Skipped: LLM response was not in the required format.

Step 5

LLM · Tool Call

Purpose

Analyze the visual evidence and reconstruction results to determine the camera motion.

Reasoning

1. **Visual Analysis**: Comparing Image 0 to Image 1, all objects (the door frame, the smoke detector, and the ceiling light) shift to the left. A shift of objects to the left in the image frame is caused by either the camera translating to the right or the camera panning to the right. 2. **Parallax Check**: I observe that the door frame (which is closer to the camera) shifts more significantly to the left than the smoke detector (which is further away). This difference in apparent motion based on depth is called parallax, and it is a definitive indicator of translation rather than rotation (panning). 3. **BEV Analysis**: The BEV plot (`result_3`) shows the camera trajectory. The red dot (Camera 0) is at the origin, and the blue triangle (Camera 1) is positioned to the right of the red dot. This confirms a physical displacement of the camera center in the positive X direction (right). 4. **Conclusion**: The combination of the rightward shift in the BEV and the presence of parallax in the images confirms that the camera translated to the right.

Next Goal

Provide the final answer.

Tool Call

{
  "tool": "ReturnAnswer",
  "args": {
    "answer": "A"
  }
}

Execute

Output

[ReturnAnswer] Answer submitted: A

Answer

SpaceTools

✗ Incorrect

Tool decision

🛠️ Decided to use 2 tool(s)

tool_calls

[
  {
    "name": "vlm__detect_one",
    "arguments": {
      "image_index": 0,
      "obj_name": "smoke detector"
    }
  },
  {
    "name": "vlm__detect_one",
    "arguments": {
      "image_index": 1,
      "obj_name": "smoke detector"
    }
  }
]

Tool call

⚡ Executing vlm__detect_one...

tool_name

vlm__detect_one

arguments

{
  "image_index": 0,
  "obj_name": "smoke detector"
}

Tool result

✅ vlm__detect_one completed (with generated image)

tool_name

vlm__detect_one

result

Detected 1 instance(s) of 'smoke detector': [(0.7120000000000001, 0.41700000000000004)] Use $smoke_detector_detections to reference them.

Tool call

⚡ Executing vlm__detect_one...

tool_name

vlm__detect_one

arguments

{
  "image_index": 1,
  "obj_name": "smoke detector"
}

Tool result

✅ vlm__detect_one completed (with generated image)

tool_name

vlm__detect_one

result

Detected 1 instance(s) of 'smoke detector': [(0.621, 0.429)] Use $smoke_detector_detections to reference them.

Synthesizing

🧠 Synthesizing results from 2 tool(s)...

Reasoning

Final answer

Final response

SpatialClaw

✓ Correct

Plan

Task Analysis

The goal is to determine the camera's movement between two consecutive frames. The options are translation (right or left) or panning (left or right).

Coordinate System Resolution:

Translate Right: The camera's 3D position moves in the direction of its own local X-axis (right).
Translate Left: The camera's 3D position moves in the opposite direction of its local X-axis.
Pan Left: The camera rotates around its local Y-axis (yaw) to look toward its left.
Pan Right: The camera rotates around its local Y-axis (yaw) to look toward its right.

This is a 3D spatial reasoning task requiring the analysis of the camera's extrinsic parameters (pose) and visual verification of the scene's motion.

Information Needs

Camera Poses: The 4x4 camera-to-world matrices for both frames are needed to calculate the relative transformation.
Relative Transformation: The translation vector and rotation matrix between the two poses in the camera's local coordinate system.
Visual Evidence: Side-by-side images and a BEV trajectory plot to confirm whether the movement is a translation (characterized by parallax) or a pan (characterized by rotation around a point).

Computation Plan

3D Reconstruction: Perform reconstruction on the two frames to extract camera extrinsics. recon = tools.Reconstruct.Reconstruct(InputImages)
Pose Extraction: Get the camera-to-world matrices for the first and second frames. pose0 = recon.extrinsics[recon.frame_indices[0]] pose1 = recon.extrinsics[recon.frame_indices[1]]
Relative Motion Calculation: Compute the relative transformation $T_{rel} = T_0^{-1} \cdot T_1$.
- The translation component $t_{rel} = [x, y, z]$ describes the movement of the camera in its local frame at $t=0$.
- The rotation component $R_{rel}$ describes the change in orientation.
Quantitative Analysis:
- Translation: Check the magnitude of $t_{rel}$. If $|x|$ is dominant and significantly larger than the rotation effect:
  - $x > 0 \rightarrow$ Translate Right.
  - $x < 0 \rightarrow$ Translate Left.
- Rotation (Panning): If the rotation around the local Y-axis is dominant and translation is minimal:
  - Determine the direction of rotation (Yaw). A rotation that turns the camera's forward vector towards its local right is a Pan Right; towards local left is a Pan Left.
Visual Verification:
- show(InputImages[0], InputImages[1]): Inspect the images. Translation causes parallax (objects at different depths move at different speeds), while panning causes a more uniform shift of the scene.
- recon.render_bev(ego_trajectory=True): Visualize the camera's path in the world frame to see if it is a straight line (translation) or a pivot (rotation).
- vlm.ask_with_thinking([InputImages[0], InputImages[1]], "Describe the camera movement between these two images. Did the camera shift its position sideways, or did it stay in place and rotate its view?")

Verification Checklist

[
  {"priority": "HIGH", "description": "Verify that the relative translation vector x-component matches the visual shift in the images."},
  {"priority": "HIGH", "description": "Distinguish between translation and panning by checking for parallax in the visual evidence."},
  {"priority": "MEDIUM", "description": "Ensure the correct frame indices are used for the relative pose calculation."},
  {"priority": "MEDIUM", "description": "Cross-check the geometric result with the VLM's qualitative description of the motion."}
]

Fallbacks

Reconstruction Failure: If tools.Reconstruct fails due to insufficient overlap or motion, I will rely on vlm.ask_with_thinking and a manual analysis of feature point movement (using vlm.locate on static objects in both frames to see if they move consistently with a pan or translation).
Ambiguous Motion: If both translation and rotation are present, I will determine which component is dominant in terms of magnitude (meters vs. degrees) relative to typical camera movements.

Step 0

LLM

Purpose

Perform 3D reconstruction to extract camera extrinsics for the two frames.

Reasoning

To quantitatively determine if the camera translated or panned, I need its precise pose (position and orientation) in 3D space. The `tools.Reconstruct` tool provides these 4x4 camera-to-world matrices. I will also display the images to visually inspect the scene.

Next Goal

Extract the camera poses and compute the relative transformation between the first and second frames.

Execute

Code

# Perform 3D reconstruction on all available frames
recon = tools.Reconstruct.Reconstruct(InputImages)

# Display the two images for visual reference
show([InputImages[0], InputImages[1]])

# Print basic reconstruction info
print(f"Reconstructed frames: {recon.frame_indices}")
print(f"Metric scale: {recon.metric_scale}")

Output

 [image: show_0001] 
Reconstructed frames: [0, 1]
Metric scale: 1.0730397701263428

show_0001

Feedback

=== Step 0 Execution Feedback ===

[SUCCESS] Code executed in 5.11s.

[Output]
Reconstructed frames: [0, 1]
Metric scale: 1.0730397701263428

[Variables] New/changed:
  recon: Reconstruction, frames=[0, 1]

[Inline Images] 2 image(s) attached below.
  Labels: [InputImages[0], InputImages[1]]

[Tool Calls] 1 tool/VLM call(s) this step.
[show() budget] 248/250 images remaining.

Step 1

LLM

Purpose

Calculate the relative transformation between the camera poses of the first and second frames to determine the movement.

Reasoning

By computing $T_{rel} = T_0^{-1} T_1$, I can find the translation and rotation of the camera in its own local coordinate system. The translation component $(x, y, z)$ will tell me if the camera shifted (translate right/left), and the rotation will tell me if it panned.

Next Goal

Analyze the results to distinguish between translation and panning and determine the direction.

Execute

Code

import numpy as np

# Get absolute frame indices
fi0 = recon.frame_indices[0]
fi1 = recon.frame_indices[1]

# Get camera-to-world matrices
pose0 = recon.extrinsics[fi0]
pose1 = recon.extrinsics[fi1]

# Calculate relative transformation: T_rel = T0_inv * T1
# T_rel represents the movement of camera 1 relative to camera 0, in camera 0's local frame.
pose0_inv = np.linalg.inv(pose0)
t_rel_mat = pose0_inv @ pose1

# Extract translation vector
t_rel = t_rel_mat[:3, 3]
print(f"Relative translation (x, y, z) in meters: {t_rel}")

# Extract rotation matrix
r_rel = t_rel_mat[:3, :3]

# Calculate the angle of rotation (yaw) around the local Y-axis (down vector)
# In OpenCV/Reconstruct convention: Y is down.
# A pan right is a positive rotation around Y (looking right).
# The rotation angle can be found from the trace of the rotation matrix or specific elements.
# For a pure yaw (rotation around Y), R[0,0] = cos(theta), R[0,2] = sin(theta), R[2,0] = -sin(theta), R[2,2] = cos(theta)
yaw = np.arctan2(r_rel[0, 2], r_rel[2, 2])
yaw_deg = np.degrees(yaw)
print(f"Relative yaw (rotation around local Y) in degrees: {yaw_deg}")

# Visual verification: BEV trajectory
show(recon.render_bev(ego_trajectory=True))

Output

Relative translation (x, y, z) in meters: [0.02013801 0.00985252 0.00288236]
Relative yaw (rotation around local Y) in degrees: 7.973368881007472
 [image: show_0002]

show_0002

Feedback

=== Step 1 Execution Feedback ===

[SUCCESS] Code executed in 0.24s.

[Output]
Relative translation (x, y, z) in meters: [0.02013801 0.00985252 0.00288236]
Relative yaw (rotation around local Y) in degrees: 7.973368881007472

[Variables] New/changed:
  fi0: int
  fi1: int
  pose0: ndarray, shape=(4, 4), dtype=float64
  pose1: ndarray, shape=(4, 4), dtype=float64
  pose0_inv: ndarray, shape=(4, 4), dtype=float64
  t_rel_mat: ndarray, shape=(4, 4), dtype=float64
  t_rel: ndarray, shape=(3,), dtype=float64
  r_rel: ndarray, shape=(3, 3), dtype=float64
  yaw: float64, shape=(), dtype=float64
  yaw_deg: float64, shape=(), dtype=float64

[Inline Images] 1 image(s) attached below.
  Labels: recon.render_bev(ego_trajectory=True)
[show() budget] 247/250 images remaining.

Step 2

LLM

Purpose

Analyze the computed relative transformation to determine the dominant camera movement.

Reasoning

1. **Quantitative Analysis**: - The relative translation is approximately `[0.02, 0.01, 0.003]` meters. The lateral shift (x-axis) is only 2 cm. - The relative yaw (rotation around the local Y-axis) is `+7.97` degrees. - In typical indoor environments, a rotation of $\sim 8^\circ$ produces a much more significant visual shift than a 2 cm translation. For an object 1 meter away, an $8^\circ$ pan shifts the image by about 14 cm, whereas a 2 cm translation only shifts it by 2 cm. 2. **Visual Analysis**: - Comparing `InputImages[0]` and `InputImages[1]`, the entire scene (the door frame and the ceiling fixture) shifts to the left. A scene shifting left corresponds to the camera moving or rotating to the right. - The BEV plot shows the camera positions (red dot and blue triangle) are nearly coincident, but the orientation of the blue triangle is rotated relative to the red one. This confirms that the movement is primarily a rotation (pan) rather than a translation. 3. **Direction**: - A positive yaw (rotation around the local Y-axis in the provided convention) indicates the camera is turning its gaze toward its own right. - Therefore, the dominant movement is a pan right.

Next Goal

Provide the final answer.

Execute

Code

ReturnAnswer("D")

Output

[ReturnAnswer] Answer submitted: D

Answer

Evaluation

Ground truth: D · Agent: D