DSI-Bench — Gemma4-31B — sample reverse_hflip__row_000197

# Show the first and last frames to identify the horse and its general movement
show([InputImages[0], InputImages[-1]])

 [image: show_0001] 

# Segment all horses in the video
seg = tools.SAM3.segment_video_by_text(prompts=["horse"], labels=["horse"])

# Verify segmentation on first, middle, and last frames
fi_start = seg.frame_indices[0]
fi_mid = seg.frame_indices[len(seg.frame_indices)//2]
fi_end = seg.frame_indices[-1]

# Visualize masks
vis_start = seg.visualize(fi_start)
vis_mid = seg.visualize(fi_mid)
vis_end = seg.visualize(fi_end)

show([
    InputImages[0], vis_start, 
    InputImages[len(InputImages)//2], vis_mid, 
    InputImages[-1], vis_end
])

# Print number of objects detected
print(f"Number of horses detected: {seg.num_objects}")

 [image: show_0002] 
Number of horses detected: 1

# 1. Locate the horse in the foreground on the left
bbox_norm_str = vlm.locate(InputImages[0], "Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the horse in the foreground on the left. Reply with ONLY the numbers.")
print(f"VLM Bbox string: {bbox_norm_str}")

# 2. Parse and convert normalized coordinates to pixels
W, H = InputImages[0].width, InputImages[0].height
try:
    # Handle potential non-numeric characters in the response
    import re
    nums = re.findall(r"[-+]?\d*\.\d+|\d+", bbox_norm_str)
    bbox_norm = [float(x) for x in nums]
    
    if len(bbox_norm) == 4:
        px_bbox = tools.Geometry.normalized_to_pixel(bbox_norm, W, H)
        print(f"Pixel Bbox: {px_bbox}")
        
        # 3. Track this horse across the video
        seg = tools.SAM3.segment_video_by_box(boxes=[px_bbox], labels=["horse"])
        
        # 4. Verify the segmentation
        fi_start = seg.frame_indices[0]
        fi_mid = seg.frame_indices[len(seg.frame_indices)//2]
        fi_end = seg.frame_indices[-1]
        
        vis_start = seg.visualize(fi_start)
        vis_mid = seg.visualize(fi_mid)
        vis_end = seg.visualize(fi_end)
        
        show([
            InputImages[0], vis_start, 
            InputImages[len(InputImages)//2], vis_mid, 
            InputImages[-1], vis_end
        ])
        print(f"Successfully tracked horse. Number of objects: {seg.num_objects}")
    else:
        print(f"Failed to parse 4 coordinates from VLM response: {bbox_norm_str}")
except Exception as e:
    print(f"Error during segmentation: {e}")

[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the horse in the foreground on the left. Reply with ONLY the numbers.
[VLM A | locate] 415, 505, 486, 534
VLM Bbox string: 415, 505, 486, 534
Pixel Bbox: [318.71999999999997, 218.16, 373.248, 230.68800000000002]
 [image: show_0003] 
Successfully tracked horse. Number of objects: 1

# 1. Locate the center point of the horse in the foreground on the left
point_norm_str = vlm.locate(InputImages[0], "Give the (x, y) center coordinates in 0-1000 normalized scale for the horse in the foreground on the left. Reply with ONLY the numbers.")
print(f"VLM Point string: {point_norm_str}")

# 2. Parse and convert normalized coordinates to pixels
W, H = InputImages[0].width, InputImages[0].height
try:
    import re
    nums = re.findall(r"[-+]?\d*\.\d+|\d+", point_norm_str)
    point_norm = [float(x) for x in nums]
    
    if len(point_norm) == 2:
        px_point = tools.Geometry.normalized_to_pixel(point_norm, W, H)
        print(f"Pixel Point: {px_point}")
        
        # 3. Track this horse across the video using point prompt
        # points_per_object is a list of lists of points: [[ [x, y] ]]
        # point_labels_per_object is a list of lists of labels: [[ 1 ]]
        seg = tools.SAM3.segment_video_by_points(
            points_per_object=[[px_point]], 
            point_labels_per_object=[[1]], 
            labels=["horse"]
        )
        
        # 4. Verify the segmentation
        fi_start = seg.frame_indices[0]
        fi_mid = seg.frame_indices[len(seg.frame_indices)//2]
        fi_end = seg.frame_indices[-1]
        
        vis_start = seg.visualize(fi_start)
        vis_mid = seg.visualize(fi_mid)
        vis_end = seg.visualize(fi_end)
        
        show([
            InputImages[0], vis_start, 
            InputImages[len(InputImages)//2], vis_mid, 
            InputImages[-1], vis_end
        ])
        print(f"Successfully tracked horse. Number of objects: {seg.num_objects}")
    else:
        print(f"Failed to parse 2 coordinates from VLM response: {point_norm_str}")
except Exception as e:
    print(f"Error during segmentation: {e}")

[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the (x, y) center coordinates in 0-1000 normalized scale for the horse in the foreground on the left. Reply with ONLY the numbers.
[VLM A | locate] 520 453
VLM Point string: 520 453
Pixel Point: [399.36, 195.696]
 [image: show_0004] 
Successfully tracked horse. Number of objects: 1

# Get the intersection of frames available in both segmentation and reconstruction
common_frames = sorted(list(set(seg.frame_indices) & set(recon.frame_indices)))
print(f"Common frames for 3D analysis: {common_frames}")

# Extract the 3D trajectory
trajectory_3d = []
valid_frames = []

for fi in common_frames:
    centroid = seg.get_centroid_3d(recon, frame=fi, object=0)
    if centroid is not None:
        trajectory_3d.append(centroid)
        valid_frames.append(fi)

trajectory_3d = np.array(trajectory_3d)
print(f"Successfully extracted 3D centroids for {len(trajectory_3d)} frames.")

if len(trajectory_3d) >= 2:
    start_pos = trajectory_3d[0]
    end_pos = trajectory_3d[-1]
    displacement = end_pos - start_pos
    print(f"Start position: {start_pos}")
    print(f"End position: {end_pos}")
    print(f"Total displacement vector: {displacement}")
    print(f"Total distance moved: {np.linalg.norm(displacement):.2f} meters")
else:
    print("Not enough 3D points to compute displacement.")

Common frames for 3D analysis: [0, 6, 12, 18, 24, 30, 36, 42, 48, 54, 60, 66, 72, 78, 84]
Successfully extracted 3D centroids for 15 frames.
Start position: [ 0.2795395  1.6255766 -3.8305395]
End position: [ 1.080781   3.3300242 -7.225006 ]
Total displacement vector: [ 0.8012415  1.7044476 -3.3944666]
Total distance moved: 3.88 meters

# 1. Render BEV to visualize horse trajectory and camera motion
bev_vis = recon.render_bev(masks=seg, ego_trajectory=True)
show(bev_vis)

# 2. Determine horse's facing direction in the first frame
# The horse is facing the camera, so its heading is roughly opposite to the camera's forward vector.
facing_desc = vlm.ask_with_thinking(InputImages[0], "In the first frame, which direction is the horse in the foreground on the left facing? Is it facing towards the camera, away from the camera, or to the side?")
print(f"VLM Facing Description: {facing_desc}")

# 3. Quantitative analysis
# Camera pose at frame 0
pose_0 = recon.extrinsics[common_frames[0]]
cam_fwd_0 = pose_0[:3, 2] # Camera forward vector in world space

# If horse is facing the camera, its heading vector h is -cam_fwd_0
h = -cam_fwd_0
h = h / np.linalg.norm(h)

# Displacement vector from start to end
d = displacement

# Forward/Backward score: dot product of displacement and heading
score_fwd = np.dot(d, h)

# Right vector: h cross world-up [0, 1, 0]
world_up = np.array([0, 1, 0])
r = np.cross(h, world_up)
r = r / np.linalg.norm(r)

# Right/Left score: dot product of displacement and right vector
score_right = np.dot(d, r)

print(f"Forward score: {score_fwd:.3f}")
print(f"Right score: {score_right:.3f}")

# Determine dominant direction
if abs(score_fwd) > abs(score_right):
    direction = "Forward" if score_fwd > 0 else "Backward"
else:
    direction = "Right" if score_right > 0 else "Left"

print(f"Dominant relative direction: {direction}")

 [image: show_0005] 
[VLM Q | thinking] In the first frame, which direction is the horse in the foreground on the left facing? Is it facing towards the camera, away from the camera, or to the side?
[VLM A | thinking] The horse in the foreground on the left is facing towards the camera.
VLM Facing Description: The horse in the foreground on the left is facing towards the camera.
Forward score: -3.394
Right score: -0.801
Dominant relative direction: Backward

ReturnAnswer("D")

[ReturnAnswer] Answer submitted: D