VSI-Bench-U — Gemma4-31B — sample 5068

# Identify frames and locate objects
# Lamp: InputImages[123] is a great shot.
# Bookshelf: InputImages[1230] (around KF 11) seems good.
# Door: InputImages[0] or InputImages[3820] (KF 32).

# Let's refine the frames first using vlm.ask_with_thinking
frames_to_check = [InputImages[0], InputImages[123], InputImages[1230], InputImages[3820]]
vlm_query = "In these frames, identify which ones clearly show: 1) the door the robot starts at, 2) the lamp the robot faces initially, 3) the bookshelf destination. Return the frame indices from InputImages."
best_frames = vlm.ask_with_thinking(frames_to_check, vlm_query)
print(f"Best frames for objects: {best_frames}")

# Now locate them in specific frames
# Lamp in InputImages[123]
lamp_box = vlm.locate(InputImages[123], "Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the red lamp. Reply with ONLY the numbers.")
print(f"Lamp box: {lamp_box}")

# Bookshelf in InputImages[1230] (or similar)
bookshelf_box = vlm.locate(InputImages[1230], "Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the wooden bookshelf. Reply with ONLY the numbers.")
print(f"Bookshelf box: {bookshelf_box}")

# Door in InputImages[3820] (it's clearly visible here)
door_box = vlm.locate(InputImages[3820], "Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the brown door. Reply with ONLY the numbers.")
print(f"Door box: {door_box}")

# Visual check of the selected frames
show([InputImages[123], InputImages[1230], InputImages[3820]])

[VLM Q | thinking] In these frames, identify which ones clearly show: 1) the door the robot starts at, 2) the lamp the robot faces initially, 3) the bookshelf destination. Return the frame indices from InputImages.
[VLM A | thinking] The requested items are visible in the following frames:
1) The door the robot starts at: Frame 0
2) The lamp the robot faces initially: Frame 1
3) The bookshelf destination: Frame 2
Best frames for objects: The requested items are visible in the following frames:
1) The door the robot starts at: Frame 0
2) The lamp the robot faces initially: Frame 1
3) The bookshelf destination: Frame 2
[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the red lamp. Reply with ONLY the numbers.
[VLM A | locate] 300 110 635 825
Lamp box: 300 110 635 825
[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the wooden bookshelf. Reply with ONLY the numbers.
[VLM A | locate] 600 208 935 1000
Bookshelf box: 600 208 935 1000
[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not v

# Combine masks for BEV rendering
# I need to create a PerFrameMask object that contains all three objects
# Since I have individual masks, I'll manually create a combined one or just pass them as a list.
# The render_bev method can take a list of masks and labels.

# Let's create a combined mask for a single frame for simplicity, 
# or just use the existing PerFrameMasks if they were on the same frame.
# Actually, render_bev takes 'masks' as PerFrameMask or (N, N_obj, H, W).
# I will create a simple mask array for the reference frame.

ref_fi = recon.frame_indices[0] # Use first frame as reference
# Since my masks are on different frames, I can't easily combine them into one PerFrameMask 
# without tracking. Let's use a simpler approach: 
# I'll just render the BEV with the ego trajectory to see the room's general shape, 
# and then I'll use the 3D centroids to plot the paths.

# To make render_bev work with my specific objects, I'll use a dummy PerFrameMask 
# that just has the masks I've already found, but mapped to the ref_frame.
# Actually, the best way is to just use the centroids and a plot.

# Let's try to render BEV with the masks I have. 
# I'll just pass the individual masks as a list if possible, or just use the centroids.
# The render_bev documentation says it needs masks or ego_trajectory.
bev_vis = recon.render_bev(ego_trajectory=True)
show(bev_vis)

# Let's also plot the centroids on a 2D plane (X, Z) to reason about the turns.
plt.figure(figsize=(8, 8))
plt.scatter(door_centroid[0], door_centroid[2], c='blue', label='Door', s=100)
plt.scatter(lamp_centroid[0], lamp_centroid[2], c='red', label='Lamp', s=100)
plt.scatter(bookshelf_centroid[0], bookshelf_centroid[2], c='green', label='Bookshelf', s=100)
plt.xlabel('World X')
plt.ylabel('World Z')
plt.title('Room Layout (Top-Down)')
plt.legend()
plt.grid(True)
plt.axis('equal')
plt.show()

 [image: show_0002] 
 [image: Room Layout (Top-Down)] 

import numpy as np

def normalize(v):
    norm = np.linalg.norm(v)
    return v / norm if norm > 0 else v

# Centroids from previous steps
p_door = np.array([2.4115696, -1.2300391]) # X, Z
p_lamp = np.array([-0.05827279, -1.4638959]) # X, Z
p_bookshelf = np.array([2.1416948, -0.05152784]) # X, Z

# Initial facing: Door -> Lamp
v_init = normalize(p_lamp - p_door)

def simulate_path(turn1_type, turn2_type):
    curr_p = p_door.copy()
    curr_v = v_init.copy()
    
    # Turn 1
    if turn1_type == 'right':
        curr_v = np.array([curr_v[1], -curr_v[0]])
    elif turn1_type == 'left':
        curr_v = np.array([-curr_v[1], curr_v[0]])
    elif turn1_type == 'back':
        curr_v = -curr_v
        
    # Forward to wall (simplified as a small movement or just changing position)
    # In this room, turning back from the door hits the wall almost immediately.
    # We'll assume the robot stays roughly at p_door for the turn.
    
    # Turn 2
    if turn2_type == 'right':
        curr_v = np.array([curr_v[1], -curr_v[0]])
    elif turn2_type == 'left':
        curr_v = np.array([-curr_v[1], curr_v[0]])
    elif turn2_type == 'back':
        curr_v = -curr_v
        
    # Check if curr_v points towards the bookshelf
    vec_to_target = normalize(p_bookshelf - curr_p)
    alignment = np.dot(curr_v, vec_to_target)
    return alignment

# Options:
# A: Turn Back, Turn Right
# B: Turn Left, Turn Left
# C: Turn Left, Turn Right
# D: Turn Back, Turn Left

results = {
    "A": simulate_path('back', 'right'),
    "B": simulate_path('left', 'left'),
    "C": simulate_path('left', 'right'),
    "D": simulate_path('back', 'left'),
}

print(f"Alignment with bookshelf: {results}")
best_option = max(results, key=results.get)
print(f"Best option: {best_option}")

ReturnAnswer("D")

Alignment with bookshelf: {'A': -0.9914694067241677, 'B': -0.13033961611891837, 'C': 0.13033961611891837, 'D': 0.9914694067241677}
Best option: D
[ReturnAnswer] Answer submitted: D