ViewSpatial — Gemma4-31B — sample 5155

# Identify best frames for table, window, and whiteboard
question = "Which frames clearly show the following objects: 'table', 'window', and 'whiteboard'? Please list the frame indices (0-10) for each."
answer = vlm.ask_with_thinking(InputImages, question)
print(answer)

[VLM Q | thinking] Which frames clearly show the following objects: 'table', 'window', and 'whiteboard'? Please list the frame indices (0-10) for each.
[VLM A | thinking] The objects are clearly visible in the following frames:
- table: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
- window: 0, 1
- whiteboard: 8, 9, 10
The objects are clearly visible in the following frames:
- table: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
- window: 0, 1
- whiteboard: 8, 9, 10

# Select frames for segmentation
f_window = 0
f_table = 5
f_whiteboard = 9

# Segment objects
seg_window = tools.SAM3.segment_image_by_text(InputImages[f_window], "window", label="window")
seg_table = tools.SAM3.segment_image_by_text(InputImages[f_table], "table", label="table")
seg_whiteboard = tools.SAM3.segment_image_by_text(InputImages[f_whiteboard], "whiteboard", label="whiteboard")

# Verify masks
vis_window = seg_window.visualize(seg_window.frame_indices[0])
vis_table = seg_table.visualize(seg_table.frame_indices[0])
vis_whiteboard = seg_whiteboard.visualize(seg_whiteboard.frame_indices[0])

show([
    InputImages[f_window], vis_window,
    InputImages[f_table], vis_table,
    InputImages[f_whiteboard], vis_whiteboard
])

# Programmatic check for empty masks
for seg, label in [(seg_window, "window"), (seg_table, "table"), (seg_whiteboard, "whiteboard")]:
    fi = seg.frame_indices[0]
    mask = seg.get_mask(frame=fi, object=0)
    area = mask.sum()
    print(f"{label} mask area: {area} pixels")

 [image: show_0001] 
window mask area: 22271 pixels
table mask area: 31715 pixels
whiteboard mask area: 7895 pixels

# 1. Perform 3D Reconstruction
recon = tools.Reconstruct.Reconstruct(InputImages)

# 2. Refine Whiteboard Mask in frame 9
f_whiteboard = 9
img_wb = InputImages[f_whiteboard]
W, H = img_wb.width, img_wb.height

# Get bounding box for the whiteboard
wb_box_str = vlm.locate(img_wb, "Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the large white board on the wall. Reply with ONLY the numbers.")
print(f"Whiteboard normalized box string: {wb_box_str}")

# Parse the string into a list of integers
try:
    wb_box_norm = [int(x.strip()) for x in wb_box_str.split(',')]
except ValueError:
    # Fallback if there are other characters in the string
    import re
    wb_box_norm = [int(x) for x in re.findall(r'\d+', wb_box_str)]

# Convert to pixels
wb_box_px = tools.Geometry.normalized_to_pixel(wb_box_norm, W, H)
print(f"Whiteboard pixel box: {wb_box_px}")

# Segment using the box
seg_whiteboard_refined = tools.SAM3.segment_image_by_box(img_wb, wb_box_px, label="whiteboard")

# Verify the refined mask
vis_wb_refined = seg_whiteboard_refined.visualize(seg_whiteboard_refined.frame_indices[0])
show([img_wb, vis_wb_refined])

# Programmatic check
fi_wb = seg_whiteboard_refined.frame_indices[0]
wb_area = seg_whiteboard_refined.get_mask(frame=fi_wb, object=0).sum()
print(f"Refined whiteboard mask area: {wb_area} pixels")

[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the large white board on the wall. Reply with ONLY the numbers.
[VLM A | locate] 252 0 801 202
Whiteboard normalized box string: 252 0 801 202
Whiteboard pixel box: [193.536, 0.0, 615.168, 116.352]
 [image: show_0002] 
Refined whiteboard mask area: 39621 pixels

# 1. Get 3D Centroids
# Table (from frame 5)
fi_table = seg_table.frame_indices[0]
p_table = seg_table.get_centroid_3d(recon, frame=fi_table, object=0)

# Window (from frame 0)
fi_window = seg_window.frame_indices[0]
p_window = seg_window.get_centroid_3d(recon, frame=fi_window, object=0)

# Whiteboard (from frame 9)
fi_wb = seg_whiteboard_refined.frame_indices[0]
p_whiteboard = seg_whiteboard_refined.get_centroid_3d(recon, frame=fi_wb, object=0)

print(f"P_table: {p_table}")
print(f"P_window: {p_window}")
print(f"P_whiteboard: {p_whiteboard}")

# 2. Reference Frame Construction
# Front vector: from table to window
v_front = p_window - p_table
v_front_norm = v_front / np.linalg.norm(v_front)

# Right vector: Front x Up [0, 1, 0]
v_up = np.array([0, 1, 0])
v_right = np.cross(v_front_norm, v_up)
v_right_norm = v_right / np.linalg.norm(v_right)

# 3. Relative Position Calculation
v_target = p_whiteboard - p_table

dot_front = np.dot(v_target, v_front_norm)
dot_right = np.dot(v_target, v_right_norm)

front_back = "front" if dot_front > 0 else "back"
left_right = "right" if dot_right > 0 else "left"

print(f"Dot Front: {dot_front:.3f} ({front_back})")
print(f"Dot Right: {dot_right:.3f} ({left_right})")
print(f"Relative Position: {front_back}-{left_right}")

P_table: [-1.3598428   1.2031621  -0.76202554]
P_window: [ 0.9969112  1.7155263 -0.8370082]
P_whiteboard: [-0.71387523  0.50180614  2.120764  ]
Dot Front: 0.392 (front)
Dot Right: 2.902 (right)
Relative Position: front-right

# Create a combined segmentation for BEV
# Since SAM3 masks are per-frame, we can combine them into a list or a dictionary
# for render_bev if we provide the masks and labels.
# However, render_bev takes a PerFrameMask or (N, N_obj, H, W) array.
# I'll just use the existing masks and a list of labels.

# To use render_bev, I need masks for the same set of frames.
# I'll create a simple mask array for the reconstructed frames.
num_frames = recon.num_frames
H, W = recon.depth[recon.frame_indices[0]].shape
masks_bev = np.zeros((num_frames, 3, H, W), dtype=bool)
labels_bev = ["table", "window", "whiteboard"]

# Fill in the masks from our previous segmentations
# Note: we must map the absolute frame index to the local index in recon.frame_indices
frame_to_local = {fi: i for i, fi in enumerate(recon.frame_indices)}

# Table
fi_t = seg_table.frame_indices[0]
if fi_t in frame_to_local:
    masks_bev[frame_to_local[fi_t], 0] = seg_table.get_mask(frame=fi_t, object=0)

# Window
fi_w = seg_window.frame_indices[0]
if fi_w in frame_to_local:
    masks_bev[frame_to_local[fi_w], 1] = seg_window.get_mask(frame=fi_w, object=0)

# Whiteboard
fi_wb = seg_whiteboard_refined.frame_indices[0]
if fi_wb in frame_to_local:
    masks_bev[frame_to_local[fi_wb], 2] = seg_whiteboard_refined.get_mask(frame=fi_wb, object=0)

# Render BEV
bev_vis = recon.render_bev(masks=masks_bev, labels=labels_bev)
show(bev_vis)

# Qualitative VLM check
vlm_q = "Look at the images. If a person is sitting at the table and facing the window, where is the whiteboard located relative to them? (e.g., front-left, back-right, etc.)"
vlm_a = vlm.ask_with_thinking(InputImages, vlm_q)
print(f"VLM Qualitative Answer: {vlm_a}")

 [image: show_0003] 
[VLM Q | thinking] Look at the images. If a person is sitting at the table and facing the window, where is the whiteboard located relative to them? (e.g., front-left, back-right, etc.)
[VLM A | thinking] Based on the visual information across the frames:

1.  The **windows** are located on the right wall of the room (as seen in frames 0-4).
2.  The **whiteboard** is located on the wall opposite the camera's view in frames 8-10, which is perpendicular to the window wall.
3.  If a person is sitting at the table and facing the window (facing right), the whiteboard on the adjacent wall would be located to their **left**.
VLM Qualitative Answer: Based on the visual information across the frames:

1.  The **windows** are located on the right wall of the room (as seen in frames 0-4).
2.  The **whiteboard** is located on the wall opposite the camera's view in frames 8-10, which is perpendicular to the window wall.
3.  If a person is sitting at the table and facing the window (facing right), the whiteboard on the adjacent wall would be located to their **left**.

ReturnAnswer("B")

[ReturnAnswer] Answer submitted: B