MMSI-Video — Gemma4-31B — sample question_0184

# Select a representative set of frames to analyze the layout
frames_to_analyze = [
    InputImages[0],   # Start
    InputImages[111], # Dining area (around key frame 37)
    InputImages[144], # Kitchen area (around key frame 48)
    InputImages[188]  # End
]

# Ask VLM to identify the best frames for each marker
question = """
Based on these frames, please identify:
1. Which frame best shows the round dining table and the floor-to-ceiling window?
2. Which frame best shows the kitchen (e.g., the stove or the island)?
3. Which frame best shows the living room (e.g., the sofa)?
Please provide the frame index relative to the provided list (0-3) and a brief description of why.
"""

answer = vlm.ask_with_thinking(frames_to_analyze, question)
print(answer)

# Also show these frames to visually confirm
show(frames_to_analyze)

[VLM Q | thinking] 
Based on these frames, please identify:
1. Which frame best shows the round dining table and the floor-to-ceiling window?
2. Which frame best shows the kitchen (e.g., the stove or the island)?
3. Which frame best shows the living room (e.g., the sofa)?
Please provide the frame index relative to the provided list (0-3) and a brief description of why.

[VLM A | thinking] 1. **Frame 1**: This frame clearly shows a round dining table with chairs and large floor-to-ceiling sliding glass doors leading outside.
2. **Frame 2**: This frame shows the kitchen, including a large white island and a stove with a range hood.
3. **Cannot determine from the images**: A sofa or a clear view of the living room is not visible in any of the provided frames.
1. **Frame 1**: This frame clearly shows a round dining table with chairs and large floor-to-ceiling sliding glass doors leading outside.
2. **Frame 2**: This frame shows the kitchen, including a large white island and a stove with a range hood.
3. **Cannot determine from the images**: A sofa or a clear view of the living room is not visible in any of the provided frames.
 [image: show_0001] 

# 1. Retry sofa segmentation in a better frame
frame_living_new = InputImages[149]
seg_living = tools.SAM3.segment_image_by_text(frame_living_new, "white sofa", label="living")

# 2. Perform 3D Reconstruction
# Subsample InputImages to stay under 64 frames (189 / 3 = 63)
recon_frames = InputImages[::3]
recon = tools.Reconstruct.Reconstruct(recon_frames)

# 3. Verify masks and print areas
masks_to_verify = {
    "table": (seg_table, 111),
    "window": (seg_window, 111),
    "kitchen": (seg_kitchen, 144),
    "living": (seg_living, 149)
}

for label, (seg, fi) in masks_to_verify.items():
    mask = seg.get_mask(frame=fi, object=0) # Using index 0 since text prompt usually returns 1 object or we take the first
    area = mask.sum()
    print(f"{label} mask area at frame {fi}: {area} pixels")

# Visual verification of the sofa mask
show([frame_living_new, seg_living.visualize(149)])

table mask area at frame 111: 2922 pixels
window mask area at frame 111: 26109 pixels
kitchen mask area at frame 144: 70297 pixels
living mask area at frame 149: 5107 pixels
 [image: show_0002] 

import numpy as np

# 1. Get the sofa mask using VLM grounding on frame 147
frame_147 = InputImages[147]
W, H = frame_147.width, frame_147.height
bbox_norm_str = vlm.locate(frame_147, "Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the white sofa in the background. Reply with ONLY the numbers.")
print(f"Sofa bbox norm: {bbox_norm_str}")

# Parse coordinates and convert to pixels
try:
    cleaned_bbox = bbox_norm_str.replace('[', '').replace(']', '').replace('(', '').replace(')', '').replace(',', ' ')
    bbox_coords = [int(float(x)) for x in cleaned_bbox.split()]
    px_bbox = tools.Geometry.normalized_to_pixel(bbox_coords, W, H)
    seg_living = tools.SAM3.segment_image_by_box(frame_147, px_bbox, label="living")
    print("Sofa segmentation successful.")
except Exception as e:
    print(f"Sofa segmentation failed: {e}")
    # Fallback to a known working frame if possible, but 147 is a multiple of 3.
    # If this fails, I'll have to rethink the living room marker.
    seg_living = None

if seg_living:
    # 2. Extract 3D centroids
    p_table = seg_table.get_centroid_3d(recon, frame=111, object=0)
    p_window = seg_window.get_centroid_3d(recon, frame=111, object=0)
    p_kitchen = seg_kitchen.get_centroid_3d(recon, frame=144, object=0)
    p_living = seg_living.get_centroid_3d(recon, frame=147, object=0)

    print(f"P_table: {p_table}")
    print(f"P_window: {p_window}")
    print(f"P_kitchen: {p_kitchen}")
    print(f"P_living: {p_living}")

    # 3. Compute Cardinal Directions
    u_south = p_window - p_table
    u_south /= np.linalg.norm(u_south)
    u_north = -u_south
    u_up = np.array([0, 1, 0])
    u_east = np.cross(u_up, u_south)
    u_east /= np.linalg.norm(u_east)

    def get_direction(p_target, p_ref, u_north, u_east):
        v_rel = p_target - p_ref
        ns = np.dot(v_rel, u_north)
        ew = np.dot(v_rel, u_east)
        
        dir_ns = "North" if ns > 0 else "South"
        dir_ew = "East" if ew > 0 else "West"
        return f"{dir_ns}{dir_ew}" if abs(ns) > 0.1 and abs(ew) > 0.1 else (dir_ns if abs(ew) <= 0.1 else dir_ew)

    dir_kitchen = get_direction(p_kitchen, p_table, u_north, u_east)
    dir_living = get_direction(p_living, p_table, u_north, u_east)

    print(f"Kitchen direction: {dir_kitchen}")
    print(f"Living room direction: {dir_living}")
else:
    print("Could not proceed without living room segmentation.")

[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the white sofa in the background. Reply with ONLY the numbers.
[VLM A | locate] 447 441 632 564
Sofa bbox norm: 447 441 632 564
Sofa segmentation successful.
P_table: [ 3.2590525  1.6480582 -5.004488 ]
P_window: [ 3.3722017  2.9420273 -6.996321 ]
P_kitchen: [-0.7721114  1.8272421 -4.029684 ]
P_living: [ -3.0373914   3.9206772 -10.664564 ]
Kitchen direction: NorthEast
Living room direction: SouthEast

ReturnAnswer("B")

[ReturnAnswer] Answer submitted: B