ViewSpatial — Gemma4-31B — sample 5163

# Select a few diverse frames to survey the scene
survey_frames = [InputImages[0], InputImages[1], InputImages[11], InputImages[21], InputImages[23]]
show(survey_frames)

# Ask the VLM to identify the frames containing the target objects
question = "Identify which of the provided frames contain a television, a toilet, and a desk. For each object, list the frame indices (0-indexed relative to the provided list) where it is most clearly visible."
answer = vlm.ask_with_thinking(survey_frames, question)
print(f"VLM Survey Answer: {answer}")

 [image: show_0001] 
[VLM Q | thinking] Identify which of the provided frames contain a television, a toilet, and a desk. For each object, list the frame indices (0-indexed relative to the provided list) where it is most clearly visible.
[VLM A | thinking] Based on the provided images:
- **Television**: Not visible in any frame.
- **Toilet**: Not visible in any frame.
- **Desk**: Frame 1 (also partially visible in frame 0).
VLM Survey Answer: Based on the provided images:
- **Television**: Not visible in any frame.
- **Toilet**: Not visible in any frame.
- **Desk**: Frame 1 (also partially visible in frame 0).

# Targeted frames for each object
tv_frame = InputImages[21]
bathroom_frame = InputImages[23]
desk_frame = InputImages[1]
targeted_frames = [tv_frame, bathroom_frame, desk_frame]

show(targeted_frames)

# Ask the VLM to confirm the objects in these specific frames
question = "In these three images: 1) Is there a television in the first image? 2) Is there a toilet or bathroom area in the second image? 3) Is there a desk in the third image? Please confirm and provide a brief description of their appearance."
answer = vlm.ask_with_thinking(targeted_frames, question)
print(f"VLM Targeted Survey Answer: {answer}")

 [image: show_0002] 
[VLM Q | thinking] In these three images: 1) Is there a television in the first image? 2) Is there a toilet or bathroom area in the second image? 3) Is there a desk in the third image? Please confirm and provide a brief description of their appearance.
[VLM A | thinking] 1) No, there is no television in the first image.
2) Yes, there is a bathroom area in the second image, where a white bathtub and a metal grab bar are visible through an open doorway.
3) Yes, there is a desk in the third image; it is a dark-colored desk with a black mesh office chair and various items on top, including papers and a coffee maker.
VLM Targeted Survey Answer: 1) No, there is no television in the first image.
2) Yes, there is a bathroom area in the second image, where a white bathtub and a metal grab bar are visible through an open doorway.
3) Yes, there is a desk in the third image; it is a dark-colored desk with a black mesh office chair and various items on top, including papers and a coffee maker.

# 1. Get TV mask from frame 19 (which worked previously)
seg_tv = tools.SAM3.segment_image_by_text(InputImages[19], "television")

# 2. 3D Reconstruction
# Use a representative subset of frames for better quality
recon_frames = InputImages[::2] # Subsample to stay within 64 limit and improve speed
recon = tools.Reconstruct.Reconstruct(recon_frames)

# 3. Extract 3D Centroids
def get_robust_centroid(seg, recon, label):
    centroids = []
    for fi in seg.frame_indices:
        # Ensure the frame is part of the reconstruction
        if fi in recon.frame_indices:
            c = seg.get_centroid_3d(recon, frame=fi, object=label)
            if c is not None:
                centroids.append(c)
    if not centroids:
        return None
    return np.median(centroids, axis=0)

p_tv = get_robust_centroid(seg_tv, recon, "television")
p_desk = get_robust_centroid(seg_desk, recon, "desk")
p_toilet = get_robust_centroid(seg_toilet, recon, "toilet")

print(f"TV position: {p_tv}")
print(f"Desk position: {p_desk}")
print(f"Toilet position: {p_toilet}")

# 4. Relative Direction Computation
if p_tv is not None and p_toilet is not None and p_desk is not None:
    # Forward vector: TV -> Toilet
    v_fwd = p_toilet - p_tv
    v_fwd /= np.linalg.norm(v_fwd)
    
    # World up vector
    v_up = np.array([0, 1, 0])
    
    # Right vector: fwd x up
    v_right = np.cross(v_fwd, v_up)
    v_right /= np.linalg.norm(v_right)
    
    # Vector to desk: TV -> Desk
    v_desk = p_desk - p_tv
    
    proj_fwd = np.dot(v_desk, v_fwd)
    proj_right = np.dot(v_desk, v_right)
    
    fwd_dir = "front" if proj_fwd > 0 else "back"
    right_dir = "right" if proj_right > 0 else "left"
    
    print(f"Relative direction: {fwd_dir}-{right_dir}")
    print(f"proj_fwd: {proj_fwd}, proj_right: {proj_right}")

# 5. Visual Cross-Validation with BEV
# Combine masks into one for BEV rendering
# Note: render_bev expects a PerFrameMask or (N, N_obj, H, W)
# We'll create a simple combined mask array for the first frame of the reconstruction
fi_ref = recon.frame_indices[0]
# We need to make sure we have masks for this specific frame for BEV
# Since our masks are single-frame, we can't easily use them in a multi-frame BEV 
# unless we segment all frames. Let's just use the centroids to verify in a plot if needed,
# or just trust the math if the points are stable.
# For now, let's render BEV with just the masks we have if possible.
# Actually, let's just rely on the centroids for this specific calculation.

TV position: None
Desk position: None
Toilet position: None

# 1. Reconstruct using ALL frames to ensure masks match
recon = tools.Reconstruct.Reconstruct(InputImages)

# 2. Extract 3D Centroids using the existing masks
def get_robust_centroid(seg, recon, label):
    centroids = []
    for fi in seg.frame_indices:
        if fi in recon.frame_indices:
            c = seg.get_centroid_3d(recon, frame=fi, object=label)
            if c is not None:
                centroids.append(c)
    if not centroids:
        return None
    return np.median(centroids, axis=0)

p_tv = get_robust_centroid(seg_tv, recon, "television")
p_desk = get_robust_centroid(seg_desk, recon, "desk")
p_toilet = get_robust_centroid(seg_toilet, recon, "toilet")

print(f"TV position: {p_tv}")
print(f"Desk position: {p_desk}")
print(f"Toilet position: {p_toilet}")

# 3. Relative Direction Computation
if p_tv is not None and p_toilet is not None and p_desk is not None:
    # Forward vector: TV -> Toilet
    v_fwd = p_toilet - p_tv
    v_fwd /= np.linalg.norm(v_fwd)
    
    # World up vector
    v_up = np.array([0, 1, 0])
    
    # Right vector: fwd x up
    v_right = np.cross(v_fwd, v_up)
    v_right /= np.linalg.norm(v_right)
    
    # Vector to desk: TV -> Desk
    v_desk = p_desk - p_tv
    
    proj_fwd = np.dot(v_desk, v_fwd)
    proj_right = np.dot(v_desk, v_right)
    
    fwd_dir = "front" if proj_fwd > 0 else "back"
    right_dir = "right" if proj_right > 0 else "left"
    
    print(f"Relative direction: {fwd_dir}-{right_dir}")
    print(f"proj_fwd: {proj_fwd}, proj_right: {proj_right}")
else:
    print("Failed to compute all centroids. Checking for None...")

TV position: [-2.2225423  2.298991   1.2168118]
Desk position: [-1.5259507   2.3376565   0.42188838]
Toilet position: [-3.2268865  -0.14823127  5.454604  ]
Relative direction: back-left
proj_fwd: -0.8333178162574768, proj_right: -0.49449942839419836

ReturnAnswer("B")

[ReturnAnswer] Answer submitted: B