MMSI — Gemma4-31B — sample 538

sun_direction = vlm.ask_with_thinking(InputImages, "Identify the direction of the sunlight or shadows. From the camera's perspective in each frame, where is the sun located? (e.g., 'coming from the right', 'from the left', 'from behind the camera', 'from the front'). Provide a consistent summary of where the light source is relative to the room's layout.")
print(sun_direction)

[VLM Q | thinking] Identify the direction of the sunlight or shadows. From the camera's perspective in each frame, where is the sun located? (e.g., 'coming from the right', 'from the left', 'from behind the camera', 'from the front'). Provide a consistent summary of where the light source is relative to the room's layout.
[VLM A | thinking] Based on the shadows and highlights in the images, the sunlight is consistently coming from the right side of the camera's perspective in every frame:

*   **Frame 0:** Bright light hits the right side of the chair and rug, casting shadows to the left.
*   **Frame 1:** Shadows from the furniture are cast towards the left.
*   **Frame 2:** Light hits the right side of the sofa, casting shadows to the left.
*   **Frame 3:** Highlights are on the right side of the mannequin and piano, with shadows cast to the left.
*   **Frame 4:** Light hits the right side of the table and chairs, casting shadows to the left.
*   **Frame 5:** Shadows from the cabinet and furniture are cast towards the left.

**Summary:** The light source is coming from the right.
Based on the shadows and highlights in the images, the sunlight is consistently coming from the right side of the camera's perspective in every frame:

*   **Frame 0:** Bright light hits the right side of the chair and rug, casting shadows to the left.
*   **Frame 1:** Shadows from the furniture are cast towards the left.
*   **Frame 2:** Light hits the right side of the sofa, casting shadows to the left.
*   **Frame 3:** Highlights are on the right side of the mannequin and piano, with shadows cast to the left.
*   **Frame 4:** Light hits the right side of the table and chairs, casting shadows to the left.
*   **Frame 5:** Shadows from the cabinet and furniture are cast towards the left.

**Summary:** The light source is coming from the right.

# 1. Perform 3D Reconstruction
recon = tools.Reconstruct.Reconstruct(InputImages)

# 2. Robust Segmentation
object_labels = ["glass cabinet", "dining table", "piano", "door"]
all_segs = {}

for label in object_labels:
    combined_masks = []
    # We'll store the results as a list of (frame_index, mask)
    for img in InputImages:
        try:
            seg = tools.SAM3.segment_image_by_text(img, label)
            # seg is a PerFrameMask. It might contain multiple instances.
            # For simplicity, we'll take the first instance if multiple are found.
            fi = seg.frame_indices[0]
            mask = seg.get_mask(frame=fi, object=0)
            combined_masks.append((fi, mask))
        except Exception as e:
            # Object not found in this frame, skip
            continue
    
    if combined_masks:
        all_segs[label] = combined_masks

# 3. Verification: Print which objects were found in which frames
for label, masks in all_segs.items():
    print(f"{label} found in frames: {[m[0] for m in masks]}")

# Visual check for one object to ensure quality
if "piano" in all_segs:
    fi, mask = all_segs["piano"][0]
    # Find which InputImage corresponds to this fi
    img_idx = -1
    for i, img in enumerate(InputImages):
        if img.frame_index == fi:
            img_idx = i
            break
    show([InputImages[img_idx], tools.Draw.draw_bbox(InputImages[img_idx], tools.Mask.bounding_box(mask), colors="red")])

glass cabinet found in frames: [0, 5]
dining table found in frames: [4]
piano found in frames: [3, 4]
door found in frames: [5]
 [image: show_0001] 

# Use the first frame as the reference for "the person"
fi_ref = recon.frame_indices[0]
pose_ref = recon.extrinsics[fi_ref]
cam_pos_ref = pose_ref[:3, 3]
cam_right_ref = pose_ref[:3, 0]
cam_fwd_ref = pose_ref[:3, 2]

# East is the direction of the sun (to the right in Frame 0)
east_vec = cam_right_ref / np.linalg.norm(cam_right_ref)
world_up = np.array([0, 1, 0])
north_vec = np.cross(east_vec, world_up)
north_vec /= np.linalg.norm(north_vec)

print(f"Reference Camera Pos: {cam_pos_ref}")
print(f"East Vector: {east_vec}")
print(f"North Vector: {north_vec}")

# Re-compute centroids with more stability
final_centroids = {}
for label, masks in all_segs.items():
    all_pts = []
    for fi, mask in masks:
        pts = recon.points[fi][mask]
        conf = recon.points.confidence[fi][mask]
        valid_pts = pts[conf > 0.5]
        if len(valid_pts) > 0:
            all_pts.append(np.median(valid_pts, axis=0))
    if all_pts:
        final_centroids[label] = np.median(np.array(all_pts), axis=0)

for label, center in final_centroids.items():
    print(f"Centroid {label}: {center}")

# Evaluate D: Piano is East of the person
if "piano" in final_centroids:
    v_piano = final_centroids["piano"] - cam_pos_ref
    dot_piano_east = np.dot(v_piano, east_vec)
    print(f"Piano relative to person: {v_piano}, Dot with East: {dot_piano_east:.3f}")

# Evaluate B: Dining table is SW of piano
if "piano" in final_centroids and "dining table" in final_centroids:
    v_p_to_t = final_centroids["dining table"] - final_centroids["piano"]
    proj_east = np.dot(v_p_to_t, east_vec)
    proj_north = np.dot(v_p_to_t, north_vec)
    print(f"Table relative to piano: {v_p_to_t}, East_proj: {proj_east:.3f}, North_proj: {proj_north:.3f}")

# Evaluate A & C: In front of person
if "glass cabinet" in final_centroids:
    v_cab = final_centroids["glass cabinet"] - cam_pos_ref
    dot_cab_fwd = np.dot(v_cab / np.linalg.norm(v_cab), cam_fwd_ref)
    print(f"Cabinet dot with fwd: {dot_cab_fwd:.3f}")

if "door" in final_centroids:
    v_door = final_centroids["door"] - cam_pos_ref
    dot_door_fwd = np.dot(v_door / np.linalg.norm(v_door), cam_fwd_ref)
    print(f"Door dot with fwd: {dot_door_fwd:.3f}")

Reference Camera Pos: [-0.00302395  2.07924647 -0.00848874]
East Vector: [ 1.00000000e+00  8.31261705e-08 -5.73610987e-08]
North Vector: [ 5.73610987e-08 -0.00000000e+00  1.00000000e+00]
Centroid glass cabinet: [-0.79935974  2.3692575  -1.714575  ]
Centroid dining table: [-1.8540796  -0.15084767  0.89975154]
Centroid piano: [-0.6615849   0.31840062  0.5062278 ]
Centroid door: [-1.5246018  1.9994826 -1.3836036]
Piano relative to person: [-0.65856096 -1.76084584  0.51471653], Dot with East: -0.659
Table relative to piano: [-1.1924946  -0.4692483   0.39352375], East_proj: -1.192, North_proj: 0.394
Cabinet dot with fwd: 0.896
Door dot with fwd: 0.670

ReturnAnswer("A")

[ReturnAnswer] Answer submitted: A