MMSI-Video — Gemma4-31B — sample question_0057

# Select key frames for identification: first, middle, and last
frames_to_inspect = [InputImages[0], InputImages[len(InputImages)//2], InputImages[-1]]

# Ask the VLM to identify moved objects and the staircase
question = "Identify the objects that the person is moving in this video. Also, describe the location and appearance of the staircase. Be specific about the objects (e.g., 'green plastic bag')."
answer = vlm.ask_with_thinking(frames_to_inspect, question)

print(f"VLM Answer: {answer}")

# Show the frames for visual confirmation
show(frames_to_inspect)

[VLM Q | thinking] Identify the objects that the person is moving in this video. Also, describe the location and appearance of the staircase. Be specific about the objects (e.g., 'green plastic bag').
[VLM A | thinking] The person is holding a green plastic bag while sitting on the stairs (frame 2).

The staircase is located indoors and consists of white-painted steps with light pink or peach-colored walls. It features white handrails on both sides and a framed sign that says "HOME" on the wall at the top. Several bags, including a blue multi-colored one and a red one, are hanging on the left wall.
VLM Answer: The person is holding a green plastic bag while sitting on the stairs (frame 2).

The staircase is located indoors and consists of white-painted steps with light pink or peach-colored walls. It features white handrails on both sides and a framed sign that says "HOME" on the wall at the top. Several bags, including a blue multi-colored one and a red one, are hanging on the left wall.
 [image: show_0001] 

# Segment the green plastic bag and the staircase
# Using text prompts for simplicity; if these fail, I'll use VLM grounding + boxes.
prompts = ["green plastic bag", "white stairs"]
labels = ["green bag", "staircase"]

seg = tools.SAM3.segment_video_by_text(prompts, labels=labels)

# Programmatic check for mask existence and area
for i, label in enumerate(seg.labels):
    print(f"Checking masks for {label}:")
    valid_frames = 0
    for fi in seg.frame_indices:
        mask = seg.get_mask(frame=fi, object=i)
        if mask.sum() > 0:
            valid_frames += 1
    print(f"  Object {label} found in {valid_frames}/{seg.num_frames} frames.")

# Visual verification of masks on a few frames
# Frame 0: Bag might not be there yet, but stairs should be.
# Frame 40: Person is sitting with the bag.
# Frame 80: Person is moving again.
test_frames = [0, 40, 80]
# Ensure test frames are within the range of InputImages
test_frames = [f for f in test_frames if f < len(InputImages)]

vis_list = []
for tf in test_frames:
    # Find the corresponding absolute frame index in seg.frame_indices
    # Since InputImages[tf] is used, we need to check the actual frame index
    abs_fi = InputImages[tf].frame_index
    if abs_fi in seg.frame_indices:
        vis_list.append(seg.visualize(abs_fi))
    else:
        print(f"  Frame {abs_fi} not in segmentation indices.")

show([InputImages[tf] for tf in test_frames] + vis_list)

Checking masks for green bag:
  Object green bag found in 667/667 frames.
Checking masks for staircase:
  Object staircase found in 0/667 frames.
 [image: show_0002] 

# Select a clear frame for grounding
prompt_frame_idx = 40
image = InputImages[prompt_frame_idx]
W, H = image.width, image.height

# Get bounding boxes from VLM
bag_box_norm = vlm.locate(image, "Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the green plastic bag. Reply with ONLY the numbers.")
stairs_box_norm = vlm.locate(image, "Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the white staircase. Reply with ONLY the numbers.")

print(f"Bag box (norm): {bag_box_norm}")
print(f"Stairs box (norm): {stairs_box_norm}")

# Parse and convert to pixels
def parse_box(box_str):
    return [float(x) for x in box_str.replace('[', '').replace(']', '').replace(',', ' ').split()]

bag_box_px = tools.Geometry.normalized_to_pixel(parse_box(bag_box_norm), W, H)
stairs_box_px = tools.Geometry.normalized_to_pixel(parse_box(stairs_box_norm), W, H)

# Segment video using boxes
# Note: prompt_frame_idx in segment_video_by_box is local to the range. 
# Since we use the whole video, and we chose InputImages[40], we need to find the local index.
# InputImages[40] is the 41st image in the list.
# However, segment_video_by_box takes the prompt_frame_idx as an index within the video.
# Let's use the absolute frame index of InputImages[40].
abs_prompt_fi = image.frame_index

# We need to specify the local index within the provided range. 
# If we don't specify start/end_frame, it's the whole video.
# The prompt_frame_idx is the index in the video (0 to total_frames-1).
seg = tools.SAM3.segment_video_by_box(
    boxes=[bag_box_px, stairs_box_px],
    labels=["green bag", "staircase"],
    prompt_frame_idx=abs_prompt_fi
)

# Programmatic check
for i, label in enumerate(seg.labels):
    print(f"Checking masks for {label}:")
    valid_frames = 0
    for fi in seg.frame_indices:
        mask = seg.get_mask(frame=fi, object=i)
        if mask.sum() > 0:
            valid_frames += 1
    print(f"  Object {label} found in {valid_frames}/{seg.num_frames} frames.")

# Visual verification
test_frames = [0, 40, 80]
test_frames = [f for f in test_frames if f < len(InputImages)]
vis_list = []
for tf in test_frames:
    abs_fi = InputImages[tf].frame_index
    if abs_fi in seg.frame_indices:
        vis_list.append(seg.visualize(abs_fi))

show([InputImages[tf] for tf in test_frames] + vis_list)

[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the green plastic bag. Reply with ONLY the numbers.
[VLM A | locate] 515 454 684 666
[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the white staircase. Reply with ONLY the numbers.
[VLM A | locate] 60 640 860 1000
Bag box (norm): 515 454 684 666
Stairs box (norm): 60 640 860 1000
Checking masks for green bag:
  Object green bag found in 667/667 frames.
Checking masks for staircase:
  Object staircase found in 667/667 frames.

# Select a clear frame for grounding
prompt_frame_idx = 40
image = InputImages[prompt_frame_idx]
W, H = image.width, image.height

# Get center points from VLM
bag_pt_norm = vlm.locate(image, "Give the (x, y) center coordinates in 0-1000 normalized scale for the green plastic bag. Reply with ONLY the numbers.")
stairs_pt_norm = vlm.locate(image, "Give the (x, y) center coordinates in 0-1000 normalized scale for the middle of the white stairs. Reply with ONLY the numbers.")

print(f"Bag point (norm): {bag_pt_norm}")
print(f"Stairs point (norm): {stairs_pt_norm}")

# Parse and convert to pixels
def parse_pt(pt_str):
    return [float(x) for x in pt_str.replace('(', '').replace(')', '').replace('[', '').replace(']', '').replace(',', ' ').split()]

bag_pt_px = tools.Geometry.normalized_to_pixel(parse_pt(bag_pt_norm), W, H)
stairs_pt_px = tools.Geometry.normalized_to_pixel(parse_pt(stairs_pt_norm), W, H)

# Prepare points and labels for SAM3
# points_per_object: list of point lists, e.g. [[[x, y]], [[x, y]]]
points_per_object = [[bag_pt_px], [stairs_pt_px]]
point_labels_per_object = [[1], [1]] # 1 for foreground
labels = ["green bag", "staircase"]

# Use absolute frame index for the prompt frame
abs_prompt_fi = image.frame_index

seg = tools.SAM3.segment_video_by_points(
    points_per_object=points_per_object,
    point_labels_per_object=point_labels_per_object,
    labels=labels,
    prompt_frame_idx=abs_prompt_fi
)

# Programmatic check
for i, label in enumerate(seg.labels):
    print(f"Checking masks for {label}:")
    valid_frames = 0
    for fi in seg.frame_indices:
        mask = seg.get_mask(frame=fi, object=i)
        if mask.sum() > 0:
            valid_frames += 1
    print(f"  Object {label} found in {valid_frames}/{seg.num_frames} frames.")

# Visual verification
test_frames = [0, 40, 80]
test_frames = [f for f in test_frames if f < len(InputImages)]
vis_list = []
for tf in test_frames:
    abs_fi = InputImages[tf].frame_index
    if abs_fi in seg.frame_indices:
        vis_list.append(seg.visualize(abs_fi))

show([InputImages[tf] for tf in test_frames] + vis_list)

[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the (x, y) center coordinates in 0-1000 normalized scale for the green plastic bag. Reply with ONLY the numbers.
[VLM A | locate] 600 558
[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the (x, y) center coordinates in 0-1000 normalized scale for the middle of the white stairs. Reply with ONLY the numbers.
[VLM A | locate] 465 825
Bag point (norm): 600 558
Stairs point (norm): 465 825
Checking masks for green bag:
  Object green bag found in 667/667 frames.
Checking masks for staircase:
  Object staircase found in 667/667 frames.

# Select a frame where the bag is distinct and the stairs are clear
prompt_frame_idx = 60
image = InputImages[prompt_frame_idx]
W, H = image.width, image.height

# Get tight bounding boxes from VLM
# Specifically asking for a "tight" box for the bag to avoid including the person
bag_box_norm = vlm.locate(image, "Give a very tight bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for ONLY the green plastic bag. Reply with ONLY the numbers.")
stairs_box_norm = vlm.locate(image, "Give a bounding box (x1, y1, x2, y2) in 0-1000 normalized scale that covers the entire visible white staircase. Reply with ONLY the numbers.")

print(f"Bag box (norm): {bag_box_norm}")
print(f"Stairs box (norm): {stairs_box_norm}")

# Parse and convert to pixels
def parse_box(box_str):
    return [float(x) for x in box_str.replace('[', '').replace(']', '').replace(',', ' ').split()]

bag_box_px = tools.Geometry.normalized_to_pixel(parse_box(bag_box_norm), W, H)
stairs_box_px = tools.Geometry.normalized_to_pixel(parse_box(stairs_box_norm), W, H)

# Segment video using boxes
abs_prompt_fi = image.frame_index
seg = tools.SAM3.segment_video_by_box(
    boxes=[bag_box_px, stairs_box_px],
    labels=["green bag", "staircase"],
    prompt_frame_idx=abs_prompt_fi
)

# Visual verification on a few frames
test_frames = [0, 40, 60, 80]
test_frames = [f for f in test_frames if f < len(InputImages)]
vis_list = []
for tf in test_frames:
    abs_fi = InputImages[tf].frame_index
    if abs_fi in seg.frame_indices:
        vis_list.append(seg.visualize(abs_fi))

show([InputImages[tf] for tf in test_frames] + vis_list)

[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give a very tight bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for ONLY the green plastic bag. Reply with ONLY the numbers.
[VLM A | locate] 441, 501, 761, 771
[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give a bounding box (x1, y1, x2, y2) in 0-1000 normalized scale that covers the entire visible white staircase. Reply with ONLY the numbers.
[VLM A | locate] 60 630 940 1000
Bag box (norm): 441, 501, 761, 771
Stairs box (norm): 60 630 940 1000

import numpy as np

# 1. Define the North axis
# Pick a reference frame (Frame 0)
ref_fi = InputImages[0].frame_index
cam_pos = recon.extrinsics[ref_fi][:3, 3]

# To get a point on the staircase, we can use the VLM to find the center of the stairs in Frame 0
# and then get the 3D point from the reconstruction.
stairs_pt_norm = vlm.locate(InputImages[0], "Give the (x, y) center coordinates in 0-1000 normalized scale for the middle of the white stairs. Reply with ONLY the numbers.")
# Parse and convert to pixels
def parse_pt(pt_str):
    return [float(x) for x in pt_str.replace('(', '').replace(')', '').replace('[', '').replace(']', '').replace(',', ' ').split()]

stairs_px = tools.Geometry.normalized_to_pixel(parse_pt(stairs_pt_norm), InputImages[0].width, InputImages[0].height)
# Get 3D point from reconstruction (using the sampled frame index in recon)
# Find the index in recon.frame_indices that matches ref_fi
recon_idx = recon.frame_indices.index(ref_fi)
stair_3d = recon.points.points[recon_idx][int(stairs_px[1]), int(stairs_px[0])]

# North vector: Staircase -> Camera
north_vec = cam_pos - stair_3d
north_vec = north_vec / np.linalg.norm(north_vec)

# East vector: North x Up
up_vec = np.array([0, 1, 0])
east_vec = np.cross(north_vec, up_vec)
east_vec = east_vec / np.linalg.norm(east_vec)

print(f"North Vector: {north_vec}")
print(f"East Vector: {east_vec}")

# 2. Sample bag positions at key frames
# We'll use frames where the bag is clearly visible and at different stages
sample_indices = [0, 15, 40, 75, 94] 
bag_positions = []

for idx in sample_indices:
    img = InputImages[idx]
    fi = img.frame_index
    # Find the frame index in recon
    if fi not in recon.frame_indices:
        # Find nearest
        recon_idx = min(range(len(recon.frame_indices)), key=lambda i: abs(recon.frame_indices[i] - fi))
    else:
        recon_idx = recon.frame_indices.index(fi)
    
    # Get bag center
    coord_str = vlm.locate(img, "Give the (x, y) center coordinates in 0-1000 normalized scale for the green plastic bag. Reply with ONLY the numbers.")
    if "Not visible" in coord_str:
        print(f"Frame {fi}: Bag not visible")
        bag_positions.append(None)
        continue
        
    px = tools.Geometry.normalized_to_pixel(parse_pt(coord_str), img.width, img.height)
    pos_3d = recon.points.points[recon_idx][int(px[1]), int(px[0])]
    bag_positions.append(pos_3d)
    print(f"Frame {fi} Bag Pos: {pos_3d}")

# 3. Analyze displacements
# Filter out None values
valid_pos = [p for p in bag_positions if p is not None]
valid_indices = [i for i, p in enumerate(bag_positions) if p is not None]

if len(valid_pos) >= 3:
    # Displacement 1: Start -> Bottom
    d1 = valid_pos[1] - valid_pos[0]
    proj_n1 = np.dot(d1, north_vec)
    print(f"Movement 1 North Projection: {proj_n1:.3f} ({'North' if proj_n1 > 0 else 'South'})")

    # Displacement 2: Bottom -> Top
    # Find the transition from bottom to top (e.g., from index 1 to 3 or 4)
    d2 = valid_pos[-2] - valid_pos[1] # Bottom to near-end
    proj_n2 = np.dot(d2, north_vec)
    print(f"Movement 2 North Projection: {proj_n2:.3f} ({'North' if proj_n2 > 0 else 'South'})")

    # Displacement 3: Top -> Final
    d3 = valid_pos[-1] - valid_pos[-2]
    proj_n3 = np.dot(d3, north_vec)
    proj_e3 = np.dot(d3, east_vec)
    print(f"Movement 3 North Projection: {proj_n3:.3f}, East Projection: {proj_e3:.3f} ({'East' if proj_e3 > 0 else 'West'})")
else:
    print("Not enough valid positions to analyze trajectory.")

[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the (x, y) center coordinates in 0-1000 normalized scale for the middle of the white stairs. Reply with ONLY the numbers.
[VLM A | locate] 500 775
North Vector: [2.52912577e-08 1.99708856e-01 9.79855282e-01]
East Vector: [-1.00000000e+00  0.00000000e+00  2.58112174e-08]
[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the (x, y) center coordinates in 0-1000 normalized scale for the green plastic bag. Reply with ONLY the numbers.
[VLM A | locate] 200 545
Frame 0 Bag Pos: [-0.3488108  0.6492406 -2.344051 ]
[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the (x, y) center coordinates in 0-1000 normalized scale for the green plastic bag. Reply with ONLY the numbers.
[VLM A | locate] 320 370
Frame 15 Bag Pos: [-0.18988638  1.0158672  -2.5169325 ]
[VLM Q | locate] Before answering, first check whether what the question 

ReturnAnswer("A")

[ReturnAnswer] Answer submitted: A