ViewSpatial — Gemma4-31B — sample 4770

# Use VLM to identify the objects and their locations
frames_to_check = [InputImages[0], InputImages[4], InputImages[9]]
question = "In these images, identify which object is the 'nightstand', which is the 'paper', and which are the 'clothes'. Tell me which frames they appear in and describe them."
answer = vlm.ask_with_thinking(frames_to_check, question)
print(answer)

# Also let's just show the frames to be sure
show(frames_to_check)

[VLM Q | thinking] In these images, identify which object is the 'nightstand', which is the 'paper', and which are the 'clothes'. Tell me which frames they appear in and describe them.
[VLM A | thinking] Based on the images provided:

*   **Nightstand**: The dark-colored chest of drawers in **frame 0**, which holds a laptop, a digital clock, and picture frames.
*   **Paper**: A white sheet of paper resting on the wooden desk in **frame 2**.
*   **Clothes**: The striped fabric draped over the back of the chair in **frame 2**.
Based on the images provided:

*   **Nightstand**: The dark-colored chest of drawers in **frame 0**, which holds a laptop, a digital clock, and picture frames.
*   **Paper**: A white sheet of paper resting on the wooden desk in **frame 2**.
*   **Clothes**: The striped fabric draped over the back of the chair in **frame 2**.
 [image: show_0001] 

# 1. Locate objects in representative frames
# Nightstand in frame 0
nightstand_box_norm = vlm.locate(InputImages[0], "Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the dark chest of drawers. Reply with ONLY the numbers.")
# Paper in frame 9
paper_box_norm = vlm.locate(InputImages[9], "Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the white sheet of paper on the desk. Reply with ONLY the numbers.")
# Clothes in frame 9
clothes_box_norm = vlm.locate(InputImages[9], "Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the striped fabric draped over the chair. Reply with ONLY the numbers.")

print(f"Nightstand box: {nightstand_box_norm}")
print(f"Paper box: {paper_box_norm}")
print(f"Clothes box: {clothes_box_norm}")

def get_mask_from_norm_box(image, box_str, label):
    # Parse the box string
    try:
        box_norm = [float(x) for x in box_str.replace('[', '').replace(']', '').split(',')]
    except ValueError:
        return None
    
    # Convert to pixels
    W, H = image.width, image.height
    box_px = tools.Geometry.normalized_to_pixel(box_norm, W, H)
    
    # Segment
    return tools.SAM3.segment_image_by_box(image, box_px, label)

# Generate masks for a few frames for robustness
# Nightstand: frames 0, 1
seg_nightstand = []
for i in [0, 1]:
    m = get_mask_from_norm_box(InputImages[i], nightstand_box_norm, "nightstand")
    if m: seg_nightstand.append(m)

# Paper: frames 8, 9
seg_paper = []
for i in [8, 9]:
    m = get_mask_from_norm_box(InputImages[i], paper_box_norm, "paper")
    if m: seg_paper.append(m)

# Clothes: frames 8, 9
seg_clothes = []
for i in [8, 9]:
    m = get_mask_from_norm_box(InputImages[i], clothes_box_norm, "clothes")
    if m: seg_clothes.append(m)

# Visual verification
if seg_nightstand:
    show([InputImages[0], seg_nightstand[0].visualize(seg_nightstand[0].frame_indices[0])])
if seg_paper:
    show([InputImages[9], seg_paper[-1].visualize(seg_paper[-1].frame_indices[0])])
if seg_clothes:
    show([InputImages[9], seg_clothes[-1].visualize(seg_clothes[-1].frame_indices[0])])

[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the dark chest of drawers. Reply with ONLY the numbers.
[VLM A | locate] 445 108 1000 666
[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the white sheet of paper on the desk. Reply with ONLY the numbers.
[VLM A | locate] 205 352 345 417
[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the striped fabric draped over the chair. Reply with ONLY the numbers.
[VLM A | locate] 453 261 701 713
Nightstand box: 445 108 1000 666
Paper box: 205 352 345 417
Clothes box: 453 261 701 713

# 1. Generate masks in the frames they were located
# Nightstand was located in frame 0
seg_nightstand = get_mask_from_norm_box(InputImages[0], nightstand_box_norm, "nightstand")
# Paper and clothes were located in frame 9
seg_paper = get_mask_from_norm_box(InputImages[9], paper_box_norm, "paper")
seg_clothes = get_mask_from_norm_box(InputImages[9], clothes_box_norm, "clothes")

# 2. Perform 3D Reconstruction
recon = tools.Reconstruct.Reconstruct(InputImages)

# 3. Extract 3D centroids
def get_robust_centroid(seg, recon):
    if seg is None:
        return None
    fi = seg.frame_indices[0]
    # We only have one frame per object here, so just get the centroid
    return seg.get_centroid_3d(recon, frame=fi, object=0)

p_nightstand = get_robust_centroid(seg_nightstand, recon)
p_paper = get_robust_centroid(seg_paper, recon)
p_clothes = get_robust_centroid(seg_clothes, recon)

print(f"Nightstand 3D: {p_nightstand}")
print(f"Paper 3D: {p_paper}")
print(f"Clothes 3D: {p_clothes}")

# Visual verification of masks
show([
    InputImages[0], seg_nightstand.visualize(0) if seg_nightstand else None,
    InputImages[9], seg_paper.visualize(9) if seg_paper else None,
    InputImages[9], seg_clothes.visualize(9) if seg_clothes else None
])

Nightstand 3D: [ 0.6642834  1.6383868 -1.40848  ]
Paper 3D: [0.60412234 0.5010488  1.1185184 ]
Clothes 3D: [-0.045558    0.37122524  1.1123338 ]
 [image: show_0002] 

# 1. Re-locate the nightstand (dark chest of drawers) in frame 0
# Visually, the dark chest of drawers is on the left side of the image.
nightstand_box_norm = vlm.locate(InputImages[0], "Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the large dark-colored chest of drawers on the left. Reply with ONLY the numbers.")
print(f"Corrected Nightstand box: {nightstand_box_norm}")

# 2. Generate corrected mask
seg_nightstand = get_mask_from_norm_box(InputImages[0], nightstand_box_norm, "nightstand")

# 3. Extract corrected 3D centroid
p_nightstand = get_robust_centroid(seg_nightstand, recon)
print(f"Corrected Nightstand 3D: {p_nightstand}")

# 4. Relative Direction Calculation
# We already have p_paper and p_clothes from the previous step
# p_paper: [0.60412234, 0.5010488, 1.1185184]
# p_clothes: [-0.045558, 0.37122524, 1.1123338]

# Forward vector: Nightstand -> Paper
v_fwd_raw = p_paper - p_nightstand
v_fwd = v_fwd_raw / np.linalg.norm(v_fwd_raw)

# Up vector (World Y)
v_up = np.array([0, 1, 0])

# Right vector: Forward x Up
v_right = np.cross(v_fwd, v_up)
v_right = v_right / np.linalg.norm(v_right)

# Target vector: Nightstand -> Clothes
v_target = p_clothes - p_nightstand

# Projections
d_fwd = np.dot(v_target, v_fwd)
d_right = np.dot(v_target, v_right)

print(f"d_fwd: {d_fwd}, d_right: {d_right}")

# Determine direction
if d_fwd < -0.1:
    direction = "back"
elif d_fwd > 0.1:
    if d_right < -0.1:
        direction = "front-left"
    elif d_right > 0.1:
        direction = "front-right"
    else:
        direction = "front"
else: # d_fwd approx 0
    if d_right < -0.1:
        direction = "left"
    elif d_right > 0.1:
        direction = "right"
    else:
        direction = "center"

print(f"Calculated Direction: {direction}")

# Visual verification of corrected mask
show([InputImages[0], seg_nightstand.visualize(0) if seg_nightstand else None])

[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the large dark-colored chest of drawers on the left. Reply with ONLY the numbers.
[VLM A | locate] 108 448 668 1000
Corrected Nightstand box: 108 448 668 1000
Corrected Nightstand 3D: [-0.2737373  1.2563334 -1.4894302]
d_fwd: 2.682345151901245, d_right: 0.6137600637223601
Calculated Direction: front-right
 [image: show_0003] 

ReturnAnswer("A")

[ReturnAnswer] Answer submitted: A