PerceptComp — Gemma4-31B — sample 1001

# Select a few informative frames to verify the conditions
test_frames = [InputImages[0], InputImages[8], InputImages[12]]

# Ask the VLM to identify the frame that best satisfies the conditions
question = (
    "In which of these frames are the following conditions all met: "
    "1) There are two people (a man and a woman) in the scene. "
    "2) The woman is looking at the man. "
    "3) There is a bright lamp positioned to the rear-left of the man. "
    "Please analyze each frame and tell me which one fits best, and why."
)

answer = vlm.ask_with_thinking(test_frames, question)
print(answer)

# Also show the candidate frame for my own visual confirmation
show(InputImages[8])

[VLM Q | thinking] In which of these frames are the following conditions all met: 1) There are two people (a man and a woman) in the scene. 2) The woman is looking at the man. 3) There is a bright lamp positioned to the rear-left of the man. Please analyze each frame and tell me which one fits best, and why.
[VLM A | thinking] Based on the provided frames, here is the analysis:

*   **Frame 0:** There are three people visible (the man, the woman on the left, and another person in the background window). There is no bright lamp to the rear-left of the man.
*   **Frame 1:** All conditions are met. There are exactly two people (a man and a woman). The woman is looking directly at the man, and there is a bright lamp positioned to the rear-left of the man.
*   **Frame 2:** There are three people visible (the man, the woman, and another person in the background window on the right).

Therefore, **Frame 1** is the only frame where all conditions are met.
Based on the provided frames, here is the analysis:

*   **Frame 0:** There are three people visible (the man, the woman on the left, and another person in the background window). There is no bright lamp to the rear-left of the man.
*   **Frame 1:** All conditions are met. There are exactly two people (a man and a woman). The woman is looking directly at the man, and there is a bright lamp positioned to the rear-left of the man.
*   **Frame 2:** There are three people visible (the man, the woman, and another person in the background window on the right).

Therefore, **Frame 1** is the only frame where all conditions are met.
 [image: show_0001] 

img = InputImages[8]
W, H = img.width, img.height

# Define objects to locate
objects = ["the man", "the woman", "the bright lamp", "the microphone"]
bboxes = []

for obj in objects:
    prompt = f"Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for {obj}. Reply with ONLY the numbers."
    res = vlm.locate(img, prompt)
    print(f"{obj}: {res}")
    
    # Parse coordinates
    try:
        coords = [float(x) for x in res.replace(',', ' ').split()]
        if len(coords) == 4:
            px_coords = tools.Geometry.normalized_to_pixel(coords, W, H)
            bboxes.append(px_coords)
        else:
            print(f"Warning: Could not parse 4 coords for {obj}")
            bboxes.append(None)
    except Exception as e:
        print(f"Error parsing {obj}: {e}")
        bboxes.append(None)

# Store bboxes in a dictionary for clarity
grounding = {obj: box for obj, box in zip(objects, bboxes)}
print("Final grounding:", grounding)

# Visualize bboxes to sanity check
annotated = img.copy()
colors = ["red", "green", "yellow", "blue"]
for (obj, box), color in zip(grounding.items(), colors):
    if box:
        annotated = tools.Draw.draw_bbox(annotated, box, colors=color)

show(annotated)

[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the man. Reply with ONLY the numbers.
[VLM A | locate] Not visible
Two men are present; selection is ambiguous.
the man: Not visible
Two men are present; selection is ambiguous.
Error parsing the man: could not convert string to float: 'Not'
[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the woman. Reply with ONLY the numbers.
[VLM A | locate] 561 232 805 823
the woman: 561 232 805 823
[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the bright lamp. Reply with ONLY the numbers.
[VLM A | locate] 398 138 464 238
the bright lamp: 398 138 464 238
[VLM Q | locate] Before answering, first check whether what the question descri

img = InputImages[8]
W, H = img.width, img.height

# Specifically locate the man in the foreground
prompt = "Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the man in the foreground wearing a hat. Reply with ONLY the numbers."
res = vlm.locate(img, prompt)
print(f"the man (refined): {res}")

try:
    coords = [float(x) for x in res.replace(',', ' ').split()]
    if len(coords) == 4:
        man_box = tools.Geometry.normalized_to_pixel(coords, W, H)
        grounding['the man'] = man_box
        print("Successfully located the man.")
    else:
        print(f"Warning: Could not parse 4 coords for the man: {res}")
except Exception as e:
    print(f"Error parsing the man: {e}")

# Visualize all bboxes again to confirm
annotated = img.copy()
colors = ["red", "green", "yellow", "blue"]
for (obj, box), color in zip(grounding.items(), colors):
    if box:
        annotated = tools.Draw.draw_bbox(annotated, box, colors=color)

show(annotated)

[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the man in the foreground wearing a hat. Reply with ONLY the numbers.
[VLM A | locate] 52 166 564 999
the man (refined): 52 166 564 999
Successfully located the man.
 [image: show_0003] 

img = InputImages[8]
seg_results = {}

# Segment each object using the grounding bboxes
for obj, box in grounding.items():
    if box:
        # Generate mask for the object
        mask_obj = tools.SAM3.segment_image_by_box(img, box, label=obj)
        
        # Programmatic check for empty mask
        fi = mask_obj.frame_indices[0]
        area = mask_obj.get_mask(frame=fi, object=0).sum()
        print(f"{obj}: area = {area} pixels")
        
        if area > 0:
            seg_results[obj] = mask_obj
        else:
            print(f"Warning: Mask for {obj} is empty.")

# Visual verification of all masks
# Since SAM3 returns PerFrameMask, I'll combine them or show them individually.
# For simplicity, I'll visualize a few key ones.
vis_list = [img]
for obj, seg in seg_results.items():
    fi = seg.frame_indices[0]
    vis_list.append(seg.visualize(fi))

show(vis_list)

the man: area = 114614 pixels
the woman: area = 41178 pixels
the bright lamp: area = 2100 pixels
the microphone: area = 3137 pixels
 [image: show_0004] 

# Reconstruct from a range of frames around the reference frame InputImages[8]
# Using frames 0 to 20 to provide enough baseline for reconstruction
recon = tools.Reconstruct.Reconstruct(InputImages[0:20])

# Reference frame index (absolute video frame index)
fi = InputImages[8].frame_index

# Extract 3D centroids for the relevant objects
# Note: I need to pass the masks from the same frame to get_centroid_3d
p_man = seg_results['the man'].get_centroid_3d(recon, frame=fi)
p_lamp = seg_results['the bright lamp'].get_centroid_3d(recon, frame=fi)
p_mic = seg_results['the microphone'].get_centroid_3d(recon, frame=fi)

print(f"Frame index: {fi}")
print(f"Man centroid: {p_man}")
print(f"Lamp centroid: {p_lamp}")
print(f"Mic centroid: {p_mic}")

# Sanity check: are any centroids NaN?
if any(p is None for p in [p_man, p_lamp, p_mic]):
    print("Warning: One or more centroids are None. Checking mask overlap with reconstructed points.")

Frame index: 192
Man centroid: [-1.2746447   0.52724266 -3.4145799 ]
Lamp centroid: [-1.878898   1.5535543 -5.8480186]
Mic centroid: [-0.5790228  0.813005  -4.9251924]

# Get camera pose at the reference frame
pose = recon.extrinsics[fi]
cam_right = pose[:3, 0]
cam_down = pose[:3, 1]
cam_fwd = pose[:3, 2]

# The man is facing right in the image
f_man = cam_right / np.linalg.norm(cam_right)
# Right vector r = f x up. World up is [0, 1, 0]
up = np.array([0, 1, 0])
r_man = np.cross(f_man, up)
r_man /= np.linalg.norm(r_man)
# Orthogonalize f_man again
f_man = np.cross(up, r_man)

# Relative vectors
v_lamp = p_lamp - p_man
v_mic = p_mic - p_man

# Verify lamp is rear-left
dot_f_lamp = np.dot(v_lamp, f_man)
dot_r_lamp = np.dot(v_lamp, r_man)

print(f"Verification - Lamp relative to man: fwd={dot_f_lamp:.2f}, right={dot_r_lamp:.2f}")
# Rear-left means dot_f < 0 and dot_r < 0.

# Calculate mic relative to man
dot_f_mic = np.dot(v_mic, f_man)
dot_r_mic = np.dot(v_mic, r_man)
print(f"Mic relative to man: fwd={dot_f_mic:.2f}, right={dot_r_mic:.2f}")

# Determine quadrant
if dot_f_mic > 0 and dot_r_mic > 0:
    res_dir = "Front-right"
elif dot_f_mic < 0 and dot_r_mic < 0:
    res_dir = "Rear-left"
elif dot_f_mic < 0 and dot_r_mic > 0:
    res_dir = "Rear-right"
elif dot_f_mic < 0 and abs(dot_r_mic) < 0.1:
    res_dir = "Directly behind"
elif dot_f_mic > 0 and dot_r_mic < 0:
    res_dir = "Front-left"
else:
    res_dir = "Unknown"

print(f"Final Direction: {res_dir}")

# Map to choice
mapping = {
    "Front-right": "A",
    "Rear-left": "B",
    "Rear-right": "C",
    "Directly behind": "D",
    "Front-left": "E"
}
ReturnAnswer(mapping.get(res_dir, "Unknown"))

Verification - Lamp relative to man: fwd=0.22, right=-2.50
Mic relative to man: fwd=1.15, right=-1.20
Final Direction: Front-left
[ReturnAnswer] Answer submitted: E