PAI-Bench — Gemma4-31B — sample 712

# Show the first, middle, and last frames to get a sense of the sequence
show([InputImages[0], InputImages[72], InputImages[-1]])

 [image: show_0001] 

# Use a subset of frames to analyze the action
frames_for_vlm = [InputImages[0], InputImages[30], InputImages[60], InputImages[90], InputImages[120], InputImages[-1]]

question = """
1. What is the agent currently holding? Which arm (left or right) is holding it?
2. Where are the bottled coconut milk and the crispy potato chips located?
3. What is the right arm doing? Does it appear to have a barcode scanner attached?
4. Based on the motion from the first to the last frame, what is the most plausible immediate next action?
"""

answer = vlm.ask_with_thinking(frames_for_vlm, question)
print(answer)

[VLM Q | thinking] 
1. What is the agent currently holding? Which arm (left or right) is holding it?
2. Where are the bottled coconut milk and the crispy potato chips located?
3. What is the right arm doing? Does it appear to have a barcode scanner attached?
4. Based on the motion from the first to the last frame, what is the most plausible immediate next action?

[VLM A | thinking] 1. The agent is holding a white bottle with its **left arm**.
2. The **bottled coconut milk** is being held by the agent's left arm, and the **crispy potato chips** (purple package) are located on the wooden table.
3. The right arm is moving toward the bottle to scan it; it has a black **barcode scanner** attached to its end-effector.
4. Since the right arm has finished scanning the bottle and is moving away (frames 4-5), the most plausible next action is to either place the bottle down or pick up and scan the **crispy potato chips** remaining on the table.
1. The agent is holding a white bottle with its **left arm**.
2. The **bottled coconut milk** is being held by the agent's left arm, and the **crispy potato chips** (purple package) are located on the wooden table.
3. The right arm is moving toward the bottle to scan it; it has a black **barcode scanner** attached to its end-effector.
4. Since the right arm has finished scanning the bottle and is moving away (frames 4-5), the most plausible next action is to either place the bottle down or pick up and scan the **crispy potato chips** remaining on the table.

# Subsample frames for reconstruction to stay under the 64-frame limit
subsampled_indices = np.linspace(0, len(InputImages) - 1, 32, dtype=int)
recon_frames = [InputImages[i] for i in subsampled_indices]
recon = tools.Reconstruct.Reconstruct(recon_frames)

# Segment the key objects across the video
# We'll use a subset of frames for segmentation to be efficient, 
# then track them if needed, but for now, let's try text segmentation on a few key frames.
# Actually, segment_video_by_text is better for tracking.
# We'll use a temporal window.
seg_coconut = tools.SAM3.segment_video_by_text(["bottled coconut milk"], labels=["coconut_milk"], prompt_frame_idx=0)
seg_chips = tools.SAM3.segment_video_by_text(["crispy potato chips"], labels=["chips"], prompt_frame_idx=0)
seg_scanner = tools.SAM3.segment_video_by_text(["barcode scanner"], labels=["scanner"], prompt_frame_idx=0)

# Combine masks into one PerFrameMask object if possible, or just use them separately.
# To use render_bev, we need a combined mask or a list of labels.
# Since we have three separate PerFrameMasks, we can combine them manually for the BEV.

# Let's first verify the masks on a few frames.
fi_start = seg_coconut.frame_indices[0]
fi_mid = seg_coconut.frame_indices[len(seg_coconut.frame_indices)//2]
fi_end = seg_coconut.frame_indices[-1]

show([
    InputImages[fi_start], seg_coconut.visualize(fi_start),
    InputImages[fi_mid], seg_coconut.visualize(fi_mid),
    InputImages[fi_end], seg_coconut.visualize(fi_end)
])

 [image: show_0002] 

import matplotlib.pyplot as plt

# Get the frame indices that were used for reconstruction
# Note: recon.frame_indices contains the absolute frame indices of the 32 subsampled frames
distances = []
validity = []

for fi in recon.frame_indices:
    c_scanner = seg_scanner.get_centroid_3d(recon, frame=fi, object=0)
    c_coconut = seg_coconut.get_centroid_3d(recon, frame=fi, object=0)
    
    if c_scanner is not None and c_coconut is not None:
        dist = tools.Geometry.euclidean_distance(c_scanner, c_coconut)
        distances.append(dist)
        validity.append(True)
    else:
        distances.append(np.nan)
        validity.append(False)

distances = np.array(distances)
validity = np.array(validity)

# Plot the distance to visualize the scan motion
chart = tools.Graph.plot(distances, validity=validity, y_label="Distance (m)", title="Scanner to Bottle Distance")
show(chart)
print(chart.description)

[WARNING] get_centroid_3d: mask for 'scanner' at frame 0 is empty — returning [nan, nan, nan]
[WARNING] get_centroid_3d: mask for 'coconut_milk' at frame 0 is empty — returning [nan, nan, nan]
[WARNING] get_centroid_3d: mask for 'scanner' at frame 4 is empty — returning [nan, nan, nan]
[WARNING] get_centroid_3d: mask for 'coconut_milk' at frame 4 is empty — returning [nan, nan, nan]
[WARNING] get_centroid_3d: mask for 'scanner' at frame 9 is empty — returning [nan, nan, nan]
[WARNING] get_centroid_3d: mask for 'coconut_milk' at frame 9 is empty — returning [nan, nan, nan]
[WARNING] get_centroid_3d: mask for 'scanner' at frame 13 is empty — returning [nan, nan, nan]
[WARNING] get_centroid_3d: mask for 'coconut_milk' at frame 13 is empty — returning [nan, nan, nan]
[WARNING] get_centroid_3d: mask for 'scanner' at frame 18 is empty — returning [nan, nan, nan]
[WARNING] get_centroid_3d: mask for 'coconut_milk' at frame 18 is empty — returning [nan, nan, nan]
[WARNING] get_centroid_3d: mask for 'scanner' at frame 23 is empty — returning [nan, nan, nan]
[WARNING] get_centroid_3d: mask for 'coconut_milk' at frame 23 is empty — returning [nan, nan, nan]
[WARNING] get_centroid_3d: mask for 'scanner' at frame 27 is empty — returning [nan, nan, nan]
[WARNING] get_centroid_3d: mask for 'coconut_milk' at frame 27 is empty — returning [nan, nan, nan]
[WARNING] get_centroid_3d: mask for 'scanner' at frame 32 is empty — returning [nan, nan, nan]
[WARNING] get_centroid_3d: mask for 'coconut_milk' at frame 32 is empty — returning [nan, nan, nan]
[WARNING] get_centroid_3d: mask for 'scanner' at frame 37 is empty — returning [nan, nan, nan]
[WARNING] get_centroid_3d: mask for 'coconut_milk' at frame 37 is empty — returning [nan, nan, nan]
[WARNING] get_centroid_3d: mask for 'scanner' at frame 41 is empty — returning [nan, nan, nan]
[WARNING] get_centroid_3d: mask for 'coconut_milk' at frame 41 is empty — returning [nan, nan, nan]
[WARNING] get_centroid_3d: mask for 'scanner' at frame 4

# Use frame 72 for grounding as both arm and objects are clearly visible
grounding_frame = InputImages[72]
W, H = grounding_frame.width, grounding_frame.height

# Locate the three key objects
box_coconut = vlm.locate(grounding_frame, "Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the white bottled coconut milk held by the left arm. Reply with ONLY the numbers.")
box_scanner = vlm.locate(grounding_frame, "Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the black barcode scanner on the right arm. Reply with ONLY the numbers.")
box_chips = vlm.locate(grounding_frame, "Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the purple package of crispy potato chips on the table. Reply with ONLY the numbers.")

print(f"Coconut box: {box_coconut}")
print(f"Scanner box: {box_scanner}")
print(f"Chips box: {box_chips}")

# Convert normalized coordinates to pixels
def norm_to_px(box_str, w, h):
    try:
        coords = [float(x) for x in box_str.replace(',', ' ').split()]
        return tools.Geometry.normalized_to_pixel(coords, w, h)
    except:
        return None

px_coconut = norm_to_px(box_coconut, W, H)
px_scanner = norm_to_px(box_scanner, W, H)
px_chips = norm_to_px(box_chips, W, H)

# Track objects using the boxes
# We use prompt_frame_idx=0 relative to the window. 
# Since we are using the whole video, we need to find the local index of frame 72.
# InputImages[72] is the 73rd frame.
prompt_idx = 72 

seg_coconut = tools.SAM3.segment_video_by_box([px_coconut], ["coconut_milk"], prompt_frame_idx=prompt_idx)
seg_scanner = tools.SAM3.segment_video_by_box([px_scanner], ["scanner"], prompt_frame_idx=prompt_idx)
seg_chips = tools.SAM3.segment_video_by_box([px_chips], ["chips"], prompt_frame_idx=prompt_idx)

# Verify the masks on frame 72
show([
    InputImages[72], 
    seg_coconut.visualize(72), 
    seg_scanner.visualize(72), 
    seg_chips.visualize(72)
])

[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the white bottled coconut milk held by the left arm. Reply with ONLY the numbers.
[VLM A | locate] 535 438 583 535
[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the black barcode scanner on the right arm. Reply with ONLY the numbers.
[VLM A | locate] 584 413 748 493
[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the purple package of crispy potato chips on the table. Reply with ONLY the numbers.
[VLM A | locate] 552 537 614 566
Coconut box: 535 438 583 535
Scanner box: 584 413 748 493
Chips box: 552 537 614 566

# Re-calculate distances with the new, reliable masks
distances = []
validity = []

# Use the reconstructed frames
for fi in recon.frame_indices:
    c_scanner = seg_scanner.get_centroid_3d(recon, frame=fi, object=0)
    c_coconut = seg_coconut.get_centroid_3d(recon, frame=fi, object=0)
    
    if c_scanner is not None and c_coconut is not None:
        dist = tools.Geometry.euclidean_distance(c_scanner, c_coconut)
        distances.append(dist)
        validity.append(True)
    else:
        distances.append(np.nan)
        validity.append(False)

distances = np.array(distances)
validity = np.array(validity)

# Plot the distance to visualize the scan motion
chart = tools.Graph.plot(distances, validity=validity, y_label="Distance (m)", title="Scanner to Bottle Distance")
show(chart)
print(chart.description)

 [image: show_0005] 
Plot of Distance (m) vs Frame: min=0.3652, max=0.4439, mean=0.3892, first=0.4071, last=0.4229, trend=increasing

ReturnAnswer("B")

[ReturnAnswer] Answer submitted: B