Video-MME-v2 — Gemma4-31B — sample 448-1

# Show the first few frames to visually identify the foxes
show([InputImages[0], InputImages[1], InputImages[2]])

# Use VLM to identify the stationary fox and its facing direction
# We use the first 5 frames to provide temporal context for "stationary" vs "running"
question = (
    "In these frames, there are three little foxes. "
    "One is stationary at the entrance of the den, and two are running. "
    "1. Which fox is the stationary one? Describe its position. "
    "2. In which direction is the stationary fox facing? (e.g., facing the camera, facing right, etc.) "
    "3. In which general direction are the other two foxes running relative to the frame?"
)
vlm_answer = vlm.ask_with_thinking(InputImages[:5], question)
print(f"VLM Answer:\n{vlm_answer}")

 [image: show_0001] 
[VLM Q | thinking] In these frames, there are three little foxes. One is stationary at the entrance of the den, and two are running. 1. Which fox is the stationary one? Describe its position. 2. In which direction is the stationary fox facing? (e.g., facing the camera, facing right, etc.) 3. In which general direction are the other two foxes running relative to the frame?
[VLM A | thinking] Based on the provided frames:

1. **The stationary fox** is the small pup located at the bottom center, peeking out from the entrance of the den (the gap under the logs).
2. The stationary fox is **facing the camera**.
3. The other foxes are running toward the **right** side of the frame (visible in frame 0).
VLM Answer:
Based on the provided frames:

1. **The stationary fox** is the small pup located at the bottom center, peeking out from the entrance of the den (the gap under the logs).
2. The stationary fox is **facing the camera**.
3. The other foxes are running toward the **right** side of the frame (visible in frame 0).

# Segment foxes in the first 16 frames
# start_frame and end_frame are absolute video frame indices
start_f = InputImages[0].frame_index
end_f = InputImages[15].frame_index

seg = tools.SAM3.segment_video_by_text(
    prompts=["fox"], 
    prompt_frame_idx=0, 
    start_frame=start_f, 
    end_frame=end_f
)

print(f"Number of objects detected: {seg.num_objects}")

# Verify the masks on the first frame
fi = seg.frame_indices[0]
vis = seg.visualize(fi)
show([InputImages[0], vis])

# Programmatic check for mask areas in the first frame
for i in range(seg.num_objects):
    mask = seg.get_mask(frame=fi, object=i)
    area = mask.sum()
    print(f"Object {i} area: {area} pixels")

Number of objects detected: 1
 [image: show_0002] 
Object 0 area: 974 pixels

# Locate the three foxes in the first frame
# 1. Stationary fox
# 2. Running fox 1
# 3. Running fox 2
img = InputImages[0]
W, H = img.width, img.height

box_stationary = vlm.locate(img, "Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the stationary fox peeking from the den. Reply with ONLY the numbers.")
box_run1 = vlm.locate(img, "Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the running fox on the right. Reply with ONLY the numbers.")
box_run2 = vlm.locate(img, "Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the other running fox. Reply with ONLY the numbers.")

print(f"Stationary box: {box_stationary}")
print(f"Running 1 box: {box_run1}")
print(f"Running 2 box: {box_run2}")

def parse_box(box_str):
    # Remove any non-numeric characters except commas/spaces/dots
    import re
    nums = re.findall(r"[-+]?\d*\.\d+|\d+", box_str)
    return [float(n) for n in nums]

# Convert normalized to pixel coordinates
try:
    b_stat_norm = parse_box(box_stationary)
    b_run1_norm = parse_box(box_run1)
    b_run2_norm = parse_box(box_run2)
    
    b_stat = tools.Geometry.normalized_to_pixel(b_stat_norm, W, H)
    b_run1 = tools.Geometry.normalized_to_pixel(b_run1_norm, W, H)
    b_run2 = tools.Geometry.normalized_to_pixel(b_run2_norm, W, H)
    
    boxes = [b_stat, b_run1, b_run2]
    labels = ["stationary_fox", "running_fox_1", "running_fox_2"]
    
    # Segment and track the foxes
    seg = tools.SAM3.segment_video_by_box(
        boxes=boxes, 
        labels=labels, 
        prompt_frame_idx=0, 
        start_frame=start_f, 
        end_frame=end_f
    )
    
    # Verify masks on the first frame
    fi = seg.frame_indices[0]
    vis = seg.visualize(fi)
    show([img, vis])
    
    for i, label in enumerate(seg.labels):
        area = seg.get_mask(frame=fi, object=i).sum()
        print(f"{label} area: {area} pixels")

except Exception as e:
    print(f"Error during segmentation: {e}")

[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the stationary fox peeking from the den. Reply with ONLY the numbers.
[VLM A | locate] 422 692 483 786
[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the running fox on the right. Reply with ONLY the numbers.
[VLM A | locate] 682, 546, 1000, 807
[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the other running fox. Reply with ONLY the numbers.
[VLM A | locate] Not visible
Only one running fox is present; the other fox is peeking from under the logs.
Stationary box: 422 692 483 786
Running 1 box: 682, 546, 1000, 807
Running 2 box: Not visible
Only one running fox is present; the other fox is peeking from under the logs.
E

# Use VLM to find all three foxes across the first 16 frames
question = (
    "In the first 16 frames, there are three little foxes: one stationary at the den and two running. "
    "Please identify the frames where all three are clearly visible. "
    "For those frames, describe the positions of the three foxes (stationary, running 1, running 2) "
    "relative to the frame (e.g., center, right, far right)."
)
vlm_fox_analysis = vlm.ask_with_thinking(InputImages[:16], question)
print(f"Fox Analysis:\n{vlm_fox_analysis}")

# Let's also just show a few more frames to see them ourselves
show([InputImages[0], InputImages[4], InputImages[8], InputImages[12]])

[VLM Q | thinking] In the first 16 frames, there are three little foxes: one stationary at the den and two running. Please identify the frames where all three are clearly visible. For those frames, describe the positions of the three foxes (stationary, running 1, running 2) relative to the frame (e.g., center, right, far right).
[VLM A | thinking] Based on the provided frames, there are no frames where three little foxes are clearly visible. In frame 0, only two foxes are visible: one stationary in the den (center) and one running to the right (far right). In frames 1 and 2, only the stationary fox in the den is visible. Frames 3 through 5 show a close-up of a single fox, and the remaining frames feature different animals.
Fox Analysis:
Based on the provided frames, there are no frames where three little foxes are clearly visible. In frame 0, only two foxes are visible: one stationary in the den (center) and one running to the right (far right). In frames 1 and 2, only the stationary fox in the den is visible. Frames 3 through 5 show a close-up of a single fox, and the remaining frames feature different animals.
 [image: show_0003] 

ReturnAnswer("A")

[ReturnAnswer] Answer submitted: A