Omni3D — Gemma4-31B — sample 146

# Ground the right-most chair and the wooden cabinet
chair_box_norm = vlm.locate(InputImages[0], "Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the right-most chair. Reply with ONLY the numbers.")
cabinet_box_norm = vlm.locate(InputImages[0], "Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the wooden cabinet. Reply with ONLY the numbers.")

print(f"Chair box (norm): {chair_box_norm}")
print(f"Cabinet box (norm): {cabinet_box_norm}")

[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the right-most chair. Reply with ONLY the numbers.
[VLM A | locate] 621 533 961 1000
[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the bounding box (x1, y1, x2, y2) in 0-1000 normalized scale for the wooden cabinet. Reply with ONLY the numbers.
[VLM A | locate] 238, 517, 704, 993
Chair box (norm): 621 533 961 1000
Cabinet box (norm): 238, 517, 704, 993

# Parse normalized coordinates
def parse_coords(coords_str):
    return [int(c) for c in coords_str.replace(',', ' ').split()]

chair_norm = parse_coords(chair_box_norm)
cabinet_norm = parse_coords(cabinet_box_norm)

# Convert to pixels
W, H = InputImages[0].width, InputImages[0].height
chair_px = tools.Geometry.normalized_to_pixel(chair_norm, W, H)
cabinet_px = tools.Geometry.normalized_to_pixel(cabinet_norm, W, H)

# Segment the objects
seg_chair = tools.SAM3.segment_image_by_box(InputImages[0], chair_px, label="chair")
seg_cabinet = tools.SAM3.segment_image_by_box(InputImages[0], cabinet_px, label="cabinet")

# Verify masks
fi = InputImages[0].frame_index
vis_chair = seg_chair.visualize(fi)
vis_cabinet = seg_cabinet.visualize(fi)

show([InputImages[0], vis_chair, vis_cabinet])

# Programmatic check for mask area
for seg, name in [(seg_chair, "chair"), (seg_cabinet, "cabinet")]:
    area = seg.get_mask(frame=fi, object=0).sum()
    print(f"{name} mask area: {area} pixels")

 [image: show_0001] 
chair mask area: 33289 pixels
cabinet mask area: 62819 pixels

# Perform 3D reconstruction for the single frame
recon = tools.Reconstruct.Reconstruct(InputImages[:1])
print("Reconstruction complete.")

Reconstruction complete.

import numpy as np

# 1. Re-establish coordinates and masks
chair_box_norm_str = "621 533 961 1000"
cabinet_box_norm_str = "238, 517, 704, 993"

def parse_coords(coords_str):
    return [int(c) for c in coords_str.replace(',', ' ').split()]

W, H = InputImages[0].width, InputImages[0].height
chair_px = tools.Geometry.normalized_to_pixel(parse_coords(chair_box_norm_str), W, H)
cabinet_px = tools.Geometry.normalized_to_pixel(parse_coords(cabinet_box_norm_str), W, H)

seg_chair = tools.SAM3.segment_image_by_box(InputImages[0], chair_px, label="chair")
seg_cabinet = tools.SAM3.segment_image_by_box(InputImages[0], cabinet_px, label="cabinet")

# 2. Extract 3D points
fi = InputImages[0].frame_index
# Get points and confidence
chair_pts = seg_chair.get_masked_points(recon, frame=fi, object=0)
cabinet_pts = seg_cabinet.get_masked_points(recon, frame=fi, object=0)

# Filter by confidence to remove outliers
# recon.points[fi] is (H, W, 3), recon.points.confidence[fi] is (H, W)
# We need to find which points in the mask are high confidence.
# A simpler way: just use the points returned by get_masked_points if they are already filtered,
# but let's check the confidence of the points if possible.
# Since get_masked_points returns world points, we can't easily map them back to the confidence map
# without the mask. Let's use the mask to filter the point cloud directly.

def get_filtered_points(recon, seg, frame_idx, obj_idx, conf_thresh=0.5):
    mask = seg.get_mask(frame=frame_idx, object=obj_idx)
    points = recon.points[frame_idx]
    conf = recon.points.confidence[frame_idx]
    
    # Mask the points and the confidence
    valid_mask = mask & (conf > conf_thresh)
    filtered_pts = points[valid_mask]
    return filtered_pts

chair_pts_filt = get_filtered_points(recon, seg_chair, fi, 0)
cabinet_pts_filt = get_filtered_points(recon, seg_cabinet, fi, 0)

# 3. Calculate dimensions
# Chair width: extent in XZ plane
if len(chair_pts_filt) > 0:
    chair_xz = chair_pts_filt[:, [0, 2]]
    # Width is the max distance between any two points in XZ plane, 
    # or more simply for a chair, the distance between the min and max X/Z
    # Since the chair is roughly facing us, width is likely along X.
    # Let's use the max distance between any two points for robustness.
    from scipy.spatial.distance import pdist
    if len(chair_xz) > 1:
        w_measured = np.max(pdist(chair_xz))
    else:
        w_measured = 0.0
else:
    w_measured = 0.0

# Cabinet height: extent in Y axis
if len(cabinet_pts_filt) > 0:
    y_coords = cabinet_pts_filt[:, 1]
    h_measured = np.max(y_coords) - np.min(y_coords)
else:
    h_measured = 0.0

print(f"Filtered chair points: {len(chair_pts_filt)}")
print(f"Filtered cabinet points: {len(cabinet_pts_filt)}")
print(f"Measured chair width (recon units): {w_measured}")
print(f"Measured cabinet height (recon units): {h_measured}")

Filtered chair points: 33289
Filtered cabinet points: 62819
Measured chair width (recon units): 0.7388629031471092
Measured cabinet height (recon units): 0.7693432569503784

import numpy as np
from sklearn.decomposition import PCA

# 1. Refine Chair Width Measurement using PCA
# We use the XZ plane to find the horizontal dimensions
chair_xz = chair_pts_filt[:, [0, 2]]
pca = PCA(n_components=2)
pca.fit(chair_xz)

# The first component is the length (depth), the second is the width
width_axis = pca.components_[1] 
# Project points onto the width axis
projections = chair_xz @ width_axis
w_measured_refined = np.max(projections) - np.min(projections)

# 2. Refine Cabinet Height Measurement
# Use the Y-coordinates of the filtered points
cabinet_y = cabinet_pts_filt[:, 1]
h_measured_refined = np.max(cabinet_y) - np.min(cabinet_y)

# 3. Calculate the actual height
# Given: Actual Chair Width = 0.2m
w_actual = 0.2
scale_factor = w_actual / w_measured_refined
h_actual = h_measured_refined * scale_factor

print(f"Refined measured chair width (recon units): {w_measured_refined:.4f}")
print(f"Refined measured cabinet height (recon units): {h_measured_refined:.4f}")
print(f"Scale factor: {scale_factor:.4f}")
print(f"Actual cabinet height: {h_actual:.4f} m")

# Sanity check: ratio of height to width
print(f"Ratio H_cab / W_chair: {h_measured_refined / w_measured_refined:.4f}")

Refined measured chair width (recon units): 0.6181
Refined measured cabinet height (recon units): 0.7693
Scale factor: 0.3236
Actual cabinet height: 0.2489 m
Ratio H_cab / W_chair: 1.2447

# 1. Locate specific boundary points for more accurate measurement
# Chair width: left and right edges of the seat
chair_l_norm = vlm.locate(InputImages[0], "Give the (x, y) center coordinates in 0-1000 normalized scale for the left edge of the seat of the right-most chair. Reply with ONLY the numbers.")
chair_r_norm = vlm.locate(InputImages[0], "Give the (x, y) center coordinates in 0-1000 normalized scale for the right edge of the seat of the right-most chair. Reply with ONLY the numbers.")

# Cabinet height: top and bottom edges
cab_top_norm = vlm.locate(InputImages[0], "Give the (x, y) center coordinates in 0-1000 normalized scale for the top edge of the wooden cabinet. Reply with ONLY the numbers.")
cab_bot_norm = vlm.locate(InputImages[0], "Give the (x, y) center coordinates in 0-1000 normalized scale for the bottom edge of the wooden cabinet. Reply with ONLY the numbers.")

def get_3d_point(norm_coords_str):
    coords = [int(c) for c in norm_coords_str.replace(',', ' ').split()]
    px, py = tools.Geometry.normalized_to_pixel(coords, W, H)
    # Ensure coordinates are within image bounds
    px = int(np.clip(px, 0, W - 1))
    py = int(np.clip(py, 0, H - 1))
    return recon.points[fi][py, px]

p_chair_l = get_3d_point(chair_l_norm)
p_chair_r = get_3d_point(chair_r_norm)
p_cab_top = get_3d_point(cab_top_norm)
p_cab_bot = get_3d_point(cab_bot_norm)

# 2. Calculate distances
w_measured_pts = tools.Geometry.euclidean_distance(p_chair_l, p_chair_r)
h_measured_pts = tools.Geometry.euclidean_distance(p_cab_top, p_cab_bot)

# 3. Final Metric Calculation
w_actual = 0.2
scale_factor = w_actual / w_measured_pts
h_actual = h_measured_pts * scale_factor

print(f"Chair L: {p_chair_l}, Chair R: {p_chair_r}")
print(f"Cab Top: {p_cab_top}, Cab Bot: {p_cab_bot}")
print(f"Measured chair width (pts): {w_measured_pts:.4f}")
print(f"Measured cabinet height (pts): {h_measured_pts:.4f}")
print(f"Ratio H/W: {h_measured_pts / w_measured_pts:.4f}")
print(f"Actual cabinet height: {h_actual:.4f} m")

[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the (x, y) center coordinates in 0-1000 normalized scale for the left edge of the seat of the right-most chair. Reply with ONLY the numbers.
[VLM A | locate] 640 920
[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the (x, y) center coordinates in 0-1000 normalized scale for the right edge of the seat of the right-most chair. Reply with ONLY the numbers.
[VLM A | locate] 910 920
[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line explaining what you saw instead) — do NOT return coordinates in that case. Only if the requested target is clearly and unambiguously present, answer the request below.

Request: Give the (x, y) center coordinates in 0-1000 normalized scale for the top edge of the wooden cabinet. Reply with ONLY the numbers.
[VLM A | locate] 472 520
[VLM Q | locate] Before answering, first check whether what the question describes is clearly visible in the provided image(s). If it is absent or ambiguous, reply EXACTLY `Not visible` on its own line (optionally followed by one short line exp

ReturnAnswer(0.35)

[ReturnAnswer] Answer submitted: 0.35