Boats from broadcast video

A computer vision pipeline that takes a YouTube sailing broadcast and recovers, for every frame, the camera's pose and each boat's position and heading in metres, with no calibration target, no GPS and no human in the loop. Final project for Brown's computer vision course, spring 2026, built by a team of three. Code: github.com/quinnbrighton/vision-final-project-quinn-ali-nish.

172 / 172held-out frames solved
3.88 pxmedian reprojection error
6.18 pxworst frame (target 15)
416training frames labelled

What it does

A solved broadcast frame: detected keypoints, the reprojected boat template, recovered hull and mast, and heading arrows
One solved frame. Cyan rings are the detected keypoints (mast tip, mast base, bow, stern); orange dots are the boat template reprojected through the solved camera; white is the recovered hull, cyan the mast, green the heading.

The input is the broadcast of the ICSA Open Team Race Nationals 2025, a single moving camera on a boat with no known position, height or lens. The output for each frame is the camera (pitch, yaw, height, focal length) and, for each boat, its x, y, heading and heel in a metric world frame, from which the race can be replayed from above.

How it works

  1. Features. A frozen DINOv3 vision transformer (ViT-S/16 at stride 8) turns each frame into a feature map. Nothing in the backbone is trained.
  2. Three keypoint heads on top of the same features: a tile head that produces a heatmap of mast tips for every visible boat; an attention head that finds the mast base for each tip, attending around the tip location; and a second attention head that finds bow and stern from the base and tip context. Trained on 416 hand-labelled frames from the first eight minutes of the broadcast, with self-distillation to extend the labels.
  3. Per-frame least-squares solver. With four pixels per boat and a boat template (hull 4.2 m, mast 5.2 m), the solver jointly fits the camera's six parameters and each boat's four so that the template reprojects onto the detections, with iterative outlier rejection: drop the worst keypoint and refit until every residual is under 15 px. Each frame is solved independently; no temporal smoothing is used, which makes the per-frame numbers an honest test.
A second solved frame with several boats on a crowded start line
A crowded frame. Boats partly occluded by one another are still solved from whichever keypoints survive outlier rejection.

Results

On the default 43-second held-out clip, re-sampled at 4 frames per second:

MetricValue
Frames solved172 of 172
Median reprojection RMS3.88 px
25th / 75th percentile3.48 / 4.46 px
90th percentile4.98 px
Worst frame6.18 px

Every solved frame is under 7 px of reprojection error on 1080p video, against a 15 px target. The pipeline runs end to end from a YouTube URL and a time range: download the section, extract frames, run the heads, solve, and write a JSON and an overlay per frame plus a summary.

A third solved frame showing the recovered headings of several boats
Recovered headings on a leg. The solved world coordinates are what a tactical replay or a boat-speed comparison would be built on.

Limits