cs.CVAug 19, 2026
SaveTeach a Molmo2Fish: Towards interactive fish tracking with natural language guidance
Organizations: Massachusetts Institute of Technology · Massachusetts Institute of Technology, Cambridge MA 02139, USA
Abstract
Computer vision is increasingly used to automate recognition tasks in large ecological datasets, but more complex tasks such as multi-object tracking continue to pose challenges. As researchers seek to incorporate vision models in ecology workflows, various lines of research have explored how to make imperfect predictions useful through human-in-the-loop processes. We propose a new approach to working with imperfect tracking predictions through an interactive prediction correction workflow taking place as a conversation with a multimodal large language model, which we tailor to a sonar fish tracking dataset as an initial proof of concept. We investigate the performance of the tool, Molmo2Fish, across guided and unguided tasks, correcting its own predicted tracks and external tracks. We find that Molmo2Fish achieves high performance on fish tracking and track correction tasks, but there is still much room to improve on incorporating natural language guidance. The code and data are publicly available at https://github.com/tidalove/molmo2fish.
Figures & tables
Figure 1: Molmo2Fish is an interactive tool for creating and editing multi-object tracking predictions. Users interact with a multimodal LLM that has been fine-tuned for fish tracking (top) and correcting tracks with natural language (bottom).
Figure 2: Molmo2Fish architecture. Molmo2Fish consists of the same three components as the Molmo2 MLLM we build upon: a video encoder, a LLM, and a “connector” that projects video representations into the LLM’s representation space. Video frames and prompts are encoded and processed together, and outputs take the form of text token predictions. We extend the training protocol of the original Molmo2 to enable multi-turn corrections by constructing prompts as illustrated: first, the original video frames and a general prompt such as “Track all fish”; then, a simulated text response from Molmo2 containing erroneous predictions; and finally a second textual correction prompt.
| Name | Description | Splits |
| Pure tracking | Generic “track all fish” task, target is all points on all fish on all frames | train, val |
| Guided tracking | “Track X fish”, where X is some descriptor: “track the last fish”, “track the top three fish”, etc; target is points only on those identified fish | train, val |
| Synthetic track corrections | Tracking corrections generated synthetically | train, val |
| Targeted track corrections | Tracking corrections fixing only target mistakes | train, val |
| Real track corrections | Tracking corrections generated from real model predictions: | |
| YOLO+SORT | Tracking corrections with pre-correction prediction from YOLO+SORT predictions | val only |
Table 1: Dataset tasks. Overview of all tasks used in the dataset, for both training and evaluation.
Figure 3: From a video and its corresponding ground truth annotations, we construct three sets of predictions to use as the “pre-correction” step during training and validation: “synthetic” which applies a corruption function to ground truth, and “Molmo-low” and “Molmo-high” which come from a model’s tracking predictions on the clip.
| Corruption | Function |
|---|---|
| add | Adds a spurious track |
| delete | Removes one track |
| fragment | Splits a track into two, assigning one track ID before a chosen frame and another after |
| merge | Picks two tracks to assign to the same track ID |
| deviate | Applies a spatial displacement |
| partial_delete | Removes a portion of a track |
Table 2: All possible corruptions to apply to ground truth tracks (possibly in combination) to generate the trajectories in synthetic correction dataset.
Figure 4: An illustration of the add , delete , fragment , merge , and deviate corruptions, with line color corresponding to predicted track ID.
Figure 5: HOTA of pre-correction (dashed lines) and post-correction (bars) predictions on each river. Pre-correction predictions are generated by Molmo and YOLO+SORT respectively; post-correction predictions are generated by Molmo2Fish from either the Molmo-high or YOLO+SORT baseline.
| Molmo2 | LLM | LLM+Connector | LLM+Connector+ViT | |
|---|---|---|---|---|
| Pure Tracking | 0.049 | 0.608 | 0.656 | 0.737 |
| Synthetic Corrections | 0.484 | 0.484 | 0.484 | 0.484 |
| No-info | — | 0.757 | 0.769 | 0.799 |
| Full | 0.164 | 0.767 | 0.780 | 0.817 |
| Targeted Corrections | 0.843 | 0.843 | 0.843 | |
| No-info | — | 0.623 | 0.632 | 0.640 |
Table 3: Fine-tuning ablation. HOTA on each validation set for Molmo2 out-of-the-box (no fine-tuning) and Molmo2 trained by fine-tuning the LLM only, LLM and connector only, and all components. For all correction tasks, the pre-correction HOTA of the step-1, corrupted tracks is listed in gray next to the task name, and the reported score is HOTA on the entire validation set after a single correction step on each video.
| Pure tracking | + synthetic | + Molmo-low | + Molmo-high | +120 steps | |
|---|---|---|---|---|---|
| Pure Tracking | 0.780 | 0.763 | 0.737 | 0.729 | 0.793 |
| Synthetic Corrections | 0.484 | 0.484 | 0.484 | 0.484 | 0.484 |
| No-info | — | 0.820 | 0.799 | 0.766 | 0.803 |
| Full | 0.415 | 0.833 | 0.817 | 0.811 | 0.848 |
| Molmo-low Corrections | 0.482 | 0.482 | 0.482 | 0.482 | |
| No-info | — | 0.473 | 0.615 | 0.567 | 0.576 |
Table 4: Training dataset ablation. HOTA after LoRA fine-tuning all model components for 300 steps on each dataset mixture, for a model trained only on the tracking task; only on tracking and synthetic corrections; on tracking, synthetic corrections, and the Molmo-low set; and on all training sets. (We include an extra rightmost column reflecting a training mixture containing all sets showing that performance can be further improved by training for more steps, but opted to compare all other models at 300 steps.) For all correction tasks, the pre-correction HOTA of the step-1, corrupted tracks is listed in gray next to the task name, and the reported score is HOTA on the entire validation set after a single correction step on each video. Only the models that saw Molmo-low data during training make appreciable corrections to Molmo-low trajectories. Despite training on Molmo-high, the rightmost columns’ models still fail to make significant corrections to Molmo-high trajectories.
Figure 6: HOTA before and after correction, with no-info prompts (top row) and full prompts (bottom row), for each subset of the track correction evaluation data. Each dot represents a single trajectory. For the targeted track correction task (where HOTA of 1.0 corresponds to the partially-corrupted ground truth as described in Sec. 4.2 ), a generic “Fix all tracks” prompt results in a large aggregate drop in HOTA (top row), while Molmo2Fish makes the targeted correction successfully when the prompt specifies what to fix (bottom row). In other tasks, the change between no-info and full corrections is subtler.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Examples of Molmo2Fish ’s tracking predictions (middle row) compared against YOLO+SORT’s tracking predictions (top row) on two clips, from the Rightbank (left group) and Nushagak (right group) locations. Left: Molmo2Fish successfully carries some tracks through the whole video that YOLO+SORT drops on some frames, but Molmo2Fish ’s predicted track on the topmost fish stalls and stops following it, while YOLO+SORT’s prediction doesn’t. Right: Molmo2Fish successfully localizes fish on more frames for more of the video but there are so many fish that it “loses track” and begins to extrapolate tracks horizontally rather than ending them.
Figure 8: Example of Molmo2Fish making a better correction (second row) from the same pre-correction step (top row) when provided with a detailed prompt (right) vs when provided with a generic prompt (left). The ground truth tracks are displayed on the same frame sequence in the bottom row for comparison. Here, Molmo2Fish recognizes the upper left blob as a fish unless explicitly told by the user that it is a false detection.
Figure 9: Similar to Fig. 8 , an example where Molmo2Fish makes a better correction, as measured by HOTA, when provided with a detailed prompt vs without. Molmo2Fish thinks the mistake is that the top fish (blue) is a false detection: actually the mistake pointed out by the user is that the first track (red) must be present longer.
Figure 10: Example of Molmo2Fish making a worse correction when provided with a detailed prompt vs without. Since Molmo2Fish can’t recognize the presence of the difficult-to-localize fish that appears early in the video, it merely refrains from making a correction at all when prompted to “Adjust the tracks if anything seems incorrect”, whereas it puts a point down randomly (and incorrectly) when told that an additional fish does exist in that region.
Figure 11: Example of Molmo2Fish making a worse correction when provided with a detailed prompt vs without. Here Molmo2Fish is diverted from its initially correct prediction of the first fish’s track by the user prompt.
| Trained on ID only | Trained on all | ||
|---|---|---|---|
| OOD set | Pre-correction baseline | 0.628 | 0.628 |
| No-info | 0.810 | 0.856 | |
| Full | 0.824 | 0.882 | |
| ID set | Pre-correction baseline | 0.432 | 0.432 |
| No-info | 0.805 | 0.806 | |
| Full | 0.823 | 0.815 |
Table 5: HOTA on evaluation of models trained on pure tracking and either a) ID set, only synthetic trajectories not including held-out corruptions or b) all synthetic trajectories. Both models perform similarly evaluated on the ID set, which contains corruptions both saw during training. A large performance gap exists between the two models when evaluated on the OOD set, which is not recovered by natural language guidance.
| Kenai | Rightbank | Elwha | Nushagak | ||
| Train | |||||
| Clips | 482 | 512 | 175 | 55 | |
| Sliced clips | 858 | 844 | 273 | 154 | |
| Examples | Pure tracking | 858 | 844 | 273 | 154 |
| Guided tracking | 2048 | 2130 | 492 | 449 | |
| Synthetic corrections | 2104 | 2116 | 404 | 602 | |
Table 6: Full dataset statistics. Split by location, for the train and validation sets. “Clips” refers to number of clips in original CFC dataset: we randomly allocate clips to the train or validation set before slicing to fit in Molmo2Fish ’s context window.
More in this topic
Visual Object Tracking
Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.
See all 175 papers