cs.CVMar 28, 2026
SaveFalcon Perception
Organizations: Falcon Vision Team, TII
Abstract
Perception-centric systems are typically implemented with a modular encoder-decoder pipeline: a vision backbone for feature extraction and a separate decoder (or late-fusion module) for task prediction. This raises a central question: is this architectural separation essential or can a single early-fusion stack do both perception and task modeling at scale? We introduce Falcon Perception, a unified dense Transformer that processes image patches and text tokens in a shared parameter space from the first layer, using a hybrid attention pattern (bidirectional among image tokens, causal for prediction tokens) to combine global visual context with autoregressive, variable-length instance generation. To keep dense outputs practical, Falcon Perception retains a lightweight token interface and decodes continuous spatial outputs with specialized heads, enabling parallel high-resolution mask prediction. Our design promotes simplicity: we keep a single scalable backbone and shift complexity toward data and training signals, adding only small heads where outputs are continuous and dense. On SA-Co, Falcon Perception improves mask quality to 68.0 Macro-F compared to 62.3 of SAM3. We also introduce PBench, a benchmark targeting compositional prompts (OCR, spatial constraints, relations) and dense long-context regimes, where the model shows better gains. Finally, we extend the same early-fusion recipe to Falcon OCR: a compact 300M-parameter model which attains 80.3% on olmOCR and 88.64 on OmniDocBench.
Figures & tables
Figure 1: Falcon Perception Architecture: A single autoregressive Transformer processes a unified sequence of image patches, text, and task tokens. The model predicts object properties in a fixed order: <coord> <size> <segm> . Bounding boxes’ coordinate and size tokens are decoded via specialized heads and re-injected as Fourier features to condition subsequent steps. High resolution segmentation masks are generated by a dot product between the <segm> token of each instance and the upsampled image features, leveraging the early-fusion backbone for instance-aware localization. Visual data flow is shown in green , coordinates in blue , size in orange , and segmentation in purple.
Figure 2: Training forward pass.
Figure 3: Prompt progression across complexity levels. We keep the image fixed and vary the text prompt to isolate the capabilities targeted by Levels 0–4 (Table 1 ). Each panel shows the predicted masks for the same scene under progressively more specific prompts: from generic object classes (Level 0, e.g . “box”), to attribute binding (Level 1, e.g . “purple box”), to OCR-based identification (Level 2, e.g . “ACME box”), and then spatial/layout constraints (Level 3, e.g . “bottle on the right”) and fine-grained relational descriptions (Level 4). Falcon Perception remains stable as prompts become more compositional and text-dependent, while SAM 3 starts to fail on OCR-driven queries and higher-level constraints, either returning no masks or segmenting visually plausible but semantically wrong instances.
| Level | Primary capability | Prompt ingredients (single-skill by construction) | Examples |
| 0 | General object classes | Rules: whole objects only; machine-detectable categories; bbox-drawable; clear boundaries; no abstract concepts. | car person tree |
| 1 | Fine-grained attributes & subtypes | Attributes: color; size; shape; material. State: old/new; clean/dirty; broken/cracked; worn/well-maintained. Subtype: sedan/SUV/pickup; oak/palm/rose; tabby/golden retriever. Functional: open/closed; on/off; active/inactive. Components: sunglasses; cracked windshield; neon sign; petals. | red car broken fence open door |
| 2 | Text as identifier (OCR) | Brand/product: Coca-Cola; Nike; iPhone. Variant: Diet vs Regular; gluten-free vs regular. Store/location: Starbucks; McDonald’s; Barnes & Noble. Signage/tags: sale tag; prescription label; emergency exit. | Diet Coke bottle Nike shoes Emergency exit door |
| 3 | Spatial relationships & layout | Relative: left/right/behind; above/below. Depth: foreground/middleground/background. Ordering: first/third/last ( e.g . , third from left). Grouping: in a row/line. Containment: inside/outside; boundary interactions. | car on the left third window from left person in the foreground |
| 4 | Relationships & interactions | Actions: holding; wearing; pulling. Functional links: key for door; charger for phone. Comparisons: tallest/smallest/most damaged. Physical: resting on; hanging on; attached to. Groupings: dining set; lock and key; tools for repair. | person holding umbrella car pulling trailer tallest building in the row |
Table 1: Complexity-level taxonomy for our internal referring expression segmentation benchmark. Each sample is assigned a single level label by construction, enabling per-level capability profiling rather than a saturated aggregate score.
Figure 4: Muon optimizer results in lower training losses for coord, size and cross-entropy, in comparison to AdamW, leading to improved performance (Tab. 3 ).
Figure 5: Raster ordering of instances results in lower training loss and better performance on both benchmarks, compared to random and size orderings.
Figure 6: Qualitative comparison across PBench complexity levels. Each column shows a different image and a prompt written to target one dominant capability (Levels 0–4; Table 1 )). The top row shows Falcon Perception predictions and the bottom row shows SAM 3. Falcon Perception produces cleaner, more instance-specific masks as prompts become more compositional (OCR, spatial, relational), while SAM 3 more often fragments the target or returns extra/incorrect masks on higher-level prompts.
| Model | Average | Metaclip | SA-1B | Crowded | Food&Drink | Sports Equip. | Attributes | Wiki-Common | ||||||||||||||||
| F 1 | pmF 1 | MCC | F 1 | pmF 1 | MCC | F 1 | pmF 1 | MCC | F 1 | pmF 1 | MCC | F 1 | pmF 1 | MCC | F 1 | pmF 1 | MCC | F 1 | pmF 1 | MCC | F 1 | pmF 1 | MCC | |
| Baselines | ||||||||||||||||||||||||
| gDino-T Liu et al. (2023) | - | 16.2 | 0.15 | - | 13.9 | 0.21 | - | 15.4 | 0.20 | - | 3.4 | 0.08 | - | 9.8 | 0.10 | - | 11.2 | 0.10 | - | 47.3 | 0.29 | - | 12.1 | 0.06 |
| OWLv2* Minderer et al. (2023) | - | 42.0 | 0.57 | - | 34.3 | 0.52 | - | 26.8 | 0.50 | - | 30.7 | 0.51 | - | 49.4 | 0.65 | - | 56.2 | 0.64 | - | 56.2 | 0.63 | - | 40.3 | 0.54 |
| OWLv2 Minderer et al. (2023) | - | 36.8 | 0.46 | - | 31.3 | 0.39 | - | 21.7 | 0.45 | - | 24.8 | 0.36 | - | 47.9 | 0.51 | - | 47.0 | 0.52 | - | 48.2 | 0.54 | - | 36.6 | 0.49 |
| LLMDet-L Fu et al. (2025) | - | 27.3 | 0.21 | - | 19.4 | 0.23 | - | 22.8 | 0.23 | - | 13.7 | 0.18 | - | 29.1 | 0.19 | - | 25.3 | 0.17 | - | 57.1 | 0.39 | - | 23.3 | 0.16 |
Table 6: SA-Co Benchmark Results. Comparison of Falcon Perception against state-of-the-art open-vocabulary segmentation models. We report Macro F 1 , Positive Micro F 1 (pmF 1 ), and Image-Level MCC across all splits.
Figure 7: Resolution Scaling. Performance as a function of image resolution. While semantic tasks ( e.g . , Attributes) are relatively robust to lower resolutions, dense and small-object tasks ( e.g . , Crowded, Sports, Dense) exhibit a phase transition, requiring high resolution to resolve individual instances.
Figure 8: Resizing Strategy. Comparison of Adaptive vs . Fixed ( ) resizing on SA-Co. Upscaling all images to a fixed high resolution consistently improves performance, particularly for crowded scenes.
Figure 9: Pass@k Performance scaling. Evolution of detection metrics across SACO (left) and PBench (right) as the number of samples increases. Performance consistently scales with , particularly on the most challenging subsets.
| Model | Average | Metaclip | SA-1B | Crowded | Food&Drink | Sports Equip. | Attributes | Wiki-Common | ||||||||||||||||
| cgF 1 | pmF 1 | MCC | cgF 1 | pmF 1 | MCC | cgF 1 | pmF 1 | MCC | cgF 1 | pmF 1 | MCC | cgF 1 | pmF 1 | MCC | cgF 1 | pmF 1 | MCC | cgF 1 | pmF 1 | MCC | cgF 1 | pmF 1 | MCC | |
| State-of-the-Art | ||||||||||||||||||||||||
| SAM 3 | 54.2 | 66.1 | 0.82 | 47.5 | 58.6 | 0.81 | 53.8 | 62.6 | 0.86 | 60.9 | 67.7 | 0.90 | 53.2 | 67.3 | 0.79 | 65.7 | 73.8 | 0.89 | 54.7 | 72.0 | 0.76 | 40.2 | 60.9 | 0.66 |
| Falcon Perception | ||||||||||||||||||||||||
| Det. (Baseline) | 34.7 | 62.9 | 0.55 | 31.0 | 55.1 | 0.56 | 32.2 | 49.8 | 0.65 | 34.3 | 60.5 | 0.57 | 38.8 | 68.6 | 0.57 | 48.3 | 72.2 | 0.67 | 39.4 | 73.1 | 0.54 | 19.3 | 60.9 | 0.32 |
| Pass@2 | 40.5 | 64.6 | 0.63 | 37.0 | 57.3 | 0.65 | 36.5 | 51.6 | 0.71 | 39.4 | 61.1 | 0.65 | 44.2 | 69.9 | 0.63 | 54.4 | 73.0 | 0.74 | 45.9 | 74.5 | 0.62 | 25.8 | 64.9 | 0.40 |
Table 9: Multi-Pass SA-Co Benchmark Results. Comparison of Falcon Perception across different baseline detection passes against SAM 3.
Figure 10: Impact of initialization: Random weight initialization results in lower performance, compared to distillation initialization, even after being trained long enough for 230K steps over pretraining and decay (stages 1 and 2) combined.
| Model | Average | ArXiv Math | Base | Hdr/Ftr | TinyTxt | MultCol | OldScan | OldMath | Tables |
| Mistral OCR 3 ( Mistral AI, 2025a ) | 81.7 | 85.4 | 99.9 | 93.8 | 88.9 | 82.1 | 48.8 | 68.3 | 86.1 |
| Chandra ( Datalab, 2025 ) | 82.0 | 81.4 | 99.8 | 88.8 | 91.9 | 82.9 | 49.2 | 73.6 | 88.2 |
| Gemini 3 Pro ( Google DeepMind, 2026 ) | 80.2 | 70.6 | 99.8 | 84.0 | 90.3 | 79.2 | 47.5 | 84.9 | 84.9 |
| PaddleOCR VL 1.5 ( Cui et al., 2026 ) | 79.3 | 85.4 | 98.8 | 96.9 | 80.8 | 82.6 | 39.2 | 66.4 | 84.1 |
| PaddleOCR VL ( Cui et al., 2025a ) | 79.2 | 85.4 | 98.6 | 96.9 | 80.8 | 82.5 | 38.8 | 66.4 | 83.9 |
| DeepSeek OCR v2 ( Wei et al., 2026 ) | 78.8 | 81.9 | 99.8 | 95.6 | 88.7 | 83.6 | 33.7 | 68.8 | 78.1 |
Table 10: olmOCR Benchmark Results. Category-wise performance comparison of FalconOCR against state-of-the-art OCR models. We report accuracy (%) across all category splits.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1 : Impact of architectural choices: Width scaling at fixed depth ( ).
Figure A.2 : Impact of architectural choices: Iso-compute depth–width trade-off.
Figure A.3 : PCA maps before and after upsampling: Image features before upsampling already distinguish multiple objects, e.g . , the coffee cup, notebook , and pen in the fourth row on the left. Similarly, chopsticks along with various food items in second row on the right and the four cups in the bottom right row can be distinguished. The feature upsampler enhances these representations, making instances more visible, suggesting that the pre-upsampling features capture sufficient scene information for instance segmentation and demonstrating the effectiveness of the early-fusion design.
Figure A.4 : Impact of the upsampling factor during inference: Bilinearly upsampling the image features at the perception model output (16x downsampled) and computing the instance masks results in degraded quality (leftmost column) with holes inside the instances and mask regions outside the instances. Processing the model output via the Anyup upsampler with different upsampling factors, followed by mask computation and bilinear upsampling of mask to image size significantly improves the mask quality (columns 2 to 6). Particularly, as the Anyup upsampling factor is increased, the instance boundaries are better refined. E.g . , in the top row, the boundaries of the fourth apple instance from left improves as the factor increases to 8 and 16. Similarly, in the third row, the beak of left penguin is incomplete for factors 1 to 4, while it is accurate for higher factors.
Figure A.5 : Level 0: While our Falcon Perception predicts correct masks, SAM 3 incorrectly predicts cupcakes in the masks for the cupcakes cream example on top row and an additional jar in the middle (middle row).
Figure A.6 : Level 1: Our Falcon perception misses few marker pens in the top row, clearly distinguishes the plant pots on the window sill from the other plant pots in the middle row. In contrast, SAM 3 identifies all the marker pens correctly but fails to distinguish between plant pots in the middle row by predicting false positives for pots not placed on window sill.
Figure A.7 : Level 2: Different to SAM 3 that incorrectly predicts masks for each bottle (top row) and each sign (middle row), our Falcon Perception model correctly identifies the 168 wine bottles (marked with 168) and the Honolulu direction sign , indicating that our model can better relate the query prompt to the OCR text in the image.
Figure A.8 : Level 3: In contrast to SAM 3 predicting false positives for meat (top row), black car (middle row) and Belgian flag (bottom row), our model correctly identifies the lower meat skewer on left grill , black car to the right of red car at bottom and Belgian flag on the left in the three examples. This shows that our Falcon Perception has better spatial understanding of the scene.
Figure A.9 : Level 4: SAM 3 wrongly outputs masks for all bread (top row), persons (middle and bottom rows). However, our Falcon Perception model correctly identifies the object instances for pastry next to brown round bread , person holding phone and person holding helmet in hand , thereby showcasing the ability to understand the object interaction in the scene.
Figure A.10 : Dense Split: While SAM 3 is capable of identifying many instances of an object, it fails to predict masks when number of instances exceeds 200 due to the fixed number of query tokens in the decoder. Our Falcon Perception, following the autoregressive approach for prediction, is not limited by this and can easily scale higher, as shown for snow goose , pigeon and colorful canned drinks in the three rows.
Figure A.11 : Effect of sampling on the quality of predictions: In the left column, we show the deterministic predictions, selecting the most likely next token, center coordinate, or size value. On the right, we show one of the predictions we obtain by sampling tokens, center, and size with a temperature of . Oftentimes, we can achieve better predictions for instances that require reasoning over the query, fine-grained perception, and small or distant objects.
Figure A.12 : Example OCR output for inputs comprising formulae, tables and handwriting (from top to bottom). Best viewed zoomed-in.
Figure A.13 : FalconOCR outputs for samples with arxiv papers, scientific formulae. Best viewed zoomed-in.
Figure A.14 : Example outputs for inputs with variations in structure of equations. Best viewed zoomed-in.
Figure A.15 : Example outputs for inputs with variations in structure of tables - sparse and dense with multi-column headers. Best viewed zoomed-in.