Perception-centric systems are typically implemented with a modular encoder-decoder pipeline: a vision backbone for feature extraction and a separate decoder (or late-fusion module) for task prediction. This raises a central question: is this architectural separation essential or can a single early-fusion stack do both perception and task modeling at scale? We introduce Falcon Perception, a unified dense Transformer that processes image patches and text tokens in a shared parameter space from the first layer, using a hybrid attention pattern (bidirectional among image tokens, causal for prediction tokens) to combine global visual context with autoregressive, variable-length instance generation. To keep dense outputs practical, Falcon Perception retains a lightweight token interface and decodes continuous spatial outputs with specialized heads, enabling parallel high-resolution mask prediction. Our design promotes simplicity: we keep a single scalable backbone and shift complexity toward data and training signals, adding only small heads where outputs are continuous and dense. On SA-Co, Falcon Perception improves mask quality to 68.0 Macro-F1 compared to 62.3 of SAM3. We also introduce PBench, a benchmark targeting compositional prompts (OCR, spatial constraints, relations) and dense long-context regimes, where the model shows better gains. Finally, we extend the same early-fusion recipe to Falcon OCR: a compact 300M-parameter model which attains 80.3% on olmOCR and 88.64 on OmniDocBench.
Figures & tables
Figure 1: Falcon Perception Architecture: A single autoregressive Transformer processes a unified sequence of image patches, text, and task tokens. The model predicts object properties in a fixed order: <coord> → <size> → <segm> . Bounding boxes’ coordinate and size tokens are decoded via specialized heads and re-injected as Fourier features to condition subsequent steps. High resolution segmentation masks are generated by a dot product between the <segm> token of each instance and the upsampled image features, leveraging the early-fusion backbone for instance-aware localization. Visual data flow is shown in green , coordinates in blue , size in orange , and segmentation in purple.
Figure 2: Training forward pass.
Figure 3: Prompt progression across complexity levels. We keep the image fixed and vary the text prompt to isolate the capabilities targeted by Levels 0–4 (Table 1 ). Each panel shows the predicted masks for the same scene under progressively more specific prompts: from generic object classes (Level 0, e.g . “box”), to attribute binding (Level 1, e.g . “purple box”), to OCR-based identification (Level 2, e.g . “ACME box”), and then spatial/layout constraints (Level 3, e.g . “bottle on the right”) and fine-grained relational descriptions (Level 4). Falcon Perception remains stable as prompts become more compositional and text-dependent, while SAM 3 starts to fail on OCR-driven queries and higher-level constraints, either returning no masks or segmenting visually plausible but semantically wrong instances.
Brand/product: Coca-Cola; Nike; iPhone. Variant: Diet vs Regular; gluten-free vs regular. Store/location: Starbucks; McDonald’s; Barnes & Noble. Signage/tags: sale tag; prescription label; emergency exit.
Diet Coke bottle Nike shoes Emergency exit door
3
Spatial relationships & layout
Relative: left/right/behind; above/below. Depth: foreground/middleground/background. Ordering: first/third/last ( e.g . , third from left). Grouping: in a row/line. Containment: inside/outside; boundary interactions.
car on the left third window from left person in the foreground
4
Relationships & interactions
Actions: holding; wearing; pulling. Functional links: key for door; charger for phone. Comparisons: tallest/smallest/most damaged. Physical: resting on; hanging on; attached to. Groupings: dining set; lock and key; tools for repair.
person holding umbrella car pulling trailer tallest building in the row
Table 1: Complexity-level taxonomy for our internal referring expression segmentation benchmark. Each sample is assigned a single level label by construction, enabling per-level capability profiling rather than a saturated aggregate score.
Figure 4: Muon optimizer results in lower training losses for coord, size and cross-entropy, in comparison to AdamW, leading to improved performance (Tab. 3 ).
Table 6
Figure 5: Raster ordering of instances results in lower training loss and better performance on both benchmarks, compared to random and size orderings.
Figure 6: Qualitative comparison across PBench complexity levels. Each column shows a different image and a prompt written to target one dominant capability (Levels 0–4; Table 1 )). The top row shows Falcon Perception predictions and the bottom row shows SAM 3. Falcon Perception produces cleaner, more instance-specific masks as prompts become more compositional (OCR, spatial, relational), while SAM 3 more often fragments the target or returns extra/incorrect masks on higher-level prompts.
Model
Average
Metaclip
SA-1B
Crowded
Food&Drink
Sports Equip.
Attributes
Wiki-Common
F 1
pmF 1
MCC
F 1
pmF 1
MCC
F 1
pmF 1
MCC
F 1
pmF 1
MCC
F 1
pmF 1
MCC
F 1
pmF 1
MCC
F 1
pmF 1
MCC
F 1
pmF 1
MCC
Baselines
gDino-T Liu et al. (2023)
-
16.2
0.15
-
13.9
0.21
-
15.4
0.20
-
3.4
0.08
-
9.8
0.10
-
11.2
0.10
-
47.3
0.29
-
12.1
0.06
OWLv2* Minderer et al. (2023)
-
42.0
0.57
-
34.3
0.52
-
26.8
0.50
-
30.7
0.51
-
49.4
0.65
-
56.2
0.64
-
56.2
0.63
-
40.3
0.54
OWLv2 Minderer et al. (2023)
-
36.8
0.46
-
31.3
0.39
-
21.7
0.45
-
24.8
0.36
-
47.9
0.51
-
47.0
0.52
-
48.2
0.54
-
36.6
0.49
LLMDet-L Fu et al. (2025)
-
27.3
0.21
-
19.4
0.23
-
22.8
0.23
-
13.7
0.18
-
29.1
0.19
-
25.3
0.17
-
57.1
0.39
-
23.3
0.16
Table 6: SA-Co Benchmark Results. Comparison of Falcon Perception against state-of-the-art open-vocabulary segmentation models. We report Macro F 1 , Positive Micro F 1 (pmF 1 ), and Image-Level MCC across all splits.
Figure 7: Resolution Scaling. Performance as a function of image resolution. While semantic tasks ( e.g . , Attributes) are relatively robust to lower resolutions, dense and small-object tasks ( e.g . , Crowded, Sports, Dense) exhibit a phase transition, requiring high resolution to resolve individual instances.
Figure 8: Resizing Strategy. Comparison of Adaptive vs . Fixed ( 10242 ) resizing on SA-Co. Upscaling all images to a fixed high resolution consistently improves performance, particularly for crowded scenes.
Figure 9: Pass@k Performance scaling. Evolution of detection metrics across SACO (left) and PBench (right) as the number of samples k increases. Performance consistently scales with k , particularly on the most challenging subsets.
Model
Average
Metaclip
SA-1B
Crowded
Food&Drink
Sports Equip.
Attributes
Wiki-Common
cgF 1
pmF 1
MCC
cgF 1
pmF 1
MCC
cgF 1
pmF 1
MCC
cgF 1
pmF 1
MCC
cgF 1
pmF 1
MCC
cgF 1
pmF 1
MCC
cgF 1
pmF 1
MCC
cgF 1
pmF 1
MCC
State-of-the-Art
SAM 3
54.2
66.1
0.82
47.5
58.6
0.81
53.8
62.6
0.86
60.9
67.7
0.90
53.2
67.3
0.79
65.7
73.8
0.89
54.7
72.0
0.76
40.2
60.9
0.66
Falcon Perception
Det. (Baseline)
34.7
62.9
0.55
31.0
55.1
0.56
32.2
49.8
0.65
34.3
60.5
0.57
38.8
68.6
0.57
48.3
72.2
0.67
39.4
73.1
0.54
19.3
60.9
0.32
Pass@2
40.5
64.6
0.63
37.0
57.3
0.65
36.5
51.6
0.71
39.4
61.1
0.65
44.2
69.9
0.63
54.4
73.0
0.74
45.9
74.5
0.62
25.8
64.9
0.40
Table 9: Multi-Pass SA-Co Benchmark Results. Comparison of Falcon Perception across different baseline detection passes against SAM 3.
Figure 10: Impact of initialization: Random weight initialization results in lower performance, compared to distillation initialization, even after being trained long enough for 230K steps over pretraining and decay (stages 1 and 2) combined.
Model
Average
ArXiv Math
Base
Hdr/Ftr
TinyTxt
MultCol
OldScan
OldMath
Tables
Mistral OCR 3 ( Mistral AI, 2025a )
81.7
85.4
99.9
93.8
88.9
82.1
48.8
68.3
86.1
Chandra ( Datalab, 2025 )
82.0
81.4
99.8
88.8
91.9
82.9
49.2
73.6
88.2
Gemini 3 Pro ( Google DeepMind, 2026 )
80.2
70.6
99.8
84.0
90.3
79.2
47.5
84.9
84.9
PaddleOCR VL 1.5 ( Cui et al., 2026 )
79.3
85.4
98.8
96.9
80.8
82.6
39.2
66.4
84.1
PaddleOCR VL ( Cui et al., 2025a )
79.2
85.4
98.6
96.9
80.8
82.5
38.8
66.4
83.9
DeepSeek OCR v2 ( Wei et al., 2026 )
78.8
81.9
99.8
95.6
88.7
83.6
33.7
68.8
78.1
Table 10: olmOCR Benchmark Results. Category-wise performance comparison of FalconOCR against state-of-the-art OCR models. We report accuracy (%) across all category splits.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1 : Impact of architectural choices: Width scaling at fixed depth ( L=20 ).
Figure A.2 : Impact of architectural choices: Iso-compute depth–width trade-off.
Figure A.3 : PCA maps before and after upsampling: Image features before upsampling already distinguish multiple objects, e.g . , the coffee cup, notebook , and pen in the fourth row on the left. Similarly, chopsticks along with various food items in second row on the right and the four cups in the bottom right row can be distinguished. The feature upsampler enhances these representations, making instances more visible, suggesting that the pre-upsampling features capture sufficient scene information for instance segmentation and demonstrating the effectiveness of the early-fusion design.
Figure A.4 : Impact of the upsampling factor during inference: Bilinearly upsampling the image features at the perception model output (16x downsampled) and computing the instance masks results in degraded quality (leftmost column) with holes inside the instances and mask regions outside the instances. Processing the model output via the Anyup upsampler with different upsampling factors, followed by mask computation and bilinear upsampling of mask to image size significantly improves the mask quality (columns 2 to 6). Particularly, as the Anyup upsampling factor is increased, the instance boundaries are better refined. E.g . , in the top row, the boundaries of the fourth apple instance from left improves as the factor increases to 8 and 16. Similarly, in the third row, the beak of left penguin is incomplete for factors 1 to 4, while it is accurate for higher factors.
Figure A.5 : Level 0: While our Falcon Perception predicts correct masks, SAM 3 incorrectly predicts cupcakes in the masks for the cupcakes cream example on top row and an additional jar in the middle (middle row).
Figure A.6 : Level 1: Our Falcon perception misses few marker pens in the top row, clearly distinguishes the plant pots on the window sill from the other plant pots in the middle row. In contrast, SAM 3 identifies all the marker pens correctly but fails to distinguish between plant pots in the middle row by predicting false positives for pots not placed on window sill.
Figure A.7 : Level 2: Different to SAM 3 that incorrectly predicts masks for each bottle (top row) and each sign (middle row), our Falcon Perception model correctly identifies the 168 wine bottles (marked with 168) and the Honolulu direction sign , indicating that our model can better relate the query prompt to the OCR text in the image.
Figure A.8 : Level 3: In contrast to SAM 3 predicting false positives for meat (top row), black car (middle row) and Belgian flag (bottom row), our model correctly identifies the lower meat skewer on left grill , black car to the right of red car at bottom and Belgian flag on the left in the three examples. This shows that our Falcon Perception has better spatial understanding of the scene.
Figure A.9 : Level 4: SAM 3 wrongly outputs masks for all bread (top row), persons (middle and bottom rows). However, our Falcon Perception model correctly identifies the object instances for pastry next to brown round bread , person holding phone and person holding helmet in hand , thereby showcasing the ability to understand the object interaction in the scene.
Figure A.10 : Dense Split: While SAM 3 is capable of identifying many instances of an object, it fails to predict masks when number of instances exceeds 200 due to the fixed number of query tokens in the decoder. Our Falcon Perception, following the autoregressive approach for prediction, is not limited by this and can easily scale higher, as shown for snow goose , pigeon and colorful canned drinks in the three rows.
Figure A.11 : Effect of sampling on the quality of predictions: In the left column, we show the deterministic predictions, selecting the most likely next token, center coordinate, or size value. On the right, we show one of the predictions we obtain by sampling tokens, center, and size with a temperature of 0.7 . Oftentimes, we can achieve better predictions for instances that require reasoning over the query, fine-grained perception, and small or distant objects.
Figure A.12 : Example OCR output for inputs comprising formulae, tables and handwriting (from top to bottom). Best viewed zoomed-in.
Figure A.13 : FalconOCR outputs for samples with arxiv papers, scientific formulae. Best viewed zoomed-in.
Figure A.14 : Example outputs for inputs with variations in structure of equations. Best viewed zoomed-in.
Figure A.15 : Example outputs for inputs with variations in structure of tables - sparse and dense with multi-column headers. Best viewed zoomed-in.
Autoregressive perception models trained to localize visual entities under the open-vocabulary setting are mostly trained using Supervised fine-tuning (SFT) with maximum likelihood, yet it optimizes a proxy objective (per-token cross-entropy) that is fundamentally misaligned with perception metrics such as precision and recall. In this paper, we explore post-training reinforcement learning (RL), specifically GRPO, to directly align these models with their evaluation metrics. Building up on the recently introduced Falcon Perception, we design an RL framework that addresses perception-specific challenges: reward design for set-structured outputs and multi-head sampling control. We discover multiple benefits from RL for perception: first, RL unlocks state-of-the-art performance in very dense scenes (up to 500 objects per scene), a regime where most existing systems degrade sharply or collapse; furthermore it fixes common issues in autoregressive perception models like mask repetitions and removes almost entirely the need for NMS and coordinate deduplication, which improve both performance and efficiency and remove the need for hyperparameters tuning; overall, we notice improvements on all levels of difficulties in referring expression segmentation (on PBench and SACO-Gold), and we find an elegant way to preserve the knowledge of whether an object exists or not (as evaluated by MCC) without training on negative samples. We show that a simple reward that penalizes false negatives and positives is sufficient. We develop two hybrid self-annotation pipelines, respectively tailored for difficult referring expressions and very dense scenes, and show their benefits on RL-training. Model weights are released as a Falcon Perception revision~\footnote{https://huggingface.co/tiiuae/Falcon-Perception}. Datasets will be published.
Vision-Language Models (VLMs) have achieved impressive results on general vision-language tasks, yet they suffer from hallucination, imprecise localization, and prohibitive computational cost when applied to dedicated OCR scenarios. This paper presents PP-OCRv6, a lightweight OCR system that combines architectural innovation with data-centric optimization. PP-OCRv6 redesigns the backbone, detection neck, and recognition neck around a unified MetaFormer-style building block with structural reparameterization, decoupling spatial token mixing from channel mixing and supporting both tasks through task-specific stride configurations. Three model tiers (medium, small, tiny) share the same block primitives, covering deployment scenarios from server to edge. On our in-house benchmarks, PP-OCRv6_medium achieves 83.2% recognition accuracy and 86.2% detection Hmean, outperforming PP-OCRv5_server by +5.1% and +4.6% respectively while surpassing Qwen3-VL-235B, GPT-5.5, and Gemini-3.1-Pro with orders of magnitude fewer parameters. The tiny tier achieves 3.9× faster inference than PP-OCRv5_mobile on Intel Xeon CPU while maintaining comparable accuracy.
Vision-language models (VLMs) can read text from images, but where does this optical character recognition (OCR) information enter the language processing stream? We investigate the OCR routing mechanism across three architecture families (Qwen3-VL, Phi-4, InternVL3.5) using causal interventions. By computing activation differences between original images and text-inpainted versions, we identify architecture-specific OCR bottlenecks whose dominant location depends on the vision-language integration strategy: DeepStack models (Qwen) show peak sensitivity at mid-depth (about 50%) for scene text, while single-stage projection models (Phi-4, InternVL) peak at early layers (6-25%), though the exact layer of maximum effect varies across datasets. The OCR signal is remarkably low-dimensional: PC1 captures up to 72.9% of variance. Crucially, principal component analysis (PCA) directions learned on one dataset transfer to others, demonstrating shared text-processing pathways. Surprisingly, in models with modular OCR circuits (notably Qwen3-VL-4B), OCR removal can improve counting performance (up to +6.9 percentage points), suggesting OCR interferes with other visual processing in sufficiently modular architectures.