Pointing-Based Object Recognition

Lukáš Hajdúch; Viktor Kocur

cs.CVMar 16, 2026

Pointing-Based Object Recognition

Authors: Lukáš Hajdúch, Viktor Kocur

Organizations: Comenius University, Bratislava, Slovakia

Abstract

This paper presents a comprehensive pipeline for recognizing objects targeted by human pointing gestures using RGB images. As human-robot interaction moves toward more intuitive interfaces, the ability to identify targets of non-verbal communication becomes crucial. Our proposed system integrates several existing state-of-the-art methods, including object detection, body pose estimation, monocular depth estimation, and vision-language models. We evaluate the impact of 3D spatial information reconstructed from a single image and the utility of image captioning models in correcting classification errors. Experimental results on a custom dataset show that incorporating depth information significantly improves target identification, especially in complex scenes with overlapping objects. The modularity of the approach allows for deployment in environments where specialized depth sensors are unavailable.

Figures & tables

Fig. 1: The full pipeline of the proposed system. The input image (a) is fed into three parallel branches: pose estimation (b), object detection (c) and depth estimation (d). The outputs of the three branches get combined to first classify whether pointing occurs and which detected object is pointed at (e). Finally, an image captioning module (f) is used to provide a better description of the object and to potentially fix an incorrect object label assignment.

Panel of Figure 2 — Fig. 2: Sample images from our dataset. We use different types of gestures to indicate pointing at objects and neutral poses (no pointing). The dataset captures different levels of complexity in terms of item positions.

Case	MoGe	Features	Precision	Recall	F1	Accuracy
Easy	$\checkmark$	kpOnly	0.53	0.86	0.66	0.77
Easy	$\checkmark$	full	0.56	0.91	0.69	0.80
Hard	$\checkmark$	kpOnly	0.48	0.83	0.61	0.71
Hard	$\checkmark$	full	0.47	0.80	0.59	0.70
Easy	$\times$	kpOnly	0.50	0.70	0.59	0.67
Easy	$\times$	full	0.56	0.67	0.61	0.72

TABLE I: The classification accuracy results for determining whether pointing occurs in an image (see Sec. III-E ). The results shown are for combination with YOLO11s [ 2 ] with (

\checkmark

) and without (

\times

) MoGe depth estimation [ 5 ] .

Easy Cases
Model	MoGe	Precision	Recall	F1	Accuracy
YOLOv8s	$\times$	0.42	0.49	0.45	0.63
YOLOv8s	$\checkmark$	0.49	0.67	0.56	0.74
YOLOv8n	$\times$	0.54	0.44	0.49	0.70
YOLOv8n	$\checkmark$	0.51	0.62	0.56	0.74
YOLO11n	$\times$	0.46	0.39	0.43	0.65

TABLE II: Comparison of object recognition performance across various detection models with (

\checkmark

) and without (

\times

) MoGe depth estimation [ 5 ] . Results are categorized into Easy Cases (horizontal separation) and Hard Cases (depth-wise overlap).

Case	Captioning	Precision	Recall	F1	Accuracy
Easy	$\times$	0.56	0.91	0.69	0.80
	BLIP	0.58	0.92	0.71	0.81
	ViT-GPT2	0.56	0.91	0.69	0.80
	GIT	0.58	0.92	0.71	0.81
Hard	$\times$	0.47	0.80	0.59	0.70
	BLIP	0.51	0.81	0.62	0.71

TABLE III: Pointed-at object recognition results for YOLO11s [ 2 ] object detector combined with different image captioning methods used for correcting badly classified objects (see Sec. III-F ).

Human-Robot Interaction

Human-robot interaction studies how people and robots communicate, coordinate, and safely work or live together.

See all 452 papers

PointVG-R: Internalizing Geometric Reasoning in MLLMs for Precise Pointing Localization via Visual Chain of Thought
Jun 23, 2026Ling Li, Bowen Liu, Zinuo Zhan +5Visual Spatial Reasoning Vision-Language Grounding
Tsinghua University Beijing Shi, China · Dalian University of Technology Dalian Shi, China · Northwestern Polytechnical University Xi’an Shi, China +1
VistaRef: Boosting Visual Spatial Orientation Awareness for Pointing-to-Object Detection
Jun 23, 2026Ling Li, Zhizhen Cai, Xinkun Wu +4Visual Spatial Reasoning Detection Transformer
Tsinghua University Beijing Shi, China · Beijing University of Posts and Telecommunications Beijing Shi, China · Tsinghua Shenzhen International Graduate School Shenzhen, Guangdong, China +1
GIVE: Grounding Human Gestures in Vision-Language-Action Models
Jun 11, 2026Pengfei Liu, Gen Li, Junqiao Fan +4Language-Conditioned Robot Manipulation Goal-Conditioned Robotic Manipulation
1MARS Lab, Nanyang Technological University

Jun 23, 2026J/K move · Enter open · S save

PointVG-R: Internalizing Geometric Reasoning in MLLMs for Precise Pointing Localization via Visual Chain of Thought

Ling Li, Bowen Liu, Zinuo Zhan, Jianhui Zhong+4

Tsinghua University Beijing Shi, China · Dalian University of Technology Dalian Shi, China · Northwestern Polytechnical University Xi’an Shi, China+1

Visual Spatial Reasoning Vision-Language Grounding Visually Grounded Reasoning Visual Grounding Visual CoT

Open paper AskPDF arXiv

Pointing-based visual grounding requires models to precisely locate target objects by deciphering complex spatial relationships between the visual scene and pointing gestures. Traditional methods typically encode input images into static feature representations and perform reasoning primarily within the linguistic domain, often overlooking the rich perceptual cues and explicit spatial geometry inherent in images. In this study, we aim to mitigate the cognitive vulnerability of models in interpreting gestural spatial relations by proposing PointVG-R, a reasoning-guided Multi-modal Large Language Model (MLLM). PointVG-R introduces geometric-aware reasoning for pointing-based grounding, enabling the model to think with images through the strategic integration of Reinforcement Learning (RL) and cold-start data. Specifically, we design a novel geometric reasoning pipeline that simulates the iterative cognitive process humans employ when interpreting pointing gestures. Furthermore, we construct EgoPoint-CoT, a high-quality visual Chain-of-Thought (CoT) dataset featuring detailed reasoning trajectories to guide the model via Supervised Fine-Tuning (SFT) and RL. To address the varying quality of learning signals encountered during training, we further propose an Adaptive Importance Weighting strategy based on Group Variance, which dynamically adjusts reward signals to optimize the learning process. Experimental results demonstrate that PointVG-R achieves SOTA performance, outperforming the baseline by

\textbf{15.86}

points in mIoU. Extensive ablation studies further validate the efficacy of our proposed modules. Code: https://github.com/lingli1724/PointVG-R.

Abstract

Human-Robot Interaction

Explore similar work

PointVG-R: Internalizing Geometric Reasoning in MLLMs for Precise Pointing Localization via Visual Chain of Thought

VistaRef: Boosting Visual Spatial Orientation Awareness for Pointing-to-Object Detection

GIVE: Grounding Human Gestures in Vision-Language-Action Models