Real-Time Localisation of Dorsal Anatomical Regions on Dairy Cows in Commercial Rotary Parlours from RGB-D

Vineet Prasad; Shiron Melvin; Sheikh Azid; Hai Wang

cs.CVOct 23, 2026

Real-Time Localisation of Dorsal Anatomical Regions on Dairy Cows in Commercial Rotary Parlours from RGB-D

Authors: Vineet Prasad, Shiron Melvin, Sheikh Azid, Hai Wang

Organizations: School of Engineering and Energy, Murdoch University, Perth, Australia

Abstract

Automated monitoring of dairy cows increasingly measures one part of the body, such as the rump for temperature or the back for posture, yet the region itself is usually placed by hand. Existing systems either score the cow as a whole or detect sparse keypoints, often in controlled walkways, and neither yields the named region masks that such measurements need on a crowded commercial rotary platform. We present a perception front end that finds the target cow, ten bony landmarks and four named dorsal regions (withers, back, rump and tail-head) in a single forward pass over a whole RGB-D frame from a commercial rotary parlour. Two encoders, one for colour and one for depth, are fused late and share one decoder with three heads. On an anatomical region dataset that we built on public rotary-parlour RGB-D images (538 annotated frames, cow-disjoint test set), the network reaches PCK@10 0.905 for landmarks and region IoU 0.70 to 0.91, and runs end to end in 26.4 ms per frame on a T4 GPU, against an 11 fps camera. Dense region supervision is what makes the landmarks accurate: removing the region head costs 0.15 PCK@10. Agreement between the two heads flags unreliable frames without labels. On our own rig at the largest dairy farm in Western Australia, the model transfers with ten labelled frames (PCK@10 0.84), and its regions are carried through depth into a radiometric thermal camera, giving one repeatable temperature per region and cow.

Figures & tables

Figure 1: Landmarks K1 to K10 and the four rule regions on a test frame: withers (blue), back (orange), rump (green) and tail-head (yellow). The target cow is in the centre stall, between two neighbours.

Figure 2: The proposed framework. A whole RGB-D frame from the parlour camera passes once through two encoders whose feature pyramids are summed (late fusion) and decoded to stride 4. Three heads give the target-cow mask, ten landmarks with visibility and four named region masks. The agreement between the region head and the regions built from the predicted landmarks serves as a label-free quality flag. The regions are lifted to 3D with the depth behind each pixel and projected into any co-calibrated sensor, here a radiometric thermal camera. (a) to (c) show a test frame from the dataset; (d) shows the temperatures inside each projected region on one farm-rig frame, coloured on the thermal scale, with the 95th percentile that serves as the region’s temperature (pixels below 25 ∘ C, rails and floor at the region edges, are not drawn).

			Median	IoU
Model	PCK@5	PCK@10	error (px)	Withers	Back	Rump	Tail-head	Cow
RGB only	0.341 $\pm$ 0.012	0.669 $\pm$ 0.013	25.7	0.65	0.85	0.87	0.72	0.94
Depth only	0.362 $\pm$ 0.003	0.701 $\pm$ 0.018	24.4	0.65	0.88	0.89	0.73	0.95
RGB-D, early fusion	0.338 $\pm$ 0.022	0.687 $\pm$ 0.021	25.6	0.67	0.87	0.88	0.73	0.96
RGB-D, late fusion (full model)	0.421 $\pm$ 0.014	0.753 $\pm$ 0.007	21.4	0.68	0.88	0.90	0.75	0.96
without consistency loss	0.415 $\pm$ 0.015	0.745 $\pm$ 0.007	21.9	0.68	0.88	0.90	0.75	0.96

Table 1: Inputs, fusion and heads (ResNet-18 backbone; mean

\pm

SD over three seeds; 69 cow-disjoint test frames unless stated). Region IoU is against the rule regions built from the labels; for the landmarks-only model, regions are built by rule from its predicted landmarks.

Figure 3: Four test cows from the dataset, of different coat colour, size and body condition score (BCS 3 to 4.75), from three sessions (ConvNeXt-T). Top: input colour frames with each cow’s BCS. Bottom: predicted regions and landmarks with the frame’s median landmark error. (d) is a cow standing at an angle in the stall.

	Params			Median	IoU	IoU	Network	End to end
Backbone	(M)	PCK@5	PCK@10	error (px)	back / rump	cow	(ms)	(ms, fps)
MobileNetV3-L	6.7	0.427 $\pm$ 0.002	0.750 $\pm$ 0.007	21.5	0.89 / 0.90	0.97	17.9	22.4 (45)
ResNet-18	23.1	0.421 $\pm$ 0.014	0.753 $\pm$ 0.007	21.4	0.88 / 0.90	0.96	10.7	13.8 (73)
PVTv2-b1	27.7	0.518 $\pm$ 0.004	0.843 $\pm$ 0.003	17.4	0.90 / 0.91	0.97	25.7	28.6 (35)
ConvNeXt-T, shared recipe	56.4	0.458 $\pm$ 0.262	0.700 $\pm$ 0.330	33.1	0.88 / 0.90	0.97	23.5	26.4 (38)
ConvNeXt-T, own recipe	56.4	0.645 $\pm$ 0.012	0.905 $\pm$ 0.008	13.3	0.91 / 0.91	0.98	23.5	26.4 (38)

Table 2: Backbones (RGB-D late fusion, all heads; three seeds; 69 test frames). Times are per frame on a T4 GPU at batch size 1 in FP16, median of three sessions; end to end includes upload, GPU pre-processing and decoding. The camera delivers a frame every 90 ms.

Figure 4: Label-free quality flag (ResNet-18, 69 test frames). Frames whose two heads disagree are the frames with large landmark errors; the shaded band is the lowest 20% of agreement.

Figure 5: Per-landmark PCK@10 (test set: mean of three seeds; farm rig: 4-fold cross-validation). On the test set the rear of the withers (K2) is the hardest landmark for both backbones; on the farm rig’s steeper view the same ConvNeXt-T backbone finds both withers landmarks more reliably.

Figure 6: Two farm passes held out from the model shown (ConvNeXt-T). (a) The model trained on the dataset, unchanged. (b) After fine-tuning on 39 farm frames; red crosses are the labels. (c) The fine-tuned regions lifted to 3D with the RGB-D depth and projected into the thermal frame of the same moment. (d) Temperatures inside each projected region, coloured on the thermal scale, with the 95th percentile marked. In these passes the tail-head lies outside the thermal camera’s narrower field of view; a strictly overhead mount with matched fields of view will close this gap.

		Median	IoU
Setting	PCK@10	error (px)	Back	Rump
Zero-shot (no farm labels)	0.545	16.7	0.55	0.41
Fine-tuned, 10 frames	0.839	9.1	0.86	0.86
Fine-tuned, 20 frames	0.905	8.0	0.87	0.88
Fine-tuned, 39 frames	0.913	7.6	0.88	0.89
ImageNet start, 39 frames	0.545	16.9	0.76	0.82

Table 3: Transfer to the farm rig (ConvNeXt-T; 52 labelled frames, 4-fold cross-validation by pass; errors in farm pixels). Every row except the last starts from the model trained on the dataset.

Figure 7: Region temperatures on the farm rig. (a) One 95th-percentile value per cow and region (box: quartiles; line: median). (b) The same value frame by frame through one pass; shaded frames are used.

Anatomical Landmark Detection

See all 18 papers

CowCare: A Unified Visual Framework for Viewpoint-Robust Cattle Tracking and Spatiotemporal Heat-Stress Analysis
Oct 10, 2026Xin Liu, Weiran Li, Minghao Zhu +2Precision Livestock Farming Agricultural Image Analysis
College of Information and Electrical Engineering, China Agricultural University, Beijing, 100083, China · National Innovation Center for Digital Fishery, Ministry of Agriculture and Rural Affairs, Beijing, 100083, China · Key Laboratory of Smart Farming Technologies for Aquatic Animal and Livestock, Ministry of Agriculture and Rural Affairs, Beijing, 100083, China +1
Deep Learning Approach for RealTime Contraction Detection in Laboring Dairy Cattle
Oct 9, 2026Sebastian Puchala, Dominik Gronkiewicz, Marcin Gołębiewski +2Agricultural Image Analysis Video Action Recognition
Promity, Warsaw, Poland · Institute of Animal Science, Warsaw University of Life Sciences, Warsaw, Poland · Faculty of Electronics and Information Technology, Warsaw University of Technology, Warsaw, Poland
FSMC-Pose: Frequency and Spatial Fusion with Multiscale Self-calibration for Cattle Mounting Pose Estimation
Mar 17, 2026Fangjing Li, Zhihai Wang, Xinxin Ding +5Object Pose Estimation Agricultural Image Analysis
Beijing Jiaotong University · NERCITA · Tsinghua University +1

Oct 10, 2026J/K move · Enter open · S save

CowCare: A Unified Visual Framework for Viewpoint-Robust Cattle Tracking and Spatiotemporal Heat-Stress Analysis

Xin Liu, Weiran Li, Minghao Zhu, Mina Han+1

College of Information and Electrical Engineering, China Agricultural University, Beijing, 100083, China · National Innovation Center for Digital Fishery, Ministry of Agriculture and Rural Affairs, Beijing, 100083, China · Key Laboratory of Smart Farming Technologies for Aquatic Animal and Livestock, Ministry of Agriculture and Rural Affairs, Beijing, 100083, China+1

Precision Livestock Farming Agricultural Image Analysis Multi-Object Tracking Video Understanding

Open paper AskPDF arXiv

Reliable individual monitoring is essential for dairy cattle welfare and early heat stress detection in Precision Livestock Farming (PLF). However, translating raw multi-camera videos into automated diagnostics remains challenging due to disconnected tracking and recognition pipelines and complex physical environments. Dense mutual occlusions and cross-camera viewpoint variations fragment identities, while environmental noise, including camera vibrations, and limited agricultural data obscure subtle heat stress cues such as rapid breathing and flank undulations. To bridge low-level visual features and high-level behavioral diagnostics, we propose CowCare, a unified framework combining viewpoint-robust tracking with spatiotemporal heat stress recognition. Its tracking module, CowTrack, maintains cross-camera identity consistency through a View-Decoupled Aggregation Network (VDA-Net). Instead of relying on static global representations, VDA-Net aggregates localized textures under heavy occlusion and uses a Viewpoint Decoupling Transformer (VDT) to suppress transient viewpoint variations. CowTrack further incorporates a causal online self-supervision mechanism for continuous adaptation to dynamic farm scenes. The resulting tracklets are subsequently processed by TSDF-Net, a fine-grained video understanding network. Its Temporal Stabilization and Aggregation Module (TSAM) suppresses high-frequency jitter, while a Diffusion Semantic Prior (DSP) uses frozen pre-trained diffusion semantics to enhance subtle behavioral signals and mitigate overfitting. Extensive evaluations on two real-world farm benchmarks demonstrate the framework's effectiveness: CowTrack achieves 83.48% HOTA and 87.72% IDF1 for global multi-object tracking on MVCow, and TSDF-Net achieves 87.72% Top-1 accuracy on HSCow. These results establish CowCare as a robust end-to-end framework for automated livestock health monitoring.

Abstract

Anatomical Landmark Detection

Explore similar work

CowCare: A Unified Visual Framework for Viewpoint-Robust Cattle Tracking and Spatiotemporal Heat-Stress Analysis

Deep Learning Approach for RealTime Contraction Detection in Laboring Dairy Cattle

FSMC-Pose: Frequency and Spatial Fusion with Multiscale Self-calibration for Cattle Mounting Pose Estimation