Asking a Multimodal LLM Whether Someone Fell: Zero-Shot Screening of Low-Resolution Thermal Video in a Laboratory and an Occupied Home

Marcos Junca; Mauro Junca

cs.CVOct 21, 2026

Asking a Multimodal LLM Whether Someone Fell: Zero-Shot Screening of Low-Resolution Thermal Video in a Laboratory and an Occupied Home

Authors: Marcos Junca, Mauro Junca

Organizations: Senecta Inc.

Abstract

Thermal cameras can watch over older people living alone without capturing faces, and multimodal large language models (LLMs) can judge a short video without task-specific training. We evaluate a production multimodal LLM, Gemini 3.8 Flash at temperature 0, as the decision stage of an event-triggered fall screener, used exactly as a deployed prototype uses it: one request per short thermal clip, with its sound track when there is one, and its predecessor Gemini 3.7 Flash on the same requests. On 438 staged falls and 563 non-fall windows that we located by hand in a public 32x24 thermal dataset, the deployed request called 69% of the held-out subjects' falls (67% over all subjects; bed falls excluded) and none of 499 non-fall windows, including people lying on the floor and getting up. Over 76 h of ordinary life in an occupied room, recorded at 160x120 with sound, it called none of 395 clips a fall (95% upper bound 1.2 false alarms per day) and caught 4 of the 9 staged falls the camera could see. One cautionary sentence in the prompt ("a person who goes out of view ... is not a fall") accounted for many of the misses: removing it raised sensitivity on the held-out subjects from 69% to 83% on 3.8 Flash and from 80% to 92% on Gemini 3.7 Flash, and in the home from 4 to 9 of the 9 visible falls, with no false call in ordinary living but at the price of calling deliberate lie-downs a fall. Sensitivity collapsed at 8x6 pixels (34%); identical requests disagreed on 4% of windows; and when the model called a fall, its explanation placed it within a second of the annotated landing 99% of the time. Two conventional detectors trained on the laboratory data caught nearly every held-out laboratory fall but raised 14.2 and 23.9 false alarms per day in the home. We release the frame-level fall annotations of the public dataset and every model answer on it.

Figures & tables

Fig. 1 : The prototype. A thermal camera and a microphone stream continuously; a wake gate combines sound and thermal movement; each wake becomes one short clip (3 s before, 15 s after, extended by further wakes); the clip is rendered as a video with its sound track and reviewed once by a multimodal LLM, which answers with a fixed JSON form. This paper evaluates the last two boxes.

Fig. 2 : The home recording, 6–13 September 2026: clips the deployed gate and pipeline sent for review, per hour, with the staged sessions (staged falls, lie-downs and tests of the response) and the hours when the bench unit was not recording.

		Gemini 3.8 Flash		Gemini 3.7 Flash
Condition	Change from the deployed request	lab	home	lab	home
base	none: v8 prompt, inferno, $480\times 360$ , sound, low thinking	1,001	443	1,001	121
no out-of-view	v8 without its third sentence	360	121	1,001	121
v5	previous prompt and answer form	360	121	360	121
repeat	identical request sent again	1,001	121	360	–
res $32\times 24$	frames block-averaged to $32\times 24$ (home)	–	–	–	121

TABLE I : Conditions, each changing one factor of the deployed request, and what each was run on: laboratory windows (all 1,001, or the 360 of the two held-out subjects) and home clips (the 395 ordinary-living clips, or the fixed sample of 73 of them, together with the 48 staged clips; or the staged clips alone). “–”: not run, or not finished when the paper was built. Gemini 3.5 Flash was run with the base request on 360 laboratory windows.

Fig. 3 : Laboratory falls called a fall, by fall type (all six subjects): Gemini 3.8 Flash and 3.7 Flash with the deployed prompt (v8), and 3.7 Flash with v8 without its out-of-view sentence. Bed falls are shown but excluded from the other laboratory numbers.

Situation	Answer	Reason given by the model
Laboratory fall, called (backward)	fall, floor	“An upright person standing on the right side of the frame begins to collapse around t=2.6s, falling down to the floor where they remain lying horizontally in view through the end of the clip.”
Laboratory fall, missed (backward when trying to sit down)	no fall, none	“An upright individual performs a controlled squat or crouch down to floor level around t=2.5s to t=3.5s, remains briefly in a low position, and then stands back up by the end of the clip without an uncontrolled collapse or fall.”
Laboratory fall, missed (backward)	no fall, out_of_view	“The thermal footage shows a person standing in the center of the frame who descends downwards out of view below the bottom edge between 00:02.375 and 00:03.250. Because the subject moves out of view and does not remain clearly visible collapsing onto the floor within the frame, this cannot be confirmed as a fall.”
Home, visible fall, called	fall, floor	“An upright individual walks across the room, loses balance around 22.8 seconds, and falls directly onto the floor with an audible impact at approximately 24.0 seconds, remaining on the ground.”
Home, visible fall, missed	no fall, out_of_view	“The person drops below the camera frame and goes out of view, which according to the guidelines does not qualify as an observable fall.”
Home, occluded fall	no fall, out_of_view	“The individual loses balance around 7.5 seconds and drops behind furniture out of view, which does not qualify as an observed fall onto the floor per guidelines.”

TABLE II : The model’s answers, verbatim (deployed request; the laboratory answers’ opening “Audio is unavailable” is omitted). Chosen to illustrate each situation, not sampled.

Fig. 4 : One staged fall (a slow backward faint) from the public laboratory data [ 6 ] (CC BY 4.0), in the colour map the model saw, seconds from the landing, at the sensor’s $32\times 24$ and block-averaged to $16\times 12$ and $8\times 6$ . The performer is back on their feet within six seconds.

Fig. 5 : Effect of sensor resolution (block averaging, same field of view, same display size). (a) Laboratory falls and non-falls called a fall, held-out subjects, Wilson 95% intervals. (b) Home: visible staged falls called (of 9) and false calls on sampled ordinary-living clips.

Fig. 6 : Each screener as one point: laboratory falls called (held-out subjects, Wilson 95% interval) against false fall calls per 24 h of ordinary living in the home (exact Poisson 95% interval). The conventional detectors were built on the laboratory data and run at

32\times 24

	Gemini 3.8 Flash (deployed)			Gemini 3.7 Flash
Condition	held-out	all	non-falls	held-out	all	non-falls
v8, deployed	69	67	0/499	80	77	0/499
v8 without the out-of-view sentence	83	–	0/182	92	88	4/499
v5 (previous prompt)	79	–	2/182	81	–	0/182
same request, sent again	72	68	0/499	76	–	1/182
resolution $16\times 12$	48	–	0/182	61	–	0/182

TABLE III : Laboratory windows (eHomeSeniors,

32\times 24

, no sound), 14 fall types (bed falls apart, Section VI-A ). Falls called (%) for the two held-out subjects (

n=140

) and for all six subjects (

n=414

; “–” where a condition was run on the held-out subjects only), and non-fall windows called a fall (upright, recovery and rest windows together; all subjects, or held-out where only those were run).

	Gemini 3.8 Flash (deployed)			Gemini 3.7 Flash
Condition	false calls	visible	occluded	false calls	visible	occluded
v8, deployed	0/395	4/9	0/15	0/73	6/9	2/15
v8 without the out-of-view sentence	0/73	9/9	13/15	1/73	9/9	11/15
v5 (previous prompt)	0/73	5/9	10/15	0/73	6/9	11/15
v8, sound removed	–	2/9	0/15	–	5/9	0/15
v5, sound removed	–	6/9	5/15	–	8/9	8/15

TABLE IV : The occupied home (

160\times 120

with sound). False calls: ordinary-living clips called a fall, all of them false alarms, out of the clips reviewed (every clip for the deployed request on Gemini 3.8 Flash; a fixed 20% sample otherwise; “–” for the conditions run on the staged clips only). Visible: staged falls the camera saw, called. Occluded: staged falls whose landing the camera did not see, called. Clips that began with a person already on the floor, and deliberate lie-downs, are reported in the text.

	Laboratory (held-out), %		Home, clips called a fall
Screener	falls	non-falls	visible falls	ordinary living	occluded falls
3.8 Flash, v8 (deployed)	69	0.0	4/9	0/395	0/15
3.8 Flash, no out-of-view	83	0.0	9/9	0/73	13/15
3.7 Flash, v8	80	0.0	6/9	0/73	2/15
3.7 Flash, no out-of-view	92	1.1	9/9	1/73	11/15
descent rule	94	1.1	7/9	76/395	13/15

TABLE V : The zero-shot model and two conventional detectors. Laboratory: held-out subjects, 14 fall types, falls and non-falls called (%). Home: visible staged falls called, ordinary-living clips called a fall (all false alarms), and occluded staged falls called. The detectors run at

32\times 24

and were built on the laboratory data.

Data	Model	$n$	agree	$\kappa$	flips
laboratory	3.8 Flash	1,001	95.9	0.90	41
laboratory	3.7 Flash	360	95.8	0.91	15
home	3.8 Flash	121	97.5	0.56	3

TABLE VI : Test–retest at temperature 0: the deployed request sent twice to the same model. Agreement (%), Cohen’s

\kappa

and the number of windows or clips whose verdict flipped.

Fall Detection

Fall detection identifies falls and impact events from video, body-worn sensors, radar, and other signals to support timely assistance.

See all 20 papers

Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMs
May 13, 2026Trung Nguyen Quang, Yiming Gao, Fanyi Pu +3Multimodal Grounding Multimodal Large Language Models
Nanyang Technological University · LMMs-Lab Team · Johns Hopkins University
Right Answer, Wrong Heat: Explanation-Aware Evaluation and Thermal-Grounded Feedback for MLLMs on Infrared Images
Aug 10, 2026Yongsong Huang, Xiaofeng Liu, Tomo Miyazaki +2Vision-Language Models Vision-Language Grounding
Tohoku University · Yale University
OmniFall: From Staged Through Synthetic to Wild, A Unified Multi-Domain Dataset for Robust Fall Detection
May 26, 2025David Schneider, Zdravko Marinov, Moritz Mistol +6Synthetic-to-Real Domain Adaptation Fall Detection
Karlsruhe Institute of Technology · Mercedes-Benz Tech Innovation

May 13, 2026J/K move · Enter open · S save

Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMs

Trung Nguyen Quang, Yiming Gao, Fanyi Pu, Kaichen Zhang+2

Nanyang Technological University · LMMs-Lab Team · Johns Hopkins University

Multimodal Grounding Multimodal Large Language Models Multimodal Reasoning Multimodal Foundation Models Inference-Time Intervention Multimodal Understanding

Open paper AskPDF arXiv

When an omnimodal large language model accepts a question whose textual premise contradicts what it actually sees or hears, does the failure lie in perception or in action? Recent omnimodal models are positioned as perception-grounded agents that jointly process video, audio, and text, yet a basic form of grounding remains untested: catching a textual claim that conflicts with the model's own sensory input. We introduce IMAVB, a curated 500-clip benchmark of long-form movies with a 2x2 design crossing target modality (vision, audio) and premise condition (standard, misleading), which lets us measure conflict detection separately from ordinary multimodal comprehension. Across eight open-source omnimodal LLMs and Gemini 3.1 Pro, we document a Representation-Action Gap: hidden states reliably encode premise-perception mismatches even when the same models almost never reject the false claim in their outputs. Behaviorally, models fall into two failure modes: under-rejection, in which they answer misleading questions as if the false premise were true; and over-rejection, in which they reject more often but also reject standard questions, sacrificing ordinary comprehension accuracy. The gap is modality-asymmetric (audio grounding underperforms vision) and prompt-resistant across seven variants. As an initial diagnostic intervention, a probe-guided logit adjustment (PGLA) re-injects the encoded mismatch signal into decoding and consistently improves rejection behavior. Together, these results suggest the bottleneck for omnimodal grounding lies in translation, not perception.

Abstract

Fall Detection

Explore similar work

Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMs

Right Answer, Wrong Heat: Explanation-Aware Evaluation and Thermal-Grounded Feedback for MLLMs on Infrared Images

OmniFall: From Staged Through Synthetic to Wild, A Unified Multi-Domain Dataset for Robust Fall Detection