CueRator: Agentic Search for Symbolic Rules to Adapt Frozen Multimodal Encoders

Sunchan Park; Beomkwon Cho; Kyeongbo Kong

cs.CVOct 6, 2026

CueRator: Agentic Search for Symbolic Rules to Adapt Frozen Multimodal Encoders

Authors: Sunchan Park, Beomkwon Cho, Kyeongbo Kong

Organizations: Pusan National University

Abstract

Large language model agents have been used to search over symbolic structures such as programs and equations. We propose CueRator, an agentic framework for policy-aware decision-rule discovery, which adapts frozen contrastive multimodal encoders by searching for the decision rule that converts their cross-modal similarities into predictions. We validate it on open-vocabulary audio-visual event perception, where existing methods involve a trade-off between adaptivity and generalization to unseen categories: trained modules adapt at the cost of generalization, and fixed rules the reverse. The framework pairs a symbolic formulation for generalization with a lightweight policy that predicts its parameters per video for adaptivity. A report-guided multi-agent loop discovers the formulation offline, evaluating each candidate on its expressive ceiling and on whether a trained policy can realize it. On OV-AVEBench, CueRator raises the total average from 57.8 to 60.2 and unseen-category performance from 55.8 to 59.9 over the best existing method, reducing the seen-unseen gap from 7.1 to 1.2. Ablations attribute the gains to both the formulation and the policy and show that both feedback signals are necessary for effective search. CueRator also improves over the respective baselines on two further audio-visual event perception tasks, and the discovered rule remains competitive across encoders with only the policy retrained. Code is available at https://github.com/cvsp-lab/cuerator.

Figures & tables

Figure 1: Overview of CueRator’s report-guided agentic search. At each iteration, the Plan Agent sets the search direction from accumulated reports, the Design Agent proposes an executable formulation verified against formal constraints, and the Oracle and Policy Agents evaluate the candidate on how well it can perform and whether its parameters can be learned. The Oracle Agent estimates the expressive ceiling through per-video parameter search and ablations, while a policy network is trained for the candidate and diagnosed by the Policy Agent. All reports are appended to memory for the next iteration; no LLM agent is used at test time.

Figure 2: Overview of the OV-AVEL pipeline.

Training-free
	Seen				Unseen				Total
Method	Acc.	Seg.	Eve.	Avg.	Acc.	Seg.	Eve.	Avg.	Acc.	Seg.	Eve.	Avg.
Video-LLaMA2	50.1	40.6	32.0	40.9	48.5	38.5	29.0	38.6	48.9	39.1	29.8	39.3
CLIP&CLAP	51.4	41.4	31.9	41.6	51.6	42.2	31.6	41.8	51.5	41.9	31.7	41.7
ImageBind-TF	57.5	45.0	34.0	45.5	59.8	47.3	34.0	47.0	59.2	46.7	34.0	46.6
Fine-tuning

Table 1: Performance comparison on OV-AVEBench. Baseline scores are from ( Zhou et al., 2025a ) except AV 2 A, which we adapt to ImageBind. Fine-tuning rows are OV-AVEBench reimplementations of ( Zhou et al., 2025a ) , which share the same frozen ImageBind representations and differ only in the trainable module on top. AV 2 A hyperparameters are tuned on a 200-video validation subset following its original protocol (Appendix B ). AV 2 A-Policy retains the AV 2 A rule but replaces its global parameters with CueRator’s per-video policy network (Appendix B ). CueRator is the validation-selected candidate from three search sessions (mean

\pm

std over sessions in Table 2 ).

Variant	Oracle	Test
Variant	Acc.	Acc.	Avg.
CueRator (full)	88.7 $\pm$ 0.9	68.3 $\pm$ 0.8	59.4 $\pm$ 0.9
w/o policy agent	91.5 $\pm$ 2.3	66.7 $\pm$ 0.9	56.3 $\pm$ 1.7
w/o oracle agent	87.3 $\pm$ 5.0	66.0 $\pm$ 0.2	55.3 $\pm$ 0.5
score-only loop	88.4 $\pm$ 1.3	64.1 $\pm$ 1.1	53.6 $\pm$ 0.7
no feedback	86.0 $\pm$ 3.5	62.2 $\pm$ 2.2	51.8 $\pm$ 2.2

Table 2: Ablation study. Mean

\pm

std. over three sessions. Oracle : accuracy with per-video oracle parameters on the training set. Test : test-set Acc. and Avg. For global threshold, Oracle uses a single global

\boldsymbol{\theta}

. The validation-selected candidate attains 60.2 (Table 1 ).

Figure 3: Best score trajectory. Running-best Val Acc. and Train Oracle Acc. over 20 iterations for the validation-selected session of each variant; Oracle Acc. is measured on training data. With oracle–policy feedback, CueRator attains the highest Val Acc. despite a lower oracle ceiling.

Benchmark	Baseline	Metric	Seen:Unseen	Performance
Benchmark	Baseline	Metric	Seen:Unseen	Baseline	CueRator
OV-DAVEL	Open-DAVTR	mAP	75:25	22.9	32.0
OV-DAVEL	Open-DAVTR	mAP	50:50	19.4	32.7
OV-AVVP	AV 2 A	Seg. Type@AV	17:8	52.4	55.6

Table 3: Cross-task generalization. Baseline scores are as published. For each task, CueRator re-runs the formulation search with the same frozen encoders as the corresponding baseline; search and policy training use seen categories only. Full protocols are in Appendix C .

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Time (h)	Share
Agent (LLM inference)	4.0	41%
Oracle search	1.8	18%
Policy training & evaluation	4.0	41%
Total	9.8	100%

Appendix

Table 4: Per-session wall-clock breakdown of the agentic search (

20

iterations), averaged over

3

sessions on a single RTX 3090.

Term	Meaning
Formulation $F$	A closed-form rule computing per-segment thresholds from the audio, visual, and text embeddings (via their similarities), with $P$ tunable parameters $\boldsymbol{\theta}$ .
Oracle ceiling	The performance of $F$ when $\boldsymbol{\theta}$ is set optimally for each video—an upper bound on what the rule’s structure can express.
Policy network $\pi_{\phi}$	A small network predicting $\boldsymbol{\theta}$ from the audio and visual embeddings, called a policy because it is trained with REINFORCE (thresholding is non-differentiable).
Learnability	Whether the optimal $\boldsymbol{\theta}$ is predictable from the video—measured as the gap between the oracle ceiling and the policy’s performance.
Reports / Memory / Directives	The agents’ written analyses ( reports ) accumulate across iterations ( memory ); the Plan Agent reads them and issues instructions for the next iteration ( directives ).

Appendix

Table 5: Glossary of CueRator search terminology.

Agent	Receives	Performs	Writes to memory
Plan	Prior candidate summaries, per-candidate records (formulations, parameter ranges, oracle scores, policy validation scores, agent reports).	Diagnoses the search state; decides whether to refine, abandon, or redirect formulation families; assigns parameter roles, range changes, ablation targets, and analysis requests.	Design directive, oracle ablation protocol, policy analysis protocol, and one-line planning summary.
Design	Plan directive, formulation interface, symbolic-form constraints, and prior formulation code patterns.	Proposes an executable formulation $F_{i}(\cdot;\theta)$ ; declares parameter roles and ranges $\Theta_{i}$ ; revises the code until AST checks pass.	Candidate code, parameter ranges, parameter role descriptions, validity status, and design summary.
Oracle	Verified formulation $F_{i}$ , parameter ranges $\Theta_{i}$ , training split, and requested ablations.	Searches per-video oracle parameters; estimates expressive capacity; repeats per-video oracle search for each ablation; analyzes parameter distributions, boundary saturation, and component impact.	Oracle score $\mathcal{O}(F_{i})$ , ablation statistics, parameter diagnostics, raw-analysis artifacts, and oracle report.
Policy	Verified formulation $F_{i}$ , parameter ranges $\Theta_{i}$ , trained policy outputs, validation metrics, and requested diagnostics.	Analyzes the policy $\pi_{\phi_{i}}$ trained for the candidate: policy-instantiated behavior, predicted parameter distributions, and failure modes.	Validation score $\mathcal{V}(F_{i},\pi_{\phi_{i}})$ , predicted-parameter statistics, learnability diagnostics, and policy report.

Appendix

Table 6: Input–operation–output view of one CueRator search iteration. The LLM agents are used only during offline formulation discovery; test-time prediction uses the selected formulation and policy network.

Figure 4: Condensed input prompt for the Plan Agent.

Figure 5: Condensed Strategy directive produced by the Plan Agent for the Design Agent.

Figure 6: Oracle ablation protocol produced by the Plan Agent.

Figure 7: Policy analysis protocol produced by the Plan Agent.

Figure 8: Condensed input prompt for the Design Agent.

Figure 9: Condensed formulation code produced by the Design Agent.

Figure 10: Condensed input prompt for the Oracle Agent.

Figure 11: Condensed output report of the Oracle Agent.

Figure 12: Condensed input prompt for the Policy Agent.

Figure 13: Condensed output report of the Policy Agent.

Parameter	Multiplied term	Range	Code index
$\theta_{0}$	audio bias	$[-0.1,\,0.5]$	params[0]
$\theta_{1}$	$\mathcal{G}(\mathbf{s}^{v})_{tc}$	$[0.0,\,2.0]$	params[2]
$\theta_{2}$	$\mathcal{G}(\mathbf{b}^{a})_{tc}$	$[0.0,\,2.0]$	params[1]
$\theta_{3}$	$\mathcal{G}(\bar{\mathbf{b}}^{a})_{c}$	$[0.0,\,1.8]$	params[3]
$\theta_{4}$	$\beta_{tc}^{a}$	$[0.0,\,1.8]$	params[4]
$\theta_{5}$	visual bias	$[0.0,\,0.8]$	params[5]

Appendix

Table 7: Parameter ranges for the discovered formulation. Indices are renumbered relative to the code in Fig. 9 so that audio and visual terms align (

\theta_{k}\leftrightarrow\theta_{k+5}

); the code index gives the position in params .

Cue group	Terms	Interpretation
Cross-modal support	$\theta_{1}\mathcal{G}(\mathbf{s}^{v})_{tc}$ , $\theta_{6}\mathcal{G}(\mathbf{s}^{a})_{tc}$	Each modality’s threshold uses the other modality’s category ranking at the same segment.
Local AV agreement	$\theta_{2}\mathcal{G}(\mathbf{b}^{a})_{tc}$ , $\theta_{7}\mathcal{G}(\mathbf{b}^{v})_{tc}$	$\mathbf{b}^{m}=\tfrac{1}{2}(\mathbf{s}^{m}+\min(\mathbf{s}^{a},\mathbf{s}^{v}))$ favors categories supported by both modalities.
Clip-level competition	$\theta_{3}\mathcal{G}(\bar{\mathbf{b}}^{a})_{c}$ , $\theta_{8}\mathcal{G}(\bar{\mathbf{b}}^{v})_{c}$	Compares the category against its competitors over the full video.
Temporal calibration	$\theta_{4}\beta^{a}_{tc}$ , $\theta_{9}\beta^{v}_{tc}$	Compares the current segment with the clip-level evidence for the same category.

Appendix

Table 8: Cue groups in the discovered formulation

F^{\star}

Encoder	AV 2 A	Transferred $F^{\star}$
LanguageBind	50.6	58.6
CLIP+CLAP	48.9	54.5

Appendix

Table 9: Cross-encoder transfer on OV-AVEBench (Total Avg.). AV 2 A uses its published per-encoder hyperparameters; the transferred

F^{\star}

keeps the ImageBind-discovered formulation and retrains only the policy.

Benchmark	Metric	Baseline	Task-specific	Transferred $F^{\star}$
OV-DAVEL (50:50)	mAP	19.4	32.7	30.3
OV-AVVP (17:8)	Seg. Type@AV	52.4	55.6	49.5

Appendix

Table 10: Cross-task transfer: task-specific discovery versus transferring the fixed OV-AVEBench formulation

F^{\star}

with the policy retrained and the task-specific auxiliary parameters re-fitted.

Backbone	Access	Test Avg.
GPT-5.6 Sol	Closed	59.5 $\pm$ 0.9
GPT-5.4	Closed	59.4 $\pm$ 0.9
Claude Opus 4.8	Closed	57.4 $\pm$ 1.1
Qwen3.7-Max	Closed	57.1 $\pm$ 1.8
GLM-5.2	Open-weight	56.7 $\pm$ 1.0

Appendix

Table 11: Sensitivity to the LLM backbone (Test Avg., mean

\pm

std. over three sessions).

Setting	Oracle Acc.	Val Acc.	Test Avg.
CueRator (default)	88.7 $\pm$ 0.9	67.8 $\pm$ 0.2	59.4 $\pm$ 0.9
w/o linear form	88.4 $\pm$ 0.3	65.2 $\pm$ 0.6	55.1 $\pm$ 0.8
w/o constant set	85.6 $\pm$ 4.1	66.4 $\pm$ 0.8	56.6 $\pm$ 1.7
w/o statement budget	86.6 $\pm$ 5.6	66.6 $\pm$ 0.5	57.0 $\pm$ 0.2

Appendix

Table 12: Sensitivity to the Design Agent’s formal constraints (mean

\pm

std. over three sessions).

$P$	Terms per modality (incl. bias)	Search time (h/session)	Test Avg.
6	3	8.2 $\pm$ 0.2	54.6 $\pm$ 0.3
8	4	9.0 $\pm$ 0.2	58.3 $\pm$ 1.1
10	5	9.8 $\pm$ 0.2	59.4 $\pm$ 0.9
12	6	10.0 $\pm$ 0.1	57.2 $\pm$ 1.0
14	7	10.6 $\pm$ 0.4	55.3 $\pm$ 1.9

Appendix

Table 13: Sensitivity to the number of formulation parameters

P

(mean

\pm

std. over three sessions).

Reward	Acc.	Seg.	Eve.	Avg.
Acc. (default)	68.6	59.4	52.7	60.2
Metric average	67.2	59.2	53.0	59.8

Appendix

Table 14: Effect of the policy reward on the selected formulation

F^{\star}

(OV-AVEBench test, Total).

Figure 14: Per-video adaptive thresholds across diverse event types. Each panel shows audio (A) and visual (V) signals, their similarities to the target category (A-T, V-T), and CueRator’s per-segment dynamic thresholds (red dashed). Green regions denote similarity > threshold; Pred and GT bars below give predicted and ground-truth labels.

Figure 15: Additional qualitative results (set 1). (a) Vacuum Cleaner : both modalities fade after the event; thresholds rise to mark the post-event background. (b) Basketball Bounce : visual stays consistent while audio builds up from silence; the audio threshold lowers accordingly, capturing the onset. (c) Race Car : the visual switches to the target as audio fades in mid-clip; both thresholds drop, marking the event window. (d) Dog Barking : a brief silent gap within the event is detected as background as the audio threshold rises above A-T there.

Figure 16: Additional qualitative results (set 2). (a) Playing Bassoon : strong audio at start fades out together with the visual; thresholds rise jointly afterward. (b) Baby Laughter : audio onset is delayed; thresholds drop only when both modalities support the event. (c) Chicken : a brief late audio pulse aligns with a target-consistent visual; the threshold isolates the short event window precisely. (d) Dog Barking : multiple disjoint events with a brief background gap; thresholds track audio pulses while the persistent visual stays above its threshold.

Figure 17: Additional qualitative results (set 3). (a) Airplane Flyby : a full-clip event with gradual audio buildup; both thresholds stay below A-T and V-T throughout, sustaining continuous detection. (b) Playing Mandolin : a stable full-clip event with clean cues; both similarities remain above their thresholds throughout, with no spurious flips. (c) Background : a brief V-T spike does not trigger a false positive; A-T stays below its threshold, so the AV-agreement rule rejects the segment. (d) Background : both A-T and V-T stay below their thresholds throughout, keeping the entire clip correctly classified as background.

Figure 18: Failure cases. (a) Female Speech : a brief background gap within the speech is missed; the audio threshold tracks the dipping A-T instead of rising, so neither modality flips. (b) Vacuum Cleaner : the event window is detected with delayed onset and early offset; both thresholds drop only once the audio is fully established. (c) Chicken Crowing : a mid-clip segment is briefly misclassified as a semantically close category ( Turkey ); the encoder’s similarity favors the wrong class within that window. (d) Dog Barking : a few boundary segments around the event are missed; the audio threshold tracks A-T closely and crosses above it at minor dips.

Explore similar work

Oct 1, 2026cs.CV

OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning

We present OmniSeek, an agentic framework that transforms an Omni Large Language Model (Omni-LLM) into an active, multi-turn reasoning agent with native tool use. Rather than passively processing an entire audio-visual sequence in a single forward pass, OmniSeek makes evidence acquisition part of the reasoning process: it dynamically decides whether to look or listen, and over which temporal window, to retrieve sparse but critical evidence across different modalities within long contexts. Through an iterative multi-turn protocol, the retrieved raw audio or visual segments are appended back into the context to support subsequent reasoning. To cold-start this capability, we build a data engine that synthesizes OmniTraj-170K, a corpus of multi-hop Chain-of-Thought trajectories with interleaved audio and visual evidence. We first supervise the model on these trajectories to instill multi-turn tool-use behavior, and then further optimize the policy via a two-stage reinforcement learning with verifiable rewards. Moreover, we introduce an Audio-Visual Necessity objective that explicitly rewards successful trajectories whose reasoning depends on both modalities, discouraging single-modality shortcuts. Extensive experiments across a wide range of benchmarks demonstrate that OmniSeek learns adaptive cross-modal evidence seeking and consistently improves audio-visual reasoning performance.

Haibo Wang, Jiteng Mu, Jialu Li +5

University of California, Davis · Adobe Research

Audio-Visual Reasoning Omni-Modal Omnichem Chain-of-Thought Reasoning

Ask PDF

May 28, 2026cs.CV

AgentCVR: Active Multi-Agent Cross-Video Reasoning via Script-Simulated Reinforcement Learning

Cross-Video Reasoning (CVR) has emerged as a critical frontier in multimodal intelligence, requiring models to retrieve, align, and aggregate evidence distributed across multiple videos. Current Multimodal Large Language Models (MLLMs) often struggle with CVR, as simple single-pass strategies encode multiple videos into a shared compressed context, potentially obscuring rare but critical evidence. In this paper, we propose AgentCVR, a multi-agent framework that treats CVR as an active evidence-acquisition task. AgentCVR employs a Master Agent to iteratively coordinate specialized Visual and Audio Agents for targeted evidence extraction. To ensure efficient training, we introduce Script-Simulated RL, which optimizes the agent's policy with LLM-generated semantic scripts and a lightweight text-based simulator, bypassing costly multimodal inference during online exploration. Experimental results on a comprehensive CVR benchmark show that AgentCVR outperforms single-pass baselines and achieves comparable performance to state-of-the-art closed-source systems, particularly in complex cross-video alignment and localization. To ensure reproducibility, our code is available at https://github.com/wang-jh24/AgentCVR.

Yilun Qiu, Jiahe Wang, Cilin Yan +4

Xiaohongshu Inc. · Tsinghua Shenzhen International Graduate School, Tsinghua University

Multimodal Agents Simulation-Based Reinforcement Learning

Ask PDF

Oct 4, 2026cs.CV

A Strong Baseline for Evaluating Vision Encoders in Multimodal Large Language Models

Evaluating vision encoders requires metrics that reliably predict their downstream performance in multimodal large language models (MLLMs). Although recent studies have shown that cross-modal metrics can better capture such performance, unimodal metrics remain the dominant choice in practice. In this work, we revisit cross-modal evaluation of vision encoders through large-scale experiments. We identify important limitations in both the experimental design and methodological formulation of prior approaches. After addressing these limitations and introducing simple improvements, we propose RAVEL, a training-free method based on cross-modal nearest-neighbor retrieval. Despite its simplicity, RAVEL achieves state-of-the-art performance across our experiments, outperforming prior methods by a substantial margin. Our results demonstrate that simple cross-modal metrics, when evaluated under a careful and comprehensive setup, can provide a strong basis for evaluating vision encoders for MLLMs.

Yilin Yang, Jun-Tao Tang, Kengyi Wang +3

School of Artificial Intelligence, Shanghai Jiao Tong University · Nanjing University · Fudan University +1

Vision Encoders

Ask PDF