Stress Testing

Video-Language Models (VidLMs) achieve strong benchmark scores, yet these scores often hide whether models use the video at all. We show that VidLM failures follow two pathways: some visual signals are never reliably encoded, while others are encoded but overridden by model priors. We introduce REVEAL, a diagnostic stress-test benchmark for quantifying when and why VidLMs under-use visual evidence. REVEAL contains five controlled probes: camera-motion sensitivity, cross-frame integration, video sycophancy, language-only shortcuts, and temporal expectation bias. Together, they test whether models encode basic video signals, combine evidence across frames, and preserve visual evidence against user assertions, language cues, and learned event expectations. Across 12 VidLMs we find systematic failures along both pathways, with most models falling below chance on the binary and six-way probes that humans solve at 78--100% accuracy. Under assertive prompts, a model's output distribution becomes nearly invariant to whether it is shown a real video or random noise, making visual evidence effectively causally inert. We further carry out mechanistic probes to identify where these failures arise in the model pipeline and why visual evidence is lost. REVEAL provides a scalable, human-verified framework for moving beyond aggregate scores toward structured, reproducible evaluation of multimodal reliability.

Echocardiography interpretation requires integrating multi-view temporal evidence with quantitative measurements and guideline-grounded reasoning, yet existing foundation-model pipelines largely solve isolated subtasks and fail when tool outputs are noisy or values fall near clinical cutoffs. We propose Echo-CoPilot, an end-to-end agentic framework that combines a multi-perspective workflow with knowledge-graph-guided measurement selection. Echo-CoPilot runs three independent ReAct-style agents, structural, pathological, and quantitative, that invoke specialized echocardiography tools to extract parameters while querying EchoKG to determine which measurements are required for the clinical question and which should be avoided. A self-contrast language model then compares the evidence-grounded perspectives, generates a discrepancy checklist, and re-queries EchoKG to apply the appropriate guideline thresholds and resolve conflicts, reducing hallucinated measurement selection and borderline flip-flops. On MIMICEchoQA, Echo-CoPilot provides higher accuracy compared to SOTA baselines and, under a stochasticity stress test, achieves higher reliability through more consistent conclusions and fewer answer changes across repeated runs. Our code is publicly available at https://github.com/moeinheidari7829/Echo-CoPilot.

Horizontal federated learning (HFL) backdoor audits often summarize model behavior through clean accuracy (CA), mean attack success rate (ASR), or a single known-trigger test. Such summaries can hide a different failure mode, in which one target label is activated by many trigger realizations. We study this failure mode with FLAT, a latent-conditioned reliability stress test for HFL backdoors. In FLAT, compromised clients still submit ordinary classifier updates to the server, while an attacker-side generator $G(x,t,z)$ separates target intent $t$ from trigger realization $z$. This separation shifts the audit question from whether one known trigger succeeds to how the hidden behavior varies across targets, latent samples, defenses, and post-stop rounds. On CIFAR-10, CIFAR-100, and Tiny-ImageNet, FLAT preserves clean utility while reaching 99.49%, 99.66%, and 94.10% single-target FedAvg ASR. The evaluation also reveals non-uniform defense responses, where a server rule can suppress one target mode while leaving another active. These observations motivate HFL backdoor audits that report target-wise ASR, worst-target ASR, target coverage, latent-sampled behavior, post-stop persistence, and defense response.

Multi-agent LLM systems that generate structured workflows from natural-language requests are now deployed in production across cloud automation, DevOps, and enterprise orchestration. Operating them exposes a recurring change-management problem. Routine updates, such as re-running an input, swapping the LLM, or refactoring an agent's prompt or orchestration code, often produce workflows that differ substantially from validated references. Engineers then lack a principled way to decide whether a change is safe to ship. Automatic workflow evaluation is the natural tool, but in practice metric scores are poorly calibrated, and a numeric change rarely communicates the severity of the underlying degradation. We introduce WorkflowPerturb, a controlled benchmark that applies realistic, graded perturbations to golden workflows. It contains 4,973 golden workflows and 44,757 perturbed variants across three perturbation types (Missing Steps, Compressed Steps, Description Changes) at severities of 10%, 30%, and 50%. We benchmark multiple metric families, analyzing their sensitivity and calibration using expected score trajectories and instance-level alert rates. Our results characterize systematic differences across families and support severity-aware interpretation of workflow evaluation scores in change-management settings. WorkflowPerturb is publicly available at https://huggingface.co/datasets/microsoft/WorkflowPerturb .

Generated operational programs are often validated with either a few hand-written examples or exhaustive regression suites. The former can miss sparse boundary and interaction faults, while the latter can be unnecessarily expensive. We introduce FaultLens, a method for learning compact behavioral test suites while preserving an auditable connection to executed evidence. It executes a rich probe domain once, stores the fault-probe kill relation as a sparse outcome cache, and learns probe orderings only from earlier program generations. A fault-driven greedy component exploits known kill structure, while a mutation-independent diversity component covers probe families, cases, templates, and temporal bins. Their alternating hybrid remains useful when a new program contains a fault mechanism absent from ordering construction. We evaluate twenty generated operational policies across four environments, ten execution seeds, 1,200 measured run summaries, 2,160 controlled program transformations, and 4,120,200 executed program-probe pairs. Of 1,960 intended faulty transformations, 1,779 alter a contract or output somewhere in the finite audit domain; 200 additional controls preserve behavior. A 32-probe hybrid learned on generations 1-3 covers 576/582 (99.0%) dynamically killable faults in generations 4-5 using 1.2-2.0% of the exhaustive domain. With an entire fault family withheld from training, diversity raises scenario-family macro coverage from 84.6% to 94.9%. In a downstream deployment study, a conservative admission rule reduces severe tail regressions from 15/20 program-environment groups to 0/20. FaultLens provides a prioritized evidence mechanism, not a proof of correctness, and makes its budget, evidence source, generalization split, and misses explicit.