Feedback

Follow this topic Ask about this topic

Momentum

70 papers in the last four weeks, up 250% on the four weeks before. 0.7% of all new papers.

Jul 13Week of Sep 28

Latest papers 341

Apr 28, 2026cs.CL

LongSumEval: Question-Answering Based Evaluation and Feedback-Driven Refinement for Long Document Summarization

Evaluating long document summaries remains the primary bottleneck in summarization research. Existing metrics correlate weakly with human judgments and produce aggregate scores without explaining deficiencies or guiding improvement, preventing effective refinement in applications requiring verifiable accuracy. We introduce LongSumEval, a unified framework bridging evaluation and generation through structured question-answering feedback. The framework operationalizes summary quality as answerability and factual alignment of question-answer pairs, generating interpretable scores and actionable feedback that identifies coverage gaps and factual inconsistencies. This resolves the misalignment where evaluation operates independently of generation objectives. Meta-evaluation of our QA-based evaluation module across seven benchmarks demonstrates substantially stronger agreement with human judgments compared to established metrics. Structured feedback enables significant quality improvements through self-refinement without retraining. By demonstrating that evaluation feedback can serve as executable instructions for generation, this work establishes a generalizable paradigm for aligning assessment with improvement, with direct implications for controllable text generation requiring verifiable accuracy and transparent quality control. All code and datasets will be released in GitHub for reproducibility.

Huyen Nguyen, Haoxuan Zhang, Yang Zhang +2

Cigna Group Evernorth Health Services Austin, Texas, USA · dept. of Information Science University of North Texas Denton, Texas, USA · dept. of Data Science University of North Texas Denton, Texas, USA

Summarization Long-Form Generation Question-Answer Pairs Feedback Text Generation

Apr 27, 2026cs.CV

Interactive Episodic Memory with User Feedback

In episodic memory with natural language queries (EM-NLQ), a user may ask a question (e.g., "Where did I place the mug?") that requires searching a long egocentric video, captured from the user's perspective, to find the moment that answers it. However, queries can be ambiguous or incomplete, leading to incorrect responses. Current methods ignore this key aspect and address EM-NLQ in a one-shot setup, limiting their applicability in real-world scenarios. In this work, we address this gap and introduce the Episodic Memory with Questions and Feedback task (EM-QnF). Here, the user can provide feedback on the model's initial prediction or add more information (e.g., "Before this. I'm looking for the big blue mug not the white one"), helping the model refine its predictions interactively. To this end, we collect datasets for feedback-based interaction and propose a lightweight training scheme that avoids expensive sequential optimization. We also introduce a plug-and-play Feedback ALignment Module (FALM) that enables existing EM-NLQ models to incorporate user feedback effectively. Our approach significantly improves over the state of the art on three challenging benchmarks and is better than or competitive with commercial large vision-language models while remaining efficient. Evaluation with human-generated feedback shows that it generalizes well to real-world scenarios.

Nikesh Subedi, Loris Bazzani, Ziad Al-Halah

University of Utah · University of Verona

Episodic Memory Feedback Recent Vision-Language Models Egocentric Video

Apr 26, 2026cs.CL

DRACULA: Hunting for the Actions Users Want Deep Research Agents to Execute

Scientific Deep Research (DR) agents answer user queries by synthesizing research papers into multi-section reports. User feedback can improve their utility, but existing protocols only score the final report, making it hard to study and learn which intermediate actions DR agents should take to improve reports. We collect DRACULA, the first dataset with user feedback on intermediate actions for DR. Over five weeks, nineteen expert CS researchers ask queries to a DR system that proposes actions (e.g., "Add a section on datasets"). Our users select actions they prefer, then judge whether an output report applied their selections successfully, yielding 8,103 action preferences and 5,230 execution judgments. After confirming a DR agent can execute DRACULA's actions, we study the predictability of user-preferred actions via simulation-how well LLMs predict the actions users select-a step toward learning to generate useful actions. We discover: (1) LLM judges initially struggle to predict action selections, but improve most when using a user's full selection history, rather than self-reported or extrapolated user context signals; (2) Users' selections for the same query differ based on unstated goals, bottlenecking simulation and motivating affordances that let users steer reports; and (3) Our simulation results inform an online intervention that generates new actions based on the user's past interactions, which users pick most often in follow-up studies. Overall, while work extensively studies execution, DRACULA reveals a key challenge is deciding which actions to execute in the first place. We open-source DRACULA's study design, user feedback, and simulation tasks to spur future work on action feedback for long-horizon agents.

Nishant Balepur, Malachi Hamada, Varsha Kishore +9

University of Maryland · New York University · Allen Institute for Artificial Intelligence (Ai2)

Action Selection Long-Horizon Agents Feedback Double-Edged Sword Scientific Discovery

Apr 25, 2026cs.AI

CAP-CoT: Cycle Adversarial Prompt for Improving Chain of Thoughts in LLM Reasoning

Chain-of-Thought (CoT) prompting has emerged as a simple and effective way to elicit step-by-step solutions from large language models (LLMs). However, CoT reasoning can be unstable across runs on long, multi-step problems, leading to inconsistent answers for unchanged task. Most prior work focuses on improving the forward reasoning chain within a single pass, with less attention to iterative and contrastive correction. To address this gap, we propose CAP-CoT, a Cycle Adversarial Prompt optimization framework designed to improve both CoT reasoning accuracy and stability of a single deployed solver. In each cycle, a forward solver generates candidate reasoning chains, an adversarial challenger constructs plausible but deliberately flawed chains using targeted error strategies, and a feedback agent contrasts the two chains and produces step-aligned structured feedback. This feedback closes the optimization loop in two directions, including updating the solver prompt based on errors exposed by the challenger, and updating the challenger prompt to generate increasingly targeted errors in subsequent cycles. Unlike safety-oriented adversarial prompting such as jailbreak or prompt-injection attacks, our adversarial component is task-semantic and aims to expose logical vulnerabilities in reasoning chains. Experiments across six benchmarks and four LLM backbones demonstrate that within two to three adversarial prompt optimization cycles, CAP-CoT consistently reduces variability across runs while improving reasoning accuracy and robustness to prompt perturbations.

Shuxu Chen, Yitian Zhou, Jiaquan Zhang +6

Department of Electronics and Information Convergence Engineering, Kyung Hee University, Republic of Korea · University of Electronic Science and Technology of China, Chengdu, China · East China Jiaotong University, Nanchang, China +1

Adversarial Prompts LLM Reasoning Strategies Reasoning Chain Chain-of-Thought Reasoning Thoughts Mitigation+1

Apr 25, 2026stat.ML

MOCA: A Transformer-based Modular Causal Inference Framework with One-way Cross-attention and Cutting Feedback

Causal effect estimation from observational data requires careful adjustment for confounding. Classical estimators such as inverse probability weighting and augmented inverse probability weighting are effective under favorable model specification, but may become unstable when treatment assignment and outcome mechanisms are complex, non-linear, and high-dimensional. Machine learning and representation learning approaches improve flexibility, yet joint training can allow outcome-related information to influence treatment-side representations, which is undesirable from a causal perspective. We propose MOCA (Modular One-way Causal Attention), a transformer-based framework that separates treatment and outcome modeling through a modular design, and performs confounder adjustment using a one-way attention mechanism. A cutting-feedback strategy, implemented via gradient detachment, prevents the outcome loss from updating the treatment module. This design preserves directional information flow while retaining the representational power of transformer architectures for causal inference. Across multiple simulated scenarios, including linear, nonlinear, heavy-tailed, hidden confounding, and high-dimensional settings, MOCA shows competitive or improved performance relative to IPW, AIPW, X-learner, TARNet, and DragonNet. We further illustrate the method on the Infant Health and Development Program dataset and the Dehejia-Wahba dataset as real-world benchmarks. These results suggest that modular attention with one-way information flow provides a promising and interpretable direction for causal inference with modern deep learning models.

Lei Wang, Debashis Ghosh

Department of Biostatistics and Informatics Colorado School of Public Health

Causal Inferences Latent Confounders Causal Attention Cross-Attention Feedback

Apr 24, 2026cs.SE

From Natural Language to Verified Code: Toward AI Assisted Problem-to-Code Generation with Dafny-Based Formal Verification

Large Language Models (LLMs) show promise in automated software engineering, yet their guarantee of correctness is frequently undermined by erroneous or hallucinated code. To enforce model honesty, formal verification requires LLMs to synthesize implementation logic alongside formal specifications that are subsequently proven correct by a mathematical verifier. However, the transition from informal natural language to precise formal specification remains an arduous task. Our work addresses this by providing the NaturalLanguage2VerifiedCode (NL2VC)-60 dataset: a collection of 60 complex algorithmic problems. We evaluate 11 randomly selected problem sets across seven open-weight LLMs using a tiered prompting strategy: contextless prompts, signature prompts providing structural anchors, and self-healing prompts utilizing iterative feedback from the Dafny verifier. To address vacuous verification, where models satisfy verifiers with trivial specifications, we integrate the uDebug platform to ensure functional validation. Our results show that while contextless prompting leads to near-universal failure, structural signatures and iterative self-healing facilitate a dramatic performance turnaround. Specifically, Gemma 4-31B achieved a 90.91% verification success rate, while GPT-OSS 120B rose from zero to 81.82% success with signature-guided feedback. These findings indicate that formal verification is now attainable for open-weight LLMs, which serve as effective apprentices for synthesizing complex annotations and facilitating high-assurance software development.

Md Erfan, Md Kamal Hossain Chowdhury, Ahmed Ryan +1

Department of Computer Science, The University of Alabama, Tuscaloosa, USA · Alabama Water Institute, The University of Alabama, Tuscaloosa, USA

Formal Verification Code Generation Natural Language Feedback

Apr 24, 2026cs.LG

Accelerating Frequency Domain Diffusion Models with Error-Feedback Event-Driven Caching

Diffusion models achieve remarkable success in time series generation. However, slow inference limits their practical deployment. We propose E

^2

-CRF (Error-Feedback Event-Driven Cumulative Residual Feature caching) to accelerate frequency domain diffusion models. Our method exploits two structural properties: (1) spectral localization, where signal energy concentrates in low frequencies, and (2) mirror symmetry, which halves the effective frequency dimension. E

^2

-CRF uses a closed-loop error-feedback system that adaptively caches transformer KV features across diffusion steps. We trigger recomputation using event-driven residual dynamics instead of fixed schedules. Our method selectively recomputes high-energy or rapidly-changing tokens while reusing cached features for stable high-frequency components. E

^2

-CRF achieves ~2.2 speedup while maintaining sample quality. We demonstrate effectiveness on 5 datasets. Our caching strategy naturally aligns with the diffusion process's structure-to-detail progression. We include sufficient-condition error and complexity bounds under standard regularity assumptions (Appendix), alongside empirical validation. Our code is available at https://github.com/NoakLiu/FastFourierDiffusion and is also integrated in https://github.com/NoakLiu/FastCache-xDiT.

Dong Liu, Haisheng Wang, Yanxuan Yu

University of California, Los Angeles · Yale University · East China Normal University +1

Diffusion Models Fast Inference Feedback

Apr 23, 2026cs.SE

Feedback Over Form: Why Execution Feedback Matters More Than Pipeline Topology in 1-3B Code Generation

Small language models (1-3B) are practical to run locally, but individually limited on harder code generation tasks. We ask whether composing them into pipelines can recover some of that lost capability. We study code generation pipelines built from 1-3B models with execution feedback, and use a NEAT-inspired evolutionary search to test whether more complex pipeline structure helps beyond a simple refinement loop. We evaluate on HumanEval (164 problems) and sanitized MBPP (427 problems), all with local inference on a single laptop. Self-refinement with execution feedback improves code generation by more than 4 standard deviations on both benchmarks. The gains are narrow in mechanism: refinement fixes many runtime errors (especially NameError and SyntaxError), but rarely fixes logic errors such as AssertionError. Within our tested general-purpose model pool, generator identity mattered less than refiner capability: a 1.5B generator paired with a 3B refiner matched a 3B model doing both roles. Early stopping is essential; without it, every iteration is net-negative. The code-specialized models outperform every general-purpose pipeline configuration, suggesting model specialization matters more than pipeline architecture. Preliminary text-only pipeline experiments without execution feedback did not show gains at this scale. In our constrained search space, evolutionary search mostly rediscovered the same simple generate-execute-refine loop we found manually, with no clearly significant gain from added topology. Single-evaluation fitness inflates results by 5-7 percent, selecting lucky genomes over good ones. On these benchmarks at 1-3B scale, execution feedback mattered more than added pipeline complexity in determining whether composition helped.

Charles Junichi McAndrews

Code Generation Large Language Model Generation Feedback

Apr 22, 2026cs.CL

Exploiting LLM-as-a-Judge Disposition on Free Text Legal QA via Prompt Optimization

This work explores the role of prompt design and judge selection in LLM-as-a-Judge evaluations of free text legal question answering. We examine whether automatic task prompt optimization improves over human-centered design, whether optimization effectiveness varies by judge feedback style, and whether optimized prompts transfer across judges. We systematically address these questions on the LEXam benchmark by optimizing task prompts using the ProTeGi method with feedback from two judges (Qwen3-32B, DeepSeek-V3) across four task models, and then testing cross-judge transfer. Automatic optimization consistently outperforms the baseline, with lenient judge feedback yielding higher and more consistent gains than strict judge feedback. Prompts optimized with lenient feedback transfer better to strict judges than the reverse direction. Analysis reveals that lenient judges provide permissive feedback, yielding prompts with broader applicability, whereas strict judges produce restrictive feedback, leading to judge-specific overfitting. Our findings demonstrate algorithmically optimizing prompts on training data can outperform human-centered prompt design and that judges' dispositions during optimization shape prompt generalizability.

Mohamed Hesham Elganayni, Runsheng Chen, Sebastian Nagl +1

Technical University of Munich Germany

Llm-As-A-Judge Prompt Optimization Feedback Free Overfitting

Apr 22, 2026cs.CL

WebGen-R1: Incentivizing Large Language Models to Generate Functional and Aesthetic Websites with Reinforcement Learning

While Large Language Models (LLMs) excel at function-level code generation, project-level tasks such as generating functional and visually aesthetic multi-page websites remain highly challenging. Existing works are often limited to single-page static websites, while agentic frameworks typically rely on multi-turn execution with proprietary models, leading to substantial token costs, high latency, and brittle integration. Training a small LLM end-to-end with reinforcement learning (RL) is a promising alternative, yet it faces a critical bottleneck in designing reliable and computationally feasible rewards for website generation. Unlike single-file coding tasks that can be verified by unit tests, website generation requires evaluating inherently subjective aesthetics, cross-page interactions, and functional correctness. To this end, we propose WebGen-R1, an end-to-end RL framework tailored for project-level website generation. We first introduce a scaffold-driven structured generation paradigm that constrains the large open-ended action space and preserves architectural integrity. We then design a novel cascaded multimodal reward that seamlessly couples structural guarantees with execution-grounded functional feedback and vision-based aesthetic supervision. Extensive experiments demonstrate that our WebGen-R1 substantially transforms a 7B base model from generating nearly nonfunctional websites into producing deployable, aesthetically aligned multi-page websites. Remarkably, our WebGen-R1 not only consistently outperforms heavily scaled open-source models (up to 72B), but also rivals the state-of-the-art DeepSeek-R1 (671B) in functional success, while substantially exceeding it in valid rendering and aesthetic alignment. These results position WebGen-R1 as a viable path for scaling small open models from function-level code generation to project-level web application generation.

Juyong Jiang, Chenglin Cai, Chansung Park +4

The Hong Kong University of Science and Technology (Guangzhou) · Alibaba · Tongyi Lab, Alibaba Group +3

Web Large Language Model Generation Aesthetics Feedback

Apr 22, 2026cs.CL

SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks

Skills have become the de facto way to enable LLM agents to perform complex real-world tasks with customized instructions, workflows, and tools, but how to learn them automatically and effectively remains unclear. We introduce SkillLearnBench, the first benchmark for evaluating continual skill learning methods, comprising 20 verified, skill-dependent tasks across 15 sub-domains derived from a real-world skill taxonomy , evaluated at three levels: skill quality, execution trajectory, and task outcome. Using this benchmark, we evaluate recent continual learning techniques, those leveraging one-shot, self/teacher feedback, and skill creator to generate skills from agent experiences. We find that all continual learning methods improve over the no-skill baseline, yet consistent gains remain elusive: no method leads across all tasks and LLMs, and scaling to stronger LLMs does not reliably help. Continual learning improves tasks with clear, reusable workflows but struggles on open-ended tasks, and using stronger LLM backbones does not consistently produce better skills. Our analysis also revealed that multiple iterations in continual learning facilitate genuine improvement via external feedback, whereas self-feedback alone induces recursive drift. Our data and code are open-source at https://github.com/cxcscmu/SkillLearnBench to enable further studies of automatic skill generation and continual learning techniques.

Shanshan Zhong, Yi Lu, Jingjie Ning +7

Carnegie Mellon University · Amazon AGI

Agent Skill Retrieval Skills Large Language Model Agents Continual Learning Feedback Real-World

Apr 20, 2026cs.LG

Online Conformal Prediction with Adversarial Semi-bandit Feedback via Regret Minimization

Uncertainty quantification is crucial in safety-critical systems, where decisions must be made under uncertainty. In particular, we consider the problem of online uncertainty quantification, where data points arrive sequentially. Online conformal prediction is a principled online uncertainty quantification method that dynamically constructs a prediction set at each time step. While existing methods for online conformal prediction provide long-run coverage guarantees without any distributional assumptions, they typically assume a full feedback setting in which the true label is always observed. In this paper, we propose a novel learning method for online conformal prediction with partial feedback from an adaptive adversary-a more challenging setup where the true label is revealed only when it lies inside the constructed prediction set. Specifically, we formulate online conformal prediction as an adversarial bandit problem by treating each candidate prediction set as an arm. Building on an existing algorithm for adversarial bandits, our method achieves a long-run coverage guarantee by explicitly establishing its connection to the regret of the learner. Finally, we empirically demonstrate the effectiveness of our method in both independent and identically distributed (i.i.d.) and non-i.i.d. settings, showing that it successfully controls the miscoverage rate while maintaining a reasonable size of the prediction set.

Junyoung Yang, Kyungmin Kim, Sangdon Park

GSAI POSTECH · GSAI & CSE POSTECH

Online Conformal Prediction Long-Run Coverage Guarantees Learning-Augmented Algorithms Uncertainty Quantification Feedback

Apr 18, 2026cs.LG

Untrained CNNs Match Backpropagation at V1: A Systematic RSA Comparison of Four Learning Rules Against Human fMRI

A central question in computational neuroscience is whether the learning rule used to train a neural network determines how well its internal representations align with those of the human visual cortex. We present a systematic comparison of four learning rules (backpropagation (BP), feedback alignment (FA), predictive coding (PC), and spike-timing-dependent plasticity (STDP)) applied to identical convolutional architectures and evaluated against human fMRI data from the THINGS-fMRI dataset (720 stimuli, 3 subjects) using Representational Similarity Analysis (RSA). All models process stimuli at 224 x 224 resolution; results are averaged across 5 random seeds. Crucially, we include an untrained random-weights baseline that reveals the dominant role of architecture. At V1/V2, the untrained baseline exceeds backpropagation (rho = 0.076 vs. rho = 0.034; Delta-rho = +0.044, p < 0.001), and STDP achieves the highest V1 alignment among trained rules (rho = 0.064). At LOC, only BP reliably exceeds the random baseline (rho = 0.012 vs. -0.005, p < 0.001). At IT, all five conditions converge (rho = 0.008-0.014) with no significant pairwise differences among trained rules (p > 0.05, FDR-corrected). FA consistently produces the lowest alignment at V1, V2, and LOC (rho = 0.012 at V1, below all other conditions). Partial RSA confirms all effects survive pixel-similarity control. Seed variability is small relative to between-rule differences at V1/V2. These results demonstrate that early visual alignment is architecture-driven, learning rules differentiate only at intermediate areas, and all rules converge at the highest levels of the hierarchy.

Nils Leutenegger

Independent Researcher, Switzerland

Human Visual Cortex Representational Similarity Analysis Backpropagation Neural Representations Functional Magnetic Resonance Imaging Convolutional Neural Networks+1

Apr 18, 2026cs.LG

Self-Reinforcing Controllable Synthesis of Rare Relational Data via Bayesian Calibration

Imbalanced data are commonly present in real-world applications. While data synthesis can effectively mitigate data scarcity for rare classes, and LLMs have revolutionized text generation, the application of LLMs to the synthesis of relational/structured tabular data remains underexplored. Moreover, existing approaches lack an effective feedback mechanism to guide LLMs in continuously optimizing the quality of the generated data throughout the synthesis process. In this work, we propose RDDG, Relational Data generator with Dynamic Guidance, which is a unified in-context learning framework that employs progressive chain-of-thought (CoT) steps to generate tabular data for enhancing downstream imbalanced classification performance. RDDG first uses core set selection to identify representative samples from the original data, then utilizes in-context learning to discover the inherent patterns and correlations among attributes within the core set, and subsequently generates tabular data while preserving the aforementioned constraints. More importantly, it incorporates a self-reinforcing feedback mechanism that provides automatic assessments of the quality of the generated data, enabling continuous quality optimization throughout the generation process. Experimental results on multiple real and synthetic datasets demonstrate that RDDG outperforms existing approaches in both data fidelity and downstream imbalanced classification performance. We make our code available at https://github.com/cszhangLMU/RDDG.

Chongsheng Zhang, Hao Wang, Zelong Yu +7

Henan University, China · Department of Statistics, LMU Munich · MCML Munich +1

Synthetic Tabular Data Synthetic Data Imbalanced Classification Relational Databases Synthesis Relational+1

Apr 17, 2026cs.CL

GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflows

The development of general-purpose agents requires a shift from executing simple instructions to completing complex, real-world productivity workflows. However, current tool-use benchmarks remain misaligned with real-world requirements, relying on AI-generated queries, dummy tools, and limited system-level coordination. To address this, we propose GTA-2, a hierarchical benchmark for General Tool Agents (GTA) spanning atomic tool use and open-ended workflows. Built on real-world authenticity, it leverages real user queries, deployed tools, and multimodal contexts. (i) GTA-Atomic, inherited from our prior GTA benchmark, evaluates short-horizon, closed-ended tool-use precision. (ii) GTA-Workflow introduces long-horizon, open-ended tasks for realistic end-to-end completion. To evaluate open-ended deliverables, we propose a recursive checkpoint-based evaluation mechanism that decomposes objectives into verifiable sub-goals, enabling unified evaluation of both model capabilities and agent execution frameworks (i.e., execution harnesses). Experiments reveal a pronounced capability cliff: while frontier models already struggle on atomic tasks (below 50%), they largely fail on workflows, with top models achieving only 14.39% success. Further analysis shows that checkpoint-guided feedback improves performance, while advanced frameworks such as Manus and OpenClaw substantially enhance workflow completion, highlighting the importance of execution harness design beyond the underlying model capacity. These findings provide guidance for developing reliable personal and professional assistants. Dataset and code will be available at https://github.com/open-compass/GTA.

Jize Wang, Xuanxuan Liu, Yining Li +7

1Shanghai Jiao Tong University · 2Shanghai AI Laboratory · 3Tencent +1

Agentic Benchmarks Agentic Workflow Design Feedback

Apr 3, 2026cs.LG

Marginal-Contribution Policy Gradients under Filtered Feedback for Multi-Agent LLMs

We develop a unified treatment of credit assignment for RL training in multi-agent LLM systems. We show that observed reward alone cannot distinguish an agent that determines it from one that never affects it, and that standard shared-reward training performs exact gradient ascent on each agent's private utility rather than system performance. Moreover, we prove no single scalar per agent can consistently account for joint performance once agents interact. We thus develop the unique background-dependent notion of marginal contribution satisfying natural consistency requirements. From it we derive gradient-correct marginal contribution training signals, identify them from filtered feedback, and optimally allocate a budget of exact counterfactual evaluations against learned-signal error. Instantiated in GRPO, our signal improves routed GSM8K accuracy over winner-take-all training at no extra generation cost.

Elai Ben-Gal, Stela Tong

Department of Mathematics Stanford University · Graduate School of Business Stanford University

Credit Assignment Multi-Agent Large Language Model Systems Policy Gradient Feedback Gradient Descent

Mar 29, 2026math.OC

Separation is Optimal for LQR under Intermittent Feedback

We study finite-horizon linear-quadratic regulation of a scalar linear system with intermittent state feedback under an average communication-rate constraint. In this setting, the scheduling policy and controller are generally coupled through the dual effect: transmission decisions shape future estimation errors, while control actions influence the information available for scheduling. Existing treatments often recover tractability by restricting attention to symmetric scheduling policies, but the optimality of this restriction has remained unclear. We show that, for i.i.d. zero-mean disturbances, symmetric policies are optimal. Consequently, the communication-constrained LQR problem admits a separation structure. The optimal controller is a linear feedback law independent of the scheduling policy, while the optimal scheduler is obtained from a dynamic program. We further show that the optimal scheduling rule is a symmetric threshold policy in the accumulated disturbance since the most recent update.

Abdullah Y. Etcibasi, C. Emre Koksal, Eylem Ekici

The Ohio State University

Linear Quadratic Regulator Optimal Policies Schedulers Feedback Separation

Mar 19, 2026cs.CL

Beyond Scalar Rewards: Dense Feedback for LLM Policy Synthesis in Sequential Social Dilemmas

We propose an LLM harness that generates code-based policy functions for multi-agent environments, evaluates them with self-play, and refines them using feedback from previous iterations. Following the recent line of work in feedback engineering (the design of which information signals are shown to the LLM during refinement), we compare sparse feedback (scalar reward only) with dense feedback (reward plus social metrics: efficiency, equality, sustainability, peace). In two Sequential Social Dilemmas (Gathering and Cleanup) and with two frontier LLMs (Claude Sonnet 4.6, Gemini 3.1 Pro), dense feedback improves over or matches sparse feedback on all metrics. We explain this asymmetry via feedback aliasing: when the scalar reward maps distinct failure modes into the same value (e.g., under- vs. over-cleaning), social metrics disambiguate and allow the LLM to diagnose which direction of improvement to take. We conclude that social metrics act as a coordination signal, leading to strategies such as Voronoi territory partitioning and adaptive cleaner schedules. Code at https://github.com/vicgalle/llm-policies-social-dilemmas.

Víctor Gallego

Komorebi AI Technologies, Spain

Large Language Model Policy Optimization Reward Functions Feedback

Feb 24, 2026cs.SE

SpecMind: Cognitively Inspired, Interactive Multi-Turn Framework for Postcondition Inference

Specifications are vital for ensuring program correctness, yet writing them manually remains challenging and time-intensive. Recent large language model (LLM)-based methods have shown successes in generating specifications such as postconditions, but existing single-pass prompting often yields inaccurate results. In this paper, we present SpecMind, a novel framework for postcondition generation that treats LLMs as interactive and exploratory reasoners rather than one-shot generators. SpecMind employs feedback-driven multi-turn prompting approaches, enabling the model to iteratively refine candidate postconditions by incorporating implicit and explicit correctness feedback, while autonomously deciding when to stop. This process fosters deeper code comprehension and improves alignment with true program behavior via exploratory attempts. Our empirical evaluation shows that SpecMind significantly outperforms state-of-the-art approaches in both accuracy and completeness of generated postconditions.

Cuong Chi Le, Minh V. T Pham, Tung Vu Duy +4

The University of Texas at Dallas · FPT Software AI Center · VinUniversity +1

Llm-Driven Code Synthesis Self-Speculative Multi-Turn Interactions Feedback

Feb 13, 2026cs.IR

Self-EvolveRec: Self-Evolving Recommender Systems with LLM-based Directional Feedback

Traditional methods for automating recommender system design, such as Neural Architecture Search (NAS), are often constrained by a fixed search space defined by human priors, limiting innovation to pre-defined operators. While recent LLM-driven code evolution frameworks shift fixed search space target to open-ended program spaces, they primarily rely on scalar metrics (e.g., NDCG, Hit Ratio) that fail to provide qualitative insights into model failures or directional guidance for improvement. To address this, we propose Self-EvolveRec, a novel framework that establishes a directional feedback loop by integrating a User Simulator for qualitative critiques and a Model Diagnosis Tool for quantitative internal verification. Furthermore, we introduce a Diagnosis Tool - Model Co-Evolution strategy to ensure that evaluation criteria dynamically adapt as the recommendation architecture evolves. Extensive experiments demonstrate that Self-EvolveRec significantly outperforms state-of-the-art NAS and LLM-driven code evolution baselines in both recommendation performance and user satisfaction. Our code is available at https://github.com/Sein-Kim/self_evolverec.

Sein Kim, Sangwu Park, Hongseok Kang +6

KAIST · Microsoft

Self-Evolution Neural Architecture Search Feedback

Feb 12, 2026cs.CL

Investigating Learner-Aware Design of LLM-Generated Educational Feedback

Although large language models (LLMs) show promise for generating educational feedback, it remains unclear how feedback should be designed (e.g., tone and coverage) to support answer revision and learner evaluations across learner profiles. We define six feedback designs for multiple-choice biology questions, including a baseline design and five variants with additional feedback elements, and conduct an empirical study with 321 high school students. We evaluate feedback using immediate revision performance and six subjective evaluation criteria, and analyze differences in subjective evaluations across learner profiles based on personality traits. Our results show that presenting task-relevant information clearly is associated with better immediate revision performance and is favorably evaluated across learner profiles, while we observe descriptive differences in evaluation patterns, particularly for informational novelty and affective framing. These findings support further investigation of personalized LLM feedback design.

Momoka Furuhashi, Kouta Nakayama, Noboru Kawai +3

Tohoku University · Research and Development Center for Large Language Models, National Institute of Informatics · Osaka Kyoiku University +2

Feedback Multiple-Choice Questions Answer

Feb 4, 2026cs.LG

Improved Dimension Dependence for Bandit Convex Optimization with Gradient Variations

Gradient-variation online learning has drawn increasing attention due to its deep connections to game theory and optimization. It has been studied extensively in the full-information setting, but is underexplored with bandit feedback. In this work, we focus on gradient variation in Bandit Convex Optimization (BCO) with two-point feedback. By proposing a refined analysis of the non-consecutive gradient variation, a fundamental quantity in gradient variation with bandit feedback, we improve the dimension dependence for both convex and strongly convex functions compared with the best known results (Chiang et al., 2013). Our improved analysis of the non-consecutive gradient variation also implies other favorable problem-dependent guarantees, such as gradient-variance and small-loss regret bounds. Beyond the two-point setup, we demonstrate the versatility of our technique by achieving the first gradient-variation bound for one-point bandit linear optimization over hyper-rectangular domains. Finally, we validate the effectiveness of our results in more challenging tasks such as dynamic and universal regret minimization, establishing the first gradient-variation dynamic and universal regret bounds for two-point BCO.

Hang Yu, Yu-Hu Yan, Peng Zhao

State Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China

Jan 28, 2026cs.AI

Policy of Thoughts: Scaling Test-Time Training for LLM Reasoning via Online Policy Evolution

Large language models (LLMs) struggle with complex, long-horizon reasoning due to instability caused by their frozen policy assumption. Current test-time scaling methods treat execution feedback merely as an external signal for filtering or rewriting trajectories, without internalizing it to improve the underlying reasoning strategy. Inspired by Popper's epistemology of "conjectures and refutations," we argue that intelligence requires real-time evolution of the model's policy through learning from failed attempts. We introduce Policy of Thoughts (PoT), a framework that recasts reasoning as a within-instance online optimization process. PoT first generates diverse candidate solutions via an efficient exploration mechanism, then uses Group Relative Policy Optimization (GRPO) to update a transient LoRA adapter based on execution feedback. This closed-loop design enables dynamic, instance-specific refinement of the model's reasoning priors. Experiments show that PoT dramatically boosts performance: a 4B model achieves 49.71% accuracy on LiveCodeBench, outperforming GPT-4o and DeepSeek-V3 despite being over 50 smaller.

Zhengbo Jiao, Hongyu Xian, Qinglong Wang +5

Zhejiang University · South China Normal University · Shanghai University of Finance and Economics +2

LLM Reasoning Strategies Large Language Model Policy Optimization Thoughts Test-Time Training Long-Horizon Task Planning Feedback

Dec 11, 2025physics.ed-ph

Developing an LLM-Based Feedback System Grounded in Evidence-Centered Design to Support Physics Problem Solving

Generative AI offers new opportunities for individualized and adaptive learning, e.g., through large language model (LLM)-based feedback systems. While LLMs can produce factually correct feedback for relatively straightforward conceptual tasks, delivering high-quality feedback for tasks that require advanced domain expertise, such as physics problem solving, remains a substantial challenge. This study presents the design and implementation of an LLM-based feedback system for physics problem solving grounded in evidence-centered design and reports a first evaluation within the German Physics Olympiad. Participants rated the usefulness and correctness of the generated feedback for each implemented problem. The collected ratings indicate that the feedback was generally perceived as useful and highly correct. However, an in-depth analysis revealed that the feedback contained errors in 20% of cases; errors that often went unnoticed by the students. We discuss the risks associated with uncritical reliance on LLM-based feedback and outline potential directions for generating more adaptive and reliable LLM-based feedback in the future.

Holger Maus, Fabian Kieser, Stefan Petersen +2

Department of Physics Education, Leibniz Institute for Science and Mathematics Education Kiel, Germany · Department of Physics Education Research, Free University of Berlin Berlin, Germany · Department of Physics and Physics Education Research, Ludwigsburg University of Education Ludwigsburg, Germany

Feedback Feedback Loop Problem Solving Capability Generative Artificial Intelligence German

Dec 1, 2025cs.HC

A Comparative Study of Student Perspectives on Technical Writing Feedback Quality: Evaluating LLMs, SLMs, and Humans in Computer Science Topics

To address the scalability of feedback in computer science while mitigating the privacy and cost limitations of commercial Large Language Models (LLMs), this study evaluates a locally hosted Small Language Model (SLM). We deployed a quantized Llama-3.1, GPT-4, and human instructors across introductory programming (N=176), operating systems (N=80), and a writing seminar (N=7). Mixed-methods analysis of student perceptions reveals that while the local SLM matched commercial LLMs and was rated higher by students for readability and actionability in technical courses, human feedback remained more favoured for highly specialized writing tasks. We demonstrate that local SLMs offer a privacy-preserving, zero-marginal-cost alternative for foundational feedback, supporting a tiered pedagogical framework where AI handles structural guidance while instructors focus on high-level conceptual scaffolding.

Suqing Liu, Runlong Ye, Christopher Eaton +2

McMaster University, Hamilton, Ontario, Canada · Department of Computer Science, University of Toronto, Toronto, Ontario, Canada · Research Institute for the Study of University Pedagogy, University of Toronto Mississauga, Mississauga, Ontario, Canada +1

Computer Science Ai-Assisted Programming Tasks Student Small Large Language Models Feedback High-Level Source Code

Nov 23, 2025cs.CV

EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning

Egocentric video understanding requires procedural reasoning under partial observability and continuously shifting viewpoints. Current multimodal large language models (MLLMs) struggle with this setting, often generating plausible but visually inconsistent or weakly grounded responses. We introduce

\textbf{EgoVITA}

, a framework that decomposes egocentric video reasoning into a structured

\textit{plan-then-verify}

process. The model first generates an

\textbf{egocentric plan}

: a causal sequence of anticipated actions from a first-person perspective. This plan is then evaluated by an

\textbf{exocentric verification}

stage that uses third-person reasoning over the same video to verify its spatiotemporal and logical consistency, without exocentric video input. This decomposition enables cross-perspective feedback without requiring paired ego-exo supervision. To train this reasoning process, we adopt Group Relative Policy Optimization (GRPO) with two dense reward signals: one that grounds anticipated actions in subsequent visual observations and another that reinforces consistent third-person verification.

\textbf{EgoVITA}

achieves state-of-the-art performance on egocentric reasoning benchmarks, outperforming Qwen2.5-VL-7B by

\mathbf{+7.7}

on EgoBlind and

\mathbf{+4.4}

on EgoOrient, while maintaining strong generalization on exocentric video tasks with only

52k

training samples.

Yogesh Kulkarni, Pooyan Fazli

Arizona State University

Egocentric Video Fine-Grained Video Understanding Egocentric Vision Reasoning Skills Feedback

Oct 2, 2025cs.CV

Oracle-RLAIF: An Improved Fine-Tuning Framework for Multi-modal Video Models using Reinforcement Learning from Ranking Feedback

Recent advances in large video-language models (VLMs) rely on extensive fine-tuning techniques that strengthen alignment between textual and visual comprehension. Leading pipelines typically pair supervised fine-tuning (SFT) with reinforcement learning from preference data to enhance video comprehension. However, as VLMs scale in parameter size, so does the cost of gathering enough human feedback. To make fine-tuning more cost-effective, recent frameworks explore reinforcement learning with AI feedback (RLAIF), which replace human preference with AI as a judge. Current RLAIF frameworks rely on a specialized reward model trained with video narratives to create calibrated scalar rewards -- an expensive and restrictive pipeline. We propose Oracle-RLAIF, a novel framework that replaces the trained reward model with a more general Oracle ranker which acts as a drop-in model ranking candidate model responses rather than scoring them. Alongside Oracle-RLAIF, we introduce

GRPO_{rank}

, a novel rank-based loss function based on Group Relative Policy Optimization (GRPO) that directly optimizes ordinal feedback with rank-aware advantages. Empirically, we demonstrate that Oracle-RLAIF consistently outperforms leading VLMs using existing fine-tuning methods when evaluated across various video comprehension benchmarks. Oracle-RLAIF paves the path to creating flexible and data-efficient frameworks for aligning large multi-modal video models with reinforcement learning from rank rather than score.

Derek Shi, Ruben Glatt, Christine Klymko +5

Stanford University, CA, USA · Lawrence Livermore National Laboratory, CA, USA · Microsoft, WA, USA

Video Understanding Model Fine-Tuning Oracles Feedback

Sep 22, 2025cs.CL

Interactive In-Meeting Speaker Correction with Human Feedback

Most automatic speech processing systems operate in ``open loop'' mode without user feedback about who said what, yet human-in-the-loop workflows can potentially enable higher accuracy. We propose an LLM-assisted in-meeting speaker correction system that lets users fix speaker attribution errors through brief corrective feedback. After performing streaming ASR and diarization, the system presents concise LLM-generated summaries to help users identify important speaker errors, and it incorporates user feedback by updating the speaker-attributed transcript and adding online speaker enrollments. To make this workflow effective despite errors in speech processing, LLM analysis, and user feedback, we developed several mechanisms to identify the intended correction more precisely. Further, we built an LLM-driven user feedback simulation to evaluate the workflow reprodubilty and at scale. Applied to the AMI headset test set, our system substantially reduces the DER from a streaming baseline (Google ASR + ECAPA) by 31.99% and speaker substitution error by 52.68%. Results of a pilot usability study suggest several avenues to improve the user experience.

Xinlu He, Yiwen Guan, Badrivishal Paurana +3

Worcester Polytechnic Institute

Speaker Correction Feedback User Simulation Reproducibility

Aug 18, 2025cs.CV

CountLoop: Training-Free High-Instance Image Generation via Iterative Agent Guidance

Diffusion models excel at photorealistic synthesis but struggle with object count fidelity, especially in high-density settings. We introduce COUNTLOOP, a training-free framework that achieves structured instance and count control through iterative, structured feedback. Our method alternates between synthesis and evaluation: a VLM-based planner generates structured scene layouts, while a VLM-based critic provides explicit feedback on object counts, spatial arrangements, and visual quality to refine the layout iteratively. Instance-driven attention masking and cumulative attention composition further prevent semantic leakage, ensuring clear object separation even in densely occluded scenes. Evaluations on COCO-Count, T2I-CompBench, and two newly introduced high instance benchmarks show that COUNTLOOP reduces counting error by up to 57% and achieves the highest or comparable spatial quality scores across all benchmarks, while maintaining photorealism. Project page is at https://mondalanindya.github.io/CountLoop/.

Anindya Mondal, Sauradip Nag, Ayan Banerjee +3

University of Surrey · Simon Fraser University · Universitat Autònoma de Barcelona

Multi-Reference Image Generation Indoor Scene Generation Counting Feedback

Jul 29, 2025cs.CL

Post-Training Large Language Models via Reinforcement Learning from Self-Feedback

Large Language Models (LLMs) often produce plausible but poorly-calibrated answers, limiting their reliability on reasoning-intensive tasks. Recent research suggests that Chain-of-Thought (CoT) reasoning paths are inherent in pre-trained LLMs and can be elicited by simply altering the decoding process, where the presence of a CoT path correlates with higher answer confidence. Building on these insights, we present Reinforcement Learning from Self-Feedback (RLSF), a post-training stage that utilises the model's intrinsic confidence as a self-generated reward. By generating multiple CoT decoding beams from a frozen LLM, we compute the confidence of each final answer span and rank the resulting traces accordingly to create synthetic preferences. These preferences are subsequently utilised to fine-tune the policy through standard preference optimisation, requiring no human labels, gold answers, or externally curated rewards. RLSF simultaneously (i) refines the model's probability estimates--restoring well-behaved calibration--and (ii) strengthens step-by-step reasoning, yielding improved performance on arithmetic reasoning and multiple-choice question answering. By converting a model's own uncertainty into structured self-feedback, RLSF affirms reinforcement learning on intrinsic model behaviour as a principled and data-efficient component of the LLM post-training pipeline. Our results demonstrate that leveraging these inherent reasoning capabilities provides a robust path for enhancing model reliability without manual prompt engineering or external supervision.

Carel van Niekerk, Renato Vukovic, Benjamin Ruppik +3

Heinrich Heine University Düsseldorf, Germany

Large Language Model Reinforcement Learning LLM Reasoning Strategies Reasoning Paths Feedback

Jul 26, 2025math.OC

Douglas-Rachford Splitting for Group-Sparse Feedback Linear-Quadratic Control

In this paper, we study the distributed linear quadratic problem with fixed communication topology (DFT-LQ) and the sparse feedback linear quadratic (SF-LQ) problem through a unified optimization framework. Specifically, both problems are formulated as a nonconvex, nonsmooth optimization problem equipped with an

\ell_0

-penalty under affine constraints. To solve this problem, we first investigate the application of the Douglas-Rachford (DR) splitting algorithm. Under the local condition that the generated iterates remain on a fixed smooth manifold, we establish the convergence of the DR splitting to a stationary point. Furthermore, we characterize this stationary point as the global minimizer of a corresponding DFT-LQ problem. To bypass the restriction of the smooth manifold assumption, we introduce a projected subgradient descent algorithm that achieves global convergence without relying on smooth-manifold structures. This algorithm may serve as a warm-start mechanism that effectively drives the iterates toward the desired smooth manifolds, thereby establishing a favorable initialization where the convergence theory of the DR splitting algorithm becomes fully applicable. Numerical experiments shed light on the effectiveness of the proposed methods in distributed group-sparse controller design.

Lechen Feng, Xun Li, Yuan-Hua Ni

Department of Applied Mathematics, The Hong Kong Polytechnic University, China · College of Artificial Intelligence, Nankai University, Tianjin, China

Linear Quadratic Regulator Communication Topologies Feedback

May 16, 2025cs.CV

Conditioning Residuals for Diffusion Models via Representation Feedback

Diffusion models now serve as a common foundation for multimedia generation, and useful intermediate representations emerge during their generative training. Standard architectures, however, propagate these representations through the main feature stream, without explicitly reintroducing their encoded semantics to later denoising layers. Meanwhile, such backbones already provide a conditioning pathway for global modulation by predefined inputs. This work examines whether this native pathway can also route internally inferred semantics as evolving, sample-dependent cues. We propose Conditioning Residuals, a lightweight feedback mechanism that converts aggregated features into residuals added to condition embeddings. By feeding back compact feature summaries, it provides adaptive generative guidance and encourages a tighter semantic bottleneck, without external encoders, auxiliary objectives, or sampling-time changes. It supports feedback at one or multiple depths in UNet and DiT backbones, with negligible overhead. Across diffusion formulations, backbone configurations, and datasets, experiments show consistent gains in generative performance, along with stronger representations in downstream linear probing and segmentation. Mechanistic analyses reveal improved generative training dynamics and reshaped feature structure, suggesting a grounded, generalizable way to enhance diffusion backbones from within.

Weilai Xiang, Hongyu Yang, Di Huang +1

State Key Laboratory of Virtual Reality Technology and Systems, School of Computer Science and Engineering, Beihang University, Beijing 100191, China · Institute of Artificial Intelligence, Beihang University, Beijing 100191, China · School of Computer Science and Engineering, Beihang University, Beijing 100191, China

Diffusion Models Generative Models Feedback Contextual Cues

Apr 16, 2025cs.LG

Reinforcement Learning from Human Feedback

Reinforcement learning from human feedback (RLHF) has become a crucial tool to build the latest machine learning systems at scale. The field grew around the core methods of RLHF into today's broader suite of post-training techniques. In this book, we give a comprehensive introduction to the core methods for post-training models for people with some level of quantitative background, organized around the canonical RLHF recipe. The book starts with what RLHF does and why it was created, with seminal technical milestones in its young history and a primer on reinforcement learning context needed to understand the book. The core of the book details every optimization stage in using RLHF, from starting with instruction tuning to training a reward model and finally all of rejection sampling, reinforcement learning, on-policy distillation, and direct alignment algorithms. The book also discusses broader topics, such as the origins of RLHF -- both in recent literature and in a convergence of disparate fields of science in economics, philosophy, and optimal control. The book concludes with advanced topics -- understudied or emerging research questions in synthetic data, tool-use, character training, and evaluation -- and open questions for the field. The book is released with a variety of companion resources, including a codebase, a library to compare model completions from within post-training stages, and an educational course, to be a one-stop shop for learning all foundational concepts for post-training language models.

Nathan Lambert

Reinforcement Learning From Human Feedback Post-Training Large Language Model Training Feedback

Mar 19, 2025cs.LG

PB-OEL: A Performance-Bounded Online Ensemble Learning Framework With Mixed Feedback for Real-Time Safety Assessment

Real-time safety assessment is critical for ensuring the reliable operation of complex dynamic systems. However, obtaining full safety labels in real time is often prohibitively expensive, resulting in a challenging mixed-feedback scenario dominated by partial feedback, especially under concept drift. Furthermore, existing online ensemble methods typically rely on heuristic weight allocation, lacking provable performance guarantees under such limited-feedback conditions. To address these challenges, we propose PB-OEL, a performance-bounded online ensemble learning framework designed for real-time safety assessment under mixed feedback. At the ensemble level, a theoretical framework is established to bound the performance of the ensemble classifier relative to its base classifiers across varying feedback ratios. By formally defining the form of expert advice, the bound guarantees that the ensemble outperforms any individual base classifier over a sufficiently large data stream. At the base-classifier level, a penalty-based update strategy is introduced, enabling base models to explicitly leverage misclassified samples rather than simply discarding them. Extensive experiments on the real-world Jiaolong manned submersible dataset demonstrate that PB-OEL maintains robust predictive performance and outperforms state-of-the-art methods.

Songqiao Hu, Zeyi Liu, Lufeng Hao +2

Department of Automation, and the Institute for Embodied Intelligence and Robotics, Tsinghua University, Beijing 100084, China · China Ship Research and Development Academy, Beijing 100192, China

Probabilistic Safety Safety-Critical Scenarios Ensemble Feedback

Feb 3, 2025cs.LG

Memento No More: Coaching AI Agents to Master Multiple Tasks via Hints Internalization

As the general capabilities of artificial intelligence (AI) agents continue to evolve, their ability to learn to master multiple complex tasks through experience remains a key challenge. Current LLM agents, particularly those based on proprietary language models, typically rely on prompts to incorporate knowledge about the target tasks. This approach does not allow the agent to internalize this information and instead relies on ever-expanding prompts to sustain its functionality in diverse scenarios. This resembles a system of notes used by a person affected by anterograde amnesia, the inability to form new memories. In this paper, we propose a novel method to train AI agents to incorporate knowledge and skills for multiple tasks without the need for either cumbersome note systems or prior high-quality demonstration data. Our approach employs an iterative process where the agent collects new experiences, receives corrective feedback from humans in the form of hints, and integrates this feedback into its weights via a context distillation training procedure. We demonstrate the efficacy of our approach by implementing it in a Llama-3-based agent that, after only a few rounds of feedback, outperforms advanced models GPT-4o and DeepSeek-V3 in tasksets requiring correct sequencing of information retrieval, tool use, and question answering.

Minttu Alakuijala, Ya Gao, Georgy Ananov +4

Department of Computer Science, Aalto University · Department of Computer Science, University of Manchester · System 2 AI

Artificial Intelligence Agents Internalization New Tasks Memo Dataset Distillation Feedback

Jan 14, 2025cs.LG

Impatient Bandits: Optimizing for the Long-Term Without Delay

Increasingly, recommender systems are tasked with improving users' long-term satisfaction. In this context, we study a content exploration task, which we formalize as a bandit problem with delayed rewards. There is an apparent trade-off in choosing the learning signal: waiting for the full reward to become available might take several weeks, slowing the rate of learning, whereas using short-term proxy rewards reflects the actual long-term goal only imperfectly. First, we develop a predictive model of delayed rewards that incorporates all information obtained to date. Rewards as well as shorter-term surrogate outcomes are combined through a Bayesian filter to obtain a probabilistic belief. Second, we devise a bandit algorithm that quickly learns to identify content aligned with long-term success using this new predictive model. We prove a regret bound for our algorithm that depends on the Value of Progressive Feedback, an information-theoretic metric that captures the quality of short-term leading indicators that are observed prior to the long-term reward. We apply our approach to a podcast recommendation problem, where we seek to recommend shows that users engage with repeatedly over two months. We empirically validate that our approach significantly outperforms methods that optimize for short-term proxies or rely solely on delayed rewards, as demonstrated by an A/B test in a recommendation system that serves hundreds of millions of users.

Kelly W. Zhang, Thomas Baldwin-McDonald, Kamil Ciosek +2

Imperial College London · University of Manchester · Spotify +2

Bandits Long-Term Feedback

Dec 19, 2024cs.CL

LDC: Learning to Generate Research Idea with Dynamic Control

Recent advancements in large language models (LLMs) have demonstrated their potential in automating the scientific research ideation. Existing approaches primarily focus on prompting techniques, often producing ideas misaligned with expert standards - novelty, feasibility, and effectiveness, which are widely recognized by the research community as the three key subdimensions of high-quality ideas. Also, balancing these dimensions remains challenging due to their inherent trade-offs. To address these limitations, we propose the first framework that employs a two-stage approach combining Supervised Fine-Tuning (SFT) and controllable Reinforcement Learning (RL) for the task. In the SFT stage, the model learns foundational patterns from pairs of research papers and their corresponding follow-up ideas. In the RL stage, multi-dimensional reward models guided by fine-grained feedback evaluate and optimize the model across key dimensions. During inference, dimensional controllers coordinated by a sentence-level decoder enable dynamic context-aware steering of the idea generation process. Our framework provides a balanced approach to research idea generation, achieving high-quality outcomes in the experiment by dynamically navigating the trade-offs among novelty, feasibility, and effectiveness.

Ruochen Li, Liqiang Jing, Chi Han +2

University of Texas at Dallas · UIUC · Stony Brook University

Scientific Ideation Research Automation Large Language Model Generation Scientific Discovery Code Generation Feedback

Nov 3, 2024cs.CL

A Course Intelligence Platform for Higher Education: Lessons from AI-Assisted Course Evaluation

The rapid adoption of generative AI has created new opportunities for teaching, learning, and quality assurance. Existing applications, however, remain largely student-facing, with comparatively limited attention to institution-level needs. This paper presents a course intelligence platform deployed across more than 100 universities and serving over 10,000 instructors in China. By linking competency requirements, knowledge structures, teaching activities, and assessment evidence, it establishes a shared foundation for knowledge organization, instructional design, learning assessment, and quality evaluation. The course evaluation module is examined as a representative institution-facing application of the platform, which integrates national evaluation standards, structured educational evidence, customized prompting strategies, and domain-adapted LLMs to generate quantitative scores and qualitative feedback. A case study involving 100 authentic university courses is conducted to evaluate its alignment with expert judgments and the interpretability of its outputs. Statistical analyses show substantial agreement between AI-generated assessments and expert ratings, while qualitative results highlight the credibility of the feedback. The findings further suggest that AI-assisted evaluation requires not only capable models but also structured domain knowledge and transparent criteria. In this context, human ratings should be treated as an informative reference rather than an error-free gold standard, and the objective is to achieve consistent, interpretable, and defensible judgments instead of merely replicating expert scores.

Bo Yuan, Jiazi Hu, Haimei Zhao

University of Queensland, Brisbane, QLD 4072, Australia · AI U Course, Kunming, Yunnan 650000, P.R. China

Computerized Adaptive Testing Artificial Intelligence Evaluation Feedback

Date pendingcs.RO

Modular Sensory Stream for Integrating Physical Feedback in Vision-Language-Action Models

Humans understand and interact with the real world by relying on diverse physical feedback beyond visual perception. Motivated by this, recent approaches attempt to incorporate physical sensory signals into Vision-Language-Action models (VLAs). However, they typically focus on a single type of physical signal, failing to capture the heterogeneous and complementary nature of real-world interactions. In this paper, we propose MoSS, a modular sensory stream framework that adapts VLAs to leverage multiple sensory signals for action prediction. Specifically, we introduce decoupled modality streams that integrate heterogeneous physical signals into the action stream via joint cross-modal self-attention. To enable stable incorporation of new modalities, we adopt a two-stage training scheme that freezes pretrained VLA parameters in the early stage. Furthermore, to better capture contact interaction dynamics, we incorporate an auxiliary task that predicts future physical signals. Through extensive real-world experiments, we demonstrate that MoSS successfully augments VLAs to leverage diverse physical signals (i.e., tactile, force, and torque), integrating multiple signals to achieve synergistic performance gains.

Jimin Lee, Huiwon Jang, Myungkyu Koo +2

KAIST · RLWRLD

Diffusion-Based Vision-Language-Actions Sensory Feedback Cross-Modal Attention Photometric Supervision Modalities Feedback

Date pendingcs.LG

SaFeR-Steer: Evolving Multi-Turn MLLMs via Synthetic Bootstrapping and Feedback Dynamics

MLLMs are increasingly deployed in multi-turn settings, where attackers can escalate unsafe intent through the evolving visual-text history and exploit long-context safety decay. Yet safety alignment is still dominated by single-turn data and fixed-template dialogues, leaving a mismatch between training and deployment. To bridge this gap, we propose SaFeR-Steer, a progressive multi-turn alignment framework that combines staged synthetic bootstrapping with tutor-in-the-loop GRPO to train a single student under adaptive, on-policy attacks. We also introduce Trajectory-Consistent Summative Reward (TCSR), which aggregates the historical minimum and average of turn rewards so that any low-quality turn affects the trajectory-level return. I. Dataset. We release STEER, a multi-turn multimodal safety dataset with STEER-SFT (12,934), STEER-RL (2,000), and STEER-Bench (3,227) dialogues spanning 1-10 turns. II. Experiment. Starting from Qwen2.5-VL-3B/7B, SaFeR-Steer substantially improves Safety/Helpfulness on both single-turn (48.30/45.86

\rightarrow

81.84/70.77 for 3B; 56.21/60.32

\rightarrow

87.89/77.40 for 7B) and multi-turn benchmarks (12.55/27.13

\rightarrow

55.58/70.27 for 3B; 24.66/46.48

\rightarrow

64.89/72.35 for 7B), shifting failures to later turns and yielding robustness beyond scaling alone. Code is available at https://github.com/Ed-Bg/SaFeR-Steer-full

Haolong Hu, Hanyu Li, Tiancheng He +6

Huazhong University of Science and Technology · Beijing University of Posts and Telecommunications · West China Biomedical Big Data Center, Sichuan University +2

Mllm-Based Large Language Model Alignment Multimodal Large Language Models Unsafe Pluralistic Alignment Feedback+1

Date pendingcs.RO

A Dynamic Toolkit for Transmission Characteristics of Precision Reducers with Explicit Contact Geometry

Precision reducers couple contact geometry, bearing support, structural deformation, and loading history. This paper presents a dynamic toolkit connecting distributed local contacts to the complete mechanical reaction path. Work-conjugate maps transfer displacement, reaction, and tangent contributions between contacts and rigid or reduced coordinates, with explicit allocation of contact and body elasticity. An implicit generalized-alpha solution distinguishes trial evaluations from accepted history. Contact records then support performance protocols and configured geometric or constitutive feedback. The numerical studies focus on mechanical coupling and pressure recovery. An idealized annular housing retains physical interfaces while its structural coordinates are reduced. Two reductions with similar static errors have cross-port response errors of 24.975 and 0.312 percent over the same frequency band relative to a common parent model. In a shared-pin example, a 20 micrometer radial displacement of one wheel changes the load on a second, fixed wheel by approximately 75 N. Removing cross-station compliance removes this incremental transfer on the tested sleeve-seating branch. A double-wheel cycloidal assembly relates torsional branch response to aggregate contact-load variation and normalized pressure fields. The pressure maxima remain sensitive to resolution despite small discrete force residuals. Further formulations specify how motion and contact records support precision, vibration, heat, wear, and durability models with their required inputs. The framework separates model representation and numerical resolution while retaining common definitions of motion, force, and observation.

Jiacheng Miao, Chao Liu, Qiliang Wang +2

Intelligent Equipment Division, CRRC Qishuyan Institute Co., Ltd, Changzhou, 213011, China · School of Mechanical Engineering, Dalian Jiaotong University, Dalian, 116028, China · Research Institute, CRRC Qishuyan Institute Co., Ltd, Changzhou, 213011, China

Rigid-Body Dynamics Precision Transmission Feedback