Model-Free

Follow this topic Ask about this topic

Momentum

1 paper in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Jul 6Week of Sep 21

Latest papers 18

Oct 1, 2026cs.AI

Q-Learning for Reachability in MEC-Free MDPs

Reinforcement learning (RL) for reachability specifications is fundamental to sequential decision-making. Prior work establishes asymptotic convergence to optimal policies, but only through model-based methods that must explicitly estimate the transition probabilities of the underlying Markov Decision Process (MDP). We present Quasar, the first model-free algorithm with asymptotic guarantees for reachability on the fragment of MDPs free of non-terminal maximal end components (MECs), a building block to which every MDP reduces by the standard MEC quotient. Our algorithm follows the classical Q-learning approach, using temporal-difference updates to converge to an optimal policy without ever learning the transition probabilities. The resulting learner reduces the memory footprint from the O(|S|^2|A|) that model-based methods require to O(|S||A|). On the standardized Quantitative Verification Benchmark Set, our algorithm converges to the optimal policy with orders of magnitude fewer samples than the previous model-based state-of-the-art. Together these results are a concrete step toward the practical deployment of reachability learning and, with it, of specification-guided RL.

Lu-Chin Chang, Suguman Bansal

Georgia Institute of Technology

Q-Learning Markov Decision Processes Reachability Model-Free Offline Reinforcement Learning

Sep 29, 2026cs.AI

Decode-Latency Feedback Prefill: A Model-Free Controller and Its Generalization Limits

Concurrent autoregressive inference creates a fundamental interference problem: prefilling a newly arrived long prompt can delay tokens for requests that are already decoding. Fixed prefill chunks reduce this interference, but the best chunk size depends on the model, hardware, load, and latency objective. We introduce Decode-Latency Feedback Prefill (DLFP), a model-free controller that changes only prefill work that overlaps active decodes. After a guarded scheduling cycle, DLFP uses the observed interval as proportional feedback to resize the next prefill chunk; isolated prefills remain unrestricted. We implement DLFP in vLLM and evaluate it with open-loop Poisson arrivals, exact token accounting, raw request traces, and NVIDIA telemetry. On Qwen3-0.6B in BF16 on one A100 80 GB GPU, three paired 100-request trials reduce P99 inter-token latency by 24.8%, 30.1%, and 28.2% (mean 27.7%, paired 95% confidence interval 21.0% to 34.3%) with exact output agreement, no failures, and unchanged SLO compliance. The benefit is not free: mean P99 time to first token increases 34.8% while remaining inside the declared SLO. Crucially, the mechanism does not generalize to Qwen3-8B, Qwen3-32B, or a two-GPU tensor-parallel configuration. We trace the failure to an asynchronous scheduler-call interval that is only a proxy for completed GPU iteration time. This negative result defines the boundary of the contribution and motivates a completion-timed controller for concurrent CPU and on-device inference. We do not claim mobile-device performance; the present work is a reproducible proof-of-concept and generalization study.

Gaurav Agarwal, Ashish Garg, Isha Singhal

Time-To-First-Token Graphics Processing Unit Resources Autoregressive Model Feedback Model-Free

Sep 1, 2026cs.CV

Separating perception from reasoning in vision-language models: a model-free render ceiling for crystal structures

Multimodal evaluations cannot say whether a vision-language model misread an image or misreasoned about it, because every existing method for separating the two places a second model in the loop. We introduce the render ceiling, a model-free reference for benchmarks built by rendering known objects: inverting the frozen cameras and re-solving cross-view correspondence recovers exactly the answer the images support. We prove the ceiling fails only through an enumerable set of projection coincidences and certify that set empty on 2,160 rendered crystal structures, so every point of a model's deficit belongs to the model. Across fourteen vision-language models, supplying exact geometry as text lifts every model yet closes under half the gap for thirteen, while a supervised vision model with no language component reads the same images at 0.8952, above every vision-language model. The instrument exposes extraction-stage fabrication that downstream accuracy would misattribute to reasoning, yields camera-placement rules for benchmark builders, and transfers to any benchmark with an invertible forward rendering.

Can Polat, Mustafa Kurban, Erchin Serpedin +1

Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX, USA · Department of Electrical and Computer Engineering, Texas A&M University at Qatar, Doha, Qatar · Department of Prosthetics and Orthotics, Ankara University, Ankara, Turkey +1

Recent Vision-Language Models Ceiling Perception Inverse Rendering Crystal Structure Ground-Truth Correspondence+1

Jun 30, 2026cs.RO

Energy-Optimal Spatial Iterative Learning within a Virtual Tube

Due to the limited endurance of embedded energy sources such as lithium-polymer (LiPo) batteries, the flight duration and operational range of unmanned aerial vehicles (UAVs) are severely constrained. Although energy-efficient trajectory planning and control have been widely studied, most existing approaches rely on accurate system models and computationally expensive optimization procedures. This paper proposes a model-free online iterative learning (IL) framework to minimize energy consumption. Without requiring explicit models of UAV dynamics or energy consumption, the proposed method improves energy efficiency while maintaining a low computational cost. The per-iteration computational complexity is O(n), where n denotes the number of path points. In the tested cases, the proposed method is approximately 50--60 times faster than the model-based IPOPT benchmark. Simulation results and real-world flight experiments across multiple UAV platforms validate the effectiveness, computational efficiency, and practical applicability of the proposed approach.

Chen Min, Shuli Lv, Pengda Mao +3

School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, China · Tianmushan Laboratory, Beihang University, Hangzhou, 311115

Unmanned Aerial Vehicle Trajectories Energy-Efficient Generative Control Policies Model-Free Lithium-Ion Batteries Spatiotemporal Tube

Jun 25, 2026math.OC

Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Data

This paper addresses model-free continuous-time mean-field control in a setting where the population dynamics evolve continuously according to an unknown McKean-Vlasov stochastic differential equation, while only discrete-time transition data are available. In the model-based formulation, policy evaluation is naturally described by a stationary Hamilton-Jacobi-Bellman equation on

\mathcal P_2(\mathbb R^d)

, but this equation involves the drift and diffusion coefficients of the controlled McKean-Vlasov dynamics, which are not identifiable when only discrete-time data are available. On the other hand, a direct reduction to a time-discrete Bellman equation avoids the non-identifiability issue but loses the differential equation structure. To bridge these two viewpoints, we introduce a Mean-Field-PhiBE (MF-PhiBE), which incorporates discrete-time transition information into a continuous-time PDE on the Wasserstein space. The MF-PhiBE replaces the unknown infinitesimal drift and covariance in the policy-evaluation equation by one-step estimators computed from data, while preserving the generator structure of the McKean-Vlasov HJB equation. We also derive a policy-gradient theorem for entropy-regularized randomized feedback policies, expressing the actor direction through an action-wise infinitesimal advantage and the score of the policy. Combining these two ingredients yields a model-free actor-critic method. We prove a first-order consistency estimate showing that the value induced by an optimal MF-PhiBE policy approximates the optimal continuous-time value with an error of order

Δt

. In the linear-quadratic case, we show our approximation achieves second-order accuracy with only one-step data. Numerical experiments on an LQR benchmark and a crowd-aversion problem illustrate the proposed framework.

Erhan Bayraktar, Martin Hernandez, Qinxin Yan +1

Department of Mathematics, University of Michigan, Ann Arbor, MI, USA. · Department of Statistics and Data Science, University of California, Los Angeles, CA, USA. · Program in Applied and Computational Mathematics, Princeton University, Princeton, NJ, USA.

Stochastic Optimal Control Model-Free Mean-Field Limit Bellman Equation Fokker--Planck Equation Optimal Policies+2

Jun 17, 2026eess.SY

Model-Free Reinforcement Learning Control for Resilient Cyber-Physical Systems

This paper compares the performance of model-free controllers on a nonlinear system under cyberattacks, including false data injection and denial-of-service attacks. Four RL reward types are analyzed for accuracy, cost, and resilience. Results show that the Lyapunov reward offers the best resilience with low tracking error. Exponential mode also provides good trade-offs with acceptable resilience under moderate training conditions. Progressive and linear rewards converge faster but are less robust. RL-MPCs show strong steady-state resilience but require longer training times; RL-PID controllers are faster with significantly less training time. Proximal Policy Optimization outperforms Deep Deterministic Policy Gradient with a significant reduction in KPI variance. This study serves to highlight how well-designed RL rewards can improve performance and resilience against cyber threats.

Hugo O. Garcés, Alejandro J. Rojas, Bernardo A. Hernández +5

Departamento de Ingeniería Informática y Ciencias de la Computación, Universidad de Concepción, Concepción, Chile · Electrical Engineering Departament, Universidad de Talca · Department of Electrical & Computer Engineering, University of Alberta, Edmonton, T6G 1H9, Alberta, AB, Canada +2

Cyber-Physical System Model-Free Reinforcement Learning Model-Free Resilience Cyberattacks

Jun 15, 2026cs.LG

Learning Policy from a Single Trajectory in Average-Reward Markov Decision Process

While there is an extensive body of work characterizing the sample complexity of discounted cumulative-reward MDPs, finite sample analyses for average-reward MDPs have been limited, and most existing works rely on restrictive assumptions such as ergodicity or access to a generative model. In this work, we establish the first finite sample complexity guarantees from a single trajectory for weakly communicating average-reward MDPs. To this end, we study the dynamics of a single trajectory in weakly communicating MDPs and based on this analysis, we develop novel model-free methods. Notably, our value-based and policy-based methods provide finite sample complexity guarantees of

\widetilde{O}(1/\varepsilon^2)

and

\widetilde{O}(1/\varepsilon^4)

from a single trajectory in weakly communicating MDPs, respectively. Furthermore, we introduce the first model-free method that requires no prior knowledge of problem-dependent quantities for communicating MDPs.

Jongmin Lee, Ernest K. Ryu, Vaneet Aggarwal

Seoul National University · UCLA · Purdue University

Markov Decision Processes Model-Free Learning Policy Optimal Sample Complexity Single Trajectory

Jun 13, 2026cs.RO

A Hybrid Model-Based and Model-Free Framework for Active Multi-View Viewpoint Optimization in Sonar Target Recognition

This paper presents a hybrid model-based and model-free framework for active multi-view target recognition using forward-looking sonar. A convolutional neural network (CNN) provides data-driven observation likelihoods, while Radon-based orientation estimation enables viewpoint-aware sensing without requiring angle annotations. During training, an information-gain-based reward guides a Proximal Policy Optimization (PPO) agent to learn a belief-aware viewpoint selection policy offline. At deployment, the learned policy performs real-time viewpoint selection using only CNN-based belief updates, eliminating the need for computationally expensive online POMDP tree search. Experiments on a marine-debris forward-looking sonar dataset demonstrate that the proposed approach achieves competitive recognition accuracy while reducing sensing steps and motion cost compared to model-based baselines.

Yongkyoon Park, Jane Shin

University of Florida Gainesville, FL, United States of America

Forward-Looking Sonar Multi-View Model-Free Convolutional Neural Networks

Jun 4, 2026cs.CL

AdaPLD: Adaptive Retrieval and Reuse for Efficient Model-Free Speculative Decoding

Speculative decoding accelerates generation by verifying multiple drafted tokens in a single target-model forward pass, reducing sequential decoding iterations. Model-free variants avoid auxiliary draft models by reusing text and model states already available during generation, but their speedup depends on the reliability of the constructed drafts. We identify two limitations of existing reuse-based methods: lexically anchored retrieval has limited recall under surface-form variation, and deterministic span copying can be brittle when the retrieved context does not uniquely determine the continuation. We propose \emph{AdaPLD}, a training-free method that adaptively improves both retrieval and draft construction. AdaPLD preserves high-precision lexical reuse while using semantic similarity to recover additional reuse opportunities when lexical matching fails. It further constructs branched reuse hypotheses to account for continuation uncertainty, rather than relying on a single copied span. Across diverse benchmarks, AdaPLD reduces target-model forward passes and achieves up to

3.10\times

decoding speedup.

Runheng Liu, Jincheng Xie, Wen Hu +2

School of Computer Science and Technology, Beijing Institute of Technology · Department of Mathematical Sciences, Tsinghua University · JDT AI Infra

Speculative Decoding Drafts Reuse Model-Free

May 30, 2026q-bio.NC

Cortex and subcortex play distinct roles over learning when cortical memory is limited

It has been proposed that the brain integrates flexible, computationally expensive cortical processing with simpler, lower-cost subcortical mechanisms to achieve resource-efficient performance greater than that of either system alone. Despite the allure of this perspective, satisfying theoretical frameworks that explore this hypothesis are still limited. We extend existing frameworks in which a model-based module and model-free module learn in tandem by explicitly constraining the memory resources of the model-based module, and investigate the impact of this constraint in a simple decision-making setting. Memory constraints naturally give rise to strategies for allocating memory resources. We evaluate the performance of different strategies in different situations and demonstrate that when the rewarded states change often, it can be advantageous for the model-based module to focus its memory resources not on exploiting the current reward, but on capturing general structure of the environment. This work provides a theoretical foundation for a functional dissociation between cortical and subcortical systems during learning: the cortex supports general structure learning, while subcortical circuits specialize in reward-based learning. We further detail how these hypotheses can be tested on experimental data.

Matthew Farrell, Taro Toyoizumi

Laboratory for Neural Computation and Adaptation RIKEN Center for Brain Science, Wako, Japan · Department of Mathematical Informatics, Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, Japan.

Human Visual Cortex Working Memory Model-Free Cortex Role

May 18, 2026cs.RO

Transfer Learning for Customized Car Racing Environments

Transfer Learning, a technique where a model/agent can use the knowledge/expertise that it gained from one task and exploit that to solve another closely-related task, is often used in tackling problems in deep learning. Through this project, we explore transfer learning in the purview of deep reinforcement learning. Specifically, we want to use transfer learning to achieve the fast lap times in OpenAI's Car racing environment by training the agent on one circuit, and racing it on other customized target environments by zero-shot transfer or by additional fine-tuning. In addition, we compare the performance of model-based and model-free approaches, and observe that model-based approaches dominate in performance and converge faster than model-free approaches in this environment. We observe that transfer learning in most setups not only boosts the performance on the target domain, but also shows high performance ability during learning.

Benedict Florance Arockiaraj, Richard Chang, Wesley Yee

SEAS

Transfer Learning Autonomous Racing Model-Free Environment Openai

May 11, 2026cs.GT

Towards Model-Free Learning in Dynamic Population Games: An Application to Karma Economies

Dynamic Population Games (DPGs) provide a tractable framework for modeling strategic interactions in large populations of self-interested agents, and have been successfully applied to the design of Karma economies, a class of fair non-monetary resource allocation mechanisms. Despite their appealing theoretical properties, existing computational tools for DPGs assume full knowledge of the game model and operate in a centralized fashion, limiting their applicability in realistic settings where agents have access only to their own private experience. This paper takes a step towards addressing this gap by studying model-free equilibrium learning in Karma DPGs. First, we analyze the setting in which a novel agent joins a Karma DPG already at its Stationary Nash Equilibrium (SNE) and learns a policy via Deep Q-Networks (DQN) without knowledge of the game model. Leveraging recent convergence results for DQN, we establish a suboptimality bound consisting of a DQN approximation error of order

O(1/\sqrt{N_s})

and a mean field perturbation error of order

O(1/N)

, where

N_s

is the replay buffer size and

N

is the population size. Second, we consider the challenging problem of learning the SNE from scratch. We show empirically that combining deep RL with fictitious play and smoothed policy iteration allows agents to converge, in a model-free fashion, to a configuration close to the centrally computed SNE. Together, these contributions support the vision of Karma economies as practical tools for fair resource allocation.

Matteo Cederle, Saverio Bolognani, Gian Antonio Susto

Department of Information Engineering University of Padova · Automatic Control Laboratory ETH Zurich

Deep Q-Networks Model-Free Model-Free Reinforcement Learning Multi-Agent Reinforcement Learning Population Dynamics

May 8, 2026cs.LG

Central Limit Theorem for Two-Time-Scale Approximate Distributionally Robust RL

Designing model-free algorithms for distributionally robust reinforcement learning (DRRL) poses fundamental challenges. The robust Bellman operator is nonlinear in the transition kernel, which makes one-sample Bellman updates biased, while the adversarial optimization underlying robustness makes robust evaluation computationally demanding. To address these difficulties, we consider the natural small-ambiguity regime under Kullback--Leibler ambiguity sets and propose an approximate DRRL framework based on a first-order expansion of the relevant robust functional. This yields an approximate robust Bellman equation that removes the adversarial optimization while remaining first-order accurate in the ambiguity radius. To learn the fixed point of this approximate equation, we propose Mean-Variance Stochastic Approximation (MVSA), a model-free algorithm that uses only one-sample updates. This is achieved via a lifted stochastic approximation dynamics and a two-time-scale design. We then prove convergence and a central limit theorem for MVSA: its main iterate satisfies a central limit theorem at the canonical

n^{-1/2}

scale, with explicitly characterized asymptotic covariances. Finally, we validate our theoretical findings with a numerical experiment.

Shengbo Wang, Zexi Zhang

Daniel J. Epstein Department of Industrial and Systems Engineering, University of Southern California

Distributional Reinforcement Learning Large Deviation Principle Model-Free Offline Reinforcement Learning Bellman Equation

May 8, 2026cs.LG

Actor-Critic Algorithm for Dynamic Expectile and CVaR

Optimizing dynamic risk with stochastic policies is challenging in both policy updates and value learning. The former typically requires transition perturbation, while the latter may rely on model-based approaches. To address these challenges, we propose a surrogate policy gradient without transition perturbation under softmax policy parameterization. We further develop model-free value learning methods for dynamic expectile and conditional value-at-risk by leveraging elicitability. Finally, inspired by Expected SARSA and Expected Policy Gradient, a model-free off-policy actor-critic algorithm is constructed. Empirical results in domains with verifiable risk-averse behavior show that our algorithm can learn risk-averse policy and consistently outperforms other existing methods.

Yudong Luo, Erick Delage

1GERAD & Department of Decision Sciences, HEC Montr´eal, Canada · 2Mila - Quebec AI Institute, Canada

Conditional-Value-At-Risk Soft Actor-Critic Model-Free

Apr 27, 2026cs.LG

Model-Free Inference of Investor Preferences: A Relative Entropy IRL Approach

We present a framework using Relative Entropy Inverse Reinforcement Learning (RE-IRL) to recover investor reward functions from observed investment actions and market conditions. Unlike traditional IRL algorithms, RE-IRL is employed to account for environments where transition probabilities are unknown or inaccessible. To address the challenge of data sparsity, we utilize a

K

-nearest neighbor approach to estimate the observed behavior policy. Furthermore, we propose a statistical testing framework to evaluate the validity and robustness of the estimated results.

Chen Xu

Department of Engineering, Shenzhen MSU-BIT University. Address: 1 International University Park Road, Longgang District, Shenzhen, Guangdong, China.

Reward Functions Model-Free Financial Strategy Research

Feb 26, 2026cs.AI

A Model-Free Universal AI

In general reinforcement learning, all established optimal agents, including AIXI, are model-based, explicitly maintaining and using environment models. This paper introduces Universal AI with Q-Induction (AIQI), the first model-free agent proven to be asymptotically

\varepsilon

-optimal in general RL. AIQI performs universal induction over distributional action-value functions, instead of policies or environments like previous works. Under a grain of truth condition, we prove that AIQI is strong asymptotically

\varepsilon

-optimal and asymptotically

\varepsilon

-Bayes-optimal. We also apply our novel proof techniques to show asymptotic

\varepsilon

-optimality of Self-AIXI without any ad-hoc assumptions. Our results significantly expand the diversity of known universal agents.

Yegon Kim, Juho Lee

Graduate School of AI, KAIST, Seoul, South Korea

Model-Free Offline Reinforcement Learning Artificial Intelligence Agents Artificial Intelligence Systems Induction Optimality

Sep 22, 2025eess.SY

Policy Gradient with Self-Attention for Model-Free Distributed Nonlinear Multi-Agent Games

Multi-agent games in dynamic nonlinear settings are challenging due to the time-varying interactions among the agents and the non-stationarity of the (potential) Nash equilibria. In this paper we consider model-free games, where agent transitions and costs are observed without knowledge of the transition and cost functions that generate them. We propose a novel distributed policy structure that follows the communication constraints in multi-team games, with multiple agents per team, and learned through policy gradients. Our formulation is inspired by the structure of distributed policies in linear quadratic games, which take the form of time-varying linear feedback gains. In the nonlinear case, we model the policies as nonlinear feedback gains, parameterized by self-attention layers to account for the time-varying multi-agent communication topology. We demonstrate that our approach achieves strong performance in several settings, including distributed linear and nonlinear regulation, and simulated and real multi-robot pursuit-and-evasion games.

Eduardo Sebastián, Maitrayee Keskar, Eeman Iqbal +3

Department of Computer Science and Technology, University of Cambridge, UK · Department of Electrical and Computer Engineering, University of California San Diego, USA · RoPeRt group, at DIIS - I3A, Universidad de Zaragoza, Spain

Multi-Agent Reinforcement Learning Policy Gradient Neural Policies Model-Free Nonlinearity Linear Attention

May 18, 2025cs.LG

Model-Free Robust Average-Reward Reinforcement Learning with Sample Complexity Analysis

Robust reinforcement learning (RL) under the average-reward criterion is essential for long-term decision-making, particularly when the environment may differ from its training dynamics. However, most existing studies focus on model-based settings and provide only asymptotic guarantees, hindering their principled understanding and practical deployment, especially in data-limited scenarios. We aim to close this gap by proposing a model-free algorithm, \textbf{Robust Halpern Iteration (RHI)}. We first design our algorithm based on a black-box sampling oracle, which can estimate the worst-case performance accurately. We then derive the finite sample complexity of RHI under the generative model setting, assuming the sampling oracle. To concretely design such an oracle, we propose a

K

-order multi-level Monte-Carlo estimator, which is shown to have a lower bias compared to prior methods. We further instantiate our design for multiple uncertainty models, including KL and

χ^2

divergence sets, and show that our RHI algorithm achieves an

\varepsilon

-optimal robust policy with a sample complexity of

\tilde{\mathcal{O}}\left( \frac{SA\mathcal{H}^2}{\varepsilon^{(2+o(1))}}\right)

, where

S,A

are the number of states and actions, and

\mathcal{H}

is the robust optimal span. Our result asymptotically matches the best complexity in robust average reward RL.

Zachary Roch, George Atia, Yue Wang

Department of Electrical and Computer Engineering, University of Central Florida, Orlando, Florida, USA. · Department of Computer Science, University of Central Florida, Orlando, Florida, USA.

Model-Free Offline Reinforcement Learning Optimal Policies Sample Complexity Newton