Offline reinforcement learning is fundamentally limited by the coverage of the available data, while collecting additional trajectories through online exploration may be costly, unsafe, or infeasible. We study whether this limitation can be overcome without directly interacting with the underlying environment. Motivated by practical settings such as collaborator-mediated simulation, where a collaborator can provide information complementary to the offline data, we introduce query-based collaborative reinforcement learning to study how to acquire this information efficiently. The learner starts with a fixed offline dataset and adaptively queries state-time pairs; the collaborator samples an action according to a fixed, possibly suboptimal policy, and the simulator returns the corresponding reward and next state. For finite-horizon tabular problems, we derive an instance-dependent information-theoretic lower bound on the additional queries required to identify an ε-optimal policy and develop a local quadratic approximation. Guided by this characterization, we develop an adaptive query-allocation algorithm whose query complexity nearly matches the local benchmark under suitable conditions. Experiments on multiple synthetic and semi-synthetic environments demonstrate the empirical efficiency of the proposed method.
Figures & tables
Figure 1: Query-based collaborative RL. The learner augments offline data with simulated experience, choosing state-time queries but not the actions sampled by the fixed-policy collaborator.
Symbol
Definition
Meaning
LM(X)
Law of a random variable X under M
Probability law
νhsaM
LM(Rh(s,a),Sh+1∣Sh=s,Ah=a)
Observation law
KLhsa(M,M)
KL(νhsaM∥νhsaM)
Local information
ΔH∣S∣
{λ∈R+H×∣S∣:\sumop\displaylimitsh,sλhs=1}
Query allocations
qhsaλ
λhsπhc(a∣s)
Sampling proportion
Jμ(M,M)
\sumop\displaylimitsh,s,adhsaμKLhsa(M,M)
Offline information
Table 1: Auxiliary notation used in Section 4 .
Figure 2: Simulation setups. a. Contextual bandit with reward gap Δ . b. Sequential decisions ( H=5 ) with transition probabilities under (a=0,a=1) . c. Long-term consequences (varying H ) where the value gap 0.02+(H−1)θ grows with the horizon. Blue marks the hard decisions.
Figure 3: Simulations 1–3 at n0=500 , 300 trials, and H=15 for Simulation 3 for the efficient implementation (solid) and the theory-aligned algorithm of Section 5 (dashed). a. Fraction of trials returning the optimal policy under fixed budget collaborator queries. b. Fraction of trials whose stopping rule has returned, each method under its own allocation. c. Allocation of the theory-aligned queries.
Figure 4: PRISM application over 400 trials under incidence (left) and equal (right) lineage weights. a. Fraction of trials returning an ε -optimal policy ( ε=0.01 ) under fixed budgets. b. Fraction of trials whose stopping rule has returned. c. Each method’s share of the theory-aligned version’s queries by lineage.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Quantity
Definition and role
δtrim
Failure probability used in the two-fold trimming step: δtrim:=δ/48 .
ℓtrim
Logarithmic factor used in the trimming bound: ℓtrim:=log(δtrim(n0∨1)H) .
wk
Fraction of the inference-side offline sample assigned to fold k : wk:=6/(π2k2) , with \sumop\displaylimitsk⩾1wk=1 .
D0G,D0,kD
Historical part of the trimmed offline sample and the k th fresh inference fold. The complete table of row counts N0,k,hsaD is recorded during preprocessing, while the corresponding reward–transition observations are not used for inference until epoch k .
A known lower bound on the minimum positive collaborator-action probability: 0<pmin⩽min(h,s,a)∈/Ucπhc(a∣s) .
Appendix
Table 2: Notation and constants used in Algorithm A.3 .
Figure 5: Simulation environments in full, in the style of Figure 2 . Circles are states, boxes are actions, and arrows are labelled with transition probabilities. a. Simulation 1: each context’s probability, the Bernoulli reward means of its two actions, and the gap Δs . Contexts 2-7 are identical and drawn once; Context 1 is drawn separately because the behavior policy selects its optimal action with probability 0.01 , compared with 0.20 elsewhere. b. Simulation 2: v and q have a single action; reward is 1 only at g , and q never reaches g . c. Simulation 3: the absorbing states g and b pay 1 and 0 at every step until the horizon H . Blue marks the hard decisions.
Figure 6: Mean collaborator queries until each method’s stopping rule returns (log scale; ±1.96 standard errors, mostly smaller than the markers), for every setting and offline sample size, under the efficient (filled) and theory-aligned (open) versions of the algorithm of Section 5 , both with C0=0.5 ; numbers below the markers give adaptive’s mean cost. No stopping run returned a policy that is not ε -optimal.
Figure 7: Simulation 1 at every offline sample size n0 (columns; 300 paired trials each), for the efficient (solid) and theory-aligned (dashed) versions of the algorithm of Section 5 , both with C0=0.5 . a. Fraction of trials returning the optimal policy and b. mean regret V1⋆−V1π of the returned policy at each epoch end, against mean collaborator queries (top: efficient; bottom: theory-aligned; the four rows of a column share the x -axis); offline-only levels outside the axis are printed in the panel. c. Fraction of trials whose stopping rule has returned, each method under its own allocation; a curve starts at the last epoch with no certified trial and ends once all have certified. d. Each method’s share of the efficient (top) and theory-aligned (bottom) version’s queries after that version’s last epoch before adaptive typically stops (epochs 5 and 7). Bands: pointwise 95% Wilson intervals (recovery, certification) and mean ±1.96 standard errors (regret).
Figure 8: Simulation 2 at every offline sample size; layout and conventions as in Figure 7 . All methods can query the nine state–time pairs. Query shares ( d. ) after epochs 6 (efficient) and 8 (theory-aligned).
Figure 9: Simulation 3 with H=5 at every offline sample size; layout and conventions as in Figure 7 . All methods can query the nine state–time pairs. Query shares ( d. ) after epochs 4 (efficient) and 7 (theory-aligned).
Figure 10: Simulation 3 with H=10 at every offline sample size; layout and conventions as in Figure 7 . All methods can query the 19 state–time pairs. Query shares ( d. ) after epochs 4 (efficient) and 7 (theory-aligned).
Figure 11: Simulation 3 with H=15 at every offline sample size; layout and conventions as in Figure 7 . All methods can query the 29 state–time pairs. Query shares ( d. ) after epochs 4 (efficient) and 7 (theory-aligned).
Figure 12: Mean collaborator queries until each method’s stopping rule returns with C0=1 ; layout and conventions as in Figure 6 . No stopping run returned a policy that is not ε -optimal.
Figure 13: Simulation 1 with C0=1 at every offline sample size; layout and conventions as in Figure 7 . Query shares ( d. ) after epochs 5 (efficient) and 7 (theory-aligned).
Figure 14: Simulation 2 with C0=1 at every offline sample size; layout and conventions as in Figure 7 . Query shares ( d. ) after epochs 6 (efficient) and 8 (theory-aligned).
Figure 15: Simulation 3 with H=5 and C0=1 at every offline sample size; layout and conventions as in Figure 7 . Query shares ( d. ) after epochs 4 (efficient) and 7 (theory-aligned).
Figure 16: Simulation 3 with H=10 and C0=1 at every offline sample size; layout and conventions as in Figure 7 . Query shares ( d. ) after epochs 4 (efficient) and 7 (theory-aligned).
Figure 17: Simulation 3 with H=15 and C0=1 at every offline sample size; layout and conventions as in Figure 7 . Query shares ( d. ) after epochs 4 (efficient) and 7 (theory-aligned).
Lines
Responders
Weight (%)
Cost of a wrong choice
Lineage
Eligible
Benchmark
D
S
Better
Gap
Incidence
Equal
Incidence
Equal
Lung
98
40
29
7
D
0.550
42.1
16.7
23.1
9.2
Skin
43
18
10
12
S
0.111
5.6
16.7
0.6
1.9
CNS
40
16
8
0
D
0.500
5.5
16.7
2.7
8.3
Ovary
35
14
14
2
D
0.857
5.5
16.7
4.7
14.3
Colorectal
34
14
7
8
S
0.071
32.7
16.7
2.3
1.2
Appendix
Table 3: The PRISM benchmark. Responders are benchmark lines with log2 fold change at most −1 at 2.5μ M; D: dasatinib, S: selumetinib. Weights are in percent. The cost of a wrong choice is the lineage’s weight times its gap, reported in percentage points of response rate ( ε=0.01 corresponds to one percentage point).
Figure 18: PRISM application at C0=0.5 (left pair of columns) and C0=1 (right pair), each under incidence and equal lineage weights (400 paired trials per column), for the efficient implementation (solid) and the theory-aligned algorithm of Section 5 (dashed). a. Fraction of trials returning an ε -optimal policy and b. mean regret of the returned policy, in response-rate units, at each epoch end, against mean collaborator queries (top: efficient; bottom: theory-aligned; the four rows of a column share the x -axis); offline-only levels outside the axis are printed in the panel. c. Fraction of trials whose stopping rule has returned, each method under its own allocation. d. Each method’s share of the efficient (top) and theory-aligned (bottom) version’s queries after that version’s last epoch before adaptive typically stops. Bands: pointwise 95% Wilson intervals (recovery, certification) and mean ±1.96 standard errors (regret).
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
We study federated reinforcement learning in which multiple agents interact with a common Markov decision process and communicate through a central server to collaboratively learn the optimal state-action value function. Our goal is to understand whether the sample-efficiency benefits of collaboration can be retained when a fraction of the agents behave adversarially and transmit arbitrarily corrupted information. To address this problem, we introduce Robust Async-Fed-Q, an epoch-based federated learning algorithm that combines variance-reduced estimation of the Bellman optimality operator at the agents with robust aggregation at the server. We establish high-probability finite-time guarantees showing that the proposed method preserves the statistical gains of collaboration among the honest agents while tolerating adversarial corruption. In particular, the effect of the adversarial agents decreases as the amount of data collected by each honest agent grows and eventually vanishes in the infinite-sample limit. We complement these guarantees with information-theoretic lower bounds that characterize the unavoidable statistical cost of adversarial corruption, leading to the first nearly matching upper and lower bounds for adversarially robust federated reinforcement learning. We further extend our framework to accommodate single-trajectory Markovian sampling and heterogeneous partial coverage, where different agents may explore different regions of the state-action space and learning relies on their collective coverage. Finally, our epoch-based design substantially improves the best known communication complexity for federated Q-learning under asynchronous sampling.
Sreejeet Maity, Aritra Mitra
Department of Electrical and Computer Engineering North Carolina State University, Raleigh
We consider a federated reinforcement learning setting involving M agents, all of whom interact with a common Markov Decision Process (MDP). The agents exchange information via a central server to learn the optimal value function. Our goal is to understand to what extent one can hope for collaborative sample-complexity speedups in such a setting, when a small fraction of the agents are adversarial and can act arbitrarily. To that end, we propose Robust Fed-Q}, a federated Q-learning algorithm that blends ideas from both model-based and model-free RL, along with the median-of-means device from robust statistics. We prove that despite corruption, with high-probability, Robust Fed-Q (i) guarantees exact convergence to the optimal value function in the limit of infinite samples, and (ii) enjoys near-optimal finite-time rates that benefit from collaboration. In addition, our approach requires just O~(1) rounds of communication to achieve each of the above guarantees, a feature of independent interest in FL where communication is the major bottleneck.
Sreejeet Maity, Aritra Mitra
Department of Electrical and Computer Engineering, North Carolina State University
Offline reinforcement learning (RL) aims to learn an effective policy from a static dataset, but its performance is fundamentally limited by dataset coverage. Action preference queries leverage expert feedback without additional environment interaction, enabling policy improvement during offline training. However, existing methods still face two key challenges: selecting informative preference queries and effectively exploiting the collected feedback. Current approaches typically rely only on the distance between policy actions and dataset actions for query selection, while enforcing fixed constraints that keep the policy close to queried preferences. Such strategies often lead to unstable policy updates and integrate poorly with value regularization. To address these limitations, we propose Conservative Query and Adaptive Regularization under Uncertainty Estimation, a lightweight framework that jointly improves preference querying and preference exploitation. Specifically, we employ a Morse network to estimate the uncertainty of policy actions with respect to the offline dataset. Based on this uncertainty, we introduce a conservative query strategy that selectively queries actions near the dataset to preserve Bellman-update stability, together with an uncertainty-aware adaptive regularization scheme that dynamically adjusts data-level constraints during policy optimization. We integrate our framework with CQL and evaluate it extensively on the D4RL benchmark. Experimental results demonstrate superior or competitive performance across a wide range of tasks.
Li-Rong Zhou, Qin-Wen Luo, Sheng-Jun Huang
Nanjing University of Aeronautics and Astronautics, Nanjing, China