Ask to Learn: Query-Based Collaborative Reinforcement Learning

Kaicheng Zhang; Hongqian Niu; Laixi Shi; Didong Li; Guannan Qu; Zhun Deng

stat.MLOct 20, 2026

Ask to Learn: Query-Based Collaborative Reinforcement Learning

Authors: Kaicheng Zhang, Hongqian Niu, Laixi Shi, Didong Li, Guannan Qu, Zhun Deng

Organizations: University of North Carolina at Chapel Hill · Johns Hopkins University · Carnegie Mellon University

Abstract

Offline reinforcement learning is fundamentally limited by the coverage of the available data, while collecting additional trajectories through online exploration may be costly, unsafe, or infeasible. We study whether this limitation can be overcome without directly interacting with the underlying environment. Motivated by practical settings such as collaborator-mediated simulation, where a collaborator can provide information complementary to the offline data, we introduce query-based collaborative reinforcement learning to study how to acquire this information efficiently. The learner starts with a fixed offline dataset and adaptively queries state-time pairs; the collaborator samples an action according to a fixed, possibly suboptimal policy, and the simulator returns the corresponding reward and next state. For finite-horizon tabular problems, we derive an instance-dependent information-theoretic lower bound on the additional queries required to identify an $ε$ -optimal policy and develop a local quadratic approximation. Guided by this characterization, we develop an adaptive query-allocation algorithm whose query complexity nearly matches the local benchmark under suitable conditions. Experiments on multiple synthetic and semi-synthetic environments demonstrate the empirical efficiency of the proposed method.

Figures & tables

Figure 1: Query-based collaborative RL. The learner augments offline data with simulated experience, choosing state-time queries but not the actions sampled by the fixed-policy collaborator.

Symbol	Definition	Meaning
$\mathcal{L}_{\mathbf{\mathcal{M}}}(X)$	Law of a random variable $X$ under $\mathbf{\mathcal{M}}$	Probability law
$\nu_{hsa}^{\mathbf{\mathcal{M}}}$	$\mathcal{L}_{\mathbf{\mathcal{M}}}(R_{h}(s,a),S_{h+1}\mid S_{h}=s,A_{h}=a)$	Observation law
$\mathrm{KL}_{hsa}(\mathbf{\mathcal{M}},\widetilde{\mathbf{\mathcal{M}}})$	$\mathrm{KL}(\nu_{hsa}^{\mathbf{\mathcal{M}}}\\|\nu_{hsa}^{\widetilde{\mathbf{\mathcal{M}}}})$	Local information
$\Delta_{H\|\mathcal{S}\|}$	$\{\mathbf{\lambda}\in\mathbb R_{+}^{H\times\|\mathcal{S}\|}:\sumop\displaylimits_{h,s}\lambda_{hs}=1\}$	Query allocations
$q_{hsa}^{\mathbf{\lambda}}$	$\lambda_{hs}\pi_{h}^{c}(a\mid s)$	Sampling proportion
$J_{\mu}(\mathbf{\mathcal{M}},\widetilde{\mathbf{\mathcal{M}}})$	$\sumop\displaylimits_{h,s,a}d_{hsa}^{\mu}\mathrm{KL}_{hsa}(\mathbf{\mathcal{M}},\widetilde{\mathbf{\mathcal{M}}})$	Offline information

Table 1: Auxiliary notation used in Section 4 .

Figure 2: Simulation setups. a. Contextual bandit with reward gap

\Delta

. b. Sequential decisions (

H=5

) with transition probabilities under

(a=0,a=1)

. c. Long-term consequences (varying

H

) where the value gap

0.02+(H-1)\theta

grows with the horizon. Blue marks the hard decisions.

Figure 3: Simulations 1–3 at

n_{0}=500

, 300 trials, and

H=15

for Simulation 3 for the efficient implementation (solid) and the theory-aligned algorithm of Section 5 (dashed). a. Fraction of trials returning the optimal policy under fixed budget collaborator queries. b. Fraction of trials whose stopping rule has returned, each method under its own allocation. c. Allocation of the theory-aligned queries.

Figure 4: PRISM application over 400 trials under incidence (left) and equal (right) lineage weights. a. Fraction of trials returning an

\varepsilon

-optimal policy (

\varepsilon=0.01

) under fixed budgets. b. Fraction of trials whose stopping rule has returned. c. Each method’s share of the theory-aligned version’s queries by lineage.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Quantity	Definition and role
$\delta_{\rm trim}$	Failure probability used in the two-fold trimming step: $\delta_{\rm trim}:=\delta/48$ .
$\ell_{\rm trim}$	Logarithmic factor used in the trimming bound: $\ell_{\rm trim}:=\log\!\left(\frac{(n_{0}\vee 1)H}{\delta_{\rm trim}}\right)$ .
$w_{k}$	Fraction of the inference-side offline sample assigned to fold $k$ : $w_{k}:=6/(\pi^{2}k^{2})$ , with $\sumop\displaylimits_{k\geqslant 1}w_{k}=1$ .
$\mathcal{D}_{0}^{\mathcal{G}},\mathcal{D}_{0,k}^{\mathcal{D}}$	Historical part of the trimmed offline sample and the $k$ th fresh inference fold. The complete table of row counts $N_{0,k,hsa}^{\mathcal{D}}$ is recorded during preprocessing, while the corresponding reward–transition observations are not used for inference until epoch $k$ .
$\mathcal{U}_{c}$	Collaborator-unsupported rows: $\mathcal{U}_{c}:=\{(h,s,a):\pi_{h}^{c}(a\mid s)=0\}$ .
$p_{\min}$	A known lower bound on the minimum positive collaborator-action probability: $0<p_{\min}\leqslant\min_{(h,s,a)\notin\mathcal{U}_{c}}\pi_{h}^{c}(a\mid s)$ .

Appendix

Table 2: Notation and constants used in Algorithm A.3 .

Figure 5: Simulation environments in full, in the style of Figure 2 . Circles are states, boxes are actions, and arrows are labelled with transition probabilities. a. Simulation 1: each context’s probability, the Bernoulli reward means of its two actions, and the gap

\Delta_{s}

. Contexts 2-7 are identical and drawn once; Context 1 is drawn separately because the behavior policy selects its optimal action with probability

0.01

, compared with

0.20

elsewhere. b. Simulation 2:

v

and

q

have a single action; reward is 1 only at

g

, and

q

never reaches

g

. c. Simulation 3: the absorbing states

g

and

b

pay 1 and 0 at every step until the horizon

H

. Blue marks the hard decisions.

Figure 6: Mean collaborator queries until each method’s stopping rule returns (log scale;

\pm 1.96

standard errors, mostly smaller than the markers), for every setting and offline sample size, under the efficient (filled) and theory-aligned (open) versions of the algorithm of Section 5 , both with

C_{0}=0.5

; numbers below the markers give adaptive’s mean cost. No stopping run returned a policy that is not

\varepsilon

-optimal.

Figure 7: Simulation 1 at every offline sample size

n_{0}

(columns; 300 paired trials each), for the efficient (solid) and theory-aligned (dashed) versions of the algorithm of Section 5 , both with

C_{0}=0.5

. a. Fraction of trials returning the optimal policy and b. mean regret

V^{\star}_{1}-V^{\widehat{\pi}}_{1}

of the returned policy at each epoch end, against mean collaborator queries (top: efficient; bottom: theory-aligned; the four rows of a column share the

x

-axis); offline-only levels outside the axis are printed in the panel. c. Fraction of trials whose stopping rule has returned, each method under its own allocation; a curve starts at the last epoch with no certified trial and ends once all have certified. d. Each method’s share of the efficient (top) and theory-aligned (bottom) version’s queries after that version’s last epoch before adaptive typically stops (epochs 5 and 7). Bands: pointwise 95% Wilson intervals (recovery, certification) and mean

\pm 1.96

standard errors (regret).

Figure 8: Simulation 2 at every offline sample size; layout and conventions as in Figure 7 . All methods can query the nine state–time pairs. Query shares ( d. ) after epochs 6 (efficient) and 8 (theory-aligned).

Figure 9: Simulation 3 with

H=5

at every offline sample size; layout and conventions as in Figure 7 . All methods can query the nine state–time pairs. Query shares ( d. ) after epochs 4 (efficient) and 7 (theory-aligned).

Figure 10: Simulation 3 with

H=10

at every offline sample size; layout and conventions as in Figure 7 . All methods can query the 19 state–time pairs. Query shares ( d. ) after epochs 4 (efficient) and 7 (theory-aligned).

Figure 11: Simulation 3 with

H=15

at every offline sample size; layout and conventions as in Figure 7 . All methods can query the 29 state–time pairs. Query shares ( d. ) after epochs 4 (efficient) and 7 (theory-aligned).

Figure 12: Mean collaborator queries until each method’s stopping rule returns with

C_{0}=1

; layout and conventions as in Figure 6 . No stopping run returned a policy that is not

\varepsilon

-optimal.

Figure 13: Simulation 1 with

C_{0}=1

at every offline sample size; layout and conventions as in Figure 7 . Query shares ( d. ) after epochs 5 (efficient) and 7 (theory-aligned).

Figure 14: Simulation 2 with

C_{0}=1

at every offline sample size; layout and conventions as in Figure 7 . Query shares ( d. ) after epochs 6 (efficient) and 8 (theory-aligned).

Figure 15: Simulation 3 with

H=5

and

C_{0}=1

at every offline sample size; layout and conventions as in Figure 7 . Query shares ( d. ) after epochs 4 (efficient) and 7 (theory-aligned).

Figure 16: Simulation 3 with

H=10

and

C_{0}=1

at every offline sample size; layout and conventions as in Figure 7 . Query shares ( d. ) after epochs 4 (efficient) and 7 (theory-aligned).

Figure 17: Simulation 3 with

H=15

and

C_{0}=1

at every offline sample size; layout and conventions as in Figure 7 . Query shares ( d. ) after epochs 4 (efficient) and 7 (theory-aligned).

	Lines		Responders				Weight (%)		Cost of a wrong choice
Lineage	Eligible	Benchmark	D	S	Better	Gap	Incidence	Equal	Incidence	Equal
Lung	98	40	29	7	D	0.550	42.1	16.7	23.1	9.2
Skin	43	18	10	12	S	0.111	5.6	16.7	0.6	1.9
CNS	40	16	8	0	D	0.500	5.5	16.7	2.7	8.3
Ovary	35	14	14	2	D	0.857	5.5	16.7	4.7	14.3
Colorectal	34	14	7	8	S	0.071	32.7	16.7	2.3	1.2

Appendix

Table 3: The PRISM benchmark. Responders are benchmark lines with

\log_{2}

fold change at most

-1

2.5\,\mu

M; D: dasatinib, S: selumetinib. Weights are in percent. The cost of a wrong choice is the lineage’s weight times its gap, reported in percentage points of response rate (

\varepsilon=0.01

corresponds to one percentage point).

Figure 18: PRISM application at

C_{0}=0.5

(left pair of columns) and

C_{0}=1

(right pair), each under incidence and equal lineage weights (400 paired trials per column), for the efficient implementation (solid) and the theory-aligned algorithm of Section 5 (dashed). a. Fraction of trials returning an

\varepsilon

-optimal policy and b. mean regret of the returned policy, in response-rate units, at each epoch end, against mean collaborator queries (top: efficient; bottom: theory-aligned; the four rows of a column share the

x

-axis); offline-only levels outside the axis are printed in the panel. c. Fraction of trials whose stopping rule has returned, each method under its own allocation. d. Each method’s share of the efficient (top) and theory-aligned (bottom) version’s queries after that version’s last epoch before adaptive typically stops. Bands: pointwise 95% Wilson intervals (recovery, certification) and mean

\pm 1.96

standard errors (regret).

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,742 papers

Learning from Unreliable Trajectories: Adversarially-Robust Federated Q-Learning
Oct 2, 2026Sreejeet Maity, Aritra MitraFederated RL Byzantine-Robust Federated Learning
Department of Electrical and Computer Engineering North Carolina State University, Raleigh
Robust Federated Q-Learning with Almost No Communication
Aug 24, 2026Sreejeet Maity, Aritra MitraFederated RL Q-Learning
Department of Electrical and Computer Engineering, North Carolina State University
Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation
Jul 21, 2026Li-Rong Zhou, Qin-Wen Luo, Sheng-Jun HuangReinforcement Learning Offline RL
Nanjing University of Aeronautics and Astronautics, Nanjing, China

Oct 2, 2026J/K move · Enter open · S save

Learning from Unreliable Trajectories: Adversarially-Robust Federated Q-Learning

Sreejeet Maity, Aritra Mitra

Department of Electrical and Computer Engineering North Carolina State University, Raleigh

Federated RL Byzantine-Robust Federated Learning Asynchronous Federated Learning

Open paper AskPDF arXiv

We study federated reinforcement learning in which multiple agents interact with a common Markov decision process and communicate through a central server to collaboratively learn the optimal state-action value function. Our goal is to understand whether the sample-efficiency benefits of collaboration can be retained when a fraction of the agents behave adversarially and transmit arbitrarily corrupted information. To address this problem, we introduce Robust Async-Fed-Q, an epoch-based federated learning algorithm that combines variance-reduced estimation of the Bellman optimality operator at the agents with robust aggregation at the server. We establish high-probability finite-time guarantees showing that the proposed method preserves the statistical gains of collaboration among the honest agents while tolerating adversarial corruption. In particular, the effect of the adversarial agents decreases as the amount of data collected by each honest agent grows and eventually vanishes in the infinite-sample limit. We complement these guarantees with information-theoretic lower bounds that characterize the unavoidable statistical cost of adversarial corruption, leading to the first nearly matching upper and lower bounds for adversarially robust federated reinforcement learning. We further extend our framework to accommodate single-trajectory Markovian sampling and heterogeneous partial coverage, where different agents may explore different regions of the state-action space and learning relies on their collective coverage. Finally, our epoch-based design substantially improves the best known communication complexity for federated Q-learning under asynchronous sampling.

Abstract

Reinforcement Learning

Explore similar work

Learning from Unreliable Trajectories: Adversarially-Robust Federated Q-Learning

Robust Federated Q-Learning with Almost No Communication

Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation