Privacy-Preserving Prompted Policy Search for Robotic Control
Authors: Ali Irshayyid, Feng Lin, Chong Li, Jun Chen
Organizations: Department of Electrical and Computer Engineering, Oakland University, Rochester, MI 48309 USA · Department of Electrical and Computer Engineering, Wayne State University, Detroit, MI 48202, USA · OORT and the Department of Electrical Engineering, Columbia University, NYC, 10027, USA
Large language models (LLMs) have recently demonstrated promising capabilities as in-context policy optimizers for Reinforcement Learning (RL), enabling policy search driven by both numerical reward signals and natural language reasoning. However, deploying such methods in practice requires transmitting raw policy parameters and rewards history to cloud-based LLM APIs, exposing proprietary control strategies to third-party service providers. To address this issue, this paper introduces Privacy-Preserving Prompted Policy Search (PP-ProPS), a framework that enables LLM-guided policy optimization while keeping policy and environmental parameters confidential. PP-ProPS encodes policy parameters and reward values using secret client-side transformations before they are included in each API request, ensuring that the LLM provider observes only encoded policy parameters and scaled reward information. Furthermore, unlike Vanilla ProPS, the proposed framework does not require the true optimal episodic return to be known or disclosed to the LLM. Beyond protecting the optimization data, PP-ProPS improves the search process in two ways. First, it provides the LLM with individual reward components instead of only a single total return, offering more informative feedback about each candidate policy. Second, it uses a bounded history that prevents the prompt from growing indefinitely, improving search with high-dimensional policies and supporting the use of open-weight LLMs. The proposed PP-ProPS is evaluated on both continuous and discrete control problems spanning Multi-Joint dynamics with Contact (MuJoCo) locomotion, classic control, highway driving, and robotic arm manipulation. Compared to Vanilla ProPS, the proposed PP-ProPS outperforms ProPS in seven of the ten evaluated tasks, and surpasses conventional RL methods including PPO, SAC, and TRPO, in five of the six tasks.
Figures & tables
Fig. 1: Overview of the proposed PP-ProPS framework. The LLM searches in the encoded parameter space θ , while decoding, policy execution, reward scaling, and storage of the secret variables A and c are performed on the trusted client.
Fig. 2: Summary of the PP-ProPS prompt structure.
Fig. 3: Environments used to evaluate the proposed PP-ProPS framework.
Environment
Action type
na
ns
Detailed feedback
D
Ant-v5 (Ultra)
Continuous
8
10
Forward-progress reward, healthy reward, control-cost penalty, and contact-force penalty
88
Ant-v5 (Joints)
Continuous
8
16
Same as Ant-v5 (Ultra)
136
Ant-v5 (Torso)
Continuous
8
27
Same as Ant-v5 (Ultra)
224
Humanoid-v5 (Ultra)
Continuous
17
11
Forward-progress reward, healthy reward, control-cost penalty, and contact-force penalty
204
Humanoid-v5 (Joints)
Continuous
17
26
Same as Humanoid-v5 (Ultra)
459
Humanoid-v5 (Torso)
Continuous
17
45
Same as Humanoid-v5 (Ultra)
782
TABLE I: Evaluated environments, policy dimensions, and detailed feedback provided to the LLM
Fig. 4: Optimization performance of the evaluated ProPS configurations across two environments. Curves represent the mean episodic rewards and shaded regions indicate the standard deviation over five independent runs. Iterations preceding i=0 correspond to the 10-policy random warm-up phase.
Environment
Vanilla ProPS
R2PO
PE-ProPS
DF-ProPS
PP-ProPS
Ant-Ultra
942.75 (59.11)
Context overflow
998.64 (4.75)
998.9 (37.5)
994.55 (10.72)
Ant-Joints
966.58 (48.61)
Context overflow
982.62 (26.45)
996.15 (2.79)
989.68 (17.11)
Ant-Torso
929.19 (61.73)
Context overflow
982.41 (28.62)
993.76 (0.27)
997.08 (7.69)
Humanoid-Ultra
448.2 (51.65)
443.7 (80.5)
382.51 (73.79)
475.91 (72.61)
375.52 (96.64)
Humanoid-Joints
394.38 (90.88)
293.2 (77.1)
364.5 (94.03)
387 (69.3)
361.59 (97.2)
Humanoid-Torso
307.38 (63.7)
260.92 (20.74) †
251 (54.93)
326.43 (87.67)
290.68 (43.57)
TABLE II: Comparison of Episodic Rewards for Vanilla ProPS, R2PO, PE-ProPS, DF-ProPS, and PP-ProPS across all evaluated environments
Fig. 5: LLM response time for Ant-Ultra ( D=88 ) versus retained history size and the corresponding prompt size. Highlighted points denote bounded ( K=10 ) and unbounded history representations.
D
Bounded history ( K=10 )
Unbounded history
Diff (%)
88
942.75 (59.11)
950.75 (32.1)
−0.84
136
966.58 (48.61)
966.9 (15.7)
−0.03
224
929.19 (61.73)
932.7 (53.7)
−0.38
TABLE III: Comparison of the bounded and unbounded history representations across the Ant policy dimensions
Approach
Ant
Humanoid
Acrobot
CartPole
Highway
FetchReachDense
PPO
1000.16 (15.06)
192.94 (16.38)
-303.95 (170.33)
127.16 (179.77)
107.01 (26.51)
-7.65 (0.46)
A2C
233.61 (126.42)
118.82 (75.77)
-209.57 (158.84)
161.69 (177.61)
73.68(33.65)
-12.80 (8.19)
SAC
-52.42 (46.51)
78.24 (3.10)
–
–
–
-8.55 (0.99)
TD3
938.04 (240.49)
81.86 (34.78)
–
–
–
-8.06 (1.79)
DDPG
-6.55 (72.99)
89.59 (40.25)
–
–
–
-7.39 (2.45)
TRPO
965.23 (10.99)
193.21 (16.80)
-284.65 (171.60)
127.04 (187.54)
109.09 (30.69)
-8.76 (0.93)
TABLE IV: Performance Comparison of the Proposed PP-ProPS and Standard RL Baselines
We study when large language models (LLMs) can serve as effective black-box policy optimizers for reinforcement learning (RL) tasks, i.e., when can we replace classical RL algorithms with an LLM? We explore this question by introducing Prompted Policy Optimization (PromptPO), an iterative method that prompts an LLM with Python descriptions of the state space, action space, and reward function, then has it generate and refine executable policies based on rollout feedback. Across hard exploration environments, Meta-World robotics tasks, and several real-world control problems, PromptPO often matches or exceeds the performance of standard RL baselines while using substantially fewer environment interactions. To maximize expected return, and without further explicit prompting, the policies PromptPO outputs range from tuned proportional controllers or rule-based plans to policies that run planning algorithms like value iteration. Our results demonstrate that LLM-based policy optimization is sufficient when the LLM can leverage prior knowledge about the environment or optimization strategy. PromptPO underperforms standard RL baselines in MuJoCo domains. This demonstrates possible limitations of LLM-based policy optimization to settings that requiring fine-grained continuous control.
Stephane Hatgis-Kessell, Emma Brunskill
Department of Computer Science, Stanford University
Reinforcement Learning with Verifiable Rewards (RLVR) enhances Large Language Model (LLM) reasoning but is suffer from advantage collapse: when all rollouts of a query receive identical rewards, the group variance vanishes, most damagingly on hard samples, where scaling rollout budgets yields little. We introduce Joint Policy and Prompt Optimization (P O) to mitigate this collapse by alternating continuous policy updates with discrete prompt evolution. P O mines hard samples with a success-rate threshold, evolves reasoning prompts for them with GEPA, and internalizes the elicited trajectories via context distillation, which optimizes each trajectory under the original query and thus removes inference-time prompting, with a Context Ratio Mask (CRM) filtering out extreme likelihood ratios. P O restores critical advantage signals and surpasses the GRPO baseline by up to 8.2 points in average accuracy on six held-out benchmarks across all training datasets and backbones, while also outperforming DAPO and other baselines. The gains are especially pronounced on hard benchmarks, reaching up to 16.3 points above GRPO on average across AIME24 and AIME25. Our findings expose the limits of standard exploration in sparse-reward environments, illuminating the potential of unifying evolutionary algorithms with reinforcement learning. This integration of discrete semantic search and continuous parameter updates provides a self-reinforcing framework that facilitates more effective LLM alignment.
Xinyu Lu, Kaiqi Zhang, Jinglin Yang +6
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Institute of Information Engineering, Chinese Academy of Sciences +2
Exploration is essential to RL since a policy cannot improve by repeatedly sampling the behaviors it already prefers. Standard methods inject stochasticity in the action space, but such jitter only yields rollouts close to the original. Escaping a weak policy often requires global perturbations that action noise cannot produce. Large language models (LLMs) and vision-language-action (VLA) models offer a pathway: they condition the policy on a natural language prompt, and since the rollout follows from it, modifying the prompt induces global changes. The challenge is finding prompts that induce useful global changes. With a weak policy that rarely succeeds, reward is too sparse to select on. Our idea is to refine prompts from the rollouts themselves: a vision-language model (VLM) reasons over the rollout video, diagnoses how the policy responded, and rewrites the prompt to elicit better behavior next time. This procedure resembles posterior sampling, a classical RL exploration framework, at the level of prompts: the VLM maintains an implicit distribution over useful prompts and updates it from observed rollouts. We call this strategy Prompt-Driven Exploration (PDE). Across manipulation and reasoning tasks, PDE enables RL to learn successful policies even from zero-reward starts, and improves sample efficiency more broadly. Our website is available at https://xinyunsunshine.github.io/prompt-rl.
Sunshine Jiang, John Marangola, David Zhang +6
Massachusetts Institute of Technology · Improbable AI Lab · MIT-IBM Computing Research Lab