The Trajectory Alignment Coefficient in Two Acts: From Reward Tuning to Reward Learning
Organizations: Sony AI · University of Alberta · The University of Texas at Austin
Abstract
The success of RL is fundamentally tied to having a reward function that accurately reflects the task objective. Yet, designing reward functions is time-consuming and prone to misspecification. To address this, our first goal is to understand how to support RL practitioners in specifying reward functions. We leverage the Trajectory Alignment Coefficient (TAC), a metric that evaluates how closely a reward function's induced preferences match those of a domain expert. To test whether TAC provides meaningful support, we evaluated its impact in a user study where practitioners designed reward functions for Lunar Lander. We found that iterative TAC feedback allowed participants to design more performant reward functions while reporting lower cognitive workload than unguided reward design. However, the study also underscored that manual reward design, even with TAC, remains labor-intensive. This motivated our second goal: to learn a reward model that maximizes TAC directly. Specifically, we propose Soft-TAC, a differentiable approximation of TAC that can be used as a loss function to train reward models from human preferences. Our theoretical analysis guarantees that under Soft-TAC optimization: (1) the optimal reward function belongs exclusively to the set of human-aligned reward functions, and (2) this optimal reward remains invariant whether trained on clean or noisy preference data. Empirically, we first evaluate reward models trained with Soft-TAC across three robotic tasks, demonstrating that they consistently match or outperform those trained with the Cross-Entropy loss. Next, we evaluate Soft-TAC in the GT7 driving simulator, and find that Soft-TAC-trained reward models more successfully capture preference-specific driving styles. Ultimately, this work demonstrates that TAC can serve as both a practical tool for guiding reward design and a reward learning objective in complex domains.
Figures & tables
| Method | BIAI | Lap | Incomplete |
| Ratio | Time | Laps | |
| Soft- | 0.971 0.003 | 270.72 0.73 | 1.17 0.40 |
| TAC | |||
| Cross- | 0.996 0.001 | 277.71 0.31 | 11.00 1.95 |
| Entropy | |||
| Default | 0.991 0.001 | 276.03 0.21 | 5.00 1.44 |
Appendix figures & tables37 assets
Supplementary material from the paper’s appendix.
Appendix
| Metric | P1 | P2 | P3 | P4 | P5 | P6 |
| Control Condition | ||||||
| Reward Weights | ||||||
| Distance | -1.0 | -0.4 | -55.5 | 1.0 | -2000.0 | – |
| Speed | 0.0 | -0.5 | 0.0 | 1.0 | -10.0 | – |
| Tilt | -1.0 | -0.4 | 0.0 | 0.0 | -10.0 | – |
| Leg Contact | 1.0 | 0.0 | 10.0 | 1.0 | 0.0 | – |
| Alignment | Control | Alignment vs. Control | ||||
| Metric | Mean SE | Mean SE | ||||
| Landing-pad success rate | 0.0832 | 0.0001 | 0.0416 | |||
| Alignment | Control | Alignment vs. Control | ||||
| Metric | Mean SE | Mean SE | Statistic | |||
| Experience | 0.2117 | 0.4399 | 0.7905 | |||
| Average Workload | 0.8304 | 0.6557 | 0.0178 | |||
| Reward Part | Versus | Time-Trial |
| Course Progress | 1.0 | 1.0 |
| Velocity-based Collision | -0.5 | NA |
| Overtaking Attempts | 0.5 | NA |
| Participant | Original # Ratings | # Binary Preference Pairs |
| Reacher | ||
| P1 | 159 | 9,195 |
| P2 | 159 | 10,103 |
| P3 | 155 | 8,880 |
| P4 | 159 | 9,781 |
| P5 | 159 | 9,482 |
| Hyperparameter / Setting | GT7 (Aggressive–Timid) | GT7 (Fast Driving) |
| Model Type | Linear | Linear |
| Weight Initialization | ||
| Weight Clipping Bounds | ||
| Learning Rate Grid | ||
| Batch Size Grid | ||
| Max Epochs | 500 | 500 |
| Hyperparameter / Setting | Lunar Lander |
| Model Type | Linear |
| Weight Initialization | |
| Weight Clipping Bounds | |
| Learning Rate Grid | |
| Batch Size Grid | |
| Max Epochs | 5000 |
| Hyperparameter / Setting | Reacher | Hopper |
| Model Type | Neural Network Ensemble | Neural Network Ensemble |
| Ensemble Size ( ) | 3 | 3 |
| Hidden Layers | 3 | 3 |
| Hidden Units per Layer | 128 | 128 |
| Hidden Layer Activation | LeakyReLU | LeakyReLU |
| Output Layer Activation | Tanh | Tanh |
| Method | Learning Rate | Batch Size |
| Lunar Lander | ||
| Soft-TAC | 0.05 | 16 |
| Cross-Entropy | 0.05 | 8 |
| GT Fast Driving | ||
| Soft-TAC | 0.05 | 8 |
| Cross-Entropy | 0.03 | 8 |
| Method | Participant | Learning Rate | Batch Size |
| Cross-Entropy | |||
| Cross-Entropy | P1 | 0.0001 | 512 |
| Cross-Entropy | P2 | 0.0010 | 256 |
| Cross-Entropy | P3 | 0.0010 | 128 |
| Cross-Entropy | P4 | 0.0001 | 512 |
| Cross-Entropy | P5 | 0.0010 | 128 |
| Method | Participant | Learning Rate | Batch Size |
| Cross-Entropy | |||
| Cross-Entropy | P1 | 0.0001 | 256 |
| Cross-Entropy | P2 | 0.0010 | 256 |
| Cross-Entropy | P3 | 0.0001 | 512 |
| Cross-Entropy | P4 | 0.0001 | 512 |
| Cross-Entropy | P5 | 0.0001 | 256 |
| Distance | Speed | Tilt | Leg Contact | Side Engine | Main Engine | Crash | Safe Landing | TAC | |
| Soft-TAC | |||||||||
| -6.34 | 0.16 | -18.02 | 4.81 | -0.26 | 0.18 | -32.92 | 99.29 | 0.88 | |
| -4.58 | 0.84 | -9.00 | 4.19 | -0.18 | 0.14 | -25.38 | 37.63 | 0.86 | |
| -6.03 | -1.81 | 0.08 | 3.95 | -0.07 | -0.08 | -15.42 | 22.79 | 0.85 | |
| -10.79 | -3.74 | -3.95 | -0.72 | -0.29 | 0.07 | -27.93 | 60.92 | 0.84 | |
| -4.98 | -1.35 | -3.95 | 4.74 | -0.08 | -0.03 | -15.98 | 24.70 | 0.84 | |
| Experiment | Method | Overtaking | Collision | TAC |
| Aggressive | Soft-TAC | 2.73 | 0.05 | 0.77 |
| 2.80 | 0.09 | 0.77 | ||
| 2.75 | 0.03 | 0.77 | ||
| 2.76 | 0.00 | 0.77 | ||
| 2.82 | 0.10 | 0.77 | ||
| 2.81 | 0.09 | 0.77 |
| Method | Progress | TAC |
| Soft-TAC | 7.98 | 0.97 |
| 7.23 | 0.97 | |
| 7.62 | 0.97 | |
| 8.18 | 0.97 | |
| 3.94 | 0.97 | |
| 8.20 | 0.97 |
| Hyperparameter | GT7 | Lunar Lander |
| RL Algorithm | QR-SAC ( Wurman et al. 2022 ) | SAC ( Haarnoja et al. 2018 ) |
| Optimizer | Adam ( Kingma and Ba 2015 ) | Adam |
| Number of Quantiles | 32 | NA |
| Number of Critic Networks | 2 | 2 |
| Number of Target Critic Networks | 2 | 2 |
| Number of Policy Networks | 1 | 1 |
| Hyperparameter | Reacher | Hopper |
| RL Algorithm | SAC ( Haarnoja et al. 2018 ) | SAC ( Haarnoja et al. 2018 ) |
| Optimizer | Adam ( Kingma and Ba 2015 ) | Adam ( Kingma and Ba 2015 ) |
| Optimizer Betas ( ) | (0.9, 0.999) | (0.9, 0.999) |
| Number of Critic Networks | 2 | 2 |
| Number of Target Critic Networks | 2 | 2 |
| Number of Policy Networks | 1 | 1 |
| Environment | Evaluation Seeds | Eval Episodes / Policy | Eval Policies (Episode Checkpoints) |
| GT7 (Timid Driving) | 6 | 1 | N/A (special policy selection) |
| GT7 (Fast Driving) | 6 | 1 | N/A (special policy selection) |
| GT7 (Aggressive Driving) | 9 | 1 | N/A (special policy selection) |
| Lunar Lander | 10 | 5 | 5 (Epochs 8000, 8500, 9000, 9500, 10000) |
| Lunar Lander (User Study) | 5 | 5 | 5 (Epochs 8000, 8500, 9000, 9500, 10000) |
| Reacher | 10 | 10 | 10 (Last 10 Epochs: 4910–5000, step 10) |
| ST | CE | ST vs. CE | |||||
| Metric | Mean SE | Mean SE | Stat | Raw | |||
| Landing-pad success rate | 0.006 | 0.000 | 0.287 | 0.287 | |||
| Outside landing success rate | 0.000 | 1.000 | 0.039 | 0.078 | |||
| ST | Tuned Condition | ST vs. Tuned | |||||
| Comparison | Mean SE | Mean SE | Stat | Raw | |||
| ST vs. Alignment | 0.006 | 0.083 | 0.412 | 0.412 | |||
| ST vs. Control | 0.006 | 0.000 | 0.019 | 0.038 | |||
| Final Return (Mean SE) | ||
| Participant | Cross-Entropy (CE) | Soft-TAC (ST) |
| Reacher | ||
| P1 | ||
| P2 | ||
| P3 | ||
| P4 | ||
| Environment | Normality ( ) | Test Statistic | -value |
| Reacher ( ) | 0.042 | 0.055 | |
| Hopper ( ) | 0.083 | 0.401 |
| Descriptive Statistics (Mean SE) | Normality Tests ( ) | |||||
| Metric | CE | ST | Default | CE | ST | Default |
| Lap Time (s) | 0.820 | 0.001 | 0.047 | |||
| BIAI Ratio | 0.562 | 0.001 | 0.117 | |||
| Incomplete Laps | 0.596 | 0.031 | 0.489 | |||
| ST vs. CE | ST vs. Default | |||||
| Metric | Stat | Raw | Stat | Raw | ||
| Descriptive Statistics (Mean SE) | Normality Tests ( ) | |||||
| Metric | CE | ST | Default | CE | ST | Default |
| Overtaking Attempts | 0.693 | 0.240 | 0.184 | |||
| Collisions | 0.724 | 0.869 | 0.232 | |||
| Off-Track Instances | 0.188 | 0.328 | 0.048 | |||
| Final Place | 0.850 | 0.904 | 0.406 | |||
| ST vs. CE | ST vs. Default | |||||
| Descriptive Statistics (Mean SE) | Normality Tests ( ) | |||||
| Metric | CE | ST | Default | CE | ST | Default |
| Overtaking Attempts | 0.979 | 0.330 | 0.635 | |||
| Collisions | 0.369 | 0.705 | 0.725 | |||
| Off-Track Instances | 0.020 | 0.136 | 0.463 | |||
| Final Place | 0.949 | 0.573 | 0.775 | |||
| ST vs. CE | ST vs. Default | |||||
| Descriptive Statistics (Mean SE) | Normality Tests ( ) | ST Aggressive vs. ST Timid | |||||
| Metric | ST Aggressive | ST Timid | ST Aggressive | ST Timid | Stat | Raw | |
| Overtaking Attempts | 0.240 | 0.330 | 0.002 | 0.002 | |||
| Collisions | 0.869 | 0.705 | |||||
| Off-Track Instances | 0.328 | 0.136 | |||||
| Final Place | 0.904 | 0.573 | 0.004 | 0.004 | |||
| Method | BIAI | Lap | Lap Time |
| Ratio | Time | 1800 | |
| Soft- | 0.980 0.00 | 135.85 0.20 | 0 |
| TAC | |||
| Cross- | 0.990 0.00 | 137.39 0.31 | 15 |
| Entropy | |||
| Default | 0.983 0.00 | 136.33 0.11 | 1 |
Reward Design for RL
Reward design for reinforcement learning studies how to specify, learn, and adapt signals that guide agents toward intended behavior.
See all 289 papers