TRR: A Thermal Interaction Engine for Robot Simulation and RGBT Data Generation
Organizations: Beijing Institute of Technology
Abstract
A robot's actions change the thermal conditions of its next interaction: heat retained after release, carried with objects or stored beneath the surface can change a later contact's outcome despite similar current temperatures. Training trajectories must capture how these effects connect earlier actions to later observations and task outcomes. We present TRR, a Thermal Robot Renderer that attaches spatial and depth-resolved thermal state to both contact partners, updates local heat exchange through contact, transport and separation, and generates synchronized RGB--thermal (RGBT) observations and event-local task checks on one execution clock. Controlled interventions show why retained heat, spatial transport and internal storage matter for subsequent contact outcomes. From the same state, TRR produces a paired six-task benchmark of 2,224 static--dynamic pairs, that is, 4,448 trajectories with synchronized dual-view observations, states and actions. Visible thermal history raises safe contact-site selection from 75.0% to 91.7% on a dedicated learning corpus, and training on dynamic histories raises simulated transfer completion by 50 percentage points under a new heating procedure with shared feedback execution. We demonstrate RGBT observation adapters on MuJoCo and Isaac Sim, report an engineering inventory of ten prepared robot configurations and six demonstrated camera viewpoints, and describe a developer workflow that carries an existing robot scene to exported RGBT records. The benchmark, the workflow and the platform resources described in Section 6 form the first items of a planned public release. Demonstrations and resource updates are available on the project page.
Figures & tables
| Task | Pairs | Pass | Fail | Unresolved |
|---|---|---|---|---|
| Cooler mug | 555 | 555 | 0 | 0 |
| Hotter mug | 300 | 300 | 0 | 0 |
| Bottle, cold | 225 | 225 | 0 | 0 |
| Bottle, hot | 425 | 200 | 225 | 0 |
| Sugar box, cold | 300 | 264 | 0 | 36 |
| Sugar box, hot | 419 | 191 | 219 | 9 |
| Input / model | Safe (%) | Site match (%) |
|---|---|---|
| Full RGBT sequence | 91.7 | 80.8 |
| Final-frame repeat | 75.0 | 62.7 |
| Frozen input | 75.0 | 62.7 |
| Per-frame spatial mean | 72.7 | 60.2 |
| RGB only | 72.5 | 60.0 |
| Global-mean GRU | 62.5 | 50.0 |
| Training input | Seed | Same heating procedure | New heating procedure |
|---|---|---|---|
| Dynamic (D) | 0–2 | 12 | 2 | 2 | 13 | 1 | 2 |
| Static (S) | 0–2 | 6 | 9 | 1 | 5 | 9 | 2 |
| Visible-frame repeat (C) | 0, 1 | 12 | 2 | 2 | 13 | 1 | 2 |
| Visible-frame repeat (C) | 2 | 12 | 2 | 2 | 11 | 3 | 2 |
| Record | Key fields | Content |
|---|---|---|
| Observation | observation time, camera id, view | RGB frame, camera temperature matrix , grayscale image |
| Robot state | observation time | robot state |
| Action interval | start and end time | first control target after the observation, with the full native interval retained |
| Event check | event time, check type, region | object, region, timing or mechanical outcome: pass, fail or unresolved |
| Trajectory | trajectory id, task, program | task label from the checks, contact channels and durations |
| Resource | Inventory | Role and examples |
|---|---|---|
| Robot configurations | 10 configurations | Articulated scene setups, including Piper, Panda, FR3 and xArm7. |
| Thermal caches | 16 categories | Nine robot caches, four object caches and three environment caches. |
| Measured robot asset | One Piper asset | Piper with D435 camera and mounting bracket; demonstrated reuse across ten poses. |
| Simulation adapters | Two backends | MuJoCo and Isaac Sim, with RGB and thermal observation examples. |
| Task scene families | Three families | Cup, sugar box and bottle, covering six hot/cold task conditions. |
| Camera placements | Six demonstrated views | Three external and three wrist-mounted viewpoints with RGBT output. |
| Demonstrated capability | Use it enables |
|---|---|
| Heat is retained on both contact partners through separation | Study how repeated grasping and recontact change object and support temperatures, and why an equally warm object can meet a differently warm surface. |
| Material regions move with the object | Simulate the local thermal condition of different grasp sites and predict which arriving region meets the next interface. |
| Internal storage behind a similar surface | Treat the visible surface as incomplete evidence: two objects at the same surface temperature can lead to different contact peaks and different safety outcomes. |
| Identical motions with different task labels | Construct and analyze data under dynamic thermal constraints; 444 of 2,224 paired motions move from pass to violation as heat evolves. |
| Thermal history improves contact-site choice | Quantify the value of temporal thermal observations for manipulation decisions, with a matched final-frame control. |
| Dynamic data improves simulated transfer | Provide evidence that dynamic thermal training data changes execution outcomes under a shared feedback executor. |
| Resource | Technical status | Access status | Content and supporting evidence |
|---|---|---|---|
| Project demonstrations | Published | Public | Preview GIF and overview images on the project page and repository. |
| Paired RGBT benchmark | Demonstrated | Internal | 2,224 source-condition pairs, 4,448 trajectories and 447,318 synchronized dual-view instants across six tasks (Table 1 ). |
| Contact-site learning corpus | Reported study | Internal | 240 source groups in four contact-history families; primary comparison on 128 held-out decisions in 64 groups. |
| Regional-decision transfer corpus | Reported study | Internal | Regional choices with shared feedback execution, 16 test conditions per heating track and three initializations. |
| Simulation adapters | Demonstrated | Internal | MuJoCo and Isaac Sim RGBT output (Figure 3 ). |
| Robot configurations | Prepared | Internal | Ten configurations, including Piper, Panda, FR3 and xArm7. |
| Measured cost of one coupled run | |
|---|---|
| Thermal nodes | 165,997 |
| Update steps | 885 |
| Simulated time | 115.5 s |
| Coupled payload wall-clock | 291.1 s |
| Contact-off wall-clock | 292.4 s |
| Cup sweep wall-clock | 294.2 s (mean of six runs) |
Appendix figures & tables32 assets
Supplementary material from the paper’s appendix.
Appendix
| Workflow | Thermal state | Interaction coupling | IR output | Measured evidence | Robot tasks |
|---|---|---|---|---|---|
| iGibson 2.0 [ 24 ] | Object states | Temperature-dependent state changes | — | — | |
| BEHAVIOR-1K / OmniGibson [ 25 ] | Object temperature | Source and ambient exchange | — | — | |
| HAZARD [ 57 ] | Object temperature | Evolving fire and object temperatures | — | — | |
| RFUniverse [ 8 ] | Particle material state | Contact-dependent material change | — | — | |
| Tactile Genesis [ 7 ] | Link temperature voxels | Sensing voxels coupled to an opposing link scalar | — | — | |
| Stonefish [ 11 ] | Prescribed surface texture | — | Thermal-camera image | — |
| Stage / parameter | Setting |
|---|---|
| Graph-field fitting | |
| Graph data loss | Huber threshold C; four reweighting iterations |
| Graph reference length | mm |
| Graph regularization | Matrix strength ; energy coefficients in Eq. ( 25 ) are |
| Prior / gradient anchor | with at least three supporting pose groups, otherwise 0.15; gradient anchor matrix strength |
| Surface node spacing | 3 mm on detail surfaces; 4.5 mm on the D435 main housing; reference link-2 main field uses 6 mm |
| True emissivity | 0.80 | 0.85 | 0.90 | 0.95 | 1.00 |
| Shell temperature at trigger ( ∘ C) | 41.73 | 40.82 | 40.00 | 39.26 | 38.59 |
| Uncorrected camera | Corrected camera | ||||
|---|---|---|---|---|---|
| Initial shell ( ∘ C) | State-trigger time (s) | Delay (s) | Shell at start ( ∘ C) | Delay (s) | Shell at start ( ∘ C) |
| 60 | 122.8 | -17.3 | 41.49 | +0.2 | 39.98 |
| 70 | 168.8 | -18.8 | 41.53 | +0.2 | 39.98 |
| 80 | 206.7 | -19.2 | 41.50 | +0.3 | 39.98 |
| Reference | Resolution | Max. error ( ∘ C) |
|---|---|---|
| Contact exchange | s | 0.2531 |
| Contact exchange | s | 0.1253 |
| Contact exchange | s | 0.0624 |
| Slab diffusion | 8 cells | 0.0434 |
| Slab diffusion | 16 cells | 0.0112 |
| Slab diffusion | 32 cells | 0.0030 |
| Pair | Target region | Initial / frozen field | At contact | After hold | Change from initial |
|---|---|---|---|---|---|
| 1 | Cooler | 9.01 | 8.85 | 7.58 | |
| 2 | Cooler | 7.47 | 7.33 | 6.29 | |
| 3 | Cooler | 6.76 | 6.64 | 5.69 | |
| 4 | Warmer | 3.85 | 3.79 | 3.25 | |
| 5 | Warmer | 3.25 | 3.20 | 2.74 |
| Task outcomes (count) | Component passes | ||||||
| Input | Seed | Completed | Failed | Unresolved | Success bounds (%) | Mechanical | Correct object |
| Same heating procedure; held-out placements | |||||||
| D | 0 | 12 | 2 | 2 | 14/16 | 14/16 | |
| D | 1 | 12 | 2 | 2 | 14/16 | 14/16 | |
| D | 2 | 12 | 2 | 2 | 14/16 | 14/16 | |
| S | 0 | 6 | 9 | 1 | 14/16 | 14/16 | |
| Apparent temperature ( ∘ C) | Display grayscale (0–255) | ||||||
|---|---|---|---|---|---|---|---|
| Pose | Mask coverage (%) | MAE | RMSE | Bias | MAE | RMSE | Bias |
| 1 | 58.88 | 1.528 | 2.179 | +0.480 | 45.76 | 52.91 | -40.77 |
| 2 | 58.08 | 1.546 | 2.270 | +0.682 | 48.15 | 54.97 | -42.27 |
| 3 | 56.54 | 1.475 | 2.213 | +0.713 | 50.73 | 57.11 | -44.71 |
| 4 | 55.03 | 1.583 | 2.508 | +0.938 | 59.30 | 65.64 | -51.91 |
| 5 | 55.54 | 1.279 | 2.099 | +0.802 | 58.74 | 64.65 | -54.13 |
| Initialization depth | Layered TRR | Scalar cooling | Frozen field |
|---|---|---|---|
| 1 mm | 1.33 | 1.30 | 2.05 |
| 2 mm (nominal) | 0.97 | 0.93 | 2.05 |
| 4 mm | 0.52 | 0.45 | 2.05 |
| Full RGBT sequence | Final-frame repeat | |||
|---|---|---|---|---|
| Seed | Safe (%) | Site match (%) | Safe (%) | Site match (%) |
| 0 | 95.31 | 92.19 | 75.00 | 62.50 |
| 1 | 94.53 | 82.03 | 75.00 | 62.50 |
| 2 | 92.97 | 79.69 | 75.00 | 62.50 |
| 3 | 88.28 | 75.00 | 75.00 | 63.28 |
| 4 | 87.50 | 75.00 | 75.00 | 62.50 |
| Input / model | Safe (%) | Site match (%) |
|---|---|---|
| Full RGBT sequence | 90.00 | 77.34 |
| Thermal only | 99.06 | 99.06 |
| RGB only | 73.91 | 61.41 |
| Frozen input | 75.00 | 62.50 |
| Per-frame spatial mean | 73.13 | 60.63 |
| Uniform global mean | 71.88 | 59.38 |
| Split | Sequence safe (%) | Final-frame safe (%) | Paired gain (points) | 95% group-bootstrap interval (points) |
|---|---|---|---|---|
| s0 | 92.97 | 75.00 | +17.97 | [11.72,24.22] |
| s1 | 90.63 | 75.00 | +15.63 | [10.16,21.09] |
| s2 | 92.19 | 75.00 | +17.19 | [11.72,22.66] |
| s3 | 90.63 | 75.00 | +15.63 | [10.16,21.09] |
| s4 | 91.41 | 75.00 | +16.41 | [10.16,22.66] |
| Mean | 91.56 | 75.00 | +16.56 | — |
| Input | Test accuracy (M2) | Balanced test accuracy (M2) | Validation accuracy (M3) |
|---|---|---|---|
| 4-frame sequence | 99.5 | 99.5 | 99.0 |
| Last frame | 50.0 | 50.0 | 50.0 |
| Last visible frame (privileged metadata) | 100.0 | 100.0 | 100.0 |
| Target object | Dynamic (D) | Static (S) | Final frame (L) | D–S gain (points) |
|---|---|---|---|---|
| Mug | ||||
| Sugar box | ||||
| Overall |
| Study | Setting | Safe choices (%) |
| Training data fraction | 71.3 | |
| 83.3 | ||
| 86.3 | ||
| 87.5 | ||
| Observation perturbation | Nominal | 94.4 |
| Thermal noise C | 91.7 |
| Initial vessel ( ∘ C) | Peak body mean ( ∘ C) | Local peak ( ∘ C) | Scalar: action / outcome | TRR: action / outcome |
|---|---|---|---|---|
| C | 25.14 | 27.57 | place / safe | place / safe |
| C | 25.23 | 29.26 | place / safe | place / safe |
| C | 25.31 | 30.93 | place / safe | place / safe |
| C | 25.39 | 32.58 | place / safe | place / safe |
| C | 25.47 | 34.22 | place / too hot | avoid / safe |
| C | 25.56 | 35.85 | place / too hot | avoid / safe |
| Run | Thermal nodes | Update steps | Simulated time (s) | Elapsed time (s) |
|---|---|---|---|---|
| Payload, coupled | 165,997 | 885 | 115.5 | 291.1 |
| Payload, contact-off | 165,997 | 885 | 115.5 | 292.4 |
| Cup sweep, mean (6 runs) | 165,997 | 885 | 115.5 | 294.2 |
| Active contact links | Peak body mean ( ∘ C) | Local peak ( ∘ C) |
|---|---|---|
| All links (coupled) | 25.555 | 35.846 |
| Vessel–phone only | 25.569 | 34.945 |
| Phone–desk only | 25.008 | 25.016 |
| No contact heat | 25.000 | 25.000 |
| Factor | Tested range | Peak body mean ( ∘ C) | Local peak ( ∘ C) |
|---|---|---|---|
| Contact resistance (m 2 K/W) | 0.001–0.004 | 25.56–25.58 | 34.83–35.14 |
| Conductivity | – | 25.36–25.84 | 34.17–34.84 |
| Heat capacity | – | 25.48–25.63 | 34.21–35.63 |
| Emissivity | – | 25.56–25.59 | 34.87–35.25 |
| Offset (m) | Body mean ( ∘ C) | half mean ( ∘ C) | half mean ( ∘ C) | Local maximum ( ∘ C) | Cooler half |
|---|---|---|---|---|---|
| 25.504 | 25.770 | 25.238 | 39.163 | ||
| 25.499 | 25.757 | 25.240 | 38.916 | ||
| 25.541 | 25.846 | 25.235 | 35.181 | ||
| 25.582 | 25.885 | 25.280 | 34.866 | ||
| 25.571 | 25.803 | 25.338 | 35.014 | ||
| 25.535 | 25.260 | 25.809 | 34.592 |
| State representation | Correct / total | Decision input |
|---|---|---|
| Plate-mean scalar | 2/10 | Phone mean only |
| Reset field | 5/10 | Uniform field after separation |
| Full TRR | 10/10 | Two half means |
| Input | Average precision | 95% interval | Recall |
|---|---|---|---|
| RGB | 0.261 | 0.188–0.367 | 0.056 |
| Frozen-field RGBT | 0.337 | 0.245–0.493 | 0.333 |
| Dynamic, early thermal | 0.941 | 0.842–0.995 | 0.889 |
| Dynamic, late thermal | 0.942 | 0.843–0.995 | 0.861 |
| Dynamic, both thermal | 0.939 | 0.838–0.995 | 0.889 |
| Future contact peak ( ∘ C) | ||||||||
| Reference | Full layers | Uniform depth | ||||||
| Target ( ∘ C) | Release (s) | Pad capacity (J/K) | Cold | Warm | Cold | Warm | Cold | Warm |
| 31 | 0.05 | 0.5 | 30.984 | 42.073 | 30.990 | 42.073 | 30.990 | 30.978 |
| 31 | 0.05 | 1.0 | 30.984 | 42.033 | 30.990 | 42.033 | 30.990 | 30.978 |
| 31 | 0.10 | 0.5 | 30.984 | 39.161 | 30.986 | 39.161 | 30.986 | 30.973 |
| 31 | 0.10 | 1.0 | 30.984 | 39.127 | 30.986 | 39.128 | 30.986 | 30.973 |
| Contact peak ( ∘ C) | Safe contact? | |||
|---|---|---|---|---|
| Model | Cold history | Warm history | Cold history | Warm history |
| Independent reference | 31.905 | 40.976 | yes | no |
| Full-layer native | 31.905 | 40.976 | yes | no |
| Uniform-depth native | 31.905 | 31.936 | yes | yes |
| Contact exchange disabled | TRR coupled query | ||||||
|---|---|---|---|---|---|---|---|
| Vessel ( ∘ C) | Peak if placed ( ∘ C) | Predicted peak ( ∘ C) | Action | Outcome | Predicted peak ( ∘ C) | Action | Outcome |
| 40 ∘ C | 27.57 | 25.00 | Place | Safe | 27.57 | Place | Safe |
| 50 ∘ C | 29.26 | 25.00 | Place | Safe | 29.26 | Place | Safe |
| 60 ∘ C | 30.93 | 25.00 | Place | Safe | 30.93 | Place | Safe |
| 70 ∘ C | 32.58 | 25.00 | Place | Safe | 32.58 | Place | Safe |
| 80 ∘ C | 34.22 | 25.00 | Place | Too hot | 34.22 | Avoid | Safe |
Robotics Simulation
Robotics simulation builds virtual robots and environments to model, train, and evaluate autonomous systems before real-world deployment.
See all 348 papers