A robot's actions change the thermal conditions of its next interaction: heat retained after release, carried with objects or stored beneath the surface can change a later contact's outcome despite similar current temperatures. Training trajectories must capture how these effects connect earlier actions to later observations and task outcomes. We present TRR, a Thermal Robot Renderer that attaches spatial and depth-resolved thermal state to both contact partners, updates local heat exchange through contact, transport and separation, and generates synchronized RGB--thermal (RGBT) observations and event-local task checks on one execution clock. Controlled interventions show why retained heat, spatial transport and internal storage matter for subsequent contact outcomes. From the same state, TRR produces a paired six-task benchmark of 2,224 static--dynamic pairs, that is, 4,448 trajectories with synchronized dual-view observations, states and actions. Visible thermal history raises safe contact-site selection from 75.0% to 91.7% on a dedicated learning corpus, and training on dynamic histories raises simulated transfer completion by 50 percentage points under a new heating procedure with shared feedback execution. We demonstrate RGBT observation adapters on MuJoCo and Isaac Sim, report an engineering inventory of ten prepared robot configurations and six demonstrated camera viewpoints, and describe a developer workflow that carries an existing robot scene to exported RGBT records. The benchmark, the workflow and the platform resources described in Section 6 form the first items of a planned public release. Demonstrations and resource updates are available on the project page.
Figures & tables
Figure 1: TRR overview. Local exchange leaves thermal traces on the contacted surfaces. The evolving fields supply camera observations and task-event checks during the same execution, while each body retains the thermal effects of earlier contacts.
Figure 2: What the engine records. RGB and thermal frames from one simulated transfer of a hot aluminium shell (Piper in MuJoCo), taken at the stages of contact, transport, regrasp and release. Thermal frames share a 25–70 ∘ C scale, so the warming gripper and support surfaces appear in the same images as the shell. The full sequence, including cooling after separation and residual heat at supports A and B once the shell has left them, appears at a 25–35 ∘ C scale in Figure 13 .
Conceptual pseudocode for the workflow of Figure 11 . The panel names the inputs of each stage and the records a run exports; t is simulation time, qn the robot state, Yn the camera temperature matrix and In the grayscale image at observation time tn .
Figure 3: RGBT examples from MuJoCo (Piper, left) and Isaac Sim (Franka, right). Temperature views share a scale; insets show same-state RGB. Gray marks unavailable temperature samples.
Figure 4: Retained support heat changes recontact. The shell has the same precontact temperature in the retained and reset runs, while the receiving surface and contact power differ. Shading marks contact; no exchange keeps the support at ambient.
Figure 5: Spatial history at a later contact. (a) Prior contact at L or R leaves different local fields despite equal plate-wide means. (b) Heating, translating and rotating a plate changes which material patch reaches the fixed receiver; dashed boxes mark its footprint. The two interventions have their own contact schedules and temperature scales for their respective contact configurations.
Figure 6: Similar current surface, different internal storage. (a) Cold and warm decision-depth profiles. (b) Instantaneous object-side contact temperature under the same finite pad. Reference and native curves nearly overlap. Both surface-matched uniform predictions are shown in gray and nearly coincide; the dashed line marks the 34∘ C contact threshold for both histories.
Figure 7: Measured and rendered steady-state observations for Piper pose 7. Temperature images share a scale; grayscale uses the calibrated display range for the measured capture setup.
Task
Pairs
Pass
Fail
Unresolved
Cooler mug
555
555
0
0
Hotter mug
300
300
0
0
Bottle, cold
225
225
0
0
Bottle, hot
425
200
225
0
Sugar box, cold
300
264
0
36
Sugar box, hot
419
191
219
9
Table 1: Paired benchmark composition. All static trajectories pass; the columns show dynamic outcomes. The archive contains 4,448 trajectories and 447,318 synchronized dual-view instants.
Figure 8: Validity and timing of paired thermal tasks. (a) Worst supported margin per condition. (b) First violation delay among detected violations. Dashed curves denote bottle tasks.
Figure 9: Thermal history resolves an occluded contact choice. Left: shared RGB and thermal crops for two histories (24–58 ∘ C), timed relative to the decision; the shutter hides the final view. Right: independent contact-surface temperatures, timed from contact, and seed-0 choices. Contact begins 0.18 s after the decision; the dashed line is 34 ∘ C. A/B are sites 0/1 at wait 0.
Input / model
Safe (%)
Site match (%)
Full RGBT sequence
91.7
80.8
Final-frame repeat
75.0
62.7
Frozen input
75.0
62.7
Per-frame spatial mean
72.7
60.2
RGB only
72.5
60.0
Global-mean GRU
62.5
50.0
Table 2: Image-to-site results on 128 held-out decisions (64 groups), averaged over seeds 0–4. Safe is the percentage of certified-safe choices; site match is exact agreement with the expert site. Bold and underline mark the highest and second-highest rates.
Training input
Seed
Same heating procedure
New heating procedure
Dynamic (D)
0–2
12 | 2 | 2
13 | 1 | 2
Static (S)
0–2
6 | 9 | 1
5 | 9 | 2
Visible-frame repeat (C)
0, 1
12 | 2 | 2
13 | 1 | 2
Visible-frame repeat (C)
2
12 | 2 | 2
11 | 3 | 2
Table 3: Transfer outcomes per 16 conditions. Cells list completed, failed and unresolved runs, in that order. Seeds are learner initializations; all test placements are held out.
Figure 10: Simulated transfer under a new heating procedure. (a) Latest visible IR and D/S region choices. (b) Lower/upper bounds on the maximum qualifying-contact point temperature; gaps indicate no qualifying contact. D passes; S violates the 28.5 ∘ C limit. Callouts mark the candidate regions.
Figure 11: TRR developer workflow. A robot scene is configured with thermal assets and a camera model, executed with coupled thermal interaction, and exported. Execute writes three parallel record streams and each stream feeds Export; the streams are synchronized by the same execution and are not derived from one another.
Record
Key fields
Content
Observation
observation time, camera id, view
RGB frame, camera temperature matrix Yn , grayscale image In
Robot state
observation time
robot state qn
Action interval
start and end time
first control target after the observation, with the full native interval retained
Event check
event time, check type, region
object, region, timing or mechanical outcome: pass, fail or unresolved
Trajectory
trajectory id, task, program
task label from the checks, contact channels and durations
Table 4: Content of a TRR run record. Fields follow the record convention described in Section 3.5 .
Resource
Inventory
Role and examples
Robot configurations
10 configurations
Articulated scene setups, including Piper, Panda, FR3 and xArm7.
Thermal caches
16 categories
Nine robot caches, four object caches and three environment caches.
Measured robot asset
One Piper asset
Piper with D435 camera and mounting bracket; demonstrated reuse across ten poses.
Simulation adapters
Two backends
MuJoCo and Isaac Sim, with RGB and thermal observation examples.
Task scene families
Three families
Cup, sugar box and bottle, covering six hot/cold task conditions.
Camera placements
Six demonstrated views
Three external and three wrist-mounted viewpoints with RGBT output.
Table 5: Engineering resource inventory. Asset and interface counts were recorded on October 12, 2026; thermal-process demonstrations were updated on October 14. Counts describe the corresponding resource type.
Demonstrated capability
Use it enables
Heat is retained on both contact partners through separation
Study how repeated grasping and recontact change object and support temperatures, and why an equally warm object can meet a differently warm surface.
Material regions move with the object
Simulate the local thermal condition of different grasp sites and predict which arriving region meets the next interface.
Internal storage behind a similar surface
Treat the visible surface as incomplete evidence: two objects at the same surface temperature can lead to different contact peaks and different safety outcomes.
Identical motions with different task labels
Construct and analyze data under dynamic thermal constraints; 444 of 2,224 paired motions move from pass to violation as heat evolves.
Thermal history improves contact-site choice
Quantify the value of temporal thermal observations for manipulation decisions, with a matched final-frame control.
Dynamic data improves simulated transfer
Provide evidence that dynamic thermal training data changes execution outcomes under a shared feedback executor.
Table 6: What the research results mean for a user of TRR. Each row names a demonstrated capability and the use it enables.
Resource
Technical status
Access status
Content and supporting evidence
Project demonstrations
Published
Public
Preview GIF and overview images on the project page and repository.
Paired RGBT benchmark
Demonstrated
Internal
2,224 source-condition pairs, 4,448 trajectories and 447,318 synchronized dual-view instants across six tasks (Table 1 ).
Contact-site learning corpus
Reported study
Internal
240 source groups in four contact-history families; primary comparison on 128 held-out decisions in 64 groups.
Regional-decision transfer corpus
Reported study
Internal
Regional choices with shared feedback execution, 16 test conditions per heating track and three initializations.
Simulation adapters
Demonstrated
Internal
MuJoCo and Isaac Sim RGBT output (Figure 3 ).
Robot configurations
Prepared
Internal
Ten configurations, including Piper, Panda, FR3 and xArm7.
Table 7: Resource status, with technical status separated from access status. Platform resources follow the October 2026 inventory of Table 5 ; the benchmark and learning corpora are reported with their own results. Research resources retain their own access status; public preview media are listed separately.
Measured cost of one coupled run
Thermal nodes
165,997
Update steps
885
Simulated time
115.5 s
Coupled payload wall-clock
291.1 s
Contact-off wall-clock
292.4 s
Cup sweep wall-clock
294.2 s (mean of six runs)
Table 8: Integration scope. Measured payload workload, followed by model scope and extension directions. The thermal network runs on the host CPU.
Appendix figures & tables32 assets
Supplementary material from the paper’s appendix.
Appendix
Workflow
Thermal state
Interaction coupling
IR output
Measured evidence
Robot tasks
iGibson 2.0 [ 24 ]
Object states
Temperature-dependent state changes
—
—
✓
BEHAVIOR-1K / OmniGibson [ 25 ]
Object temperature
Source and ambient exchange
—
—
✓
HAZARD [ 57 ]
Object temperature
Evolving fire and object temperatures
—
—
✓
RFUniverse [ 8 ]
Particle material state
Contact-dependent material change
—
—
✓
Tactile Genesis [ 7 ]
Link temperature voxels
Sensing voxels coupled to an opposing link scalar
—
—
✓
Stonefish [ 11 ]
Prescribed surface texture
—
Thermal-camera image
—
✓
Appendix
Table 9: Thermal state, interaction coupling and infrared output in representative workflows.
Stage / parameter
Setting
Graph-field fitting
Graph data loss
Huber threshold 0.35∘ C; four reweighting iterations
Graph reference length
ℓs=3 mm
Graph regularization
Matrix strength s∈{0.003,0.01,0.03} ; energy coefficients in Eq. ( 25 ) are λc=λg=s2
Prior / gradient anchor
ηn=0.003 with at least three supporting pose groups, otherwise 0.15; gradient anchor matrix strength 10−6
Surface node spacing
3 mm on detail surfaces; 4.5 mm on the D435 main housing; reference link-2 main field uses 6 mm
Appendix
Table 10: Fitting settings for the measured Piper asset. Graph parameters apply to the adopted detail patches; the two local refinements retain surrounding asset values.
Figure 12: Recorded contact-channel totals for the transfer execution. Energy is accumulated from object-to-partner exchange; duration is accumulated separately for each active channel. Concurrent channels can contribute during the same interval.
Figure 13: Transfer and regrasp (top); cooling and removal (bottom). Full thermal views share a 25–70 ∘ C scale; insets show residual heat at A and B on a 25–35 ∘ C scale.
True emissivity ϵ
0.80
0.85
0.90
0.95
1.00
Shell temperature at trigger ( ∘ C)
41.73
40.82
40.00
39.26
38.59
Appendix
Table 11: Physical trigger temperature when the emissivity-corrected estimate reaches 40 ∘ C with assumed ϵ0=0.90 .
Uncorrected camera
Corrected camera
Initial shell ( ∘ C)
State-trigger time (s)
Delay (s)
Shell at start ( ∘ C)
Delay (s)
Shell at start ( ∘ C)
60
122.8
-17.3
41.49
+0.2
39.98
70
168.8
-18.8
41.53
+0.2
39.98
80
206.7
-19.2
41.50
+0.3
39.98
Appendix
Table 12: Camera-triggered transfer. Delay is measured relative to the state trigger. Temperature is the physical shell temperature at motion start.
Reference
Resolution
Max. error ( ∘ C)
Contact exchange
Δt=0.20 s
0.2531
Contact exchange
Δt=0.10 s
0.1253
Contact exchange
Δt=0.05 s
0.0624
Slab diffusion
8 cells
0.0434
Slab diffusion
16 cells
0.0112
Slab diffusion
32 cells
0.0030
Appendix
Table 13: Maximum temperature error against insulated analytical references.
Figure 14: Benchmark composition by task and generation family. Bars show dynamic task labels; counts are source-condition pairs.
Figure 15: Paired observations before the common pinch. Both thermal images use the same apparent-temperature scale.
Pair
Target region
Initial / frozen field
At contact
After hold
Change from initial
1
Cooler
9.01
8.85
7.58
−1.42
2
Cooler
7.47
7.33
6.29
−1.18
3
Cooler
6.76
6.64
5.69
−1.07
4
Warmer
3.85
3.79
3.25
−0.61
5
Warmer
3.25
3.20
2.74
−0.51
Appendix
Table 14: Warm-minus-cool contrast across simulated closures ( ∘ C). Changes are computed relative to initialization before rounding.
Task outcomes (count)
Component passes
Input
Seed
Completed
Failed
Unresolved
Success bounds (%)
Mechanical
Correct object
Same heating procedure; held-out placements
D
0
12
2
2
[75.00,87.50]
14/16
14/16
D
1
12
2
2
[75.00,87.50]
14/16
14/16
D
2
12
2
2
[75.00,87.50]
14/16
14/16
S
0
6
9
1
[37.50,43.75]
14/16
14/16
Appendix
Table 15: Complete transfer results by heating procedure and learner initialization. D: dynamic training; S: static training; C: latest-visible-frame repeat; R: geometry–temperature rule. Each row has 16 planned conditions in eight pose clusters. Success bounds span certified completions through completions plus unresolved runs. The final columns count mechanical and correct-object passes. A planning failure acts on no object; its false object predicate is not evidence of choosing a distractor.
Apparent temperature ( ∘ C)
Display grayscale (0–255)
Pose
Mask coverage (%)
MAE
RMSE
Bias
MAE
RMSE
Bias
1
58.88
1.528
2.179
+0.480
45.76
52.91
-40.77
2
58.08
1.546
2.270
+0.682
48.15
54.97
-42.27
3
56.54
1.475
2.213
+0.713
50.73
57.11
-44.71
4
55.03
1.583
2.508
+0.938
59.30
65.64
-51.91
5
55.54
1.279
2.099
+0.802
58.74
64.65
-54.13
Appendix
Table 16: Observation errors across ten Piper configurations. Bias is prediction minus measurement; means weight configurations equally.
Figure 16: Measured and TRR observations for Piper poses 1–5. Shared scales: temperature 22 – 40∘ C, absolute error 0 – 8∘ C, grayscale 0–255. White error-map regions lie outside the evaluation masks.
Figure 17: Measured and TRR observations for Piper configurations 6–10, with columns and scales as in Figure 16 . All quantitative results: Table 16 .
Initialization depth
Layered TRR
Scalar cooling
Frozen field
1 mm
1.33
1.30
2.05
2 mm (nominal)
0.97
0.93
2.05
4 mm
0.52
0.45
2.05
Appendix
Table 17: Pooled region-mean MAE ( ∘ C) against 24 measured cooling frames. Bold marks the lowest error at each initialization depth.
Full RGBT sequence
Final-frame repeat
Seed
Safe (%)
Site match (%)
Safe (%)
Site match (%)
0
95.31
92.19
75.00
62.50
1
94.53
82.03
75.00
62.50
2
92.97
79.69
75.00
62.50
3
88.28
75.00
75.00
63.28
4
87.50
75.00
75.00
62.50
Appendix
Table 18: Image-to-site test on 128 held-out decisions (64 groups), by learner seed. All rates are percentages. Safe uses the independent finite-difference certificate; site match is exact expert-site agreement. The full and final-frame-repeat arms share the ResNet/GRU/head.
Input / model
Safe (%)
Site match (%)
Full RGBT sequence
90.00
77.34
Thermal only
99.06
99.06
RGB only
73.91
61.41
Frozen input
75.00
62.50
Per-frame spatial mean
73.13
60.63
Uniform global mean
71.88
59.38
Appendix
Table 19: Matched channel study, held-out test (128 decisions; seeds 0–4). Bold and underline mark the highest and second-highest observed percentages. The legacy last-frame path differs in BN/input handling; the corrected frame-only five-seed control is reported separately.
Split
Sequence safe (%)
Final-frame safe (%)
Paired gain (points)
95% group-bootstrap interval (points)
s0
92.97
75.00
+17.97
[11.72,24.22]
s1
90.63
75.00
+15.63
[10.16,21.09]
s2
92.19
75.00
+17.19
[11.72,22.66]
s3
90.63
75.00
+15.63
[10.16,21.09]
s4
91.41
75.00
+16.41
[10.16,22.66]
Mean
91.56
75.00
+16.56
—
Appendix
Table 20: Compact-feature policy across five source-group partitions. Each test contains 128 decisions from 64 groups; the mean weights the five fits equally.
Input
Test accuracy (M2)
Balanced test accuracy (M2)
Validation accuracy (M3)
4-frame sequence
99.5
99.5
99.0
Last frame
50.0
50.0
50.0
Last visible frame (privileged metadata)
100.0
100.0
100.0
Appendix
Table 21: Material-held-out corpus (300 groups / 1200 episodes). The coupon geometry is unchanged; material set M1 is used for training, M3 for validation and M2 for testing. Bold and underline mark the highest and second-highest accuracies (%).
Target object
Dynamic (D)
Static (S)
Final frame (L)
D–S gain (points)
Mug
64/64
0/64
32/64
+100.0
Sugar box
64/64
0/64
32/64
+100.0
Overall
128/128
0/128
64/128
+100.0
Appendix
Table 22: Shared-candidate transfer from coupon source training to two target objects. Each object supplies 32 groups and 64 paired-history decisions. Cells report safe choices; exact agreement has the same counts.
Study
Setting
Safe choices (%)
Training data fraction
10%
71.3
25%
83.3
50%
86.3
100%
87.5
Observation perturbation
Nominal
94.4
Thermal noise 0.5∘ C
91.7
Appendix
Table 23: Training-set scaling and observation perturbations for the compact-feature policy. Training fractions use the trend and occluded families; perturbations use the archived 144-decision evaluation. Each block retains all its tested settings.
Initial vessel ( ∘ C)
Peak body mean ( ∘ C)
Local peak ( ∘ C)
Scalar: action / outcome
TRR: action / outcome
40∘ C
25.14
27.57
place / safe
place / safe
50∘ C
25.23
29.26
place / safe
place / safe
60∘ C
25.31
30.93
place / safe
place / safe
70∘ C
25.39
32.58
place / safe
place / safe
80∘ C
25.47
34.22
place / too hot
avoid / safe
90∘ C
25.56
35.85
place / too hot
avoid / safe
Appendix
Table 24: Plate-mean scalar baseline for the safety decision. The scalar is fitted to the same cup–phone–support traces; it predicts the payload mean, whereas the decision uses the localized peak.
Run
Thermal nodes
Update steps
Simulated time (s)
Elapsed time (s)
Payload, coupled
165,997
885
115.5
291.1
Payload, contact-off
165,997
885
115.5
292.4
Cup sweep, mean (6 runs)
165,997
885
115.5
294.2
Appendix
Table 25: Payload workload and measured wall-clock. Timing is single-process on one A6000 node; the thermal network is CPU-side split-explicit.
Active contact links
Peak body mean ( ∘ C)
Local peak ( ∘ C)
All links (coupled)
25.555
35.846
Vessel–phone only
25.569
34.945
Phone–desk only
25.008
25.016
No contact heat
25.000
25.000
Appendix
Table 26: Path ablation for the 90 ∘ C payload hold. Only the indicated body-pair contact links are removed.
Factor
Tested range
Peak body mean ( ∘ C)
Local peak ( ∘ C)
Contact resistance (m 2 K/W)
0.001–0.004
25.56–25.58
34.83–35.14
Conductivity
0.5 – 2×
25.36–25.84
34.17–34.84
Heat capacity
0.7 – 1.5×
25.48–25.63
34.21–35.63
Emissivity
0.8 – 1.0
25.56–25.59
34.87–35.25
Appendix
Table 27: Bounded material/contact sensitivity at the 90 ∘ C decision. Each row changes one factor from the configured baseline (max peak 35.85 ∘ C); all spatial decisions remain over-temperature.
Offset (m)
Body mean ( ∘ C)
y− half mean ( ∘ C)
y+ half mean ( ∘ C)
Local maximum ( ∘ C)
Cooler half
−0.018
25.504
25.770
25.238
39.163
y+
−0.016
25.499
25.757
25.240
38.916
y+
−0.012
25.541
25.846
25.235
35.181
y+
−0.008
25.582
25.885
25.280
34.866
y+
−0.006
25.571
25.803
25.338
35.014
y+
+0.006
25.535
25.260
25.809
34.592
y−
Appendix
Table 28: Matched-mean history probe. Contact offset is along the phone’s y axis; positive offsets heat the y+ half. Decisions select the cooler half.
State representation
Correct / total
Decision input
Plate-mean scalar
2/10
Phone mean only
Reset field
5/10
Uniform field after separation
Full TRR
10/10
Two half means
Appendix
Table 29: Decision arms on the ten matched histories. Scalar uses only the phone mean with a leave-one-out threshold; reset removes the half pattern after separation; full TRR reads the two half means.
Input
Average precision
95% interval
Recall
RGB
0.261
0.188–0.367
0.056
Frozen-field RGBT
0.337
0.245–0.493
0.333
Dynamic, early thermal
0.941
0.842–0.995
0.889
Dynamic, late thermal
0.942
0.843–0.995
0.861
Dynamic, both thermal
0.939
0.838–0.995
0.889
Appendix
Table 30: Precontact forecast by modality and frame time. Average precision (AP) intervals use source-group bootstrap resampling; Bold and underline mark the two highest distinct point estimates per metric.
Future contact peak ( ∘ C)
Reference
Full layers
Uniform depth
Target ( ∘ C)
Release (s)
Pad capacity (J/K)
Cold
Warm
Cold
Warm
Cold
Warm
31
0.05
0.5
30.984
42.073
30.990
42.073
30.990
30.978
31
0.05
1.0
30.984
42.033
30.990
42.033
30.990
30.978
31
0.10
0.5
30.984
39.161
30.986
39.161
30.986
30.973
31
0.10
1.0
30.984
39.127
30.986
39.128
30.986
30.973
Appendix
Table 31: Hidden-depth sensitivity across 16 fixed combinations of target surface temperature, release duration and pad heat capacity. Cold and warm denote the two internal-storage histories. The four rows marked unreachable cannot attain the target surface temperature after release; the remaining rows compare future contact peaks.
Figure 18: Continuous thermal interaction record. (a–d) Transfer, contact detail, contact hold and cooling. (e,f) Source and destination surface histories averaged along y . Shading marks interaction intervals; dashed lines mark the trigger and uncovering.
Contact peak ( ∘ C)
Safe contact?
Model
Cold history
Warm history
Cold history
Warm history
Independent reference
31.905
40.976
yes
no
Full-layer native
31.905
40.976
yes
no
Uniform-depth native
31.905
31.936
yes
yes
Appendix
Table 32: Surface-matched uniform state versus the full layer state on identical finite-pad recontact. Safe uses the 34∘ C peak rule. The uniform state is fitted to the current surface and is not energy-matched.
Contact exchange disabled
TRR coupled query
Vessel ( ∘ C)
Peak if placed ( ∘ C)
Predicted peak ( ∘ C)
Action
Outcome
Predicted peak ( ∘ C)
Action
Outcome
40 ∘ C
27.57
25.00
Place
Safe
27.57
Place
Safe
50 ∘ C
29.26
25.00
Place
Safe
29.26
Place
Safe
60 ∘ C
30.93
25.00
Place
Safe
30.93
Place
Safe
70 ∘ C
32.58
25.00
Place
Safe
32.58
Place
Safe
80 ∘ C
34.22
25.00
Place
Too hot
34.22
Avoid
Safe
Appendix
Table 33: Placement decisions under a common 33 ∘ C payload-peak rule. The coupled scene supplies the outcome; the query uses evolving contact exchange or contact-off.
Language-guided manipulation can depend on physical properties that visible appearance does not reveal. Temperature is one such property, but object datasets for robot learning rarely associate measured temperatures with object appearance and geometry. We present HEARTH, an object-centric RGB-thermal-3D dataset of 90 physical objects from 18 everyday categories, comprising 145 captured object states. Our pipeline maps apparent surface temperatures onto reconstructed meshes through camera calibration and pose transfer. The dataset includes raw temperature measurements, camera parameters, RGB-textured meshes, and thermal textures for simulation. We use these assets to construct three LIBERO-derived tasks and collect 1,200 demonstrations for fine-tuning a pretrained vision-language-action (VLA) model, π0.5. In an ablation study, adding thermal observations to the VLA increases success on temperature-dependent object-selection tasks from 35.0% for the RGB-only baseline to 75.0%. These results demonstrate the utility of HEARTH for training robot policies to follow temperature-related instructions.
In-context robot learning enables robots to acquire new behaviors from demonstrations without test-time parameter updates. However, visual and kinematic specifications alone may not distinguish contact-rich behaviors that share similar motions but require different interaction intensities or patterns. We propose Contact Trajectory Prompting (CTP), a framework for one-shot transfer of contact-rich manipulation behaviors. CTP encodes motion and contact information from a single demonstration into an ordered contact trajectory prompt that captures their temporal evolution and specifies the desired physical interaction behavior. During execution, the policy uses online sensory feedback to retrieve interaction-relevant information from the prompt, conditioning closed-loop action generation on the demonstrated contact behavior. Across real-world wiping, rotation, and insertion tasks on objects excluded from training, CTP achieves 78.7% overall success, outperforming the strongest evaluated baseline by 13.0 percentage points. Changing only the reference demonstration enables CTP to follow contact specifications for both intensity and direction, including an intermediate wiping force level and clockwise or counterclockwise cap rotation. These results demonstrate that contact trajectories serve as effective behavioral specifications, extending in-context robot learning beyond motion imitation to physical interaction transfer. Project website: https://ctp-robotics.github.io/project_page.
Xian Nie, Yujie Zang, Yuhang Zheng +8
Shanghai Jiao Tong University · TARS Robotics · National University of Singapore +2
Recent Vision-Language-Action (VLA) models have begun incorporating thermal sensing, but existing approaches typically treat thermal images as ordinary visual inputs, providing limited support for temperature-aware relational reasoning in manipulation. To address this limitation, we propose Thermo-VLA with a ThermoRes Encoder, which decomposes thermal images into cold and hot attribute representations and injects them into the RGB token through language-conditioned residual modulation, enabling temperature-driven target selection and continuous manipulation. We further construct six real-world robotic tasks on a dual-arm Unitree G1 platform, spanning basic grounding to relational reasoning. Experiments show that Thermo-VLA achieves an average success rate of 0.86 across four thermal-dependent tasks, outperforming a pi_0.5 baseline that directly incorporates RGB-thermal observations by 18 percentage points. Under unseen-object and scene variations, Thermo-VLA maintains an average success rate of 0.78, achieving the best generalisation performance among the compared methods. Beyond thermal sensing, we transfer the same residual modulation mechanism to colour attributes and observe consistent improvements in both simulation and real-robot experiments, indicating its potential to generalise beyond thermal attributes. Project website: https://j-x-chen.github.io/Thermo-VLA-web/