HolidayFactory 1.0: Simulating, Generating, and Evaluating Industrial Robot Tasks
Abstract
Robots are typically evaluated in episodes that begin from a reset, while industrial work consists of repeated jobs whose outcomes affect subsequent operations. Evaluating such work at scale requires simulation of task-critical physical interactions, construction of industrial environments, and measurements of performance as scene state persists. We present HOLIDAYFACTORY 1.0, a framework for simulating, generating, and evaluating industrial robot work. Given a human-specified workplace and task, the framework defines a world pairing a scenario of scene variants and process logic with an evaluation defining a job, metrics, and a rubric. HOLIDAYFACTORY 1.0 comprises three components. HOLIDAY-SIM extends Newton's SDF-based hydroelastic contact with additional constitutive laws and a semi-implicit primal-dual solver, couples stateful actuator dynamics to articulated motion, and derives tactile and joint observations from simulated states. Agentic world construction generates simulation-ready assets and scene variants using geometric and interaction checks, and proposes task-conditioned evaluations for human confirmation and executable checks. 24/7-BENCH introduces seven industrial tasks in which jobs are evaluated without resetting the scene, measuring completed, failed, and unfinished jobs, throughput, latency, and motor energy. Controlled studies characterize the proposed contact, actuator, and tactile models and examine the effectiveness of verification-guided world construction. Experiments on scene and evaluation generation further reveal the limitations of model critics and the difficulty of ensuring that generated evaluations faithfully capture task requirements. We examine job-level evaluations with scripted trajectories (e.g., code-as-policy) but have not yet compared policies over persistent runs.
Figures & tables
| \tablehead Symbol | \tablehead Meaning |
| Numbers of scene variants and metric definitions. | |
| Simulator state, policy action, and observation at run step . | |
| Policy mapping observation history and task to an action. | |
| State transition under scene and process logic , and observation model for . | |
| Recorded trajectory and its actual number of steps. | |
| Evaluation report under specification . |
| \tablehead Benchmark | \tablehead Scale | \tablehead Environment | \tablehead Horizon | \tablehead Simulator | \tablehead Scored unit | \tablehead Metrics |
| LIBERO ( Liu et al., 2023a ) | 130 tasks | Tabletop | 30 s | MuJoCo (robosuite) | Episode | Success rate |
| ManiSkill3 ( Tao et al., 2025 ) | 12 domains | Tabletop Indoor | 2.5-12.5 s † | PhysX (SAPIEN) | Episode | Success rate Episode return |
| RoboCasa365 ( Nasiriany et al., 2026 ) | 365 tasks | Kitchen | 15 s-6 min | MuJoCo (robosuite) | Episode | Success rate |
| BEHAVIOR-1K ( Li et al., 2024a ) | 50 tasks | Household | 2.4-29 min † | PhysX (OmniGibson) | Episode | Goal completion Execution time Motion distance |
| RoboTwin 2.0 ( Chen et al., 2026a ) | 50 tasks | Tabletop | 400-1,700 steps † | PhysX (SAPIEN) | Episode | Success rate |
| HumanoidBench ( Sferrazza et al., 2024 ) | 27 tasks | Obstacle courses Indoor | 10-20 s | MuJoCo | Episode | Success rate Episode return |
| \tablehead System | \tablehead Simulator engine | \tablehead Contact geometry | \tablehead Contact physics | \tablehead Contact solver |
| LIBERO ( Liu et al., 2023a ) | MuJoCo (robosuite) | Convex decomp. | Compliant point | Newton ∗ |
| ManiSkill3 ( Tao et al., 2025 ) | PhysX (SAPIEN) | Convex decomp. | Point | TGS ∗ |
| RoboCasa365 ( Nasiriany et al., 2026 ) | MuJoCo (robosuite) | Convex decomp. | Compliant point | Newton ∗ |
| BEHAVIOR-1K ( Li et al., 2024a ) | PhysX (OmniGibson / Isaac Sim) | Convex decomp. | Point | TGS ∗ |
| RoboTwin 2.0 ( Chen et al., 2026a ) | PhysX (SAPIEN) | Convex decomp. | Point | TGS ∗ |
| HumanoidBench ( Sferrazza et al., 2024 ) | MuJoCo | Convex decomp. | Compliant point | Newton ∗ |
| Backend | Iter. cap | Lifted |
| MuJoCo | 15 | 1/20 |
| MuJoCo | 100 | 0/20 |
| CDMM | 32 | 20/20 |
| \tablehead System | \tablehead Actuation model | \tablehead Actuation characteristics | \tablehead Internal actuator states |
| LIBERO | Ideal torque/servo actuation | Torque limits | None modeled |
| RoboCasa365 | Ideal torque/servo actuation | Torque limits | None modeled |
| ManiSkill3 | Ideal PD drives | Torque limits | None modeled |
| HolidayFactory 1.0 | Embedded control motor/transmission | Electrical limits; transmission and friction dynamics | Electrical, thermal, friction, hydraulic, and controller states |
| Check | Measurement | Action on failure |
| Resolution admissibility | geometric voxel-size bounds | choose a finer admissible voxel size |
| SDF fidelity | surface, boundary, and topology checks | refine and rebake |
| Layout preservation | per-part wall and surface retention | keep failed part separate |
| Probe | Evaluation unit | Mesh | Code-native | Selected generator |
| Visual shape fidelity | VLM judgments | 80% | 0% | Mesh |
| Tote: vertical walls, no handles | geometric checks | 80% | 100% | Code |
| Tote: wall mm | geometric checks | 75% | 100% | Code |
| Rack / peg: requested pocket count | structural checks | 37.5% | 100% | Code |
| Rack / peg: diameter within mm | geometric checks | 0% | 100% | Code |
| Rack / peg: all parts fully seated | functional checks | 0% | 100% | Code |
| Metric | Independent regeneration | Coarse feedback | Eris |
| 0.63 | 0.63 | 0.63 | |
| 0.33 (4/12) | 0.17 (2/12) | 0.92 (11/12) | |
| 0.75 | 0.69 | 0.97 | |
| 0.38 (3/8) | 0.40 (4/10) | 1.00 (1/1) | |
| 0.84 | 0.81 | 1.00 |
| LLM backbone | Raw | Eris | |
| claude-opus-5-5 (high) | 0.65 (26/40) | 1.00 (40/40) | +0.35 |
| claude-fable-5-1 | 0.70 (28/40) | 0.95 (38/40) | +0.25 |
| gpt-6-astra | 0.65 (26/40) | 0.95 (38/40) | +0.30 |
| gpt-6.1-sol | 0.58 (23/40) | 0.93 (37/40) | +0.35 |
| gpt-6-luna | 0.43 (17/40) | 0.75 (30/40) | +0.33 |
| Metric | One-shot | Independent regeneration | Eris |
| 18/40 | 26/40 | 40/40 | |
| Overall efficiency | |||
| Rounds / solved | 2.22 | 3.81 | 1.70 |
| Cost / solved | $2.59 | $5.00 | $2.48 |
| Repair efficiency | |||
| Rounds / recovered | – | 7.4 | 1.3 |
| \tablehead Component | \tablehead Output | \tablehead Verification | \tablehead Reported measures |
| Task planning | Task plan, a predicate-based graph of work steps | Deterministic plan checks, then plan critic | Acceptance, cost |
| Scene generation | Scene specification and resolved variants | Deterministic scene checks, then scene critics | Acceptance, validity, fidelity, cost |
| Background generation | Panorama, Gaussian splat, work-cell placement | Image, view, and stability checks; model assessment | Rubric, stability |
| Process logic authoring | Logic with effects, events, queues, and idle joint targets | Schema validation when the scenario is loaded | - |
| \tablehead Planner | \tablehead Critic | \tablehead Plan | \tablehead Scene spec. | \tablehead Checks | \tablehead Scene | \tablehead Min |
| gpt-6.1-sol | Own | 8/10 | 8/10 | 5/10 | 1/10 | 27 |
| gpt-6.1-sol | claude-opus-5-5 | 7/10 | 7/10 | 4/10 | 3/10 | 28 |
| claude-sonnet-5-5 | Own | 10/10 | 10/10 | 10/10 | 7/10 | 12 |
| claude-sonnet-5-5 | claude-opus-5-5 | 9/10 | 9/10 | 9/10 | 3/10 | 22 |
| \tablehead Specification generation | \tablehead Full ISG | ||||||||
| \tablehead Model (effort) | Easy | Medium | Hard | 1st | Attempts | Plan | Checks | Scene | Min |
| gpt-6-astra (high) | 3/3 | 3/3 | 3/3 | 2/9 | 2.44 | 5/10 | 0/10 | 0/10 | 29 |
| gpt-6.1-sol (medium) | 3/3 | 3/3 | 1/3 | 0/9 | 2.57 | 8/10 | 5/10 | 1/10 | 27 |
| gpt-6-luna (medium) | 2/3 | 0/3 | 1/3 | 0/9 | 3.33 | 3/10 | 2/10 | 0/10 | 13 |
| claude-opus-5-5 (high) | 1/3 | 3/3 | 3/3 | 0/9 | 3.29 | 9/10 | 8/10 | 2/10 | 29 |
| claude-sonnet-5-5 (medium) | 3/3 | 2/3 | 3/3 | 2/9 | 3.00 | 10/10 | 10/10 | 7/10 | 12 |
| \tablehead Condition | \tablehead Plan | \tablehead Scene spec. | \tablehead Checks | \tablehead Scene |
| Default budget (40 min, 40 calls) | 8/10 | 8/10 | 5/10 | 1/10 |
| + Larger budget (2 h, 80 calls) | 9/10 | 9/10 | 2/10 | 1/10 |
| + Asset generation | 9/10 | 9/10 | 2/10 | 0/10 |
| + Affordance format in prompt | 9/10 | 9/10 | 3/10 | 2/10 |
| \tablehead Step | \tablehead Content |
| Task 1 | Single tote transfer : “Pick up the tote presented on the pallet and place it upright on the running conveyor.” |
| Task 2 | Conveyor loading : “Pick up each tote presented on the pallet and place it upright on the running conveyor.” |
| Question 1 | Does a job complete upon placement on the belt or entry into the X-ray inlet? Entry into the inlet. |
| Question 2 | What tilt is allowed? At most . |
| Question 3 | When does a run end? After one job or 1 min for Task 1, and after 5 min for Task 2. |
| Shared job | Starts when a tote is presented in the pick region, requires upright placement supported by the conveyor, and completes upon entry into the X-ray inlet. All conditions use simulator state. |
| \tablehead Check | \tablehead Execution | \tablehead Rejects when | \tablehead Does not detect |
| Static | No simulation | Invalid definitions, unjustified event rules, or time limits beyond the run | Incorrect regions or tolerances |
| Reset | 4 s after reset | Expected job units fail to satisfy or maintain their start conditions | Overly broad start regions or later state changes |
| Idle | Rollout to termination | Jobs complete without robot action, the run passes, or start events recur during open jobs | Errors requiring policy actions or task-intent labels |
| \tablehead Family | \tablehead Task | \tablehead One job | \tablehead Carried state | \tablehead Robot application | \tablehead Occupation |
| Material flow | Conveyor loading | Tote, pallet belt | Totes on pallet, belt | Picking, placing | Material movers |
| Flow-rack loading | Container, stack lane | Filled lanes, stack | Picking, placing | Material movers | |
| Cart push | Carton, table dock | Cart loads, docks | Transport † | Material movers | |
| Assembly | Jig loading | Part in stored jig; empty jig seated | Filled jigs | Assembling | Assemblers |
| Machine tending | Press tending | Stamped part on outfeed | Press, stamped parts | Press tending | Press operators |
| Process operation | Valve operation | Beaker, spout output | Valve, filled beakers | – | Plant operators |
| \tablehead Task | \tablehead Start | \tablehead Required | \tablehead Completion |
| Conveyor loading | On pallet | Upright on belt | Entered tunnel |
| Flow-rack loading | In stack | In aisle | Upright in lane |
| Cart push | Staged | Left table, crossed aisle | In dock, loading deck clear |
| Jig loading | In tray | Left tray, in bench jig | Jig stored, empty jig seated |
| Press tending | On infeed | Nested, two-hand, stamped | Part on outfeed |
| Valve operation | On input side | Filled | On output side, valve shut |
Appendix figures & tables33 assets
Supplementary material from the paper’s appendix.
Appendix
| \tablehead Time step | \tablehead CDMM, at most 32 | \tablehead MuJoCo, at most 15 | \tablehead MuJoCo, at most 100 |
| s | 5 / 0 | 0 / 0 | 0 / 0 |
| s | 5 / 0 | 0 / 0 | 0 / 0 |
| s | 5 / 0 | 0 / 0 | 0 / 0 |
| s | 5 / 0 | 1 / 1 | 0 / 0 |
| \tablehead Parameter | \tablehead Meaning | \tablehead Default |
| Normal compliance | ||
| Radial compliance | ||
| Local/global radial blend | ||
| Lateral response scale | ||
| Depth attenuation length | ||
| Configured magnet depth |
| Category | Representative constraint types | Description |
| Property | dimension , part_count , min_thickness , appearance | Dimensions, counts, wall thickness, and appearance |
| Access | open_top , access , graspable | Openings and manipulation accessibility |
| Fit | fits , clearance , aligns , … | Geometric compatibility, fit, and clearance |
| Articulation | dof , actuated , reaches_state , … | Relative motion and actuation |
| Support | supports , retains , stable , … | Load support, retention, and stability |
| Transport | transports , guides , presents , … | Object transport and presentation |
| Branch | Generator | Model setting | Budget |
| Mesh | FLUX.2-klein-9B BiRefNet SAM 3D PaMO | pipeline defaults | |
| Code-Rigid | executable Blender program | claude-opus-5-5 (high) | |
| Code-Articulated | Blender program + joint declaration | claude-opus-5-5 (high) |
| Task class | Wall-clock / unit | Cost / unit |
| Static | 1.9 min | $1.11 |
| Insertion | 1.8 min | $0.80 |
| Articulated | 12.5 min | $3.13 |
| Transport | 21.1 min | $4.29 |
| Machinery | 11.2 min | $2.78 |
| Overall | 10.1 min | $2.48 |
| Stage | Parameters | Values |
| Resolution admissibility | ||
| Supported voxel sizes | mm | |
| SDF fidelity | ||
| Collider layout | ||
| Maximum collider-layout optimization rounds |
| \tablehead Stage | \tablehead Model (effort) | \tablehead Budget and settings |
| Inverse scene generation | gpt-6-astra (high), gpt-6.1-sol (medium), gpt-6-luna (medium), claude-opus-5-5 (high), claude-sonnet-5-5 (medium), gemini-4-argon (high) | 900 s per call, 2,400 s and 40 calls per run, or 7,200 s and 80 calls with the larger budget, at most 6 scene-specification, 8 plan, and 3 outline rounds, 2 scene critics in fresh contexts, asset generation off unless stated |
| Asset generation in the loop | claude-opus-5-5 (high), claude-sonnet-5-5 fallback | Parametric path of the asset generator, at most 1,800 s per request, used only in the last two rows of Table 18 |
| \tablehead Full ISG | \tablehead Scene description | \tablehead Human review | |
| Input | Task and workplace | Scene description | Scene description |
| Task-plan stage | Generated and checked | Not run | Not run |
| Scene specification | Agent-authored | Agent-authored | Agent-authored |
| Deterministic checks | Plan and scene checks | Scene checks | Static verification |
| Assessment | Plan critic and scene critics | Reference-based fidelity checks | Human review |
| \tablehead Model (effort) | \tablehead Candidates | \tablehead Static | \tablehead All checks | \tablehead Calls | \tablehead Min |
| gpt-6-astra (high) | 1 | 0 | 0 | – | – |
| gpt-6.1-sol (medium) | 14 | 10 | 8 | 186 | 274 |
| gpt-6-luna (medium) | 11 | 6 | 6 | – | – |
| claude-opus-5-5 (high) | 24 | 21 | 16 | 105 | 147 |
| claude-sonnet-5-5 (medium) | 40 | 34 | 31 | 28 | 17 |
| gemini-4-argon (high) | 31 | 17 | 8 | 32 | 37 |
| \tablehead Stage | \tablehead Model | \tablehead Parameters |
| Shared | ||
| Reference image | gemini-3.1-flash-image | Optional, 16:9, 1K, at most three attempts, not used in the paired comparison |
| Image and view checks | gpt-5.6-sol (medium) | Panorama with four crops, five renders of the splat |
| Rubric judge | gpt-5.6-sol (medium) | Nine matched frames for each background in the paired comparison, four criteria scored 1 to 5, three shuffled repeats, pairwise comparison in both orders |
| Proprietary | ||
| Background | marble-1.1-plus | Image input, seed 17, full resolution, 4.32 M Gaussians |
| \tablehead Group | \tablehead Factors |
| Structure | Ceiling (5), floor (6), wall (6), window (5), door (5) |
| Lighting | Lighting (5), ceiling equipment (4) |
| Perimeter | Conveyor (5), shelving (5), cart (4), boxes (5), production equipment (6), safety (5) |
| Arrangement | Layout (5), density (3), condition (5), palette (7) |
| \tablehead Step | \tablehead Method | \tablehead Defaults |
| Faces | Floor and ceiling by RANSAC on the lowest and highest deciles, walls by blocked fraction | Normal within , |
| Alignment | Floor to , yaw from wall azimuths folded by | – |
| Floaters | Eq. ( 42 ) | : 0.2 (floor, ceiling), 0.1 (walls) of center distance |
| Placement region | Largest free rectangle, with the origin moved to its center | Band 0.2–1.2 m, wall margin 1 m |
| \tablehead Rubric score (1 to 5) | ||||||||
| \tablehead Background | \tablehead Mean | \tablehead Detail | \tablehead Integrity | \tablehead Realism | \tablehead Consistency | \tablehead Preferred (of 18) | \tablehead Usable (of 27) | \tablehead Stability |
| Proprietary API | 2.85 | 3.00 | 2.67 | 2.41 | 3.33 | 18 | 0 | 0.37 |
| Open-source chain | 1.78 | 1.67 | 1.78 | 1.33 | 2.33 | 0 | 0 | 0.98 |
| \tablehead Variants | \tablehead Per chunk | \tablehead Peak memory (GiB) | \tablehead Images/s |
| 4 | 4 | 4.5 | 129.0 |
| 16 | 16 | 9.8 | 135.0 |
| 32 | 32 | 15.9 | 140.6 |
| 32 | 8 | 6.4 | 130.3 |
| \tablehead Check | \tablehead Threshold |
| Static geometry | Fixture bounding-box overlap and ground penetration at most 5 mm, specified clearance of an item above its support at most 1 cm |
| Plan | Up to reachable predicate states, critic score at least 0.8 with no open concern |
| Variants | At least two distinct resolved layouts when variation is requested, every variant collision-checked |
| Specification fidelity | Position within 3 cm, contact gap within 1 cm, penetration within 5 mm, at most five attempts |
| Background prompt | Ten requirements present |
| Panorama geometry | Aspect ratio , width at least 1,024 pixels, seam ratio at most 4.0, pole standard deviation at most 24 (8-bit) |
| \tablehead Criterion | \tablehead Question to the judge |
| Scenes | |
| process_realism | Would this cell, its equipment, and its material flow exist on a real factory floor? |
| task_conditioning | Does the scene enable exactly the requested work, no more and no less? |
| entity_asset_fit | Is each entity the right kind of thing, at the right scale, with an asset whose category matches its role? |
| layout_coherence | Are positions, supports, clearances, access, and reach coherent? |
| meaningful_variation | Do the variation axes match what the request asks to vary, with meaningful ranges? |
| \tablehead Group | \tablehead The proposal is rejected when |
| Names | a name is missing from the scenario, is not declared in the proposal, does not resolve to an object in the scene, or is reserved for the evaluator although the task gives it to the policy. |
| Predicates and metrics | a predicate has invalid parameters, or a metric is not one the evaluator can compute. |
| Job lifecycle | the start, completion, and failure events do not form a valid job, or the rubric of an episodic task ends on anything other than one job. |
| Authority | a visual requirement is not bound to events, a success criterion refers to a visual or unmeasured requirement, or a requirement that the task needs is labeled unmeasured. |
| \tablehead Rule | \tablehead Rejects when | \tablehead Admits or only reports when |
| Single-step completion | never | the completion event holds for one step |
| Events on both edges | a predicate has events on its rising and its falling edge, or its declared justification names no condition of the evaluation | the rubric tolerates no protocol violation |
| Nonpositive drop threshold | a drop predicate uses a height threshold and no region failure is declared | the evaluation declares it a guard only, which is reported |
| Time limit beyond the run | the job timeout or the stall limit exceeds the length of the run | the limit equals the length of the run, which is reported |
| No failure event | never | no failure event is declared, which is reported |
| Case | Expected | Static | Reset | Idle | Accepted |
| Conveyor-loading control | None | Pass | Pass | Pass | Yes |
| Undeclared completion event | Static | Fail | NR | NR | No |
| Static-rule control (two-frame completion) | None | Pass | NR | NR | – |
| Add falling start-predicate event | Static | Fail | NR | NR | No |
| Add drop failure at floor ( ) | Static | Fail | NR | NR | No |
| Stall limit: 3,600 to 3,601 steps | Static | Fail | NR | NR | No |
| Model | Condition | Accepted | TP | FN | FP | TN |
| claude-opus-5-5 | Direct, no feedback | 3/3 | 12 | 0 | 3 | 30 |
| Direct, with feedback | 3/3 | 12 | 0 | 3 | 30 | |
| Questions, no feedback | 3/3 | 12 | 0 | 0 | 33 | |
| Questions, with feedback | 3/3 | 12 | 0 | 0 | 33 | |
| gpt-6.1-sol | Direct, no feedback | 0/3 | 12 | 0 | 3 | 30 |
| Direct, with feedback | 3/3 | 12 | 0 | 3 | 30 |
| Stage | Relevant content |
| Candidate 2 | transfer_complete combines support, on-top geometry, uprightness, and inlet membership in one conjunction. required_before_complete is empty, and the separately defined transfer_qualification is unused. |
| Simulated-user feedback | Candidate 2 accepts qualification only at entry and not earlier, and it also completes when qualification first holds after entry. Record qualification as an event, require it before or on completion, and complete on the rising inlet-entry event. |
| Candidate 3 | transfer_qualified becomes a one-step rising event for the support/upright conjunction and is added to required_before_complete . transfer_complete becomes inlet membership alone, with completion on its one-step rising event. |
| \tablehead Region | \tablehead Robots installed | \tablehead Robot density | \tablehead Handling workers | \tablehead Production workers |
| China | 354,000 | 166 | 6.1% | 15.8% |
| European Union | 60,500 | 231 | 5.33 M (2.6%) | 5.90 M (2.8%) |
| Japan | 36,200 | 446 | 5.05 M (7.4%) | 8.44 M (12.4%) |
| South Korea | 30,000 | 1,220 | 3.92 M (13.6%) | 2.94 M (10.2%) |
| United States | 38,400 | 307 | 2.95 M (1.9%) | 8.57 M (5.5%) |
| \tablehead Component | \tablehead Description |
| Mobile base | Differential drive, two driven wheels on prismatic suspension, four passive casters |
| Torso | Five-joint waist |
| Arms | Two arms with seven revolute joints each |
| Hands | Two five-finger hands with 20 joints each |
| Head | Three joints |
| Active joints | 64 (2 wheels, 5 waist, 14 arm, 40 hand, and 3 head joints) |
| \tablehead Setting | \tablehead Configuration |
| Frame/control period | s |
| Control interface | Joint position and velocity targets |
| Physics substeps | 4 per frame ( s each) |
| Evaluation rate | 60 Hz |
| \tablehead Group | \tablehead Predicates |
| Object state | In a region, upright within a tilt bound, below a height, at rest |
| Relation | Touching, supported by, on top of, next to, stable on (on top of, supported by, and at rest) |
| Joint | Within a range, completed a stroke |
| Composition | Conjunction |
| \tablehead Task | \tablehead | \tablehead (min) | \tablehead (min) |
| Conveyor loading | – | 5 | – |
| Flow-rack loading | 3 | 10 | 7 |
| Cart push | 8 | 20 | 18 |
| Jig loading | 8 | 20 | 18 |
| Press tending | 3 | 12 | 9 |
| Valve operation | 3 | 10 | 7 |
| \tablehead Task | \tablehead Region checks | \tablehead Start probes | \tablehead Replay traces |
| Flow-rack loading | 33/33 | 3/3 | 5/5 |
| Cart push | 30/30 | 4/4 | 5/5 |
| Jig loading | 25/25 | 3/3 | 5/5 |
| Press tending | 23/23 | 3/3 | 6/6 |
| Valve operation | 15/15 | 3/3 | 7/7 |
| Barcode routing | 59/59 | 4/4 | 4/4 |
| \tablehead Judge (effort) | \tablehead Frames | Step-up | Agreement | False success | |
| Time ramp | Ordered | 0.53 | – | – | – |
| gpt-6.1-sol (medium) | Ordered | 0.68 | 27% | 78% | 29% |
| Shuffled | 0.73 | 52% | – | – | |
| gpt-6-astra (high) | Ordered | 0.69 | 26% | 78% | 29% |
| Shuffled | 0.73 | 58% | – | – | |
| claude-opus-5-5 (high) | Ordered | 0.55 | 50% | 76% | 29% |
Robotics Simulation
Robotics simulation builds virtual robots and environments to model, train, and evaluate autonomous systems before real-world deployment.
See all 339 papers