A Benchmark and Layered Evaluation Protocol for Evidence-Grounded Shared Belief Updating in Connected Robotics
Organizations: Tohoku University, Sendai, Japan
Abstract
Connected physical systems fuse observations from robots, IoT sensors, and edge-AI modules into a shared account of the world, yet most evaluations score only the final answer or action and cannot tell whether a method attributed the evidence, updated its belief, handled contradictions, and justified its decision. We argue that shared reasoning in Connected Robotics should be evaluated as an evidence-to-belief trajectory, and introduce the Connected Robotics Evidence Graph (CREG) Benchmark to make that trajectory measurable. CREG comprises 350 deterministically generated scenarios across seven regimes (justified revision, belief preservation, relation association, contradiction onset and resolution, stale/delayed evidence, and source reliability), each exposing a normative posterior reference at every evidence prefix, validated by an independent log-odds checker and screened by a blinded expert audit. Nine representation conditions separate relation content from serialization, ordering, and metadata. Evaluating four open-weight instruction models and six transparent fusion baselines yields three findings. First, entity grounding accounts for a substantial part of the weak-model deficit. Second, deleting semantic relations costs 15.3 points of strict belief-state accuracy on relation-dependent scenarios (Holm-corrected p<0.001), but an information-matched structured list outperforms the full graph and shuffling the graph does not hurt: relation content matters where required, while graph serialization confers no consistent advantage. Third, belief accuracy is strongly non-monotonic along the evidence sequence, with mid-trajectory failures that endpoint-only evaluation cannot detect. All scenarios, references, prompts, metrics, and analysis code are released for offline, hardware-free reproduction.
Figures & tables
| Belief-R | BeliefTrack † | BayesBench † | ALFRED/TEACh/B-1K | OpenEQA | CREG | |
| [ 17 ] | [ 18 ] | [ 19 ] | [ 20 , 21 , 22 ] | [ 23 ] | (ours) | |
| Evidence setting | ||||||
| Sequential evidence with revision pressure | ✓ | ✓ | ✓ | – | – | ✓ |
| Reports attributed to distinct physical sources with reliability | – | – | – | – | – | ✓ |
| Per-report confidence, provenance, and timestamps | – | – | – | – | – | ✓ |
| Stale, delayed, and superseded evidence semantics | – | – | – | – | – | ✓ |
| System | Trajectory score [95% CI] | Strict state acc. | Grounding | Strict valid |
|---|---|---|---|---|
| Qwen-1.5B / JSON (diagnostic) | 0.397 [0.380, 0.413] | 0.331 | 0.786 | 0.807 |
| Mistral-7B / JSON (diagnostic) | 0.239 [0.229, 0.248] | 0.221 | 0.694 | 0.897 |
| Qwen-7B / JSON | 0.797 [0.789, 0.806] | 0.831 | 1.000 | 1.000 |
| Gemma-12B / JSON | 0.801 [0.794, 0.808] | 0.729 | 1.000 | 0.999 |
| Qwen-7B / structured list | 0.842 [0.829, 0.854] | 0.682 | 0.966 | 0.999 |
| Qwen-7B / flat edge-removed | 0.774 [0.760, 0.788] | 0.605 | 0.917 | 0.999 |
| Contrast (A B) | Metric | 95% CI | Holm | |
|---|---|---|---|---|
| Relation content | ||||
| Full graph graph w/o relations | traj. score | |||
| same, relation-association regime | traj. score | |||
| same, relation-association regime | strict acc. | |||
| Serialization and order | ||||
| Full graph flat edge-removed | traj. score | |||
| Fusion baseline | Traj. score | Strict acc. | Preserv. | Revision |
|---|---|---|---|---|
| unweighted majority | 0.680 | 0.631 | 0.333 | 0.655 |
| confidence weighted | 0.688 | 0.768 | 0.667 | 0.786 |
| source reliability weighted | 0.700 | 0.856 | 1.000 | 0.856 |
| recency weighted | 0.691 | 0.815 | 0.667 | 0.857 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Budget | Strict valid | Repair | Grounding (pooled) | Grounding (core) | Truncated | Mean traj. | Proj. hours |
|---|---|---|---|---|---|---|---|
| 256 | 0.928 | 0.021 | 0.928 | 0.997 | 0.000 | 0.572 | 39.4 |
| 384 | 0.954 | 0.000 | 0.912 | 0.990 | 0.000 | 0.573 | 43.6 |
| 512 | 0.961 | 0.003 | 0.930 | 0.986 | 0.000 | 0.589 | 45.8 |
| Condition | Mean chars | Max chars | Rel. to full graph |
|---|---|---|---|
| canonical_json | 5284 | 6090 | 0.843 |
| structured_list | 5651 | 6513 | 0.901 |
| flat-edge-removed-relations | 6298 | 7657 | 1.004 |
| graph_without_relations | 6058 | 7430 | 0.966 |
| graph | 6271 | 7624 | 1.000 |
| no-provenance | 5834 | 6821 | 0.930 |
| Contrast | Metric | 95% CI | Holm | ||
|---|---|---|---|---|---|
| graph_vs_graph_without_relations | trajectory_score | +0.088 | [+0.072, +0.103] | 0.0000 | 0.0001 |
| graph_vs_graph_without_relations @ relation_association | trajectory_score | +0.020 | [+0.002, +0.037] | 0.0288 | 0.0577 |
| graph_vs_edge_removed | trajectory_score | +0.041 | [+0.028, +0.053] | 0.0000 | 0.0001 |
| graph_vs_canonical_json | trajectory_score | +0.017 | [+0.010, +0.024] | 0.0000 | 0.0001 |
| structured_list_vs_canonical_json | trajectory_score | +0.045 | [+0.033, +0.056] | 0.0000 | 0.0001 |
| shuffled_vs_graph | trajectory_score | +0.034 | [+0.025, +0.042] | 0.0000 | 0.0001 |
| Regime | Constraint | Scenarios |
|---|---|---|
| justified revision | applicable evidence reverses the top belief | 50 |
| belief preservation | weak counterevidence does not reverse the top belief | 50 |
| relation association | measures relation is necessary to associate sensor evidence with the target | 50 |
| contradiction onset | opposing claims in one epoch create an active contradiction | 50 |
| contradiction resolution | correction evidence supersedes a conflicting claim | 50 |
| stale delayed evidence | older-epoch late evidence is stale while delayed current-epoch evidence applies | 50 |
| Diagnostic | Mean std (3 seeds) |
|---|---|
| Final policy accuracy (= deterministic row; validated action) | 0.357 0.000 |
| Override rate | 0.637 0.003 |
| Correct-override rate (unjustified proposal corrected) | 0.143 0.000 |
| Harmful-override rate (defensible proposal overridden) | 0.494 0.003 |
| Proposals accepted without correction | 0.363 0.003 |
| Explanation coverage (model-side) | 0.354 0.016 |
| System | Prefixes | Bin agreement | Pred. high-bin | Ref. high-bin |
|---|---|---|---|---|
| gemma-3-12b-it-canonical_json | 3297 | 0.787 | 0.730 | 0.711 |
| gemma-3-12b-it-graph | 3276 | 0.885 | 0.764 | 0.693 |
| gemma-3-12b-it-graph_without_relations | 3249 | 0.864 | 0.758 | 0.692 |
| gemma-3-12b-it-structured_list | 3271 | 0.874 | 0.688 | 0.654 |
| mistral-7b-v0.3-canonical_json | 1054 | 0.710 | 0.310 | 0.226 |
| qwen2.5-1.5b-canonical_json | 1727 | 0.768 | 0.660 | 0.540 |
| System | prem. rev. | fail. rev. | stale dom. | unrel. dom. | contr. om. | inc. attr. | rel. fail. | malformed | miscal. |
|---|---|---|---|---|---|---|---|---|---|
| gemma-3-12b-it-canonical_json | 673 | 252 | 148 | 1 | 292 | 551 | 223 | 3 | 141 |
| gemma-3-12b-it-graph | 523 | 168 | 37 | 17 | 300 | 671 | 258 | 24 | 286 |
| gemma-3-12b-it-graph_without_relations | 551 | 157 | 59 | 25 | 302 | 684 | 229 | 51 | 295 |
| gemma-3-12b-it-structured_list | 530 | 186 | 58 | 7 | 391 | 687 | 280 | 29 | 210 |
| mistral-7b-v0.3-canonical_json | 1549 | 1153 | 150 | 317 | 764 | 640 | 394 | 2344 | 2536 |
| qwen2.5-1.5b-canonical_json | 1340 | 950 | 130 | 250 | 994 | 848 | 291 | 1635 | 1868 |
Evidence-Grounded Reasoning
Evidence-grounded reasoning studies how AI systems connect conclusions to traceable, relevant evidence and recognize when support is incomplete.
See all 93 papers