cs.AINov 2, 2026
SaveBeyond Answers, Traces, and Efficiency: Evidence for Adaptive Reasoning
Organizations: Chungnam National University · Gwangju Institute of Science and Technology
Abstract
Current AI systems are evaluated as reasoners through final-answer accuracy, reasoning traces, process supervision, and computational or human-relative efficiency. Yet these measures do not establish whether reasoning adapts when tasks or evidence change. We distinguish across-task adaptivity, where acquired abstractions transfer or recombine across novel problems, fromwithin-task adaptivity, where hypotheses and actions are revised in response to new observations, feedback, or counterexamples. Using ARC-AGI as a diagnostic case study rather than an alignment benchmark, we examine what existing measures leave unresolved when test-time search, sampling, or adaptation affects performance. We propose intervention-oriented evaluation that tests abstraction transfer and hypothesis revision under controlled changes to tasks and evidence, with explicit reporting of test-time procedures and budgets. This perspective supports evaluating systems for robustness, responsiveness to corrective feedback, and auditability under changing conditions.
Figures & tables
| Evidence | What it establishes / leaves unresolved |
|---|---|
| Outcome | Measures task success; does not identify how the solution was obtained. |
| Trace | Exposes expressed reasoning; does not establish behavioral grounding or adaptation. |
| Process | Evaluates intermediate-step quality; does not test responsiveness to changed evidence. |
| Resource | Reports compute and search budgets; does not separate stronger reasoning from larger search. |
| Human-relative | Measures efficiency against humans; does not identify what was acquired or revised. |
Table 1: Complementary evidence provided by existing reasoning evaluations and the questions each leaves unresolved.
| Evidence | Minimum reporting |
|---|---|
| Outcome | Task success pre- and post-intervention. |
| Reasoning artifact | Relevant trace, hypothesis, proposal, program, action, or other observable intermediate decision. |
| Test-time budget | Samples, tokens, search steps, refinement rounds, interactions, tools, or other inference-time resources. |
| Across-task | What structure is preserved or changed, the paired task variants, and the corresponding behavioral response. |
| Within-task | What evidence is added or modified, the decision before and after the intervention, and subsequent behavior. |
| Human reference | Action efficiency, trajectory, or strategy information when used as a diagnostic comparison. |
Table 2: Minimum reporting for interpreting intervention-oriented evidence of adaptive reasoning.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Approach | Variation or intervention | Main evaluative target | Connection to adaptive reasoning |
|---|---|---|---|
| Contrast Sets | Local changes to an input or task instance | Sensitivity of model predictions to meaningful local contrasts | Provides a basis for constructing controlled task variants, which we use to test whether behavior follows preserved or changed task structure. |
| CheckList | Behavioral test conditions and targeted input perturbations | Capability-specific behavioral testing | Motivates targeted tests and controls that isolate the behavior associated with a specified capability. |
| ConceptARC | Concept-centered families of ARC tasks | Generalization across tasks that share a concept | Closely related to across-task adaptivity, which additionally distinguishes concept-preserving from concept-changing interventions. |
| Belief-R | Newly supplied textual evidence | Revision or maintenance of a prior conclusion | Closely related to within-task adaptivity, which applies the same distinction to observable decisions beyond textual conclusions. |
| Thinking-draft faithfulness | Intermediate reasoning content | Causal influence of reasoning on later reasoning or answers | Tests dependence on intervened reasoning content, whereas our within-task evaluation changes task-relevant evidence and examines the resulting revision. |
| Our framework | Task structure or task-specific evidence | Abstraction transfer and evidence-sensitive revision | Evaluates across-task and within-task adaptivity under controlled changes while making test-time resources explicit. |
Table 3: Relationship to closely related evaluation paradigms. The comparison focuses on the information varied by each approach and the behavioral question it is designed to answer.
Figure 3: Across-task adaptivity illustrated with related ARC-AGI-1 tasks. The base task, 1e0a9b12 , applies pixel-level downward relocation. Under the evaluator-specified abstraction, 3618c87e is concept-preserving: the manipulation unit and downward-relocation rule remain unchanged despite the introduction of a structured gray background. In 25ff71a9 , the manipulation unit changes from individual pixels to a connected object, yielding an object-level downward-relocation rule and a concept-changing variant.
Figure 4: Within-task adaptivity in ARC-AGI-3 task r11L . (a) The initial state of Level 1. (b) Clicking a black cell moves the current white endpoint, providing confirming evidence about endpoint control. (c) Clicking the dark-gray endpoint swaps the white and dark-gray roles, contradicting the hypothesis that the dark-gray endpoint is fixed and motivating a targeted revision. (d) After Level 1 is cleared, Level 2 introduces additional structures while previously acquired interaction rules remain useful. The sequence illustrates how supported knowledge can be retained while task-specific hypotheses are revised or extended as new evidence becomes available.
| Branch | Evidence | Expected response |
|---|---|---|
| Contradictory | Clicking the dark-gray endpoint swaps the white and dark-gray roles. | Revise the claim that the dark-gray endpoint is fixed. |
| Confirming | Clicking a black cell moves the white endpoint to that cell. | Retain the claim that the white endpoint is movable. |
| Irrelevant | Clicking a light-gray obstacle produces no endpoint movement. | Preserve the endpoint-role hypothesis. |
| No added evidence | No diagnostic interaction occurs. | Continue under the same test-time budget. |
Table 4: Same-history branching in ARC-AGI-3 task r11L . Each branch specifies the expected response to different evidence about the current hypothesis.
| Stage | Across-task | Within-task |
|---|---|---|
| Before intervention | Specify the target structure and the properties to be preserved. | Record the available evidence and an observable decision , such as a hypothesis, program, plan, action, or tool call. |
| Intervention | Modify a surface property or the target structure while controlling other relevant properties. | Add or modify evidence to produce and identify which part of the new evidence bears on. |
| Expected response | Specify whether behavior should remain stable, transform predictably, or change with the modified structure. | Specify whether the implicated decision should be revised or retained given the new evidence. |
| Observed response | Compare behavior across the paired task variants. | Compare the decision before and after the intervention and record subsequent behavior . |
| Controls | Use matched variants and report the test-time budget. | Use no-intervention or irrelevant-evidence branches when possible, and report the test-time budget. |
Table 5: Operationalization of across-task and within-task adaptivity.