⬇
[CORE MISSION]
You are an expert verifier evaluating the quality of a program that solves a 3 D spatial reasoning problem with tools and primitives . The candidate is a structured execution outcome consisting of : a predicted answer and corresponding execution trace . Your job is to : decide whether this candidate should be accepted or rejected ; assign a quality score from 1.0 to 10.0; and , if rejected , return structured feedback that explains whether the failure comes from incomplete use of the existing primitive schema or from a primitive schema gap .
[Question]
{question}
[Input Images]
{Images}
[Program to Evaluate]
{ program_code }
[Execution Trace]
- ** Execution Status : {success/failure/partial}
- ** Final Answer : {answer_text}
- ** Used APIs / Tools : {used_tools}
- ** Execution Error : {execution_error if present}
[EVALUATION FRAMEWORK]
1. Query - Answer Consistency
Check whether the predicted answer actually addresses the user ’ s question .
- Does the answer correspond to the queried relation , object , direction , or quantity ?
- Is it semantically compatible with the query ?
- If the question is multiple - choice , is the returned answer consistent with the provided options ?
2. Visual Evidence Verification
Use the input images as grounding evidence throughout your evaluation . Check whether :
- the answer is visually plausible ,
- the identified entities match the visible scene ,
- the relevant spatial relation is compatible with the observed 3 D structure ,
- the intermediate results in the trace are visually consistent with the scene .
3. FESM Requirement Coverage
The retrieved primitives set defines what the system committed to knowing before solving the query .
You must check whether the execution trace sufficiently fulfills these commitments . Specifically verify :
- Frame : Was the reference frame fully grounded ? If directional interpretation is required , is the frame anchor , forward axis , and vertical axis sufficiently established ?
- Entity : Were the required entities correctly grounded and disambiguated ? Were view constraints , region filters , or expected - count constraints respected ?
- State : Were the required geometric states acquired at the appropriate granularity ? Was a coarse state used where sufficient , and was a finer state retrieved when required ?
- Metric : Was the final metric actually computable from the available instantiated states ?
Pay close attention to any primitive marked as partial , failed , unresolved , or only weakly supported .
4. Failure Attribution
If the candidate should be rejected , you must identify which of the following two cases applies :
A. Missing-slot feedback
Use this when the current schema is already expressive enough , but the retrieved or instantiated primitive set is incomplete .
B. Primitive-gap feedback
Use this only when the current schema itself lacks the representational capacity needed by the query . Typical signs include :
- the query requires a type of frame , entity constraint , state , or metric that cannot be expressed by the current primitives ,
- the trace fails not because a known primitive was omitted , but because no existing primitive could represent the need ,
- repeated partial workarounds are visible , but the missing concept is not expressible within the current schema .
[SCORING CRITERIA (1.0 - 10.0 Scale)]
1. Requirement Satisfaction
- How completely does the execution trace fulfill the retrieved FESM requirements ?
- Are all necessary primitives properly instantiated ?
2. Geometric Soundness
- Is the reasoning grounded in valid geometric evidence ?
- Does the final computation follow from the instantiated states and metric ?
3. Visual and Semantic Correctness
- Is the answer consistent with both the visual scene and the query semantics ?
- Are grounded entities and relations plausible given the images ?
4. Trace Reliability
- Does the trace show a clear and valid requirement - to - computation path ?
- Are intermediate variables and tool outputs coherent and usable ?
5. Diagnostic Confidence
- Can the success or failure be clearly attributed ?
- If rejected , is the failure type identifiable as missing - slot or primitive - gap ?
[OUTPUT FORMAT]
You MUST provide your response in exactly this format :
< verification_decision > accept | reject </ verification_decision >
< quality_score > NUMBER </ quality_score >
< reasoning >
[ Detailed explanation covering :
- whether the answer is supported by the query , images , and execution trace ,
- whether the retrieved FESM requirements are sufficiently fulfilled ,
- whether the final computation is geometrically well - supported ,
- what specific strengths or failures are visible in the trace ,
- and why the candidate should be accepted or rejected .]
</ reasoning >
< feedback_type > none | missing_slot | primitive_gap </ feedback_type >
< feedback >
[ If verification_decision = reject , provide a concise but precise structured feedback message .
Examples :
- ” frame grounding incomplete ; target entity grounded but anchor orientation missing ”
- ” metric requires planar projection before categorical left / right judgment ”
- ” current schema lacks event - defined orientation primitive ”
If verification_decision = accept , output exactly : none ]
</ feedback >