Appendix C Quality Control and Human Validation
C.1 Workspace Validation Programmatic checks. We first check that each generated output follows the required JSON format and contains the required fields. For extracted numerical claims, we compare the reported values with the source text. We also check that referenced facts, collaborators, and channels exist, that events follow the planned order, and that referenced documents are available. For events involving errors and corrections, we check that each error concerns the intended fact and uses the specified incorrect value. We then verify that the error is introduced, detected, and corrected in the planned order. For Slack messages, we check that each planned event has exactly one corresponding message, sent by the specified collaborator at the specified time. Regeneration with validation feedback. When an output fails validation, we provide the model with the original generation inputs and accumulated validation errors. The model returns a revised output and this process continues until the output passes validation or the retry limit is reached. ⬇ — USER — Your answer was rejected ({{ rejection_count_phrase }} so far ). Every one of these must hold at once – the latest rejection is marked *: {{ validation_error_list }} Fix every listed problem without reintroducing an earlier one , and return the full corrected JSON object . Conversational naturalness. For each channel and day with at least six planned events, an LLM reviews the naturalness of the generated Slack conversation. A score of 2 or lower on a five-point scale, together with reported problems, triggers one additional rewrite. The rewrite uses the original Slack generation prompt with the review feedback, and the resulting messages undergo the same programmatic checks. ⬇ You review a day of chat messages for naturalness : do they read like real coworkers , or like a script reading numbers aloud ? — USER — Channel {{ channel }}: {{ messages }} Score 1-5 (5 = indistinguishable from a real team chat ). List concrete problems ( stilted phrasing , everybody narrating , no reactions , repeated boilerplate , values recited without context ). Return {" score ": n , " problems ": […]}. The review feedback is appended to the channel context in the Slack generation prompt as follows. ⬇ A reviewer flagged the previous draft : {{ reviewer_problems }}. Fix these .
C.2 QA Validation Prompts We validate each QA item through three complementary stages: evidence-grounded answering, candidate–gold comparison, and full-project validation. The first stage produces an independent answer from the cited records, the second compares it with the reference answer, and the third examines the item against all project records. Together, these stages assess evidence sufficiency, answer correctness, question ambiguity, and answerability. C.2.1 Evidence-grounded Answering An independent solver answers the question using only the cited records, without seeing the reference answer. It returns an answer and indicates whether the supplied evidence settles the question. This response provides an initial check of whether the cited records support the requested answer. ⬇ The records are the internal records ( chat , email , Notion tracker / decisions / meetings , attached documents and images ) of one fictional company ’ s project . The benchmark asks questions as the whole team would answer them after the project ended , with every record visible . Output exactly one JSON object and nothing else . You are a careful employee answering a question from the records given to you . Use only these records ; do not guess from general knowledge . Give exact values , names and times as they appear . If the records do not settle the question , say so and set knowable = false ( name the closest recorded fact , but do not present it as the answer ). Question : { q } Cited records : { cited records } C.2.2 Candidate–Gold Comparison The comparison stage evaluates the independent answer against the reference answer on the specific object requested by the question. It allows equivalent wording and appropriate rounding while distinguishing substantive disagreements, missing required qualifiers, and incorrect abstention. The resulting judgment is passed to the full-project validation stage alongside the solver’s answer. ⬇ You grade one candidate answer against the gold answer . Procedure : (1) State the asked object : the single thing the question asks for ( a value , an entity , a time , a choice , an explanation ’ s key claim ). (2) Compare candidate and gold ONLY on the asked object . Tolerate wording , rounding at the precision the question implies , extra correct detail , and missing secondary explanation . (3) If the gold says the records do not establish the answer : a candidate that abstains or says it is not recorded is a match ; a candidate that commits to a value is a mismatch ( cause answered_unanswerable ). If the gold gives an answer and the candidate abstains : mismatch ( cause candidate_abstained ). Return {" asked_object ": str , " verdict ": " match " | " partial " | " mismatch ", " cause ": " none " | " different_value " | " wrong_entity " | " missing_required_part " | " candidate_abstained " | " answered_unanswerable " | " other ", " reason ": str }. partial = asked object right but a required qualifier ( unit , one of two required items , direction ) wrong or missing . Question : { q } Gold answer : { gold } Gold status : { answerable | unanswerable } Candidate answer : { Stage -1 answer } Candidate says knowable : { Stage -1 knowable } C.2.3 Full-project Validation The final stage examines the question, reference answer, and cited evidence against all records of the project. It checks whether the wording admits an equally valid competing answer, whether the reference answer’s values are supported, whether the answerability label is consistent with the records, and whether the cited evidence is sufficient. The independent solver’s response informs this assessment, but does not determine it. The validator returns a pass, fix, or drop decision and specifies concrete edits when the item can be repaired. ⬇ You audit one benchmark item against ALL records of its project . You also see what an independent solver answered when given only the cited records . Decide , reading the records and images yourself : 1. competing_answer : Could a careful employee defend a materially different answer to the question AS WORDED ( a different referent , version , time , or reading of an ambiguous term )? equally_valid is true only if the wording does not let a reasonable employee decide which is meant . 2. numbers : For every number , date and time in the gold , check that the records give that value for the thing the gold attaches it to ; list any that are wrong or unverifiable . 3. status : If the gold says the records do not establish the answer , check whether some record actually gives what is asked . If the gold gives an answer , check whether the question presupposes something the records contradict . 4. evidence_sufficient : Can the asked object be derived from the cited records alone ? The solver ’ s result is a hint , not proof . Then give verdict : pass ( item is fine ) | fix ( salvageable by a concrete edit : wording , gold detail , evidence list ) | drop ( gold wrong or the question cannot be made to pick one answer ). category : gold_wrong | ambiguous_question | evidence_insufficient | unanswerable_is_answerable | leak | none . Return {" competing_answer ": {" answer ": str | null , " records ": [ ids ], " equally_valid ": bool }, " numbers ": {" wrong_or_unverified ": [ str ], " note ": str }, " status ": {" ok ": bool , " note ": str }, " evidence_sufficient ": bool , " verdict ": str , " category ": str , " fix ": {" q ": str | null , " gold ": str | null , " evidence_add ": [ ids ]} | null , " note ": str } Question : { q } Gold answer : { gold } Gold status : { answerable | unanswerable } Solution : { solution - stage JSON } Cited records : { evidence ids + solution record_ids , deduplicated } Stage -1 solver : answer =’{ Stage -1 answer }’ knowable ={…} -> grade { match | partial | mismatch } ({ cause }): { reason } Cited records : { cited records } All project records : { full project records }