| Test | AcCoRD -Shop prompt | AcCoRD -Travel prompt |
| test000: Response relevance | Does the candidate response engage with at least one thing the assistant said or asked in its last message? ( yes if the response addresses something from the assistant’s last message; no if the response is about an unrelated topic) | Does the candidate response engage with at least one thing the assistant said or asked in its last message? ( yes if the response addresses something from the assistant’s last message; no if the response is about an unrelated topic) |
| test001: Preference value consistency | If the assistant asks the user for a preference value and the user’s instructions specify a value for that preference, are all preference values stated in the candidate response consistent with the user’s instructions ? ( yes if every stated value matches the instructions; no if any stated value contradicts the instructions; n/a if the assistant did not ask for a preference value, or the response did not address the request) | If the assistant asks the user for a preference value and the scenario specifies a value for that preference, are all preference values stated in the candidate response consistent with the scenario ? ( yes if every stated value matches the scenario; no if any stated value contradicts the scenario; n/a if the assistant did not ask for a preference value, or the response did not address the request) |
| test002: Contradiction acknowledgment | If the assistant describes a specific product whose attribute contradicts a preference the user has already revealed earlier in this conversation, does the candidate response point out that contradiction? ( yes if the contradiction is pointed out; no if the contradiction is ignored or the response accepts the product; n/a if the assistant did not describe a specific product, or no described attribute contradicts an already-revealed preference) | If the assistant describes a specific hotel/flight/activity option whose attribute contradicts a preference the user has already revealed earlier in this conversation, does the candidate response point out that contradiction? ( yes if the contradiction is pointed out; no if the contradiction is ignored or the response accepts the option; n/a if the assistant did not describe a specific option, or no described attribute contradicts an already-revealed preference) |
| test003: Backoff acceptance | If the assistant has indicated that the user’s primary preference is unavailable AND the user’s instructions define a backoff for that preference, does the candidate response accept the backoff rather than continuing to insist on the unavailable primary? ( yes if the response accepts the backoff or otherwise shows appropriate flexibility; no if the response continues to insist on the unavailable primary; n/a if the assistant has not indicated that the primary is unavailable, or the user’s instructions define no backoff for that preference) | If the assistant has indicated that the user’s primary preference is unavailable AND the scenario defines a backoff for that preference, does the candidate response accept the backoff rather than continuing to insist on the unavailable primary? ( yes if the response accepts the backoff or otherwise shows appropriate flexibility; no if the response continues to insist on the unavailable primary; n/a if the assistant has not indicated that the primary is unavailable, or the scenario defines no backoff for that preference) |
| test004: Premature backoff disclosure | If the candidate response reveals a backoff preference, had the assistant already indicated earlier in the conversation that the primary preference could not be satisfied? ( yes if the backoff is only revealed after the primary was confirmed unavailable; no if the backoff was revealed before the primary was confirmed unavailable; n/a if the response does not reveal a backoff preference) | If the candidate response reveals a backoff preference, had the assistant already indicated earlier in the conversation that the primary preference could not be satisfied? ( yes if the backoff is only revealed after the primary was confirmed unavailable; no if the backoff was revealed before the primary was confirmed unavailable; n/a if the response does not reveal a backoff preference) |
| test005: Triggered preference revelation | If the user’s instructions include a preference marked ‘only reveal if mentioned’ AND the assistant’s last message explicitly surfaces the specific target value for that preference (e.g., names the exact value, or presents a product whose attribute equals that value), does the candidate response reveal that preference? ( yes if the matching preference is revealed; no if the response withholds or misstates the preference when it should have been revealed; n/a if no such preference applies, or the assistant did not surface the specific target value) | If the scenario includes a preference marked ‘only reveal if mentioned’ AND the assistant’s last message explicitly surfaces the specific target value for that preference (e.g., names the exact value, or describes an option whose attribute equals that value), does the candidate response reveal that preference? ( yes if the matching preference is revealed; no if the response withholds or misstates the preference when it should have been revealed; n/a if no such preference applies, or the assistant did not surface the specific target value) |