Large language model (LLM) assistants increasingly help users with everyday activities that involve time-dependent constraints. However, users may not mention these constraints in later requests, requiring assistants to recognize and adjust their advice accordingly. To evaluate such capability, we introduce TASC (Temporal Action Safety in Conversation), a benchmark that evaluates whether LLMs apply time-dependent constraints when responding to conversational requests. TASC consists of 3,418 inputs derived from 128 templates, along with a separate set of 33 scenarios designed to compare models' ability to calculate elapsed time with their ability to apply that information. Across 13 models, assistants account for time-dependent constraints in only 30.5% of cases where the proposed action violates a rule, while rarely warning users unnecessarily. More importantly, models fail to apply these constraints in 62.1% of cases despite correctly calculating the elapsed time. Explicitly providing the timing resolves only about half of these failures, revealing a gap between temporal reasoning and its application in conversational advice.
Figures & tables
Figure 1: Four examples from the Tasc-p of Tasc . Tasc covers 23 everyday domains, and we evaluate 13 models. The “Rule ∣ Time” row gives the time-dependent rule that the model should apply and the time elapsed (or remaining) at the final user message. (i) : the conversation history shared by all conditions. (ii) ( Condition A ): asked directly how much time has passed, models answer correctly in 85.0% of cases. (iii) ( Condition B ): given an ordinary request in which the user is about to break, or has already broken, the rule, models flag the violation in only 33.2% of cases and usually just answer the request. (iv) ( Condition C ): even when the timing is stated explicitly in the system prompt, models still miss 37.4% of violations.
Factor
Level
# Inputs (%)
# Templates
Timing
Before
1,081 (31.6)
116
Near
1,497 (43.8)
119
After
820 (24.0)
116
Prior conflict
20 (0.6)
10
Request
Ongoing
1,213 (35.5)
110
Done
1,206 (35.3)
113
Table 1: Distribution of the 3,418 Tasc-m inputs over the construction factors. Templates counts the distinct templates with at least one input at that level; one template contributes to several levels.
Table 3: Tasc-m scores (%) on the 3,397 inputs answered by all models. Note that in non-violation controls \labelbox catchcolorCatch rate represents false alarms, so the metric is the lower the better. Rows are sorted by Condition B \labelbox catchcolorCatch rate. Cells are darkened proportionally to their value. Average is the unweighted mean over the 13 models.
Condition A outcome
Condition B misses
Condition C recovered
Still missed
Correct
226
119 (52.7%)
107
Incorrect
60
12 (20.0%)
48
Table 4: Tasc-p recovery under Condition C , split by the outcome of Condition A . Percentages use the Condition B misses in each row.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Four more examples of different domains from Tasc .
Dimension
# Scen.
n
Condition A
Condition B
Condition C
Miss ∣ Condition A
Rec.
Long history
13
168
88.7
25.0
54.2
71.8
41.3
Implicit anchor
6
78
83.3
51.3
73.1
41.5
47.4
Interleaved
5
65
80.0
33.8
67.7
61.5
51.2
Incorrect claim
3
39
79.5
35.9
66.7
54.8
48.0
Timer reset
3
39
92.3
48.7
71.8
47.2
50.0
Schedule drift
2
26
84.6
11.5
53.8
86.4
47.8
Appendix
Table 5: Tasc-p scores (%) by structural dimension. # Scen. counts templates and n counts complete model–scenario tuples. Miss ∣ Condition A is the share of correct Condition A answers with a Condition B miss. Rec. (recovery) is the share of Condition B misses caught under Condition C .
Experiment
Condition A
Condition B
Condition C
Non-violation
–
Metric
Accuracy ↑
\labelbox catchcolorCatch rate ↑
\labelbox catchcolorCatch rate ↑
\labelbox catchcolorCatch rate ↓
Recovery ↑
Claude Opus 5
100.0
100.0
100.0
0.0
–
Kimi K3
75.8
18.2
45.5
0.0
37.0
Gemini 3.7 Flash
97.0
63.6
87.9
0.0
66.7
Claude Sonnet 5
93.9
36.4
84.8
3.0
81.0
GPT-5.5
100.0
48.5
90.9
0.0
82.4
Appendix
Table 6: Tasc-p scores (%) by model. Condition A measures accuracy; Condition B and Condition C calculate \labelbox catchcolorCatch rates. Note that in non-violation controls \labelbox catchcolorCatch rate represents false alarms, so the metric is the lower the better. Recovery is the share of Condition B misses caught under Condition C (– when a model has no Condition B misses). Rows follow the order of Table 3 . Cells are darkened proportionally to their value. Each model has 33 complete tuples except DeepSeek-V4-Pro, which has 32.
Figure 3: Tasc-p scores (%) for the 13 models. Condition A measures accuracy; Condition B and Condition C calculate \labelbox catchcolorCatch rates. Each model has 33 complete tuples except DeepSeek-V4-Pro, which has 32.
Rule / direction
\labelbox catchcolorCatch / n
Rate (%)
Minimum wait / early
1,991 / 5,551
35.9
Deadline / late
944 / 3,003
31.4
Target window / early
325 / 1,677
19.4
Target window / late
189 / 1,040
18.2
Appendix
Table 7: Tasc-m Condition B \labelbox catchcolorCatch rates on violations by rule type and direction, using the shared scored inputs. n counts model–input responses.
\labelbox catchcolorCatch
\labelbox hedgecolorHedge
\labelbox askcolorAsk
\labelbox proceedcolorProceed
\labelbox catchcolorCatch
401
1
0
23
\labelbox hedgecolorHedge
5
0
1
9
\labelbox askcolorAsk
0
0
0
0
\labelbox proceedcolorProceed
5
2
1
538
Appendix
Table 8: Original judge labels. Rows: Claude Sonnet 5; columns: GPT-5.6 Luna.
κ
95% CI
n
Agr. (%)
All sampled (4-label)
0.708
0.60–0.82
144
84.7
Judge-labeled, vs. final
0.735
0.62–0.84
126
85.7
Violations ( Condition B & Condition C )
0.768
0.65–0.88
114
87.7
Condition C
0.846
0.71–0.96
49
91.8
Condition B violations
0.707
0.54–0.87
65
84.6
Non-violation controls
0.330
− 0.06–0.69
30
73.3
Appendix
Table 9: Human–LLM agreement with the recorded labels. κ is Cohen’s κ ; Agr. is agreement.
Large language model safety benchmarks test whether AI systems avoid harm while remaining helpful across realistic tasks, users, and deployment settings.
Department of Computer Science, Princeton University · Department of Operations Research and Financial Engineering, Princeton University · Department of Industrial Engineering and Operations Research, Columbia University +3