Large language model (LLM) assistants increasingly help users with everyday activities that involve time-dependent constraints. However, users may not mention these constraints in later requests, requiring assistants to recognize and adjust their advice accordingly. To evaluate such capability, we introduce TASC (Temporal Action Safety in Conversation), a benchmark that evaluates whether LLMs apply time-dependent constraints when responding to conversational requests. TASC consists of 3,418 inputs derived from 128 templates, along with a separate set of 33 scenarios designed to compare models' ability to calculate elapsed time with their ability to apply that information. Across 13 models, assistants account for time-dependent constraints in only 30.5% of cases where the proposed action violates a rule, while rarely warning users unnecessarily. More importantly, models fail to apply these constraints in 62.1% of cases despite correctly calculating the elapsed time. Explicitly providing the timing resolves only about half of these failures, revealing a gap between temporal reasoning and its application in conversational advice.
Figures & tables
Figure 1: Four examples from the Tasc-p of Tasc . Tasc covers 23 everyday domains, and we evaluate 13 models. The “Rule ∣ Time” row gives the time-dependent rule that the model should apply and the time elapsed (or remaining) at the final user message. (i) : the conversation history shared by all conditions. (ii) ( Condition A ): asked directly how much time has passed, models answer correctly in 85.0% of cases. (iii) ( Condition B ): given an ordinary request in which the user is about to break, or has already broken, the rule, models flag the violation in only 33.2% of cases and usually just answer the request. (iv) ( Condition C ): even when the timing is stated explicitly in the system prompt, models still miss 37.4% of violations.
Factor
Level
# Inputs (%)
# Templates
Timing
Before
1,081 (31.6)
116
Near
1,497 (43.8)
119
After
820 (24.0)
116
Prior conflict
20 (0.6)
10
Request
Ongoing
1,213 (35.5)
110
Done
1,206 (35.3)
113
Table 1: Distribution of the 3,418 Tasc-m inputs over the construction factors. Templates counts the distinct templates with at least one input at that level; one template contributes to several levels.
Table 3: Tasc-m scores (%) on the 3,397 inputs answered by all models. Note that in non-violation controls \labelbox catchcolorCatch rate represents false alarms, so the metric is the lower the better. Rows are sorted by Condition B \labelbox catchcolorCatch rate. Cells are darkened proportionally to their value. Average is the unweighted mean over the 13 models.
Condition A outcome
Condition B misses
Condition C recovered
Still missed
Correct
226
119 (52.7%)
107
Incorrect
60
12 (20.0%)
48
Table 4: Tasc-p recovery under Condition C , split by the outcome of Condition A . Percentages use the Condition B misses in each row.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Four more examples of different domains from Tasc .
Dimension
# Scen.
n
Condition A
Condition B
Condition C
Miss ∣ Condition A
Rec.
Long history
13
168
88.7
25.0
54.2
71.8
41.3
Implicit anchor
6
78
83.3
51.3
73.1
41.5
47.4
Interleaved
5
65
80.0
33.8
67.7
61.5
51.2
Incorrect claim
3
39
79.5
35.9
66.7
54.8
48.0
Timer reset
3
39
92.3
48.7
71.8
47.2
50.0
Schedule drift
2
26
84.6
11.5
53.8
86.4
47.8
Appendix
Table 5: Tasc-p scores (%) by structural dimension. # Scen. counts templates and n counts complete model–scenario tuples. Miss ∣ Condition A is the share of correct Condition A answers with a Condition B miss. Rec. (recovery) is the share of Condition B misses caught under Condition C .
Experiment
Condition A
Condition B
Condition C
Non-violation
–
Metric
Accuracy ↑
\labelbox catchcolorCatch rate ↑
\labelbox catchcolorCatch rate ↑
\labelbox catchcolorCatch rate ↓
Recovery ↑
Claude Opus 5
100.0
100.0
100.0
0.0
–
Kimi K3
75.8
18.2
45.5
0.0
37.0
Gemini 3.7 Flash
97.0
63.6
87.9
0.0
66.7
Claude Sonnet 5
93.9
36.4
84.8
3.0
81.0
GPT-5.5
100.0
48.5
90.9
0.0
82.4
Appendix
Table 6: Tasc-p scores (%) by model. Condition A measures accuracy; Condition B and Condition C calculate \labelbox catchcolorCatch rates. Note that in non-violation controls \labelbox catchcolorCatch rate represents false alarms, so the metric is the lower the better. Recovery is the share of Condition B misses caught under Condition C (– when a model has no Condition B misses). Rows follow the order of Table 3 . Cells are darkened proportionally to their value. Each model has 33 complete tuples except DeepSeek-V4-Pro, which has 32.
Figure 3: Tasc-p scores (%) for the 13 models. Condition A measures accuracy; Condition B and Condition C calculate \labelbox catchcolorCatch rates. Each model has 33 complete tuples except DeepSeek-V4-Pro, which has 32.
Rule / direction
\labelbox catchcolorCatch / n
Rate (%)
Minimum wait / early
1,991 / 5,551
35.9
Deadline / late
944 / 3,003
31.4
Target window / early
325 / 1,677
19.4
Target window / late
189 / 1,040
18.2
Appendix
Table 7: Tasc-m Condition B \labelbox catchcolorCatch rates on violations by rule type and direction, using the shared scored inputs. n counts model–input responses.
\labelbox catchcolorCatch
\labelbox hedgecolorHedge
\labelbox askcolorAsk
\labelbox proceedcolorProceed
\labelbox catchcolorCatch
401
1
0
23
\labelbox hedgecolorHedge
5
0
1
9
\labelbox askcolorAsk
0
0
0
0
\labelbox proceedcolorProceed
5
2
1
538
Appendix
Table 8: Original judge labels. Rows: Claude Sonnet 5; columns: GPT-5.6 Luna.
κ
95% CI
n
Agr. (%)
All sampled (4-label)
0.708
0.60–0.82
144
84.7
Judge-labeled, vs. final
0.735
0.62–0.84
126
85.7
Violations ( Condition B & Condition C )
0.768
0.65–0.88
114
87.7
Condition C
0.846
0.71–0.96
49
91.8
Condition B violations
0.707
0.54–0.87
65
84.6
Non-violation controls
0.330
− 0.06–0.69
30
73.3
Appendix
Table 9: Human–LLM agreement with the recorded labels. κ is Cohen’s κ ; Agr. is agreement.
Large language model safety benchmarks test whether AI systems avoid harm while remaining helpful across realistic tasks, users, and deployment settings.
Today's large language models (LLMs) increasingly interact with time: AI agents schedule tasks, coordinate with people and other AI systems, and even decide how long to think. Yet LLMs only ever encounter time secondhand---in descriptions rather than experience. Does this hinder their reasoning and decisions that involve time? Our paper traces this question from a model's knowledge to its actions through four successive evaluations. We find that models know the rules of time, which can be read, but struggle to predict durations, which must be lived; while they are highly accurate (92.0-97.5%) on a collection of questions built from the theorems of various time-related literatures, they underestimate their own completion times by up to 99.5% and fail to identify which tasks will take them longer to run. And while they are better at estimating durations of human tasks, their duration judgments change based on how an event is worded, magnifying similar human biases by up to 1.3×. Moving from estimation to action, we find that some LLMs disregard time costs completely when answering user queries, ignoring a 1,200-fold increase in wait time entirely. Together, our experiments reveal new challenges posed to LLMs as they navigate a dimension they cannot directly perceive, and highlight how their blind spots can even be simple cognitive abilities that humans take for granted.
Ryan Liu, David Lee, Icey Siyi Ai +3
Department of Computer Science, Princeton University · Department of Operations Research and Financial Engineering, Princeton University · Department of Industrial Engineering and Operations Research, Columbia University +3
Large language models cannot estimate how long their own tasks take. We investigate this limitation through four experiments across 68 tasks and four model families. Pre-task estimates overshoot actual duration by 4--7× (p<0.001), with models predicting human-scale minutes for tasks completing in seconds. Relative ordering fares no better: on task pairs designed to expose heuristic reliance, models score at or below chance (GPT-5: 18% on counter-intuitive pairs, p=0.033), systematically failing when complexity labels mislead. Post-hoc recall is disconnected from reality -- estimates diverge from actuals by an order of magnitude in either direction. These failures persist in multi-step agentic settings, with errors of 5--10×. The models possess propositional knowledge about duration from training but lack experiential grounding in their own inference time, with practical implications for agent scheduling, planning and time-critical scenarios.
Can interactive vision-and-language agents learn not just what to say but also when to say it? Current language models rarely plan over whether and when to realize a real-time response to a user. However, providing accurate and timely support for human decision-making, such as when guiding visually impaired individuals through urban environments, requires careful real-time responsiveness--poorly timed responses can distract users or add unnecessary cognitive load. As a machine intelligence challenge for Multimodal Large Language Model (MLLM)-based agents, we introduce a large-scale multimodal benchmark for an egocentric, assistive navigation task in complex outdoor environments. Using this benchmark, we uncover a fundamental limitation of off-the-shelf MLLMs in delivering safe and time-sensitive navigation instructions, even with model fine-tuning on substantial amounts of data. We then demonstrate that a simple yet effective modification of the model, including direct supervision to predict the underlying reason for each instruction, yields significant performance gains across open-loop, closed-loop, and sim-to-real generalization settings. However, our analysis highlights persistent challenges in temporal reasoning, safety-critical object awareness, and relational and distance understanding. To advance the development of scalable assistive agents, we will release our simulation, benchmark, and code (available at the project website: https://timeli-icra.github.io/).