cs.AIOct 28, 2026
SaveLeaving, Staying, or Living There? Duration Commonsense Reasoning with Human-LLM Co-annotations
Organizations: University of North Texas · University of Arizona
Abstract
Gardening in the backyard implies living there, while checking into an Airbnb more likely means staying for a few days. Such duration judgments are important for understanding human mobility, travel behavior, and social context. However, duration is rarely stated explicitly in social media posts and must instead be inferred from noisy textual, visual, and contextual cues. In this paper, we investigate whether vision-language models possess such duration commonsense. We create a new corpus of geotagged tweets co-annotated by humans and LLMs to serve as a testbed for this task. We find that state-of-the-art models struggle with duration inference, even when guided by multi-turn Chain-of-Thought prompting. Error analyses further reveal that models often fail to integrate critical contextual information, or they overly rely on prominent visual details while overlooking subtle hashtags and semantic nuances. These findings underscore the significant challenges remaining in context-aware multimodal understanding.
Figures & tables
Fig. 1: Examples of tweets in which the author will stay at the geotagged location for a short (left) or long (right) time. The author of the left tweet will stay at Morro Bay only briefly: the text recounts kayaking before heading back, and the image shows the author in a kayak during a day trip. The author of the right tweet will stay at New Orleans for a long time; the “Love in the Time of Corona” post, indoor setting, and pet suggest the author is sheltering at home during the COVID-19 lockdown.
Fig. 2: Annotation examples. The author of the leftmost tweet is departing SeaTac (“Hopefully I will be safe this time” in the text and the crowded airport waiting area in the image), indicating a stay of 0-24 hours . The middle tweet exemplifies 1-30 days : the author visits the Gateway Arch on a #roadtrip to St Louis , and travelers typically spend less than one week at each stop. The author of the rightmost tweet lives in Halawa ; the “Yep, we’re eating again” text and a home-cooked meal in the image, together with hashtags such as #whatsfordinner and #onogrindz, suggest everyday residence rather than a brief visit, leading to more than 30 days .
| Model | Kappa Value | Interpretation |
|---|---|---|
| Qwen 2.5 | 0.1659 | Moderate agreement |
| GPT5-mini | 0.1892 | Moderate agreement |
| Grok 4 | 0.2985 | Fair agreement |
| Claude 3 haiku | 0.3256 | Fair agreement |
| Gemini 2.5 flash | 0.5302 | Substantial agreement |
| Gemini 2.5 pro | 0.5572 | Substantial agreement |
TABLE I: Inter-model agreement measured using Cohen’s Kappa.
| LLM | Human | ||
|---|---|---|---|
| 0-24 hours | 1-30 days | more than 30 days | |
| 0-24 hours | 2.00 | 0.73 | 18.36 |
| 1-30 days | 4.57 | 2.65 | 5.40 |
| more than 30 days | 32.23 | 2.37 | 31.69 |
TABLE II: Confusion matrix of duration labels assigned by human and LLM annotators. Values are percentages. Diagonal cells (in bold ) indicate consensus labels between annotators.
| R1: COVID-19 quarantine duration commonsense | R2: Duration not calculated at the city level | R3: Tweet intent unclear | R4: Author residency status unclear |
|---|---|---|---|
| Quarantine Birthday for me. #lockdown #coronavirus #carrotcake #nyc #manhattan #cumplea 3:55 AM Apr 28, 2020 from Manhattan, NY | VVBS buying @tahlequahumc #vvbs #vbs2020 @ Tahlequah United Methodist Church 3:26 PM Jul 1, 2020 from Tahlequah, OK | gloom my day #boardwalkclosed #fencedin #locked #gates #cold #covid_19 #epedemic #truphotos 4:13 PM Mar 29, 2020 from Seaside Heights, NJ | Im a Barbie girl… in a Covid world #transisbeautiful #girlslikeus #transvisibility #saturdayvibes #saturdaynight #covid19 12:42 AM Sep 13, 2020 from Dallas, TX |
| LLM: 1--30 days The reason is that the tweet text explicitly uses hashtags like #lockdown and #coronavirus. During the COVID-19 pandemic, a quarantine period was a mandated or recommended period of isolation, typically lasting for 14 days. This falls within the 1–30 day range, making it the most logical duration for the author’s stay in Manhattan. | LLM: more than 30 days The reason is that the tweet is about buying supplies for a Vacation Bible School (#vbs2020) at the Tahlequah United Methodist Church (@tahlequahumc). Organizing and participating in such a local church event strongly suggests that the person is a resident of Tahlequah or the surrounding area, not a temporary visitor. Therefore, they are likely to be in the city for an extended period. | LLM: more than 30 days The image and hashtags like #boardwalkclosed, #locked, and #fencedin show that public areas were shut down. During this period, travel was heavily restricted, and people were generally under stay-at-home orders. It is most likely that the author is a resident of Seaside Heights documenting the unusual state of their local area during the lockdown, rather than a tourist visiting a closed-down town. | LLM: 1--30 days The tweet includes the hashtags #saturdayvibes and #saturdaynight, indicating that the author is dressed up for a specific weekend event. This, combined with being in a specific city “Dallas”, suggests that she is likely on a short trip, probably for the weekend. A weekend trip typically lasts for a few days, which falls into the 1–30 day category. |
| Human: more than 30 days | Human: 0--24 hours | Human: 0--24 hours | Human: more than 30 days |
TABLE III: Common system errors made by LLM and human annotators.
| Prediction: <LLM, Human> | R1 | R2 | R3 | R4 | # Tweets |
|---|---|---|---|---|---|
| <0-24 hours, 1-30 days> | 0 | 0 | 3(D) | 5(D) | 8 |
| <0-24 hours, more than 30 days> | 0 | 20(H) | 191(D) | 0 | 201 |
| <1-30 days, 0-24 hours> | 0 | 0 | 26(D) | 24(D) | 50 |
| <1-30 days, more than 30 days> | 8(H) | 0 | 0 | 51(D) | 59 |
| <more than 30 days, 0-24 hours> | 0 | 266(L) | 87(D) | 0 | 353 |
| <more than 30 days, 1-30 days> | 3(L) | 0 | 0 | 23(D) | 26 |
TABLE IV: Prediction Pair Analysis. ‘L’ indicates that the LLM’s prediction is the final decision, ‘H’ indicates that the human annotator’s prediction is the final decision, and ‘D’ indicates that the annotation was discarded.
Fig. 3: Label distribution.
| Methods | Backbone | Input | ZS-ST | ZS-MT | FS-ST | FS-MT |
|---|---|---|---|---|---|---|
| Human | Human reference | human judgment | 0.37 | |||
| PTLM | RoBERTa-Large 340M | text | 0.18 | 0.21 | – | – |
| ViT-Large/16 | image | 0.05 | 0.07 | – | – | |
| RoBERTa-ViT-Concatenation | text + image | 0.22 | 0.24 | – | – | |
| RoBERTa-ViT-Addition | text + image | 0.16 | 0.17 | – | – | |
| RoBERTa-ViT-Multiplication | text + image | 0.13 | 0.15 | – | – | |
TABLE V: Weighted-average F1 scores of different model families on duration inference. ZS-ST, FS-ST, and FS-MT denote zero-shot single-turn, few-shot single-turn, and few-shot multi-turn settings, respectively. Text-only LLMs use tweet captions as input. The multi-turn setting separates visual analysis and duration inference into two sequential dialogue turns. The best score within each method group is underlined , and the best score in each evaluation setting is boldfaced . A dash indicates that the setting is not applicable. Further details are provided in Section V-B and Table VI .
| Model | Content | Single-turn prompt | Multiturn prompt |
|---|---|---|---|
| LLM (Text Only) | Tweet text: Corona virus approved activities at the Glass House! #COVID19 #backtobasics #puzzlesrock #winelife #deerfootwinery #staycationfunwiththewife @ Wyomissing Park, Reading Tweet image description: an indoor home setting with a jigsaw puzzle spread across the floor, wine glasses, and a decorated Christmas tree | System / user prompt: You are a helpful assistant for inferring the duration of human activities. Please read the tweet and determine how long the author of the tweet will stay in Reading after posting this tweet. Options: 1. 0–24 hours 2. 1–30 days 3. more than 30 days Answer: 2. 1–30 days. The hashtag #staycationfunwiththewife suggests a temporary leisure stay rather than permanent residence. The tweet describes home-based activities such as doing puzzles and drinking wine, which are compatible with a short staycation lasting multiple days but do not by themselves imply long-term residence. | Round 1 prompt: You are a helpful assistant for inferring the duration of human activities. Analyze how the tweet’s image description relates to its text and hashtags. Answer: The image description validates the tweet’s narrative: a pandemic-safe, low-tech, couple-focused staycation. The immediate scene (puzzle + wine) likely represents a 2–4 hour evening activity, but the hashtag #staycationfunwiththewife suggests this isn’t a one-off event but a multi-day to week-long pattern of home-based leisure Round 2 prompt: Please read the tweet and determine how long the author of the tweet will stay in Reading after posting this tweet. Options: 1. 0–24 hours 2. 1–30 days 3. more than 30 days Answer: 2. 1–30 days. The combination of the #staycation hashtag, pandemic-era context, and home-based leisure activities strongly implies the author is settled in Reading for a short-to-medium-term local break. The most evidence-supported inference is a stay lasting between 1 and 30 days after posting. |
| VLM | Tweet text: Corona virus approved activities at the Glass House! #COVID19 #backtobasics #puzzlesrock #winelife #deerfootwinery #staycationfunwiththewife @ Wyomissing Park, Reading Tweet image: | System / user prompt: You are a helpful assistant for inferring the duration of human activities. Please read the tweet and determine how long the author of the tweet will stay in Reading after posting this tweet. Options: 1. 0–24 hours 2. 1–30 days 3. more than 30 days Answer: 3. more than 30 days. The tweet refers to “Corona virus approved activities” and includes #COVID19, suggesting stay-at-home behavior during the pandemic. The image shows an indoor domestic setting, and #staycationfunwiththewife further implies that the author is spending time at or near home. Taken together, these cues suggest that the author is likely a resident of Reading and will remain there for an extended period. | Round 1 prompt: You are a helpful assistant for inferring the duration of human activities. Describe the tweet image in detail, and analyze its relationship to the tweet text and hashtags. Answer: The image shows an indoor home environment with wooden flooring. A large jigsaw puzzle is spread across the floor, and wine glasses are visible nearby. These visual details align with #puzzlesrock and #winelife. The hashtags #COVID19 and #staycationfunwiththewife suggest home-based leisure during the pandemic, while #backtobasics indicates simple indoor activities. Round 2 prompt: Please read the tweet and determine how long the author of the tweet will stay in Reading after posting this tweet. Options: 1. 0–24 hours 2. 1–30 days 3. more than 30 days Answer: 3. more than 30 days. The tweet and image together indicate a home-based staycation during COVID-19. Because the activity appears to take place in the author’s home environment, the author is likely staying in Reading long-term rather than visiting briefly. |
TABLE VI: Representative prompts used for different model settings. For text-only models, the image is converted into a textual description. For VLM models, the original image is provided directly. In the multiturn CoT setting, the model first interprets the image and its relation to the tweet, and then predicts the duration label.
| Context | Visual | Semantic |
|---|---|---|
| I carried the memories of my dad on a Social Distancing (SoDi) Half Marathon for Father’s Day. #MNStrong #AllOutin2020 #Minnesota #FathersDay #fitgay 3:12 AM Jun 23, 2020 from St Joseph, MN | Love me a good day in #oceancitymaryland. The weather was perfect for a walk. Most folks were #socialdistancing. Just wish #oceangallery was open. The #clouds and sky were amazing. 12:35 AM Jun 21, 2020 from Ocean City, MD | Glad to see one of my best friends Raul from Chicago and college in town! #LewisUniversity #Chicago #Flyers #Botines #Liverpool @ Corona del Mar State Beach 12:29 AM Aug 18, 2020 from Newport Beach, CA |
| Generated: 0--24 hours The image and the tweet describe the author completing a half marathon run on a trail for Father’s Day. This is a specific, time-bound activity that typically lasts a few hours. After finishing the run and posting the photo, it is most likely that the author would return home. The nature of the activity does not suggest a multi-day vacation or residency. | Generated: 0--24 hours The tweet states, "Just wish #oceangallery was open". This implies that the author had intended to visit the Ocean Gallery during their current stay but couldn’t because it was closed. The sentiment of "just wish" suggests a missed opportunity that they won’t be able to fulfill on this trip, strongly indicating that their stay in Ocean City is 0 hours. | Generated: 1--30 days The reason is that the tweet mentions the author is meeting his friend from Chicago who is “in town”. A friend visiting from another state usually stays for a few days or a short vacation, which falls within the 1–30 day range. It is unlikely to be a visit for less than 24 hours due to the travel distance, and there is no indication of a long-term stay of over 30 days. |
| Gold: more than 30 days The reason is that the tweet indicates the author is running a half marathon in Minnesota, with hashtags like #MNStrong and #Minnesota, suggesting he is a local resident. Therefore, it is highly probable that he lives in or near St. Joseph and will remain there for an extended period. | Gold: 1--30 days The reason is that the tweet mentions having "a good day in #oceancitymaryland" and wishing a local gallery was open. The image shows that weather was good for walking, most people were keeping distance from each other. This suggest that the author is on a vacation or a short trip, which usually lasts from a few days to a couple of weeks. | Gold: more than 30 days The tweet states, “Glad to see one of my best friends Raul from Chicago and college in town!” The phrase “in town” implies that Raul is visiting the city where the author is located. This suggests that the author either resides in Newport Beach/Corona del Mar or has a prolonged presence there. If the author lives in the city, their stay is continuous and indefinite, which falls under “more than 30 days”. |
| Analysis: Fails to recognize the hashtag #MNStrong, which indicates he is a local resident. | Analysis: Fails to distinguish the actual beach walk, which is visible in the image, from the unfulfilled Ocean Gallery visit. | Analysis: Correctly identifies the scenario as the author hosting a friend from another state, but incorrectly reports the friend’s duration of stay as the final answer. |
TABLE VII: Error analysis of all evaluated models
Fig. 4: Distribution of error types across model categories