Previous work has examined the capacity of deep neural networks (DNNs), particularly transformers, to predict human sentence acceptability judgments, both independently of context, and in document contexts. We consider the effect of prior exposure to visual images (i.e., visual context) on these judgments for humans and large language models (LLMs). Our results suggest that, in contrast to textual context, visual images appear to have little if any impact on human acceptability ratings. However, LLMs display the compression effect seen in previous work on human judgments in document contexts. Different sorts of LLMs are able to predict human acceptability judgments to a high degree of accuracy, but in general, their performance is slightly better when visual contexts are removed. Moreover, the distribution of LLM judgments varies among models, with Qwen resembling human patterns, and others diverging from them. LLM-generated predictions on sentence acceptability are highly correlated with their normalised log probabilities in general. However, the correlations decrease when visual contexts are present, suggesting that a higher gap exists between the internal representations of LLMs and their generated predictions in the presence of visual contexts. Our experimental work suggests interesting points of similarity and of difference between human and LLM processing of sentences in multimodal contexts.
Figures & tables
Book
Orig
One piece of felt, folded in the corner, will be spliced, unfurled and dangled from the ceiling like a canopy.
Mod
A piece in a corner, spliced were put forward and believe that the upper ceiling as a canopy.
News
Orig
But the answer seems to be no: Reeves lets it be known she requests no costings on raising the three forbidden taxes.
Mod
But the reply does not seem: reeves suggests that it would not cost to raise the three banned taxes.
Wiki
Orig
In response, the US deployed an additional 170,000 troops during the 2007 troop surge, which helped stabilize parts of the country.
Mod
In response, we deployed with further 170 000 soldiers in 2007 increase troops, which help to stabilize parts of the country.
Table 1: Examples of sentences used in our experiment, sourced from books, news, and Wikipedia in original form (Orig) and modified (Mod) through round-trip machine translation with Moses 3.0.
Original Sentence
Relevant Image
Irrelevant Image
One piece of felt, folded in the corner, will be spliced, unfurled and dangled from the ceiling like a canopy.
Table 2: An example sentence with relevant and irrelevant image as context.
Original
Modified
N
R
I
N
R
I
mean
3.54
3.54
3.53
1.96
2.02
1.94
sd
0.76
0.75
0.77
1.05
1.03
1.01
Table 3: Descriptive statistics for human ratings on sentence acceptability (1 - 4) in null (N), relevant (R), and irrelevant (I) conditions.
Figure 1: Human ratings of sentence acceptability in different conditions with regression lines generated from total least square regression. P-value alpha: 0.017 (Bonferroni correction).
Figure 3: Human ratings of sentence acceptability in different textual context. “Context = None” means null context, “Context = Real” means relevant context and “Context = Random” means irrelevant context. Image reproduced from Lau et al. (2020) .
Books
News
Wikipedia
mean
1.95
1.63
1.70
sd
1.03
0.86
0.92
Table 4: Mean abstractness scores reported by human participants on original English sentences from books, news, and Wikipedia.
All
N
R
I
LLM ratings ∼ Human ratings
gpt-4o
0.87
0.89
0.87
0.88
gpt-4o-mini
0.89
0.89
0.89
0.88
InternVL3-1B
0.32
0.66
0.26
0.16
InternVL3-8B
0.83
0.88
0.85
0.85
Qwen2.5-3B
0.65
0.70
0.61
0.66
Qwen2.5-7B
0.78
0.84
0.75
0.76
Table 5: Spearman ( ρ ) correlations between average human sentence acceptability ratings (1-4) and LLM-prompted ratings (1-4) in null (N), relevant (R) and irrelevant (I) conditions. All correlations significant (p < 0.001). Highest correlations per row marked in bold.
All
N
R
I
MeanLP ∼ Human ratings
InternVL3-1B
0.70
0.72
0.70
0.70
InternVL3-8B
0.70
0.72
0.71
0.70
Qwen2.5-3B
0.61
0.70
0.61
0.57
Qwen2.5-7B
0.73
0.79
0.76
0.71
llava-1.5-7b
0.74
0.74
0.76
0.73
Table 6: Spearman ( ρ ) correlations between Mean Logprobs from open-source LLMs and average human sentence acceptability ratings. All correlations significant (p < 0.001). Highest correlations per row marked in bold.
Figure 4: Sentence acceptability ratings generated by gpt-4o.
Figure 5: Sentence acceptability ratings generated by InternVL3-8B.
Figure 6: Sentence acceptability ratings generated by Qwen2.5-7B.
All
N
R
I
MeanLP ∼ Model ratings
InternVL3-1B
0.30
0.58
0.30
0.17
InternVL3-8B
0.70
0.75
0.74
0.71
Qwen2.5-3B
0.52
0.56
0.48
0.45
Qwen2.5-7B
0.70
0.76
0.71
0.68
llava-1.5-7b
0.28
0.35
0.24
0.22
Table 7: Spearman ( ρ ) correlations between Mean Logprobs from open-source LLMs and model prompted ratings. All correlations significant (p < 0.001). Highest correlations per row marked in bold.