Paper ID: 2403.17196

GPT-4 Understands Discourse at Least as Well as Humans Do

Thomas Shultz, Jamie Wise, Ardavan Salehi Nobandegani

We test whether a leading AI system GPT-4 understands discourse as well as humans do, using a standardized test of discourse comprehension. Participants are presented with brief stories and then answer eight yes/no questions probing their comprehension of the story. The questions are formatted to assess the separate impacts of directness (stated vs. implied) and salience (main idea vs. details). GPT-4 performs slightly, but not statistically significantly, better than humans given the very high level of human performance. Both GPT-4 and humans exhibit a strong ability to make inferences about information that is not explicitly stated in a story, a critical test of understanding.

Submitted: Mar 25, 2024