Computational Hermeneutics: Evaluating generative AI as a cultural technology
Authors: Cody Kommers, Ruth Ahnert, Maria Antoniak, Emmanouil Benetos, Steve Benford, Mercedes Bunz, Baptiste Caramiaux, Shauna Concannon, +30 more
Organizations: The Alan Turing Institute · Queen Mary University of London · University of Colorado · University of Nottingham · King’s College London · Sorbonne Université · Durham University · University of Edinburgh · Dartmouth College · Gibran AI · University of Coventry · Technische Universität Darmstadt · University of Cambridge · McGill University · Cornell University · University of Utah · University of Chicago · Princeton University · London School of Economics · University of Southampton · Rice University · University of Glasgow · Cardiff University · University of Exeter · University of Brighton · The Alan Turing Institute & University of Southampton · University of Illinois Urbana-Champaign · University College London · University of Sheffield · The Alan Turing Institute & University of Edinburgh
Generative AI systems are increasingly recognized as cultural technologies, yet current evaluation frameworks often treat culture as a variable to be measured rather than fundamental to the system's operation. Drawing on hermeneutic theory from the humanities, we argue that GenAI systems function as "context machines" that must inherently address three interpretive challenges: situatedness (meaning only emerges in context), plurality (multiple valid interpretations coexist), and ambiguity (interpretations naturally conflict). We present computational hermeneutics as an emerging framework offering an interpretive account of what GenAI systems do, and how they might do it better. We offer three principles for hermeneutic evaluation -- that benchmarks should be iterative, not one-off; include people, not just machines; and measure cultural context, not just model output. This perspective offers a nascent paradigm for designing and evaluating contemporary AI systems: shifting from standardized questions about accuracy to contextual ones about meaning.