Autonomous AI Agents Systematically Diverge From Human Preferences in Proof Formalization
Organizations: Institute for Complex Social Dynamics, Carnegie Mellon University · Department of Social and Decision Sciences, Carnegie Mellon University · Santa Fe Institute
Abstract
Artificial intelligence (AI) systems have ushered in a new age of proof abundance, but helping humans understand these proofs requires new attention to the differences in how AI agents and humans pursue a proof-making task. We present "simulated collaboration", a methodology that allows us to examine two key questions: (1) the relative attention AI agents pay to surface syntax or, conversely, deeper underlying semantics, and (2) how and where AI agent preferences for what to do next diverge from human preferences. On a set of nine major human collaborations in Lean with 4.7M definitions across more than 6K commits, we find that, over the course of proof construction, AI agents (1) are far more strongly drawn than humans to explicitly marked gaps; (2) show a relative aversion to densely connected proof centers and to complex or surprising proof goals; and (3) are relatively attracted to surprising proof bodies, which they tend to regularize toward baseline. Our results highlight important blind spots and potential interventions for future human-AI collaboration in mathematics.
Figures & tables
| Actor | Added | Modified | Removed | Total | ||||||
| Count | % | Median chars. | Count | % | Median chars. | Count | % | Median chars. | Count | |
| Human | 10,618 | 0.90 | 133 | 6,975 | 0.59 | 20 | 4,913 | 0.41 | 129 | 1,185,143 |
| Claude | 4,640 | 0.39 | 171 | 6,178 | 0.52 | 149.5 | 408 | 0.03 | 81 | 1,176,797 |
| Gemini | 1,974 | 0.17 | 193 | 4,876 | 0.42 | 84 | 618 | 0.05 | 104 | 1,174,808 |
| Codex | 2,056 | 0.17 | 135 | 1,394 | 0.12 | 103 | 57 | 0.00 | 110 | 1,175,224 |
| Overall | 19,288 | 0.41 | 145 | 19,423 | 0.41 | 72 | 5,996 | 0.13 | 120 | 4,711,972 |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Repo | Mathematical Topic | LoC | Editors | Commits | Years |
| apap | Arithmetic progressions, almost periodicity [ Dil+23 , BS23 ] | 7,509 | 11 | 472 | 3.26 |
| duality | Linear-optimization duality (Farkas) [ DK26 ] | 2,582 | 1 | 177 | 1.91 |
| finlin | Linearity of functions on finite sets [ Bac25 ] | 2,055 | 2 | 57 | 0.66 |
| flt | Fermat’s Last Theorem [ Buz+23 ] | 28,031 | 61 | 1,614 | 2.57 |
| ordsemi | Hölder’s theorem for ordered semigroups [ Pau24 ] | 3,438 | 2 | 125 | 1.88 |
| pdl | Propositional Dynamic Logic [ Gat+23 , Bor+25 ] | 23,873 | 13 | 1,041 | 2.70 |
| Definitions | Dependency edges | Degree rank corr. | Graph features | |||||||||
| Repository | Modules | LCC % | ||||||||||
| apap | 40/40 | 835 | 0.998 | 2,173 | 0.812 | 0.830 | 0.821 | 0.79 | 0.84 | 0.70 | 0.20 | 91 |
| duality | 7/7 | 184 | 1.000 | 951 | 0.979 | 0.432 | 0.600 | 0.80 | 0.57 | 0.62 | 0.30 | 99 |
| finlin | 2/2 | 15 | 1.000 | 25 | 1.000 | 1.000 | 1.000 | 1.00 | 1.00 | 1.00 | 1.00 | 100 |
| flt | 171/174 | 1,845 | 1.000 | 5,117 | 0.715 | 0.532 | 0.610 | 0.76 | 0.64 | 0.50 | 0.32 | 80 |
| ordsemi | 16/16 | 228 | 1.000 | 1,122 | 0.986 | 0.608 | 0.752 | 0.84 | 0.88 | 0.84 | 0.67 | 98 |
| Predictor | DeepSeek-Prover-V2-7B | Goedel-Prover-V2-32B | Kimina-Prover-72B | |
| Surprise, signature | Human | |||
| Claude | {\color[rgb]{0.8359,0.3672,0}-0.877^{***}\,(0.085)} | {\color[rgb]{0.8359,0.3672,0}-0.528^{***}\,(0.083)} | {\color[rgb]{0.8359,0.3672,0}-0.531^{***}\,(0.082)} | |
| Gemini | {\color[rgb]{0.8359,0.3672,0}-0.338^{***}\,(0.089)} | {\color[rgb]{0.8359,0.3672,0}-0.191^{*}\,(0.087)} | ||
| Codex | {\color[rgb]{0.8359,0.3672,0}-0.390^{**}\,(0.138)} | {\color[rgb]{0.8359,0.3672,0}-0.528^{***}\,(0.139)} | {\color[rgb]{0.8359,0.3672,0}-0.498^{***}\,(0.137)} | |
| Surprise, body | Human | |||
| Claude | {\color[rgb]{0,0.4453,0.6992}\phantom{-}1.029^{***}\,(0.084)} | {\color[rgb]{0,0.4453,0.6992}\phantom{-}0.995^{***}\,(0.080)} | {\color[rgb]{0,0.4453,0.6992}\phantom{-}0.672^{***}\,(0.079)} |
| Actor | Parent body | Goedel-Prover-V2-32B | Kimina-Prover-72B | DeepSeek-Prover-V2-7B |
| Human | with sorry | (1,232) | (1,232) | (1,232) |
| without sorry | (5,129) | (5,129) | (5,129) | |
| Claude | with sorry | (4,464) | (4,464) | (4,464) |
| without sorry | (367) | (367) | (367) | |
| Gemini | with sorry | (2,855) | (2,855) | (2,855) |
| without sorry | (718) | (718) | (718) |
| Predictor | Human | Claude | Gemini | Codex |
| Intercept | {\color[rgb]{0.8359,0.3672,0}-1.119^{*\phantom{**}}\,(0.546)} | {\color[rgb]{0.8359,0.3672,0}-1.185^{*\phantom{**}}\,(0.546)} | ||
| Has sorry | {\color[rgb]{0,0.4453,0.6992}\phantom{-}4.556^{***}\,(0.071)} | {\color[rgb]{0,0.4453,0.6992}\phantom{-}3.074^{***}\,(0.060)} | {\color[rgb]{0,0.4453,0.6992}\phantom{-}1.714^{***}\,(0.084)} | |
| Surprise, signature (rank) | {\color[rgb]{0.8359,0.3672,0}-0.877^{***}\,(0.085)} | {\color[rgb]{0.8359,0.3672,0}-0.338^{***}\,(0.089)} | {\color[rgb]{0.8359,0.3672,0}-0.390^{**\phantom{*}}\,(0.138)} | |
| Surprise, body (rank) | {\color[rgb]{0,0.4453,0.6992}\phantom{-}1.029^{***}\,(0.084)} | {\color[rgb]{0,0.4453,0.6992}\phantom{-}0.854^{***}\,(0.091)} | {\color[rgb]{0,0.4453,0.6992}\phantom{-}0.522^{***}\,(0.135)} | |
| Max leaf (rank) | {\color[rgb]{0.8359,0.3672,0}-4.745^{***}\,(0.329)} | {\color[rgb]{0.8359,0.3672,0}-1.584^{***}\,(0.356)} | {\color[rgb]{0.8359,0.3672,0}-3.533^{***}\,(0.549)} | |
| Max leaf (rank) 2 | {\color[rgb]{0,0.4453,0.6992}\phantom{-}4.054^{***}\,(0.296)} | {\color[rgb]{0,0.4453,0.6992}\phantom{-}1.424^{***}\,(0.315)} | {\color[rgb]{0,0.4453,0.6992}\phantom{-}4.223^{***}\,(0.481)} |
Human-AI Collaboration
Human-AI collaboration studies how people and AI systems divide, coordinate, and adapt work to support decisions, creativity, and learning.
See all 473 papers