The ability to integrate new information and deduce its consequences is central to learning, understanding, and reasoning in large language models (LLMs). Whether information arrives during training or in-context, models must not only store local facts, but propagate their long-range implications globally. To explore information integration in LLMs, we introduce two complementary natural language benchmarks for large-scale evaluation of deductive reasoning as a function of inference depth. The first is a formal graph benchmark enabling controlled evaluation over a wide range of graph topologies and integration tasks; the second is a knowledge composition benchmark grounded in real-world knowledge graphs. We use these benchmarks to compare knowledge integration from information encoded in context versus information encoded in model weights during pretraining, evaluating a set of open models implicitly, without chain of thought. We find that accuracy decays sharply with increased composition length for both pretrained and in-context knowledge. Information presented in-context, however, substantially outperforms deduction over information encoded in weights. These results quantify a fundamental limitation of current models with direct implications for prompt-based control of agent behavior and for continual learning strategies, and they provide a foundation for developing training, prompting, and integration methods that support deeper reasoning.
Figures & tables
Figure 1: Models receive graph adjacency in-context but integrate the information poorly, failing at long-range inferences. DeepSeek, Kimi, Nemotron, and 12 other models answer questions about a random graph of 50–60 nodes given in the prompt as a list of edges, with reasoning off. Each question concerns the nodes exactly k hops from a start node, so k=1 questions require only direct-neighbor information. a) Verify: is a given node exactly k hops away? Accuracy falls from 93 – 100% at k=1 to 49 – 56% at k=3 , close to chance. b) Count: how many nodes are exactly k hops away? Models count correctly 43 – 69% of the time at k=1 and ≤16% from k=2 . c) Name one: name a node exactly k hops away. Models name a neighbor at 99 – 100% , but a node two hops away at only 9 – 40% . d) Enumerate: list every node exactly k hops away. Models list all neighbors at 71 – 100% , but no model lists more than 12% of sets correctly from k=2 .
Figure 2: Models compose facts from the prompt far better than from memory. DeepSeek, Kimi, and Nemotron answer k -hop Wikidata questions with reasoning off and information given in context, in memory, or both. a) The chain’s facts are given in the prompt under invented names, among 16 decoy chains with the same relations. All models compose at ≥94% up to k=3 , but at k=6 , Kimi composes at 29% , while DeepSeek and Nemotron Ultra fall to chance. b) The facts are recalled from memory without supporting facts in the prompt. Composition reaches only 37 – 64% at k=2 and ≤12% from k=4 , although every fact is known. c) The facts are given in the prompt under their real names, among 16 real decoy chains, so the model can use both the prompt and memory. Accuracy reaches 36 – 71% at k=6 .
Large language models increasingly serve as persistent knowledge workers, with in-context memory - facts stored in the prompt - as the default strategy. We benchmark in-context memory against Knowledge Objects (KOs), discrete hash-addressed tuples with O(1) retrieval. Within the context window, Claude Sonnet 4.5 achieves 100% exact-match accuracy from 10 to 7,000 facts (97.5% of its 200K window). However, production deployment reveals three failure modes: capacity limits (prompts overflow at 8,000 facts), compaction loss (summarization destroys 60% of facts), and goal drift (cascading compaction erodes 54% of project constraints while the model continues with full confidence). KOs achieve 100% accuracy across all conditions at 252x lower cost. On multi-hop reasoning, KOs reach 78.9% versus 31.6% for in-context. Cross-model replication across four frontier models confirms compaction loss is architectural, not model-specific. We additionally show that embedding retrieval fails on adversarial facts (20% precision at 1) and that neural memory (Titans) stores facts but fails to retrieve them on demand. We introduce density-adaptive retrieval as a switching mechanism and release the benchmark suite.
Large language models (LLMs) read knowledge-graph facts ranked by relevance, leaving open whether they suffice or conflict. Our checks use relation knowledge, written by hand or learned from training questions labelled with their relation paths. It selects a compact context of facts yielding an answer, and derives it; a derivability check tests whether an answer follows, a consistency check whether facts conflict, and a validation-chosen routing gate decides where the derived answer replaces the LLM's. Seven open LLMs read the compact context better in exact match than relevance-ranked facts; on ComplexWebQuestions (CWQ) they match SubgraphRAG's facts at a quarter of the prompt tokens, and stronger LLMs read SubgraphRAG's better. Where the routing gate passes, ours leads SubgraphRAG's in exact match in all 53 LLM, benchmark and prompt settings. A precision gate, returning the derived answer only when the check passes with exactly one derived entity, answered 18 and 5 percent of held-out WebQSP and CWQ questions in a pre-specified test, right for 90 and 96 percent. The consistency check judges facts by path evidence only where another path joins their entities, true of 2.5 percent of named-entity facts in our compact WebQSP and CWQ contexts.
Seunghui Jwa, Minsu Oh, Sangwoo Hong +1
Department of Artificial Intelligence, Jeju National University · Department of Computer Science and Engineering, Konkuk University
Large language models and LLM-based agents are widely used as personal chat assistants, enterprise copilots, and autonomous workflow agents. In all these applications, memory (the ability to retain, access, and reason over information accumulated over long contexts and multiple interactions) plays a crucial role in determining the reliability of any agent. We introduce RECON (Reasoning over Extended Contexts with Obfuscated Narratives), a benchmark for evaluating compositional reasoning over long contexts. RECON spans 24 case files across three domains (criminal, medical, and financial), each ranging from 50k to 100k tokens, and tests agents on six memory intensive tasks: reconstructing multi-hop evidence chains, propagating cascading invalidations, resolving source conflicts, counterfactual reasoning, satisfying temporal constraints, and temporal fact retrieval. Recent memory benchmarks evaluate whether agents can retrieve scattered facts or detect if a fact has changed whereas RECON evaluates what happens after the change, whether agents can trace which downstream conclusions are affected, which survive through independent support, and how alternative timelines would have unfolded. Our evaluation reveals substantial limitations across current architectures: even the strongest non-Oracle system reaches only 22.4% Accuracy, with retrieval and reasoning each surfacing as challenges.
Mihir Shriniwas Arya
Department of Computer Science and Engineering RV College of Engineering