Abstract

Reasoning-optimized LLMs improve answer quality by generating long internal chains of thought, which raises token consumption and therefore cost and latency. This paper presents compositional cognition: encapsulating recurring reasoning as reusable higher-order functions over atomic operations [1, 2, 3], passing summarized results between modules rather than carrying an exhaustive reasoning trail in a single context window. The approach rests on three established efficiency results: knowledge distillation, which compresses behavior into a cheaper form [4]; IO-aware attention, which reduces the systems cost of exact attention [5]; and self-consistency, which spends samples deliberately rather than exhaustively [6]. The reference point is reasoning-model inference, whose token cost grows with chain length; the token arithmetic here is illustrative under stated assumptions, and the condition that decides the saving — whether a reused abstraction fits the problem at hand — is named explicitly rather than buried.

1. Introduction

Reasoning models earn accuracy by "thinking longer" — emitting extended chains of intermediate tokens before answering. The trade is real and useful, and its cost scales with the amount of intermediate text produced, which is why long-context, deep-reasoning inference is expensive.

Compositional cognition takes a different emphasis: rather than re-deriving reasoning token-by-token per problem, encapsulate recurring reasoning as reusable higher-order functions and pass compact summaries between modules, spending tokens on decisions that change the output and reusing prior results elsewhere. This is abstraction and reuse over exhaustive iteration — an engineering direction built on three efficiency results already in production use.

2. Related Work

Reasoning models and token cost. Transformer LLMs [7, 8] that reason by generating long intermediate chains pay a token cost proportional to the length of those chains. This is the cost the design targets.

Efficiency lineage the design rests on. Knowledge distillation [4] compresses a large model's behavior into a smaller, cheaper one — the canonical "do the same with less compute" result, and the closest established analogue to encapsulating reasoning into a reusable function. FlashAttention [5] is a benchmarked technique for reducing the memory/IO cost of exact attention; it is the citation for any claim about making attention cheaper. Self-consistency [6] samples several reasoning chains and votes, improving reliability by spending samples deliberately; it adds samples rather than removing them — a reliability technique, and the compositional argument treats it as one, not as a source of token savings.

Grounding as a context-reduction lever. Retrieval-augmented generation [9] lets a model consult external memory instead of holding everything in context — the same summarize-and-reference intuition the design applies to reasoning state; it reduces, without removing [10], the need to keep exhaustive trails in-window.

Composition. Encapsulating reasoning as higher-order functions over atomic operations is the functional-composition discipline formalized by Backus, Hughes, and Strachey [1, 2, 3]; compositional cognition applies it to reasoning state.

3. The Design: Abstraction over Iteration

Claim. Recurring reasoning is encapsulated as reusable higher-order functions over atomic operations; modules pass compact summaries rather than full reasoning trails; and reasoning strategy is selected per task rather than applied uniformly. The effect is to spend tokens on decisions that change the output and to reuse prior results instead of re-deriving them.

The mechanism is analogy to established practice: distillation compresses behavior into a cheaper form [4]; RAG-style summarize-and-reference reduces what must be held in context [9]; efficient attention lowers the per-token systems cost [5]. The saving these compound into is governed by one condition — that the reused abstraction fits the problem it is applied to — which the next section makes precise.

4. Illustrative Token Arithmetic

[ILLUSTRATIVE] arithmetic, under stated assumptions: if a reasoning model emits t intermediate tokens per problem and a compositional approach reuses a cached abstraction so that it emits only s tokens plus a fixed overhead c, then the per-problem token ratio is (s + c) / t. The values of t, s, and c are inputs to the arithmetic, not measurements; the saving is real exactly when the abstraction is reusable and correct for the problem at hand. That condition — reuse fit — is the substantive variable, the way sparse-subset selection is the substantive variable for a retrieval index.

5. Evidence & Scope

Compositional cognition is a design with one decisive variable. Its efficiency citations bound components, not the composite: distillation [4], FlashAttention [5], and RAG [9] are each measured, and the compositional layer inherits their guarantees where a reused abstraction fits the task. Two trade-offs follow directly and are made explicit by the design rather than hidden: a cached abstraction applied to a problem it does not fit yields a fast wrong answer, so reuse is gated on fit; and passing summaries between modules instead of full trails drops detail a later step may need [10], the same boundary retrieval systems operate under. The reference point is reasoning-model inference, whose token cost grows with chain length; reuse fit is the variable that decides whether the compositional pipeline comes in under it. "HOF cognition" and "compositional cognition" name the composition of distillation, retrieval, efficient attention, and functional composition [4, 5, 9, 1, 2, 3].

References

  1. John Backus (1978). Can Programming Be Liberated from the von Neumann Style? A Functional Style and Its Algebra of Programs. Communications of the ACM. [1977 ACM Turing Award Lecture]
  2. John Hughes (1989). Why Functional Programming Matters. The Computer Journal.
  3. Christopher Strachey (2000). Fundamental Concepts in Programming Languages. Higher-Order and Symbolic Computation. [Reprint of 1967 lecture notes]
  4. Geoffrey Hinton et al. (2015). Distilling the Knowledge in a Neural Network. NeurIPS Deep Learning and Representation Learning Workshop. arXiv:1503.02531.
  5. Tri Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. Advances in Neural Information Processing Systems (NeurIPS). arXiv:2205.14135.
  6. Xuezhi Wang et al. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. International Conference on Learning Representations (ICLR). arXiv:2203.11171.
  7. Ashish Vaswani et al. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems (NeurIPS). arXiv:1706.03762.
  8. Tom B. Brown et al. (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems (NeurIPS). arXiv:2005.14165. [GPT-3]
  9. Patrick Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems (NeurIPS). arXiv:2005.11401. [RAG]
  10. Ziwei Ji et al. (2023). Survey of Hallucination in Natural Language Generation. ACM Computing Surveys. arXiv:2202.03629.