Self-Attention Mechanization for HOF Cognition

Abstract

Self-attention [1] is the core operation of the transformer architecture and its scaled descendants [2, 3]. This article states the mechanism — query/key/value projection, scaled dot-product scoring, softmax normalization, and value aggregation — and reads it as a higher-order function: an operation parameterized by a learned weighting over a set of representations, in the sense of function-level programming [4, 5, 6]. Attention factors into three reusable pieces — project, weight, aggregate — and that structure is what lets it serve as a building block in larger cognitive pipelines. This is an expository account: the mechanism is the transformer's [1], and the compositional reading is the contribution.

1. Introduction

The transformer [1] replaced recurrence with attention, allowing every position in a sequence to be related directly to every other position in a single, parallelizable operation. This is the property that scaled models such as BERT [2] and GPT-3 [3] are built on.

Attention is usually presented as a fixed architectural component. For building composable cognition it is more useful to read it as a higher-order operation: a function parameterized by a learned scoring function over a set of representations. That reading is the subject of this article — the mechanism stated exactly, then factored into the reusable pieces (projection, weighting, aggregation) that compose.

2. Related Work

Self-attention and transformers. The scaled dot-product attention operation and the multi-head transformer are due to Vaswani et al. [1]; pretrained transformer encoders [2] and decoder-only language models [3] established that a single attention-based backbone transfers across many tasks. The quadratic time and memory cost of exact attention in sequence length is a documented property of this design [1] and the motivation for the efficiency literature discussed in companion articles [7].

Functional composition. Reading attention as a higher-order operation — a function that takes a set of representations and a learned scoring function and returns a composed representation — applies the standard vocabulary of function-level programming: Backus's Turing-award account of programming without the von Neumann bottleneck [4], Hughes on why higher-order functions and composition aid modularity [5], and Strachey's foundational treatment of functions as first-class values [6]. Attention is a natural instance: its weighting step produces a function over the value set, and its aggregation step applies it.

3. The Mechanism

Given an input sequence encoded as a matrix X whose rows are per-position embeddings, self-attention computes three linear projections with learned weight matrices W_Q, W_K, W_V:

• Q = X · W_Q (queries), K = X · W_K (keys), V = X · W_V (values).

Scores are the scaled dot products of queries against keys, normalized by softmax into a row-stochastic weighting, which is then applied to the values:

Attention(Q, K, V) = softmax( (Q · Kᵀ) / √d_k ) · V

where d_k is the key dimensionality and the 1/√d_k scaling controls the variance of the dot products before the softmax [1]. Stacking multiple such layers (and multiple heads per layer) lets deeper layers attend over the representations produced by shallower ones.

4. Attention as a Higher-Order Operation (IPO framing)

Attention fits an Input–Process–Output (IPO) description whose Process step is itself parameterized by a function, which is what makes it higher-order in the functional-programming sense [4, 5].

• Input. A set of position representations X (no ordering assumption beyond the positional encoding carried in X).
• Process. (1) project X → Q, K, V; (2) form a weighting A = softmax(QKᵀ/√d_k); (3) aggregate A · V. Step (2) produces a function over the value set — the weighting — which step (3) then applies. The operation is thus a composition: aggregate ∘ weight ∘ project, in the compositional style of [4, 6].
• Output. One contextualized representation per input position.

The framing makes explicit which parts of attention are reusable composable pieces — projection, weighting, aggregation — when building larger cognitive pipelines, the same modularity higher-order functions provide generally [5].

5. Established Properties

From the cited literature, the following properties of self-attention carry into the compositional reading:

• Direct long-range interaction. Any two positions interact in one layer, without the path-length growth of recurrence [1].
• Parallelism. The core operation is a pair of matrix multiplies with a softmax between, parallelizable across positions [1].
• Transferable backbones. A single attention backbone adapts across tasks via pretraining and fine-tuning [2, 3].

Evidence & Scope

This is an expository account. The mechanism is the transformer's [1], stated exactly; the contribution is the compositional reading that factors attention into projection, weighting, and aggregation and names them as reusable higher-order pieces [4, 5, 6]. The properties it carries forward — direct long-range interaction, parallelism, transferable backbones — are the ones established in the cited literature [1, 2, 3]. Exact self-attention is quadratic in sequence length in time and memory [1]; the IO-aware kernels that address that cost without approximation [7] are treated in companion work.

References

1. Ashish Vaswani et al. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems (NeurIPS). arXiv:1706.03762.
2. Jacob Devlin et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of NAACL-HLT. arXiv:1810.04805.
3. Tom B. Brown et al. (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems (NeurIPS). arXiv:2005.14165. [GPT-3]
4. John Backus (1978). Can Programming Be Liberated from the von Neumann Style? A Functional Style and Its Algebra of Programs. Communications of the ACM. [1977 ACM Turing Award Lecture]
5. John Hughes (1989). Why Functional Programming Matters. The Computer Journal.
6. Christopher Strachey (2000). Fundamental Concepts in Programming Languages. Higher-Order and Symbolic Computation. [Reprint of 1967 lecture notes]
7. Tri Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. Advances in Neural Information Processing Systems (NeurIPS). arXiv:2205.14135.