RAAFT: Rank-Adapted Atomic Function Tokenization

Abstract

Rank-Adapted Atomic Function Tokenization (RAAFT) is a parameter-efficient fine-tuning (PEFT) framework that combines two mechanisms: (1) context-conditioned adaptation rank — the subject of the companion HORA proposal, and the established premise of AdaLoRA [1] — and (2) composing each weight update from a small library of reusable atomic function tokens (scale, rotate, project, shift) via higher-order composition [2, 3, 4]. RAAFT sits in the PEFT lineage of LoRA [5], AdaLoRA [1], and adapter tuning [6]. This paper specifies the framework, gives an Input–Process–Output (IPO) worked example of the update procedure, and states what the named, composable token library buys: interpretability and modularity by construction — every weight update is a named, inspectable transformation — at competitive adaptation rank.

1. Introduction

Full fine-tuning is expensive and risks catastrophic interference [7, 8]. PEFT methods reduce cost by learning few parameters over a frozen base. LoRA [5] adds a fixed-rank low-rank update; AdaLoRA [1] allocates rank adaptively across modules; adapters [6] insert fixed bottleneck modules. RAAFT (a) lets the rank be a function of context, as in HORA and AdaLoRA, and (b) builds the update out of a small, named, reusable set of parameterized transformations ("atomic function tokens") that are composed rather than learned monolithically. The benefit of (b) is interpretability and modularity — each token is a nameable transformation. That is the property the framework is built to expose.

2. Related Work

PEFT. LoRA [5] (fixed-rank low-rank updates), AdaLoRA [1] (adaptive per-module rank budget — the reference point for any adaptive-rank claim), and adapter tuning [6] (bottleneck modules) are the PEFT reference points. RAAFT's rank-adaptation half builds on AdaLoRA's context-conditioned-rank premise [1]; its atomic-token composition is what has no direct precedent.

Catastrophic forgetting. The reason to prefer localized adaptation over full retraining is sequential-learning interference [7] and its partial mitigations [8]; RAAFT freezes base weights and inherits PEFT's usual protection.

Functional composition. The "atomic function token → higher-order composition" vocabulary is the functional-programming tradition formalized by Backus [2], Hughes [3], and Strachey [4]. RAAFT applies that composition to structure the weight update itself.

3. The RAAFT Framework

RAAFT composes a layer's weight update from context-conditioned factors and a token library:

ΔW(C) = 𝓕( f₁(C)·T₁·g₁(C), …, f_k(C)·T_k·g_k(C) )

where C is the context descriptor (gradients, activations, layer type, epoch), each Tᵢ is an atomic function token drawn from a small library {scale, rotate, project, shift}, fᵢ/gᵢ produce context-conditioned low-rank factors at a rank r = H(C), and 𝓕 is the higher-order composition that combines the token contributions. Every symbol is notation for the framework. Because each Tᵢ names a transformation, every contribution to ΔW is inspectable — the interpretability property the design is built around.

4. Worked Example (IPO)

The update procedure in IPO cognitive-DSL form — a higher-order function over atomic functions. ○[ILLUSTRATIVE] — a specification of the intended control flow.

raaft_update_HOF:
 description: "Build one layer's weight update from context-conditioned atomic function tokens."
 input: "layer_context" # gradients, activations, layer type, epoch
 process:
 - extract_context:
 type: atomic_function
 input: "layer_context"
 process: "summarize_layer_and_signal_state"
 output: "context_descriptor"
 - select_rank:
 type: atomic_function
 input: "context_descriptor"
 process: "map_context_to_rank" # H(C); cf. AdaLoRA budget allocation
 output: "rank"
 - assign_tokens:
 type: atomic_function
 input: ["context_descriptor", "rank"]
 process: "select_atomic_tokens_and_factors" # {scale, rotate, project, shift}
 output: "token_terms"
 - compose_update:
 type: atomic_function
 input: "token_terms"
 process: "higher_order_compose_terms" # the F(...) composition
 output: "delta_W"
 output: "delta_W" # applied additively to the frozen base weights

5. Design Considerations

• Modularity. Because the tokens and the maps H/f/g are separate atomic functions, they can be swapped without rewriting the framework — a maintainability property [3].
• Overhead. Context extraction and per-token composition add runtime cost; whether a lower effective rank amortizes it is a design tradeoff.
• Named updates. Each token is a nameable transformation, so a RAAFT update is auditable and swappable where a monolithic low-rank delta is opaque.

6. What RAAFT Contributes

The contribution is the atomic-token composition: each weight update is built from a named library of reusable transformations — scale, rotate, project, shift — composed as higher-order functions [2, 3], so every update is interpretable and modular by construction rather than an opaque low-rank delta. Naming each transformation in the update makes it auditable and swappable — the interpretability and modularity a low-rank delta cannot offer, and the part of RAAFT without PEFT precedent.

Evidence & Scope

RAAFT's contribution is the named atomic-token composition — the component without direct PEFT precedent. Its interpretability and modularity follow from naming each transformation in the update. The context-conditioned rank is compared for adaptive rank on the GLUE/SuperGLUE [9] families at equal parameter budget. "RAAFT," "Atomic Function Token," and "AFT" are original to this lineage.

7. Conclusion

RAAFT is a PEFT framework built on a compositional token library [2, 3] over a context-conditioned rank [1]. Its interpretability and modularity follow from naming each transformation in the update — the atomic-token composition, named and composable by construction, is the contribution.

References

1. Qingru Zhang et al. (2023). AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. International Conference on Learning Representations (ICLR). arXiv:2303.10512.
2. John Backus (1978). Can Programming Be Liberated from the von Neumann Style? A Functional Style and Its Algebra of Programs. Communications of the ACM. [1977 ACM Turing Award Lecture]
3. John Hughes (1989). Why Functional Programming Matters. The Computer Journal.
4. Christopher Strachey (2000). Fundamental Concepts in Programming Languages. Higher-Order and Symbolic Computation. [Reprint of 1967 lecture notes]
5. Edward J. Hu et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. International Conference on Learning Representations (ICLR). arXiv:2106.09685.
6. Neil Houlsby et al. (2019). Parameter-Efficient Transfer Learning for NLP. Proceedings of the 36th International Conference on Machine Learning (ICML). arXiv:1902.00751.
7. Michael McCloskey & Neal J. Cohen (1989). Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem. Psychology of Learning and Motivation.
8. James Kirkpatrick et al. (2017). Overcoming Catastrophic Forgetting in Neural Networks. Proceedings of the National Academy of Sciences (PNAS). arXiv:1612.00796. [Elastic Weight Consolidation (EWC)]
9. Alex Wang et al. (2019). SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. Advances in Neural Information Processing Systems (NeurIPS). arXiv:1905.00537.