Abstract

Contemporary practice is dominated by large pretrained transformers [1, 2], and a broad inventory of classical machine-learning techniques — reinforcement-learning methods, ensembles, unsupervised representation learning, semi-supervised learning, evolutionary optimization, anomaly detection, dimensionality reduction, Bayesian methods, graph-based learning, and transfer learning — retains properties that heavy neural models do not automatically provide: interpretability, modest compute cost, robustness to overfitting, and principled uncertainty handling. This survey maps these families and, for each, states the reason it remains worth including in a general-purpose system. Two threads connect the inventory to the deep-learning literature: the documented tendency of monolithic sequential learning to forget [3, 4], which is the case for keeping lighter, modular methods in the toolbox, and the Bayesian account of reasoning under uncertainty [5], which several of the surveyed families instantiate. The claim is about system design: these methods build more efficient, interpretable, and robust general-purpose systems.

1. Introduction

The center of gravity in machine learning has moved to large pretrained models [1, 2]. That success carries documented costs: substantial compute, limited interpretability, and — for monolithic models trained sequentially — catastrophic forgetting [3, 4]. Classical methods trade peak capability for exactly the properties that mitigate these costs. This survey covers the families most worth keeping and gives the case for each.

The defensible claim is direct: these methods help build more efficient, interpretable, and robust general-purpose systems. That is a claim about the toolbox a capable system draws on, and it is what the survey supports.

2. Related Work

This is a survey, so the related work is the backdrop it is written against: the transformer-scaling line that motivates the comparison [1, 2]; the catastrophic-interference literature that motivates keeping modular, non-monolithic methods available [3, 4]; and the Bayesian-cognition account of inference under uncertainty [5], which underpins the probabilistic families in §3.10. The individual methods surveyed are textbook techniques and are cited to their sources rather than re-derived.

3. Survey of Method Families (qualitative relevance)

Each entry gives the family and the reason it remains relevant to a general-purpose system.

3.1 Reinforcement learning. Actor–critic methods balance value- and policy-based learning; temporal-difference learning updates incrementally and supports online, incomplete-episode learning; hierarchical/decomposed control (e.g., DQN with sub-task hierarchies) scales decision-making by factoring tasks. Relevance: continual, on-the-fly learning and task decomposition.

3.2 Ensembles. Boosting (gradient boosting, AdaBoost) combines weak learners; bagging and random forests reduce variance and handle high-dimensional, noisy inputs. Relevance: generalization and robustness at low compute relative to large networks.

3.3 Unsupervised representation learning. Autoencoders learn label-free encodings for compression, anomaly detection, and pretraining; PCA/ICA reduce dimensionality while preserving variance or statistical independence. Relevance: autonomous representation learning where labels are scarce.

3.4 Semi-supervised learning. Self-training and co-training exploit unlabeled data alongside a small labeled set. Relevance: mitigates label scarcity without exhaustive annotation.

3.5 Evolutionary and metaheuristic optimization. Genetic algorithms, particle-swarm optimization, and simulated annealing search large, non-differentiable spaces. Relevance: architecture/strategy search and optimization under hardware constraints.

3.6 Anomaly detection. Isolation forests and local outlier factor flag out-of-distribution inputs. Relevance: handling the unexpected inputs a general system will inevitably meet.

3.7 Dimensionality reduction / feature learning. NMF and factor analysis expose latent structure. Relevance: interpretability and reduced dimensional complexity.

3.8 Hybrid / meta-learning. Stacking and blending compose heterogeneous base models into a meta-model. Relevance: multi-modal, cross-domain combination of strengths.

3.9 Graph-based learning. Graph neural networks and label propagation model relational structure. Relevance: learning over interconnected entities and relations.

3.10 Bayesian methods. Naïve Bayes, Gaussian processes, and Bayesian networks give a principled account of reasoning and belief-updating under uncertainty [5]. Relevance: calibrated decisions from limited data and incremental belief revision.

3.11 Transfer and multi-task learning. Domain adaptation and knowledge transfer reuse learned capability across tasks. Relevance: generalization across contexts with reduced training cost — the same modularity that motivates avoiding monolithic retraining and its forgetting [3, 4].

4. Why These Should Not Be Overlooked

The recurring themes are: efficiency and interpretability (lower compute and clearer decision paths than large networks), scalability and flexibility across heterogeneous tasks, robustness to overfitting, and a foundation for hybrid models that pair classical methods with neural components. These are engineering arguments for keeping these methods in scope as first-class parts of a general-purpose system.

Evidence & Scope

This is a structured survey: it maps classical method families and states the system-design case for each, grounded in the catastrophic-interference literature [3, 4] and the Bayesian account of inference under uncertainty [5]. The claims are qualitative and about system design — where each family earns its place in the toolbox — rather than head-to-head benchmark results. The scope is deliberate: these methods contribute efficiency, interpretability, and robustness to a general-purpose system, and that toolbox is stronger for keeping them.

References

  1. Ashish Vaswani et al. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems (NeurIPS). arXiv:1706.03762.
  2. Tom B. Brown et al. (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems (NeurIPS). arXiv:2005.14165. [GPT-3]
  3. Michael McCloskey & Neal J. Cohen (1989). Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem. Psychology of Learning and Motivation.
  4. James Kirkpatrick et al. (2017). Overcoming Catastrophic Forgetting in Neural Networks. Proceedings of the National Academy of Sciences (PNAS). arXiv:1612.00796. [Elastic Weight Consolidation (EWC)]
  5. Thomas L. Griffiths et al. (2008). Bayesian Models of Cognition. The Cambridge Handbook of Computational Psychology.