The foundations of machine learning
The vocabulary every ML practitioner needs — from what a neural network actually is to how gradient descent updates weights. Start here if you're building an ML mental model from scratch.
Foundational ML is the bedrock of every modern AI system. Before you can meaningfully discuss transformers, RAG, or RLHF, you need the vocabulary in this category. A large language model is, ultimately, a very large neural network trained with gradient descent on billions of examples — every one of those concepts starts here.
These 80 terms are organized into five sub-topics: the learning process itself, optimization methods that make training work, regularization techniques that prevent overfitting, evaluation approaches, and the data that fuels everything. Every entry links to prerequisites, so if a term uses vocabulary you don't know, you can drill down until you hit familiar ground.
The most important terms in Foundational ML
Start here if you're new. These entries explain the foundational vocabulary in depth.
Neural Network
The core computational structure of modern AI — layers of connected nodes with weights adjusted during training. Every other term in this category ultimately describes something about how neural networks learn.
Read the full entry →OptimizationGradient Descent
The optimization algorithm that adjusts model weights to reduce error. Variations like SGD, Adam, and AdamW power essentially all deep learning training.
Read the full entry →LearningBackpropagation
The algorithm that calculates how much each weight contributed to the error, enabling gradient descent. The 1986 paper that made deep learning possible.
Read the full entry →LearningLoss Function
A numerical measure of how wrong a model's predictions are. Cross-entropy for classification, MSE for regression — the choice shapes what the model learns.
Read the full entry →ArchitectureActivation Function
The non-linearity applied at each neuron — ReLU, GELU, SwiGLU, Tanh. Without them, deep networks would collapse into simple linear models.
Read the full entry →EvaluationOverfitting
When a model memorizes training data instead of learning generalizable patterns. Detected by test-set performance dropping below training performance.
Read the full entry →All Foundational ML terms, organized
Grouped into sub-topics so you can find neighbors and prerequisites, not just alphabetical entries.
Learning fundamentals
Optimization
Regularization
Evaluation & metrics
Data & preprocessing
Related term categories
These categories connect naturally to Foundational ML — many terms cross-reference between them.
Go deeper on Foundational ML
📖 Related concept tutorials
Long-form guides that walk through how these concepts actually work.
- How a Neural Network Works
End-to-end walkthrough with math and code.
- Understanding Gradient Descent
Why the algorithm works and when it fails.
- How Backpropagation Works
The chain rule in action, step by step.
🎯 Related models & tools
Real products and models where you'll encounter these terms.
- Every AI model
Built on these foundational principles.
- Best open-weight models
Study the architectures freely.
- AI development tools
PyTorch, JAX, TensorFlow, and more.
Questions about Foundational ML
Not to use them — anyone can prompt ChatGPT or Claude. To build production applications, understand cost, or debug why a model fails, yes — the vocabulary here becomes essential. Start with our AI for Developers path.
Yes. Bigger models are still trained with gradient descent, still overfit, still need regularization, still use activation functions. The scale changes but the fundamentals persist.
Each term entry starts non-mathematically and adds math for readers who want it. Our concept tutorials go deeper with equations, but the term entries themselves are approachable.
Neural Network → Loss Function → Gradient Descent → Backpropagation → Overfitting → Regularization. Our AI from Zero path walks the sequence with exercises.
Basic linear algebra (vectors, matrices), calculus (derivatives, chain rule), and probability (distributions, expectation) help. But every term entry links to the prerequisites you need, so you can drill down as far as you need.