Neural network (the foundation of modern AI)

Definition

A neural network is a computational model composed of interconnected units called neurons, organized into layers. Each neuron computes a weighted sum of its inputs, adds a bias, and applies a non-linear activation function. Neurons pass their output to the next layer, and the final layer produces the network's answer. The 'learning' happens by adjusting the weights and biases so the network's outputs match the desired outputs on training examples.

The core learning algorithm is backpropagation, published by Rumelhart, Hinton, and Williams in 1986. Backprop uses the chain rule of calculus to compute how much each weight contributed to the error, and gradient descent then updates weights to reduce that error. Repeat this on millions of examples and the network learns to map inputs to outputs even for tasks that resist explicit programming.

Modern neural networks are 'deep' — they have many layers (hence 'deep learning'). A modern large language model is a neural network with 60-100+ layers and hundreds of billions of parameters. An image generation model like Stable Diffusion is a neural network. Every specialized architecture — Transformer, CNN, RNN, Mamba — is a specific pattern of neural network structure.

What makes neural networks powerful is universal approximation: given enough width and depth, they can approximate any continuous function. What makes them practical is that gradient descent finds surprisingly good solutions even in enormous parameter spaces. What makes them dominant in 2026 is the combination of scale (billions to trillions of parameters), massive training data, and specialized hardware (GPUs, TPUs) — the perfect confluence that produced the current era of AI.

Real-world example

A minimal neural network — classifying handwritten digits

The classic first neural network task is MNIST — classifying 28×28 pixel images of handwritten digits (0-9). A simple network takes 784 input pixels (flattened), passes them through a hidden layer of, say, 128 neurons with ReLU activation, then outputs 10 numbers (one per digit class). Softmax turns those numbers into probabilities. Trained on 60,000 examples with backpropagation, this simple architecture reaches about 98% accuracy on unseen digits. From here, everything scales up — bigger networks, more data, deeper layers, better architectures.

# Minimal neural network in PyTorch
import torch
import torch.nn as nn
import torch.nn.functional as F

class SimpleNN(nn.Module):
    def __init__(self):
        super().__init__()
        # 784 input pixels → 128 hidden neurons → 10 output classes
        self.fc1 = nn.Linear(784, 128)
        self.fc2 = nn.Linear(128, 10)

    def forward(self, x):
        # Flatten input
        x = x.view(-1, 784)
        # First layer with ReLU activation
        x = F.relu(self.fc1(x))
        # Output layer (logits — softmax applied in loss)
        x = self.fc2(x)
        return x

# Training loop (simplified)
model = SimpleNN()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

for images, labels in train_loader:
    # Forward pass
    logits = model(images)
    loss = F.cross_entropy(logits, labels)
    # Backward pass (backpropagation happens here)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
# After enough epochs: ~98% accuracy on MNIST test set

When you'll encounter this

• You're using any AI product — every one is a neural network under the hood.
• You're reading AI research — neural network vocabulary is assumed throughout the field.
• You're building AI features — understanding the foundation helps you reason about capabilities and limitations.
• You're training models — every training decision (learning rate, batch size, architecture) is a neural network hyperparameter.
• You're debugging model behavior — understanding that models 'learn from examples' rather than 'know rules' explains many failure modes.

How it works

Common misconceptions

Related terms

Where you'll see this in practice

Learn more

• Read the deep-dive: How a Neural Network Works
• Take the AI from Zero learning path
• Read: Understanding Gradient Descent
• Browse Foundational ML category