A neural network is a machine learning model made of many simple calculation units, called neurons, arranged in layers. Each neuron multiplies its inputs by adjustable numbers called weights, adds them up, and passes the result through a simple function. Data flows through the layers to produce a prediction, and the network improves by comparing predictions with correct answers and nudging its weights to reduce the error. Repeat that enough times on enough examples, and the network learns to recognize faces, translate languages, or flag fraud.
The name suggests a digital brain, but the reality is more modest and more useful to understand. A neural network is arithmetic: lots of multiplication and addition, organized cleverly and tuned automatically. This article builds the idea from a single neuron up, shows how the network learns, runs a small example you can try yourself, and explains where neural networks excel and where they struggle.
The Building Block: One Artificial Neuron
IBM describes neural networks as stacks of simple neurons that learn weights and biases from data to map inputs to outputs. A single neuron does three things:
- Weighs the inputs. Each input is multiplied by a weight. A large weight means that input matters a lot, a weight near zero means it is mostly ignored, and a negative weight pushes the result the opposite way.
- Adds a bias. The bias is an extra number added to the weighted sum. It shifts the neuron’s threshold, so the neuron can respond even when its inputs are weak or stay quiet unless they are strong.
- Applies an activation function. The sum is passed through a function that reshapes it before sending it onward.
IBM’s spam-detection example makes this concrete: a word like “prize” might receive a larger weight than a word like “hello,” because it is more telling.
Here is a tiny calculation. Suppose a neuron receives inputs 2 and 1, with weights 0.5 and -1, and a bias of 0.5. The weighted sum is (2 × 0.5) + (1 × -1) + 0.5 = 0.5. Passed through ReLU, an activation function that keeps positive values and turns negative ones into zero, the output is 0.5. That is all a neuron does. Networks get their power from combining huge numbers of them.
Why activation functions matter
Common activation functions include ReLU, tanh (which squeezes values between -1 and 1), and sigmoid (which squeezes them between 0 and 1, handy for probabilities). They are not decoration. Without a nonlinear activation, stacking layers would accomplish nothing extra, because a chain of purely linear steps collapses into a single linear step. The nonlinearity is what lets deeper networks capture curved, complicated relationships.
Layers: Input, Hidden, and Output
Neurons are organized into layers:
- Input layer: holds the raw features, such as pixel brightness values, word encodings, or sensor readings.
- Hidden layers: neurons that combine the previous layer’s outputs and pass results forward, transforming the data into new representations.
- Output layer: produces the final prediction. For predicting a number, it outputs that number. For classification, it often outputs probabilities, one per category.
IBM notes that networks with several layers of this kind are the basis of deep learning. In a fully connected network, each neuron receives every output from the previous layer, so the number of weights grows quickly. A modest network can have thousands of weights, and the largest modern networks have billions.
The forward pass
Running data through the network from input to output is called the forward pass. Layer by layer, each neuron computes its weighted sum and activation, and its output becomes an input to the next layer. Libraries implement this with matrix multiplication so that many neurons are computed at once. Early in training, a network’s weights are small random values, so the forward pass produces essentially random predictions.
How a Neural Network Learns
Learning means finding good values for the weights and biases. IBM and other explainers describe the same training cycle:
- Forward pass: the network makes a prediction.
- Error calculation: a loss function scores how far the prediction is from the correct answer.
- Backpropagation: an algorithm works out how much each weight and bias contributed to the error.
- Update: each weight is adjusted slightly in the direction that reduces the error, typically using gradient descent.
Backpropagation is the clever part. A network may have millions of weights, and you need to know how to change every one of them. Backpropagation applies the chain rule from calculus to pass the error backward from the output layer, assigning each weight its share of the blame. A 1986 paper by Rumelhart, Hinton, and Williams showed that this method repeatedly adjusts connection strengths to bring actual outputs closer to target outputs and that doing so causes hidden units to learn useful features of the task. The authors highlighted that ability to construct new features as the key difference from simpler earlier methods.
Gradient descent then takes a small step against the gradient, scaled by a learning rate. One full pass over the training data is called an epoch, and the loss should generally fall as epochs accumulate.
A Worked Example: Teaching a Network XOR
XOR is a classic test. Given two inputs that are each 0 or 1, the output should be 1 only when exactly one input is 1. It sounds trivial, but no single neuron can solve it, because no straight line can separate the cases that should output 1 from those that should output 0. A network with a hidden layer can.
This example builds a network with 2 inputs, 4 hidden neurons using tanh, and 1 output neuron using sigmoid, then trains it with backpropagation and gradient descent in plain NumPy.
Python
import numpy as np
# XOR: output is 1 only when exactly one input is 1
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
y = np.array([[0], [1], [1], [0]], dtype=float)
rng = np.random.default_rng(1)
W1 = rng.normal(0, 1, (2, 4)) # 2 inputs -> 4 hidden neurons
b1 = np.zeros((1, 4))
W2 = rng.normal(0, 1, (4, 1)) # 4 hidden neurons -> 1 output neuron
b2 = np.zeros((1, 1))
lr = 0.5
def sigmoid(z):
return 1 / (1 + np.exp(-z))
for epoch in range(10001):
# forward pass
h = np.tanh(X @ W1 + b1)
p = sigmoid(h @ W2 + b2)
loss = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))
# backward pass: the chain rule, one layer at a time
dz = (p - y) / len(X)
dW2 = h.T @ dz
db2 = dz.sum(axis=0, keepdims=True)
dh = (dz @ W2.T) * (1 - h ** 2)
dW1 = X.T @ dh
db1 = dh.sum(axis=0, keepdims=True)
# gradient descent update
W1 -= lr * dW1; b1 -= lr * db1
W2 -= lr * dW2; b2 -= lr * db2
if epoch in (0, 1000, 10000):
print(f"epoch {epoch:5d} loss={loss:.4f}")
print("predictions:", np.round(p.ravel(), 3))Running it with NumPy produces this output:
epoch 0 loss=0.6712
epoch 1000 loss=0.0051
epoch 10000 loss=0.0005
predictions: [0. 0.999 0.999 0.001]The predictions for the inputs (0,0), (0,1), (1,0), and (1,1) come out as roughly 0, 1, 1, 0, which is exactly XOR. The loss started around 0.67, which is what random guessing looks like, and fell to nearly zero.
One honest detail: training is not guaranteed to succeed. When I tested the same setup with a different random starting point (seed 0), the network got stuck, producing roughly 0, 0.5, 1, 0.5 after the same number of epochs. Where the weights begin affects where gradient descent ends up. This is a small version of a real issue: neural network training can be sensitive to initialization, learning rate, and architecture, so practitioners often run several trials.
Why Hidden Layers Make Networks Powerful
Hidden layers let the network build its own intermediate features instead of relying on features designed by hand. In an image model, early layers tend to respond to simple patterns such as edges, later ones to parts such as textures or shapes, and the last layers to whole objects. Nobody writes those features. They emerge during training, which is the property the 1986 paper emphasized.
A well-known theoretical result supports the idea that these networks are very expressive. The universal approximation theorem says that a feedforward network with a single hidden layer can approximate any continuous function on a bounded region to any desired accuracy, provided the activation function is non-polynomial. Early proofs date to Cybenko in 1989 and to Hornik, Stinchcombe, and White in the same year.
The theorem is easy to over-read, though. It says a suitable network exists. It does not say how to find its weights, how many neurons the task requires, or how the network behaves outside the region it was trained on. Training with backpropagation is a separate matter, and the theorem does not guarantee it will succeed.
Common Types of Neural Networks
IBM lists several major families, each suited to a different kind of data:
| Type | Designed for | Typical uses |
|---|---|---|
| Feedforward (multilayer perceptron) | General tabular or simple prediction tasks | Basic classification and regression |
| Convolutional (CNN) | Grid-like data such as images | Image and facial recognition, medical imaging |
| Recurrent (RNN) | Sequences, using feedback loops to carry information over time | Speech recognition, time series forecasting |
| Transformer | Sequences, using attention mechanisms | Language models such as GPT |
IBM notes that transformers have replaced recurrent networks for many sequence tasks. The large language models behind modern chatbots are transformer-based neural networks, which makes them deep learning systems.
A Short History
- 1943: Warren McCulloch and Walter Pitts propose the first mathematical model of a neuron.
- 1958: Frank Rosenblatt introduces the perceptron for pattern recognition, which IBM calls the historical ancestor of today’s networks.
- 1986: Rumelhart, Hinton, and Williams publish their work on learning by back-propagating errors.
- 2012: A deep convolutional network called AlexNet wins the ImageNet competition by a wide margin, after training on a large dataset using graphics cards, and draws wide attention to deep learning.
The ideas are old. What changed was data and computing power.
Strengths and Limits
Where neural networks shine: unstructured data such as images, audio, and text, where they learn useful features automatically and complex patterns that are hard to describe with hand-written rules.
Where they struggle:
- Data and compute: large networks typically need large datasets and substantial computing resources.
- Overfitting: IBM identifies this as a key challenge. A network that grows too complex can fit its training data, noise included, and then predict poorly on new data.
- Opacity: with millions of weights, it is hard to explain exactly why a network produced a given output, which matters in settings like lending or medicine.
- Inherited bias: a network learns whatever patterns its data contains, including unfair ones.
- Brittleness outside its training range: a network can behave unpredictably on inputs unlike anything it has seen.
Neural networks are also not brains. They borrow a loose metaphor from biology but are mathematical functions with adjustable numbers inside.
Practical Takeaways
- Match the architecture to the data. Images suggest a convolutional network, text suggests a transformer, and tabular business data often does well with simpler models.
- Start small. A small network or a classical model makes a useful baseline before reaching for something large.
- Test on held-out data. Judge a network by performance on examples it never trained on.
- Expect some trial and error. Learning rate, initialization, and layer sizes all affect whether training works.
- Keep humans in the loop for high-stakes decisions. A confident output is not the same as a correct one.
Conclusion
A neural network is layers of simple neurons, each computing a weighted sum plus a bias and passing it through an activation function. It learns by running the training loop repeatedly: predict, measure the error, use backpropagation to assign blame to each weight, and take a small gradient descent step. Hidden layers let it discover its own useful features, which is why neural networks became the engine of modern AI.
The mechanism is simple enough to build in under forty lines of code, and powerful enough to write essays when scaled up enormously. Keeping both facts in mind protects against treating neural networks as either magic or toys.