Part 21: Backpropagation: How Neural Networks Learn From Their Mistakes

Imagine you are learning to shoot basketball hoops. 🏀

You take a shot, but the ball misses the basket.

Instead of simply trying again randomly, you think:

“I aimed too far to the left. Next time, I should adjust my aim slightly to the right.”

You take another shot, make a small correction, and gradually improve.

This is the basic idea behind backpropagation.

A neural network also makes predictions, makes mistakes, and then learns from those mistakes by adjusting its weights.

Let's understand how this works.

🤔 What Is Backpropagation?

Backpropagation is the process a neural network uses to determine how much each weight contributed to the error and then adjust those weights to improve future predictions.

In simple words:

Prediction → Find Error → Trace Error Backward → Update Weights → Improve

It is one of the key processes that allows neural networks to learn.

🎯 Step 1: The Network Makes a Prediction

Before backpropagation can happen, the neural network first makes a prediction.

For example, imagine we want a neural network to recognize whether an image contains a cat 🐱 or a dog 🐶.

We give it an image of a cat.

The network processes the image through its layers and predicts:

Prediction: 0.30 → Cat

But suppose the correct answer is:

Actual: 1.00 → Cat

The prediction isn't very accurate.

So, the network needs to learn from this mistake.

❌ Step 2: Find the Error

The difference between what the network predicted and the correct answer is measured using a loss function.

For a simple example, we can represent the error as:

For example:

Think of the loss function as a teacher giving the student a score.

If the prediction is close to the correct answer, the loss is small.

If the prediction is far away, the loss is large.

For example:

Actual answer: 1.0

Prediction: 0.3

Loss: High ❌

The network now knows:

“My prediction was not good enough.”

But knowing that there is an error isn't enough.

The network needs to figure out what caused the error.

That's where backpropagation comes in.

🔙 Step 3: The Error Travels Backward

The word backpropagation comes from:

Backward + Propagation

The network starts from the output and works backward through the layers.

It calculates how much each weight contributed to the final error.

Formula:

This tells the network how much the loss changes when a particular weight changes.

In simple terms, it helps the network understand which weights need to be changed and in which direction.

Imagine you are studying for an exam and get a question wrong.

Instead of saying, “I got it wrong,” you ask:

• Did I misunderstand the concept?

• Did I use the wrong formula?

• Did I make a calculation mistake?

You trace the mistake back to its source.

Backpropagation does something similar.

It identifies which weights need bigger changes and which need smaller changes.

⚙️ Step 4: Update the Weights

Once the network knows how each weight contributed to the error, it adjusts them.

This is called a weight update.

The basic weight-update formula is:

Here:

Don't worry if the formula looks complicated.

The important idea is:

If a weight is contributing to the error, change it in a direction that reduces the error.

The learning rate controls how big the adjustment should be.

Think about adjusting the temperature of an oven. 🔥

If your food is slightly undercooked, you might increase the temperature a little.

You wouldn't suddenly increase it to the maximum.

Similarly, a neural network usually makes controlled adjustments instead of changing its weights drastically.

🔁 Step 5: Learning Happens Again and Again

One correction isn't enough.

The network repeats the process:

This process happens many times during training.

Gradually, the network's predictions become better and the loss decreases.

🧠 A Simple Real-Life Analogy

Think about learning to ride a bicycle. 🚲

At first, you may lose your balance.

You notice:

“I'm leaning too much to the right.”

You correct your balance.

You try again.

Maybe now you lean too far left, so you make another correction.

After many attempts, your brain learns the right movements.

Neural networks learn in a similar trial-and-correction process.

The difference is that instead of consciously thinking about the mistakes, the neural network uses loss, gradients, and weight updates to make those corrections.

🚀 Final Takeaway

Backpropagation is essentially a neural network's learning-from-mistakes mechanism.

It helps the network answer three important questions:

❌How wrong was my prediction?

→ Loss function

🔍Which weights contributed to the mistake?

→ Backpropagation

🔧How should I change those weights?

→ Weight updates using gradients

By repeating this process thousands or millions of times, a neural network can gradually learn patterns and make better predictions.

And just like learning to ride a bicycle, the network doesn't become perfect in one attempt—it improves through repeated corrections. 🚲

🔮 Coming Up Next

We know how a neural network learns from its mistakes—but how does it know which direction to move to reduce the error?

In the next blog, we’ll explore Gradient Descent, the optimization technique that helps neural networks find better weights and improve their predictions.