If we want to build a model that can generate images, the first question is not really “what neural network should we use?” The more basic question is:

Where should generation start?

This post starts from that question and gradually derives the core idea behind diffusion models.

The First Question: Where Does the Model Start?

Suppose we have a model whose goal is to generate an image. In the end, it outputs pixels, or more generally, a high-dimensional vector.

But before the model can output an image, it needs some input.

If the input is a sketch, we are doing image-to-image generation. If the input is a text prompt, we are doing text-to-image generation. But if we only consider the simplest unconditional image generation setting, what should the input be?

A natural but imprecise answer is: start from nothing.

The problem is that “nothing” is not a computable input. A neural network needs an actual tensor. If we feed it the same fixed input every time, it may keep producing the same kind of image, or even the same image. That would not give us real generative diversity.

So a generative model needs a random starting point.

We Need Randomness

Randomness gives the model freedom to produce different outcomes.

For example, we may want the model to sometimes generate a cat and sometimes a dog. Sometimes it should generate a street during the day, and sometimes a city at night. If the input is always exactly the same and the model itself is deterministic, it has no reason to output different images.

So a generative model usually does not learn a simple function like:

image = model()

Instead, it learns a function with random input:

image = model(randomness)

The random input itself does not need to have semantic meaning. It only needs to provide enough possibilities so that the model can map different random points to different images.

The Simplest Random Starting Point: Gaussian Noise

The easiest random object to sample is Gaussian noise.

It has several useful properties:

  1. It is easy to generate.
  2. Its mathematical properties are simple.
  3. In high-dimensional space, it provides rich random variation.

So we can imagine sampling a Gaussian noise image with the same shape as the target image, then asking a model to turn that noise into a real image.

In form, we want to learn:

noise -> image

But a difficulty appears immediately: pure noise and real images are very far apart.

Going From Noise to Image in One Step Is Too Hard

Asking a neural network to transform Gaussian noise into a complete image in one step means asking it to solve everything at once:

  1. What objects should appear in the image.
  2. How those objects should be arranged.
  3. How edges, textures, and colors should form.
  4. How local details should stay consistent with the global semantics.

This is not impossible, but the task is extremely steep. The model must jump directly from an unstructured input to a highly structured output.

It is like asking someone to turn a ball of tangled lines directly into a finished illustration. An easier way is not to draw everything at once, but to sketch the rough shape first and then refine the details.

This is the key intuition behind diffusion models: do not treat generation as one huge leap. Break it into many small steps.

Break One-Step Generation Into Many Small Steps

If going from pure noise to an image is too hard, we can insert many intermediate states:

noise -> slightly less noisy image -> ... -> clean image

Each step does one relatively simple thing: remove a little bit of noise.

The problem changes from “how do we generate an image directly from noise?” to “how do we make a slightly noisy image a little cleaner?”

The second problem is much easier. In intermediate states, some image structure may already be faintly present. The model does not need to invent the whole image from scratch. It only needs to decide which direction the current sample should move.

This is the generative view of diffusion models: generation is not completed instantly. It is a gradual denoising process.

Small Denoising Steps Are Easier

Why are small denoising steps easier?

Because the difference between two neighboring steps is small.

If an image is only corrupted by a small amount of noise, humans can usually still recognize what it roughly is. A model can learn a similar ability: look at a noisy image and predict the noise inside it, or predict a cleaner version.

If each step removes only a little noise, the model’s task becomes much more stable:

x_t -> x_{t-1}

Here, x_t is the image at noise level t, and x_{t-1} is the image with slightly less noise.

When we chain many of these steps together, we can eventually move from pure noise back to a clean image.

But We Do Not Have Denoising Paths

Now a new problem appears.

If we want to train a model to do:

noisy image -> less noisy image

we need many training examples of that form. In other words, we need to know what an image looks like at different noise levels, and what the target should be after each denoising step.

But real-world datasets usually only give us clean images:

clean image

They do not tell us which intermediate states this image would pass through if it were generated from noise. We do not naturally have a “generation path.”

This looks like a dead end: we want to train a denoising process, but we do not have denoising-process data.

The beautiful move in diffusion models is to reverse the problem.

Reverse the Direction: Add Noise to Real Images

We may not have a path from noise to image, but we can easily construct a path from image to noise.

Given a real image x_0, we can add Gaussian noise to it step by step:

clean image -> slightly noisy image -> ... -> pure noise

This is called the forward diffusion process.

It is not learned by the model. We define it manually. At each step, we add a little noise to the image. As the number of steps increases, the original image contains less and less recoverable information, until it becomes almost pure Gaussian noise.

Now we have many training examples:

clean image + chosen noise level -> noisy image

More importantly, because we added the noise ourselves, we know the correct answer. We know the exact noise that was added, and we also know the original clean image.

This creates the training data we need.

Train the Model to Remove Noise Step by Step

Once we have the noising process, the training objective becomes clear.

We take a real image x_0 from the dataset, randomly choose a timestep t, and add the corresponding amount of noise to obtain x_t. Then we ask the model to predict the noise from x_t and t.

In form:

predicted_noise = model(x_t, t)

During training, we know the true noise that was added, so we can directly compare the predicted noise with the real noise.

This is the common training strategy in DDPM: the model does not necessarily predict the clean image directly. Instead, it predicts the noise. Once we know the noise, we know what should be removed from the current image.

Intuitively, the model learns to answer:

At this noise level, which parts of this image look like noise, and which parts look like real image structure?

After training, the model has learned how to denoise step by step.

Generation: Start From Noise and Repeatedly Call the Model

During training, we start from real images and keep adding noise.

During generation, we reverse the direction: first sample a pure Gaussian noise image, then denoise it step by step.

The rough process is:

x_T = gaussian_noise

for t = T, T-1, ..., 1:
    noise = model(x_t, t)
    x_{t-1} = remove_a_little_noise(x_t, noise, t)

return x_0

At the beginning, x_T looks like complete noise. After several steps, some large-scale structure starts to appear. As denoising continues, shapes, colors, and details gradually stabilize. Finally, we obtain an image that looks like it came from the real data distribution.

This is the basic loop of a diffusion model:

  1. During training, gradually add noise to real images to construct supervised signals.
  2. Train the model to predict noise at arbitrary noise levels.
  3. During generation, start from Gaussian noise and repeatedly predict and remove noise.

Summary

The core idea of a diffusion model is not mysterious. It starts from a simple question: where should image generation begin?

If we start from a fixed input, we do not have enough randomness. If we start from Gaussian noise, generating the whole image in one step is too hard. So we break the large problem into many small problems: each step only removes a little noise.

The training-data problem is also solved by reversing the direction. We do not naturally have denoising paths, but we can start from real images and add noise ourselves. That gives us a path from image to noise. We then train the model to walk this path backward, so generation becomes the process of going from noise back to image.

The whole idea can be summarized in three sentences:

  1. During training, we add noise to real images step by step until they become close to Gaussian noise.
  2. The model learns to predict which parts of the current image are noise at different noise levels.
  3. During generation, we start from Gaussian noise and repeatedly call the model to remove noise little by little.

That is the most important idea behind diffusion models: they do not draw an image all at once. They gradually recover structure from noise. They turn the difficult problem of “generating an image from nothing” into the more learnable problem of “performing many small denoising steps.”

What’s Next (TODOs)

  • Add visual illustrations for the forward and reverse diffusion processes.
  • Connect these concepts to real-world models like Stable Diffusion and Midjourney.
  • Dive into the mathematical details (loss function, reparameterization trick) in a future post.

References