Text Diffusion

A Visual Explanation of Text Diffusion

How language models can draft many words at once, then refine them together.

Why text diffusion?

An alternative to next-token generation

Large language models are now everywhere—from chatbots to coding tools—but almost all of them write in the same way: one word at a time, committing each word before moving on to the next. Recent projects such as Google DeepMind’s Gemini Diffusion (opens in a new tab) and the open-weight DiffusionGemma (opens in a new tab) explore a different possibility: generating a whole block of text as a rough draft, then repeatedly revising it until it becomes coherent. Researchers care because this approach could produce text faster on suitable hardware, reconsider uncertain words, and support more flexible forms of writing and editing. This article shows how that works—and why it is not yet an obvious replacement for today’s language models.

LLMs are typically autoregressive, commonly understood as next-token predictors. This means that the model generates one token at a time, conditioning each prediction on all previously generated text. In contrast, text diffusion models follow a fundamentally different generation process. Rather than constructing a sequence strictly from left to right, they begin with a highly uncertain or corrupted representation of the entire sequence, for example, random noise (Diffusion-LM (opens in a new tab)) or masked tokens (D3PM (opens in a new tab)), and progressively refine it through a series of denoising steps. At each step, the model can simultaneously reconsider many or all token positions, gradually transforming an initially vague representation into a coherent sequence of text. In this sense, autoregressive models generate text by extending a sequence, whereas text diffusion models generate text by iteratively refining an entire sequence.

One at a time, or many at once

Both sides arrive at the same final output. Autoregressive decoding extends it one position at a time; diffusion repeatedly revises many positions together.

The autoregressive canvas reveals positions sequentially because each new position depends on the preceding output. The diffusion canvas sketches every position at once and then revises many positions together on each pass, until both canvases show the same result.

Animation illustrates dependency structure, not measured generation speed.

Both models are aiming at the same sentence: “A red balloon floated across the clear blue sky.” Neither has written anything yet.

The autoregressive model commits one word at a time, left to right: first “A,” then “red,” then “balloon.”

Every committed word becomes context for the next one. Nothing already written is reconsidered.

The text diffusion model starts elsewhere. One pass proposes a word for all nine positions at once — a rough draft of the whole sentence.

Later passes revise several positions together, reading the rest of the draft as context: “kite” becomes “balloon,” “drifted” becomes “floated.”

A few more revisions and both reach the same sentence — one by extending it word by word, the other by refining all of it at once.

One word at a timeAutoregressive

0 of 9 words committed

Revise a rough draftText diffusion

Draft · 0 of 9 words resolved

Now try it yourself!

Select any word to examine it, or step through either process yourself

One word at a timeAutoregressive

9 of 9 words committed

Revise a rough draftText diffusion

Pass 4 · 9 of 9 words resolved

Why it matters

Faster responses
Updating many positions during one pass may use parallel hardware more effectively and reduce latency for an individual request.
Revisable drafts
A word can be reconsidered after the rest of the passage becomes clearer. This enables better high-level, overall structural controllability, such as control over syntactic structure, sentence organization, and global coherence. Because text diffusion models refine many token positions jointly, an early provisional choice does not have to be treated as final. More broadly, the model can revise local wording in response to emerging global structure, allowing constraints that span multiple parts of a sentence or passage to be enforced more naturally than in strictly left-to-right generation. This makes diffusion particularly appealing for generation tasks that require coordinated global planning, iterative refinement, or explicit control over the structure of the final text.
Flexible context
An active position can use information on both sides, supporting patterns such as filling between fixed text.
This also enables more flexible use of bidirectional context. An active token position can use information from both its left and right sides, rather than relying only on previously generated text. This naturally supports tasks such as filling in missing text between two fixed passages, revising a phrase while preserving its surrounding context, or satisfying constraints specified at multiple positions in a sequence. More generally, diffusion models can treat partially specified text as a global constraint and refine the unresolved regions around it.

Diffusion is promising, but not automatically faster. Multiple refinement passes still require computation, and autoregressive models retain major advantages in streaming, caching, batching, and deployment maturity. Actual speed depends on the model, sampler, hardware, output length, and workload.

Different generation patterns, different constraints.

Neither approach is universally better. The better choice depends on how text should be produced, revised, and delivered.

CharacteristicAutoregressiveDiffusion
Generation orderStrictly left to rightIterative within an active block
RevisionCommitted history stays fixedTentative positions can be reopened
Active contextTokens to the leftBoth sides within the active canvas
StreamingNaturally emits a growing sequenceCommits completed blocks
Computational patternRepeated single-position extensionRepeated multi-position refinement

Both approaches can produce the same sentence. What changes is the shape of the process.

See how the two processes work

Part 1

The shape of the process

What is actually different about writing a whole passage at once — and what has to be true of the network for that to be possible at all.

  1. 1One word at a time
  2. 2The whole canvas
  3. 3Both sides visible
Autoregressive

One word leads to the next.

Autoregressive model factorizes a sequence of L tokens into a series of conditional predictions, where each token is predicted based on all preceding tokens.

Sequence0 / 9
Diffusion

The whole line stays in play.

In diffusion models, at each timestep t, one forward pass predicts a distribution for every position at once. Positions that clear the acceptance threshold commit; the rest are re-predicted on the next pass, now with more of the sentence settled around them.

Canvas · pass 1 of 50 committed · 0 open
What happened on this pass

The canvas starts as nine masked positions. Nothing has been proposed yet, and no position has any confidence.

Select any position above to see what caused it.

Pass 1 of 5. The canvas starts as nine masked positions. Nothing has been proposed yet, and no position has any confidence. The canvas reads: [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK]

Bidirectional context

The blank sees in both directions.

Because the whole canvas is present at once, a position can be predicted from the words after it as well as the words before it.

A different sentence this time, because the balloon never had an ambiguous word. Read only left to right and “suitcase” is the obvious guess—it is the nearest noun. The words that settle it are all on the right: something that did not fit and was too large is the object, not the container. Bidirectional context is done trough controlling model's attention mask between tokens:

One masked position, both sides visiblemodel’s pick
Which word fits, and why

The model puts 78% on “trophy” and 14% on “suitcase”. Pick one to see the words that decide it.

Part 2During training

How it learns

Diffusion model blurs part of a real sentence and score the network ability to put it back. Repeated over millions of sentences.

  1. 1Hide some words
  2. 2Predict every position
  3. 3Score against the truth
Training on unblurring noise

Hide part of a real sentence.

Training starts with a real sentence, samples a diffusion timestep, and obscures a fraction of its tokens using a reserved placeholder, [MASK]. At timestep t, whether a token remains in its original clean state x_0 or is replaced by [MASK] is modeled as a discrete probability distribution over these two possible states:

No model inference is involved at this stage. The corruption step is computationally cheap and directly defines the training targets: the tokens that were masked are exactly the ground-truth tokens the model is trained to recover.

q(xₜ | x₀)t · 0% hidden
The clean sentence, x₀

Before any noise is applied, the canvas is just a sentence from the training data.

Learning to denoise

One guess, scored at every position.

The network sees the hidden canvas once and guesses every position at the same time. Each guess is scored against the word that was actually there. This is done by maximizing log likelihood of recovering each original token x^i_0 given partially corruped / masked version:

This is a single guess, not an iterative denoising process. The repeated refinement typically associated with diffusion occurs only during generation. During training, the model repeats this one-step prediction task over millions of examples, each time receiving a corrupted sequence and being scored on how accurately it reconstructs the hidden tokens.

pθ(x₀ | xₜ, t)input canvas
The input

Eight of nine positions are hidden. Only “red” survived, so that is the model’s single piece of evidence about this sentence.

Select any position to see how it was scored.

Part 3During generation

How it writes

That same trained prediction, run over and over on a canvas that starts empty.

  1. 1Predict every position
  2. 2Accept the confident ones
  3. 3Remask the rest, repeat
Iterative denoising

Predict, commit, repeat.

Generation repeatedly applies this trained prediction process. At each pass, the model commits to the tokens it is most confident about while leaving the remaining positions unresolved. As a result, each successive pass has more settled context to condition on than the previous one.

This is why most revisions tend to occur in the middle passes. Early in generation, too little context has been resolved for meaningful corrections; near the end, most positions have already been committed, leaving little left to revise.

Pass 1 of 5no revisions
What happened on this pass

The canvas starts as nine masked positions. Nothing has been proposed yet, and no position has any confidence.

Select any position above to see what caused it.

Reconsideration

Uncertain words can be reopened.

A position that never cleared the threshold can be hidden again and re-predicted. Nothing in an autoregressive decoder can do this.

Every round of reconsideration requires another full pass over the sequence. What this buys is flexibility: a provisional choice can be wrong early without remaining wrong forever.

Step 1 of 4revisable
Position 8 · green

Below the threshold, so the sampler leaves it revisable instead of committing it.

“green” is the current proposal at position 8, but at 0.52 it never cleared the threshold.

Playground

Open the loop and inspect one sampling run.

Sampling settingsNoise 70% · τ 0.8 · threshold 0.72
xₜ → pθ(x₀ | xₜ,t)7 hidden · 4 accepted
Reverse step 2 / 5

Reverse step 2 of 5. 4 of 9 positions accepted.

Explore classifier guidance

Block diffusion · classifier guidance

Steer the active block while it denoises.

Before a diffusion block is decoded into words, an external classifier can evaluate its intermediate vectors. The classifier’s gradient makes a small adjustment toward the requested property, then denoising continues from that adjusted state.

  1. Base denoising step

  6. words
Without guidance

The model predicts a slightly cleaner version of the current text representation.

1 of 5

Continue to DiffusionGemma

Chapter 02 · In the wild

From text diffusion to DiffusionGemma

So far we described masked discrete diffusion: hide tokens behind a placeholder, predict every position at once, commit the confident ones, re-predict the rest. (A uniform-state variant replaces tokens with random words instead of a mask; the recent open models use masking.) But everything happened to one nine-word sentence, in a simulation built to be watched—nothing about a toy proves the loop survives real vocabulary, real document lengths, real tasks.

DiffusionGemma is that loop in the wild: 256-token canvases driven by a 26B-parameter mixture-of-experts backbone that activates roughly 4B parameters per step, pointed at the work where revision earns its cost—code, structured edits, long drafts. Scale changes two things, and they are this chapter: the canvas now answers to a fixed prompt, and finished blocks commit into cached context so the next canvas can begin.

Prompt and context

Fixed context guides the active block.

The prompt and completed text stay fixed while the next canvas is repeatedly refined.

Committed context

Continue the scene: A red balloon floated across the clear blue sky.

Active canvasDenoising pass 1
Block generation

Commit a block, start the next.

DiffusionGemma denoises a 256-token canvas, commits it, then adds that block to the cached context for the next canvas.

Token blocksBlock 1 of 3
Parallel work

A pass can update one position or many.

Autoregressive decoding repeatedly loads weights for a small one-token workload. A full canvas gives the accelerator parallel work per pass.

Same active passagePass 1
What this enables

The canvas supports more than one writing pattern.

Generate

Ordinary text generation

Aredballoonfloatedacross
Fill between

Fixed context on both sides

A balloonclimbedthe sky.
Revisit

Revise uncertain positions

thegreensky.

Conclusion

A revisable canvas makes parallel refinement, filling between fixed context, uncertain-token revision, and blockwise streaming possible—while trading simple left-to-right streaming for repeated work across active positions.

References

  1. [1]Google AI for Developers. “DiffusionGemma model overview.” 2026.
  2. [2]Google AI for Developers. “Diffusion in Text Generation Explained.” 2026.
  3. [3]Google Developers Blog. “DiffusionGemma: The Developer Guide.” 2026.
  4. [4]Arriola et al. “(BD3-LM) Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models.” ICLR 2025.
  5. [5]Austin et al. “(D3PM) Structured Denoising Diffusion Models in Discrete State-Spaces.” NeurIPS 2021.
  6. [6]Li et al. “Diffusion-LM Improves Controllable Text Generation.” NeurIPS 2022.
  7. [7]Google DeepMind. “Gemini Diffusion.” 2025.
  8. [8]Nie et al. “Large Language Diffusion Models (LLaDA).” ICML 2025.
  9. [9]Zhang et al. “PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model.” NeurIPS 2023.
  10. [10]Apple Machine Learning Research. “Enhancing Paragraph Generation with a Latent Language Diffusion Model.” 2024.

How to cite

@article{li2026textdiffusion,
  title = {A Visual Explanation of Text Diffusion},
  author = {Li, Susie S.Y. and Li, Mingwei S.G. and Chang, Remco},
  year = {2026},
  month = {June},
  url = {https://susiesyli.com/text-diffusion-explainer/}
}