How Transformers Learn Where Words Are: A Visual Guide to Positional Encodings

October 13, 2026  ·  transformers positional-encoding LLMs tutorial

Imagine handing someone three cards: dog, bites, and man.

You ask what happened. They cannot tell you. Did the dog bite the man, or did the man bite the dog? The words are all there. What is missing is their order.

dog bites man An unfortunate afternoon.
man bites dog A rather different story.
Same words. Different order. Different meaning.

We see that order immediately because we read from left to right. A Transformer needs a way to use it too. Its attention mechanism is excellent at comparing words and gathering relevant information. But the basic, unrestricted attention calculation does not come with a built-in idea of “first,” “before,” or “three places back.”

That is the problem positional encodings solve. They give the model information about where things are.

The interesting part is how that idea developed. The first approach sounds obvious: number the words. Then comes a better question: do we need the number, or do we need the distance between two words? Then comes a surprisingly elegant answer involving rotating arrows.

Let’s build those ideas one at a time. Each method now has a worked walkthrough: the same sentence, the same small embeddings, and the point where position changes the calculation. RoPE gets a longer two-clock example. You can follow the numbers without running code. Read the intuition first; the numbered steps make it concrete.

Start with attention: what should this word look at?

Consider this sentence:

The animal didn’t cross the street because it was tired.

To make sense of it, we look back at animal. The tired thing is probably the animal, not the street. Attention gives the model a way to make connections like this. Each word compares itself with other words, then gathers more information from the ones that seem useful.

Here is a picture of what we would like it to learn. The connections are illustrative, not measurements from a model.

it Who are we talking about?
animal A promising connection
street Less useful here
Attention helps a word gather context. Different attention heads can learn different kinds of connections.

But relevance is not just about meaning. In the red ball, “red” describes the word next to it. In a long paragraph, a nearby noun might be a better candidate than another noun several sentences away. Sometimes we need to look far back. Sometimes the word immediately before us matters most.

So the model needs both: what a word means and where it sits relative to other words.

A small vocabulary note: models actually work with tokens, which can be words, pieces of words, or punctuation. We’ll use whole words in the pictures so we can concentrate on the idea.

Our shared example: seven words, four numbers per word

We will use the same sentence for every walkthrough. The numbers are deliberately small, invented teaching values—not a trained model’s embeddings. Each word is one token in this example, and positions start at 0.

Position 0 1 2 3 4 5 6
Word the cat sat on the soft mat

Notice the two copies of the. At the initial lookup, the same token gets the same embedding. Let’s see exactly where each design makes their journeys different.

Position Word Embedding x · four numbers
0 the (1.000, 0.000, 1.000, 0.000)
1 cat (0.000, 1.000, 0.000, 1.000)
2 sat (1.000, 1.000, 0.000, 0.000)
3 on (0.000, 0.000, 1.000, 1.000)
4 the (1.000, 0.000, 1.000, 0.000)
5 soft (1.000, 0.000, 0.000, 1.000)
6 mat (0.000, 1.000, 1.000, 0.000)

Seven tokens give us a 7 × 4 embedding matrix. More generally, n tokens with d numbers each give n × d. The columns are learned coordinates, not named properties such as “animal” or “action.”

A small head we can follow without hiding the numbers

Real models learn weight matrices that make queries, keys, and values from each row. For this toy head we choose especially simple matrices: queries and keys copy the input row; values reverse its four coordinates. This is a legitimate choice of weights, but a deliberately simple one—not how trained weights usually look. We write row vectors, so the convention is q = x W_Q, k = x W_K, v = x W_V.

Our chosen weight transformations · the same in every main walkthrough
W_Q and W_K · copy the row
1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1
W_V · reverse the row
0 0 0 1 0 0 1 0 0 1 0 0 1 0 0 0
For “the”: x = (1, 0, 1, 0), q = k = (1, 0, 1, 0), and v = (0, 1, 0, 1). Other layers and heads would normally use different learned weights.
Word q = k before position v before position
the (1.000, 0.000, 1.000, 0.000) (0.000, 1.000, 0.000, 1.000)
cat (0.000, 1.000, 0.000, 1.000) (1.000, 0.000, 1.000, 0.000)
sat (1.000, 1.000, 0.000, 0.000) (0.000, 0.000, 1.000, 1.000)
on (0.000, 0.000, 1.000, 1.000) (1.000, 1.000, 0.000, 0.000)
soft (1.000, 0.000, 0.000, 1.000) (1.000, 0.000, 0.000, 1.000)
mat (0.000, 1.000, 1.000, 0.000) (0.000, 1.000, 1.000, 0.000)

We will follow the second “the,” at position 4, when it looks backward. It may use positions 0 through 4; soft and mat are hidden because they are in the future. We apply this same causal mask to compare the designs, even though some of the original models also use attention that looks both ways.

To compare a query with a key, multiply corresponding coordinates and add. For example, “the” looking at “sat” gives 1×1 + 0×1 + 1×0 + 0×0 = 1. With four query/key coordinates, ordinary scaled attention divides this by sqrt(4) = 2, giving a score of 0.5. DeBERTa’s three-term example below uses a different scale.

For n tokens, one head builds n query rows, n key rows, and n value rows. Comparing all queries with all keys makes an n × n score grid. We will display one row of that grid so we can see the arithmetic clearly.

The scores become attention weights through softmax: exponentiate the allowed scores, then divide each by their sum. Those weights mix the value rows, giving one updated row for each input token.

7 embedding rows
7 q / k / v rows
7 × 7 scores + mask
weights mix values
7 updated rows

For reference, without any explicit position signal, the second “the” has the score row (1.000, 0.000, 0.500, 0.500, 1.000) for keys at positions 0–4. Both “the” keys tie. The causal mask itself provides some order structure, but there is no explicit distance signal in these matches.

What the second “the” gathers
the position 027.9%
cat position 110.3%
sat position 216.9%
on position 316.9%
the position 427.9%
These five weights add to 100% before rounding. “Soft” and “mat” are hidden by the causal mask.

The head output for this token is the weighted mixture of the five value rows: (0.272, 0.728, 0.272, 0.728).

This is one head’s output before an output projection, residual addition, normalization, or feed-forward block. Repeating the calculation for every query gives seven output rows, each with four numbers.

Now we can change one ingredient at a time. In the panels below, blue changes the input, amber changes relationships or scores, and purple changes query/key rotations. All calculations use full precision and are displayed rounded to three decimal places.

1. Give each word a seat number

Suppose you wanted to restore the order of our three cards. The simplest thing you could do is write a number on each one.

dog seat 1 bites seat 2 man seat 3
man seat 1 bites seat 2 dog seat 3
“Dog in seat 1” and “dog in seat 3” now carry different information.

A Transformer does something similar, with one difference. It represents a word as a list of numbers, called an embedding. Instead of attaching a printed seat number, it adds another list of numbers describing that position. The model receives a mixture of the word information and the position information.

Learned absolute positions let training decide what each position’s list should be. Picture a drawer of seat labels: a label for the first token, another for the second, and so on. Models such as BERT and GPT-2 used this kind of position table. “Absolute” just means that a label describes a particular place in the sequence. The BERT paper describes its input embeddings, including position embeddings.

Worked example: learned positions

Look up x
Look up position row p
Add: h = x + p
Make q, k, v from h
Compare and gather

Step 0: Keep the sentence and choose a position table

Here is one possible toy table. We chose a tidy pattern so you can check the sums; a trained table need not follow any such pattern.

Position Word Position row p Input after adding: h = x + p
0 the (0.000, 0.000, 0.000, 0.000) (1.000, 0.000, 1.000, 0.000)
1 cat (0.100, 0.000, 0.000, 0.050) (0.100, 1.000, 0.000, 1.050)
2 sat (0.200, 0.000, 0.000, 0.100) (1.200, 1.000, 0.000, 0.100)
3 on (0.300, 0.000, 0.000, 0.150) (0.300, 0.000, 1.000, 1.150)
4 the (0.400, 0.000, 0.000, 0.200) (1.400, 0.000, 1.000, 0.200)
5 soft (0.500, 0.000, 0.000, 0.250) (1.500, 0.000, 0.000, 1.250)
6 mat (0.600, 0.000, 0.000, 0.300) (0.600, 1.000, 1.000, 0.300)

Step 1: Compare the two copies of “the”

At position 0, (1.000, 0.000, 1.000, 0.000) + (0.000, 0.000, 0.000, 0.000) = (1.000, 0.000, 1.000, 0.000).

At position 4, (1.000, 0.000, 1.000, 0.000) + (0.400, 0.000, 0.000, 0.200) = (1.400, 0.000, 1.000, 0.200).

The lookup embeddings are identical. The rows entering attention are now different.

Step 2: Make queries, keys, and values from the combined rows

Our weights give q₄ = k₄ = (1.400, 0.000, 1.000, 0.200) and v₄ = (0.200, 1.000, 0.000, 1.400). Notice that values change too: they are made from the word-plus-position input, not the original word row.

Step 3: Follow one comparison

Against the first “the”: 1.4×1 + 0×0 + 1×1 + 0.2×0 = 2.4. Divide by 2 → 1.200. Against itself the dot is 1.4² + 1² + 0.2² = 3.0 → 1.500.

Key position Word Query dot key Scaled score
0 the 2.400 1.200
1 cat 0.350 0.175
2 sat 1.700 0.850
3 on 1.650 0.825
4 the 3.000 1.500

Step 4: Convert scores to weights, then gather values

The two “the” connections no longer tie. This follows from the added position rows, not from a changed word identity.

What the second “the” gathers
the position 024.4%
cat position 18.7%
sat position 217.2%
on position 316.8%
the position 432.9%
These five weights add to 100% before rounding. “Soft” and “mat” are hidden by the causal mask.

The head output for this token is the weighted mixture of the five value rows: (0.368, 0.741, 0.259, 0.970).

This is one head’s output before an output projection, residual addition, normalization, or feed-forward block. Repeating the calculation for every query gives seven output rows, each with four numbers.

This gives the model a useful starting point. The word has an identity and a place.

But imagine that our drawer contains 512 labels. What happens when a document needs a 513th? There is no label waiting for it. Extending the table is possible, but the model has not automatically learned how to use those extra positions.

There is another question hiding here. “Red” can describe “ball” near the start of a sentence or near the end. Their seat numbers change, but being next to each other stays the same. Could our position system make that relationship easier to recognize?

Before we get there, let’s look at a clever alternative to a drawer of learned labels.

2. Use several clocks to describe a position

Think about the hands on a clock. The second hand moves quickly. The minute hand moves more slowly. The hour hand moves more slowly still. Together, they describe the time more fully than any one hand could.

Now imagine a collection of clock hands that advances every time we move one token along a sentence. One hand takes a large step. Another takes a small step. Another barely moves. A position is described by the collection of angles.

Moving from the first position to a later position turns a fast clock hand substantially, while a slow clock hand moves only a little.
The fast hand helps distinguish nearby positions. The slow hand changes more gradually over a longer stretch. Neither hand has to do the whole job alone.

Why more than one clock? A single hand eventually comes around again. Looking at the minute hand alone, you cannot tell which hour it is. Having hands that turn at different speeds gives the model a richer pattern for each position.

The original 2017 Transformer used sine and cosine functions to make these patterns. That is where the name sinusoidal positional encoding comes from. You can think of those functions as a way to write down a clock hand’s position using two numbers. The resulting numbers are added to the word embedding, just like the seat labels. Attention Is All You Need introduced this design alongside a learned-position alternative.

Worked example: sinusoidal positions

Look up x
Generate clock row PE(m)
Add: h = x + PE(m)
Make q, k, v from h
Compare and gather

Step 0: Make two clock pairs

For four dimensions and base 10,000, the row is (sin m, cos m, sin(0.01m), cos(0.01m)). Angles here are in radians. The first pair changes quickly; the second changes slowly.

Position Word Clock row Input after adding
0 the (0.000, 1.000, 0.000, 1.000) (1.000, 1.000, 1.000, 1.000)
1 cat (0.841, 0.540, 0.010, 1.000) (0.841, 1.540, 0.010, 2.000)
2 sat (0.909, -0.416, 0.020, 1.000) (1.909, 0.584, 0.020, 1.000)
3 on (0.141, -0.990, 0.030, 1.000) (0.141, -0.990, 1.030, 2.000)
4 the (-0.757, -0.654, 0.040, 0.999) (0.243, -0.654, 1.040, 0.999)
5 soft (-0.959, 0.284, 0.050, 0.999) (0.041, 0.284, 0.050, 1.999)
6 mat (-0.279, 0.960, 0.060, 0.998) (-0.279, 1.960, 1.060, 0.998)

Step 1: Follow the second “the”

At position 4, its original row (1.000, 0.000, 1.000, 0.000) receives clock row (-0.757, -0.654, 0.040, 0.999), producing (0.243, -0.654, 1.040, 0.999). The first “the” instead receives (0.000, 1.000, 0.000, 1.000). Same word, different clock reading.

Step 2: Make q, k, and v after the addition

q₄ = k₄ = (0.243, -0.654, 1.040, 0.999); v₄ = (0.999, 1.040, -0.654, 0.243). This is the same place in the pipeline as learned absolute positions. Only the source of the added position row changed.

Step 3: Multiply corresponding coordinates and add

For the first “the,” k₀ = (1.000, 1.000, 1.000, 1.000). The raw match is (0.243×1) + (-0.654×1) + (1.040×1) + (0.999×1) ≈ 1.629. Divide by 2 → 0.814.

Key position Word Raw match Scaled score
0 the 1.629 0.814
1 cat 1.207 0.603
2 sat 1.103 0.551
3 on 3.751 1.875
4 the 2.566 1.283

Step 4: Gather the changed value rows

These clock patterns influence both the matches and the values, because both were made after the input addition.

What the second “the” gathers
the position 014.2%
cat position 111.5%
sat position 210.9%
on position 340.9%
the position 422.6%
These five weights add to 100% before rounding. “Soft” and “mat” are hidden by the causal mask.

The head output for this token is the weighted mixture of the five value rows: (1.523, 0.801, -0.171, 0.558).

This is one head’s output before an output projection, residual addition, normalization, or feed-forward block. Repeating the calculation for every query gives seven output rows, each with four numbers.

The pleasant thing is that a formula can keep producing patterns beyond the positions in a fixed table. We do not run out of labels in the same way.

Still, being able to produce a label is different from the model knowing what to do with it. A model trained on short passages can struggle with the patterns and relationships in much longer ones. And we are still handing it a description of each individual position, then asking it to work out the relationship between two positions.

Let’s make that relationship explicit.

3. Describe how far apart the words are

Read these two phrases:

  • the red ball
  • after a surprisingly long afternoon, the red ball

“Red” moved. “Ball” moved. But “red” is still immediately before “ball.” That feels like the useful fact, doesn’t it?

the red position 2 1 step → ball position 3
… the red position 9 1 step → ball position 10
Absolute positions change. The relationship between the words stays the same. Position numbers here are illustrative.

This is the idea behind relative positions. Instead of telling each word only “you are in seat 9,” we give its interactions information such as “this word is one step ahead” or “that word is three steps behind.”

Notice where the information is useful. When “red” considers “ball,” attention is already asking whether they belong together. We can bring their relative position into that very comparison.

Shaw: learn what “one step away” means

In 2018, Shaw and colleagues introduced learned representations for relative offsets. Think of little relationship labels: one before, two before, one after. The same label can be used anywhere in a sentence. The model can learn what to do with it rather than learning separate versions of the relationship at every pair of seat numbers.

To keep the collection manageable, distant offsets share labels beyond a chosen cutoff. That saves space, but it also means some far-apart relationships become indistinguishable to the position system. Their method brings relative information into both the matching and the information gathered. Self-Attention with Relative Position Representations.

Worked example: Shaw relative representations

Word rows unchanged
Make q, k, v
Key + relative-key row
Match → weights
Value + relative-value row

Step 0: Use relationship labels instead of seat labels

Our gap is query position minus key position: a positive gap means the key is earlier. This reverses the original paper’s indexing convention, so we relabel the learned table consistently. Clip gaps to −2…2.

Gap label Relative-key row Relative-value row
-2 (-0.200, 0.000, 0.000, 0.000) (0.000, -0.100, 0.000, 0.000)
-1 (-0.100, 0.000, 0.000, 0.000) (0.000, -0.050, 0.000, 0.000)
0 (0.000, 0.000, 0.000, 0.000) (0.000, 0.000, 0.000, 0.000)
1 (0.100, 0.000, 0.000, 0.000) (0.000, 0.050, 0.000, 0.000)
2 (0.200, 0.000, 0.000, 0.000) (0.000, 0.100, 0.000, 0.000)

Step 1: Follow “the” looking at “cat”

The query is at 4 and the key is at 1. Gap 3 is clipped to 2. For this pair, add (0.2, 0, 0, 0) to cat’s key (0, 1, 0, 1) → (0.2, 1, 0, 1). The stored cat key has not permanently changed; the extra row belongs to this particular relationship.

Step 2: Score the relationship

The query (1, 0, 1, 0) dotted with (0.2, 1, 0, 1) gives 0.2. Divide by 2 → 0.100, rather than the content-only score 0.

Key Gap → clipped gap Content-only score Score with relative key
0: the 4 → 2 1.000 1.100
1: cat 3 → 2 0.000 0.100
2: sat 2 → 2 0.500 0.600
3: on 1 → 1 0.500 0.550
4: the 0 → 0 1.000 1.000

Step 3: Change what is gathered as well

Cat’s value is (1, 0, 1, 0). For this gap we add (0, 0.1, 0, 0) → (1, 0.1, 1, 0). Thus the weighted mixture carries relationship information as well as content.

Step 4: Mix the pair-specific values

Offsets beyond the clipping limit share a label. This head can distinguish “nearby” and “farther away,” but its position table treats gaps 2, 3, and 4 alike.

What the second “the” gathers
the position 028.9%
cat position 110.6%
sat position 217.5%
on position 316.7%
the position 426.2%
These five weights add to 100% before rounding. “Soft” and “mat” are hidden by the causal mask.

The head output for this token is the weighted mixture of the five value rows: (0.273, 0.784, 0.282, 0.727).

This is one head’s output before an output projection, residual addition, normalization, or feed-forward block. Repeating the calculation for every query gives seven output rows, each with four numbers.

Transformer-XL: keep the relationship when you turn the page

Now imagine reading a book one page at a time. You want the next page to use what you learned on the previous one. But if both pages start their numbering at 1, the old seat numbers become confusing.

Transformer-XL reused representations from earlier text segments. Relative positions helped it describe how far those remembered tokens were from the current tokens, across segment boundaries. The important development was that position had to work with a continuing memory, not just within one isolated passage. Transformer-XL, 2019.

Worked example: Transformer-XL across a segment boundary

Memory: the · cat
Current: sat · on · the · soft · mat
Current queries + all available keys
Relative-distance matches
5 current output rows

Step 0: Split our sentence into memory and current text

Remember positions 0–1 (“the cat”) from the previous segment. Process positions 2–6 (“sat on the soft mat”) now. For this first-layer toy, pretend the stored rows equal the listed token rows; real memory contains earlier hidden representations.

Step 1: Count the rows

The current segment has 5 query rows. Together, memory and current text supply 7 key and value rows, making a 5 × 7 score grid before masking. Following global position 4 gives access to keys 0–4, including the remembered “the cat.”

Step 2: Describe distance across the boundary

For the current “the” at global 4 and remembered “cat” at global 1, gap = 3. Use relative clock row (0.141, -0.990, 0.030, 1.000). Their segment-local numbers need not be compared as if they were positions in one isolated segment.

Step 3: Combine four kinds of match

In this simplified attention calculation, the content projections and relative-position projection are identities. Use learned-bias examples u = (0.1, 0, 0, 0) and b = (0, 0.1, 0, 0). The raw score is q·k + q·R(gap) + u·k + b·R(gap). These are a content match, a content–distance match, a content preference, and a distance preference. Divide their sum by 2.

Key q·k q·R u·k b·R Scaled total
0: the 2.000 -0.717 0.100 -0.065 0.659
1: cat 0.000 0.171 0.000 -0.099 0.036
2: sat 1.000 0.929 0.100 -0.042 0.994
3: on 1.000 0.851 0.000 0.054 0.953
4: the 2.000 0.000 0.100 0.100 1.100

Step 4: Gather from memory and current values

Memory extends the set of keys and values; it does not create new output rows for the old segment.

What the second “the” gathers
the position 017.2%
cat position 19.2%
sat position 224.0%
on position 323.0%
the position 426.7%
These five weights add to 100% before rounding. “Soft” and “mat” are hidden by the causal mask.

The head output for this token is the weighted mixture of the five value rows: (0.322, 0.668, 0.332, 0.678).

This is one head’s output before an output projection, residual addition, normalization, or feed-forward block. Repeating this for the five current queries gives five current output rows, each with four numbers. The two memory rows are consulted, not emitted again.

T5: we may not need a separate label for every distance

When giving directions, we often say “next door,” “a few houses away,” or “across town.” We are precise nearby and less precise far away.

T5 takes a similar approach. Nearby distances get fine distinctions. Larger distances are grouped into wider buckets. For each bucket, an attention head learns a number that nudges a connection’s score up or down.

T5 · a rough map of distance buckets
1 its own bucket
2 its own bucket
3–4 a shared bucket
5–8 a wider bucket
9–16 wider still
Illustrative ranges, not T5's exact bucket boundaries. The principle is fine detail nearby, broader groups farther away.

Different heads can learn different distance preferences: one might often look nearby, while another pays more attention farther back. These are preferences, not rules about which words must connect. T5, first released in 2019.

Worked example: T5 distance buckets

Word rows unchanged
Make q, k, v
Content scores
Add bucket biases
Weights mix original values

Step 0: Choose a small learned bias table

This is an invented table for one head, not T5’s real bucket boundaries or trained values. Direction is “backward” because our comparison uses causal attention.

Distance bucket Learned score adjustment
self 0.000
one back 0.300
two back 0.100
3–4 back -0.200
5+ back -0.400

Step 1: Keep q, k, and v as they were

For the second “the,” q₄ = (1, 0, 1, 0). Its content-only score row is (1, 0, 0.5, 0.5, 1). We have not added a position vector to any word row.

Step 2: Look up one bias for each pair

Looking from position 4 to “on” at 3 gives gap 1. Content score 0.5, bucket bias +0.3, new score 0.8. Looking back to the first “the” gives gap 4: 1.0 − 0.2 = 0.8.

Key Gap Bucket Content score Add bias New score
0: the 4 3–4 back 1.000 -0.200 0.800
1: cat 3 3–4 back 0.000 -0.200 -0.200
2: sat 2 two back 0.500 0.100 0.600
3: on 1 one back 0.500 0.300 0.800
4: the 0 self 1.000 0.000 1.000

Step 3: Gather values with the new weights

The score row changed. The query, key, and value rows did not receive any direct positional transformation in this calculation.

What the second “the” gathers
the position 022.7%
cat position 18.3%
sat position 218.6%
on position 322.7%
the position 427.7%
These five weights add to 100% before rounding. “Soft” and “mat” are hidden by the causal mask.

The head output for this token is the weighted mixture of the five value rows: (0.310, 0.731, 0.269, 0.690).

This is one head’s output before an output projection, residual addition, normalization, or feed-forward block. Repeating the calculation for every query gives seven output rows, each with four numbers.

DeBERTa: keep meaning and position separate while comparing

Earlier, we mixed the seat label into the word embedding. DeBERTa asks whether we can keep the two descriptions separate for longer.

Imagine two cards for each token: one describes its content; the other helps describe relative position. Attention can then consider content-to-content matches and how content relates to position, instead of starting from one blended description. That separation is the intuition behind disentangled attention. DeBERTa, 2020.

Worked example: DeBERTa’s separate comparisons

Content rows unchanged
Content q / k + position q / k
Three separate matches
Add and scale scores
Weights gather content values

Step 0: Give content and position their own representations

For the toy, content projections and the position-key projection are identities. The position-query projection halves the first coordinate and copies the other three. For a signed offset g, the relative-position row is P(g) = (0.1g, 0.2, 0, 0). Actual position rows and projections are learned; this formula just specifies our invented table.

Step 1: Match the content

“The” at 4 looking at “cat” at 1: content query (1, 0, 1, 0) meets content key (0, 1, 0, 1) → content–content match 0.

Step 2: Add two directional position matches

Gap = 3. The content query meets position key P(+3) = (0.3, 0.2, 0, 0) → 0.3. Project P(−3) to position query (−0.15, 0.2, 0, 0) meets cat’s content key → 0.2. Both directions participate, using opposite signed offsets.

Step 3: Combine the three matches

For cat, raw total 0 + 0.3 + 0.2 = 0.5. For this three-term, four-coordinate example, divide by sqrt(3×4) ≈ 3.464 → 0.144. We use the three-term attention scale rather than quietly treating it as a single dot product.

Key Content↔content Content→position Position→content Scaled total
0: the 2.000 0.400 -0.200 0.635
1: cat 0.000 0.300 0.200 0.144
2: sat 1.000 0.200 0.100 0.375
3: on 1.000 0.100 0.000 0.318
4: the 2.000 0.000 0.000 0.577

Step 4: Gather content values

Relative-position representations affected the score through separate matches. They were not blended into each initial token row.

What the second “the” gathers
the position 0 24.7%
cat position 1 15.1%
sat position 2 19.0%
on position 3 18.0%
the position 4 23.3%
These five weights add to 100% before rounding. “Soft” and “mat” are hidden by the causal mask.

The head output for this token is the weighted mixture of the five value rows: (0.330, 0.659, 0.341, 0.670).

This is one head’s output before an output projection, residual addition, normalization, or feed-forward block. Repeating the calculation for every query gives seven output rows, each with four numbers.

These developments are related ideas, not a ladder where every new method makes the previous one obsolete. Now let’s try a distance preference that does not need to be learned at all.

4. Let distance nudge attention: ALiBi

Suppose you are reading a paragraph and encounter “it.” Usually, you start looking for its referent nearby. You can look farther back, but you need a good reason.

ALiBi gives attention that sort of preference. A token farther in the past pays a larger distance penalty when its attention score is calculated. If two candidates have equally good content matches, the closer one wins.

ALiBi · equal content matches
1 step back
A small distance cost
3 steps back
A larger distance cost
5 steps back
A larger cost again
A longer bar means a larger penalty, not a larger attention weight. A strong content match can overcome the penalty.

Different heads pay different rates. Some have a strong preference for nearby tokens; others have a gentler penalty and can look farther back more easily. These rates are fixed rather than learned.

Why is this useful for longer text? “Charge a little more for each extra step” still makes sense at distances we did not see during training. There is no new seat label to invent. ALiBi demonstrated better length extrapolation in its experiments. The tradeoff is that a useful detail buried far back also has to overcome that distance cost. Train Short, Test Long, 2021.

Worked example: ALiBi

Word rows unchanged
Make q, k, v
Content scores
Subtract distance costs
Weights mix original values

Step 0: Choose a rate for this head

Use a fixed 0.25 per token step. This is a teaching rate; real ALiBi assigns a prescribed set of rates across heads.

Step 1: Compute the content matches first

The second “the” has q₄ = (1, 0, 1, 0). Before distance penalties, the score row is (1, 0, 0.5, 0.5, 1). Keys and values remain as listed in the shared setup.

Step 2: Subtract the price of distance

For the first “the,” gap 4 costs 0.25×4 = 1. Its score changes from 1 to 0. For “on,” gap 1 costs 0.25; its score changes from 0.5 to 0.25.

Key Gap Content score Distance penalty New score
0: the 4 1.000 -1.000 0.000
1: cat 3 0.000 -0.750 -0.750
2: sat 2 0.500 -0.500 0.000
3: on 1 0.500 -0.250 0.250
4: the 0 1.000 0.000 1.000

Step 3: Mix values according to the new weights

The identical “the” keys now receive different weights because one is four steps away and the other is at the query’s own position. A distant key could still win if its content match were strong enough.

What the second “the” gathers
the position 015.4%
cat position 17.3%
sat position 215.4%
on position 319.8%
the position 442.0%
These five weights add to 100% before rounding. “Soft” and “mat” are hidden by the causal mask.

The head output for this token is the weighted mixture of the five value rows: (0.271, 0.773, 0.227, 0.729).

This is one head’s output before an output projection, residual addition, normalization, or feed-forward block. Repeating the calculation for every query gives seven output rows, each with four numbers.

So far we have given tokens labels and given connections distance preferences. Next, we change the way the comparison is represented.

5. Turn position into a rotation: RoPE

Hold out two pencils so they point in the same direction. Now turn one a little. They are less aligned. Turn it farther, and their alignment changes again.

Now keep the angle between them fixed and turn both together. They still have the same alignment. You have changed the direction each pencil points, but not their relationship.

That simple observation is the heart of Rotary Position Embedding, or RoPE.

To connect it to attention, we need just two names. A query describes what a token is looking for. A key describes what another token offers for matching. Both are lists of numbers learned by the model. Attention compares them to decide how much information to gather from that token.

Imagine a tiny part of each query and key as an arrow on a flat sheet of paper. RoPE turns those arrows by an amount determined by their token positions before comparing them. The comparison can then use both the original content and the relative turn. RoFormer, 2021.

RoPE with d = 4 and seven words, fully worked

Look up x
Make q, k, v
Rotate q and k only
Compare rotated rows
Weights mix unrotated values

Step 0: Locate the change in the pipeline

The input embedding row is not directly modified by RoPE. Neither is v. The positional turn happens to q and k just before their dot product. Later token representations still change, because the resulting attention weights gather different information.

Step 1: Follow the two copies of “the”

At positions 0 and 4, their lookup rows are both (1, 0, 1, 0). With our chosen first-layer weights, both also have q = k = (1, 0, 1, 0) and v = (0, 1, 0, 1) before rotation. In later layers, their different contexts can already give them different content rows.

Step 2: Split each matching row into two arrows

Cut (1, 0, 1, 0) into pair A = (1, 0) and pair B = (1, 0). Each pair is an arrow in its own flat plane. Four coordinates make two pairs; 128 coordinates make 64 pairs.

One query row: (1, 0, 1, 0)
Pair A · coordinates 0–1
(1, 0) → fast arrow
Pair B · coordinates 2–3
(1, 0) → slow arrow
The arrows live in separate planes. They are not two arrows competing for one direction.

Step 3: Give the arrows different speeds

Use 90° per position for pair A and 10° per position for pair B. These are deliberately simple teaching speeds. Standard d = 4 RoPE with base 10,000 uses 1 radian and 0.01 radian per position instead.

Position Word Pair A angle Pair B angle
0 the 0° 0°
1 cat 90° 10°
2 sat 180° 20°
3 on 270° 30°
4 the 360° (same direction as 0°) 40°
5 soft 450° (same direction as 90°) 50°
6 mat 540° (same direction as 180°) 60°

Step 4: Turn an arrow, then rejoin the pairs

An arrow starting at (1, 0) becomes (cos φ, sin φ) after a turn of φ. First hold the starting content fixed and see what happens if this same “the” row occupies each possible position. This is a position-only demonstration, not a claim that every word has the same vector.

Position Fast pair A Slow pair B Rejoined “the” row
0 (1.000, 0.000) (1.000, 0.000) (1.000, 0.000, 1.000, 0.000)
1 (0.000, 1.000) (0.985, 0.174) (0.000, 1.000, 0.985, 0.174)
2 (-1.000, 0.000) (0.940, 0.342) (-1.000, 0.000, 0.940, 0.342)
3 (0.000, -1.000) (0.866, 0.500) (0.000, -1.000, 0.866, 0.500)
4 (1.000, 0.000) (0.766, 0.643) (1.000, 0.000, 0.766, 0.643)
5 (0.000, 1.000) (0.643, 0.766) (0.000, 1.000, 0.643, 0.766)
6 (-1.000, 0.000) (0.500, 0.866) (-1.000, 0.000, 0.500, 0.866)
The same two starting arrows at seven positions: the fast arrow returns to its starting direction at position four, while the slow arrow has moved forty degrees.
At positions 0 and 4 the fast hand repeats, but the slow hand does not. Together the pairs distinguish these two positions.

The actual “the” rows are now (1, 0, 1, 0) at 0 and (1, 0, 0.766, 0.643) at 4. The first pair alone repeats every four steps in this toy. The second pair separates those positions. Multiple speeds reduce this ambiguity over useful ranges; they do not promise an unlimited, unique distance code.

Step 5: Check equal gaps while keeping content fixed

Temporarily give the compared query and key the same starting vector (1, 0, 1, 0). We are isolating position, so these are not the actual “on/cat” and “mat/the” content pairs from our sentence.

Case 1: query at 3, key at 1, gap = 2.

q = (0.000, -1.000, 0.866, 0.500), k = (0.000, 1.000, 0.985, 0.174).

Fast-pair contribution -1.000 + slow-pair contribution 0.940 = raw dot product -0.060.

Case 2: query at 6, key at 4, gap = 2.

q = (-1.000, 0.000, 0.500, 0.866), k = (1.000, 0.000, 0.766, 0.643).

Fast-pair contribution -1.000 + slow-pair contribution 0.940 = raw dot product -0.060.

Both raw dots are −0.060 to three decimals. Scaled attention divides by 2, giving about −0.030, before masking and softmax. The raw dot and the scaled score are different stages of the pipeline.

Step 6: See the shortcut for identical starting arrows

Each pair contributes cos(speed × gap), so the raw dot is cos(90°×gap) + cos(10°×gap). This shortcut applies to our chosen identical unit arrows; general queries and keys also bring their starting content into the match.

Gap Fast pair Slow pair Raw dot
0 1.000 1.000 2.000
1 0.000 0.985 0.985
2 -1.000 0.940 -0.060
3 0.000 0.866 0.866
4 1.000 0.766 1.766
5 0.000 0.643 0.643
6 -1.000 0.500 -0.500

Step 7: Change the content and observe direction

Keep q = (1, 0, 1, 0) but choose k = (0, 1, 0, 1). The raw dot now follows sin(90°×gap) + sin(10°×gap), with gap = query position minus key position.

Gap Raw dot for these different starting rows
0 0.000
1 1.174
2 0.342
-1 -1.174

Now +1 gives 1.174, while −1 gives −1.174. In this example the match distinguishes which side the key is on. A causal mask will still forbid future keys; these dots alone are not permission to attend to them.

Step 8: Return to the actual seven-word sentence

Now use each word’s own starting row from the shared table. The position-only shortcut no longer applies to every pair. Rotate each q and k using its actual position; keep v unchanged.

Position Word Actual rotated q = rotated k Unrotated v
0 the (1.000, 0.000, 1.000, 0.000) (0.000, 1.000, 0.000, 1.000)
1 cat (-1.000, 0.000, -0.174, 0.985) (1.000, 0.000, 1.000, 0.000)
2 sat (-1.000, -1.000, 0.000, 0.000) (0.000, 0.000, 1.000, 1.000)
3 on (0.000, 0.000, 0.366, 1.366) (1.000, 1.000, 0.000, 0.000)
4 the (1.000, 0.000, 0.766, 0.643) (0.000, 1.000, 0.000, 1.000)
5 soft (0.000, 1.000, -0.766, 0.643) (1.000, 0.000, 0.000, 1.000)
6 mat (0.000, -1.000, 0.500, 0.866) (0.000, 1.000, 1.000, 0.000)

For the second “the,” q̂₄ = (1, 0, 0.766, 0.643). Against “cat” at 1, k̂₁ = (−1, 0, −0.174, 0.985). The raw dot is −1 + (0.766×−0.174) + (0.643×0.985) ≈ −0.500; scaled score −0.250. This is content and position working together.

Key position Word Raw dot after rotation Scaled score
0 the 1.766 0.883
1 cat -0.500 -0.250
2 sat -1.000 -0.500
3 on 1.158 0.579
4 the 2.000 1.000

Step 9: Convert scores to weights and gather the original values

The two “the” embeddings stayed the same at lookup. Their rotated keys changed the comparisons, which changes what the current token gathers.

What the second “the” gathers
the position 029.1%
cat position 19.4%
sat position 27.3%
on position 321.5%
the position 432.7%
These five weights add to 100% before rounding. “Soft” and “mat” are hidden by the causal mask.

The head output for this token is the weighted mixture of the five value rows: (0.309, 0.833, 0.167, 0.691).

This is one head’s output before an output projection, residual addition, normalization, or feed-forward block. Repeating the calculation for every query gives seven output rows, each with four numbers.

The picture to keep: two stacked turntables, one fast and one slow. Each token places its content arrows on them. Position turns the tables. Attention compares a query’s arrows with a key’s arrows; shifting both by the same amount preserves their relative turns when their starting content is held fixed.

Optional: why does the absolute position cancel?

Rotations preserve dot products. Turn q by mθ and k by nθ; undo the same mθ turn on both. You are left comparing q with k turned by (n−m)θ. The starting vectors still determine the content part of the match. In complex-number notation, the same cancellation comes from multiplying one rotated pair by the conjugate of the other.

Why many arrows, rather than just two?

Remember the fast and slow clocks? RoPE uses that same idea of several rotation speeds. Different pairs of numbers in a query or key turn at different rates. Fast turns give detailed changes over nearby positions. Slower turns give a more gradual signal over longer distances.

There is an important difference from the earlier sinusoidal method. There, clock readings were added to the word’s representation. Here, they control how the query and key are turned before a match.

Sinusoidal

Add a clock reading

The word starts its journey carrying meaning mixed with a position pattern.

RoPE

Turn the matching arrows

Position changes the comparison used to decide where attention should look.

The information being gathered, called the value, is not rotated by standard RoPE. A useful way to picture it: we adjust the way we choose a book from the shelf, rather than turning all the writing inside the book.

One thing to be careful about: RoPE is not a rule that says “nearby words always match better.” Our equal-gap demonstration fixes the starting arrows to make the geometry visible. Real queries and keys differ with content, and many rotating pairs contribute together. The model learns how to use that positional structure.

6. What happens when the document gets longer?

We have a method that describes relative relationships. Surely we can now read a document of any length?

There is a catch. The model learned from documents of certain lengths. A much longer document produces combinations of clock angles and distances that may be unfamiliar, especially for slowly turning pairs. Having a position system that accepts a large number does not guarantee that the model will use it well.

Imagine practising with a ruler marked from 0 to 4, then being handed one that runs to 16. You can read the numbers, but the patterns you learned on the short ruler may not transfer smoothly.

Position Interpolation: squeeze the longer ruler into the familiar span

One approach is to compress the position scale. To fit four times as many tokens into the old range, we treat position 12,000 like position 3,000 for the rotations.

Position Interpolation · four times the text
Familiar range
0 1 2 3 4
Longer text, compressed
0 4 8 12 16
Both rulers occupy the same span. The second fits more positions inside it, so neighbouring positions are packed closer together.

This helps keep the rotation range familiar. The cost is that nearby positions also become more similar: we squeezed the small gaps as well as the large ones. Models can be fine-tuned to adapt to the new scale. Position Interpolation, 2023.

NTK-aware scaling and YaRN: don’t slow every clock equally

If the fast clocks help distinguish neighbours, perhaps we should preserve more of their detail while slowing the clocks that describe longer stretches.

NTK-aware scaling changes the spacing of rotation speeds, stretching the slower ones more while leaving the fastest unchanged. YaRN makes a more explicit distinction: keep fast rotations, slow down slow rotations, and blend the treatment in between. It also adjusts the attention score scale.

Position Interpolation

Slow every clock

Simple compression, including the clocks that distinguish neighbours.

NTK-aware

Stretch speeds unevenly

Preserve the fastest rotation while slowing longer-range ones more.

YaRN

Treat fast and slow differently

Keep fast clocks, slow slow clocks, and blend the middle.

The aim is to gain room for long documents without losing so much detail between neighbouring tokens. YaRN, 2023, develops this frequency-selective approach.

Other approaches, such as LongRoPE, search for different stretching factors across the rotating pairs. LongRoPE develops that search-based approach. A larger RoPE base also makes the slower clocks turn more slowly. These are ways of reshaping the position scale, not promises that every distant detail will be retrieved correctly.

Where does this change our rows? All three keep the same token embeddings and the same query/key/value pipeline as RoPE. They change the schedule of turns applied to the query and key pairs. Interpolation compresses positions uniformly; NTK-aware scaling and YaRN treat rotation speeds differently. The word rows do not get longer, and no extra word rows are created.

Worked examples: extending the same four-number head

For this section we switch from the 90°/10° teaching clocks to standard d = 4 RoPE rates: 1 and 0.01 radians per position. The sentence, word embeddings, and weight matrices remain the same. Assume a training window of 4,096 tokens and a desired 4× extension. The seven words let us inspect the changed arithmetic; they do not demonstrate long-context quality.

Recipe Fast pair · radians/step Slow pair · radians/step Position 4 · fast / slow angles
Unextended standard RoPE 1.000 0.010000 4.000 / 0.040000 rad
Position Interpolation 0.250 0.002500 1.000 / 0.010000 rad
NTK-aware 1.000 0.002500 4.000 / 0.010000 rad
YaRN 1.000 0.003835 4.000 / 0.015341 rad
Example LongRoPE candidate 0.500 0.002500 2.000 / 0.010000 rad

Position Interpolation

Step 0: Compress the position scale

Position m becomes m/4. Equivalently, divide both rotation rates by 4: 1 → 0.25 and 0.01 → 0.0025. Position 4 takes the old position-1 angles.

Step 1: Turn the second “the” with the revised rates

Its pre-rotation query is still (1, 0, 1, 0). The revised rotation gives (0.540, 0.841, 1.000, 0.010). Each key receives its corresponding turn. The value rows stay as listed in the shared setup.

Step 2: Compute matches and mix values

Compare that query with the revised keys, divide raw dots by 2, then turn the scores into weights.

Key 0: the 1: cat 2: sat 3: on 4: the
Scaled score 0.770 0.345 0.679 0.501 1.000
Attention weight 21.8% 14.2% 19.9% 16.7% 27.4%

The second “the” gathers the original value rows into head output (0.309, 0.659, 0.341, 0.691). Embeddings and values were not directly changed by the positional rule. The seven query rows still produce seven output rows.

NTK-aware

Step 0: Change the base rather than every position

For the usual base-change rule with d = 4 and scale 4, the new base is 10,000 × 4^(4/(4−2)) = 160,000. Its rates are 1 and 1/sqrt(160,000) = 0.0025. The fastest rate stays put; the slower one changes.

Step 1: Turn the second “the” with the revised rates

Its pre-rotation query is still (1, 0, 1, 0). The revised rotation gives (-0.654, -0.757, 1.000, 0.010). Each key receives its corresponding turn. The value rows stay as listed in the shared setup.

Step 2: Compute matches and mix values

Compare that query with the revised keys, divide raw dots by 2, then turn the scores into weights.

Key 0: the 1: cat 2: sat 3: on 4: the
Scaled score 0.173 0.074 0.247 0.501 1.000
Attention weight 15.0% 13.6% 16.2% 20.9% 34.3%

The second “the” gathers the original value rows into head output (0.345, 0.702, 0.298, 0.655). Embeddings and values were not directly changed by the positional rule. The seven query rows still produce seven output rows.

YaRN

Step 0: Classify the two clocks by training-window rotations

The fast clock made about 651.9 turns in 4,096 positions; the slow one made about 6.52. Using YaRN’s conventional blend thresholds of 1 and 32 turns, the fast pair is preserved. The slow pair falls in the blend region, not the fully compressed region.

Step 1: Blend the slow rate and apply the attention-scale correction

The slow pair’s blend weight is (6.52 − 1) / (32 − 1) ≈ 0.178. Blend rate 0.01 with rate 0.0025 → 0.003835. The conventional attention correction multiplies q and k by 1 + 0.1 ln(4) ≈ 1.139, multiplying dot-product scores by about 1.296. We include that correction below. A four-coordinate head has no other pairs; a wide head has many intermediate cases.

Step 2: Turn the second “the” with the revised rates

Its pre-rotation query is still (1, 0, 1, 0). The revised rotation gives (-0.654, -0.757, 1.000, 0.015) before YaRN’s amplitude correction. Each key receives its corresponding turn. The value rows stay as listed in the shared setup.

Step 3: Compute matches and mix values

Compare that query with the revised keys, divide raw dots by 2 and apply the attention correction, then turn the scores into weights.

Key 0: the 1: cat 2: sat 3: on 4: the
Scaled score 0.224 0.099 0.320 0.651 1.296
Attention weight 13.5% 11.9% 14.8% 20.6% 39.3%

The second “the” gathers the original value rows into head output (0.325, 0.733, 0.267, 0.675). Embeddings and values were not directly changed by the positional rule. The seven query rows still produce seven output rows.

Example LongRoPE candidate

Step 0: Inspect a possible set of searched factors

Imagine the search selected compression factors 2 for the fast pair and 4 for the slow pair. Rates become 0.5 and 0.0025. These factors are invented to illustrate an uneven candidate; this is not a reproduction of LongRoPE’s search, position-dependent choices, or staged training.

Step 1: Turn the second “the” with the revised rates

Its pre-rotation query is still (1, 0, 1, 0). The revised rotation gives (-0.416, 0.909, 1.000, 0.010). Each key receives its corresponding turn. The value rows stay as listed in the shared setup.

Step 2: Compute matches and mix values

Compare that query with the revised keys, divide raw dots by 2, then turn the scores into weights.

Key 0: the 1: cat 2: sat 3: on 4: the
Scaled score 0.292 0.502 0.691 0.501 1.000
Attention weight 14.3% 17.7% 21.3% 17.6% 29.1%

The second “the” gathers the original value rows into head output (0.353, 0.610, 0.390, 0.647). Embeddings and values were not directly changed by the positional rule. The seven query rows still produce seven output rows.

A larger fixed base uses the same mechanics: for d = 4, base 160,000 gives rates (1, 0.0025) and the NTK-aware rows above. The distinction is how the base is selected—for example as a training configuration versus an extension rule. Long-context adaptation and evaluation still matter.

Position is one piece of long-context ability. The model also needs training, enough computation and memory, and attention patterns that can find useful information in a sea of text.

7. A picture or a video needs more than “before” and “after”

A sentence is arranged along a line. An image has rows and columns. A video adds time. Two image patches may be next to each other horizontally, vertically, or neither. A single place in a flattened list does not describe those relationships very naturally.

Multimodal RoPE assigns different groups of rotating pairs to different coordinates. Some describe horizontal position, some vertical position, and some time. Qwen2-VL uses this idea to handle text, images, and video.

Where across? → column Where down? ↓ row When? → video frame
The highlighted patch has a place across the image and down the image. In a video, it also belongs to a particular moment.

Where does this change our rows? An image is split into patches that become visual token representations. Each patch has a row of numbers, just as a text token does. Multimodal RoPE uses different groups of query/key coordinates for horizontal, vertical, and time rotations. Those groups live within the rows; a patch does not need a separate token row for each coordinate.

Worked example: a patch in a video

Step 0: Give three coordinates three pairs

This time we need six query/key coordinates: one pair for time, one for row, one for column. Four numbers can only hold two pairs. Use starting row (1, 0, 1, 0, 1, 0) for two patches to isolate geometry. This illustrates axis assignment, not Qwen2-VL’s exact dimensional allocation or frequencies.

Step 1: Choose the coordinates

Patch A is at frame 0, row 1, column 2. Patch B is at frame 0, row 0, column 2. They are in the same frame and column, one row apart. Use rates 10° per frame, 90° per row, 10° per column.

Patch Time pair Row pair Column pair Rejoined row
A (1, 0) (0, 1) (0.940, 0.342) (1, 0, 0, 1, 0.940, 0.342)
B (1, 0) (1, 0) (0.940, 0.342) (1, 0, 1, 0, 0.940, 0.342)

Step 2: Compare pair by pair

The time pair contributes 1, the row pair contributes 0, and the column pair contributes 1. Raw dot = 2. Divide by sqrt(6) → 0.816. The query and key comparisons carry three kinds of place at once.

Step 3: Gather visual information

With only those two keys for illustration, A matching itself scores 3/sqrt(6) = 1.225. A matching B scores 0.816. Softmax gives about 60.1% to A and 39.9% to B. If their unrotated value rows are respectively (1, 0) and (0, 1), A gathers (0.601, 0.399). Value width can differ from query/key width; rotations do not need to change the values.

The question is still the same one we started with: what information about place will help two pieces of content relate to each other? The answer now needs more than one direction.

8. Other questions researchers are asking

Once the main idea makes sense, a few more developments are easier to place.

Does every part of the representation need a rotation? Partial RoPE rotates only some of the query and key coordinates. The others retain a way to compare content without an explicit positional turn.

Can we keep a small position component separate? DeepSeek’s multi-head latent attention compresses stored attention information to save memory. Its decoupled RoPE design uses a separate component for the positional comparison so that rotation can coexist with that compression. DeepSeek-V2.

Do all layers need explicit position signals? A model that can look only backward already has an asymmetry: the first token sees no earlier tokens, while later tokens can see a growing history. This can provide clues about position, though it does not automatically give every token a reliable counter. Research on NoPE explores what models can learn without explicit positional encodings. The Impact of Positional Encoding on Length Generalization in Transformers.

Are token steps always the right unit? “Two sentences ago” may be more useful than “37 tokens ago.” Contextual Position Encoding, or CoPE, learns when to advance a position count based on content, rather than always counting every token equally. Contextual Position Encoding.

Design What changes in the row journey?
Partial RoPE Rotate only some query/key coordinates; leave the other coordinates unrotated.
Decoupled RoPE Use a separate query/key component for rotation alongside the compressed content path.
NoPE Add no explicit position rows, score bias, or rotation; order clues can come from the causal attention mask.
CoPE Learn content-dependent positional counts and use them in the attention comparison.

Worked examples: changing which information carries position

Partial RoPE

Step 0: Rotate one pair, leave one pair alone

Use the seven-word setup. Rotate only coordinates 0–1 at 90° per position. Coordinates 2–3 keep their content values. The second “the” at 4 becomes (1, 0, 1, 0) because its rotating pair completes a full turn; the first “the” at 0 is also (1, 0, 1, 0). In this toy they collide. The unrotated content pair does not supply another clock to separate them.

Step 1: Compare partial rotations and gather values

Only the first pair’s contribution changes with position. The other pair provides a direct content match.

Key 0: the 1: cat 2: sat 3: on 4: the
Scaled score 1.000 -0.500 -0.500 0.500 1.000
Attention weight 32.8% 7.3% 7.3% 19.9% 32.8%

The second “the” gathers the original value rows into head output (0.272, 0.854, 0.146, 0.728). Embeddings and values were not directly changed by the positional rule. The seven query rows still produce seven output rows.

Decoupled RoPE

Step 0: Use a content path and a separate positional path

Here is a tiny architecture sketch, rather than a complete DeepSeek model. Compress each four-number input to a two-number content latent c = (x₀ + x₂, x₁ + x₃). Let content queries/keys copy c; let values copy c too. A separate projected pair for rotation copies (x₀, x₁). Rotate that pair at 90° per position. The combined query/key width is four; value width is two.

Step 1: Add the two kinds of match

For the second “the,” content query is (2, 0) and its rotated positional query is (1, 0). For cat, content key is (0, 2) and its rotated positional key is (−1, 0). Their content dot is 0 and positional dot is −1: combined raw dot −1, scaled score −0.5.

Key Content dot Positional dot Combined / 2 Weight
0: the 4.000 1.000 2.500 41.5%
1: cat 0.000 -1.000 -0.500 2.1%
2: sat 2.000 -1.000 0.500 5.6%
3: on 2.000 0.000 1.000 9.3%
4: the 4.000 1.000 2.500 41.5%

Step 2: Gather the content values

The output is the two-number content mixture (1.810, 0.190). The memory idea is to cache a compressed content latent plus the positional key component, instead of all full content keys/values for every head. This toy does not implement the real multi-head projections or cache optimizations.

NoPE

Step 0: Skip explicit position additions and turns

Keep the word rows and q/k/v from the shared setup. Apply the causal mask. The second “the” sees keys 0–4 and has content scores (1, 0, 0.5, 0.5, 1). Its two “the” key matches tie. The first “the,” at 0, can only see itself.

Step 1: Compare the two output rows

The first “the” gathers 100% of its own value (0, 1, 0, 1). The second “the” mixes five available values:

Key 0: the 1: cat 2: sat 3: on 4: the
Scaled score 1.000 0.000 0.500 0.500 1.000
Attention weight 27.9% 10.3% 16.9% 16.9% 27.9%

The second “the” gathers the original value rows into head output (0.272, 0.728, 0.272, 0.728). Embeddings and values were not directly changed by the positional rule. The seven query rows still produce seven output rows.

Their contexts differ even without a position vector. The causal mask creates structure the model may learn to use; it does not guarantee a reliable numerical position counter.

CoPE

Step 0: Let the query decide how much each token counts

Use the second “the” and its base scaled scores (1, 0, 0.5, 0.5, 1). Following the paper’s implementation, sigmoid converts these into gates: (0.731, 0.500, 0.622, 0.622, 0.731). Gates are fractional and query-dependent, not hand-assigned word-category labels.

Step 1: Count gated contributions back to each key

For each allowed key, add the gates from that key through the query, including both endpoints. For “on” at 3, the contextual count is 0.622 + 0.731 = 1.354, rather than the raw token gap 1. For the first “the” it is about 3.207, rather than raw gap 4.

Step 2: Interpolate a position contribution

Choose a toy learned position-vector table e[p] = (b[p], 0, 0, 0), with b[0…4] = (0, 0.3, 0.1, −0.2, −0.4). Since q = (1, 0, 1, 0), q·e[p] is simply b[p]. At contextual position 1.354, interpolate 64.6% of b[1] + 35.4% of b[2] ≈ 0.229. Add that contribution to the base content score. This follows the score-level interpolation used in the paper’s example implementation.

Key Gate for this token Inclusive contextual count Position contribution New score
0: the 0.731 3.207 -0.241 0.759
1: cat 0.500 2.476 -0.043 -0.043
2: sat 0.622 1.976 0.105 0.605
3: on 0.622 1.354 0.229 0.729
4: the 0.731 0.731 0.219 1.219

Step 3: Gather unchanged content values

The new count controls a learned relative-position contribution. Initial word embeddings still have four numbers; we have changed the way relationships are measured.

Key 0: the 1: cat 2: sat 3: on 4: the
Scaled score 0.759 -0.043 0.605 0.729 1.219
Attention weight 20.6% 9.2% 17.6% 20.0% 32.6%

The second “the” gathers the original value rows into head output (0.292, 0.731, 0.269, 0.708). Embeddings and values were not directly changed by the positional rule. The seven query rows still produce seven output rows.

These approaches address different needs. We should not read the history as one method winning every contest. A system built for short passages, a system built for video, and a system built for long document retrieval may need different positional choices.

Put the story together

Go back to our seven-word sentence. We started with content and needed order. Here are the main answers we explored:

Idea Picture to remember What it gives the model
Learned absolute positions A drawer of seat labels A learned description of each place
Sinusoidal positions Several clock hands A position pattern generated by a formula
Relative representations “One step before me” A relationship reused throughout the text
T5 distance buckets “Next door” versus “across town” Learned preferences for groups of distances
ALiBi A cost for looking farther back A fixed preference that weakens with distance
RoPE Two arrows turning before a comparison Content matching that incorporates relative turns
RoPE extension methods Stretching a ruler, or slowing clocks A position scale adapted to longer text
Multimodal RoPE Across, down, and through time Relationships in more than one direction

The move from seat labels to relative relationships is worth remembering. Much of language depends on how words sit with respect to one another. “The adjective before this noun” is a pattern we can reuse anywhere. Rotations give attention a compact way to express those relationships, and the later work asks how to preserve them across longer documents and different kinds of input.

Compare the same token across the designs

Each row below is the head output for the second “the”, using the main walkthrough’s four-coordinate setup. Different outputs make the changed pipeline visible. These invented weights do not rank the methods by quality. Transformer-XL additionally consults the two memory tokens; its row belongs to the current segment.

Design Head output for position 4
Learned absolute (0.368, 0.741, 0.259, 0.970)
Sinusoidal (1.523, 0.801, -0.171, 0.558)
Shaw (0.273, 0.784, 0.282, 0.727)
Transformer-XL (0.322, 0.668, 0.332, 0.678)
T5 (0.310, 0.731, 0.269, 0.690)
DeBERTa (0.330, 0.659, 0.341, 0.670)
ALiBi (0.271, 0.773, 0.227, 0.729)
RoPE (90° / 10° toy) (0.309, 0.833, 0.167, 0.691)

Three small thought experiments

No calculations needed. Try answering before opening the explanation.

If we put ten words before “the red ball,” what changes?

The absolute positions change. The gap between “red” and “ball” does not. That is why relative position is a useful description of their relationship. Their full model representations can still change because the added words introduce new context.

With ALiBi, can a faraway word beat a nearby word?

Yes. The faraway word pays a larger distance penalty, but a sufficiently stronger content match can overcome it. ALiBi gives attention a preference; it does not forbid distant connections.

Why doesn't a clock formula automatically give a model unlimited context?

We can generate clock readings for a new position without having trained the model to use them well. Longer passages also bring unfamiliar relationships, more competing content, and higher computation and memory costs. A valid position label is only part of the problem.


Back to Blog