Sunday, November 22, 2026
Mobile Offer

🎁 You've Got 1 Reward Left

Check if your device is eligible for instant bonuses.

Unlock Now
Survey Cash

🧠 Discover the Simple Money Trick

This quick task could pay you today — no joke.

See It Now
Top Deals

📦 Top Freebies Available Near You

Get hot mobile rewards now. Limited time offers.

Get Started
Game Offer

🎮 Unlock Premium Game Packs

Boost your favorite game with hidden bonuses.

Claim Now
Money Offers

💸 Earn Instantly With This Task

No fees, no waiting — your earnings could be 1 click away.

Start Earning
Crypto Airdrop

🚀 Claim Free Crypto in Seconds

Register & grab real tokens now. Zero investment needed.

Get Tokens
Food Offers

🍔 Get Free Food Coupons

Claim your free fast food deals instantly.

Grab Coupons
VIP Offers

🎉 Join Our VIP Club

Access secret deals and daily giveaways.

Join Now
Mystery Offer

🎁 Mystery Gift Waiting for You

Click to reveal your surprise prize now!

Reveal Gift
App Bonus

📱 Download & Get Bonus

New apps giving out free rewards daily.

Download Now
Exclusive Deals

💎 Exclusive Offers Just for You

Unlock hidden discounts and perks.

Unlock Deals
Movie Offer

🎬 Watch Paid Movies Free

Stream your favorite flicks with no cost.

Watch Now
Prize Offer

🏆 Enter to Win Big Prizes

Join contests and win amazing rewards.

Enter Now
Life Hack

💡 Simple Life Hack to Save Cash

Try this now and watch your savings grow.

Learn More
Top Apps

📲 Top Apps Giving Gifts

Download & get rewards instantly.

Get Gifts
Summer Drinks

🍹 Summer Cocktails Recipes

Make refreshing drinks at home easily.

Get Recipes

Latest Posts

Positional Embeddings in LLMs: How Transformers Track Order


“The dog hunted the fox.” “The fox hunted the dog.” They are the same words, but the meanings are totally different. You can note the difference at once, but a transformer can’t.

This is the problem which positional embeddings solve. They inform a language model about the position of each token in the sentence. Without them, one sees a prompt merely as a collection of words.

We will explore these notions starting from basic facts. First, we will find out why transformers are unable to understand word order. Then we will look through three important types of positional embeddings: learned, sinusoidal, and RoPE.

Why Do Transformers Need Positional Embeddings?

The central concept of a transformer is attention. Every token looks at all other tokens to figure out what is relevant. However, the way attention operates is through the dot product of the vectors. Since the dot product doesn’t depend on the origin of the vectors, it loses track of which vectors were processed first.

For instance, think of a group chat without timestamps. You can see all the messages sent, but you have no idea about the order of sending. That is how attention works without a position.

Let’s demonstrate this with some code. We will take two sentences and swap some words in them to perform self-attention.

import numpy as np

np.random.seed(0)
emb = {w: np.random.randn(4) for w in ["the", "fox", "chased", "dog"]}

def attention(x):
    scores = x @ x.T / np.sqrt(x.shape[1])
    weights = np.exp(scores) / np.exp(scores).sum(axis=1, keepdims=True)
    return weights @ x

s1 = np.stack([emb[w] for w in ["fox", "chased", "dog"]])
s2 = np.stack([emb[w] for w in ["dog", "chased", "fox"]])

out1, out2 = attention(s1), attention(s2)
print(np.allclose(out1[0], out2[2])) # True

The result is True. Fox shows the same representation in both sentences. The model does not know whether the fox is chasing or being chased.

When positional embeddings are absent, the fox shows the same output.

But doesn’t the causal mask help?

Somewhat, yes. The GPT-style model employs a causal mask. Each token can see the preceding tokens only. This creates some order information.

However, its signal is not clear. It cannot differentiate between a single-token gap and a ten-token gap. Therefore, many models add explicitly coded position data on top of that.

Approach 1: Learned Absolute Positional Embeddings

The simplest solution would be to use the idea from GPT-2, where every position is assigned a unique vector. These vectors would be learned during the training phase, not dissimilar to how the word embeddings are learned.

Then the position vector gets added to the token vector.

  • For example, “fox” at position 3: embedding("fox") + position_vector[3]
  • Yet at position 7: embedding("fox") + position_vector[7]
  • In this way both “fox”s token will look different for the attention.

The analogy here would be the theater seats and their numbers.

Unfortunately, there are only 1,024 position vectors learned in GPT-2. Thus position number 1,025 simply does not exist. Hence, you would have to select the context length before the very beginning of the training.

The cost also grows very fast. The numbers observed for GPT-2 Medium:

Context length Position weights as a share of the model
1,024 tokens About 0.3%
Roughly 100,000 tokens More than 22%

That’s a lot of parameters to count and even remember.

Approach 2: Sinusoidal Positional Encoding

The initial paper on Transformers from 2017 embraced a unique approach. Rather than acquiring positional vectors, it created them through the combination of sine and cosine waves. There aren’t any learning parameters to be taken into consideration.

Various pairs of characteristics oscillate with different speed, where some oscillate rapidly while others do not oscillate a lot.

As an analogy, we can take that of a watch that has three hands. The hand that marks seconds oscillates quickly. Hand that marks minutes oscillates more slowly than the second hand. The hand that marks hours barely oscillate. Thus, when we read three hands, we know what time is.

import numpy as np

def sinusoidal_encoding(num_positions, d_model):
    pos = np.arange(num_positions)[:, None]
    i = np.arange(0, d_model, 2)[None, :] # even feature indices
    freqs = 1 / (10000 ** (i / d_model)) # fast to slow "clock hands"
    pe = np.zeros((num_positions, d_model))
    pe[:, 0::2] = np.sin(pos * freqs)
    pe[:, 1::2] = np.cos(pos * freqs)
    return pe

pe = sinusoidal_encoding(num_positions=50, d_model=8)
print(np.round(pe[:3], 3))

Consider the output rows. In the first column, the value seems to randomly bounce from one position to another; however, the other columns remain almost unchanged. This combination of speed makes every position uniquely identifiable.

Sinusoidal waves representing positional encoding fingerprints

The sinusoidal positional encoding can be represented using fast, moderate, and slow waves and a heatmap of values for 64 positions.

The upper chart presents three properties consisting of waves. The lower heatmap makes a composite chart consisting of all 64 properties. In any column, you can read the individual fingerprint of a certain position from the top to the bottom.

Two favorable effects can be observed:

  • It is possible to compute positional vectors for any position (even in instances which are more extensive than one used for training);
  • An invariable displacement related to different positions is transformed into a simple linear transformation.

That position was intended to assist the models in understanding the relative distances; in point of fact, it was partially effective.

Approach 3: Rotary Position Embeddings (RoPE)

The RoPE (Rotary Position Embeddings) emerged in 2021 with the work on RoFormer and has since become the mainstream technique in modern LLMs. The concept itself is very simple.

RoPE doesn’t introduce anything new to the token embeddings but instead simply rotates the query and key vectors in the attention mechanism according to the position of the token. This is where RoPE distinguishes itself from the previous methods. Earlier techniques introduced position once during the embedding of the tokens; RoPE is different in that it applies this technique every time there is an attention layer.

Comparison of standard positional embeddings and RoPE

Let’s illustrate this using an image. Consider each pair of dimensional features as arrows in a 2-dimensional space. The token at position 1 rotates its arrow a bit, while the token at position 5 rotates it five times more. However, the length of the arrow will remain the same; only its direction will change.

The most interesting part is something related to how attention works as it uses the dot product to compare the query and key arrows.

How does rotation encode relative position?

Let’s take the illustrative example, assuming that one pair of features is tilted by 20 degrees for each position. At first, we take query and key as (1, 5). This is the moment when “aha” comes up for everyone! Position 5 and position 2 are at a distance of 3 from each other. Position 105 and position 102 have the same distance. Both situations yield a score of 13.

Rotary position embeddings using relative angles

Rotating RoPE: at positions 5 and 2 the source and target vectors create 60 degrees between them; at 105 and 102 the vectors create 60 degrees.

The representations of different vectors look definitely different in the images. But their slopes remain equal to 60°. So, the score did not change.

If we take the key one position closer and make it position 3, we will see the change in score. The score will equal 19.92, and it means that smaller gap leads to smaller distance and larger score. The model now can “sense” the distance directly.

Here’s the whole example:

Query position Key position Gap Angle between them Attention score
No rotation No rotation n/a 0° 26.00
5 2 3 60° 13.00
105 102 3 60° 13.00
5 3 2 40° 19.92

What is the use of it? In terms of a system prompt, if you had 100 tokens of instructions in a prefix, everything still gets moved from its position, unlike in RoPE where the relation of the vectors of your later tokens does not change.

What about the other feature pairs?

The actual attentiveness of a head consists of various characteristics – for example, it can have 128. The split takes place within the concept of RoPE so that there are 64 pairs. In each pair, the rotation must be executed continuously at various speeds.

What we have here is the same trick as the clock hand. The rotating pairs are fast, and they help to investigate relationships within a close distance. Slow rotations help to pay attention to remote connections. The above-mentioned speeds are like sinusoidal encodings.

Mini Project: Build RoPE and Verify It Works

Let’s carry out RoPE to implement a full-attention head and test the shift invariance on random data.

Step 1: Write the RoPE function

This specific RoPE version relies on a “split-half” pairing. Feature 1 goes with feature 5, feature 2 with feature 6, and so forth, just as it was.

Each pair spins the same way as we have shown in previous examples but now the only difference is that they spin at a different speed.

Step 2: Test the shift invariance

We score the random queries and keys at positions 0–5 and create a faux prefix of 100 tokens.

np.random.seed(42)
seq_len, head_dim = 6, 8
q = np.random.randn(seq_len, head_dim)
k = np.random.randn(seq_len, head_dim)

pos = np.arange(seq_len)
scores = rope(q, pos) @ rope(k, pos).T

shifted = pos + 100
scores_shifted = rope(q, shifted) @ rope(k, shifted).T

print(np.allclose(scores, scores_shifted)) # True

We get true as output. Every attention score is identical after the shift. Only relative distances survive, which is exactly what we wanted.

Notice we only rotated queries and keys, never values. Position shapes who attends to whom.

What Are Modern LLMs Doing Now?

RoPE continues to lead. Nevertheless, other models have started to change it in interesting ways:

Model Positional choice What it means in plain words
MiniMax-M3 Partial RoPE Rotates only 64 of 128 features per head
SmolLM3 RoPE plus NoPE layers Skips RoPE in every fourth layer
Inkling Learned relative bias Learns distance-aware adjustments to attention scores
T5 (2019) Simpler relative bias Adds a learned number based on token distance

NoPE is somewhat peculiar, how are we able to determine position if there is no encoding? The answer is the causal mask. Previous layers also relay position. The developers of SmolLM3 have claimed that this method of combining layer signals leads to better performance in long-context situations.

My personal view is that RoPE will not lose its relevance anytime soon. However, we can see that the universal use of “RoPE everywhere” is coming to an end.

Positional Embeddings at a Glance:

Here’s how the three main approaches stack up side by side:

  Learned absolute Sinusoidal RoPE
Where position enters Added to input once Added to input once Rotates Q and K in every attention layer
Learned parameters One vector per position None None
What attention sees Absolute position Absolute position Relative distance
Beyond training length Impossible without a bigger table Computable, quality not guaranteed Computable, quality not guaranteed
Used by GPT-2, BERT Original Transformer LLaMA, Qwen, most modern LLMs

Common Misconceptions About Positional Embeddings

A few concepts confuse almost every individual the first-time round:

  • “Sinusoidal representation works for arbitrary length.” It works for any length. Anything outside the training length is different.
  • “RoPE applies position into embedding.” It hardly does. It simply rotates queries and keys.
  • “RoPE too rotates value.” It doesn’t. Values are left untouched.

Two other things to remember:

  • Pairing conventions are not the same for everyone. Original LLaMA pairs adjacent features while Qwen3-style code works split-half. Misalignment during weights loading will lead to a dead model.
  • A causal mask cannot do anything alone. It provides rough ordering but not proper distance.

Conclusion

The positional embeddings method allows transformers to know about the significance of the order of words. In the GPT-2 network, a vector for a specified location was used. In the case of the original Transformer, sine functions were applied. The idea of the RoPE function is that its queries and keys are now rotated, meaning that the attention scores are calculated based on relative distances.

Your task is to use the rope function in the first attention code snippet. After that, perform the fox-and-dog experiment again. The answer should now be negative.

Read more: Kimi K3 Architecture Explained: How a 2.8T Model Runs Efficiently

Frequently Asked Questions

Q1. Why can’t Transformers understand word order naturally?

A. Transformers use attention mechanisms based on dot products, which treat inputs as a set rather than a sequence, causing them to lose track of token order.

Q2. What is the main limitation of learned absolute positional embeddings?

A. They require a fixed context length during training and cannot handle sequences longer than the maximum number of position vectors defined initially.

Q3. How does RoPE differ from earlier positional embedding methods?

A. RoPE rotates query and key vectors within the attention mechanism based on position, rather than adding static position vectors to token embeddings.

Riya Bansal

Data Science Trainee at Analytics Vidhya
I am currently working as a Data Science Trainee at Analytics Vidhya, where I focus on building data-driven solutions and applying AI/ML techniques to solve real-world business problems. My work allows me to explore advanced analytics, machine learning, and AI applications that empower organizations to make smarter, evidence-based decisions.
With a strong foundation in computer science, software development, and data analytics, I am passionate about leveraging AI to create impactful, scalable solutions that bridge the gap between technology and business.
📩 You can also reach out to me at [email protected]

Login to continue reading and enjoy expert-curated content.



Source link

Mobile Offer

🎁 You've Got 1 Reward Left

Check if your device is eligible for instant bonuses.

Unlock Now
Survey Cash

🧠 Discover the Simple Money Trick

This quick task could pay you today — no joke.

See It Now
Top Deals

📦 Top Freebies Available Near You

Get hot mobile rewards now. Limited time offers.

Get Started
Game Offer

🎮 Unlock Premium Game Packs

Boost your favorite game with hidden bonuses.

Claim Now
Money Offers

💸 Earn Instantly With This Task

No fees, no waiting — your earnings could be 1 click away.

Start Earning
Crypto Airdrop

🚀 Claim Free Crypto in Seconds

Register & grab real tokens now. Zero investment needed.

Get Tokens
Food Offers

🍔 Get Free Food Coupons

Claim your free fast food deals instantly.

Grab Coupons
VIP Offers

🎉 Join Our VIP Club

Access secret deals and daily giveaways.

Join Now
Mystery Offer

🎁 Mystery Gift Waiting for You

Click to reveal your surprise prize now!

Reveal Gift
App Bonus

📱 Download & Get Bonus

New apps giving out free rewards daily.

Download Now
Exclusive Deals

💎 Exclusive Offers Just for You

Unlock hidden discounts and perks.

Unlock Deals
Movie Offer

🎬 Watch Paid Movies Free

Stream your favorite flicks with no cost.

Watch Now
Prize Offer

🏆 Enter to Win Big Prizes

Join contests and win amazing rewards.

Enter Now
Life Hack

💡 Simple Life Hack to Save Cash

Try this now and watch your savings grow.

Learn More
Top Apps

📲 Top Apps Giving Gifts

Download & get rewards instantly.

Get Gifts
Summer Drinks

🍹 Summer Cocktails Recipes

Make refreshing drinks at home easily.

Get Recipes

Latest Posts

Don't Miss

Stay in touch

To be updated with all the latest news, offers and special announcements.