Sequences and attention

Choose your learning level

BeginnerIntermediateAdvanced

Start with plain-language explanations. Switch levels at any time.

Beginner explanation

Word order and context matter. In “the bank beside the river,” nearby words help identify which meaning of bank is intended. Attention lets a model combine information from different positions in a sequence.

Intermediate explanation

Transformers use attention to relate representations of tokens. Positional information helps distinguish order. Causal language models mask future positions during next-token prediction; other architectures use different masking schemes for different tasks.

Advanced explanation

Scaled dot-product attention computes softmax(QKᵀ/√dₖ)V. Queries and keys determine normalized weights over value vectors. Standard full attention has quadratic pairwise interaction growth with sequence length; attention weights are not a reliable causal explanation of a model’s answer.

PUT IT IN CONTEXT

A worked example

In “The parcel did not fit in the locker because it was too small,” resolving “it” requires relationships across the sentence. A model may infer locker, but ambiguity and unfamiliar wording can still produce mistakes.

TRY IT YOURSELF

Practice the idea

Why should a next-token training example hide later tokens from a causal model? Explain using “The sky is …”.

Your working notes (not saved or sent)

Reveal the answer guide

If the target word were visible, the model could copy information it will not have at generation time. A causal mask preserves the intended prediction task by blocking future positions.

Go further: advanced challenge

Compare doubling sequence length under full attention with doubling only the hidden width. Identify which dimensions appear in QKᵀ and avoid claiming one universal runtime multiplier.

An open-ended discussion task. Explain your assumptions and evidence; there may be more than one defensible answer.

Continue the conversation

What part of this idea would you like to test or question? Compare your reasoning with the community.

Find related forum discussions ↗

Opens a topic search in the existing AstraCruz forum. Results depend on available posts; signing in may be required. No discussion or activity is fabricated.

Browse all forums