This post is based on Chapter 3 of Build a LLM from Scratch by Sebastian Raschka, with one twist: all Python examples are rewritten in Elixir. We are building LLMs attention mechanism. Attention mechanism is second part of stage 1. We have already prepared input text data. Attention mechanism help LLM to predict next token. We will implement four attention mechanism:

• simplified self-attention
• self-attention
• casual attention
• multi-head attention https://karlosmid.com/2026/03/coding-attention-mechanism-in-elixir/