The Engineer's Approach to Mastering AI

Engineers learn easily from architecture diagrams, code samples, and by working through the data flow. However, when it comes to the basics of LLMs, such as the transformer mechanism or even how attention works, things suddenly become too abstract.

Immediately, you’re thrown into a barrage of mathematical symbols, strange equations, and abstract notation that make no sense whatsoever. You can’t even work through examples to get a real sense of what’s going on.

Rather than spending time learning how to read notation and work through papers, which usually depend on other papers, we can take a more engineering approach and think in terms of constructs we know.

In this issue, I’ll pick Attention as the mechanism and try to explain it using tables (like you’d know in databases).

1 The Tokens

Whenever inputs are processed by an LLM, they’re in the form of a stream of tokens. A token can be either a word or a sub-word, but for our purposes here, we’ll assume that they’re always whole words.

Let’s put them in a database “table”. Each token appears in sequence and has the following properties:

1. Embedding: Each token is represented as a vector, which we call an embedding. This is a standard practice in NLP and not specific to transformers.

2. Query, Key, Value: These are derived from a token’s embedding using three different transformation matrices**: Q, K, V.** The resulting column values are written in lowercase as q, k, v.

All we need to understand is that we derive 3 additional values for each token, which are not the same dimensions as the embedding.

2 Query, Key, and Value

This is an important concept to understand. Let’s see why they’re there.

First, let’s introduce the notion of the current token. Attention is calculated with respect to the current token. For example: “Cats chase mice.” Attention indicates how much we should also attend to the other words in the sentence, given that the current one is “chase.”

To calculate attention, the query (q) of the current token is used. We compute a dot product with the keys (k) of every other token to obtain a weight. The weight indicates how important the other token is relative to the current token.

This is how the calculation would look in our example, assuming the current token in our table is “chase”:

Now, based on the weight, we compute a weighted average of the values (v) of those tokens. In a typical bag-of-words model, you would average all the tokens equally, but in the case of attention, it’s weighted.

This only gives you the attention output. Usually, this is added to the embedding rather than used as-is. Thus, each token’s final embedding becomes a mixture of the surrounding context as well.

3 The What vs. the Why

One thing I wanted to illustrate is this: the attention mechanism becomes very simple once you see it and work it out the right way. Otherwise, if you’re just staring at equations without the right background knowledge, it will all seem abstract and confusing.

We usually jump to the WHY of everything we want to understand. But that’s usually a fatal mistake with deep learning. Much of what we know is empirical. And the only thing we usually know is that it works, without quite knowing “why.” In the end, it’s just a bunch of numbers that add up in just the right way to optimize for a desired output.

The more important question is the WHAT. When reading papers, things can seem particularly abstract, which prevents us from getting to the WHAT. But once you work out a proper example, everything seems very obvious.

4 Where to Go from Here?

Try to focus more on the WHAT of things. The best way is to interrogate subjects using the abstractions you already know.

The next time you sit down to learn something, take a notepad and try to sketch out something that resembles the problem space. Then think in terms of that sketch instead of abstract mathematical symbols. It will surely clarify a lot.

For example, we think of Kafka as a log, and it immediately makes sense. Because you can now play around with an actual log and still understand the characteristics of Kafka, even if it’s imperfect.

Similarly, a lot of LLM concepts map very cleanly to concepts you already know. For example, the embedding lookup is just a table lookup. Attention is just weighted averages. Multiple attention blocks are similar to applying the same function recursively to an input.

That’s what an Engineer’s way of understanding anything is. Turn it into something that can be explored, queried, interrogated, and thus understood, instead of keeping it all theoretical. And that’s the exact approach that also works for AI.

That’s all for this week. See you in Issue 13!