Randomness and Determinism

A big barrier to LLM adoption is often its randomness. The answer Joe gets in the morning is different from the one in the evening, for the exact same question. And it’s not just the wording being changed, but also the underlying facts.

Let’s understand determinism better, and also how you can build robustness into your applications, especially where determinism is required.

First, let’s understand that in the world of LLMs, meaning and representation are intertwined. Although the LLM appears to give similar answers when you phrase the same question differently, a different section of the network might be getting activated more because of it. So, the answer to the following two prompts might be coming from different parts of the LLM’s network:

1. “Write me a great quote to put on a book.”

2. “Write me an epigraph.”

In humans, representation and meaning are seen as separate things. Although LLMs build higher-level representation in the vector space instead of using raw tokens, they’re still a function of tokens.

In short, in the human world, meaning came first, and then language. In the world of LLMs, language came first and then came meaning, which was merely a function of language and indistinguishable from it.

Second, the formulation of any machine learning/AI problem is usually highly statistical. For example, when we say that a traditional model’s accuracy is 98%, it does not mean that the “error” is at most 2%. Instead, what it means is that the model makes errors 2% of the time you call it, without necessarily talking about how wide those errors go.

LLMs also operate statistically in many places. But the one that concerns us post-training is the decoding process. An LLM outputs token probabilities, and the sampling of them is decoding.

Because decoding samples from token probabilities, the guarantees that we get are similar to an ML model: that for some X% of the time, the answer will make sense, and for the remaining, it won’t. But even with this, there isn’t necessarily an upper-bound on the magnitude of an individual error.

So, if you run an LLM 100 times on a problem, then many times you might get a great answer; most of the time the answers would be average; and sometimes the answers would be ridiculously wrong. That’s the goal being optimized for.

This brings us to the goal of determinism. That’s not what the LLM is being optimized for at all, and it cannot be, because meaning and representation are intertwined. Instead, our broad approach would be to:

1. Make the decoding process greedy; meaning it will always pick the same tokens in the output for the same input tokens. This is not the same as being correct, but we do try to be more stable.

2. Make the token probabilities as wide as possible, especially in the top few tokens. We don’t want the first- and second-most probable tokens to fight for first place depending on how the floating-point fluctuation goes.

3. Split the LLM calls into two parts: Robust and Creative

a. The Robust part is responsible for outputting a concise JSON, which is the core deterministic output. This is where greedy decoding and wide token probabilities matter. We’ll do this by using an indirect approach to answering questions by building a Rubric.

b. The Creative part can be a standard decoding process, which takes the output of the Robust part and builds a narrative from it. This way, we try to decouple meaning and representation to some extent, by having the Robust stage provide the meaning, and the Creative stage build the representation. Even if the Creative stage uses different words each time, the underlying answer would be stable.

To better understand the concept, let’s walk through an example.

Consider a content marketing problem. Given a user’s prompt, you want to devise a marketing strategy and then help create content around it. However, the problem is that each time you run the prompt, not only does the content change, but the overall strategy does as well. And that’s bad, because there’s no anchor for the LLM’s output.

Instead, Step 1 should be to create a list of possible strategies and their descriptions. This becomes your core taxonomy and is an important IP.

Then, Step 2 should be to tag each strategy with a fixed tagset or rubric. This is where you might add tags like growth if it’s for growth, and festival if it’s for holidays, etc. The classes and types you choose are entirely up to you and your use case.

Then, Step 3 should list all the tags you’ve used, along with a description of each tag and when it becomes relevant. This helps teach LLMs how to tag content.

Then, Step 4 should take the user’s prompt and tag it based on the above scheme using the LLM.

Then, Step 5 should be about taking these tags and the list of strategies and scoring each strategy in a stable-ordered fashion. The output would then be a single strategy, along with the tag intersection explaining why it was the highest scoring.

Step 6 could be to take those intersected tags and form a verbal explanation of them. Even if the words change, the intent behind them won’t.

In short, decomposing the big decision into smaller parts and having the LLM then “tag” those smaller features is a more reliable way to use LLMs when decisions need to be made. They’re still not guaranteed to be fully deterministic, for that will depend on your tagging choices.

And that’s one way to approach “determinism.”

See you in Issue 15!