Issue #5: Using Domain Knowledge to Steer LLMs

There’s an important rule of writing that is not widely known outside certain circles but is tremendously valuable for writing LLM prompts.

If you use LLMs enough, you’ll quickly realize they don’t always do what you want. Sometimes they veer off in the wrong direction, and you have to keep correcting the prompt to nudge them.

It could be as simple as the LLM choosing Flexbox when the problem statement clearly requires a CSS grid. And since, as of today, the grid is not as “popular” an approach, the LLM might statistically fall back to Flexbox.

One of the problems with software engineering is that for each problem, there are many plausible solutions that appear to work. The more robust and maintainable you want them to be in the long run, the fewer solutions will actually fit the bill.

Experienced engineers recognize the difficulty of nudging the LLM toward the few right solutions without spelling everything out.

The 3 Approaches to Improving LLM Output

With LLMs, you have 3 options to improve the output.

Expand the instructions to add more detail, as that is what we usually do.
Add more examples, which may or may not be feasible. Examples help because they provide richer context than elaborate instructions (counterintuitively).
Or we keep the prompt almost the same, with only a few magic fixes. This is the approach we’ll discuss.

LLMs and Knowledge

First of all, understand that most of an LLM’s knowledge is stored in the weights of the neural network. It’s not always clear how that knowledge is distributed across those weights, and the Mixture-of-Experts only complicates the matter further with additional non-linearities.

Thus, it’s important for our prompts to activate the right kind of weights, which will then drive the right outputs with high probability.

It’s also important to understand that while LLMs learn good stuff from the “world,” they also learn its mistakes. So you could get an LLM to do something you don’t know how to do directly, but you also have a high chance of inheriting side effects. For example, people trying to get LLMs to write good content often inherit the side effect of excessive em dashes or emojis.

Elaborate Prompts

Elaborate prompts are like trying to activate blotches of behavior in the neural network, each with its own side effects. Moreover, these blotches might not coalesce into something you want. For example, you could ask ChatGPT to write a bubble-sort implementation in Python in the style of Linus Torvalds, and you’ll get “C” written in Python. C-like code is not desirable in Python; similarly, a more subtle “mixture” of instructions can lead to undesirable code.

Vivid Examples

Examples, on the other hand, are more convenient. But examples need to be clearly annotated, because the LLM can sometimes confuse the core thing you’re trying to teach with the background stuff that needs to exist. And many times, it can overfit on the wrong thing. For example, asking the LLM to “write in your style” by providing a sample of content you’ve written yourself is still not a reliable way for it to fully mimic your thinking and writing. Since it’s your style, you’d soon realize that it’s only surface-level.

The Magic Fix

While elaborate instructions and examples have their place, we’ll now focus on the importance of the “magic fix” we discussed. Foundation model companies do talk about it briefly, but only in a very limited context. But we’ll take it a step further.

Before that, let’s understand one hidden rule in writing. Consider the following sentence.

“The team made the process better.”

That sentence is generic. We have a weak adjective, “better”. It doesn’t bring any vivid imagery to your mind, and it won’t even activate rich context in the case of LLMs for the same reason.

But consider this version of it:

“The team streamlined the process.”

Here, “streamlined” is a strong verb, and we got rid of the adjective “better”. It conveys more details, and does it sharply, in that we now see the process getting shorter and sharper in some fashion. It’s still not a vivid sentence, but enough for our purposes.

Thus, that rule of writing is: use strong verbs and nouns instead of resorting to weak adjectives and adverbs.

Magic Fix Applied to Programming

Similarly, when it comes to instructions for the LLM, we could write something weak like:

“Make this function robust.”

Depending on the LLM’s temperature value, you’d get several different interpretations of this each time you prompt it. Because robust is such a generic adjective, the variance in your output will be very high.

Instead, what if we said:

“Add precondition and postcondition assertions to the function.”

Immediately, there are a couple of strong words here, namely: precondition, postcondition, and assertion.

This time, although the LLMs’ outputs will still vary across runs of the same prompt, the variance will be much lower. Also, the output will be far more useful to a software engineer.

The Broader Implication

But there is one more important thing happening. It’s not that we’ve just given the LLM a more “specific” instruction. Actually, we’ve not added a whole lot of detail on how to do this task.

Instead, what we’ve done is use certain words that have broad implications and will activate the full breadth of the context we need, hopefully with most of the side effects, while also being very cohesive.

When the LLM encounters such words, it knows other words (or tokens) that are usually in the neighborhood of those words, and it will activate context accordingly. It’s like using academic terms and getting an academic-grade response, simply because of the choice of words. In short, the LLM is very much like a mirror.

What we’re accomplishing is not merely the 3 strong words we used, but also the other things those words are associated with. Remember how we used the word “Linus Trovald” and got the LLM to write C-like code in Python without having to specify any of it? In this case, we used a strong proper noun instead of elaborate instructions.

That’s also what foundation model companies talk about, but they only talk about having the LLM assume a “role”, like: “You’re an expert software architect…” That line is also accomplishing what we’re talking about.

A good role replaces tons of instructions you’d have to give otherwise, all because it’s the same principle in use.

Putting This Into Action

How does this impact how we develop software using AI? Well, consider this to be a domain-driven design approach to prompting. When you understand the domain of the problem you’re solving, as well as the domain of computing, it becomes easy to find the right words to activate the right context, so that the LLM can narrowly generate exactly what you thought it should. (Incidentally, it’s also true that LLMs don’t really “follow” instructions; they just pretend to)

Instead of asking the LLM to split large functions into smaller ones, you could ask it to do “top-down decomposition.”

Or instead of asking it to parse a file, ask it to create a “recursive-descent parser.”

In short, your domain knowledge of engineering is your biggest advantage.

Three rabbit holes people seemed to like

A few related ideas on thinking and engineering took off recently:

Closing Thoughts

I believe computer science and software engineering knowledge would become more important, albeit in different forms, as AI becomes a powerful tool. Because any powerful tool requires a good deal of judgment and knowledge to use properly.

See you in Issue 6!