The Most Important Component in an Agentic System

Most believe that the choice of model, the available tools, or even the sophistication of the prompts are the most important factors in building an agentic system.

They’re wrong.

That’s why they spend months working and reworking the agentic part of the system, and much less on product depth.

Before I can give you the most important factor, it’s important to step back and see the big picture of software architecture.

The Big Picture of Software Architecture

Software architecture is not merely a “solution” to a problem. It’s more than that. Rather, it’s an ecosystem that determines which classes of problems will exist in the first place and how their solutions might look.

For example, if you choose to use a SQL database, you’ll face a class of problems that includes normalization, relational modeling, and specific types of optimizations, like write performance.

Instead, if you choose to use a document-based database, your class of problems might include rethinking how you’d manage relationships, how you’d keep things in sync, or how you’d prevent a document from becoming too large.

A project can make a decision in its first week and then spend years solving problems in that specific direction.

As you can see, the big-picture solutions are not just things we apply and forget. They also determine which types of future problems we’ll face, each of which will require its own creative solutions.

Being Off Course

Software architecture has always been a tricky subject, and it applies to building agentic systems too. There are two opposing forces.

One is planning ahead and recognizing that a few right decisions made early can make future builds easier.

But the second is principles like YAGNI, where we try to make decisions that only make sense for today and leave the problem of adaptation to ourselves tomorrow.

However, these two are not opposing forces as they initially appear. Good architecture is always required for long-term projects, and YAGNI is best applied to the sub-problems that arise from those architectural decisions on a day-to-day basis.

In the case of AI agents, the decisions often revolve around choosing the best model or creating the best system prompts. It could also be that the whole agentic system is just a collection of specialized prompts, each targeting a specific workflow.

For example, a project could have specialized workflow prompts that allow it to sequentially create marketing campaigns. Another could review messages for adherence to brand guidelines. And so on.

However, the “collection of workflows” approach doesn’t scale well. And the more workflows you add, the more “confused” the LLM can eventually get about what you want to achieve.

That’s not a good class of problems to ask for.

The Most Important Component

The most important component in an agentic system is the harness. The harness is the loop in which you call the LLM and perform other tasks, such as calling tools.

For example, the ReACT loop is a type of simple harness. In it:

1. The harness accepts a user prompt (with a system prompt already in place).

2. The harness calls the LLM with that user prompt.

3. If the LLM responds with tool calls, the harness calls those tools and passes those results back to the LLM.

4. The harness keeps looping until the LLM’s output contains no more tool calls (or we reach the maximum iteration limit).

5. The final response is what is shown to the user and is also added to the conversation history.

Why is the harness the most important component? Because once you create the harness and treat it as a rigid structure, you can then decide what your tools should be and how they can come together to support various workflows.

In short, with a ReACT-like harness and the right tools, all your actual workflows and use cases you target become emergent behavior of the LLM. The mistake usually made here is to force and “teach” the LLM through prompts everything you want exactly, whereas the right way is to put in the guardrails and let the LLM “reason” through it. That’s how it learns to do well what you haven’t exactly specified.

Anthropic also uses a very simple harness that relies on the LLM to make the right reasoning choices, given all the right tools.

Thus, the harness is the most important decision you’ll make, and it’s highly strategic. It’s not something you can change without also impacting all the tools, prompts, and guardrails you’ve built.

A ReACT-like harness is the most flexible for building powerful agents.

Emergent Behavior

Compared to workflows, a ReACT-like structure relies heavily on emergent behavior. It’s like giving the LLM a clear guideline for the overall objectives and treating the tools as puzzle pieces, with the LLM stitching them together one at a time to reach the desired result. You can even have it write a “plan” first to create a tentative assemblage of those puzzle pieces.

Rather than explicitly prompting the LLM to write an email and then send it, you need to let the reasoning layer figure out that sending an email requires using both the “write_content” and “send_email(content)” tools. That’s where using LLMs as the reasoning layer becomes very powerful.

And as an aside, I want to note that your tools can be your most powerful assets. Tools can be entire programs, algorithms, or anything proprietary that you create. LLMs merely actuate those tools and do not eliminate the need for clever solutions designed by you. It’s just that those tools are now also accessible via an API to the LLMs for orchestration.

Guardrails

By guardrails, we don’t just mean adding safety instructions to the prompts. It’s also about designing the tools so that bad things can’t happen easily.

For example, regardless of what your API supports, you might not allow editing a live marketing campaign. You might only allow editing draft campaigns. Or, for live campaigns, you might allow editing a “draft version” of the updates that won’t be published automatically, without a solid confirmation.

Certain things you never want to happen won’t have tools at all. And even when you do, they can have strict pre-conditions and post-conditions, as well as user interventions at the right places (which the core LLM does not need to know about).

Conclusion

In conclusion, here’s how you can build an agentic system that could last:

1. Decide on a harness. Make it flexible.

2. Then design the right tools and think ahead about how those tools can work together to satisfy many of your use cases. This approach, instead of rigid workflows, scales better.

3. The system prompt must help the LLM navigate the tools well while still aiming for the objective. We’re aiming for the right workflows to emerge as behavior.

4. Note that you can always have a secondary LLM call, which has its own context/history, as another tool call.

5. Ensure guardrails in the tool design itself, the types of tools available, and the confirmation logic within those tools. You can also have clever heuristics within those tools to ensure that not everything needs confirmation (which could annoy users), but certain patterns are flagged the same way an antivirus works.

Once you have the harness, the project can sustain months or years of development focused on product depth, rather than having to revise your core framework each time a new model comes along.

See you in Issue 7!