Last Thursday, we explored the differences between LLMClient, PromptExecutor, and AIAgent in the Koog stack. Knosh uses AIAgent, mostly for its tool support.

It turns out that how you work with prompts differs depending on which of these layers you work with. You have a richer API at lower layers, mostly because AIAgent is designed to handle much of prompt management for you.

As users of coding agents, we are used to thinking of "the prompt" as being "what we tell the agent to do". So, for example:

knosh "Make me a sandwich"

(obligatory XKCD shout-out)

Here, Make me a sandwich is a prompt. It is a fairly lousy prompt, in that coding agents generally are ill-suited for meal prep, but, hey.

While that is a prompt, it is not the prompt that gets sent to the LLM. Make me a sandwich is a "user message": something you (the user) is sending to the LLM. With a one-shot agent like Knosh, that is the one and only user message. However, most LLM clients are "multi-turn", with multiple user messages interspersed with LLM replies. Plus, many LLM clients have a "system prompt": a message that is supplied by the agent's developers, that forms the first text in the prompt sent to the LLM.

You can envision this in the form of a chat transcript:

| You are a helpful coding agent.
> Make me a sandwich.
< Sorry, Dave, I can't do that. Would you like me to open the pod bay door instead?
> Um, I don't have a "pod bay door", and my name is not Dave.
< I like to think that all my users are Dave, each in their own way.
> Are you *sure* that you are coding agent?
< Well, you asked for a sandwich. I assumed that the normal "coding agent" guidance was part of some other... mythos.

Here, | represents the system prompt, > prefixes user messages, and < prefixes assistant messages (replies from the LLM).

The good news is: AIAgent handles assembling all that for you.

An available constructor parameter on AIAgent is systemPrompt, where you supply that initial message. Knosh assembles this from multiple user-supplied sources:

• AGENTS.md files
• The agent definition Markdown file
• The particular command that you are running, it it is anything other than the (now default) prompt command

To send a user message to the LLM, you simply call run() on the AIAgent, and it attaches that to the system prompt and sends the whole thing to the LLM. For a one-shot (single-turn) agent harness like Knosh, you just get back the LLM's response.

For a multi-turn agent, AIAgent will keep track of the LLM's response and add it to the growing transcript that represents the overall prompt. If the LLM's response represents a tool call, AIAgent will invoke the tool, attach that result to the overall prompt, and send the whole thing back to the LLM.

However, this leads to two challenges:

1. If the first user message is supplied to run(), how does AIAgent get further messages from the user?

2. What happens if the prompt gets large?

Knosh neatly avoids those by having only one turn. This is good, because while dealing the large prompts via Koog's history compression is reasonable, Koog's callback-centric approach to future user messages is awkward IMHO. I might talk about that awkwardness in some future issue.

One way to avoid that awkwardness is to not use AIAgent, and instead use PromptExecutor. Turn management becomes straightforward:

• You call execute() to send a prompt to the LLM, and the return value is what the LLM sent back
• You append that to the overall prompt
• You get the next user message from the user, and append that to the overall prompt
• You call execute() to send the revised prompt to the LLM, and the return value is what the LLM sent back
• Lather, rinse, repeat

Koog has a prompt DSL to assemble prompts from various pieces (system prompts, user messages, assistant responses). You wind up with a Prompt that represents the whole exchange between user and assistant, which you pass to execute():

val response = promptExecutor.execute(
    prompt = prompt("demo") { user("Summarize this.") },
    model = OpenAIModels.Chat.GPT4o
)

(the above code snippet comes courtesy of the Koog documentation)

What you lose is tool call support. Technically speaking, you can blend that in — AIAgent does that, using an underlying PromptExecutor — but that's going to be a fair bit of work.

In fact, more details about "that fair bit of work" is what I plan to talk about next week: how you set up tools and how they tie into this whole prompt thing.