As we have seen in previous "Thursdays with Koog" issues, Knosh uses Koog's AIAgent to send a prompt and get a final response, with the AIAgent framework handling tool calls. However, especially for a local Ollama model, getting a response may take some time. Not only are locally-hosted LLMs usually slower than top-tier frontier models, Ollama might have to load the model into memory, if it has not been used in a while. So, we execute() our prompt and we wait and we wait and we wait and we wait and we wait and we wait and we finally get a response. This makes for a fairly lousy UX.

Many TUI coding agent harnesses do something to indicate that the harness is in communication with the LLM and is not just stuck. One popular pattern is to use Braille dots to simulate a loading state. I wanted to add that to Knosh, with the caveat that I wanted the animation to advance as messages went back and forth between Knosh and the LLM. Just having the animation animate based on the passage of time would provide limited benefit for the user.

Sometimes, you can get a coding agent harness to display the "thinking" messages.

As a user, you tend to think of your interaction with the agent as being a message that you send out ("make the button 50% larger and a slightly darker shade of purple") and a message that you get back. Sometimes, you get multiple messages displayed before control fully returns to you, with thinking periods interspersed.

What is really happening during those thinking periods is that the LLM is talking to itself.

One of the techniques used to get better results out of LLMs is encouraging them to talk through what they are doing. Their generated output then becomes input in an updated prompt, and that helps steer them towards better solutions. This is especially true in planning situations, as a plan will evolve over many such "thinking" exchanges.

This approach was not part of the original APIs created by OpenAI and others. To accommodate this, especially with arbitrary LLM clients, they added notation to indicate "thinking" messages (not designed to be shown to users), much in the way that there is notation to identify tool call requests. And, they added "streaming" mode as a way of getting many messages back from a single REST call, with the call ending when the LLM indicates that it is done with its turn and control returns to the user.

Koog supports both simple request-and-response and streaming modes. My original vision was to have Knosh use the streaming mode, and update the Braille dots animation with each message. That was going to be just complicated enough that I elected to skip that and stick with the simple request-and-response approach. Besides, not all models support streaming, especially in the local-model space, so I did not want to rely upon it.

What I could do, though, was find out about tool calls.

The Koog API supports the notion of "installing" features into the AIAgent. These features hook into the internal pipeline that AIAgent uses for processing inbound and outbound messages. You get an installFeatures DSL when you construct the AIAgent as a trailing lambda expression, and in there you can call install() to install these optional features:

  return AIAgent(
    // lots of parameters go here
    ) {
      if (onMetrics != null) {
        install(MetricsFeature.Feature) { callback = onMetrics }
      }

      if (onStatus != null) {
        install(StatusFeature.Feature) { callback = onStatus }
      }
    }
  }

(from Knosh 0.3.0)

The StatusFeature lets you register a callback for AgentStatus events. Specifically, you can find out about Starting, CallingModel, and RunningTool events. The Knosh CLI uses that to update a StatusLine, advancing the Braille-dots animation on each status change, while also printing a message about what the status change is:

/**
 * Returns the human-readable label shown on the status line for this [AgentStatus]. The label carries no spinner frame.
 *
 * @return the label text
 * @receiver the status to describe
 */
internal fun AgentStatus.label(): String =
  when (this) {
    AgentStatus.Starting -> "starting…"
    AgentStatus.CallingModel -> "calling model…"
    is AgentStatus.RunningTool -> "running tool: $toolName"
  }

/**
 * A single-line, in-place terminal status indicator rendered to stderr.
 *
 * Each call to [update] advances an 8-frame Braille spinner by one position (cycling back to the start after the eighth
 * frame) and rewrites the line as a carriage return, the frame, the label, then a clear-to-end-of-line sequence, with
 * no trailing newline, so successive updates overwrite one another. [clear] erases the line. The spinner advances
 * independently of the label: the frame is purely a function of how many updates have occurred.
 *
 * Rendering is dependency-free raw ANSI; callers must construct a [StatusLine] only when stderr is an interactive
 * terminal.
 *
 * @param write the sink that receives raw output fragments; defaults to writing to and flushing `System.err`
 */
internal class StatusLine(
  private val write: (String) -> Unit = { fragment ->
    System.err.print(fragment)
    System.err.flush()
  }
) {
  private var frameIndex = 0

  /**
   * Advances the spinner by one frame and rewrites the status line to show [status].
   *
   * @param status the current agent status to display
   */
  fun update(status: AgentStatus) {
    val frame = SPINNER_FRAMES[frameIndex % SPINNER_FRAMES.size]

    frameIndex++
    write("\r$frame ${status.label()}$CLEAR_TO_END_OF_LINE")
  }

  /** Erases the status line, leaving the cursor at the start of an empty line. */
  fun clear() {
    write("\r$CLEAR_TO_END_OF_LINE")
  }
}

(from Knosh 0.3.0)

The other feature that Knosh uses is MetricsFeature. This registers a callback to be invoked when the full request-and-response has completed. You can find out:

• How many LLM calls were made
• How many tool calls were requested
• How many input tokens were sent to the LLM
• How many output tokens were received from the LLM

The first two can be tracked directly by Koog. The latter two require the LLM to supply that information as part of the REST API, and that is not always supported. If we get that data, Knosh will include it in the output, so you get a sense for how expensive that particular request was.

If you pull up Knosh in your favorite IDE and try looking up this Feature system, you will see that it appears to be tied to a GraphAIAgent, a specific implementation of the AIAgent abstract class. Next week, we will learn a bit about Koog's strategy system and what strategy Knosh employs.