Dukehoff Blog

dukehoff-decide: typed, calibrated decisions from small language models

2026-10-23T00:00:00Z

Many LLM calls in business applications ask a question whose answer is one of a few options. Which team handles this ticket? Is this message spam? How urgent is it? The usual way is to send the question to a large model, ask for JSON, parse the text and hope the answer is one of the allowed values. And when the model says it is sure, there is no way to tell whether that means anything.

dukehoff-decide turns such a question into a Java type and answers it with a small model that runs locally. You get a valid value of that type, a probability for every option, and a gate that decides on its own only where those probabilities are reliable. It is open source under the Apache License 2.0, and version 0.1.0 is on Maven Central.

What it looks like

@Question("Triage an incoming customer support ticket")
record Triage(
    @Question("How urgent is this ticket?") Priority priority,
    @Question("Which team should handle it?") Team team,
    @Question("Does the customer threaten to cancel?") boolean churnRisk) {}

Decision<Triage> d = triage.decide(State.of(ticketText));
d.value();                      // always a valid Triage
d.distribution("priority");     // Map<Priority, Double>

switch (triage.requireConfidence(0.9).decide(state)) {
    case Outcome.Confident<Triage>(var dec)          -> route(dec.value());
    case Outcome.Uncertain<Triage>(var dec, var why) -> sendToHuman(state, dec, why);
}

Every field becomes a question with a fixed set of options: a boolean is yes or no, an enum has one option per constant, a scale has one option per step. The model answers each question with a single token, and dukehoff-decide reads the probabilities of all options from the model's log-probabilities. There is no text to parse, so there is no invalid answer.

Why calibration matters

Raw model probabilities are usually overconfident, and small models often favour particular options. So you calibrate once, offline, with a few hundred historical cases whose answer you know, for example tickets and the team that finally handled them. dukehoff-decide fits a correction per field and writes a report: accuracy before and after, how many cases a gate at each threshold would decide automatically, how accurate those are, and which options get confused.

The report is worth reading. A warning such as "option CONTACT receives 16 of 30 errors" points to an option whose description matches too much. In our tests, rewording one such description raised accuracy from 76% to 88–90%, more than a model 4.7 times as large did.

What we measured

Customer messages from a public dataset, 11 categories, Qwen2.5 1.5B (4-bit) with llama.cpp, 300 examples for calibration and 300 different ones for the test:

calibration	before	after
category correct	87.7%	93.7%
decided automatically (threshold 0.9)	50%	71%
of those correct	94.0%	99.5%

Always guessing the most frequent category would be right 21.3% of the time.

The second test matters as much. On a dataset of support tickets where queue and priority mostly cannot be inferred from the text, the models still claimed over 99% confidence. Raw scores would have routed 42% to 65% of the tickets automatically, with under 10% of them fully correct. After calibration, none were routed automatically. That is the right answer: the gate knows when the model doesn't.

Details, both datasets and how to rerun them are in the README.

Get it

You need Java 21 and a server with an OpenAI-compatible API that returns log-probabilities. We verified it with Ollama and llama.cpp.

<dependency>
    <groupId>com.dukehoff.decide</groupId>
    <artifactId>dukehoff-decide-openai</artifactId>
    <version>0.1.0</version>
</dependency>

For Spring Boot, use dukehoff-decide-spring-boot-starter instead. It creates a DecisionTask bean per type and comes with a health check and Micrometer metrics. Getting started, the Spring Boot setup and the limitations are in the README on GitHub.

If you want to replace calls to a large model in your application with a small one you can run yourself, or need help with the labelled examples for calibration, get in touch.

Dukehoff MCP for Java: an IDE for coding agents

2026-10-18T00:00:00Z

Imagine being a Java developer in 1996, before modern IDEs, with nothing but command-line tools and maybe Notepad. To rename a method, you search for its name, then open each of maybe 22 files to check whether a hit refers to that method or to an unrelated one with the same name, and replace each occurrence with sed or manually. You get there eventually, and once it compiles, you're probably done. It just takes a lot of steps and time.

You may be surprised, but that's pretty much how coding agents work in 2026 when you ask them to rename a method.

Without the server: Claude Code with Sonnet 5.5 renames a method in Apache Commons Lang. 32 s model time, 994 tokens.

Dukehoff MCP for Java gives the agent the analysis and refactoring operations of an IDE as tools: find usages, rename, move, change signature, safe delete, inline. The agent passes the method's signature to the rename tool, and the server finds the method, renames it everywhere, overrides included, writes the files, and returns the diff plus a compile check that names any new errors.

With the server: the same task. 10 s model time, 230 tokens.

Some Java IDEs have integrated tools like this, but the way we develop software has changed. Many developers now start an agent in a terminal, let it write the code, and review the result in the pull request. Dukehoff MCP for Java runs headless, so your agent doesn't need a running IDE to use these tools. And when you do work in an IDE, the same tools are there too: in VS Code next to Copilot, or in a running NetBeans.

Be nice to your agent and give it the right tools for the tedious, repetitive and error-prone tasks. The server is open source, and for exactly these tasks it saves you and your agent turns, tokens and time.

What it saves

−82%turns: Rename a method used in 22 files9 turns instead of 51, $0.22 instead of $0.56.NetBeans, java/ folder, Opus 5.5
−91%cost: Find the callers of one overload91% less cost, 85% less time.Apache Commons Lang, Sonnet 5
−40%turns: Modernize 84 places among 144 candidates40% fewer turns, about 22% less cost and 39% less time than writing its own scripts.Told to use the rules, Sonnet 5.5

Each figure compares the same agent with and without the server.

These are real numbers we measured with Claude models. Find the details in eval/RESULTS.md.

We are not alone with this idea. Netflix and JetBrains recently reported that giving agents the headless code intelligence of IntelliJ IDEA made them typically 20–30% more efficient, and up to about 65% in the best case (JetBrains blog). Their tooling is proprietary: the editor extension needs an IntelliJ IDEA Ultimate subscription, and support for terminal agents is still to come. Dukehoff MCP for Java is open source, free, and works today with Claude Code, Codex, Copilot and others, in the terminal, in VS Code and in NetBeans.

Why we hit the Jackpot with NetBeans

Apache NetBeans is free and open source, and it understands Java with javac itself, the same compiler your build uses: types, overloads and overrides are resolved exactly, not approximated. It runs without a window just as well; its LSP server, which powers the Apache NetBeans extension for VS Code, does that every day. And it has a mature refactoring engine, built over more than twenty years.

Beyond that, NetBeans has a hidden gem: Jackpot, a rule-based rewrite engine. A Jackpot rule matches code by its structure and types, not by its text, so a migration changes what it should and leaves look-alikes alone. Agents can write such rules themselves, and we ship tested rule sets for common migrations: JUnit 4 to 5, Java modernization, and javax to jakarta.

Get it

Dukehoff MCP for Java is free and open source under the Apache License 2.0. Pick the way that fits how you work:

Claude Code, Codex, Copilot CLI and other terminal agents: download dukehoff-mcp-<version>.tar.gz (macOS, Linux) or .zip (Windows) from the latest release, unpack it, and let it print the registration for your agent: dukehoff-mcp --config claude (installation).
VS Code with GitHub Copilot: install Dukehoff MCP for Java from the Visual Studio Marketplace, or search for "Dukehoff MCP" in VS Code's Extensions view. It brings the Apache NetBeans extension along; the tools appear in Copilot's agent mode.
NetBeans: download dukehoff-mcp-netbeans-<version>.nbm and dukehoff-mcp-launcher-<version>.jar from the latest release, install the plugin under Tools > Plugins > Downloaded, and connect your terminal agent to the running IDE with --ide (how).

You need a JDK 17 or newer; VS Code 1.101 or newer for the extension, NetBeans 31 or newer for the plugin. To make your agent reach for the tools on its own, add the short section from Telling the agent when to use the tools to your CLAUDE.md or AGENTS.md.

If you have a migration where the meaning of a change depends on types, such as an in-house framework, a vendor library swap or a Java EE to Jakarta EE move beyond the APIs, we can help: get in touch.

Dukehoff MCP for Java runs on Apache NetBeans. It is not part of, nor endorsed by, the Apache NetBeans project or the Apache Software Foundation. "Apache", "NetBeans" and "Apache NetBeans" are trademarks of the Apache Software Foundation.