dukehoff-decide 0.3.0: two classifiers decide more than one
dukehoff-decide 0.3.0 adds a classifier on sentence embeddings and a cascade: the cheap classifier decides what it is sure about, the next one only sees the rest. Together they decide 9 to 10 points more cases automatically, at the same threshold.
dukehoff-decide turns a multiple-choice question in a Java application into a type, returns calibrated probabilities and hands uncertain cases to a person (the introduction). It started with a small language model; 0.2.0 added a classifier you train on your own labelled cases, which decides in under a millisecond.
The gate is what makes this useful: above a confidence threshold the application acts on its own, below it a person decides. So the number that matters is how many cases end up above the threshold while the decisions there stay right. Version 0.3.0 raises that number by combining backends.
A second classifier that reads meaning
The new module dukehoff-decide-embedding contains EmbeddingClassifier. A frozen embedding model (bge-m3, for
example) turns each text into a vector in which texts with similar meaning lie close together, even when they share
no words, and a logistic regression is trained on those vectors. It needs an embedding server such as llama.cpp
and takes 20 to 90 ms per decision. On long texts written by people, like consumer complaints, it is the most
accurate backend we have measured.
Embedder embedder = OpenAiEmbedder.builder() // any OpenAI-compatible /v1/embeddings endpoint
.baseUrl("http://localhost:8081/v1")
.model("bge-m3")
.modelId("bge-m3-q8_0")
.build();
EmbeddingClassifier model = EmbeddingClassifier.train(embedder, Triage.class, labeled);
The cascade
Word counts and embeddings are often unsure about different cases. That is the whole idea of the cascade: the cheap classifier decides what it is sure about, and the next one only sees the rest.
Cascade<Triage> triage = Cascade.of(
linearTask.requireConfidence(0.9), // each task calibrated on its own
embeddingTask.requireConfidence(0.9));
Outcome<Triage> outcome = triage.decide(state);
List<Outcome<Triage>> all = triage.decideAll(inbox); // each stage gets the open cases in one batch
Each stage is calibrated and gated on its own, so the threshold means the same thing at every stage. A case
neither stage is sure about comes back as Uncertain, as before. A language model fits as the last stage, for the
few cases both classifiers leave. There is also Ensemble, which averages the parts into one backend; it is
slightly more accurate at a given threshold, but the cascade decides more and is cheaper.
What it gains
Share of cases decided automatically at threshold 0.9, with the accuracy of those decisions in brackets. Every number is a mean over five runs, each with its own calibration, test and training examples:
| word counts | embeddings | cascade | |
|---|---|---|---|
| consumer complaints (10 products) | 44.2% (96.9%) | 40.5% (97.3%) | 54.3% (96.3%) |
| voice assistant, English (18 scenarios) | 77.1% (97.2%) | 74.2% (97.9%) | 85.9% (96.7%) |
| voice assistant, German | 73.5% (97.4%) | 70.3% (97.1%) | 82.1% (96.1%) |
The cascade decides 8.6 to 10.1 points more than word counts alone, at 0.5 to 1.3 points lower accuracy. It never did worse in a single run. And it is cheap: the embedding is only computed for the cases the word counts are unsure about, 56% of the complaints and a quarter of the voice requests.
The numbers in the README are now all means over five runs. With 300 test cases a single run moves by several points, and our first single run had been a favourable one.
Also new
- Calibration now chooses between temperature and temperature plus per-option bias by cross-validation on the training cases, so the holdout set stays untouched and its numbers are honest.
decideAllworks on gated tasks too, so backends that batch, like embedding requests, are called once per batch.- dukehoff-decide has a project page: dukehoff-gmbh.github.io/dukehoff-decide.
0.3.0 is source and binary compatible with 0.2.0.
Get it
<dependency>
<groupId>com.dukehoff.decide</groupId>
<artifactId>dukehoff-decide-embedding</artifactId>
<version>0.3.0</version>
</dependency>
Cascade and Ensemble are in dukehoff-decide-core; LinearClassifier is in dukehoff-decide-classifier.
dukehoff-decide is open source under the Apache License 2.0 and needs Java 21. Getting started and the
limitations are in the README on GitHub.
If you have a decision in your application that people make by hand today, and a history of how they made it, get in touch.