dukehoff-decide 0.2.0: decisions in under a millisecond with a trained classifier

dukehoff-decide turns a multiple-choice question in a Java application into a type. It answers the question with a small local language model, returns calibrated probabilities and hands uncertain cases to a person (the introduction). A language model is the right way to start: it needs nothing but the question, the option descriptions and a few hundred labelled examples for calibration.

An application that has run for a while has something better: thousands of cases where a person made the decision. Version 0.2.0 puts them to use. The new module dukehoff-decide-classifier contains LinearClassifier, a classifier you train on your own labelled cases. It plugs in where the language model was, so calibration, the gate and the report work unchanged, and so does the code that uses the decisions. Thanks to Ondřej Tulach for the idea of adding a linear classifier.

What it looks like

LinearClassifier model = LinearClassifier.train(Triage.class, labeled);
model.save(Path.of("src/main/resources/decide/triage.model"));

DecisionTask<Triage> raw = Decider.builder().backend(model).build().task(Triage.class);
profiles.save(Calibration.fit(raw, train, holdout).profile());   // cases the model was not trained on

At runtime, LinearClassifier.load(path) returns the backend. With Spring Boot, declare it as a ScoringBackend bean and the starter uses it instead of the language model.

Under the hood it is deliberately simple: word and character n-grams (TF-IDF) and one logistic regression per field. It has no dependencies besides dukehoff-decide-core and needs no model server.

How fast

Training on 20,000 texts takes about a minute. Deciding takes well under a millisecond. Measured on an M1 Pro:

The language model times are wall time over the whole run, with four requests in flight.

How accurate

That depends on how many labels you have. We compared both approaches on four public datasets, each with 300 test examples. The language models were calibrated, and the classifier was trained on examples that are in neither the calibration nor the test set.

• Customer messages (11 categories): the classifier gets 100% with 20,000 training examples, the calibrated models 87.0% (1.5B) and 91.3% (7B). Even with every training message removed that closely resembles a calibration or test message, it reaches 99% from 1,000 examples. This dataset names its categories with a few words ("invoice", "refund"), which is easy for word counts.
• Voice assistant requests (18 scenarios, often only implied): with the 300 labels that calibration needs anyway, the language models are clearly better. The classifier catches up with the 1.5B model at about 1,000 examples and with the 7B model at about 3,000. With all of them (about 15,900) it gets 88.3% in English and 85.0% in German, against 83.0% and 78.3% for the 7B model.
• Consumer complaints (10 financial products, written by real consumers, the longest texts): 78.0% against 74.3% for the 7B model. At a threshold of 0.9 the gate decides half of the complaints on its own, 97.3% of them correctly. With the 7B model it is 23%.

On a fourth dataset, where queue and priority mostly cannot be inferred from the text, the classifier does better than the models, but it is not good enough to route whole tickets automatically either. The calibration report says so. Details and how to rerun everything are in the README.

When to switch

Start with a language model. Its confident decisions are useful from day one, and the uncertain cases that people handle produce exactly the labels a classifier needs. Once there are a few thousand, train one, calibrate it on cases it has not seen, and compare the reports.

One difference to keep in mind: a language model reads the question and the option descriptions, so sharpening a description can make it more accurate. The classifier learns only from the labelled examples and never sees those texts. Rewording a question changes nothing for it, and adding an option means retraining. It refuses to score a task whose options no longer match the ones it was trained on.

Get it

<dependency>
    <groupId>com.dukehoff.decide</groupId>
    <artifactId>dukehoff-decide-classifier</artifactId>
    <version>0.2.0</version>
</dependency>

dukehoff-decide is open source under the Apache License 2.0 and needs Java 21. Getting started and the limitations are in the README on GitHub.

If you have a decision in your application that people make by hand today, and a history of how they made it, get in touch.

← All posts