Am I hearing this right, that they made a decision model based on Typesafe's new paradigm, and actually made a model better than Jev based on Typesafe's own ranking?
The weights have permissive licensing, but the data and training pipeline are not published to reproduce them from their proprietary Qwen starting points. Weights are not "source."
I wonder if a good usecase for this would be cloudflare's WAF rules. Give broader request context to the decider and let it pick type of challenge/block traffic directly.
Can someone explain how so many folks managed to build decision models within days or weeks after Typesafe came out with Jev? Is this concept of decision models been in the works for a while? Is it easy to copy?
Smaller models have been able to do these sorts of tasks, but a little slower, for a while now. Give a small Qwen 3.8 model a classification task and force a structured output, and it'll do a good job. I've used Qwen 0.8b for basic image classification in <500ms on my local machine for a while now.
There are a few technical details that can reduce the latency significantly (covered in the post) but the real insight has been from watching the reaction to Jev and seeing that there's enough of a market interest to offer it as a distinct thing. The underlying concept/approach was already there.
Not just structured output. Dropping down to logprobs, prompting the model to emit one word as the answer, and then ranking the output tokens to pick your answer works great on small Qwen & Gemma models.
The fascinating part to me is that Jev seems like this technique plus post-training to get multiple independent confidence values for each possible answer.
Transformers output a set of probabilities over outputs. For ChatGPT etc, those are predictions of what the next token will be. But it can also be a structured list of options or classes. Jev mostly innovated on the interface, API, and product concept around this, and made it click for a large number of people. Unfortunately for Jev, it's very easy to copy an API, and any pretrained LLM can be adapted to work in this way.
I think Jev also innovated on data & algorithms, but it remains to be seen if it's enough to be meaningfully better than traditional LLMs + a few tweaks.
Jev created accessible/programmatic ergonomics around a general purpose classifiers, and did it very well; ie intuitive api and structured data approach.
Anyone can copy that and apply to an array of models - stripped down LLMs or already slim/highly performant traditional classification architectures (just wrap inference with an api that inputs/outputs the same structured data).
Jev, I think, would say their advantage is the intelligence of their models and training data including calibration: https://medium.com/code-applied/calibrated-classifiers-makin... (which i still struggle with in the general application... there's no free lunch with these things).
What's new is "smart" decision models than you can supposedly use on anything without additional training.
If you have a very narrow use case you can train a BERT based decision model on a laptop an hour if you have good data to train it on. It'll answer faster than the roundtrip to clef/jev and use <1gb memory
You can use already trained large transformer models to make one, so it doesn't require the kind of high-scale compute, high quality data, data cleanup, reinforcement, and so on training that say an LLM does.
You just have to fine tune an LLM like Qwen on some synthetic data to do so. There was even someone that had a model that was exactly like Typesafe and published their work a year before Jev (but wasn't marketed as heavily since it was academic).
It's not a new concept, it just took someone adding on to the approach and refining it. I never deep dove it, but I assume JEV is sort of like how Sora works? They had a blog post about how it has a sort of tiny LLM, which OpenAI's small LLMs are insanely good and well defined. I think any lab tackling this with a from-scratch model could yield affordable alternatives that are highly competitive.
It seems insanely obvious at least to me, that JEV is the new hot thing for the AI field since they give you stronger output that isn't... flat out wrong, that alone is impressive.
They are not too difficult to train if you already have infra to train regular LLMs. You can typically replace a few layers train them alone and you're off to the races.
Getting training data that works well for calibrated classification objectives is difficult.
I hear conflicting opinions (including my own) about how well calibrated each of these are. Jev seems to be the best.
But the jev release made obvious the PMF for these models, and the underlying reality is that calibration really doesn't matter much when you're replacing usecases where people were using damn LM head softmax probabilities before, which are nowhere near calibrated.
So now everyone simply finetunes qwen and makes a compared-to-regular-LLM vastly cheaper decision model. And it works for majority of usecases. People mostly only care about accuracy, not confidence.
Wow, Cloudflare is definitely buying some goodwill from me. Just consistently interesting new releases alongside and solid products at great prices.
Seems nearly too good to be true.
And it's only been a few weeks.
The weights have permissive licensing, but the data and training pipeline are not published to reproduce them from their proprietary Qwen starting points. Weights are not "source."
<sad trombone sound>
Or are companies/people already building this based on say an arXiv docs? n
---
The pricing is ... hm more expensive but not at the point I won't give it a try due to the embeded vision encoding
https://www.youtube.com/watch?v=AzxoU7kxjig
Latency won't be that good, but could still work similarly. Simply force the structured output of a LLM to the given schema.
Probably also easy to train because we can use stronget LLMs to generate input/output data, or even synthetic data is easy to generate.
It's not really a new technology, it's more like a new use-case.
Perhaps that may be too costly atm
There are a few technical details that can reduce the latency significantly (covered in the post) but the real insight has been from watching the reaction to Jev and seeing that there's enough of a market interest to offer it as a distinct thing. The underlying concept/approach was already there.
The fascinating part to me is that Jev seems like this technique plus post-training to get multiple independent confidence values for each possible answer.
Anyone can copy that and apply to an array of models - stripped down LLMs or already slim/highly performant traditional classification architectures (just wrap inference with an api that inputs/outputs the same structured data).
Jev, I think, would say their advantage is the intelligence of their models and training data including calibration: https://medium.com/code-applied/calibrated-classifiers-makin... (which i still struggle with in the general application... there's no free lunch with these things).
If you have a very narrow use case you can train a BERT based decision model on a laptop an hour if you have good data to train it on. It'll answer faster than the roundtrip to clef/jev and use <1gb memory
Many people seem to have run into the same question and started working out the answer.
It seems insanely obvious at least to me, that JEV is the new hot thing for the AI field since they give you stronger output that isn't... flat out wrong, that alone is impressive.
Getting training data that works well for calibrated classification objectives is difficult.
I hear conflicting opinions (including my own) about how well calibrated each of these are. Jev seems to be the best.
But the jev release made obvious the PMF for these models, and the underlying reality is that calibration really doesn't matter much when you're replacing usecases where people were using damn LM head softmax probabilities before, which are nowhere near calibrated.
So now everyone simply finetunes qwen and makes a compared-to-regular-LLM vastly cheaper decision model. And it works for majority of usecases. People mostly only care about accuracy, not confidence.