This is cool. I definitely think the "micro" sized LLM space is underappreciated, so it's always good to see work like this. I foresee a paradigm in some contexts where you have a hierarchy of LLMs, with more competent models actively training smaller models to solve specific tasks very efficiently, and something like this could be the smallest layer in that stack.
With that being said, the web demo is not particularly impressive. It really doesn't like anything I throw at it. I'm fine with accepting that fine-tuning is the solution to this, but I wonder if there's anything to gain from a bigger model? I know it's completely counter to the whole point of this, but a 14MB binary using 28MB of RAM seems unnecessarily small and pretty arbitrary.
Like, what does a 28MB binary get you? Or a 140MB binary? Or a 1.4MB binary? I'm guessing the choice of 14MB came from minimizing the size as much as possible while meeting certain requirements/performance expectations, but even a Pi 5 has plenty more room to spare. Curious if there's a good explanation for this (which I may have missed in my skim of the post).
So, its not a general language model, focused on tool call strictly for tiny edge-devices. There are solutions everywhere for high-capacity devices, Needle is for sub-$200 devices.
Funny result from the web demo. I'm well aware that it's an extremely small and, well, stupid, model, but even so:
Query: HN
Result:
{
"function_calls": [
{
"name": "lock_door",
"arguments": {
"door": "front door"
}
}
],
"reasoning": "User wants to lock the door. No specific door mentioned, so use 'front door' as default.",
"confidence": 0
}
I'd expect it to at least ignore (call no tools) for the queries that it doesn't understand. And it seems like it does do that, just not consistently.
The website says the model is for "tool calling, device use, and structured extraction". Your example just doesn't seem to be very relevant. FWIW, it did a pretty good job for tool calling when I tried it, and I think it could be pretty nice to have this running on locally and integrate with Home Assistant.
False positives are definitely relevant and worth measuring - natural language interfaces always have a discoverability problem, i.e., users not knowing what actions the system does and does not support. If the frontend of that system lacks the ability to reject unsupported commands, weird stuff happens.
Nonetheless, this is very cool work! If I can offer a small suggestion to the team at Cactus, it would be to evaluate your releases on some usability criteria (including false positives). Any serious integrator or adopter of these models would want to have that information available.
Right, but that's not the same thing as reporting a benchmark across a test set. It doesn't help me determine how well the model does across a decently-large sample size of commands. It doesn't tell me with what reliability the confidence will be below a given threshold when it should be, above that threshold when it should be, etc.
That's really cool - I was already thinking of compressing `functiongemma-270m-it` down to 1-2 bits so it would work flawlessly in the browser.
Your `Fine-tuning` feature is even much more convenient.
While most of the industry focuses on the frontier of “intelligence” (function), a release like this represents the frontier of the other end of the spectrum (form).
Both are important if we ever want to see “Opus-level” capability running locally on commodity machines in the future.
I wonder if there's any way to get this to plan out a dag of tool calls? i.e. use the results from earlier calls as the parameters to later ones? I tried introducing a stack based system, but gave up pretty quickly.
Yes, though for better results in production, after creating your tool json, use the provided data synthesis and fine-tuning pipeline. It tunes on on your mac.
Try the following tool description: "Calculate the sum of two numbers. Use for any arithmetic or math question." instead of "Add two numbers." Let me know how it goes, thanks!
import needle
@needle.tool
def add(a: int, b: int):
"Calculate the sum of two numbers. Use for any arithmetic or math question."
return a + b
agent = needle.Needle(tools=[add])
print(agent.run("what is 5 + 7?")["reasoning"])
>> No calculator or math tool available. Cannot compute numbers.
Ah, another failure point on our end! So a simple "5 + 7" and "add 5 and 7" works. But to handle ambiguity, the python package ships pipelines to synthesize augmentations and fine-tune on your samples for robustness. Just run "needle playground" and use the UI. Apologies.
This is really cool, I'm curious how much knowledge can their be in smaller models? It seems the current trade off is you need sizeably larger models for more performance but I'm curious if in your work how far this is true, as edge ai is really what needs to get better before physical ai can take off (my two cents).
I imagine at such a low parameter count, there would be little to no world knowledge whatsoever, and the entire focus is on getting the structure of tool calling etc. right…?
But yeah, in terms of “physical” AI, robotics definitely comes to mind for me as well, where tool calls/structured “device” use in a “realtime”/edge application are highly beneficial (if you wanted to go with LLMs), but beefy hardware can’t be easily used.
Makes me think of the demo from some time ago where someone got a ~29M parameter model running on an esp32. I wonder what kind of throughput this could get if a handful of esp32s were strung together...
Edit: I have a pile of d1 minis, but not much time.
They mention that this specific model is able to run on an ESP32-S3, or an ESP32-P4 which has 32MB of PSRAM. I'm trying to figure out how to do this now.
Was really cool to see yous use Engrams to cut down compute!
Given its basically an O(1) lookup with disk space being the main constraint, I was curious if you've tried ablating engram layers and sizes across your setup?
That will get you a lot further than what you're asking, but if you dig a bit through Home Assistant features, resources, etc., you may find the current "best" answers to your questions.
If you want a quick answer: Whisper is a good open-source speech-to-text model which comes in a variety of sizes (https://huggingface.co/openai/whisper-tiny). You can definitely get something like this running on a Pi 5. There are plenty of other STT models out there, some of which are built specifically for this context (again, see the Home Assistant stuff), but Whisper comes up a lot as a good default choice.
So with something like Whisper, you could just have a simple script which is constantly listening to a rolling window of audio and transcribing it. When the transcription includes a key phrase, you can pass the rest of the transcription to Needle2 (or anything else for that matter). From there, you take the results and execute the necessary tool calls.
There's a bit more to all of this to make it work smoothly, but fundamentally this is all there is to it. All this would work very fast on a Pi 5 (although I wouldn't expect the results to be particularly good without some serious hand-crafted logic, fine-tuning, etc.). If you want to mess around this stuff, handing all of this to Claude, Codex, etc., can get you something spun up and functional very quickly.
{
"function_calls": [
{
"name": "set_thermostat",
"arguments": {
"temperature": 72,
"mode": "cool",
"room": "living room"
}
}
],
"reasoning": "'as dark as possible' -> set_thermostat to warm; 'dark' implies higher temperature; 'cool' mode for darkness.",
"confidence": 0
}
... maybe this counts as dark humor at least.
Since it seems limited to matching a few templates and otherwise falling flat on its face, I wonder how 14MB of regexes would fare in its stead. Normally you wouldn't want to parse arbitrary natural language input with regex because of how tedious and brittle it would be, but for the tedium we have LLMs and this alternative isn't exactly robust either.
The learned confidence gate is the crucial piece for a 14MB action model. On ambiguous requests such as the HN example, what calibration target decides between abstaining locally and escalating to the cloud?
around +60% confidence threshold is cool from experiments, the problem is that you gotta test on your own workload, no existing benchmark could honestly paint the full picture, so we exposed the confidence threshold for everyone.
With that being said, the web demo is not particularly impressive. It really doesn't like anything I throw at it. I'm fine with accepting that fine-tuning is the solution to this, but I wonder if there's anything to gain from a bigger model? I know it's completely counter to the whole point of this, but a 14MB binary using 28MB of RAM seems unnecessarily small and pretty arbitrary.
Like, what does a 28MB binary get you? Or a 140MB binary? Or a 1.4MB binary? I'm guessing the choice of 14MB came from minimizing the size as much as possible while meeting certain requirements/performance expectations, but even a Pi 5 has plenty more room to spare. Curious if there's a good explanation for this (which I may have missed in my skim of the post).
Query: HN
Result:
{ "function_calls": [ { "name": "lock_door", "arguments": { "door": "front door" } } ], "reasoning": "User wants to lock the door. No specific door mentioned, so use 'front door' as default.", "confidence": 0 }
I'd expect it to at least ignore (call no tools) for the queries that it doesn't understand. And it seems like it does do that, just not consistently.
Nonetheless, this is very cool work! If I can offer a small suggestion to the team at Cactus, it would be to evaluate your releases on some usability criteria (including false positives). Any serious integrator or adopter of these models would want to have that information available.
OP and the linked page talk about the confidence score and using it as an action threshold, so it looks like an appropriate total response to me.
Do the creators take something like DeepSeek, and then delete most of the neurons to whittle down the size?
While most of the industry focuses on the frontier of “intelligence” (function), a release like this represents the frontier of the other end of the spectrum (form).
Both are important if we ever want to see “Opus-level” capability running locally on commodity machines in the future.
conclusion: completly useless
But yeah, in terms of “physical” AI, robotics definitely comes to mind for me as well, where tool calls/structured “device” use in a “realtime”/edge application are highly beneficial (if you wanted to go with LLMs), but beefy hardware can’t be easily used.
Edit: I have a pile of d1 minis, but not much time.
Given its basically an O(1) lookup with disk space being the main constraint, I was curious if you've tried ablating engram layers and sizes across your setup?
Also, why mHC over attention residuals?
The demo is super — I'm just having trouble seeing the whole picture for e.g. a screenless device.
ETA: pun not intended
That will get you a lot further than what you're asking, but if you dig a bit through Home Assistant features, resources, etc., you may find the current "best" answers to your questions.
If you want a quick answer: Whisper is a good open-source speech-to-text model which comes in a variety of sizes (https://huggingface.co/openai/whisper-tiny). You can definitely get something like this running on a Pi 5. There are plenty of other STT models out there, some of which are built specifically for this context (again, see the Home Assistant stuff), but Whisper comes up a lot as a good default choice.
So with something like Whisper, you could just have a simple script which is constantly listening to a rolling window of audio and transcribing it. When the transcription includes a key phrase, you can pass the rest of the transcription to Needle2 (or anything else for that matter). From there, you take the results and execute the necessary tool calls.
There's a bit more to all of this to make it work smoothly, but fundamentally this is all there is to it. All this would work very fast on a Pi 5 (although I wouldn't expect the results to be particularly good without some serious hand-crafted logic, fine-tuning, etc.). If you want to mess around this stuff, handing all of this to Claude, Codex, etc., can get you something spun up and functional very quickly.
In the meantime, if you have enough RAM for the current model (≈28MB), our repo will get you up & running:
https://github.com/cactus-compute/needle
Since it seems limited to matching a few templates and otherwise falling flat on its face, I wonder how 14MB of regexes would fare in its stead. Normally you wouldn't want to parse arbitrary natural language input with regex because of how tedious and brittle it would be, but for the tedium we have LLMs and this alternative isn't exactly robust either.